[Mailman-Developers] Incoming Queue format

[Brad Knowles]

At 12:23 PM -0400 9/29/06, emf wrote:

>  Furthermore, many MTAs *do* understand Maildir,

MTAs should be sticking to the job of transmitting e-mail between 
themselves.  If they're spending any time mucking about with local 
mailbox formats, they're making a mistake.  That's a job for a Local 
Delivery Agent, and not an MTA.

Now, granted many mail system packages also include one or more 
sample LDAs in the tarball, either as an official part of the system 
or in some sort of contrib/ directory somewhere, but let's make sure 
that we're using the proper terms for the proper objects.

Therefore, by definition, MTA != LDA.

>                                                  and most admins
>  do as well;

If this discussion has taught me anything, it's that after all these 
years we still have virtually no one in this business that really 
does understand Maildir or any other mailbox format, although many 
claim to do so.

>              using our own queue-on-disk format means MTAs must
>  access Mailman via LTMP, pipe invocation, or the like, and if
>  there are issues with the queue the administrator likely must
>  learn our queue-on-disk format.

Our queue-on-disk format is already much simpler than Maildir, at 
least when it comes to directory structure, and the directory hashing 
schemes that I've been talking about have been around for many years. 
No new thought needs to be put into implementing them.

I even convinced Wietse that he should implement a lot of the same 
concepts, back when I first got involved in postfix in '98, when it 
was still being called VMailer.


Now, if you want to get outside of directory structure, our 
queue-on-disk format includes a lot of things that are 
Mailman-specific, such as creating message pickles, and I don't think 
that anyone is talking about getting rid of those aspects.

>  Most of the maildir phenomena you have an issue with
>  wouldn't even arise in the use case under discussion; a
>  mail would enter maildir/new , mailman would suck it out,
>  and that would be that; renaming wouldn't occur and the
>  number of elements in the queue is unlikely to become
>  large enough to pressure filesystem indexing schemes.

You really need to go back and review exactly how messages are 
created using Maildir.


With Maildir, when a message comes in, a temporary file is opened 
with truncate (with certain measures taken to try to ensure that the 
selected filename will be unique), and if that system call succeeds, 
then the system appends the incoming message and renames the file, 
before it ever closes it.

If that creat() system call fails, then there is already a file by 
that name, and the LDA has to try again.  This is how they "safely" 
create files on NFS, with an operation that is supposed to be atomic, 
and allows them to avoid file locking.

I'd have to check, but I think there are some more synchronous 
meta-data operations in here, too.  Certainly, every time you look to 
see if more messages have come in, you have to scan the entire 
directory, and you have to stat() each and every file, and if you 
want to pick up the message and move that somewhere else, you're 
going to have to do further synchronous meta-data operations that 
involve locking the entire directory structure while they are taking 
place.

Now, your application may not see those locking, scanning, and stat 
operations, you may simply see "move this file to another location", 
but the underlying filesystem code has to do a lot of work to support 
that.  And regardless of whether you're using locking or not, you 
still have race conditions that you have to code for -- or at least 
be aware of, because they are potential areas where you may have 
problems in the future.


If you don't read the comments that Mark Crispin has written about 
the weaknesses in Maildir, and you haven't read the code to see what 
it's actually doing, then I don't see how you can participate in this 
discussion.

-- 
Brad Knowles, <brad at stop.mail-abuse.org>

"Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety."

     -- Benjamin Franklin (1706-1790), reply of the Pennsylvania
     Assembly to the Governor, November 11, 1755

  Founding Individual Sponsor of LOPSA.  See <http://www.lopsa.org/>.