[Mailman-Developers] Regarding Handlers/SMTPDirect.py and "chunkify"

[Mark Sapiro]

Stefan Förster wrote:
>
>Now, the reason this change was so effective was that the number of
>(disk) I/O operations was greatly reduced: Where a posting to a list
>with 25000 subscribers would inevitably lead to the creation of 25000
>queue files (even more if message content and control information are
>kept in different files) before the change, only 25000/500 were
>created afterwards (or "should have been", see next paragraph).
>Additionally (but this has not really that much of an influence), a
>larger number of destinations is available much faster to the MTA,
>allowing it's queue manager to make decisions in a "more educated"
>way.
>
>Since the server is still quite close to it's I/O limits, I would very
>much like to further reduce the number of I/O transactions. Therefore,
>I would like to see a configurable parameter, something like
>FLAT_RECIPIENT_CHUNKIFY: Right now, if mm_cfg.SMTP_MAX_RCPTS is not
>set <= 0, "chunkify" will always (i.e. for any real world recipient
>distribution) return more than one chunk of recipients, even if the
>total number of recipients is smaller than  mm_cfg.SMTP_MAX_RCPTS.
>With the addition of the aforementioned "tunable", we could have
>chunkify simple split recipients into batches of mm_cfg.SMTP_MAX_RCPTS
>and don't do any groupings. This way, a delivery to 25000 recipients
>would really only involve (25000/mm_cfg.SMTP_MAX_RCPTS)*(number of
>queue files per message) file creation operations.


I understand what you are saying, but I wonder what the real world
difference would be. As currently written, chunkify returns at most 4
partially filled chunks. Granted, 4 is significantly bigger than one,
but given that the MTA is VERPing the deliveries, it may ultimately
create an outgoing queue entry for each recipient anyway, so the extra
3 on the inbound side doesn't seem that significant (and it might
increase parallelism in the MTA).

Given your 25000 member list, and assuming SMTP_MAX_RCPTS = 500, you
would have at most 54 chunks (and more likely 53 or 52) instead of 50.

In any case, If I were coding this, I would be inclined to not make it
an option, but just to change chunkify so it still grouped, but
continued to fill the last chunk of a group from the next group so
there would be at most one partial chunk.

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan