[Email-SIG] Handling large emails: DiskMessage and DiskFeedParser

[Menno Smits]

Barry Warsaw wrote:

>>The only feedback I have for you is: Cool!
> 
> Indeed!  Unfortunately, I don't have time right now to look at the code,
> but I'm +1 on including a feature "like this" in email 3.0.

Great to hear.

> What I'd like you to think about is a better API for hooking in features
> like this.  I'm not sure inheritance (of either the FeedParser, but
> especially of the Message object) is the right way to go.  One reason
> for that is that many applications already pass Message subclasses to
> the parser to get some extra application-specific functionality.  This
> isn't going to mix well if they'd also like to get disk storage of big
> attachments.

I definitely agree that subclasses FeedParser isn't the way to go, 
especially with the way it stands now. I'm not convinced that 
subclassing Message is a bad idea but I'm happy to bounce around other 
ideas.

What do Message subclasses typically do?

One example I can think of (MailMan) just adds some convenience 
functions to simplify access to message headers for that particular 
application. In this case changing the app to subclass from DiskMessage 
instead of Message in order to get disk caching would work and would be 
simple.

Of course I'm probably missing some more complex subclasses of Message 
here. Any other examples?

> Perhaps we can start with some use cases and then propose a better API
> for addressing these use cases?  I'll see if I can find some time over
> the weekend to think about this.

The main things I care about from my perspective:
- message payloads are never loaded entirely in to memory
- message payloads are easily accessible for manipulation and searching
- same functionality and flexibility wrt message manipulation and
   generation as is available with current email package

Looking forward to your thoughts.

Menno

-- 
Menno Smits, Senior Development Engineer
NetBox       http://netbox.biz  |  Voice        +61 500 555 357
Oxcoda       http://oxcoda.com  |  Fax          +61 500 555 358