From web11.forest at tibit.com Tue Sep 24 00:14:17 2013 From: web11.forest at tibit.com (Forest) Date: Mon, 23 Sep 2013 15:14:17 -0700 Subject: [Email-SIG] Proposed Enhanced MIME Handling API In-Reply-To: <20130831033528.E7EDC2507F5@webabinitio.net> References: <20130808162608.B8E4D2500C7@webabinitio.net> <20130831033528.E7EDC2507F5@webabinitio.net> Message-ID: On Fri, 30 Aug 2013 23:35:28 -0400, R. David Murray wrote: >No response means nobody objects, right? :) Hi, David. You can count me among those who are interested but busy. :) My main problem with the existing email package is that it requires entire message parts to be held in memory, making it unusable on constrained systems like low-power personal servers. The issue concerns me right now because I'm working on some email software that targets these systems, and it takes only a single message with a half-gigabyte attachment to fill all available RAM. I believe such systems will become more widespread in the coming years, mainly due to the availability of cheap hardware and increasing privacy concerns with big webmail providers. Budget VPS hosts, mini-notebooks, smart phones, tablets, NAS servers, plug computers, and other platforms are also affected. That leaves very few classes of common hardware on which the existing library can handle messages with midsize attachments: basically just workstations and medium-to-large servers. If python's email package is going to handle today's correspondence on a wide variety of systems, I think it's going to need support for data streams and file section references, rather than hoping that there is always enough free memory to hold the messages, bodies, and MIME parts that turn up in the real world. With that in mind, your new API looks promising. I like the fact that it opens the door to content managers that could accept/produce socket-like objects, files, (filename, offset, size) tuples, iterators, etc. Of course, such things can work only if the library's parsing and flattening code can stream data to and from those objects. I remember you mentioning in the past that data streaming was a development goal. Is it a priority yet? Even if not, would this be a good time to define a streaming interface for creating/reading/writing custom content objects, so that the standard library's internal code can eventually be rewritten to use it? I'd hate to see everyone's shiny new content managers end up blocking progress on data streaming due to a design incompatibility. Forest From rdmurray at bitdance.com Tue Sep 24 03:40:41 2013 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 23 Sep 2013 21:40:41 -0400 Subject: [Email-SIG] Proposed Enhanced MIME Handling API In-Reply-To: References: <20130808162608.B8E4D2500C7@webabinitio.net> <20130831033528.E7EDC2507F5@webabinitio.net> Message-ID: <20130924014042.8BCE6250965@webabinitio.net> On Mon, 23 Sep 2013 15:14:17 -0700, Forest wrote: > On Fri, 30 Aug 2013 23:35:28 -0400, R. David Murray wrote: > >No response means nobody objects, right? :) > > Hi, David. You can count me among those who are interested but busy. :) :) > I remember you mentioning in the past that data streaming was a development > goal. Is it a priority yet? Even if not, would this be a good time to > define a streaming interface for creating/reading/writing custom content > objects, so that the standard library's internal code can eventually be > rewritten to use it? I'd hate to see everyone's shiny new content managers > end up blocking progress on data streaming due to a design incompatibility. I would say that that was the goal that is next on my list. The problem with it is that the current parser and generator do *not* support streaming (despite the parser being named 'feedparser'...you can stream data *in* to it, but you can't get the message parts to stream *out* of it...the entire message is held in memory; although that said it will be the easier of the two). Something I wanted to do but ran out of time for was to add policy hooks to implement external storage of bodies. That is probably the key piece needed for streaming, although by itself it doesn't solve the generator problem. Figuring out how to make all this work is not going to be trivial, so I'm not sure there's any way to even get an API figured out before 3.4 beta. If you've got suggestions, though, I'm all ears. --David