[issue1441530] socket read() can cause MemoryError in Windows

[Charles-François Natali]

Charles-François Natali <neologix at free.fr> added the comment:

Digging a little deeper, here's the conclusion:
- with py3k, fragmentation is less likely: the buffered reader returned by makefile() ensures that we can allocate only one result buffer for the total number of bytes read() (thanks to socket's readinto()). It's also faster. Finally, since buffered read() guarantees to return exactly the number of bytes asked, there's no need to call it repeatedly with min(size-read, 4096): it's slower, and actually makes fragmentation more likely.
So I've attached a patch removing the looping from py3k IMAP4 read() method, which leads to simpler, faster and fragmentation-less code.
- for 2.7, it's also better to wrap the SSL socket with makefile(): the leads to simpler code (no need to loop), and also I noticed that socket's _fileobject's (returned by makefile()) read() and readline() methods already use a StringIO to avoid fragmentation.
So I've attached a second patch wrapping 2.7 IMAP4_SSL socket with makefile.

While I can't reproduce this issue on my system, I've analyzed malloc/realloc/free calls using ltrace, and I'm pretty confident this should solve fragmentation issues for both 2.7 and py3k.

By the way, nice work with the I/O stack in py3k, the new layer allows much more efficient code (reduced allocations and copies)!

----------
Added file: http://bugs.python.org/file22062/imaplib_recv_py3k.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue1441530>
_______________________________________