IMAP4 example in docs causes memory error for me (OS X 10.3, python 2.3) - help?

[Tony Meyer]

> I am running the IMAP4 example in the documentation:
> 
> http://www.python.org/doc/lib/imap4-example.html
> 
> with two small changes (different username, comment out print 
> statement). The inbox I am accessing is 42 MB and has 
> 80-someodd messages, so the messages are fairly big (all of 
> them have attachments). When I run this program, I get a 
> malloc error.

Background: I don't have access to a mac at the moment to test this there,
so this is theory based.  However, I do look after the IMAP filter script
that's part of SpamBayes, so get a fair bit of mail about that, including
from mac users.  Some of those have reported much the same error, and
googling about for information gives some, although not a lot.

The information that I found at the time indicated that this was the result
of a change introduced with OS X 10.3.  The problem always occurs at line
301 of socket.py, which is basically just retrieving the data from the
socket.  Under the hood (i.e. in the C of the socket module, around line
2024), all it's really doing is allocating memory for the amount to be
received, and receiving it.

With SpamBayes, I just wormed around the problem (i.e. I skipped those
messages).  If you do need to get hold of them, then I imagine that this
will be rather annoying to fix.  My ideas at the moment are:

 * I suspect that you could retrieve chunks of the message (with the
appropriate IMAP commands) and then put it all together afterwards.

 * You could modify imaplib (or have a subclass) that (in _getresponse())
alters the way that literals are read (i.e. change it to chunks).

 * Otherwise, particularly if you are able to run with a modified Python,
you could try fixing up socketmodule.c to do this more nicely on OS X.

I plan to look into it more at some point, but have no idea when I'll have
time.

If you do find a workable solution, please post it to the list, as it would
definitely be of interest.

> Now, I'm really new to python programming, but 
> it was my understanding that memory management was not 
> something I had to explicitly do. I'd appreciate some help in 
> understanding what I've done wrong.

FWIW, you shouldn't ever need worry about the malloc'ing with Python (unless
you're writing extensions etc).  This is a very odd case, caused by some OS
X 10.3 oddity.  I'm fairly confident that if you wrote a (correct) little C
program that asked for the same amount of memory, OS X would fail there,
too.

=Tony Meyer