append in IMAP4 from imaplib very slow

[Simon Pirschel]

On 07/26/2012 05:21 AM, Tim Chase wrote:
> On 07/25/12 12:47, Simon Pirschel wrote:
>> I'm currently experimenting with IMAP using Python 2.7.3 and
>> IMAP4 from imaplib. I noticed the performance to be very bad. I
>> read 5000 files from a directory and append them to an IMAP
>> INBOX. The hole procedure of reading and appending is taking
>> about 210 seconds.
>>
>> I set up the exact same code in Perl to check if there is a
>> general IMAP server configuration issue, since CPU and I/O isn't
>> the problem. The same amount of data on the same IMAP server is
>> done after 7.9 seconds using Perl.
>>
>> The difference is huge and I tried to narrow the issue down by
>> profiling the Python code. The profile results are, 206 seconds
>> are spent in calling socket.recv.
> While I don't know the ins and outs of the Perl code, it looks like
> the Python imaplib.py is doing some pretty straight-forward
> send/receive work.  However, IMAP4 does support both pipelining and
> multi-append extensions, so if the Perl code takes advantage of the
> features if the server offers them, it might be reaping significant
> gains over the naïve Python code.
>
> Can you get a debugging dump of the commands being sent each way?
> (particularly if you see a bunch of APPEND commands before the reply
> acknowledgement, as detailed at [1])
>
> -tkc
>
> [1]
> http://www.ietf.org/rfc/rfc3502.txt on page 4
>
I took a strace of all network related sys calls from Python and Perl. 
It seems both don't do multi appends and waiting for the OK after 
appending one mail. But the Python version is getting a completely 
different response.

Python: "* 204 EXISTS\r\n* 1 RECENT\r\nNBGC3 OK [APPENDUID 1343283946 
204] Append completed.\r\n"
Perl: "2 OK [APPENDUID 1343283946 255] Append completed.\r\n"

The Python lib seems to trigger a EXISTS and RECENT lookup on every 
append, which could be the performance difference. I couldn't figure out 
why, yet!

Python strace: http://aboutsimon.com/py.strace
Perl strace: http://aboutsimon.com/pl.strace

Strings are shortened to 40 chars.


Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20120726/c203e239/attachment.html>