append in IMAP4 from imaplib very slow

[Simon Pirschel]

On 07/26/2012 09:49 AM, Simon Pirschel wrote:
> On 07/26/2012 05:21 AM, Tim Chase wrote:
>> On 07/25/12 12:47, Simon Pirschel wrote:
>>> I'm currently experimenting with IMAP using Python 2.7.3 and
>>> IMAP4 from imaplib. I noticed the performance to be very bad. I
>>> read 5000 files from a directory and append them to an IMAP
>>> INBOX. The hole procedure of reading and appending is taking
>>> about 210 seconds.
>>>
>>> I set up the exact same code in Perl to check if there is a
>>> general IMAP server configuration issue, since CPU and I/O isn't
>>> the problem. The same amount of data on the same IMAP server is
>>> done after 7.9 seconds using Perl.
>>>
>>> The difference is huge and I tried to narrow the issue down by
>>> profiling the Python code. The profile results are, 206 seconds
>>> are spent in calling socket.recv.
>> While I don't know the ins and outs of the Perl code, it looks like
>> the Python imaplib.py is doing some pretty straight-forward
>> send/receive work.  However, IMAP4 does support both pipelining and
>> multi-append extensions, so if the Perl code takes advantage of the
>> features if the server offers them, it might be reaping significant
>> gains over the naïve Python code.
>>
>> Can you get a debugging dump of the commands being sent each way?
>> (particularly if you see a bunch of APPEND commands before the reply
>> acknowledgement, as detailed at [1])
>>
>> -tkc
>>
>> [1]
>> http://www.ietf.org/rfc/rfc3502.txt  on page 4
>>
> I took a strace of all network related sys calls from Python and Perl. 
> It seems both don't do multi appends and waiting for the OK after 
> appending one mail. But the Python version is getting a completely 
> different response.
>
> Python: "* 204 EXISTS\r\n* 1 RECENT\r\nNBGC3 OK [APPENDUID 1343283946 
> 204] Append completed.\r\n"
> Perl: "2 OK [APPENDUID 1343283946 255] Append completed.\r\n"
>
> The Python lib seems to trigger a EXISTS and RECENT lookup on every 
> append, which could be the performance difference. I couldn't figure 
> out why, yet!
>
> Python strace: http://aboutsimon.com/py.strace
> Perl strace: http://aboutsimon.com/pl.strace
>
> Strings are shortened to 40 chars.
>
>
> Simon
Ok, forget about the EXISTS and RECENT response. The server will 
response this way if you selected a mailbox, which I did in the Python 
code but not in the Perl code. I disabled selecting the mailbox in 
Python and there is no difference in the runtime. So, this isn't the issue.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20120726/7c5eaedf/attachment.html>