[Python-Dev] Mailbox module - timings and functionality changes

[Steve Holden]

Guido van Rossum wrote:
> On Tue, Jun 29, 2010 at 7:49 AM, Steve Holden <steve at holdenweb.com> wrote:
>> Steve Holden wrote:
>>> Nick Coghlan wrote:
>>>> Command line: ./python -m test.regrtest -v test_mailbox
>>>>
>>>> trunk: Ran 274 tests in 25.239s
>>>> py3k: Ran 268 tests in 26.263s
>>>>
>>>> So I don't see any substantial difference on a Kubuntu 10.04 box (both
>>>> builds are recent'ish, but not completely up to date).
>>>>
>>>> However, the underlying IO access is significantly different between
>>>> POSIX and Windows, so there could still be something pathological
>>>> happening at the filesystem manipulation layer. My comparisons are
>>>> also 2.7 vs 3.2 rather than 2.6 vs 3.1.
>>>>
>>>> Cheers,
>>>> Nick.
>>>>
>>> Thanks for all the timings! If a Windows user could do the same thing
>>> that would help ...
>>>
>> And there is *definitely a performance issue. I created a Thunderbird
>> folder of 26 Google alerts and just parsed then all after reading them
>> in from the mailbox.
>>
>> 2.5 (!):  0.78 sec
>> 3.1    : 42.80 sec
>>
>> Rather than debate the code here perhaps I should just open an issue for
>> this? I can then provide both a program and some data, which can be
>> added to the tests if appropriate. The issue can clearly stand some
>> investigation.
> 
> Since you have such a great reproducible test case, could you point
> the profiler at it? (Perhaps on a reduced dataset... The profiler
> multiples your run time by some number between 2 and 10 IIRC.)
> 
Sure. I attach the outputs of both files, as well as the program and the
data. With profiling (python -m cProfile test3.py) the run took less
than a third of a second under 2.5, and 168 seconds under 3.1. I'd say
that was problematical :)

I will leave the profiler output to speak for itself, since I can find
nothing much to say about it except that there's a hell of a lot of
decoding going on inside mailbox.iterkeys().

regards
 Steve
-- 
Steve Holden           +1 571 484 6266   +1 800 494 3119
See Python Video!       http://python.mirocommunity.org/
Holden Web LLC                 http://www.holdenweb.com/
UPCOMING EVENTS:        http://holdenweb.eventbrite.com/
"All I want for my birthday is another birthday" -
                                     Ian Dury, 1942-2000
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: test3.1.out
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100629/53064c86/attachment-0004.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: test2.5.out
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100629/53064c86/attachment-0005.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: test3.py
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100629/53064c86/attachment-0006.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: test.mailbox
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100629/53064c86/attachment-0007.ksh>