[Pythonmac-SIG] Fwd: PPC OSX vs. x86 Linux

[Bob Ippolito]

On Apr 11, 2005, at 8:00 AM, Joshua Ginsberg wrote:

>> On Apr 10, 2005, at 4:14 PM, Bob Ippolito wrote:
>>
>>>
>>> On Apr 10, 2005, at 2:46 PM, Joshua Ginsberg wrote:
>>>
>>>>> I writing some python code to do some analysis of my mail logs. I 
>>>>> took a 10,000 line snippet from them (the files are about 5-6 
>>>>> million usually) to test my code with. I'm developing it on a 
>>>>> Powerbook G4 1.2GHz with 1.25GB of RAM and the Apple distributed 
>>>>> Python* and I tested my code on the 10,000 line snippet. It took 2 
>>>>> minutes and 10 seconds to process that snippet. Way too slow -- 
>>>>> I'd be looking at about 20 hours to process a single daily log 
>>>>> file.
>>>>>
>>>>> Just for fun, I copied the same code and the same log snippet to a 
>>>>> dual-proc P3 500MHz machine running Fedora Core 2* with 1GB of RAM 
>>>>> and tested it there. This machine provides web services and domain 
>>>>> control for my network, so it's moderately utilized. The same code 
>>>>> took six seconds to execute.
>>>>>
>>>>> Granted I've got the GUI and all of that bogging down my Mac. 
>>>>> However, I had nothing else fighting for CPU cycles and 700MB of 
>>>>> RAM free when my testing was done. Even still, what would account 
>>>>> for such a wide, wide, wide variation in the time required to 
>>>>> process the data file? The code is 90% regular expressions and 
>>>>> string finds.
>>>>>
>>>>> * versions are:
>>>>> Python 2.3 (#1, Sep 13 2003, 00:49:11)
>>>>> [GCC 3.3 20030304 (Apple Computer, Inc. build 1495)] on darwin
>>>>> and
>>>>> Python 2.3.3 (#1, May  7 2004, 10:31:40)
>>>>> [GCC 3.3.3 20040412 (Red Hat Linux 3.3.3-7)] on linux2
>>>
>>> Try it with a newer version of Python on Mac OS X.  I had a similar 
>>> problem, and it turned out to be Python 2.3.0's fault.  
>>> Specifically, the implementation of the datetime module's parser was 
>>> really, really, really stupid and slow in early versions of Python 
>>> 2.3.
>>>
> Well, I compiled a fresh version of Python 2.3.5 from python.org to 
> test the datetime theory... and I'm still getting 150sec execution 
> times. :-/ I'm gonna test the string vs. strop now...

Use Python's profiling tools and/or Apple's Shark to see what's slow.

-bob