Overcoming python performance penalty for multicore CPU

[John Nagle]

Steve Holden wrote:
> John Nagle wrote:
>> Paul Rubin wrote:
>>> John Nagle <nagle at animats.com> writes:
>>>> Analysis of each domain is
>>>> performed in a separate process, but each process uses multiple
>>>> threads to read process several web pages simultaneously.
>>>>
>>>>    Some of the threads go compute-bound for a second or two at a time as
>>>> they parse web pages.  
>>> You're probably better off using separate processes for the different
>>> pages.  If I remember, you were using BeautifulSoup, which while very
>>> cool, is pretty doggone slow for use on large volumes of pages.  I don't
>>> know if there's much that can be done about that without going off on a
>>> fairly messy C or C++ coding adventure.  Maybe someday someone will do
>>> that.
>>    I already use separate processes for different domains.  I could
>> live with Python's GIL as long as moving to a multicore server
>> doesn't make performance worse.  That's why I asked about CPU dedication
>> for each process, to avoid thrashing at the GIL.
>>
> I believe it's already been said that the GIL thrashing is mostly MacOS
> specific. You might also find something in the affinity module

    No, the original analysis was MacOS oriented, but the same mechanism
applies for fighting over the GIL on all platforms.  There was some
pontification that it might be a MacOS-only issue, but no facts
were presented.  It might be cheaper on C implementations with mutexes
that don't make system calls for the non-blocking cases.

					John Nagle