[Python-Dev] Thoughts fresh after EuroPython

[Guido van Rossum]

On Mon, Jul 26, 2010 at 3:00 AM, Michael Foord
<fuzzyman at voidspace.org.uk> wrote:
> At Resolver Systems we created a "calculation system" that does large
> calculations on background threads using IronPython. Doing them on a
> background thread allows the ui to remain responsive. Several calculations
> could run simultaneously using multiple cores.
>
> As the calculation operates on a large object graph (which the ui then needs
> access to in order to display it) using multiprocessing would have imposed a
> very big overhead due to serialization / deserialization (the program runs
> on windows).
>
> Using CPython would have made the program a lot slower due to the GIL.

Sure. Note that using threads with the GIL, it is not a problem to
keep the UI responsive even if background calculations are going on
(at worst it requires some tweaking of sys.setcheckinterval() or its
new-GIL equivalent). However with the GIL multiple calculations would
be limited to a single core.

According to CSP advicates, this approach will break down when you
need more than 8-16 cores since cache coherence breaks down at 16
cores. Then you would have to figure out a message-passing approach
(but the messages would have to be very fast).

-- 
--Guido van Rossum (python.org/~guido)