[Python-Dev] Improved thread switching

[Alex Martelli]

Hmmm, sorry if I'm missing something obvious, but, if the occasional
background computations are sufficiently heavy -- why not fork, do
said computations in the child thread, and return the results via any
of the various available IPC approaches?  I've recently (at Pycon,
mostly) been playing devil's advocate (i.e., being PRO-threads, for
once) on the subject of utilizing multiple cores effectively -- but
the classic approach (using multiple _processes_ instead) actually
works quite well in many cases, and this application server would
appear to be one.  (the pyProcessing package appears to offer an easy
way to migrate threaded code to multiple-processes approaches,
although I've only played around with it, not [yet] used it for
production code).


Alex

On Wed, Mar 19, 2008 at 10:49 AM, Adam Olsen <rhamph at gmail.com> wrote:
> On Wed, Mar 19, 2008 at 11:25 AM, Stefan Ring <s.r at visotech.at> wrote:
>  > Adam Olsen <rhamph <at> gmail.com> writes:
>  >
>  >
>  > > So you want responsiveness when idle but throughput when busy?
>  >
>  >  Exactly ;)
>  >
>  >
>  >  > Are those calculations primarily python code, or does a C library do
>  >  > the grunt work?  If it's a C library you shouldn't be affected by
>  >  > safethread's increased overhead.
>  >  >
>  >
>  >  It's Python code all the way. Frankly, it's a huge mess, but it would be very
>  >  very hard to come up with a scalable solution that would allow to optimize
>  >  certain hotspots and redo them in C or C++. There isn't even anything left to
>  >  optimize in particular because all those low hanging fruit have already been
>  >  taken care of. So it's just ~30kloc Python code over which the total time spent
>  >  is quite uniformly distributed :(.
>
>  I see.  Well, at this point I think the most you can do is file a bug
>  so the problem doesn't get forgotten.  If nothing else, if my
>  safethread stuff goes in it'll very likely include a --with-gil
>  option, so I may put together a FIFO scheduler.
>
>
>  --
>  Adam Olsen, aka Rhamphoryncus
>
>
> _______________________________________________
>  Python-Dev mailing list
>  Python-Dev at python.org
>  http://mail.python.org/mailman/listinfo/python-dev
>  Unsubscribe: http://mail.python.org/mailman/options/python-dev/aleaxit%40gmail.com
>