[python-win32] PyParallel - an experimental "multicore" fork of Python 3 for Windows

[Trent Nelson]

On Sat, Aug 01, 2015 at 02:54:16PM +0000, Paul_Koning at Dell.com wrote:
> 
> > On Jul 31, 2015, at 4:14 PM, Trent Nelson <trent at snakebite.org>
> > wrote:
> > 
> > Hi folks,
> > 
> > Bit off-topic, but just wanted to let people know about an
> > experimental proof-of-concept fork of Python 3 I've been working on
> > for the past couple of years called PyParallel:
> > http://pyparallel.org.  It essentially gets around the GIL
> > limitations and allows Python code to run simultaneously in multiple
> > threads from within a single interpreter/process.
> > 
> > It's Windows only -- so I figured it might be of interest to this
> > list.
> 
> Impressive.  Is the Windows only aspect because you used Windows
> proprietary thread APIs?  Or because there is something special about
> Windows thread mechanisms that doesn’t carry over to, say, POSIX
> pthread?  I assume the former, in other words it should be adaptable
> to standard thread architectures with some effort.

It's actually closer to the latter.  I don't create a single thread with
PyParallel -- I use the threadpool facilities exclusively.  POSIX doesn't
have anything like this -- the pthread abstraction is focused around the
thread (the "worker"), not the work.  (OS X has GCD, which comes close.)

You *could* port PyParallel to POSIX, but you're either going to have to
implement all the stuff Windows provides for free, which is a pretty big
engineering effort.  Even then, you're constrained by side-effects of
the readiness-oriented I/O model (versus the completion-oriented model
of Windows), so it becomes a lot harder to saturate all your I/O channels
*and* peg all of your cores at 100% under load.  (A typical multithreaded
architecture will have a single thread dedicated to accept(), and then
simply round-robin dispatch new FDs to a manual thread pool.  A thread
picks up the FD and adds it to its epoll/kqueue set -- that FD is now
bound to that thread, which can quickly result in lop-sided resource
allocation.  The Windows I/O model revolves around the I/O request
packet, which allows the "thread agnostic" separation between the I/O
event (a write or read has completed) and the underlying thread used to
handle the callback.)

> 	paul

    Trent.