[python-win32] PyParallel - an experimental "multicore" fork of Python 3 for Windows
Trent Nelson
trent at snakebite.org
Mon Aug 3 02:10:19 CEST 2015
On Sat, Aug 01, 2015 at 02:54:16PM +0000, Paul_Koning at Dell.com wrote:
>
> > On Jul 31, 2015, at 4:14 PM, Trent Nelson <trent at snakebite.org>
> > wrote:
> >
> > Hi folks,
> >
> > Bit off-topic, but just wanted to let people know about an
> > experimental proof-of-concept fork of Python 3 I've been working on
> > for the past couple of years called PyParallel:
> > http://pyparallel.org. It essentially gets around the GIL
> > limitations and allows Python code to run simultaneously in multiple
> > threads from within a single interpreter/process.
> >
> > It's Windows only -- so I figured it might be of interest to this
> > list.
>
> Impressive. Is the Windows only aspect because you used Windows
> proprietary thread APIs? Or because there is something special about
> Windows thread mechanisms that doesn’t carry over to, say, POSIX
> pthread? I assume the former, in other words it should be adaptable
> to standard thread architectures with some effort.
It's actually closer to the latter. I don't create a single thread with
PyParallel -- I use the threadpool facilities exclusively. POSIX doesn't
have anything like this -- the pthread abstraction is focused around the
thread (the "worker"), not the work. (OS X has GCD, which comes close.)
You *could* port PyParallel to POSIX, but you're either going to have to
implement all the stuff Windows provides for free, which is a pretty big
engineering effort. Even then, you're constrained by side-effects of
the readiness-oriented I/O model (versus the completion-oriented model
of Windows), so it becomes a lot harder to saturate all your I/O channels
*and* peg all of your cores at 100% under load. (A typical multithreaded
architecture will have a single thread dedicated to accept(), and then
simply round-robin dispatch new FDs to a manual thread pool. A thread
picks up the FD and adds it to its epoll/kqueue set -- that FD is now
bound to that thread, which can quickly result in lop-sided resource
allocation. The Windows I/O model revolves around the I/O request
packet, which allows the "thread agnostic" separation between the I/O
event (a write or read has completed) and the underlying thread used to
handle the callback.)
> paul
Trent.
More information about the python-win32
mailing list