The Future of Python Threading

[Chris Mellon]

On 8/10/07, Ben Sizer <kylotan at gmail.com> wrote:
> On 10 Aug, 15:38, Ben Finney <bignose+hates-s... at benfinney.id.au>
> wrote:
> > "Justin T." <jmtull... at gmail.com> writes:
> > > The truth is that the future (and present reality) of almost every
> > > form of computing is multi-core, and there currently is no effective
> > > way of dealing with concurrency.
> >
> > Your post seems to take threading as the *only* way to write code for
> > multi-core systems, which certainly isn't so.
> >
> > Last I checked, multiple processes can run concurrently on multi-core
> > systems. That's a well-established way of structuring a program.
>
> It is, however, almost always more complex and slower-performing.
>
> Plus, it's underdocumented. Most academic study of concurrent
> programming, while referring to the separately executing units as
> 'processes', almost always assume a shared memory space and the
> associated primitives that go along with that.
>

This is simply not true. Firstly, there's a well defined difference
between  'process' and a 'thread' and that is that processes have
private memory spaces. Nobody says "process" when they mean threads of
execution within a shared memory space and if they do they're wrong.

And no, "most" academic study isn't limited to shared memory spaces.
In fact, almost every improvement in concurrency has been moving
*away* from simple shared memory - the closest thing to it is
transactional memory, which is like shared memory but with
transactional semantics instead of simple sharing. Message passing and
"shared-nothing" concurrency are very popular and extremely effective,
both in performance and reliability.

There's nothing "undocumented" about IPC. It's been around as a
technique for decades. Message passing is as old as the hills.

> > > We still worry about setting up threads, synchronization of message
> > > queues, synchronization of shared memory regions, dealing with
> > > asynchronous behaviors, and most importantly, how threaded an
> > > application should be.
> >
> > All of which is avoided by designing the program to operate as
> > discrete processes communicating via well-defined IPC mechanisms.
>
> Hardly. Sure, so you don't have to worry about contention over objects
> in memory, but it's still completely asynchronous, and there will
> still be a large degree of waiting for the other processes to respond,
> and you have to develop the protocols to communicate. Apart from
> convenient serialisation, Python doesn't exactly make IPC easy, unlike
> Java's RMI for example.
>

There's nothing that Python does to make IPC hard, either. There's
nothing in the standard library yet, but you may be interested in Pyro
(http://pyro.sf.net) or Parallel Python
(http://www.parallelpython.com/). It's not erlang, but it's not hard
either. At least, it's not any harder than using threads and locks.