multiprocessing module (PEP 371)

[sturlamolden]

I sometimes read python-dev, but never contribute. So I'll post my
rant here instead.

I completely support adding this module to the standard lib. Get it in
as soon as possible, regardless of PEP deadlines or whatever.

I don't see pyprocessing as a drop-in replacement for the threading
module. Multi-threading and multi-processing code tend to be
different, unless something like mutable objects in shared memory is
used as well (cf. Python Shared Objects). If this limitation can
educate Python programmers to use queues instead of locks and mutable
objects, even multi-threaded Python programs may actually benefit.
Some API differences between threading and multiprocessing do not
matter. Programmers should not consider processes as a drop-in
replacement for threads.

One limitation not discussed on python-dev is the lack of fork on
Win32. This makes the pyprocessing module particularly inefficient at
creating processes on this platform, as it depends on serializing
(pickling and de-pickling) a lot of Python objects. Even a non-COWfork
would be preferred. I will strongly suggest something is done to add
support for os.fork to Python on Windows. Either create a full cow
fork using ZwCreateProcess (ntdll.dll does support COWforking, but
Win32 API does not expose it), or do the same as Cygwin is doing to
fork a process without COW. Although a non-cow fork a la Cygwin is not
as efficient as a fork on Linux/FreeBSD/Unix, it is still better than
what pyprocessing is doing.