Python 2.6, multiprocessing module and BSD

[Jesse Noller]

On Wed, Oct 22, 2008 at 11:06 AM, Philip Semanchuk <philip at semanchuk.com> wrote:
>> The BSD issue was raised late in the cycle for 2.6. The problem is
>> that FBSD's support is "very experimental" as Phillip points out - and
>> OpenBSD doesn't even have them.
>>
>> Due to the lateness of the issue and a finite amount of time I have to
>> work on things, I chose to disable support for this on the various
>> *BSDs until I can cook up a stable patch or have one provided by
>> someone more familiar with the inner workings of Free-BSD. OpenBSD
>> support is a non-starter.
>
> Hi Jesse,
> I wasn't aware of the multiprocessing module. It looks slick! Well done.
>

The credit goes to R. Oudkerk, the original author of the pyprocessing
library - I'm simply a rabid user who managed to wrangle it into
Python-Core. See: http://www.python.org/dev/peps/pep-0371/

> I don't know if you clicked on the link I gave for my posix_ipc module, but
> it looks like we're duplicating effort to some degree. My module makes POSIX
> semaphore & shared memory primitives available to Python programs.
> Obviously, what you've done is much more sophisticated.
>

I actually saw your stuff cross the 'tubes - it looks darned nice as a
lower-level interface. What the MP package is meant to be is obviously
much more high level (and "thread like"), MP goes out of it's way to
hide the gritty internals of the semaphore management/etc - posix_ipc
is much more low level than that.

> One oversight I noticed the multiprocessing module docs is that a
> semaphore's acquire() method shouldn't have a timeout on OS X as
> sem_timedwait() isn't supported on that platform. (You note OS X's lack of
> support for sem_getvalue() elsewhere.)

Please file a ticket or update http://bugs.python.org/issue4012 so I
don't loose it, my memory is increasingly lossy. Good catch.

>
> A question - how do you handle the difference in error messages on various
> platforms? For instance, sem_trywait() raises error 35, "Resource
> temporarily unavailable" under OS X but error 11 under Ubuntu. Right now I'm
> just passing these up to the (Python) caller as OSErrors. This makes it
> really hard for the Python programmer to write cross-platform code.
>

If you look at the code, we're pretty much raising OSError - it's
possible we could enhance this in later versions, but given MP is
supposed to be a cross-platform as possible and protect the user from
the seedy underbelly of semaphores/pipes/etc - when an OSError does
occur, it's generally a bug in our code, not the users.

> The only solution I can think of (which I haven't coded yet) is to compile &
> run a series of small C programs during setup.py that test things like
> sem_trywait() to see what errors occur, and provide those constants to my
> main .c module so that it can detect those errors exactly and wrap them into
> a specific, custom error for the Python caller.
>
> Any thoughts on this?
>

That's actually (while feeling hacky) a possibly sensible idea, the
problem is is that you'd need to maintain documentation to tell users
the exceptions for their platform.

-jesse