multiprocessing.Queue() and missing sem_open

[Оlе Ѕtrеісhеr]

Chris Angelico <rosuav at gmail.com> writes:

> On Fri, Feb 6, 2015 at 7:22 AM, Оlе Ѕtrеісhеr <ole-usenet-spam at gmx.net> wrote:
>> I am just trying to prepare a package (astropy) for (Debian) Hurd. This
>> os lacks a sem_open() implementation. When I now try:
>>
>> import multiprocessing
>> q = multiprocessing.Queue()
>>
>> I get an ImportError with Python 2.7, but an AttributeError with Python
>> 3.4. In the documentation of multiprocessing.Queue() I couldn't find any
>> hint that it would throw this exception.
>
> Neither of those errors is being consciously thrown by the Queue
> class; the latter means that multiprocessing exists but it has no
> Queue in it, and the former means that the entire multiprocessing
> module is absent.

Trace with 3.4.2:

self = <multiprocessing.context.DefaultContext object at 0x1449b0c>, maxsize = 0

    def Queue(self, maxsize=0):
        '''Returns a queue object'''
        from .queues import Queue
>       return Queue(maxsize, ctx=self.get_context())

/usr/lib/python3.4/multiprocessing/context.py:101: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <multiprocessing.queues.Queue object at 0x511e8cc>, maxsize = 0

    def __init__(self, maxsize=0, *, ctx):
        if maxsize <= 0:
>           maxsize = _multiprocessing.SemLock.SEM_VALUE_MAX
E           AttributeError: 'module' object has no attribute 'SemLock'

/usr/lib/python3.4/multiprocessing/queues.py:38: AttributeError

Trace with 2.7.9:

maxsize = 0

    def Queue(maxsize=0):
        '''
        Returns a queue object
        '''
>       from multiprocessing.queues import Queue

/usr/lib/python2.7/multiprocessing/__init__.py:217: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    __all__ = ['Queue', 'SimpleQueue', 'JoinableQueue']
    
    import sys
    import os
    import threading
    import collections
    import time
    import atexit
    import weakref
    
    from Queue import Empty, Full
    import _multiprocessing
    from multiprocessing import Pipe
>   from multiprocessing.synchronize import Lock, BoundedSemaphore, Semaphore, Condition

/usr/lib/python2.7/multiprocessing/queues.py:48: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    'Lock', 'RLock', 'Semaphore', 'BoundedSemaphore', 'Condition', 'Event'
    ]
    
    import threading
    import os
    import sys
    
    from time import time as _time, sleep as _sleep
    
    import _multiprocessing
    from multiprocessing.process import current_process
    from multiprocessing.util import Finalize, register_after_fork, debug
    from multiprocessing.forking import assert_spawning, Popen
    
    # Try to import the mp.synchronize module cleanly, if it fails
    # raise ImportError for platforms lacking a working sem_open implementation.
    # See issue 3770
    try:
    from _multiprocessing import SemLock
    except (ImportError):
    raise ImportError("This platform lacks a functioning sem_open" +
                      " implementation, therefore, the required" +
                      " synchronization primitives needed will not" +
>                     " function, see issue 3770.")
E                     ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.

/usr/lib/python2.7/multiprocessing/synchronize.py:59: ImportError

Both are with the same Hurd version.

> I'm guessing you built Python from source?

No, I am using the precompiled version that comes with Debian. It has no
specific patch for sem_open.

> But this is the exact way that Python will normally signal that
> something isn't available. If your platform doesn't have, say,
> os.O_BINARY, because that flag has no meaning for you, then you don't
> get NotImplementedError - you simply get AttributeError.

As you can see from the trace, multiprocessing.Queue is there, but
initialization fails.

>> Can I be sure that the following works also in future?
>>
>> try
>>     q = multiprocessing.Queue()
>> except (ImportError, AttributeError)
>>     # handle the case of missing sem_open
>>
>> Or what is the correct way to catch a not working Queue caused by a
>> missing sem_open() implementation?
>
> The ImportError will come from the import statement

No.

> the AttributeError will come from the attribute lookup - the above code
> can't[1] bomb out with ImportError.

Maybe it can't but it actually does.

> So a more correct way would be something like:
>
> try:
>     import multiprocessing
>     multiprocessing.Queue
> except (ImportError, AttributeError):
>     # handle the absence

Is it sure that this will work in the future as well?

> Or alternatively, don't even bother to try/except the import. If your
> code requires multiprocessing.Queue (as opposed to just queue.Queue),
> chances are you can't cope with the absence of the module.

This code is used for a fast, asynchronious reader. If it is not
implemented, there is a (slower) fallback solution. However, we just want
to catch the "is not implemented" case here but not any other problem
that may appear during opening or reading the file.

Shall I file a bug for this?

Best

Ole