[issue41660] multiprocessing.Manager objects lose connection info

[Tim Peters]

New submission from Tim Peters <tim at python.org>:

This started on StackOverflow:

https://stackoverflow.com/questions/63623651/how-to-properly-share-manager-dict-between-processes

Here's a simpler program.

Short course: an object of a subclass of mp.Process has an attribute of seemingly any type obtained from an mp.Manager(). The report above used a Manager.dict, and the program here a Manager.Value.

When .start() is invoked, the first time that attribute is used in any way that requires communication with the Manager server, the program dies. The output below is from 3.8.5 on Windows; the report above is some Linux flavor. The tracebacks are very similar, the only obvious difference being that the implementation complains about a socket on Linux but about a named pipe on Windows.

Note that there's no problem passing such things as explicit arguments to functions across processes. The loss here appears specific to the inscrutable under-the-covers pickle dance needed to create a "self" by magic on worker processes.

The code:

from multiprocessing import Process, Manager, Event

class MyProducer(Process):
    def __init__(self, value, event):
        Process.__init__(self)
        self.val = value
        self.event = event

    def run(self):
        print("at producer start:", self.val.value)
        self.val.value = 42
        self.event.set()

class MyConsumer(Process):
    def __init__(self, value, event):
        Process.__init__(self)
        self.val = value
        self.event = event

    def run(self):
        self.event.wait()
        print("in consumer:", self.val.value)
                        
if __name__ == "__main__":
    state_value = Manager().Value('i', 666)
    print("at start:", state_value.value)
    state_ready = Event()
    producerprocess = MyProducer(state_value, state_ready)
    consumerprocess = MyConsumer(state_value, state_ready)
    producerprocess.start()
    consumerprocess.start()    

The output:

at start: 666
Process MyProducer-2:
Traceback (most recent call last):
  File "C:\Users\Tim\AppData\Local\Programs\Python\Python38\lib\multiprocessing\managers.py", line 827, in _callmethod
    conn = self._tls.connection
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Tim\AppData\Local\Programs\Python\Python38\lib\multiprocessing\process.py", line 315, in _bootstrap
    self.run()
  File "C:\Code\temp.py", line 10, in run
    print("at producer start:", self.val.value)
  File "C:\Users\Tim\AppData\Local\Programs\Python\Python38\lib\multiprocessing\managers.py", line 1154, in get
    return self._callmethod('get')
  File "C:\Users\Tim\AppData\Local\Programs\Python\Python38\lib\multiprocessing\managers.py", line 831, in _callmethod
    self._connect()
  File "C:\Users\Tim\AppData\Local\Programs\Python\Python38\lib\multiprocessing\managers.py", line 818, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "C:\Users\Tim\AppData\Local\Programs\Python\Python38\lib\multiprocessing\connection.py", line 500, in Client
    c = PipeClient(address)
  File "C:\Users\Tim\AppData\Local\Programs\Python\Python38\lib\multiprocessing\connection.py", line 702, in PipeClient
    _winapi.WaitNamedPipe(address, 1000)
FileNotFoundError: [WinError 2] The system cannot find the file specified

----------
components: Library (Lib)
messages: 376051
nosy: tim.peters
priority: normal
severity: normal
status: open
title: multiprocessing.Manager objects lose connection info
type: behavior
versions: Python 3.8

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue41660>
_______________________________________