Multiprocessing vs. concurrent.futures, Linux vs. Windows

[John Ladasky]

On Monday, May 4, 2020 at 4:09:53 PM UTC-7, Terry Reedy wrote:
> On 5/4/2020 3:26 PM, John Ladasky wrote:
> > Several years ago I built an application using multiprocessing.  It only needed to work in Linux.  I got it working fine.  At the time, concurrent.futures did not exist.
> > 
> > My current project is an application which includes a PyQt5 GUI, and a live video feed with some real-time image processing.  Running all this in one process resulted in unacceptable performance, so I'm trying to move all of the heavy video work into its own process.  I only need to define one child process.  I don't need a Pool.  The child Process can run indefinitely, and it will communicate multiple messages to the main process using a Pipe.
> > 
> > I built a working example in Linux, but it hangs in Windows.  I built a minimum example.  My problem has nothing to do with PyQt.  In Windows, my example hangs when I try to create the child Process.  Code:
> > 
> > 
> > import os
> > from multiprocessing import Pipe, Process
> > 
> > def child_app():
> >      inlet.send("Child process id = {}".format(os.getpid()))
> > 
> > if __name__ == "__main__":
> >      outlet, inlet = Pipe()
> >      print("Parent process id =", os.getpid())
> >      child_process = Process(target=child_app)  # Windows hangs here
> >      child_process.start()
> >      print(outlet.recv())
> >      child_process.join()
> >      child_process.close()
> >      print("Program complete.")
> > 
> > 
> > I'm working in Python 3.7.6 on Windows 10, and 3.7.5 on Ubuntu Linux 19.10.
> 
> Does the minimal example in the doc work for you?
> (IE, do you only have a problem with Pipe?)
> 
> from multiprocessing import Process
> 
> def f(name):
>      print('hello', name)
> 
> if __name__ == '__main__':
>      p = Process(target=f, args=('bob',))
>      p.start()
>      p.join()
> 
> How about the Pipe example?
> 
> from multiprocessing import Process, Pipe
> 
> def f(conn):
>      conn.send([42, None, 'hello'])
>      conn.close()
> 
> if __name__ == '__main__':
>      parent_conn, child_conn = Pipe()
>      p = Process(target=f, args=(child_conn,))
>      p.start()
>      print(parent_conn.recv())   # prints "[42, None, 'hello']"
>      p.join()
> 
> If this does not work on Windows, the example or doc should be changed.
> But I believe I tested it once. Note that unlike your code, the 
> child_conn is sent as argument.  The relation between the module code 
> and child processes is different on Windows than *nix.

Hi Terry,

Thanks for your reply.  I have been hacking at this for a few hours.  I have learned two things:

1. Windows hangs unless you explicitly pass any references you want to use in the subprocess through args.  That would include the Pipe connection. 
 Using multiprocessing in Linux requires the reference names to be global, however the use of args is not required.  Finally, Linux does not appear to cause any problems if args are specified.

2. Even if you fix problem 1, the parent process must be distinguished by creating the subprocess inside an "if __name__ == '__main__'" block.  Again, Linux just pushes on through, but Windows will hang if you don't do this.

The example code you posted shows exactly these two changes.  They are OS-specific, and my multiprocessing code has been (up to now) only required to run on Linux.  These recommendations can be found in the official Python docs, e.g.:

https://docs.python.org/3.7/library/multiprocessing.html#programming-guidelines