Multiprocessing question

[Roy Smith]

In article <53c34400$0$9505$c3e8da3$5496439d at news.astraweb.com>,
 Steven D'Aprano <steve+comp.lang.python at pearwood.info> wrote:

> On Sun, 13 Jul 2014 19:53:09 -0400, Paul LaFollette wrote:
> 
> > I have thrown together a little C/UNIX program that forks a child
> > process, then proceeds to let the child and parent alternate.  Either
> > can run until it pauses itself and wakes the other.
> > 
> > I would like to know if there be a way to create the same behavior in
> > Python 3, preferably in a non-platform dependent fashion.

[...]
> But the best way to solve this in a platform independent way is to use 
> one of the concurrency modules:
> 
> https://docs.python.org/3/library/concurrency.html
> 
> such as multiprocessing. Although these are written for Python 2 rather 
> than 3, you may find them useful:
> 
> http://pymotw.com/2/multiprocessing/index.html#module-multiprocessing
> 
> https://www.ibm.com/developerworks/aix/library/au-multiprocessing/

Let me toss out one other possibility.  If you are going to build some 
kind of message queue between two processes, you might want to look at 
an external queue mechanism such as http://kr.github.io/beanstalkd/ or 
http://www.celeryproject.org/.  The advantage of these over rolling your 
own using the built-in Python modules is they handle a lot of queueing, 
locking, and persistence problems which you might end up having to deal 
with yourself.

They also give you the ability to locate the two processes on different 
machines.  You might not want to do that today, but if your application 
grows, it might be how you want to scale your processing power.  It is 
easy, for example, to have one process pushing tasks onto the queue, and 
a large number of workers pulling them off.  The producer doesn't have 
to know anything about how many workers there are (nor does there need 
to be a fixed number).

They are also language independent.  You might discover at some point in 
the future that you want to rewrite one side of the queue in a different 
language for some reason (perhaps to take advantage of a library that's 
not available in Python).  A queue like Beanstalk or Celery makes that 
easy.  Of course, to do this, you would need to format your messages in 
some language-neutral way (JSON, MessagePack, etc).