Multiprocessing: don't push the pedal to the metal?

[Adam Tauno Williams]

On Mon, 2011-05-23 at 12:51 -0700, John Ladasky wrote:
> On May 23, 2:50 am, Adam Tauno Williams <awill... at whitemice.org>
> wrote:
> > I develop an app that uses multiprocessing heavily.  Remember that all
> > these processes are processes - so you can use all the OS facilities
> > regarding processes on them.  This includes setting nice values,
> > schedular options, CPU pinning, etc...
> That's interesting.  Does code exist in the Python library which
> allows the adjustment of CPU pinning and nice levels?  I just had
> another look at the multiprocessing docs, and also at os.subprocess.
> I didn't see anything that pertains to these issues.

"in the Python library" - no.  All these types of behaviors are platform
specific.

For example you can set the "nice" (priority) of a UNIX/LINUX process
using the nice method from the os module.  Our workflow engine does this
on all worker processes it starts - it sends the workers to the lowest
priority.


from os                    import nice as os_priority
...
        try:
            os_priority(20)
        except Exception, e:
            ...

I'm not aware of a tidy way to call sched_setaffinity from Python; but
my own testing indicates that the LINUX kernel is very good at figuring
this out on its own so long as it isn't swamped.  Queuing, rather than
starting, additional workflows if load average exceeds X.Y and setting
the process priority of workers to very-low seems to work very well.

There is <http://pypi.python.org/pypi/affinity> for setting affinity,
but I haven't used it.