More CPUs doen't equal more speed

[Cameron Simpson]

On 23May2019 17:04, bvdp <bob at mellowood.ca> wrote:
>Anyway, yes the problem is that I was naively using command.getoutput()
>which blocks until the command is finished. So, of course, only one process
>was being run at one time! Bad me!
>
>I guess I should be looking at subprocess.Popen(). Now, a more relevant
>question ... if I do it this way I then need to poll though a list of saved
>process IDs to see which have finished? Right? My initial thought is to
>batch them up in small groups (say CPU_COUNT-1) and wait for that batch to
>finish, etc. Would it be foolish to send send a large number (1200 in this
>case since this is the number of files) and let the OS worry about
>scheduling and have my program poll 1200 IDs?
>
>Someone mentioned the GIL. If I launch separate processes then I don't
>encounter this issue? Right?

Yes, but it becomes more painful to manage. If you're issues distinct 
separate commands anyway, dispatch many or all and then wait for them as 
a distinct step.  If the commands start thrashing the rest of the OS 
resources (such as the disc) then you may want to do some capacity 
limitation, such as a counter or semaphore to limit how many go at once.

Now, waiting for a subcommand can be done in a few ways.

If you're then parent of all the processes you can keep a set() of the 
issued process ids and then call os.wait() repeatedly, which returns the 
pid of a completed child process. Check it against your set. If you need 
to act on the specific process, use a dict to map pids to some record of 
the subprocess.

Alternatively, you can spawn a Python Thread for each subcommand, have 
the Thread dispatch the subcommand _and_ wait for it (i.e. keep your 
command.getoutput() method, but in a Thread). Main programme waits for 
the Threads by join()ing them.

Because a thread waiting for something external (the subprocess) doesn't 
hold the GIL, other stuff can proceed. Basicly, if something is handed 
off the to OS and then Python waits for that (via an os.* call or a 
Popen.wait() call etc etc) then it will release the GIL while it is 
blocked, so other Threads _will_ get to work.

This is all efficient, and there's any number of variations on the wait 
step depending what your needs are.

The GIL isn't the disaster most people seem think. It can be a 
bottleneck for pure Python compute intensive work. But Python's 
interpreted - if you _really_ want performance the core compute will be 
compiled to something more efficient (eg a C extension) or handed to 
another process (transcode video in pure Python - argh! - but call the 
ffmpeg command as a subprocess - yes!); handed off, the GIL should be 
released, allowing other Python side work to continue.

Cheers,
Cameron Simpson <cs at cskk.id.au>