More CPUs doen't equal more speed

[Chris Angelico]

On Fri, May 24, 2019 at 5:37 AM Bob van der Poel <bob at mellowood.ca> wrote:
>
> I've got a short script that loops though a number of files and processes
> them one at a time. I had a bit of time today and figured I'd rewrite the
> script to process the files 4 at a time by using 4 different instances of
> python. My basic loop is:
>
> for i in range(0, len(filelist), CPU_COUNT):
>     for z in range(i, i+CPU_COUNT):
>         doit( filelist[z])
>
> With the function doit() calling up the program to do the lifting. Setting
> CPU_COUNT to 1 or 5 (I have 6 cores) makes no difference in total speed.
> I'm processing about 1200 files and my total duration is around 2 minutes.
> No matter how many cores I use the total is within a 5 second range.

Where's the part of the code that actually runs them across multiple
CPUs? Also, are you spending your time waiting on the disk, the CPU,
IPC, or something else?

ChrisA