Multiprocessing performance question

[george trojan]

I don't know whether this is a toy example, having grid of this size is not
uncommon. True, it would make more sense to do distribute more work on each
box, if there was any. One has to find a proper balance, as with many other
things in life. I simply  responded to a question by the OP.

George

On Thu, 21 Feb 2019 at 01:30, DL Neil <PythonList at danceswithmice.info>
wrote:

> George
>
> On 21/02/19 1:15 PM, george trojan wrote:
> > def create_box(x_y):
> >      return geometry.box(x_y[0] - 1, x_y[1],  x_y[0], x_y[1] - 1)
> >
> > x_range = range(1, 1001)
> > y_range = range(1, 801)
> > x_y_range = list(itertools.product(x_range, y_range))
> >
> > grid = list(map(create_box, x_y_range))
> >
> > Which creates and populates an 800x1000 “grid” (represented as a flat
> list
> > at this point) of “boxes”, where a box is a shapely.geometry.box(). This
> > takes about 10 seconds to run.
> >
> > Looking at this, I am thinking it would lend itself well to
> > parallelization. Since the box at each “coordinate" is independent of all
> > others, it seems I should be able to simply split the list up into chunks
> > and process each chunk in parallel on a separate core. To that end, I
> > created a multiprocessing pool:
>
>
> I recall a similar discussion when folk were being encouraged to move
> away from monolithic and straight-line processing to modular functions -
> it is more (CPU-time) efficient to run in a straight line; than it is to
> repeatedly call, set-up, execute, and return-from a function or
> sub-routine! ie there is an over-head to many/all constructs!
>
> Isn't the 'problem' that it is a 'toy example'? That the amount of
> computing within each parallel process is small in relation to the
> inherent 'overhead'.
>
> Thus, if the code performed a reasonable analytical task within each box
> after it had been defined (increased CPU load), would you then notice
> the expected difference between the single- and multi-process
> implementations?
>
>
>
>  From AKL to AK
> --
> Regards =dn
>