Multiprocessing performance question

[Israel Brewster]

> On Feb 18, 2019, at 6:37 PM, Ben Finney <ben+python at benfinney.id.au> wrote:
> 
> I don't have anything to add regarding your experiments with
> multiprocessing, but:
> 
> Israel Brewster <ijbrewster at alaska.edu> writes:
> 
>> Which creates and populates an 800x1000 “grid” (represented as a flat
>> list at this point) of “boxes”, where a box is a
>> shapely.geometry.box(). This takes about 10 seconds to run.
> 
> This seems like the kind of task NumPy <URL:http://www.numpy.org/> is
> designed to address: Generating and manipulating large-to-huge arrays of
> numbers, especially numbers that are representable directly in the
> machine's basic number types (such as moderate-scale integers).
> 
> Have you tried using that library and timing the result?

Sort of. I am using that library, and in fact once I get the result I am converting it to a NumPy array for further use/processing, however I am still a NumPy newbie and have not been able to find a function that generates a numpy array from a function. There is the numpy.fromfunction() command, of course, but “…the function is called with … each parameter representing the coordinates of the array varying along a specific axis…”, which basically means (if my understanding/inital testing is correct) that my function would need to work with *arrays* of x,y coordinates. But the geometry.box() function needs individual x,y coordinates, not arrays, so I’d have to loop through the arrays and append to a new one or something to produce the output that numpy needs, which puts me back pretty much to the same code I already have.

There may be a way to make it work, but so far I haven’t been able to figure it out any better than the code I’ve got followed by converting to a numpy array. You do bring up a good point though: there is quite possibly a better way to do this, and knowing that would be just as good as knowing why multiprocessing doesn’t improve performance. Thanks!
---
Israel Brewster
Software Engineer
Alaska Volcano Observatory 
Geophysical Institute - UAF 
2156 Koyukuk Drive 
Fairbanks AK 99775-7320
Work: 907-474-5172
cell:  907-328-9145

> 
> -- 
> \     “You don't need a book of any description to help you have some |
>  `\    kind of moral awareness.” —Dr. Francesca Stavrakoloulou, bible |
> _o__)                                              scholar, 2011-05-08 |
> Ben Finney
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list