[Numpy-discussion] List/location of consecutive integers (2)
Christopher Barker
Chris.Barker at noaa.gov
Tue May 26 20:07:39 EDT 2009
Andrea Gavana wrote:
> I have tried the solutions proposed in the previous thread and it
> looks like Chris' one is the fastest for my purposes.
whoo hoo! What do I win? ;-)
> Splitting the reading process between 4 processes will require the
> exchange of 5-20 MB from the child processes to the main one: do you
> think my script will benefit from using multiprocessing?
If you are talking about multiprocessing to read the data in -- I don't
think so -- that's probably IO bound anyway. You can't make your disks
faster with multiple processors.
> Should I try another approach?
I don't know it will do anything for performance, but you might want to
look at memory mapped arrays -- it's a very cool way to work with data
files too big to want to bring into memory all at once.
-Chris
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker at noaa.gov
More information about the NumPy-Discussion
mailing list