[Numpy-discussion] IDL vs Python parallel computing

[Nathaniel Smith]

On Wed, May 7, 2014 at 7:11 PM, Sturla Molden <sturla.molden at gmail.com> wrote:
> On 03/05/14 23:56, Siegfried Gonzi wrote:
>  > I noticed IDL uses at least 400% (4 processors or cores) out of the box
>  > for simple things like reading and processing files, calculating the
>  > mean etc.
>
> The DMA controller is working at its own pace, regardless of what the
> CPU is doing. You cannot get data faster off the disk by burning the
> CPU. If you are seeing 100 % CPU usage while doing file i/o there is
> something very bad going on. If you did this to an i/o intensive server
> it would go up in a ball of smoke... The purpose of high-performance
> asynchronous i/o systems such as epoll, kqueue, IOCP is actually to keep
> the CPU usage to a minimum.

That said, reading data stored in text files is usually a CPU-bound
operation, and if someone wrote the code to make numpy's text file
readers multithreaded, and did so in a maintainable way, then we'd
probably accept the patch. The only reason this hasn't happened is
that no-one's done it.

-n

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org