[Numpy-discussion] Improving Python+MPI import performance

[Langton, Asher]

On 1/13/12 1:58 PM, Dag Sverre Seljebotn wrote:
>
>It's actually not too difficult to do something like
>
>LD_PRELOAD=myhack.so python something.py
>
>and have myhack.so intercept the filesystem calls Python makes (to libc)
>and do whatever it wants. That's a solution that doesn't interfer with
>how Python does its imports at all, it simply changes how Python
>perceives the world around it ("emulation", though much, much lighter).
>
>It does require some low-level C code, but there are several examples on
>the net. I know Ondrej Certik just implemented something similar.

One of my colleagues suggested the LD_PRELOAD trick. I asked around here
at LLNL, and I seem to recall hearing that the LD_PRELOAD trick didn't
work on BlueGene/P, which is where the import bottleneck is the worst.
That might have been incorrect though, since LD_PRELOAD is mentioned on
Argonne's BG/P wiki. I'll have to look into this some more.

-Asher