[Numpy-discussion] Starting to work on runtime plugin system for plugin (automatic sse optimization, etc...)

[David Cournapeau]

Lisandro Dalcin wrote:
> David, I briefly took a look at your code, and I have a very, very
> important observation.
>
> Your implementation make uses of low level dlopening. Then, your are
> going to have to manage all the oddities of runtime loading in the
> different systems. In this case, 'libtool' could really help. I know,
> it is GPL, but AFAIK it has some special licencing that let's you ship
> it with your code in the same licence terms than your code.
Ok, there are several issues here:
    1 cross platform runtime loading
    2 how to access the plugin capabilities (function pointer, 
interfaces, etc...)
    3 how to build

1: the implementation is not cross platform, but the API is; It took me 
~ 2 hours to refactor symbol loading, and getting an implementation for 
posix/win32/Mac os X. I don't know any OS with runtime loading 
capabilities and without the ability to load a file and a symbol from 
it; if it does not, it cannot be used by python anyway, everything would 
have to be build statically at the same time as python, which we do not 
support in numpy anway, AFAIK.

2: I studied quite a bit several approaches before using this one. That 
was my main concern at first. For plugins, you have the following 
possibilities I am aware of:
    - raw function pointer
    - COM
    - pre-defined API

Raw function pointer are the simplest, but is not really scalable. COM 
is this big monstrosity, extremely ackward to use, but can be extended 
ad nauseum, without pre-defined interface. By pre-defined API, I mean 
something like VST plugins and the co (used for music softwares, where a 
host can load may plugins to provide sound effectsl it is the de-facto 
standard, Mac OS X has its built-in thing called AudioUnit, which is the 
same thing for what matters here).

For the usage I have in mind (blas, fft, lapack, etc...), the API  
cannot be pre-defined (each one is totally different), so I quickly 
dismiss the VST-like approach. Then there is the COM thing, which is 
really complicated. Although each plugin interface is totally different 
(blas vs lapack), they are relatively fixed in stone, so I thought that 
by using generated code, the scalability problem of raw pointers could 
be alleviated.

3: libtool does not know about windows, and I think it is way too 
overkill. We can't use libtool for building (which is one of the big 
thing libtool provides). dlopen-like approach is not as portable as 
libtool, but it is as portable as python, which is good enough for numpy :)

> But, I definitely think that a betther approach would be using a stubs
> mechanism ala TCL, or wath is currently used in some extension modules
> in core python, like cStringIO. In short, you access all your
> functions from a pointer do a struct (statically or heap allocated)
> where each struct member is filled with a pointer to a function. This
> is pretty much similar to C++ virtual tables, or the Cython cdef's
> classes with cdef's methods. And then you just let Python do the work
> of dynamic loading of extension modules. The numpy C/API uses a
> similar approach, but uses an array. IMHO, the struct approach is
> cleaner.
>
> What do you think about this?

It may be cleaner, but I am not convinced it buys us much. With my 
approach, all is needed for the existing code (numpy.core, numpy.linalg, 
etc...) is a renaming of the used function (which can be done in 5 
minutes with sed), because in C, calling a function or a function 
pointer is exactly the same thing. With the approach of function 
pointers, you will have to replace all the function calls for blas, 
lapack, etc... by a system to pass the array of function pointers. That 
sounds like nightmare to me, because I don't see how to do that 
automatically. Maybe I just don't see it; in that case, what would be 
your approach ?

I have been convinced that the function pointer approach is usable by 
looking at liboil, which does exactly the thing we need:

http://liboil.freedesktop.org/wiki/

(you can look at the liboil/liboilfuncs* and liboil/liboilfunc.* files). 
Liboil's approach is more complicated, because the function pointer is 
provided by a function factory (so that each function can be initialized 
differently), but I don't think we need the factory in our case (and if 
we need, we can do it without changing anything in the code which uses 
the plugin).

cheers,

David