[Cython] Transformation of pxds

[Stefan Behnel]

Hi Romain,

CC-ing cython-devel here, as this is not an entirely trivial question to 
answer.

Romain Guillebert, 23.07.2011 06:52:
> During this week, I've been trying to find a way to implement cimport, I
> think that the cleanest way of implementing this is to compile pxds just
> like pyxs are compiled.

You are aware that there are two types of .pxd files? One next to a .pyx or 
.py file with the same name, which provides additional declarations for 
that source file and can also serve as a cimportable .pxd for other 
modules, thus providing a C-level API between modules, and a second type 
that only contains external declarations (well, and potentially inline 
functions etc.), usually for external C libraries or header files, but that 
does not correspond to a .pyx/.py source file.

The idea of materialising the external .pxd files does sound appealing. I 
imagine that this means that the resulting .py file would basically just 
set up ctypes wrappers for external declarations and store them under the 
corresponding name, right? Then a cimport from an external .pxd file would 
turn into a regular import. Plus, you'd end up with a single place where 
the external library (if any, or potentially many libraries) is loaded and 
managed.

Cool.

Even the first kind of .pxd file would fit into this quite nicely. Here, 
the corresponding .pyx/.py file would be enough to represent the module, 
and the exported declarations in the .pxd file would just be ignored. 
Consequently, the .pxd file would only be used to provide declarations for 
the module itself and serve no other purpose. Thus, in both cases, there 
would simply be an importable .py file for each cimported .pxd file, be it 
user provided or generated.


> I'm having trouble knowing where to implement that though.
>
> I think that I will implement that in the method process_pxd of Context
> in Cython/Compiler/Main.py (and it will probably call compile_single)
> but I don't know if it's a good design.
>
> What do you think ?

I think it could be done on the fly, when cimporting the .pxd file. Or 
maybe it's better to just queue up the cimported files that need to get 
compiled, and then compile them after compiling the main source file. For 
the client side of the cimport mechanism, parsing the .pxd file is enough 
to figure out what it will eventually export, so there is no actual need to 
generate a .py file for it at the same time. I think the compilation will 
be a separate operation anyway, because the .pxd file is currently parsed 
into the context of the .pyx file being compiled. Hacking an additional 
compilation onto that step may not be trivial.

I don't mind either way, i.e. if it's done while cimporting the .pxd or 
afterwards.

In any case, this means that compiling a .pyx file can potentially generate 
multiple .py files, which means that there should be a way to reuse already 
generated files from other compiler runs (i.e. if you compile multiple .pyx 
files which cimport the same .pxd files, then each .pxd file only needs to 
get compiled once). Basically, this means that the compiler would check if 
there is already a .py file for a .pxd, and not generate it in that case. 
However, this is problematic, because we do not necessarily know if the .py 
file was user provided or generated...

This makes me think that the ctypes backend may generally want to generate 
its output files into a separate target *directory*, rather than just 
providing a target file name.

Stefan