Adding new source types to distutils?

[Stefan Behnel]

Devin Jeanpierre, 28.04.2013 19:55:
> Last night I wrote a toy prototype module that lets one compile Rust
> crates into extension modules for Python. The problem is, I don't know
> the "right" way to do this. Ideally I'd just want to tell build_ext
> that there's a new source type I want to handle (.rc and .rs), and
> also tell distutils that it should handle it by running the code that
> I specify (which can compile the .rs/.rc files, remove them from the
> sources list, and add the resulting object files and such to the
> linker arguments)
> 
> The problem is that, as I understand it, the way to do this is
> subclassing and then replacing the build_ext command. At least, that's
> what Cython does. The problem is, that's what Cython does, so if I do
> that, it means you can't use Cython and Rust together -- that's bad,
> because Cython would be useful for writing bindings to Rust crates,
> too. So I don't want to write my own subclass. In place of that, I
> don't know what the right approach is.

That approach is discouraged for Cython. The compiler comes with a
cythonize() function these days, which users can simply call from their
setup.py scripts. It spits out a list of Extensions that you can pass into
setup(). So, for example, you can say

    extensions = cythonize('mypackage/*.pyx')

and it would do the right thing. You can also pass configuration options
into cythonize() to influence the way Cython translates your code.
Alternatively, you can pass in a list of Extensions and cythonize() will
process that and replace .pyx files by the compiled .c files. That also
makes it easier to build without having Cython installed, by simply
replacing the .pyx files by .c yourself and passing the Extensions directly
into setup(). And it allows for more complex Extension configurations that
Cython doesn't have to care about.

You might want to do something similar in your case. It gives users much
more flexibility when using source code preprocessors and also avoids
conflicts between packages like the one you describe above, or problems
with future versions of distutils due to fragile build setups.

Stefan