[Cython] patch for #655

[Stefan Behnel]

Felix Salfelder, 27.06.2013 23:06:
> On Thu, Jun 27, 2013 at 12:39:48PM -0700, Robert Bradshaw wrote:
>> Building Python extensions with makefiles/autotools rather than
>> distutils is less supported, but I suppose you could do that manually.
> 
> i've done that. I ran into a few peculiarities with "-I", "-w", and
> __init__.py, but nothing serious.
> 
> the only thing that will affect the user (and which i should mention)
> is: make uses file extensions to determine file types. particularly, i
> have not found a portable hack that allows the use of .pyx for both .c
> and .cpp within the scope of one makefile yet...

That's unfortunate, but not too serious either. Larger projects may end up
using both, but since it should often work to compile everything in C++
mode, and the right way to do it is distutils anyway, people who really
want to go through the hassle of using make will just have to live with it.
In the worst case, you could spell out the build targets explicitly (i.e.
not as patterns) in the makefile, as part of the dependencies.


>> Currently cythonize() allows you to determine quickly upfront what
>> needs to be compiled *without* actually compiling anything. I suppose
>> the idea is that by default you compile everything, and the next time
>> around you have some kind of artifact that lets you understand the
>> dependencies better?
> 
> yes, the dependencies for the first run are empty and filled/refreshed
> during the cython run. no upfront determination required.
> 
>> But you still need a rule for the initial run, right?
> 
> the rules are the same each time, if the target doesn't exist, it will be
> made regardless of how empty the dependency file is.
> 
>> It would still help if you posted exactly how you're using it. E.g.
>> here's a set of .pyx files, I run this to generate some make files,
>> which invokes "cython -M ..."
> 
> The basic principle to build file.so out of file.pyx is this:
> $ cat Makefile # handwritten example demo
> all: file.so
> %.so: %.cc
> 	gcc -shared -fpic $(CFLAGS) $(CPPFLAGS) $< -o $@ -M -MD -MP
> %.c: %.pyx
> 	cython $< -o $@ -MD -MP
> include file.pyx.d file.cP
> $ : > file.pyx.d 
> $ : > file.cP     # these files are initially empty

Hmm, does that mean you either have to create them manually before the
first run, and/or you have to manually collect all dependency files for the
"include" list? And then keep track of them yourself when you add new files?

I suppose you could also use a wildcard file search to build the list of
include files?


> $ vim file.pyx    # write down some code using "file.pxi" somehow
> [..]
> $ vim file.pxi    # write down some code using <file.h> somehow
> [..]
> $ make file.so    # will find the .pyx -> .c -> .so chain
> cython file.pyx -o file.c -MD -MP # creates file.c and file.pyx.d
> [..]
> gcc -shared -fpic file.so -o file.so -M -MD -MP # creates file.{so,cP}
> [..]
> $ cat file.pyx.d   # the dependency makefile cython has created
> file.c: file.pxi
> file.pxi:
> $ cat file.cP # the dependency makefile gcc has written
> file.so: /path/to/file.h
> /path/to/file.h:
> $ make # doesnt do anything after checking timestamps.
> $ touch file.pyx; make # calls cython, gcc
> [..]
> $ touch /path/to/file.h; make # just calls gcc
> [..]
> $ touch file.pxi; make # calls both
> [..]
> 
> (lets hope that this is syntactically correct and half way comprehensible)
> 
> eventually, it (i.e. what i've implemented for sage) could look more like this:
> $ ./configure # creates Makefiles from templates
> $ make
> $ CYTH file.c
> $ CC   file.lo
> $ LD   file.so
> $ touch file.h; make
> $ CC   file.lo
> $ LD   file.so
> ...
> (automake will call the linker seperately, to increase portability or
> something)

If you want this feature to go in, I think you should write up
documentation for it, so that other people can actually use it as well.
Even writing a correct makefile requires a huge amount of digging into
distutils.

Here are examples for portable makefiles:

https://github.com/cython/cython/tree/master/Demos/embed

It would be nice to have a similar demo setup for a complete make build.

The textual documentation should go here:

http://docs.cython.org/src/reference/compilation.html

(see the .rst files in docs/src/)

Stefan