[Cython] Utilities, cython.h, libcython

[Robert Bradshaw]

On Tue, Oct 4, 2011 at 2:19 PM, mark florisson
<markflorisson88 at gmail.com> wrote:
> Hey,
>
> I briefly mentioned something about this in a pull request, but maybe
> it deserves some actual discussion on the ML.
>
> So I propose that after fused types gets merged we try to move as many
> utility codes as possible to their utility code files (unless they are
> used in pending pull requests or other branches). Preferably this will
> be done in one or a few commits. How should we split up the work, any
> volunteers? Perhaps people who wrote certain utilities also want to
> move them? In that case, we should start a new branch and then merge
> that into master when it's done.
> We could actually move things before fused types get merged, as long
> as we don't touch binding_cfunc_utility_code.

+1 to moving towards this, but I don't see the urgency or need to do
it all at once (though if there's going to be a big push, lets
coordinate on a wiki or trac).

> Before we go there, Stefan, do we still want to implement the header
> .ini style which can list dependencies and such? I personally don't
> care very much about it, but memoryviews and the utility loaders are
> merged so if someone wants to take up that job, it'd be good to do
> before moving the utilities.
>
> Another issue is that Cython compile time is increasing with the
> addition of control flow and cython utilities. If you use fused types
> you're also going to combinatorially add more compile time.

Yeah, this was especially obvious with, e.g. cython.compile(...). (In
particular, some utility code was being parsed before it could even
figure out whether it needed to do a full re-compile...)

> I'm sure
> this came up earlier, but I really think we should have a libcython
> and a cython.h. libcython (a shared library) should contain any common
> Cython-specific code not meant to be inlined, and cython.h any types,
> macros and inline functions etc. This will decrease Cython and C
> compile time, and will also make executables smaller.

+1. Yes, we talked about this earlier, but nothing concrete was
planned. It's probably worth a CEP, if anything to have a concrete
plan recorded somewhere other than a series of mailing list threads
(though discussion tends to work best here).

> This could be
> enabled using a command line option to Cython, as well as with
> distutils, eventually we may decide to make it the default (lets
> figure that out later). Preferably libcython.so would be installed
> alongside libpython.so and cython.h inside the Python include
> directory. Assuming multiple versions of Cython and multiple Python
> installations, we'd need to come up with a versioning scheme for
> either.

I would propose a cython.h file that sits in Cython/Compiler/Include
(or similar), as a first step. The .pyx -> .c pass could be configured
to copy this to a specific location (for shipping just the generated
.c files).

One option is to build the shared library as a companion
_cython_x_y_z.so module which, while not as efficient as linking at
the C level, would probably be much simpler to implement in a
cross-platform way. (This perhaps merits some benchmarks, but the main
contents is likely to be things like shared classes and objects.)
Actually linking .so files from modules that cimport each other would
be a nice feature down the road anyways. Again, the associated .c file
could be (optionally) generated/copied during the .pyx -> .c step.
Installation would determine if the required module exists, and if not
build and install it.

> We could also provide a static library there, for users who want to
> link and ship a compiled and statically linked version of their code.
> For a local Cython that isn't built, we can ignore the header and
> shared library option and issue a warning or some such.
>
> Lastly, I think we also should figure out a way to serialize Entry
> objects from CythonUtilities, which could easily and swiftly be loaded
> when creating the cython scope. It's quite a pain to declare all
> entries for utilities you write manually, so what I mostly did was
> parse the utility up to and including AnalyseDeclarationsTransform,
> and then retrieve the entries from there.

This would be really nice too. Way back in the day I did some work
with trying to pickle full module scopes, but that soon became too
painful as there are so many far-reaching references. Pickling
individual Entries and re-building modules will probably be a more
tractable goal. Eventually, I'd like to see a way to cache the full
pxd pipeline.

- Robert