[Cython] Utilities, cython.h, libcython

[mark florisson]

Hey,

I briefly mentioned something about this in a pull request, but maybe
it deserves some actual discussion on the ML.

So I propose that after fused types gets merged we try to move as many
utility codes as possible to their utility code files (unless they are
used in pending pull requests or other branches). Preferably this will
be done in one or a few commits. How should we split up the work, any
volunteers? Perhaps people who wrote certain utilities also want to
move them? In that case, we should start a new branch and then merge
that into master when it's done.
We could actually move things before fused types get merged, as long
as we don't touch binding_cfunc_utility_code.

Before we go there, Stefan, do we still want to implement the header
.ini style which can list dependencies and such? I personally don't
care very much about it, but memoryviews and the utility loaders are
merged so if someone wants to take up that job, it'd be good to do
before moving the utilities.

Another issue is that Cython compile time is increasing with the
addition of control flow and cython utilities. If you use fused types
you're also going to combinatorially add more compile time. I'm sure
this came up earlier, but I really think we should have a libcython
and a cython.h. libcython (a shared library) should contain any common
Cython-specific code not meant to be inlined, and cython.h any types,
macros and inline functions etc. This will decrease Cython and C
compile time, and will also make executables smaller. This could be
enabled using a command line option to Cython, as well as with
distutils, eventually we may decide to make it the default (lets
figure that out later). Preferably libcython.so would be installed
alongside libpython.so and cython.h inside the Python include
directory. Assuming multiple versions of Cython and multiple Python
installations, we'd need to come up with a versioning scheme for
either.
We could also provide a static library there, for users who want to
link and ship a compiled and statically linked version of their code.
For a local Cython that isn't built, we can ignore the header and
shared library option and issue a warning or some such.

Lastly, I think we also should figure out a way to serialize Entry
objects from CythonUtilities, which could easily and swiftly be loaded
when creating the cython scope. It's quite a pain to declare all
entries for utilities you write manually, so what I mostly did was
parse the utility up to and including AnalyseDeclarationsTransform,
and then retrieve the entries from there.

Thoughts?

Mark