[Python-Dev] Packaging JIT-less versions of Python

[David Malcolm]

On Mon, 2010-03-01 at 15:35 -0800, Collin Winter wrote:
> Hey packaging guys,
> 
> We recently committed a change to Unladen Swallow [1] that moves all
> the JIT infrastructure into a Python extension module. The theory [2]
> behind this patch was that this would make it easier for downstream
> packagers to ship a JIT-less Python package, with the JIT compiler
> available via an optional add-on package.
> 
> Some questions for you, so we're not shooting blind here:
> - Have you guys thought about how a JIT-enabled Python 3 installation
> would be packaged by your respective distros?
I suspect that there may be architecture-specific issues both in getting
LLVM to work, and in getting the Python JIT code running (e.g. I've no
idea how well it works on, say, s390).  The answers may vary between CPU
architecture (e.g. "on by default for i386 and x86_64, off by default on
ppc", or whatever).


> - Would you prefer the default python3.x package to have a JIT, or
> would you omit the JIT by default?

I'd be inclined to turn it on by default in Fedora's "rawhide"
development branch as soon as it's meaningful to test it (with a
suitable warning on our mailing list); we like to try this kind of
thing!  If no major issues are found, I'd leave it turned on.  I'd
expect to run into "fun" CPU-arch-specific bugs, though.  Multicore
64-bit big-endian springs to mind.

I can't speak for what I'd do for RHEL at this time.  It's likely to be
affected by how experience in Fedora goes, but goes through more formal
QA, and on more "enterprisey" hardware (e.g. s390).


> - How would you prefer to build the JIT-less package (current options:
> via a ./configure flag; or by deleting _llvmjit.so from the
> JIT-enabled package)?
> - Would the two packages be able to exist side-by-side, or would they
> be mutually exclusive?

I have a particular interest in ABI compatibility: if turning JIT on and
off is going to change the ABI of extension modules, that would be a
major pain, as I hope that we will have dozens of C extension modules
available via RPM for our Python 3 stack by the time of the "great
unladen merger".

So I'm keen for the ability to toggle the JIT code in the face of bugs
and have it not affect ABI.  "-Xjit" will do this at runtime (once
that's renamed), but I think it would be useful to be able to toggle the
JIT on/off default during the build, so that I can fix a broken
architecture for non-technical users, but have individual testers opt
back in with -Xjit whilst tracking down a major bug.

In either case, I don't want to have to recompile 30 extension modules
to try with/without JIT; that would introduce too much change during
bug-hunts, and be no fun at all.


(In the blue-sky nirvana future, I'd love to be able to ship
ahead-of-time compiled versions of the stdlib, pre-optimized based on
realworld workloads.  Back in my reality, though, I have bugs to fix
before I can work on _that_ patch :( )


> My strong preference would be to have the JIT included by default so
> that it receives as much testing as possible.

Sounds reasonable.  Hope the above made sense and is useful.
Dave

> 
> Thanks,
> Collin Winter
> 
> [1] - http://code.google.com/p/unladen-swallow/source/detail?r=1110
> [2] - http://code.google.com/p/unladen-swallow/issues/detail?id=136