[Python-ideas] CPython needs to revamp/replace it's build system && Cross-compiling Python sucks

[Russell Keith-Magee]

Hi Ryan,

First off I'm no fan of autoconf, but...

On Tue, Mar 10, 2015 at 1:40 AM, Ryan Gonzalez <rymg19 at gmail.com> wrote:

> I've been trying to work on Issue 23496. And want to tear my hair out.
> I've noticed a few problems with Python's current build system:
>
> - There is no separation of the host C compiler and the target C compiler.
> When you give configure a C compiler for cross-compiling, make will happily
> try to build Parser/pgen with the cross-compiler. And try to run the
> generated executable. And fail. I know you can use `make touch` to get
> around this, but it still sucks.
>

True there is no separation of host/target compilers, but are you sure this
matters? In my cross-compiling for iOS, the pgen steps are skipped --
Parser/pgen is compiled, but not ever used. Are you sure that this failed
execution isn't an artefact of identifying the Android build a "linux"?
I've had to make a bunch of small modifications to configure.ac to make the
iOS build process not build and run things that won't work on the build
platform.


> - For some reason, autoconf decided to re-configure everything about 2
> minutes into make. But it didn't pass any of the flags I needed, resulting
> in everything failing.
>

That's the step compiling libffi/ctypes. libffi is vendored into the Python
source, and the mid-build configure step is the configuration of that
vendored source code. You'll see the mid-build configure execute regardless
of whether you're cross-compiling or not -- unless you're for OS/X, because
the build instructions for OS/X libffi are hard-coded into setup.py.


> - I do not like having to use this for configuring:
>
>    PYTHON_FOR_BUILD=`realpath ../cpython/python`
> CC=arm-linux-androideabi-clang CXX=arm-linux-androideabi-clang++
> ./configure --host=arm-linux-androideabi --build=x86_64 --disable-ipv6
> ac_cv_file__dev_ptmx=no ac_cv_file__dev_ptc=no
> ac_cv_have_long_long_format=yes
>
>
The PYTHON_FOR_BUILD specification *shouldn't* be necessary. If you're
compiling Python 3.X, as long as the build platform has a python3.X
executable on the path, PYTHON_FOR_BUILD should auto detect. It's only
necessary to manually specify PYTHON_FOR_BUILD if you *haven't* got an
appropriate python in your path.

ac_cv_file__dev_ptmx=no and ac_cv_file__dev_ptc=no are manually specified
flags because they can't be auto detected on a cross-compile (at least, not
in any way I can think of). They're checking whether the *target* device
has /dev/ptc and /dev/ptmx. In the case of Android/iOS, you could probably
hard-code the responses on the basis that the target platform is a known
quantity, but in the general case (which is the current case), you can't
auto detect the properties of the target platform.

I can't comment on ac_cv_have_long_long_format, but I'm guessing the cause
is the same - you can't compile and run the configure test program.

--disable-ipv6 is a feature flag - I don't know why it's necessary on
Android.

At the same time, I'm not quite sure how many people would want to sift
> through the 5000 lines in configure.ac or the 1650 lines in
> Makefile.pre.in.
>
> I just get the feeling that, whenever I build Python, there are too many
> holes/hacks to get around make/autoconf things.
>

Sure - configure sucks. What's the alternative?

Sometimes things are complex because... they're complex. This is a build
system that will support building everything from AiX to Android, and in
*most* cases, with little more than a ./configure && make && make install.
Android (and other cross compiling cases) are a little more complex because
you have to *tell* configure what the target looks like, because it can't
detect anything about the platform.

Yours,
Russ Magee %-)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150310/b08c26c5/attachment-0001.html>