[Python-ideas] CPython needs to revamp/replace it's build system && Cross-compiling Python sucks

[Ryan]

Russell Keith-Magee <russell at keith-magee.com> wrote:
>Hi Ryan,
>
>First off I'm no fan of autoconf, but...
>

Who is other than the one who designed it? :)

>On Tue, Mar 10, 2015 at 1:40 AM, Ryan Gonzalez <rymg19 at gmail.com>
>wrote:
>
>> I've been trying to work on Issue 23496. And want to tear my hair
>out.
>> I've noticed a few problems with Python's current build system:
>>
>> - There is no separation of the host C compiler and the target C
>compiler.
>> When you give configure a C compiler for cross-compiling, make will
>happily
>> try to build Parser/pgen with the cross-compiler. And try to run the
>> generated executable. And fail. I know you can use `make touch` to
>get
>> around this, but it still sucks.
>>
>
>True there is no separation of host/target compilers, but are you sure
>this
>matters? In my cross-compiling for iOS, the pgen steps are skipped --
>Parser/pgen is compiled, but not ever used. Are you sure that this
>failed
>execution isn't an artefact of identifying the Android build a "linux"?
>I've had to make a bunch of small modifications to configure.ac to make
>the
>iOS build process not build and run things that won't work on the build
>platform.
>

Hmmm...

But you gave an example. You shouldn't need to edit configure.ac to not try to run what can't be run because the host != the builder.

>
>> - For some reason, autoconf decided to re-configure everything about
>2
>> minutes into make. But it didn't pass any of the flags I needed,
>resulting
>> in everything failing.
>>
>
>That's the step compiling libffi/ctypes. libffi is vendored into the
>Python
>source, and the mid-build configure step is the configuration of that
>vendored source code. You'll see the mid-build configure execute
>regardless
>of whether you're cross-compiling or not -- unless you're for OS/X,
>because
>the build instructions for OS/X libffi are hard-coded into setup.py.
>

Still sucks.

>
>> - I do not like having to use this for configuring:
>>
>>    PYTHON_FOR_BUILD=`realpath ../cpython/python`
>> CC=arm-linux-androideabi-clang CXX=arm-linux-androideabi-clang++
>> ./configure --host=arm-linux-androideabi --build=x86_64
>--disable-ipv6
>> ac_cv_file__dev_ptmx=no ac_cv_file__dev_ptc=no
>> ac_cv_have_long_long_format=yes
>>
>>
>The PYTHON_FOR_BUILD specification *shouldn't* be necessary. If you're
>compiling Python 3.X, as long as the build platform has a python3.X
>executable on the path, PYTHON_FOR_BUILD should auto detect. It's only
>necessary to manually specify PYTHON_FOR_BUILD if you *haven't* got an
>appropriate python in your path.

The build was giving errors because apparently 3.4's _sre module doesn't have some constant that the build script needs (MAXGROUPS).

>
>ac_cv_file__dev_ptmx=no and ac_cv_file__dev_ptc=no are manually
>specified
>flags because they can't be auto detected on a cross-compile (at least,
>not
>in any way I can think of). They're checking whether the *target*
>device
>has /dev/ptc and /dev/ptmx. In the case of Android/iOS, you could
>probably
>hard-code the responses on the basis that the target platform is a
>known
>quantity, but in the general case (which is the current case), you
>can't
>auto detect the properties of the target platform.

It took me a while to figure out that that was the option I needed to pass, though.

>
>I can't comment on ac_cv_have_long_long_format, but I'm guessing the
>cause
>is the same - you can't compile and run the configure test program.
>
>--disable-ipv6 is a feature flag - I don't know why it's necessary on
>Android.

I don't really know why, either. Something to do with a getaddrline bug.

>
>At the same time, I'm not quite sure how many people would want to sift
>> through the 5000 lines in configure.ac or the 1650 lines in
>> Makefile.pre.in.
>>
>> I just get the feeling that, whenever I build Python, there are too
>many
>> holes/hacks to get around make/autoconf things.
>>
>
>Sure - configure sucks. What's the alternative?

All 1000 other build systems out there?

>
>Sometimes things are complex because... they're complex. This is a
>build
>system that will support building everything from AiX to Android, and
>in
>*most* cases, with little more than a ./configure && make && make
>install.
>Android (and other cross compiling cases) are a little more complex
>because
>you have to *tell* configure what the target looks like, because it
>can't
>detect anything about the platform.

But autotools tends to make things more complex than needed. When Qt (or was it KDE?) moved to CMake from autotools, the build script size dropped a large amount.

Just take a look at the code in configure.ac that manually detects the compiler on OSX systems because autoconf SOMEHOW manages to get it wrong.

>
>Yours,
>Russ Magee %-)

-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.
Check out my website: http://kirbyfan64.github.io/