Is setup.py a "good thing"? (long)

[Mike Clarkson]

setup.py was introduced in 2.1, presumably to help configure and
compile Python, so presumably it's a "good thing".  But I have an
uneasy feeling which I will try to explain. 

Perhaps it is just my lack of understanding of setup.py, or the
multitude of issues involved in building Python. If so, I apologize in
advance, and I welcome any light people can shed on the subject.

I know about configure and I know about make, and but I don't know
about setup.py - it's purpose is not clearly defined anywhere I can
see, so I worry that it's one of those things everyone can agree is a
"good thing" only because no one defined it's role, so it couldn't be
examined too closely. Because after all, the overlap between configure
and setup.py is significant, and that makes me uneasy.

It could be that setup.py is the first step in a programme for Python
to rule the world by gradually replacing configure. If so, I'm uneasy
because life is too short. Much as I hate configure (with a passion),
it's better to join it than to try and beat it. So I worry that
setup.py is duplicating what should really be in configure.

I believe a part of the reason for coming up with setup.py was to
facilitate the building of dynamically loaded extensions to Python,
which is a difficult problem for everyone: Tcl and Apache are two
other pieces of software that have faced this issue.

I'll frame my discussion in terms of adding a Tk extension to Tkinter,
which is what I was trying to do. I believe that the issues are not
Tkinter specific, and it serves as a concrete example of issues with
setup.py, and the path it leads us down. Or perhaps much of my
uneasiness is the way setup.py specifically deals with _tkinter.

Take for example compiling _tkinter, which needs to know where the tcl
libraries are. The cannonical way of doing this in Tcl would be to add
a --with-tcl configure option which lets you tell configure where the
Tcl files might be. Many modern Tcl programs use --with-tcl-config -
Tcl realized it was not just the locations of the includes and
libraries you needed to store, but also other compile settings like
--enable-shared  and --enable-stubs, because you must be coherent with
these settings too. So every tcl install generates, and every tcl
binary distribution comes with, a file tclConfig.sh which stores a
bunch of these things. Then extension configure scripts can be written
to look for these files (or be told with --with-tcl-config), and then
source them into its configure so the extension will be coherent.

setup.py uses a simplistic find_file which tries to find if the tcl
and tk libraries and include files are around. If it finds one, it
adds the include directory and library to the equivalent of
Modules/Setup and compiles _tkinter to be dynamically loading.
 
Problems:
#1)  It's black magic. I don't know where it looked or what it found.
#2) It's closed - I can't tell it where libtcl is unless I have the
courage and temerity to go hack on it (after all, maybe G wrote it!)
#3)  The algorithm is used to find things like libraries is not the
same as the one the OS will use to resolve dynamic linkages (ldd).
setup.py doesn't look at LD_LIBRARY_PATH or ld.so.cache for example.
#3b) The same is true of where Python will look to load .so files,
which I believe is partially determined by the linker -R flag, which
is set by configure, not setup.py.

Maybe you can see where I might be getting uneasy. 

Now assuming someone graciously explains to me all of the above, 
I'm still going to left feeling a little uneasy because:
#3c) The same is true of where *Tcl* will look to load .so files,
which I believe is partially determined by the linker -R flag, and
probably things like env(TCL_LIBRARY) and hence tclpkgPath.
#4) If setup.py finds a Tk extension library, it will add the WITH_FOO
flag to the compile, which will compile the extension statically, even
though the whole idea of setup.py was to facilitate the building of
dynamically loaded extensions to Python (this may just be a bug).
#5) setup.py is not dealing with one of the more important configure
options in Tcl --enable-stubs *with which is must be coherent.*
(BTW, I think Python should switch to the Tcl stubs API ASAP.)
#6) I'll have to foresake all of my setup.py hacks when I go back and
build again from scratch a static python to use freeze.py, for which I
must not use setup.py, but Modules/Setup instead. (It's problem
generic to any freezing, not just freeze.py in particular.)

Now all of this gets me thinking that the real reason was the setup.py
in the first place was the Tcl equivalent to the configure options
--enable-shared/--disable-shared, which at least makes it not black
magic because I know about 'configure --help' anyway. So I think I'd
feel less uneasy if:
#1) Someone documented what setup.py is supposed to do, and how it is
supposed to interact with configure and make.
#2) Python configure had the  configure options --enable-shared
--disable-shared so I could say explicitly what I want in the way of
static/dynamic. setup.py could then restrain itself to just looking in
the config.* files for anything it needs.  (See Apache for something
more elaborate on this point: configure --activate-module
--enable-module --disable-shared  unsw.).
#3) Python configure had the  configure options --with-tcl-config and
--with-tk-config so I could say explicitly where what I want is. This
allows Python configure to verify it is consistent with the Tcl
configure that was used to build the libraries I will be linking
against. This is relatively easy to do these days as almost all Tk
extensions have this, so that anyone who is good at configure (an
oxymoron?) can cut and paste one into the Python configure.
#4) While the oxymoron was at it, he/she/it could  cut and paste in
the section for --enable-stubs, which would be a "good thing".

Maybe I can ask the people who have a lot of experience with this
problem in Tcl like Jeff Hobbes and Cameron Laird to comment,
and hopefully lay my fears to rest.

Yours uneasi-ly,

Mike