Python Extension Building Network

[kyosohma at gmail.com]

On Nov 10, 11:03 pm, Yu-Xi Lim <y... at ece.gatech.edu> wrote:
> kyoso... at gmail.com wrote:
> > Hi,
>
> > I am trying to get a small group of volunteers together to create
> > Windows binaries for any Python extension developer that needs them,
> > much like the package/extension builders who volunteer their time to
> > create Linux RPMs.
>
> > Mike
>
> It's not entirely clear to me what you're compiling. Most of the modules
> you have there do not have native code. I'm guessing your main focus is
> building .exe installers for Windows users.


The main objective is to make it easier for Windows users to install
the modules, so that's what the .exe is for. Admittedly developers
should know how to install from source, but the newbs don't always
know how and I've had some trouble with some of the more complex ones
myself.


>
> If it's just installers of pure Python code, why the extra effort of
> using both VS and MingW? AFAIK, they both yield functionally identical
> results---the exact same .py files get installed.
>


I am using both because Steve Holden asked me to try MinGW. I was just
going to use Visual Studio. However, I thought it might be a good idea
to try it both ways to make sure it could be done in a completely open
source environment. I'm pretty sure you can use MinGW on impure Python
extensions too, although it takes more work.


> I'm also not sure how big a task this is or the issues involved, but
> here's my take based on what I've read:
>
> Throwing more manpower at the task is not the solution. The actual
> process can be and SHOULD BE highly automated. All that's needed are a
> proper distutils script, and proper tests. Both responsibilities should
> fall on the shoulders of the module authors, though I guess experienced
> volunteers can help out with the former.
>


I would like to do this and I was trying to come up with a way to do
just that. For some reason, when I mentioned that idea to one of my
more knowledgeable contacts, he didn't see the need for it.


> If the module is pure Python, there should be little need for testing
> the installer specifically on Windows. If there's a failure, then the
> module itself is buggy (e.g. making platform-specific assumptions) or
> there's similar problem with the install script.
>
> For a large number of modules on PyPI, building the installer is trivial
> (assuming pure Python, and a setup script). Compiling them can be easily
> automated: 1) check PyPI for updates 2) download the latest update 3)
> build the .exe 3) upload to a website. All that remains is the need for
> some hardware (a build farm) and the occasional manual intervention.
>
> If I'm not mistaken, distutils can build Windows installers on other
> platforms. Maybe you could set up a Linux LiveCD with the necessary
> tools installed and distribute that. Module authors who want to build
> Windows installers can easily use that CD. Volunteers wanting to help
> can do so easily without having to repeat the task of setting up the
> toolchain separately.


I've been told that distutils can build Windows installers on other
platforms, but I think that may apply only to pure Python extensions
only.


>
> I've done some work as a port maintainer (aka package maintainer) for
> FreeBSD. The BSD port system is somewhat closer to PEAK's Easy Install,
> however binary packages (similar to RPMs or DEBs) are also available.
> The port maintainer's job is solely to ensure that the Makefile (build
> script) is working and up to date. The actual testing of the code is
> done by the authors of the software itself and the port maintainer only
> ensures that the install script works right. Binary packages are built
> by an automated system using a cluster of computers for various
> architectures and supported OS versions. Errors during builds are sent
> to to port maintainers.
>
> A similar system is used for most Linux distributions with a central
> repository for packages.

I like this idea quite a bit. It's nice to know that other people are
thinking along the same lines as I am. However, I am not sure how to
do this myself. I assume when you mean by creating a Linux Live CD
with the proper tools, you mean that it should include MinGW and
Python. I suppose the only problem with that is the dependency issue.

There's a number of modules that require arbitrary modules for the
setup.py file to run. Mechanize is one such module as it require
ClientForm. Some of the others I've done required various versions of
Easy Setup or ElementTree. I'm not sure how this is resolved unless
you can just download these items during the Live CD session and
install them to memory. However, I don't think this is currently
possible, correct?

I'll float your ideas by Holden and see what he thinks though. Thanks
for the ideas.

Mike