[Distutils] First steps with distutils...

[Greg Ward]

On 04 September 2000, M.-A. Lemburg said:
> > Correct.  MANIFEST is a simple list of files, one per line.  If you want
> > to include whole directories, you should write a MANIFEST.in file.
> 
> I didn't mean that distutils should scan the goven directory --
> it should simply ignore it instead of raising an exception.

Hmm, are you really getting an exception?  When I put a directory name
into a MANIFEST file, I get:

  error: can't copy 'pkg': doesn't exist or not a regular file

and immediate termination of the setup script (hey, it's an error, not a 
warning).  I'll admit that's not as useful as it could be, but as long
as there's no traceback it's not a bug!  ;-)

Actually, I spent some time documenting the "sdist" command today, and
tracked down some bugs in the handling of the manifest files.  I also
now understand the rules a bit better -- amazing what reading the code
will tell you.  ;-)  Here are the rules as implemented in the current
code (not even checked in yet, because of the 2.0b1 code freeze):

    cases:
      1) no manifest, template exists: generate manifest
         (covered by 2a: no manifest == template newer)
      2) manifest & template exist:
         2a) template or setup script newer than manifest:
             regenerate manifest
         2b) manifest newer than both:
             do nothing (unless --force or --manifest-only)
      3) manifest exists, no template:
         do nothing (unless --force or --manifest-only)
      4) no manifest, no template: generate w/ warning ("defaults only")

In the 0.9.2 code (and all Distutils releases since at least 0.2), case
(4) wasn't handled properly.  A workaround is to use the -f
(--force-manifest) option, which causes the sdist command to
unconditionally regenerate the MANIFEST file.

Also, case (3) implies that the "default file set" -- setup.py,
README.txt, and any source code specified in the setup script -- will
*not* be used if you supply your own MANIFEST file.  Additionally, the
default "prune list" -- which strips RCS and CVS directories, as well as
Distutils-generated temporary directories -- does not apply.  IOW, if
you generate your own MANIFEST, you have to get it exactly right.  I
think this is the right thing to do, because only control freaks will
want to supply their own MANIFEST file, and taking away any control from
control freaks makes them, well, freak out.  ;-)

See the 'get_file_list()' method in distutils/command/sdist.py if you're
curious about the implementation.  (And see if you can spot the bug that
I fixed this morning.)  (No fair watching python-checkins!)

> I'm currently using my own tools for generating the MANIFEST
> file. The most important difference is that they allow per
> directory MANIFEST.in style files which are appended to the
> general MANIFEST.in logic while scanning the directory.

Hmmm, another good reason to roll your own manifest -- maybe it's not
just for control freaks?

> My version if rpm 3.0.3.
[...]
> I also noted another bug:
> When building an RPM which contains more than one Extension(),
> bdist_rpm fails on the second Extension(): it can't find
> the C file.
> """
> running build_ext
> building 'mx.DateTime.mxDateTime.mxDateTime' extension
> creating build/temp.linux2
> creating build/temp.linux2/mx
> creating build/temp.linux2/mx/DateTime
> creating build/temp.linux2/mx/DateTime/mxDateTime
> gcc -g -O2 -fpic -Imx/DateTime/mxDateTime -I/usr/local/include/python2.0 -c mx/DateTime/mxDateTime/mxDateTime.c -o build/temp.linux2/mx/DateTime/mxDateTime/mxDateTime.o -O2 -m486 -fno-strength-reduce
> creating build/lib.linux2/mx/DateTime
> creating build/lib.linux2/mx/DateTime/mxDateTime
> gcc -shared build/temp.linux2/mx/DateTime/mxDateTime/mxDateTime.o -o build/lib.linux2/mx/DateTime/mxDateTime/mxDateTime.so
> building 'mx.Proxy.mxProxy.mxProxy' extension
> error: file 'mx/Proxy/mxProxy/mxProxy.c' does not exist
> Bad exit status from /var/tmp/rpm-tmp.85854 (%build)
> error: command 'rpm' failed with exit status 1
> """

???  Very weird.  Does "python setup.py build" work on its own, ie. when 
run by you rather than by rpm in its build directory?

One simple possibility is that you forgot to include
mx/Proxy/mxProxy/mxProxy.c in your tarball -- make sure that you can
unpack your tarball in a fresh directory and run "python setup.py
build", which is just what rpm is doing.

> > For a straight Distutil build (ie. no RPM involved), setting CFLAGS
> > should work, but it only applies to extensions (not C libraries).  When
> > RPM is involved, I'm not sure what the Right Way to do it is.  Maybe
> > CFLAGS will work?  Maybe editing /etc/rpmrc or /usr/lib/rpm/rpmrc?  I
> > dunno.
> 
> Wouldn't an option to distutils be the right way for these kind
> of things ?

Yes.  Definitely.  It's on the TODO list.

Now, on to your other "bdist_rpm" failure...

> setup.py bdist_rpm output:
> 
> rpm -ba --define _topdir /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm --clean build/bdist.linux2/rpm/SPECS/mx-Extensions-BASE.spec
> Executing: %prep
[...]
> Wrote: /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/SRPMS/mx-Extensions-BASE-1.0.0-1.src.rpm
> Could not open /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/RPMS/i386/mx-Extensions-BASE-1.0.0-1.i386.rpm
> 
> error: command 'rpm' failed with exit status 1

Hmmm, this smells like the archetypal Distutils bug: someone's trying to
write a file into a directory that doesn't exist yet.  In this case, it
would be rpm's fault, since the "bdist_rpm" command doesn't know the
name of the architecture directory that rpm will try to write to.
Things to look for:
  * does /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/RPMS/i386
    exist?
  * if you create it, does the bdist_rpm command then run successfully?

I haven't had problems with rpm failing to create directories -- I have
RPM 3.0.4 on my Red Hat 6.2 box, and I just cranked out an RPM of
mxDateTime 1.3.0 without a hitch.  Perhaps RPM 3.0.3 joins RPM 2.x on
the index of forbidden versions... sigh...

        Greg
-- 
Greg Ward - Unix bigot                                  gward@python.net
http://starship.python.net/~gward/
If at first you don't succeed, redefine success.