[Distutils] symlinks in sourcetree sdist breaking easy_install?

[John Hunter]

We are experiencing a problem related to symbolic links in our source
tree/sdist that appears to breaking easy_install (using
setuptools-0.6c9).  To produce the bug::

    jdhunter at bic128:~>
PYTHONPATH=~/dev/lib/python2.5/site-packages:/home/jdhunter/devez/lib/python2.5/site-packages
~/dev/bin/easy_install --prefix=~/devez matplotlib
    Searching for matplotlib
    Reading http://pypi.python.org/simple/matplotlib/
    Reading http://matplotlib.sourceforge.net
    Reading https://sourceforge.net/project/showfiles.php?group_id=80706&package_id=278194
    Reading https://sourceforge.net/project/showfiles.php?group_id=80706&package_id=82474
    Reading http://sourceforge.net/project/showfiles.php?group_id=80706
    Best match: matplotlib 0.98.5
    Downloading
http://downloads.sourceforge.net/matplotlib/matplotlib-0.98.5.tar.gz?modtime=1229034572&big_mirror=0
    Processing matplotlib-0.98.5.tar.gz
    Running matplotlib-0.98.5/setup.py -q bdist_egg --dist-dir
/tmp/easy_install-qW11xj/matplotlib-0.98.5/egg-dist-tmp-QFgBsw
    ..snip snip
    [Edit setup.cfg to suppress the above messages]
    ============================================================================
    error: lib/matplotlib/mpl-data/matplotlib.conf.template: No such
file or directory
    Exception exceptions.OSError: (2, 'No such file or directory',
'src/image.cpp') in <bound method CleanUpFile.__del__ of
<setupext.CleanUpFile instance at 0x7f022cf378c0>> ignored
    Exception exceptions.OSError: (2, 'No such file or directory',
'src/path.cpp') in <bound method CleanUpFile.__del__ of
<setupext.CleanUpFile instance at 0x7f022cf35d40>> ignored
    Exception exceptions.OSError: (2, 'No such file or directory',
'src/backend_gdk.c') in <bound method CleanUpFile.__del__ of
<setupext.CleanUpFile instance at 0x7f022ab83e18>> ignored
    Exception exceptions.OSError: (2, 'No such file or directory',
'src/backend_agg.cpp') in <bound method CleanUpFile.__del__ of
<setupext.CleanUpFile instance at 0x7f022cf37710>> ignored
    jdhunter at bic128:~>


The file which triggers the first error is matplotlib.conf.template,
which exists in lib/matplotlib/mpl-data.  There is a symlink in
doc/mpl_data which points to lib/matplotlib/mpl-data.  Eg in the mpl
src tree::

    jdhunter at bic128:mpl> ls -ld lib/matplotlib/mpl-data/ doc/mpl_data
    lrwxrwxrwx 1 jdhunter jdhunter   27 Jun  2  2008 doc/mpl_data ->
../lib/matplotlib/mpl-data/
    drwxrwxr-x 6 jdhunter jdhunter 4096 Dec 10 13:09 lib/matplotlib/mpl-data/

When we create the sdist, here is the output matching that file::

    jdhunter at bic128:mpl> python setup.py sdist > sdist.out
    jdhunter at bic128:mpl> grep matplotlib.conf.template sdist.out
    hard linking doc/mpl_data/matplotlib.conf.template ->
matplotlib-0.98.5/doc/mpl_data
    hard linking lib/matplotlib/mpl-data/matplotlib.conf.template ->
matplotlib-0.98.5/lib/matplotlib/mpl-data

In the tarball, the link looks backwards to me: the tarball says
lib/matplotlib/mpl-data/matplotlib.conf.template links to
doc/mpl_data/matplotlib.conf.template and has zero content

    jdhunter at bic128:tmp> tar tvfz matplotlib-0.98.5.tar.gz |grep
matplotlib.conf.template
    -rw-r--r-- cmoad/staff   13838 2008-12-09 17:53
matplotlib-0.98.5/doc/mpl_data/matplotlib.conf.templatehrw-r--r--
cmoad/staff       0 2008-12-09 17:53
matplotlib-0.98.5/lib/matplotlib/mpl-data/matplotlib.conf.template
link to matplotlib-0.98.5/doc/mpl_data/matplotlib.conf.template


If you unpack the tarball using a standard incantation, a copy is made::

    jdhunter at bic128:tmp> tar xfz matplotlib-0.98.5.tar.gz
    jdhunter at bic128:tmp> cd matplotlib-0.98.5/
    jdhunter at bic128:matplotlib-0.98.5> ls -ld
doc/mpl_data/matplotlib.conf.template
lib/matplotlib/mpl-data/matplotlib.conf.template
    -rw-r--r-- 2 jdhunter jdhunter 13838 Dec  9 17:53
doc/mpl_data/matplotlib.conf.template
    -rw-r--r-- 2 jdhunter jdhunter 13838 Dec  9 17:53
lib/matplotlib/mpl-data/matplotlib.conf.template

I think (guess) what may be happening is that setuptools are reading the
tarball using the python tarfile reader and the link is getting
botched.  However, I *can* read the "missing" file using tarfile::

    In [17]: import tarfile

    In [18]: t = tarfile.open('dist/matplotlib-0.98.5.tar.gz')

    In [19]: o =
t.extractfile('matplotlib-0.98.5/lib/matplotlib/mpl-data/matplotlib.conf')

    In [20]: s = o.read()

    In [21]: print s[:80]
    # MPLConfig - plaintext (in .conf format)

    # This is a sample matplotlib configu

So I am not sure exactly what is going wrong but I think broken
handling of the link is playing a role.  I will probably work around
this by removing the links and making hard copies myself before
building the sdist and eggs, unless I hear something more intelligent
from someone or come up with something better.