[Distutils] buildout/setuptools slow as it scans the whole project dir

[Reinout van Rees]

Hi,

In some of my projects, buildout takes a looooooong time to complete. 
Sometimes the same problem occurs on the server or on another 
developer's laptop, sometimes not. The difference is something like "15 
seconds if there's no problem" and "5 minutes if the problem occurs".

It is terribly hard to pinpoint the exact problem and/or cause. An error 
in the setup.py or or MANIFEST.in is unlikely, as the very same project 
might take 5 minutes locally and 15 seconds on the server or vice versa...

Anyway, if I run "bin/buildout" as "strace -f bin/buildout", I see a lot 
of "stat" calls. Setuptools walks through all the files inside my 
project dir. Including parts/omelette/* and, if available, a 
bower_compontents/ directory full of thousands of javascript files. 
Removing some of these directories (which running the buildout 
re-creates) fixes the speed issue for one run.

I modified my local buildout copy to run "setup.py develop" with a -v 
instead of a -q option. This way I found out where it approximately happens:


/usr/bin/python /tmp/tmp6UdsMl -v develop -mxN -d 
/vagrant/sso/develop-eggs/tmpfioc1Ibuild
running develop
running egg_info
writing requirements to sso.egg-info/requires.txt
writing sso.egg-info/PKG-INFO
writing top-level names to sso.egg-info/top_level.txt
writing dependency_links to sso.egg-info/dependency_links.txt
writing entry points to sso.egg-info/entry_points.txt

### This is where the process seems to stop a couple of minutes to scan 
all the files.

reading manifest file 'sso.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'sso.egg-info/SOURCES.txt'
running build_ext
Creating /vagrant/sso/develop-eggs/tmpfioc1Ibuild/sso.egg-link (link to .)



The MANIFEST.in loooks like this:

# Include docs in the root.
include *.rst
# Include everything in our project directory (sso/views.py, 
sso/static/some.js, etc)
graft sso


It is a git project. The setup.py looks like this:


from setuptools import setup

version = '1.1.dev0'

long_description = '\n\n'.join([
     open('README.rst').read(),
     open('CREDITS.rst').read(),
     open('CHANGES.rst').read(),
     ])

install_requires = [
     'Django >= 1.4.2, < 1.7',
     'django-nose',
     'lizard-auth-server',
     'gunicorn',
     'raven',
     'werkzeug',
     'south',
     'django-auth-ldap',
     'django-mama-cas',
     ],

setup(name='sso',
       version=version,
       description="Single sign on server (and more) for lizard",
       long_description=long_description,
       # Get strings from 
http://www.python.org/pypi?%3Aaction=list_classifiers
       classifiers=['Programming Language :: Python',
                    'Framework :: Django',
                    ],
       keywords=[],
       author='Do not blame Reinout',
       author_email='reinout.vanrees at nelen-schuurmans.nl',
       url='',
       license='GPL',
       packages=['sso'],
       zip_safe=False,
       install_requires=install_requires,
       entry_points={
           'console_scripts': [
           ]},
       )



Conclusion for me: something somewhere in setuptools is reading my whole 
project folder. It does it after "writing entry points to 
sso.egg-info/entry_points.txt" and before "reading manifest file 
'sso.egg-info/SOURCES.txt'". The SOURCES.txt itself is small:


CHANGES.rst
CREDITS.rst
LICENSE.rst
MANIFEST.in
README.rst
setup.cfg
setup.py
sso/__init__.py
sso/__init__.pyc
sso/admin.py
sso/developmentsettings.py
sso/developmentsettings.pyc
sso/models.py
sso/models.pyc
sso/settings.py
sso/settings.pyc
sso/stagingsettings.py
sso/tests.py
sso/urls.py
sso/views.py
sso.egg-info/PKG-INFO
sso.egg-info/SOURCES.txt
sso.egg-info/dependency_links.txt
sso.egg-info/entry_points.txt
sso.egg-info/not-zip-safe
sso.egg-info/requires.txt


Hm. I see some .pyc files in there. Something that needs fixing, but not 
the cause, I think.

Is there something I'm missing? Where should I look? I cannot find 
"writing entry points to..." in the setuptools source code right away.



Reinout

-- 
Reinout van Rees                          http://reinout.vanrees.org/
reinout at vanrees.org                   http://www.nelen-schuurmans.nl/
"Learning history by destroying artifacts is a time-honored atrocity"