From chester_lab at fltg.net  Sun Jan  1 03:46:53 2012
From: chester_lab at fltg.net (CT)
Date: Sat, 31 Dec 2011 21:46:53 -0500
Subject: [Distutils] Problem Using Python 2.7 and Py2Exe with Pygame
	Module
References: <366_1325251790_4EFDBCCD_366_346_1_716BC830FA9A48FD8516F31AAC21DA6E@bruce956f5b965>
Message-ID: <B96801D5EEAD4E2A94AC0CB8EF54990C@bruce956f5b965>


Hi Again,

    I changed the error that happened on the compiled version which was line 
26 where I was calling a import pygame.mixer. It states at that point it can 
ot find a DLL when I run the compiled py2exe made .exe file.
    So, I changed it to not import single modules and the error below 
happened on compile. So, no .exe was even made just by changing one line.
Note:
    This program, PlayMusic.py or Playing.py, or any uncompiled version runs 
perfectly as I designed it by clicking on it or running it from the command 
line.
    The error below comes up by only importing Pygame without single imports 
such as Pygame.Mixer where it will compile but fails when running it, I mean 
running the compiled .exe file.

    Inside my batch file I check for the .exe file and if not created I 
state "No Compile" at the very end.
        Bruce

Failure when importing Pygame without specifics:
This is the last step when attempting to copy DLLS
*** copy dlls ***
copying c:\python27\lib\site-packages\pygame\SDL_mixer.dll -> C:\Make\dist
copying C:\WINDOWS\system32\python27.dll -> C:\Make\dist
setting sys.winver for 'C:\Make\dist\python27.dll' to 'py2exe'
Traceback (most recent call last):
  File "SetUp4.py", line 107, in <module>
    'script':dest,
  File "c:\python27\lib\distutils\core.py", line 152, in setup
    dist.run_commands()
  File "c:\python27\lib\distutils\dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "c:\python27\lib\distutils\dist.py", line 972, in run_command
    cmd_obj.run()
  File "c:\python27\lib\site-packages\py2exe\build_exe.py", line 243, in run
    self._run()
  File "c:\python27\lib\site-packages\py2exe\build_exe.py", line 312, in 
_run
    self.create_binaries(py_files, extensions, dlls)
  File "c:\python27\lib\site-packages\py2exe\build_exe.py", line 525, in 
create_
binaries
    self.copy_dlls(dlls)
  File "c:\python27\lib\site-packages\py2exe\build_exe.py", line 471, in 
copy_dl
ls
    self.patch_python_dll_winver(dst)
  File "c:\python27\lib\site-packages\py2exe\build_exe.py", line 1008, in 
patch_
python_dll_winver
    add_resource(unicode_name, mfest, RT_MANIFEST, 2, False)
RuntimeError: EndUpdateResource: The system cannot open the device or file 
specified.

Error! No Compile!

Sent: Friday, December 30, 2011 8:19 AM
Subject: [Distutils] Problem Using Python 2.7 and Py2Exe with Pygame Module



Hi!

    I am getting errors and missing modules now that I have loaded Python2.7
and the Pygame modules, but works fine on my old machine using Python2.5

    I would like to know what is going wrong here? I also looked on the web
and lots of people are having the same problems. Does this mean that Python
2.7 is not wise to use along with Py2Exe when using Pygame modules?

Below is the errors and at the end is my Setup file:

*** copy extensions ***
*** copy dlls ***
copying c:\python27\lib\site-packages\py2exe\run.exe ->
C:\Make\dist\MusicPlayer.exe
The following modules appear to be missing
['AppKit', 'Foundation', 'Numeric', 'OpenGL.GL', '_scproxy', 'copyreg',
'dummy.Process', 'numpy', 'pkg_resources', 'queue', 'winreg',
'pygame.sdlmain_osx']

*** binary dependencies ***
Your executable(s) also depend on these dlls which are not included,
you may or may not need to distribute them.

Make sure you have the license if you distribute any of them, and
make sure you don't distribute files belonging to the operating system.

   OLEAUT32.dll - C:\WINDOWS\system32\OLEAUT32.dll
   USER32.dll - C:\WINDOWS\system32\USER32.dll
   SHELL32.dll - C:\WINDOWS\system32\SHELL32.dll
   ole32.dll - C:\WINDOWS\system32\ole32.dll
   WINMM.DLL - C:\WINDOWS\system32\WINMM.DLL
   ADVAPI32.DLL - C:\WINDOWS\system32\ADVAPI32.DLL
   WS2_32.DLL - C:\WINDOWS\system32\WS2_32.DLL
   GDI32.dll - C:\WINDOWS\system32\GDI32.dll
   libogg-0.dll - c:\python27\lib\site-packages\pygame\libogg-0.dll
   KERNEL32.dll - C:\WINDOWS\system32\KERNEL32.dll
   SDL_ttf.dll - c:\python27\lib\site-packages\pygame\SDL_ttf.dll
The MusicPlayer.exe Has Been Made And Is Located In The Dist Directory!

Setup.py File:
PY_PROG = "music.py"

APP_NAME = "MusicPlayer"

cfg = {
    'name':APP_NAME,
    'version':'1.0',
    'description':'',
    'author':'',
    'author_email':'',
    'url':'',

    'py2exe.target':'',
#    'py2exe.icon':'icon.ico', #64x64
    'py2exe.binary':APP_NAME, #leave off the .exe, it will be added

    'py2app.target':'',
    'py2app.icon':'icon.icns', #128x128

    'cx_freeze.cmd':'~/src/cx_Freeze-3.0.3/FreezePython',
    'cx_freeze.target':'',
    'cx_freeze.binary':APP_NAME,
    }

# usage: python setup.py command
#
# sdist - build a source dist
# py2exe - build an exe
# py2app - build an app
# cx_freeze - build a linux binary (not implemented)
#
# the goods are placed in the dist dir for you to .zip up or whatever...

from distutils.core import setup, Extension
try:
    import py2exe
except:
    pass

import sys
import glob
import os
import shutil

try:
    cmd = sys.argv[1]
except IndexError:
    print 'Usage: setup.py py2exe|py2app|cx_freeze'
    raise SystemExit

# utility for adding subdirectories
def add_files( dest, generator):
    for dirpath, dirnames, filenames in generator:
        for name in 'CVS', '.svn':
            if name in dirnames:
                dirnames.remove(name)

        for name in filenames:
            if '~' in name: continue
            suffix = os.path.splitext(name)[1]
            if suffix in ('.pyc', '.pyo'): continue
            if name[0] == '.': continue
            filename = os.path.join(dirpath, name)
            dest.append(filename)

# define what is our data
data = []
add_files( data, os.walk('data'))
data.extend( glob.glob('*.txt'))
# define what is our source
src = []
add_files( src, os.walk('lib'))
src.extend( glob.glob('*.py'))

# build the sdist target
if cmd == 'sdist':
    f = open( "MANIFEST.in", "w")
    for l in data: f.write("include "+l+"\n")
    for l in src: f.write("include "+l+"\n")
    f.close()

    setup(
        name=cfg['name'],
        version=cfg['version'],
        description=cfg['description'],
        author=cfg['author'],
        author_email=cfg['author_email'],
        url=cfg['url'],
        )

# build the py2exe target
if cmd in ('py2exe',):
    dist_dir = os.path.join('dist',cfg['py2exe.target'])
    data_dir = dist_dir

    src = PY_PROG
    dest = cfg['py2exe.binary']+'.py'
    shutil.copy(src,dest)

    setup(
        options={'py2exe':{
            'dist_dir':dist_dir,
            'dll_excludes':['_dotblas.pyd','_numpy.pyd']
            }},
#        windows=[{
        console=[{
            'script':dest,
#            'icon_resources':[(1,cfg['py2exe.icon'])],
            }],
        )

# build the py2app target
if cmd == 'py2app':
    dist_dir = os.path.join('dist',cfg['py2app.target']+'.app')
    data_dir = os.path.join(dist_dir,'Contents','Resources')
    from setuptools import setup

    src = PY_PROG
    dest = cfg['py2app.target']+'.py'
    shutil.copy(src,dest)

    APP = [dest]
    DATA_FILES = []
    OPTIONS = {'argv_emulation': True, 'iconfile':cfg['py2app.icon']}

    setup(
        app=APP,
        data_files=DATA_FILES,
        options={'py2app': OPTIONS},
        setup_requires=['py2app'],
    )

# make the cx_freeze target
if cmd == 'cx_freeze':
    dist_dir = os.path.join('dist',cfg['cx_freeze.target'])
    data_dir = dist_dir
    os.system('%s --install-dir %s --target-name %s %s' %
(cfg['cx_freeze.cmd'], cfg['cx_freeze.binary'], dist_dir, PY_PROG))

# recursively make a bunch of folders
def make_dirs(dname_):
    parts = list(os.path.split(dname_))
    dname = None
    while len(parts):
        if dname == None:
            dname = parts.pop(0)
        else:
            dname = os.path.join(dname,parts.pop(0))
        if not os.path.isdir(dname):
            os.mkdir(dname)

# copy data into the binaries
if cmd in ('py2exe','cx_freeze','py2app'):
    dest = data_dir
    for fname in data:
        dname = os.path.join(dest,os.path.dirname(fname))
        make_dirs(dname)
        if not os.path.isdir(fname):
            shutil.copy(fname,dname)


_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG at python.org
http://mail.python.org/mailman/listinfo/distutils-sig 


From jim at zope.com  Sun Jan  1 16:57:01 2012
From: jim at zope.com (Jim Fulton)
Date: Sun, 1 Jan 2012 10:57:01 -0500
Subject: [Distutils] [buildout] latest bootstrap.py fails on all 3
 pythons that ship with Mac OS X Lion
In-Reply-To: <4EFC5263.8050008@simplistix.co.uk>
References: <4EFC5263.8050008@simplistix.co.uk>
Message-ID: <CAPDm-FjOxr9y8ovWbexC_NrG9TptuSMQ85xVQswjfD5+KzkCbQ@mail.gmail.com>

On Thu, Dec 29, 2011 at 6:43 AM, Chris Withers <chris at simplistix.co.uk> wrote:
> Hi All,
>
> I just upgraded a machine I want to use as a build server to Mac OS X Lion.
> It appears that Lion ships with Python 2.5, 2.6 and 2.7, but bootstrap.py
> fails with each of these:
>
> $ python2.5 bootstrap.py

Try: python2.5 -S bootstrap.py

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From chris at simplistix.co.uk  Sun Jan  1 19:26:15 2012
From: chris at simplistix.co.uk (Chris Withers)
Date: Sun, 01 Jan 2012 18:26:15 +0000
Subject: [Distutils] [buildout] latest bootstrap.py fails on all 3
 pythons that ship with Mac OS X Lion
In-Reply-To: <CAPDm-FjOxr9y8ovWbexC_NrG9TptuSMQ85xVQswjfD5+KzkCbQ@mail.gmail.com>
References: <4EFC5263.8050008@simplistix.co.uk>
	<CAPDm-FjOxr9y8ovWbexC_NrG9TptuSMQ85xVQswjfD5+KzkCbQ@mail.gmail.com>
Message-ID: <4F00A547.8050404@simplistix.co.uk>

On 01/01/2012 15:57, Jim Fulton wrote:
> On Thu, Dec 29, 2011 at 6:43 AM, Chris Withers<chris at simplistix.co.uk>  wrote:
>> Hi All,
>>
>> I just upgraded a machine I want to use as a build server to Mac OS X Lion.
>> It appears that Lion ships with Python 2.5, 2.6 and 2.7, but bootstrap.py
>> fails with each of these:
>>
>> $ python2.5 bootstrap.py
>
> Try: python2.5 -S bootstrap.py

That appears to help with 2.6 and 2.7, but with 2.5 I still get:

$ python2.5 -S bootstrap.py
The required version of setuptools (>=0.6c11) is not available, and
can't be installed while this script is running. Please install
  a more recent version first, using 'easy_install -U setuptools'.

(Currently using setuptools 0.6c9 
(/System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python))

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk

From asinghania at solarflare.com  Tue Jan  3 17:26:27 2012
From: asinghania at solarflare.com (Akhi Singhania)
Date: Tue, 3 Jan 2012 16:26:27 +0000
Subject: [Distutils] Creating a binary only python distribution of a C
 extension module and including some additional python and c files in it
Message-ID: <4F032C33.2060404@solarflare.com>

Hi,

I have a extension module in C which I want to distribute in binary 
format, ideally an rpm. Additionally, I want to include some python 
files (examples on how to use the extension module) and source for a 
library the module dynamically links to (c,h, and make files).

How do I specify the example python file in setup.py so that it will be 
included in the rpm?

If I specify it as scripts, I get the following error:

$ python setup.py bdist --format=rpm

   running build_scripts
   creating build/scripts-2.6
   error: file 'foo.py' does not exist
   error: Bad exit status from /var/tmp/rpm-tmp.yjws9x (%build)
If I specify it as data_files, I get the following error:

$ python setup.py bdist --format=rpm

   error: Installed (but unpackaged) file(s) found:
      /usr/foo.pyc
      /usr/foo.pyo
If I specify it as py_modules, I do not get errors but it is not 
included in the resulting rpm.

Specifying it as a script works if I use

$ python setup.py bdist --format=gztar
Additionally, can I control the hierarchy of how the files are laid out 
in the rpm?

Currently, I am specifying the c,h,make files as data_files and they get 
placed in /usr, which is not desirable.

Alternatively, do you know of a way to specify the directory hierarchy 
if using --format=gztar
Currently, it creates a hierarchy resembling root install into /usr.
If I can modify this hierarchy, I would have essentially what I want.

Regards,
akhi

From almar.klein at gmail.com  Thu Jan  5 11:49:24 2012
From: almar.klein at gmail.com (Almar Klein)
Date: Thu, 5 Jan 2012 11:49:24 +0100
Subject: [Distutils] Compiling Cython on Windows runs into link problems
 because of directories containing spaces
Message-ID: <CAB_T6==mVXigDfgDJ4HBM2-mE3k70D+UahNDM+7Fpj5eR1OB1A@mail.gmail.com>

Hi,

I have a problem with compiling Cython code on Windows. I think this is a
bug in distutils which is easy to solve. I have reproduced this on Python
2.6 and Python 3.2 (32 bit).

The problem occurs with the native msvc compiler. Using gcc works fine. And
I prefer using gcc, but sometimes you just need msvc :/

The problem is that the command to link the libraries does not put double
quotes around paths that have spaces in them. Unfortunately, the path where
many users have Python installed have spaces in it (e.g. c:/program
files/python26). Small example: */LIBPATH:C:\Program Files
(x86)\python32\libs*. Oh, and the include_dirs DO have double quotes around
them.

The problem is easily solved (I confirmed this) by changing
msvc9compiler.py and msvccompiler.py:

def library_dir_option(self, dir):  # OLD VERSION

     return "/LIBPATH:" + dir

def library_dir_option(self, dir): # FIXED VERSION

    if ' ' in dir and not dir.startswith('"'):

        dir = '"%s"' % dir

    return "/LIBPATH:" + dir


===== Below follows a minimal example =====

===== test_.pyx
def foo():

print('hello')


===== setup.py

import os, sys

from Cython.Distutils import build_ext

from distutils.core import setup

from distutils.extension import Extension

from numpy.distutils.misc_util import get_numpy_include_dirs

# Ugly hack so I can run setup.py in my IDE

sys.argv = ['setup.py', 'build_ext', '--inplace']

# Init include dirs

include_dirs = ['.']

include_dirs.extend(get_numpy_include_dirs())

# Creat Extensions

ext_modules = [

     Extension('test_', ['test_.pyx'],

        include_dirs=include_dirs,

        ),

     ]

# Compile

setup(

    cmdclass = {'build_ext': build_ext},

    ext_modules = ext_modules,

    )

print('Successfully compiled cython file: test_')


===== output when running setup.py

running build_ext

No module named msvccompiler in numpy.distutils; trying from distutils

cythoning test_.pyx to test_.c

building 'test_' extension

C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c /nologo
/Ox /MD /W3 /GS- /DNDEBUG -I. -I"C:\Program Files
(x86)\python32\lib\site-packages\numpy\core\include" -I"C:\Program Files
(x86)\python32\include" -I"C:\Program Files (x86)\python32\PC" /Tctest_.c
/Fobuild\temp.win32-3.2\Release\test_.obj

Found executable C:\Program Files (x86)\Microsoft Visual Studio
9.0\VC\BIN\cl.exe

C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL
/nologo /INCREMENTAL:NO */LIBPATH:C:\Program Files (x86)\python32\libs
/LIBPATH:C:\Program Files (x86)\python32\PCbuild* /EXPORT:PyInit_test_
build\temp.win32-3.2\Release\test_.obj
/OUT:C:\almar\projects\py\cmu1394\test_.pyd
/IMPLIB:build\temp.win32-3.2\Release\test_.lib
/MANIFESTFILE:build\temp.win32-3.2\Release\test_.pyd.manifest

Found executable C:\Program Files (x86)\Microsoft Visual Studio
9.0\VC\BIN\link.exe

LINK : fatal error LNK1181: cannot open input file 'Files.obj'
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120105/2be8e8c5/attachment.html>

From chris at simplistix.co.uk  Sun Jan  8 14:29:15 2012
From: chris at simplistix.co.uk (Chris Withers)
Date: Sun, 08 Jan 2012 13:29:15 +0000
Subject: [Distutils] [buildout] windows issue causes jenkins problems
Message-ID: <4F099A2B.5040504@simplistix.co.uk>

Hi All,

The weird process forking stuff that running buildout on windows 
sometimes does causes problems for Jenkins:

http://jenkins.simplistix.co.uk/job/testfixtures-buildout/32/PYTHON=2.6,label=windows/console

Is there any way I can cause this not to happen?

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk

From chris at simplistix.co.uk  Sun Jan  8 21:39:12 2012
From: chris at simplistix.co.uk (Chris Withers)
Date: Sun, 08 Jan 2012 20:39:12 +0000
Subject: [Distutils] [buildout] windows issue causes jenkins problems
In-Reply-To: <801FDD7B-35A0-4D3D-B325-69BE9DFF35CF@rzn.co.il>
References: <4F099A2B.5040504@simplistix.co.uk>
	<801FDD7B-35A0-4D3D-B325-69BE9DFF35CF@rzn.co.il>
Message-ID: <4F09FEF0.3060605@simplistix.co.uk>

> On 8 ???? 2012, at 15:29, Chris Withers <chris at simplistix.co.uk
>> The weird process forking stuff that running buildout on windows
>> sometimes does causes problems for Jenkins:
>>
>> http://jenkins.simplistix.co.uk/job/testfixtures-buildout/32/PYTHON=2.6,label=windows/console
>>
>> Is there any way I can cause this not to happen?

On 08/01/2012 20:31, Guy Rozendorn wrote:
> Use distribute 0.6.24

Why would this make a difference?

Just for confirmation, it does not:

http://jenkins.simplistix.co.uk/job/testfixtures-buildout/PYTHON=2.5,label=windows/35/console

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk

From guy at rzn.co.il  Sun Jan  8 22:18:01 2012
From: guy at rzn.co.il (Guy Rozendorn)
Date: Sun, 08 Jan 2012 23:18:01 +0200
Subject: [Distutils] [buildout] windows issue causes jenkins problems
In-Reply-To: <4F09FEF0.3060605@simplistix.co.uk>
References: <4F099A2B.5040504@simplistix.co.uk>
	<801FDD7B-35A0-4D3D-B325-69BE9DFF35CF@rzn.co.il>
	<4F09FEF0.3060605@simplistix.co.uk>
Message-ID: <4F0A0809.3020902@rzn.co.il>

My bad,
I ran into this some time I ago and did not remember the cause, and when 
I saw on log you're using setuptools, I was sure that was it.
Now that look more close it looks like you're suffering from several issues.
and now your jenkins in offline? I'll try again tomorrow

> Chris Withers <mailto:chris at simplistix.co.uk>
> January 8, 2012 10:39 PM
>> On 8 ???? 2012, at 15:29, Chris Withers <chris at simplistix.co.uk
>>> The weird process forking stuff that running buildout on windows
>>> sometimes does causes problems for Jenkins:
>>>
>>> http://jenkins.simplistix.co.uk/job/testfixtures-buildout/32/PYTHON=2.6,label=windows/console 
>>>
>>>
>>> Is there any way I can cause this not to happen?
>
> On 08/01/2012 20:31, Guy Rozendorn wrote:
>> Use distribute 0.6.24
>
> Why would this make a difference?
>
> Just for confirmation, it does not:
>
> http://jenkins.simplistix.co.uk/job/testfixtures-buildout/PYTHON=2.5,label=windows/35/console 
>
>
> cheers,
>
> Chris
>
> Guy Rozendorn <mailto:guy at rzn.co.il>
> January 8, 2012 10:31 PM
> Use distribute 0.6.24
>
> Sent from my iPhone
>
> On 8 ???? 2012, at 15:29, Chris Withers <chris at simplistix.co.uk 
> <mailto:chris at simplistix.co.uk>> wrote:
>
> Chris Withers <mailto:chris at simplistix.co.uk>
> January 8, 2012 3:29 PM
> Hi All,
>
> The weird process forking stuff that running buildout on windows 
> sometimes does causes problems for Jenkins:
>
> http://jenkins.simplistix.co.uk/job/testfixtures-buildout/32/PYTHON=2.6,label=windows/console 
>
>
> Is there any way I can cause this not to happen?
>
> cheers,
>
> Chris
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120108/bdb9b123/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: compose-unknown-contact.jpg
Type: image/jpeg
Size: 770 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120108/bdb9b123/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: postbox-contact.jpg
Type: image/jpeg
Size: 1173 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120108/bdb9b123/attachment-0001.jpg>

From chris at simplistix.co.uk  Sun Jan  8 22:21:56 2012
From: chris at simplistix.co.uk (Chris Withers)
Date: Sun, 08 Jan 2012 21:21:56 +0000
Subject: [Distutils] [buildout] windows issue causes jenkins problems
In-Reply-To: <4F0A0809.3020902@rzn.co.il>
References: <4F099A2B.5040504@simplistix.co.uk>
	<801FDD7B-35A0-4D3D-B325-69BE9DFF35CF@rzn.co.il>
	<4F09FEF0.3060605@simplistix.co.uk> <4F0A0809.3020902@rzn.co.il>
Message-ID: <4F0A08F4.6060506@simplistix.co.uk>

On 08/01/2012 21:18, Guy Rozendorn wrote:
> Now that look more close it looks like you're suffering from several issues.

Huh? On the windows builds, the only issue I see is that bootstrap.py 
does some weird forking which returns control to the terminal, so the 
shell moves onto the next command in jenkins.bat.

Of course, that command is "bin/buildout", which doesn't exist until the 
weird forked process is finished.

The linux and mac builds have test failures as a result of some 2.5 and 
2.7 incompatibilities I accidentally introduced, which is the whole 
reason I'm getting jenkins up and running...

> and now your jenkins in offline?

Not by my reckoning.

http://www.downforeveryoneorjustme.com/jenkins.simplistix.co.uk

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk

From almar.klein at gmail.com  Sun Jan  8 23:10:49 2012
From: almar.klein at gmail.com (Almar Klein)
Date: Sun, 8 Jan 2012 23:10:49 +0100
Subject: [Distutils] Compiling Cython on Windows runs into link problems
 because of directories containing spaces
In-Reply-To: <CAB_T6==mVXigDfgDJ4HBM2-mE3k70D+UahNDM+7Fpj5eR1OB1A@mail.gmail.com>
References: <CAB_T6==mVXigDfgDJ4HBM2-mE3k70D+UahNDM+7Fpj5eR1OB1A@mail.gmail.com>
Message-ID: <CAB_T6=ny1w+1KTjWQHBtWijmH1j7Kq62QVmRXat1WSqCsMs6gg@mail.gmail.com>

Since there've been no responses yet, let me make my question more clear :)

Is this is a bug, or is this a known "feature" that I should solve in
another manner. If this is a bug, can someone help me fix it?

Further, I'm a bit surprised to find this bug. I can't be the only one who
wants to compile stuff with the MS compiler and who has Python installed in
his "program files". Why is that?

Regards,
  Almar


On 5 January 2012 11:49, Almar Klein <almar.klein at gmail.com> wrote:

> Hi,
>
> I have a problem with compiling Cython code on Windows. I think this is a
> bug in distutils which is easy to solve. I have reproduced this on Python
> 2.6 and Python 3.2 (32 bit).
>
> The problem occurs with the native msvc compiler. Using gcc works fine.
> And I prefer using gcc, but sometimes you just need msvc :/
>
> The problem is that the command to link the libraries does not put double
> quotes around paths that have spaces in them. Unfortunately, the path where
> many users have Python installed have spaces in it (e.g. c:/program
> files/python26). Small example: */LIBPATH:C:\Program Files
> (x86)\python32\libs*. Oh, and the include_dirs DO have double quotes
> around them.
>
> The problem is easily solved (I confirmed this) by changing
> msvc9compiler.py and msvccompiler.py:
>
> def library_dir_option(self, dir):  # OLD VERSION
>
>      return "/LIBPATH:" + dir
>
> def library_dir_option(self, dir): # FIXED VERSION
>
>     if ' ' in dir and not dir.startswith('"'):
>
>         dir = '"%s"' % dir
>
>     return "/LIBPATH:" + dir
>
>
> ===== Below follows a minimal example =====
>
> ===== test_.pyx
> def foo():
>
> print('hello')
>
>
> ===== setup.py
>
> import os, sys
>
> from Cython.Distutils import build_ext
>
> from distutils.core import setup
>
> from distutils.extension import Extension
>
> from numpy.distutils.misc_util import get_numpy_include_dirs
>
> # Ugly hack so I can run setup.py in my IDE
>
> sys.argv = ['setup.py', 'build_ext', '--inplace']
>
> # Init include dirs
>
> include_dirs = ['.']
>
> include_dirs.extend(get_numpy_include_dirs())
>
> # Creat Extensions
>
> ext_modules = [
>
>      Extension('test_', ['test_.pyx'],
>
>         include_dirs=include_dirs,
>
>         ),
>
>      ]
>
> # Compile
>
> setup(
>
>     cmdclass = {'build_ext': build_ext},
>
>     ext_modules = ext_modules,
>
>     )
>
> print('Successfully compiled cython file: test_')
>
>
> ===== output when running setup.py
>
> running build_ext
>
> No module named msvccompiler in numpy.distutils; trying from distutils
>
> cythoning test_.pyx to test_.c
>
> building 'test_' extension
>
> C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c
> /nologo /Ox /MD /W3 /GS- /DNDEBUG -I. -I"C:\Program Files
> (x86)\python32\lib\site-packages\numpy\core\include" -I"C:\Program Files
> (x86)\python32\include" -I"C:\Program Files (x86)\python32\PC" /Tctest_.c
> /Fobuild\temp.win32-3.2\Release\test_.obj
>
> Found executable C:\Program Files (x86)\Microsoft Visual Studio
> 9.0\VC\BIN\cl.exe
>
> C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL
> /nologo /INCREMENTAL:NO */LIBPATH:C:\Program Files (x86)\python32\libs
> /LIBPATH:C:\Program Files (x86)\python32\PCbuild* /EXPORT:PyInit_test_
> build\temp.win32-3.2\Release\test_.obj
> /OUT:C:\almar\projects\py\cmu1394\test_.pyd
> /IMPLIB:build\temp.win32-3.2\Release\test_.lib
> /MANIFESTFILE:build\temp.win32-3.2\Release\test_.pyd.manifest
>
> Found executable C:\Program Files (x86)\Microsoft Visual Studio
> 9.0\VC\BIN\link.exe
>
> LINK : fatal error LNK1181: cannot open input file 'Files.obj'
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120108/b7727bd3/attachment-0001.html>

From jim at zope.com  Mon Jan  9 18:54:30 2012
From: jim at zope.com (Jim Fulton)
Date: Mon, 9 Jan 2012 12:54:30 -0500
Subject: [Distutils] [buildout] windows issue causes jenkins problems
In-Reply-To: <4F099A2B.5040504@simplistix.co.uk>
References: <4F099A2B.5040504@simplistix.co.uk>
Message-ID: <CAPDm-FjxDMhuk1Nh=5GmN=qDn=y0Euk2a_nN4BctcQ7n3QvPzQ@mail.gmail.com>

On Sun, Jan 8, 2012 at 8:29 AM, Chris Withers <chris at simplistix.co.uk> wrote:
> Hi All,
>
> The weird process forking stuff that running buildout on windows sometimes
> does causes problems for Jenkins:
>
> http://jenkins.simplistix.co.uk/job/testfixtures-buildout/32/PYTHON=2.6,label=windows/console
>
> Is there any way I can cause this not to happen?

I don't remember the details and I don't have time to look right now,
but it has something to do with Windows not letting you remove or
move open files.

You could try coming up with an alternate bootstrap that takes a different
approach.

Another idea is to keep a buildout to the side and use that to bootstrap
the buildouts your doing while testing.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From skippy.hammond at gmail.com  Wed Jan 11 05:56:18 2012
From: skippy.hammond at gmail.com (Mark Hammond)
Date: Wed, 11 Jan 2012 15:56:18 +1100
Subject: [Distutils] Compiling Cython on Windows runs into link problems
 because of directories containing spaces
In-Reply-To: <CAB_T6=ny1w+1KTjWQHBtWijmH1j7Kq62QVmRXat1WSqCsMs6gg@mail.gmail.com>
References: <CAB_T6==mVXigDfgDJ4HBM2-mE3k70D+UahNDM+7Fpj5eR1OB1A@mail.gmail.com>
	<CAB_T6=ny1w+1KTjWQHBtWijmH1j7Kq62QVmRXat1WSqCsMs6gg@mail.gmail.com>
Message-ID: <4F0D1672.4070208@gmail.com>

On 9/01/2012 9:10 AM, Almar Klein wrote:
> Since there've been no responses yet, let me make my question more clear :)
>
> Is this is a bug, or is this a known "feature" that I should solve in
> another manner. If this is a bug, can someone help me fix it?

It probably can be considered a bug - the Python issue tracker is the 
place where it should be reported (but be sure to check it hasn't 
already been reported).  From your earlier message, it sounds like you 
already have a handle on how to fix it (although I suspect it might be 
better for the caller of library_dir_option to add the quotes).

>
> Further, I'm a bit surprised to find this bug. I can't be the only one
> who wants to compile stuff with the MS compiler and who has Python
> installed in his "program files".

You may well be.  "program files" isn't the default location and even if 
it was, I suspect most developers would override it.  So I suspect that 
most people either (a) use the default location or (b) override it to 
something without spaces in the path.

Mark

> Why is that?
>
> Regards,
>    Almar
>
>
> On 5 January 2012 11:49, Almar Klein <almar.klein at gmail.com
> <mailto:almar.klein at gmail.com>> wrote:
>
>     Hi,
>
>     I have a problem with compiling Cython code on Windows. I think this
>     is a bug in distutils which is easy to solve. I have reproduced this
>     on Python 2.6 and Python 3.2 (32 bit).
>
>     The problem occurs with the native msvc compiler. Using gcc works
>     fine. And I prefer using gcc, but sometimes you just need msvc :/
>
>     The problem is that the command to link the libraries does not put
>     double quotes around paths that have spaces in them. Unfortunately,
>     the path where many users have Python installed have spaces in it
>     (e.g. c:/program files/python26). Small example:
>     _/LIBPATH:C:\Program Files (x86)\python32\libs_. Oh, and the
>     include_dirs DO have double quotes around them.
>
>     The problem is easily solved (I confirmed this) by changing
>     msvc9compiler.py and msvccompiler.py:
>
>     def library_dir_option(self, dir):  # OLD VERSION
>
>           return "/LIBPATH:" + dir
>
>
>     def library_dir_option(self, dir): # FIXED VERSION
>
>          if ' ' in dir and not dir.startswith('"'):
>
>              dir = '"%s"' % dir
>
>          return "/LIBPATH:" + dir
>
>
>
>     ===== Below follows a minimal example =====
>
>     ===== test_.pyx
>     def foo():
>
>     print('hello')
>
>
>     ===== setup.py
>
>     import os, sys
>
>     from Cython.Distutils import build_ext
>
>     from distutils.core import setup
>
>     from distutils.extension import Extension
>
>     from numpy.distutils.misc_util import get_numpy_include_dirs
>
>     # Ugly hack so I can run setup.py in my IDE
>
>     sys.argv = ['setup.py', 'build_ext', '--inplace']
>
>     # Init include dirs
>
>     include_dirs = ['.']
>
>     include_dirs.extend(get_numpy_include_dirs())
>
>     # Creat Extensions
>
>     ext_modules = [
>
>           Extension('test_', ['test_.pyx'],
>
>              include_dirs=include_dirs,
>
>              ),
>
>           ]
>
>     # Compile
>
>     setup(
>
>          cmdclass = {'build_ext': build_ext},
>
>          ext_modules = ext_modules,
>
>          )
>
>     print('Successfully compiled cython file: test_')
>
>
>     ===== output when running setup.py
>
>     running build_ext
>
>     No module named msvccompiler in numpy.distutils; trying from distutils
>
>     cythoning test_.pyx to test_.c
>
>     building 'test_' extension
>
>     C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c
>     /nologo /Ox /MD /W3 /GS- /DNDEBUG -I. -I"C:\Program Files
>     (x86)\python32\lib\site-packages\numpy\core\include" -I"C:\Program
>     Files (x86)\python32\include" -I"C:\Program Files (x86)\python32\PC"
>     /Tctest_.c /Fobuild\temp.win32-3.2\Release\test_.obj
>
>     Found executable C:\Program Files (x86)\Microsoft Visual Studio
>     9.0\VC\BIN\cl.exe
>
>     C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe
>     /DLL /nologo /INCREMENTAL:NO _/LIBPATH:C:\Program Files
>     (x86)\python32\libs /LIBPATH:C:\Program Files
>     (x86)\python32\PCbuild_ /EXPORT:PyInit_test_
>     build\temp.win32-3.2\Release\test_.obj
>     /OUT:C:\almar\projects\py\cmu1394\test_.pyd
>     /IMPLIB:build\temp.win32-3.2\Release\test_.lib
>     /MANIFESTFILE:build\temp.win32-3.2\Release\test_.pyd.manifest
>
>     Found executable C:\Program Files (x86)\Microsoft Visual Studio
>     9.0\VC\BIN\link.exe
>
>     LINK : fatal error LNK1181: cannot open input file 'Files.obj'
>
>
>
>
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig


From almar.klein at gmail.com  Wed Jan 11 10:38:54 2012
From: almar.klein at gmail.com (Almar Klein)
Date: Wed, 11 Jan 2012 10:38:54 +0100
Subject: [Distutils] Compiling Cython on Windows runs into link problems
 because of directories containing spaces
In-Reply-To: <4F0D1672.4070208@gmail.com>
References: <CAB_T6==mVXigDfgDJ4HBM2-mE3k70D+UahNDM+7Fpj5eR1OB1A@mail.gmail.com>
	<CAB_T6=ny1w+1KTjWQHBtWijmH1j7Kq62QVmRXat1WSqCsMs6gg@mail.gmail.com>
	<4F0D1672.4070208@gmail.com>
Message-ID: <CAB_T6=kUJ+jfvJbeqdL1aLFxd3D-eNq2qKXqfyd_RjK8Ht7M=Q@mail.gmail.com>

Dear Mark,

Thanks for the response.


Is this is a bug, or is this a known "feature" that I should solve in
>> another manner. If this is a bug, can someone help me fix it?
>>
>
> It probably can be considered a bug - the Python issue tracker is the
> place where it should be reported (but be sure to check it hasn't already
> been reported).  From your earlier message, it sounds like you already have
> a handle on how to fix it (although I suspect it might be better for the
> caller of library_dir_option to add the quotes).
>

Ok, I will report the bug there. I wasn't sure whether distutils had its
own issue tracker (I couldn't find it anyway).



> Further, I'm a bit surprised to find this bug. I can't be the only one
>> who wants to compile stuff with the MS compiler and who has Python
>> installed in his "program files".
>>
>
> You may well be.  "program files" isn't the default location and even if
> it was, I suspect most developers would override it.  So I suspect that
> most people either (a) use the default location or (b) override it to
> something without spaces in the path.
>

Well I didn't :)  And I can hardly be the only one. I can't imagine I'm the
first to run into this bug. Maybe because the produced link error is not
helpful at all, people did not know what to do with it?

Regards,
  Almar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120111/f71c41f0/attachment.html>

From almar.klein at gmail.com  Wed Jan 11 11:56:26 2012
From: almar.klein at gmail.com (Almar Klein)
Date: Wed, 11 Jan 2012 11:56:26 +0100
Subject: [Distutils] Compiling Cython on Windows runs into link problems
 because of directories containing spaces
In-Reply-To: <CAB_T6=kUJ+jfvJbeqdL1aLFxd3D-eNq2qKXqfyd_RjK8Ht7M=Q@mail.gmail.com>
References: <CAB_T6==mVXigDfgDJ4HBM2-mE3k70D+UahNDM+7Fpj5eR1OB1A@mail.gmail.com>
	<CAB_T6=ny1w+1KTjWQHBtWijmH1j7Kq62QVmRXat1WSqCsMs6gg@mail.gmail.com>
	<4F0D1672.4070208@gmail.com>
	<CAB_T6=kUJ+jfvJbeqdL1aLFxd3D-eNq2qKXqfyd_RjK8Ht7M=Q@mail.gmail.com>
Message-ID: <CAB_T6=m34iRrw0tMSXkVdfn6=7aPhyXgFuWpzveDapaK6SrmiQ@mail.gmail.com>

> Ok, I will report the bug there. I wasn't sure whether distutils had its
> own issue tracker (I couldn't find it anyway).
>

I reported the bug and supplied a patch: http://bugs.python.org/issue13765

  Almar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120111/806ab73d/attachment.html>

From leorochael at gmail.com  Wed Jan 11 16:21:20 2012
From: leorochael at gmail.com (Leonardo Rochael Almeida)
Date: Wed, 11 Jan 2012 16:21:20 +0100
Subject: [Distutils] buildout: overriding the global [versions] section
Message-ID: <CAMxX_oOKg-0kKkpCh-THEhc8ca+oErr4Y0LLC2jUUzjdA60Haw@mail.gmail.com>

Hi,

It seems I cannot override, for a single buildout part, the version
information of the global [versions] section.

Is this possible and I just overlooked something?

If not, I intend to add support for that on zc.recipe.egg and/or
z3c.recipe.scripts recipes.

Any comments?

And where can I find the source for z3c.recipe.scripts?

More details below:

I've frequently found myself in the situation of needing to install,
on the same buildout, different parts containing eggs with conflicting
version requirements.

It would be useful if I could specify a version information that
overrode the global [versions] information, in a part with a
zc.recipe.egg or z3c.recipe.scripts recipe.

Notice that specifying the version on the 'eggs' setting doesn't work
to override the global [versions] specification. I.e., the following
buildout.cfg:

  [buildout]
  parts =
    testbrower

  versions=versions

  [versions]
  zope.testbrowser=3.6.0a2

  [testbrower]
  recipe = zc.recipe.egg
  eggs =
    zope.testbrowser==3.11.1
-------

Fails with:

  The version, 3.6.0a2, is not consistent with the requirement,
'zope.testbrowser==3.11.1'.
  While:
    Installing testbrower.
  Error: Bad version 3.6.0a2

What would be nice is if I could say something like:

  [buildout]
  parts =
    testbrower

  versions=versions

  [versions]
  zope.testbrowser=3.6.0a2

  [testbrowser-versions]
  zope.testbrowser=3.11.1

  [testbrower]
  recipe = zc.recipe.egg
  versions = testbrowser-versions
  eggs =
    zope.testbrowser
-------

Or maybe even:

  (...)
  [testbrower]
  recipe = zc.recipe.egg
  versions =
    zope.testbrowser = 3.11.1
  eggs =
    zope.testbrowser
-------

And then have buildout pick my version override only while installing
the eggs in that part.

The buildout easy_install API can already [1] accept a local version
specification in the 'versions' keyword to the
zc.buildout.easy_install.install() function. It also specifies that
the global version specification can be retrieved and modified through
the zc.buildout.easy_install.default_versions() function.

[1] http://pypi.python.org/pypi/zc.buildout/1.5.2#specifying-version-information-independent-of-requirements

However, the docs don't say what happens if both a default_versions()
is set and a "versions=" keyword is passed to .install() containing
conflicting information.

In a worse case scenario, I could save the results of
zc.buildout.easy_install.default_versions() and restore after calling
.install() with the overridden versions.

Any comments, for or against?

Jim?

Cheers,

Leo

From jim at zope.com  Thu Jan 12 11:08:08 2012
From: jim at zope.com (Jim Fulton)
Date: Thu, 12 Jan 2012 05:08:08 -0500
Subject: [Distutils] buildout: overriding the global [versions] section
In-Reply-To: <CAMxX_oOKg-0kKkpCh-THEhc8ca+oErr4Y0LLC2jUUzjdA60Haw@mail.gmail.com>
References: <CAMxX_oOKg-0kKkpCh-THEhc8ca+oErr4Y0LLC2jUUzjdA60Haw@mail.gmail.com>
Message-ID: <CAPDm-FgzpoDs9++2t-tu3uT_uop7C_GKDfySwvqmydrEeCVZcg@mail.gmail.com>

On Wed, Jan 11, 2012 at 10:21 AM, Leonardo Rochael Almeida
<leorochael at gmail.com> wrote:
> Hi,
>
> It seems I cannot override, for a single buildout part, the version
> information of the global [versions] section.
>
> Is this possible and I just overlooked something?
>
> If not, I intend to add support for that on zc.recipe.egg and/or
> z3c.recipe.scripts recipes.
>
> Any comments?
...
> Any comments, for or against?

Overall you raise a good point.

Rather than having part-specific versions, I'd rather just let explicit
versions specified in in parts (e.g. in eggs options) override versions
specifications.

Also, there's value in having one place to see see versions used
so I it's possible some folks would object to this change.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton

From chris at simplistix.co.uk  Thu Jan 12 16:54:20 2012
From: chris at simplistix.co.uk (Chris Withers)
Date: Thu, 12 Jan 2012 15:54:20 +0000
Subject: [Distutils] buildout: overriding the global [versions] section
In-Reply-To: <CAPDm-FgzpoDs9++2t-tu3uT_uop7C_GKDfySwvqmydrEeCVZcg@mail.gmail.com>
References: <CAMxX_oOKg-0kKkpCh-THEhc8ca+oErr4Y0LLC2jUUzjdA60Haw@mail.gmail.com>
	<CAPDm-FgzpoDs9++2t-tu3uT_uop7C_GKDfySwvqmydrEeCVZcg@mail.gmail.com>
Message-ID: <4F0F022C.5080704@simplistix.co.uk>

On 12/01/2012 10:08, Jim Fulton wrote:
> Rather than having part-specific versions, I'd rather just let explicit
> versions specified in in parts (e.g. in eggs options) override versions
> specifications.
>
> Also, there's value in having one place to see see versions used
> so I it's possible some folks would object to this change.

Yeah, my take on this is that you should just split these out into 
separate buildouts.

A huge value for me is that a buildout has a consistent set of package 
versions available across all parts it has managed.

Also, bear in mind that any changes you make that allow this will cause 
problems for extensions such as buildout.dumppickedversions and 
buildout_versions.

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From st.chris at gmail.com  Mon Jan 16 16:34:25 2012
From: st.chris at gmail.com (=?UTF-8?Q?Christian_=C5=9Etef=C4=83nescu?=)
Date: Mon, 16 Jan 2012 16:34:25 +0100
Subject: [Distutils] Distribute instead of setuptools for virtualenv?
Message-ID: <CAPR7qLmpO7aCgmbs2b0-v+zcpjNjzfnnLZxkZCYzQjwrxLV0Qg@mail.gmail.com>

Hi,

I'd like to know if I can create my virtualenvs with "--distribute" without
any other changes, since distribute is advertised as being backwards
compatible with setuptools?

I'm trying to circumvent https://github.com/pypa/virtualenv/issues/202 and
Distribute comes in handy.

Also, if this works just fine, wouldn't it make sense to make Distribute
the default for virtualenv at some point?

Thanks,
Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120116/7477e43f/attachment.html>

From regebro at gmail.com  Mon Jan 16 18:23:45 2012
From: regebro at gmail.com (Lennart Regebro)
Date: Mon, 16 Jan 2012 18:23:45 +0100
Subject: [Distutils] Distribute instead of setuptools for virtualenv?
In-Reply-To: <CAPR7qLmpO7aCgmbs2b0-v+zcpjNjzfnnLZxkZCYzQjwrxLV0Qg@mail.gmail.com>
References: <CAPR7qLmpO7aCgmbs2b0-v+zcpjNjzfnnLZxkZCYzQjwrxLV0Qg@mail.gmail.com>
Message-ID: <CAL0kPAUwNupP2ZUs-Tu13tWPp6XheL02LUcZZ+8d6KB5AsouCQ@mail.gmail.com>

2012/1/16 Christian ?tef?nescu <st.chris at gmail.com>:
> I'd like to know if I can create my virtualenvs with "--distribute" without
> any other changes, since distribute is advertised as being backwards
> compatible with setuptools?

It should work as well or better, yes. If not, that's a bug.

> Also, if this works just fine, wouldn't it make sense to make Distribute the
> default for virtualenv at some point?

In my opinion, yes.

//Lennart

From me at rpatterson.net  Tue Jan 17 03:45:14 2012
From: me at rpatterson.net (Ross Patterson)
Date: Mon, 16 Jan 2012 18:45:14 -0800
Subject: [Distutils] buildout: several fold performance increases
Message-ID: <87obu3jbd1.fsf@transitory.lefae.org>

I've long been perplexed by how long a buildout takes to run with
multiple parts whose required distributions are largely similar.  Taking
a stab at it, I found two hot spots that yield several fold improvements
in performance.

First, zc.buildout.easy_install._log_requirements was doing expensive
requirements parsing and sorting even when no message would be logged.
I committed a fix for it that on a 10 part buildout with a large "eggs"
option for each part decreased update time from a cProfile run time of
93 seconds to 15 seconds:

http://svn.zope.org/zc.buildout/trunk/src/zc/buildout/easy_install.py?rev=124059&r1=122980&r2=124059

Secondly, instantiating pkg_resources.Environment, including the
setuptools.package_index.PackageIndex subclass, is very expensive and
was being done multiple times for any given part, and was being done for
parts whose environments were identical.  There was some existing global
caching for package indexes that I've duplicated for environments in the
attached patch.

Unfortunately, I haven't been able to get a clean test environment for
the life of me.  I'm using a clean Python 2.7 build from source, turning
everything in ~/.buildout/default.cfg off, and running tests in a clean
checkout of the zc.buildout/trunk buildout.  Even under those conditions
I get 17 failing tests before any changes.  With this environments
cache, I see 41 failures, but I can't make sense of it.  This patch
yields another 2-3 fold decrease to 6 seconds for the same buildout and
is driven by profiling data, not guessing.  Can someone help me get this
patch in?

Finally, it would be great to see releases of zc.buildout with these
performance improvements get out in the world.  I've been hearing more
and more complaints about buildout run times and these are easy fixes.
If we can get the second, attached patch in quickly, then I'd say we
should release with both.  If not, then it's still worth it to cut a
release for the first, already committed patch, which yields the
greatest improvement.

Thanks!
Ross

-------------- next part --------------
A non-text attachment was scrubbed...
Name: envs.diff
Type: text/x-diff
Size: 3574 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120116/6c16f531/attachment.diff>

From techtonik at gmail.com  Tue Jan 17 12:16:23 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Tue, 17 Jan 2012 14:16:23 +0300
Subject: [Distutils] Is the Packaging Guide dead?
Message-ID: <CAPkN8x+xkdWcZomK7S3x+edkoniFrg3hZ4U4fpA=nyiH9iRQsg@mail.gmail.com>

The last commit to Packaging Guide was 2010-07-06.
https://bitbucket.org/tarek/hitchhiker-guide-packaging/

The last message in the mailing list was 2010-05-05
https://groups.google.com/forum/#!forum/packaging-guide


Is the project dead or moved?
http://guide.python-distribute.org/
-- 
anatoly t.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120117/1c3fc2f3/attachment.html>

From marius at pov.lt  Tue Jan 17 16:01:09 2012
From: marius at pov.lt (Marius Gedminas)
Date: Tue, 17 Jan 2012 17:01:09 +0200
Subject: [Distutils] buildout: several fold performance increases
In-Reply-To: <87obu3jbd1.fsf@transitory.lefae.org>
References: <87obu3jbd1.fsf@transitory.lefae.org>
Message-ID: <20120117150109.GB29342@fridge.pov.lt>

On Mon, Jan 16, 2012 at 06:45:14PM -0800, Ross Patterson wrote:
> I've long been perplexed by how long a buildout takes to run with
> multiple parts whose required distributions are largely similar.  Taking
> a stab at it, I found two hot spots that yield several fold improvements
> in performance.

That is awesome!  I spend way too much time waiting for bin/buildout.

> First, zc.buildout.easy_install._log_requirements was doing expensive
> requirements parsing and sorting even when no message would be logged.
> I committed a fix for it that on a 10 part buildout with a large "eggs"
> option for each part decreased update time from a cProfile run time of
> 93 seconds to 15 seconds:
> 
> http://svn.zope.org/zc.buildout/trunk/src/zc/buildout/easy_install.py?rev=124059&r1=122980&r2=124059

cProfiler overhead is rarely linear.  What's the real speedup when run
without a profiler?

> Unfortunately, I haven't been able to get a clean test environment for
> the life of me.  I'm using a clean Python 2.7 build from source, turning
> everything in ~/.buildout/default.cfg off, and running tests in a clean
> checkout of the zc.buildout/trunk buildout.  Even under those conditions
> I get 17 failing tests before any changes.  With this environments
> cache, I see 41 failures, but I can't make sense of it.  This patch
> yields another 2-3 fold decrease to 6 seconds for the same buildout and
> is driven by profiling data, not guessing.  Can someone help me get this
> patch in?

I'd love to try, but I'm not one of the people who understand buildout
internals (or even are able to get a clean test run on trunk).

> Finally, it would be great to see releases of zc.buildout with these
> performance improvements get out in the world.  I've been hearing more
> and more complaints about buildout run times and these are easy fixes.
> If we can get the second, attached patch in quickly, then I'd say we
> should release with both.  If not, then it's still worth it to cut a
> release for the first, already committed patch, which yields the
> greatest improvement.

Marius Gedminas
-- 
We've found by experience that people who are careless and sloppy writers are
usually also careless and sloppy thinkers (often enough to bet on, anyway).
	-- ESR (http://www.tuxedo.org/~esr/faqs/smart-questions.html)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120117/f1520daf/attachment.pgp>

From merwok at netwok.org  Tue Jan 17 18:25:11 2012
From: merwok at netwok.org (=?UTF-8?Q?=C3=89ric_Araujo?=)
Date: Tue, 17 Jan 2012 18:25:11 +0100
Subject: [Distutils] =?utf-8?q?Is_the_Packaging_Guide_dead=3F?=
In-Reply-To: <CAPkN8x+xkdWcZomK7S3x+edkoniFrg3hZ4U4fpA=nyiH9iRQsg@mail.gmail.com>
References: <CAPkN8x+xkdWcZomK7S3x+edkoniFrg3hZ4U4fpA=nyiH9iRQsg@mail.gmail.com>
Message-ID: <8e5f4c2bb0c03d9257605dfca18df100@netwok.org>

> The last commit to Packaging Guide was 2010-07-06.
> https://bitbucket.org/tarek/hitchhiker-guide-packaging/
>
> The last message in the mailing list was 2010-05-05
> https://groups.google.com/forum/#!forum/packaging-guide
>
> Is the project dead or moved?

Dead.  Tarek?s weblog posts in 2011 explain how distutils2/packaging
will replace distribute in the long run.  I?m still working on the
packaging docs, and maybe when this first set of doc is in a good shape
I?ll work on updating the distribute packaging guide.

Regards

From me at rpatterson.net  Tue Jan 17 18:41:58 2012
From: me at rpatterson.net (Ross Patterson)
Date: Tue, 17 Jan 2012 09:41:58 -0800
Subject: [Distutils] buildout: several fold performance increases
References: <87obu3jbd1.fsf@transitory.lefae.org>
	<20120117150109.GB29342@fridge.pov.lt>
Message-ID: <87ipkajkex.fsf@transitory.lefae.org>

Marius Gedminas <marius at pov.lt> writes:

> On Mon, Jan 16, 2012 at 06:45:14PM -0800, Ross Patterson wrote:
>> First, zc.buildout.easy_install._log_requirements was doing expensive
>> requirements parsing and sorting even when no message would be logged.
>> I committed a fix for it that on a 10 part buildout with a large "eggs"
>> option for each part decreased update time from a cProfile run time of
>> 93 seconds to 15 seconds:
>> 
>> http://svn.zope.org/zc.buildout/trunk/src/zc/buildout/easy_install.py?rev=124059&r1=122980&r2=124059
>
> cProfiler overhead is rarely linear.  What's the real speedup when run
> without a profiler?

Didn't know that, thanks for the info.  I don't have unprofiled timings
right now and I can't do that ATM, but I'll let you know if I do.

Ross


From techtonik at gmail.com  Wed Jan 18 08:52:00 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 18 Jan 2012 10:52:00 +0300
Subject: [Distutils] Is the Packaging Guide dead?
In-Reply-To: <8e5f4c2bb0c03d9257605dfca18df100@netwok.org>
References: <CAPkN8x+xkdWcZomK7S3x+edkoniFrg3hZ4U4fpA=nyiH9iRQsg@mail.gmail.com>
	<8e5f4c2bb0c03d9257605dfca18df100@netwok.org>
Message-ID: <CAPkN8xLeXzbw6LgChSML0UydCDrmS-3Tt4LCj-vg1Uk1=DPPhA@mail.gmail.com>

On Tue, Jan 17, 2012 at 8:25 PM, ?ric Araujo <merwok at netwok.org> wrote:

>  The last commit to Packaging Guide was 2010-07-06.
>> https://bitbucket.org/tarek/**hitchhiker-guide-packaging/<https://bitbucket.org/tarek/hitchhiker-guide-packaging/>
>>
>> The last message in the mailing list was 2010-05-05
>> https://groups.google.com/**forum/#!forum/packaging-guide<https://groups.google.com/forum/#!forum/packaging-guide>
>>
>> Is the project dead or moved?
>>
>
> Dead.  Tarek?s weblog posts in 2011 explain how distutils2/packaging
> will replace distribute in the long run.  I?m still working on the
> packaging docs, and maybe when this first set of doc is in a good shape
> I?ll work on updating the distribute packaging guide.
>

It will be handy to place a redirect to the new location of `packaging`
guide in the meanwhile and describe the current focus. Then people may be
able to help.
-- 
anatoly t.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120118/fcb41646/attachment.html>

From erik.m.bray at gmail.com  Wed Jan 18 23:16:27 2012
From: erik.m.bray at gmail.com (Erik Bray)
Date: Wed, 18 Jan 2012 17:16:27 -0500
Subject: [Distutils] Distribute instead of setuptools for virtualenv?
In-Reply-To: <CAL0kPAUwNupP2ZUs-Tu13tWPp6XheL02LUcZZ+8d6KB5AsouCQ@mail.gmail.com>
References: <CAPR7qLmpO7aCgmbs2b0-v+zcpjNjzfnnLZxkZCYzQjwrxLV0Qg@mail.gmail.com>
	<CAL0kPAUwNupP2ZUs-Tu13tWPp6XheL02LUcZZ+8d6KB5AsouCQ@mail.gmail.com>
Message-ID: <CAOTD34Y00W3aBm19QCE+N74rGXvdD3y233wmj-v1AXJEUNXzJw@mail.gmail.com>

2012/1/16 Lennart Regebro <regebro at gmail.com>:
> 2012/1/16 Christian ?tef?nescu <st.chris at gmail.com>:
>> Also, if this works just fine, wouldn't it make sense to make Distribute the
>> default for virtualenv at some point?
>
> In my opinion, yes.

FWIW you can also set VIRTUALENV_USE_DISTRIBUTE=1 in your environment.
 Or, in the latest version of virtualenv I think it's just
VIRTUALENV_DISTRIBUTE.  You can also set it in a config file now; I
forget the details but it's explained in the virtualenv docs.

From Sachindeo.Chavan at ge.com  Thu Jan 19 11:55:07 2012
From: Sachindeo.Chavan at ge.com (Chavan, Sachindeo (GE Oil & Gas))
Date: Thu, 19 Jan 2012 10:55:07 -0000
Subject: [Distutils] Unable to execute Python executable on Windows 2000
Message-ID: <4FC17A4D0DED8E42B38330770A672EE10501059A@LONMLVEM05.e2k.ad.ge.com>

Hi all,

I have developed a small application using python 2.5 on XP.
I have the exe converted using py2exe tool. It executes in XP.

I am able to create exe in windows 2000 SP4.

I'm unable trying to run the exe on Windows 2000.

I get the following error

Traceback (most recent call last):
  File "UMDC.py", line 14, in <module>
  File "zipextimporter.pyo", line 82, in load_module
  File "wx\__init__.pyo", line 45, in <module>
  File "zipextimporter.pyo", line 82, in load_module
  File "wx\_core.pyo", line 4, in <module>
  File "zipextimporter.pyo", line 98, in load_module
ImportError: MemoryLoadLibrary failed loading wx\_core_.pyd

Regards,
S

From techtonik at gmail.com  Thu Jan 19 19:19:33 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Thu, 19 Jan 2012 21:19:33 +0300
Subject: [Distutils] Unable to execute Python executable on Windows 2000
In-Reply-To: <4FC17A4D0DED8E42B38330770A672EE10501059A@LONMLVEM05.e2k.ad.ge.com>
References: <4FC17A4D0DED8E42B38330770A672EE10501059A@LONMLVEM05.e2k.ad.ge.com>
Message-ID: <CAPkN8xL6zWYA=HaUgfLFBL4PZB4S23qbX4V0qQQEqrvA9nKLqg@mail.gmail.com>

On Thu, Jan 19, 2012 at 1:55 PM, Chavan, Sachindeo (GE Oil & Gas) <
Sachindeo.Chavan at ge.com> wrote:

> Hi all,
>
> I have developed a small application using python 2.5 on XP.
> I have the exe converted using py2exe tool. It executes in XP.
>
> I am able to create exe in windows 2000 SP4.
>
> I'm unable trying to run the exe on Windows 2000.
>
> I get the following error
>
> Traceback (most recent call last):
>  File "UMDC.py", line 14, in <module>
>  File "zipextimporter.pyo", line 82, in load_module
>  File "wx\__init__.pyo", line 45, in <module>
>  File "zipextimporter.pyo", line 82, in load_module
>  File "wx\_core.pyo", line 4, in <module>
>  File "zipextimporter.pyo", line 98, in load_module
> ImportError: MemoryLoadLibrary failed loading wx\_core_.pyd
>

http://www.py2exe.org/index.cgi/ProblemsToBeFixed
Is that it?
-- 
anatoly t.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120119/29176711/attachment.html>

From yegorslists at googlemail.com  Thu Jan 19 22:54:47 2012
From: yegorslists at googlemail.com (Yegor Yefremov)
Date: Thu, 19 Jan 2012 22:54:47 +0100
Subject: [Distutils] Unable to execute Python executable on Windows 2000
In-Reply-To: <4FC17A4D0DED8E42B38330770A672EE10501059A@LONMLVEM05.e2k.ad.ge.com>
References: <4FC17A4D0DED8E42B38330770A672EE10501059A@LONMLVEM05.e2k.ad.ge.com>
Message-ID: <CAGm1_ksYLZxc8OV1+RnsqtNCUrG9d--VcCNRhQYT-GUwUhq=LQ@mail.gmail.com>

On Thu, Jan 19, 2012 at 11:55 AM, Chavan, Sachindeo (GE Oil & Gas)
<Sachindeo.Chavan at ge.com> wrote:
> Hi all,
>
> I have developed a small application using python 2.5 on XP.
> I have the exe converted using py2exe tool. It executes in XP.
>
> I am able to create exe in windows 2000 SP4.
>
> I'm unable trying to run the exe on Windows 2000.
>
> I get the following error
>
> Traceback (most recent call last):
> ?File "UMDC.py", line 14, in <module>
> ?File "zipextimporter.pyo", line 82, in load_module
> ?File "wx\__init__.pyo", line 45, in <module>
> ?File "zipextimporter.pyo", line 82, in load_module
> ?File "wx\_core.pyo", line 4, in <module>
> ?File "zipextimporter.pyo", line 98, in load_module
> ImportError: MemoryLoadLibrary failed loading wx\_core_.pyd

Just in case.. Have you already tried http://www.pyinstaller.org/? I
could successfully compiled exe with pyserial, netifaces, dpkt  and
pypcap. But it was WinXP, I don't know if it will work with Win 2000.

Yegor

From me at rpatterson.net  Sat Jan 21 11:19:03 2012
From: me at rpatterson.net (Ross Patterson)
Date: Sat, 21 Jan 2012 02:19:03 -0800
Subject: [Distutils] buildout: several fold performance increases
References: <87obu3jbd1.fsf@transitory.lefae.org>
Message-ID: <87fwf98ijs.fsf@transitory.lefae.org>

I moved this patch to a branch of zc.buildout:

svn+ssh://svn.zope.org/repos/main/zc.buildout/branches/env-cache

Digging into one of the additional failures, I traced the problem to
develop eggs.  Since zc.buildout.easy_install.develop() can change what
eggs and versions a pkg_resources.Environment would find, the cached
environments were out of date after buildout processed develop eggs.  I
addressed this by finding any cached environments whose paths include
the path the develop egg is installed into and having those environments
rescan that path.  With that fix, my zc.buildout tests have no
additional failures beyond what they do on my system (with an isolated
python built from source).

I also compared buildout run times (using the 'time' command, not
cProfile) on a real world buildout with 6 identical parts and a few
other parts with very similar distribution requirements.  The time
without the patches was 1m27.513s and with the patches as applied to
zc.buildout 1.4.4 it was 0m34.674s.  Also, buildout.dumppickedversions
suffers the same logging hot spot as I fixed in zc.buildout r122980 and
before I patched it the buildout run time was 2m13s.  Can someone cut a
release of buildout.dumppickedversions?

All told, this is a 4 fold real world improvement.  With all those hot
spots addressed there are no obvious wastes I can find in the profiling
data.  I'd like to merge this into trunk and the 1.4 branch and see
releases cut of both 1.5 and 1.4.  May I begin the merging?

Ross

Ross Patterson <me at rpatterson.net> writes:

> I've long been perplexed by how long a buildout takes to run with
> multiple parts whose required distributions are largely similar.  Taking
> a stab at it, I found two hot spots that yield several fold improvements
> in performance.
>
> First, zc.buildout.easy_install._log_requirements was doing expensive
> requirements parsing and sorting even when no message would be logged.
> I committed a fix for it that on a 10 part buildout with a large "eggs"
> option for each part decreased update time from a cProfile run time of
> 93 seconds to 15 seconds:
>
> http://svn.zope.org/zc.buildout/trunk/src/zc/buildout/easy_install.py?rev=124059&r1=122980&r2=124059
>
> Secondly, instantiating pkg_resources.Environment, including the
> setuptools.package_index.PackageIndex subclass, is very expensive and
> was being done multiple times for any given part, and was being done for
> parts whose environments were identical.  There was some existing global
> caching for package indexes that I've duplicated for environments in the
> attached patch.
>
> Unfortunately, I haven't been able to get a clean test environment for
> the life of me.  I'm using a clean Python 2.7 build from source, turning
> everything in ~/.buildout/default.cfg off, and running tests in a clean
> checkout of the zc.buildout/trunk buildout.  Even under those conditions
> I get 17 failing tests before any changes.  With this environments
> cache, I see 41 failures, but I can't make sense of it.  This patch
> yields another 2-3 fold decrease to 6 seconds for the same buildout and
> is driven by profiling data, not guessing.  Can someone help me get this
> patch in?
>
> Finally, it would be great to see releases of zc.buildout with these
> performance improvements get out in the world.  I've been hearing more
> and more complaints about buildout run times and these are easy fixes.
> If we can get the second, attached patch in quickly, then I'd say we
> should release with both.  If not, then it's still worth it to cut a
> release for the first, already committed patch, which yields the
> greatest improvement.
>
> Thanks!
> Ross


From me at rpatterson.net  Sat Jan 21 13:59:41 2012
From: me at rpatterson.net (Ross Patterson)
Date: Sat, 21 Jan 2012 04:59:41 -0800
Subject: [Distutils] buildout: several fold performance increases
References: <87obu3jbd1.fsf@transitory.lefae.org>
	<87fwf98ijs.fsf@transitory.lefae.org>
Message-ID: <877h0l8b42.fsf@transitory.lefae.org>

Sorry, I spoke a little too soon.  I generated and normalized diffs of
the test output (ugh!) and found some differences that were specific to
my env caching changes.  I've resolved all those differences on my
branch.  Mostly involves clearing the appropriate cached envs when using
easy_install.develop or easy_install.build.

Ross

Ross Patterson <me at rpatterson.net> writes:

> I moved this patch to a branch of zc.buildout:
>
> svn+ssh://svn.zope.org/repos/main/zc.buildout/branches/env-cache
>
> Digging into one of the additional failures, I traced the problem to
> develop eggs.  Since zc.buildout.easy_install.develop() can change what
> eggs and versions a pkg_resources.Environment would find, the cached
> environments were out of date after buildout processed develop eggs.  I
> addressed this by finding any cached environments whose paths include
> the path the develop egg is installed into and having those environments
> rescan that path.  With that fix, my zc.buildout tests have no
> additional failures beyond what they do on my system (with an isolated
> python built from source).
>
> I also compared buildout run times (using the 'time' command, not
> cProfile) on a real world buildout with 6 identical parts and a few
> other parts with very similar distribution requirements.  The time
> without the patches was 1m27.513s and with the patches as applied to
> zc.buildout 1.4.4 it was 0m34.674s.  Also, buildout.dumppickedversions
> suffers the same logging hot spot as I fixed in zc.buildout r122980 and
> before I patched it the buildout run time was 2m13s.  Can someone cut a
> release of buildout.dumppickedversions?
>
> All told, this is a 4 fold real world improvement.  With all those hot
> spots addressed there are no obvious wastes I can find in the profiling
> data.  I'd like to merge this into trunk and the 1.4 branch and see
> releases cut of both 1.5 and 1.4.  May I begin the merging?
>
> Ross
>
> Ross Patterson <me at rpatterson.net> writes:
>
>> I've long been perplexed by how long a buildout takes to run with
>> multiple parts whose required distributions are largely similar.  Taking
>> a stab at it, I found two hot spots that yield several fold improvements
>> in performance.
>>
>> First, zc.buildout.easy_install._log_requirements was doing expensive
>> requirements parsing and sorting even when no message would be logged.
>> I committed a fix for it that on a 10 part buildout with a large "eggs"
>> option for each part decreased update time from a cProfile run time of
>> 93 seconds to 15 seconds:
>>
>> http://svn.zope.org/zc.buildout/trunk/src/zc/buildout/easy_install.py?rev=124059&r1=122980&r2=124059
>>
>> Secondly, instantiating pkg_resources.Environment, including the
>> setuptools.package_index.PackageIndex subclass, is very expensive and
>> was being done multiple times for any given part, and was being done for
>> parts whose environments were identical.  There was some existing global
>> caching for package indexes that I've duplicated for environments in the
>> attached patch.
>>
>> Unfortunately, I haven't been able to get a clean test environment for
>> the life of me.  I'm using a clean Python 2.7 build from source, turning
>> everything in ~/.buildout/default.cfg off, and running tests in a clean
>> checkout of the zc.buildout/trunk buildout.  Even under those conditions
>> I get 17 failing tests before any changes.  With this environments
>> cache, I see 41 failures, but I can't make sense of it.  This patch
>> yields another 2-3 fold decrease to 6 seconds for the same buildout and
>> is driven by profiling data, not guessing.  Can someone help me get this
>> patch in?
>>
>> Finally, it would be great to see releases of zc.buildout with these
>> performance improvements get out in the world.  I've been hearing more
>> and more complaints about buildout run times and these are easy fixes.
>> If we can get the second, attached patch in quickly, then I'd say we
>> should release with both.  If not, then it's still worth it to cut a
>> release for the first, already committed patch, which yields the
>> greatest improvement.
>>
>> Thanks!
>> Ross
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig


From marius at pov.lt  Sat Jan 21 16:00:16 2012
From: marius at pov.lt (Marius Gedminas)
Date: Sat, 21 Jan 2012 17:00:16 +0200
Subject: [Distutils] buildout: several fold performance increases
In-Reply-To: <87fwf98ijs.fsf@transitory.lefae.org>
References: <87obu3jbd1.fsf@transitory.lefae.org>
	<87fwf98ijs.fsf@transitory.lefae.org>
Message-ID: <20120121150015.GA29663@fridge.pov.lt>

On Sat, Jan 21, 2012 at 02:19:03AM -0800, Ross Patterson wrote:
> I moved this patch to a branch of zc.buildout:
> 
> svn+ssh://svn.zope.org/repos/main/zc.buildout/branches/env-cache
> 
[...snip bugfixing...]

> I also compared buildout run times (using the 'time' command, not
> cProfile) on a real world buildout with 6 identical parts and a few
> other parts with very similar distribution requirements.  The time
> without the patches was 1m27.513s and with the patches as applied to
> zc.buildout 1.4.4 it was 0m34.674s.

Ouch.  My slowest buildout takes "only" 41 seconds to do nothing.
Still, I'd welcome a 3x improvement.

> Also, buildout.dumppickedversions
> suffers the same logging hot spot as I fixed in zc.buildout r122980 and
> before I patched it the buildout run time was 2m13s.  Can someone cut a
> release of buildout.dumppickedversions?

There's also buildout-versions [1], which is a fork/reimplementation of
buildout.dumppickedversions, and which I use for my buildouts.  I wonder
if the same speedup could be applied to it.  Perhaps not: if I disable
buildout-versions in my 41s do-nothing buildout, the do-nothing time
stays exactly the same.

  [1] http://pypi.python.org/pypi/buildout-versions

Incidentally, looking at r122980 I wonder if you got the right revision
number [2]?  It seems to be a bugfix for LP#697913 (Buildout doesn't
honor exit code from scripts).

  [2] http://zope3.pov.lt/trac/changeset/122980

> All told, this is a 4 fold real world improvement.  With all those hot
> spots addressed there are no obvious wastes I can find in the profiling
> data.  I'd like to merge this into trunk and the 1.4 branch and see
> releases cut of both 1.5 and 1.4.  May I begin the merging?

I wish Jim Fulton would found some time to look at your branch.  4x
real-world speedup seems very compelling.

How can I test your branch?  It doesn't seem to be enough to add
 
  [buildout]
  develop += /path/to/your/branch/checkout
  [versions]
  zc.buildout = 

to my existing buildout.cfg.

Marius Gedminas
-- 
HOST SYSTEM NOT RESPONDING, PROBABLY DOWN. DO YOU WANT TO WAIT? (Y/N)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120121/3ace171e/attachment.pgp>

From me at rpatterson.net  Sat Jan 21 20:05:08 2012
From: me at rpatterson.net (Ross Patterson)
Date: Sat, 21 Jan 2012 11:05:08 -0800
Subject: [Distutils] buildout: several fold performance increases
References: <87obu3jbd1.fsf@transitory.lefae.org>
	<87fwf98ijs.fsf@transitory.lefae.org>
	<20120121150015.GA29663@fridge.pov.lt>
Message-ID: <87sjj8uba3.fsf@transitory.lefae.org>

Marius Gedminas <marius at pov.lt> writes:

> On Sat, Jan 21, 2012 at 02:19:03AM -0800, Ross Patterson wrote:
>> I moved this patch to a branch of zc.buildout:
>> 
>> svn+ssh://svn.zope.org/repos/main/zc.buildout/branches/env-cache

[...snip...]

>> Also, buildout.dumppickedversions
>> suffers the same logging hot spot as I fixed in zc.buildout r122980 and
>> before I patched it the buildout run time was 2m13s.  Can someone cut a
>> release of buildout.dumppickedversions?
>
> There's also buildout-versions [1], which is a fork/reimplementation of
> buildout.dumppickedversions, and which I use for my buildouts.  I wonder
> if the same speedup could be applied to it.  Perhaps not: if I disable
> buildout-versions in my 41s do-nothing buildout, the do-nothing time
> stays exactly the same.
>
>   [1] http://pypi.python.org/pypi/buildout-versions

Yup, it does suffer the same:

https://github.com/Simplistix/buildout-versions/blob/master/src/buildout_versions/__init__.py#L20

That speedup is proportional to the number of required dists, so your
do-nothing might not suffer much.  In fact, I think all of these
optimizations are at least proportional to the size of your egg cache,
IOW, the number of possible dists available on your path.

> Incidentally, looking at r122980 I wonder if you got the right revision
> number [2]?  It seems to be a bugfix for LP#697913 (Buildout doesn't
> honor exit code from scripts).
>
>   [2] http://zope3.pov.lt/trac/changeset/122980

Oops, sorry about that, it's r124059.  Don't know how I got that wrong:

http://zope3.pov.lt/trac/changeset/124059

> How can I test your branch?  It doesn't seem to be enough to add
>  
>   [buildout]
>   develop += /path/to/your/branch/checkout
>   [versions]
>   zc.buildout = 
>
> to my existing buildout.cfg.

Not sure how.  I've been patching eggs to do the 'timed' tests with
real-world buildout.  Maybe you could install the checkout as a develop
egg to the python installation?  Ick.  Maybe you could edit bin/buildout
and adjust the path for the buildout egg?  Ick.

Ross


From me at rpatterson.net  Sat Jan 21 23:48:55 2012
From: me at rpatterson.net (Ross Patterson)
Date: Sat, 21 Jan 2012 14:48:55 -0800
Subject: [Distutils] buildout: several fold performance increases
References: <87obu3jbd1.fsf@transitory.lefae.org>
	<87fwf98ijs.fsf@transitory.lefae.org>
	<20120121150015.GA29663@fridge.pov.lt>
	<87sjj8uba3.fsf@transitory.lefae.org>
Message-ID: <87r4ysoenc.fsf@transitory.lefae.org>

Ross Patterson <me at rpatterson.net> writes:

> Marius Gedminas <marius at pov.lt> writes:
>
>> On Sat, Jan 21, 2012 at 02:19:03AM -0800, Ross Patterson wrote:
>>> I moved this patch to a branch of zc.buildout:
>>> 
>>> svn+ssh://svn.zope.org/repos/main/zc.buildout/branches/env-cache

[...snip...]

>> How can I test your branch?  It doesn't seem to be enough to add
>>  
>>   [buildout]
>>   develop += /path/to/your/branch/checkout
>>   [versions]
>>   zc.buildout = 
>>
>> to my existing buildout.cfg.
>
> Not sure how.  I've been patching eggs to do the 'timed' tests with
> real-world buildout.  Maybe you could install the checkout as a develop
> egg to the python installation?  Ick.  Maybe you could edit bin/buildout
> and adjust the path for the buildout egg?  Ick.

Here's how I just managed to test my real-world buildouts using my
zc.buildout and buildout.dumppickedversions checkouts and compare those
times against the released versions (1.4 and 0.4 respectively).

To get the buildout to use the zc.buildout checkout, the bootstrap.py
script needs to be convinced to use it when there is not buildout
environment controlling what dists it finds.  As such you need to
install your zc.buildout checkout as a develop egg in the python
installation you are using for the buildout you want to test.

I recommend using a virtualenv so that when you install the develop egg
it only influences that specific buildout:

    $ cd ~/src/my-buildout
    $ virtualenv -p ~/src/buildout.python/python-2.6 --no-site-packages .
    $ bin/python2.6 bootstrap.py -d

Then time your baseline with the existing buildout release:

    $ bin/buildout -N && time bin/buildout -N

Then install the zc.buildout develop egg from where ever your checkout
it into the virtualenv python installation:

    $ cd ~/src/zc.buildout
    $ ~/src/my-buildout/bin/python2.6 setup.py develop

Then use that python with the bootstrap.py script from the checkout and
tell it to use the unreleased version with "-t":

    $ cd ~/src/my-buildout
    $ bin/python2.6 ~/src/zc.buildout/bootstrap/bootstrap.py -dt versions:zc.buildout=

Now time with the zc.buildout checkout by overriding the version pin for
zc.buildout:

    $ bin/buildout -N versions:zc.buildout= && time bin/buildout -N versions:zc.buildout=

Now, time it with the buildout.dumppickedversions checkout by
overriding both version pins and adding a develop egg for
buildout.dumppickedversions.  In the case of buildout.dumppickedversions
we *can* use buildout alone to make it use the develop egg and need not
install it into the virtualenv python installation:

    $ bin/buildout -N versions:zc.buildout= versions:buildout.dumppickedversions= buildout:develop+=/home/me/src/buildout.dumppickedversions && time bin/buildout -N versions:zc.buildout= versions:buildout.dumppickedversions= buildout:develop+=/home/me/src/buildout.dumppickedversions

Finally, cleanup the zc.buildout develop egg:

    $ bin/easy_install-2.6 -m zc.buildout

With this procedure, here are my timings with two different buildouts,
one that is development focused with lots of different parts with
slightly different to wildly different required dists, and one that is a
production buildout with many identical parts.  I did all this after
completely clearing out my egg and download caches and running each
buildout once first to re-download all eggs so that results aren't
skewed by a large, out-of-date egg cache:

+--------------------------------------+-------------+------------+
|                                      | development | production |
+--------------------------------------+-------------+------------+
| released eggs                        | 1m52.068s   | 2m2.731s   |
+--------------------------------------+-------------+------------+
| zc.buildout checkout                 | 0m56.079s   | 1m3.781s   |
+--------------------------------------+-------------+------------+
| buildout.dumppickedversions checkout | 0m16.453s   | 0m20.752s  |
+--------------------------------------+-------------+------------+

All told, roughly a 6-7 fold decrease in buildout runtime.  This really
makes re-running buildout something I no longer feel an urge to avoid.
I'd really like to see this in releases soon so that people may feel
less buildout pain.

Ross


From me at rpatterson.net  Sun Jan 22 02:37:07 2012
From: me at rpatterson.net (Ross Patterson)
Date: Sat, 21 Jan 2012 17:37:07 -0800
Subject: [Distutils] buildout: several fold performance increases
References: <87obu3jbd1.fsf@transitory.lefae.org>
	<87fwf98ijs.fsf@transitory.lefae.org>
	<20120121150015.GA29663@fridge.pov.lt>
	<87sjj8uba3.fsf@transitory.lefae.org>
	<87r4ysoenc.fsf@transitory.lefae.org>
Message-ID: <87ehusmsak.fsf@transitory.lefae.org>

Ross Patterson <me at rpatterson.net> writes:

> Ross Patterson <me at rpatterson.net> writes:
>
>> Marius Gedminas <marius at pov.lt> writes:
>>
>>> On Sat, Jan 21, 2012 at 02:19:03AM -0800, Ross Patterson wrote:
>>>> I moved this patch to a branch of zc.buildout:
>>>> 
>>>> svn+ssh://svn.zope.org/repos/main/zc.buildout/branches/env-cache

[...snip...]

> With this procedure, here are my timings with two different buildouts,
> one that is development focused with lots of different parts with
> slightly different to wildly different required dists, and one that is a
> production buildout with many identical parts.  I did all this after
> completely clearing out my egg and download caches and running each
> buildout once first to re-download all eggs so that results aren't
> skewed by a large, out-of-date egg cache:

I re-timed everything comparing zc.buildout 1.4.4, 1.5.2 and the
env-cache branch.  All of these are with the trunk version of
buildout.dumppickedversions which includes the optimizations from there.
These are all against the production buildout from before but with a
large egg cache.

+------------------+-----------+-----------+-----------+
|                  | 1.4.4     | 1.5.2     | env-cache |
+------------------+-----------+-----------+-----------+
| bin/buildout -N  | 1m28.394s | 0m34.441s | 0m20.468s |
+------------------+-----------+-----------+-----------+
| bin/buildout     | 1m51.173s | 0m57.524s | 0m42.124s |
+------------------+-----------+-----------+-----------+
| bin/buildout -vN | 2m25.618s | 1m18.324s | 1m4.896s  |
+------------------+-----------+-----------+-----------+
| bin/buildout -v  | 2m36.758s | 1m32.049s | 1m27.575s |
+------------------+-----------+-----------+-----------+

So 1.5 has a lot of improvements over 1.4 and these changes offer some
incremental improvement over that but the improvement is a bigger
relative difference when running with "-N" which I think is the more
important case.  Whatever 1.5 does differently also obviates the
_log_requirements optimization, it only applies to 1.4 so I've reverted
it on trunk.

Now that I understand what's going on with zc.buildout.easy_install and
pkg_resources, it seems like the right way to handle this is not to scan
all the paths involved in a working set for all possible dists including
those not required for the working set.  Instead, it might be better to
just scan the paths for dists we know we need.  Maybe we can subclass
pkg_resources.Environment that only stores the path in scan() and then
scans the paths for specific project_names in __getitem__().  With the
current caching, the profiling data indicates very little time spent
scanning, so there's only incremental benefit to doing this in addition
to the caching.  It me be a more correct solution to do this *instead*
of the caching.  Also, with this approach, it may be possible to do
global dist caching which can be updated as things are changed by
hooking into zc.buildout.easy_install's develop(), install(), and
build().  Thoughts?

Ross


From reinout at vanrees.org  Sun Jan 22 19:26:03 2012
From: reinout at vanrees.org (Reinout van Rees)
Date: Sun, 22 Jan 2012 19:26:03 +0100
Subject: [Distutils] buildout: several fold performance increases
In-Reply-To: <87ehusmsak.fsf@transitory.lefae.org>
References: <87obu3jbd1.fsf@transitory.lefae.org>
	<87fwf98ijs.fsf@transitory.lefae.org>
	<20120121150015.GA29663@fridge.pov.lt>
	<87sjj8uba3.fsf@transitory.lefae.org>
	<87r4ysoenc.fsf@transitory.lefae.org>
	<87ehusmsak.fsf@transitory.lefae.org>
Message-ID: <jfhkbs$emr$1@dough.gmane.org>

On 22-01-12 02:37, Ross Patterson wrote:
> Also, with this approach, it may be possible to do
> global dist caching which can be updated as things are changed by
> hooking into zc.buildout.easy_install's develop(), install(), and
> build().  Thoughts?

Would any work you do in this area make it easier to handle selected 
global packages? The hard point there seems to me to keep track of what 
is coming from where. When buildout has a firmer grip itself on where 
what dist is, doing advanced tricks with them might be easier.

Unrelated to your work as such, but as long as you're thinking about a 
global dist cache, throwing an extra possible usecase at you might help 
you design it ;-)


Reinout

-- 
Reinout van Rees                    http://reinout.vanrees.org/
reinout at vanrees.org             http://www.nelen-schuurmans.nl/
"If you're not sure what to do, make something. -- Paul Graham"


From me at rpatterson.net  Sun Jan 22 21:42:03 2012
From: me at rpatterson.net (Ross Patterson)
Date: Sun, 22 Jan 2012 12:42:03 -0800
Subject: [Distutils] buildout: several fold performance increases
References: <87obu3jbd1.fsf@transitory.lefae.org>
	<87fwf98ijs.fsf@transitory.lefae.org>
	<20120121150015.GA29663@fridge.pov.lt>
	<87sjj8uba3.fsf@transitory.lefae.org>
	<87r4ysoenc.fsf@transitory.lefae.org>
	<87ehusmsak.fsf@transitory.lefae.org> <jfhkbs$emr$1@dough.gmane.org>
Message-ID: <87k44jlbac.fsf@transitory.lefae.org>

Reinout van Rees <reinout at vanrees.org> writes:

> On 22-01-12 02:37, Ross Patterson wrote:
>> Also, with this approach, it may be possible to do
>> global dist caching which can be updated as things are changed by
>> hooking into zc.buildout.easy_install's develop(), install(), and
>> build().  Thoughts?
>
> Would any work you do in this area make it easier to handle selected
> global packages? The hard point there seems to me to keep track of
> what is coming from where. When buildout has a firmer grip itself on
> where what dist is, doing advanced tricks with them might be easier.

Not quite following you yet.  Can you elaborate?  If you could also
provide examples, that would be great.

Ross


From reinout at vanrees.org  Mon Jan 23 23:20:26 2012
From: reinout at vanrees.org (Reinout van Rees)
Date: Mon, 23 Jan 2012 23:20:26 +0100
Subject: [Distutils] buildout: several fold performance increases
In-Reply-To: <87k44jlbac.fsf@transitory.lefae.org>
References: <87obu3jbd1.fsf@transitory.lefae.org>
	<87fwf98ijs.fsf@transitory.lefae.org>
	<20120121150015.GA29663@fridge.pov.lt>
	<87sjj8uba3.fsf@transitory.lefae.org>
	<87r4ysoenc.fsf@transitory.lefae.org>
	<87ehusmsak.fsf@transitory.lefae.org>
	<jfhkbs$emr$1@dough.gmane.org>
	<87k44jlbac.fsf@transitory.lefae.org>
Message-ID: <jfkmfa$qdh$1@dough.gmane.org>

On 22-01-12 21:42, Ross Patterson wrote:
> Reinout van Rees<reinout at vanrees.org>  writes:
>
>> >  On 22-01-12 02:37, Ross Patterson wrote:
>>> >>  Also, with this approach, it may be possible to do
>>> >>  global dist caching which can be updated as things are changed by
>>> >>  hooking into zc.buildout.easy_install's develop(), install(), and
>>> >>  build().  Thoughts?
>> >
>> >  Would any work you do in this area make it easier to handle selected
>> >  global packages? The hard point there seems to me to keep track of
>> >  what is coming from where. When buildout has a firmer grip itself on
>> >  where what dist is, doing advanced tricks with them might be easier.
> Not quite following you yet.  Can you elaborate?  If you could also
> provide examples, that would be great.

Assumption: if you look at a global dist cache, that means buildout 
itself keeps really good track of where every dist is. It doesn't have 
to rely on setuptools internals, for instance. Buildout knows where 
every dist it knows about is.

Now, what I saw when trying out buildout 1.5 and telling it to use a 
couple of globally installed python packages, was that it would put the 
system python's library dir quite near the top of buildout's sys.path if 
that was where it found one of the packages that I asked it about.

Apparently it is hard to place such a dir at the end of sys.path for the 
current 1.5 buildouts.

=> My thought "if buildout really keeps track itself where everything 
is, perhaps it can build a better sys.path list".


(Perhaps it is still unclear: in that case please disregard. It was just 
a brainstormy thought.)


Reinout

-- 
Reinout van Rees                    http://reinout.vanrees.org/
reinout at vanrees.org             http://www.nelen-schuurmans.nl/
"If you're not sure what to do, make something. -- Paul Graham"


From marius at pov.lt  Tue Jan 24 15:33:40 2012
From: marius at pov.lt (Marius Gedminas)
Date: Tue, 24 Jan 2012 16:33:40 +0200
Subject: [Distutils] buildout: several fold performance increases
In-Reply-To: <87r4ysoenc.fsf@transitory.lefae.org>
References: <87obu3jbd1.fsf@transitory.lefae.org>
	<87fwf98ijs.fsf@transitory.lefae.org>
	<20120121150015.GA29663@fridge.pov.lt>
	<87sjj8uba3.fsf@transitory.lefae.org>
	<87r4ysoenc.fsf@transitory.lefae.org>
Message-ID: <20120124143340.GA30974@fridge.pov.lt>

On Sat, Jan 21, 2012 at 02:48:55PM -0800, Ross Patterson wrote:
> > Marius Gedminas <marius at pov.lt> writes:
> >> How can I test your branch?
> 
> Here's how I just managed to test my real-world buildouts using my
> zc.buildout and buildout.dumppickedversions checkouts and compare those
> times against the released versions (1.4 and 0.4 respectively).
[...]
> Now time with the zc.buildout checkout by overriding the version pin for
> zc.buildout:
> 
>     $ bin/buildout -N versions:zc.buildout= && time bin/buildout -N versions:zc.buildout=

My baseline is 41s (that includes 'newest = false' in my buildout.cfg,
which is what -N does here).

Your modified buildout's first run was 12s.  Second run decided to
recompile PIL for no reason and took 37s.  Third, fourth and fifth runs
were 10s each.

Awesome.  I want your branch merged.

Marius Gedminas
-- 
Special bonus feature: absolutely nowhere in RetchMail's code is there an
arbitrary 3-second sleep(). Wow! What other mail retriever can say that? (Hint:
not fetchmail.)
	-- http://open.nit.ca/wiki/index.php?page=RetchMail
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120124/bcb1396f/attachment.pgp>

From chris at simplistix.co.uk  Tue Jan 24 19:54:04 2012
From: chris at simplistix.co.uk (Chris Withers)
Date: Tue, 24 Jan 2012 18:54:04 +0000
Subject: [Distutils] [buildout] windows issue causes jenkins problems
In-Reply-To: <4F099A2B.5040504@simplistix.co.uk>
References: <4F099A2B.5040504@simplistix.co.uk>
Message-ID: <4F1EFE4C.8010306@simplistix.co.uk>

Olivier,

ShiningPanda plugin 0.7 looks fantastic.
Sadly, for at least one of the things I wanted to use it for there's a snag:

On 08/01/2012 13:29, Chris Withers wrote:
> The weird process forking stuff that running buildout on windows
> sometimes does causes problems for Jenkins:
>
> http://jenkins.simplistix.co.uk/job/testfixtures-buildout/32/PYTHON=2.6,label=windows/console
>
> Is there any way I can cause this not to happen?

I had an outside hope that ShiningPanda would somehow wait for the weird 
forking stuff to be done, but sadly not:

http://jenkins.simplistix.co.uk/job/testfixtures-buildout/PYTHON=2.5,label=windows/59/console

So, for now I'll have to stick with the hacky script:

https://github.com/Simplistix/testfixtures/blob/master/jenkins.bat

Any ideas on this?
(other than rewriting buildout not to do the silly forking ;-) )

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk

From me at rpatterson.net  Wed Jan 25 20:20:59 2012
From: me at rpatterson.net (Ross Patterson)
Date: Wed, 25 Jan 2012 11:20:59 -0800
Subject: [Distutils] buildout: several fold performance increases
References: <87obu3jbd1.fsf@transitory.lefae.org>
	<87fwf98ijs.fsf@transitory.lefae.org>
	<20120121150015.GA29663@fridge.pov.lt>
	<87sjj8uba3.fsf@transitory.lefae.org>
	<87r4ysoenc.fsf@transitory.lefae.org>
	<20120124143340.GA30974@fridge.pov.lt>
Message-ID: <878vkv1td0.fsf@transitory.lefae.org>

Marius Gedminas <marius at pov.lt> writes:

> On Sat, Jan 21, 2012 at 02:48:55PM -0800, Ross Patterson wrote:
>> > Marius Gedminas <marius at pov.lt> writes:
>> >> How can I test your branch?
>> 
>> Here's how I just managed to test my real-world buildouts using my
>> zc.buildout and buildout.dumppickedversions checkouts and compare those
>> times against the released versions (1.4 and 0.4 respectively).
> [...]
>> Now time with the zc.buildout checkout by overriding the version pin for
>> zc.buildout:
>> 
>>     $ bin/buildout -N versions:zc.buildout= && time bin/buildout -N versions:zc.buildout=
>
> My baseline is 41s (that includes 'newest = false' in my buildout.cfg,
> which is what -N does here).
>
> Your modified buildout's first run was 12s.  Second run decided to
> recompile PIL for no reason and took 37s.  Third, fourth and fifth runs
> were 10s each.
>
> Awesome.  I want your branch merged.

Was your baseline under zc.buildout 1.5?

Ross


From m.van.rees at zestsoftware.nl  Thu Jan 26 01:48:05 2012
From: m.van.rees at zestsoftware.nl (Maurits van Rees)
Date: Thu, 26 Jan 2012 01:48:05 +0100
Subject: [Distutils] buildout: several fold performance increases
In-Reply-To: <87sjj8uba3.fsf@transitory.lefae.org>
References: <87obu3jbd1.fsf@transitory.lefae.org>
	<87fwf98ijs.fsf@transitory.lefae.org>
	<20120121150015.GA29663@fridge.pov.lt>
	<87sjj8uba3.fsf@transitory.lefae.org>
Message-ID: <jfq7s5$kj9$1@dough.gmane.org>

Op 21-01-12 20:05, Ross Patterson schreef:
> That speedup is proportional to the number of required dists, so your
> do-nothing might not suffer much.  In fact, I think all of these
> optimizations are at least proportional to the size of your egg cache,
> IOW, the number of possible dists available on your path.

I had over 5000 eggs in an eggs directory that I share over lots of 
buildouts.  I have now removed the oldest 4000 eggs.  Then I reran 
buildout for a few projects; this added several of those eggs again as 
they were still used after all.  Then I compared the time taken by a 
buildout before and after.

A sample output of old data was this:
real    0m59.552s
user    0m29.117s
sys     0m4.197s

and new data was this:
real	0m49.364s
user	0m19.654s
sys	0m3.578s

So that is about a 30% time decrease; I have also seen about 50% decrease.

So removing unused eggs indeed helps.  That is good to know, thanks.

(Note that this is without your zc.buildout speed improvements.)


-- 
Maurits van Rees   http://maurits.vanrees.org/
Web App Programmer at Zest Software: http://zestsoftware.nl
"Logical thinking shows conclusively that logical thinking
is inconclusive." - My summary of G?del, Escher, Bach


From marius at pov.lt  Thu Jan 26 14:45:08 2012
From: marius at pov.lt (Marius Gedminas)
Date: Thu, 26 Jan 2012 15:45:08 +0200
Subject: [Distutils] buildout: several fold performance increases
In-Reply-To: <878vkv1td0.fsf@transitory.lefae.org>
References: <87obu3jbd1.fsf@transitory.lefae.org>
	<87fwf98ijs.fsf@transitory.lefae.org>
	<20120121150015.GA29663@fridge.pov.lt>
	<87sjj8uba3.fsf@transitory.lefae.org>
	<87r4ysoenc.fsf@transitory.lefae.org>
	<20120124143340.GA30974@fridge.pov.lt>
	<878vkv1td0.fsf@transitory.lefae.org>
Message-ID: <20120126134508.GA13862@fridge.pov.lt>

On Wed, Jan 25, 2012 at 11:20:59AM -0800, Ross Patterson wrote:
> Marius Gedminas <marius at pov.lt> writes:
> 
> > On Sat, Jan 21, 2012 at 02:48:55PM -0800, Ross Patterson wrote:
> >> Now time with the zc.buildout checkout by overriding the version pin for
> >> zc.buildout:
> >> 
> >>     $ bin/buildout -N versions:zc.buildout= && time bin/buildout -N versions:zc.buildout=
> >
> > My baseline is 41s (that includes 'newest = false' in my buildout.cfg,
> > which is what -N does here).
> >
> > Your modified buildout's first run was 12s.  Second run decided to
> > recompile PIL for no reason and took 37s.  Third, fourth and fifth runs
> > were 10s each.
> >
> > Awesome.  I want your branch merged.
> 
> Was your baseline under zc.buildout 1.5?

1.5.2.

Incidentally, I pinged Jim Fulton on IRC (#zope3-dev on FreeNode), but
he's very busy currently and so does not have the time to take a look at
your branch right now.  :(

Marius Gedminas
-- 
  To express oneself
  In seventeen syllables
  Is very diffic
		-- John Cooper Clark.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120126/63542989/attachment.pgp>

From marius at pov.lt  Thu Jan 26 14:48:22 2012
From: marius at pov.lt (Marius Gedminas)
Date: Thu, 26 Jan 2012 15:48:22 +0200
Subject: [Distutils] buildout: several fold performance increases
In-Reply-To: <jfq7s5$kj9$1@dough.gmane.org>
References: <87obu3jbd1.fsf@transitory.lefae.org>
	<87fwf98ijs.fsf@transitory.lefae.org>
	<20120121150015.GA29663@fridge.pov.lt>
	<87sjj8uba3.fsf@transitory.lefae.org>
	<jfq7s5$kj9$1@dough.gmane.org>
Message-ID: <20120126134822.GB13862@fridge.pov.lt>

On Thu, Jan 26, 2012 at 01:48:05AM +0100, Maurits van Rees wrote:
> Op 21-01-12 20:05, Ross Patterson schreef:
> >That speedup is proportional to the number of required dists, so your
> >do-nothing might not suffer much.  In fact, I think all of these
> >optimizations are at least proportional to the size of your egg cache,
> >IOW, the number of possible dists available on your path.
> 
> I had over 5000 eggs in an eggs directory that I share over lots of
> buildouts.  I have now removed the oldest 4000 eggs.  Then I reran
> buildout for a few projects; this added several of those eggs again
> as they were still used after all.  Then I compared the time taken
> by a buildout before and after.
> 
> A sample output of old data was this:
> real    0m59.552s
> user    0m29.117s
> sys     0m4.197s
> 
> and new data was this:
> real	0m49.364s
> user	0m19.654s
> sys	0m3.578s
> 
> So that is about a 30% time decrease; I have also seen about 50% decrease.
> 
> So removing unused eggs indeed helps.  That is good to know, thanks.

Some info about my timing data:

  $ ls ~/.buildout/eggs/|wc -l
  1791

And since I suspect there's some filename-based filtering for Python version
numbers:

  $ ls -d ~/.buildout/eggs/*-py2.6.egg|wc -l
  703

Marius Gedminas
-- 
It's my understanding that although in principle TCP can handle huge
throughputs in practice many stacks haven't been optimized for that case, so
you have to either use a utility which opens multiple TCP sessions in parallel
or do something really radical like upgrade to the latest version of the linux
kernel.
		-- Bram Cohen
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120126/1ff22c93/attachment.pgp>

From john.calcote at gmail.com  Thu Jan 26 19:49:12 2012
From: john.calcote at gmail.com (John Calcote)
Date: Thu, 26 Jan 2012 11:49:12 -0700
Subject: [Distutils] buildout: referencing sub-project with bad directory
	form
Message-ID: <02c601ccdc5b$32f36b90$98da42b0$@gmail.com>

Hi,

 

I'm new to buildout. The project I'm working on uses pywbem
(http://pywbem.sf.net), which seems to have some bad form in its project
directory structure. Since pywbem isn't up on PyPI, I've checked out the
project from sf.net as a subdirectory of my project. Here's a tree listing
of my project directory (the basic structure was built by paster using the
modern_project_template):

 

c:\...\my-project>tree /A

.

+---pywbem

|   +---irecv

|   \---testsuite

|       \---testmofs

\---src

    +---myproject

    \---my_project.egg-info

 

Note that the pywbem sub-directory has no src directory. Its setup.py looks
like this:

 

from distutils.core import setup, Extension

import sys, string, os

import shutil

 

import mof_compiler

mof_compiler._build()

 

args = {'name': 'pywbem',

        'author': 'Tim Potter',

        'author_email': 'tpot at hp.com',

        'description': 'Python WBEM client library',

        'long_description': __doc__,

        'platforms': ['any'],

        'url': 'http://pywbem.sf.net/',

        'version': '0.7.0',

        'license': 'LGPLv2',

        'packages': ['pywbem'],

        # Make packages in root dir appear in pywbem module

        'package_dir': {'pywbem': ''}, 

        # Make extensions in root dir appear in pywbem module

        'ext_package': 'pywbem',

        }

 

setup(**args)

 

The problem I'm having is that when I add pywbem as a devel directory to my
buildout.cfg file, the sys.path is configured to assume that pywbem has a
src directory so it can't find the module because '.' Is not in the
sys.path. Only my own 'src' directory - and 'pywbem', of course. 

 

I was about to reorganize the pywbem project a bit, but I thought I'd ask if
anyone has any ideas on how to get buildout to consume pywbem in its current
form. I don't own the project, and I have no developer rights to it, so I'd
like to make as few changes as possible to it.

 

Thanks in advance,

John Calcote

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120126/0626c809/attachment-0001.html>

From john.calcote at gmail.com  Thu Jan 26 21:57:00 2012
From: john.calcote at gmail.com (jcalcote)
Date: Thu, 26 Jan 2012 12:57:00 -0800 (PST)
Subject: [Distutils] buildout: referencing sub-project with bad
	directory form
In-Reply-To: <02c601ccdc5b$32f36b90$98da42b0$@gmail.com>
References: <02c601ccdc5b$32f36b90$98da42b0$@gmail.com>
Message-ID: <1327611420707-4341656.post@n6.nabble.com>

So, I discovered a way of doing this - I moved the 'pywbem' directory into my
project's 'src' directory:

c:\...\my-project>tree /A
?
\---src
    +---myproject
    +---my_project.egg-info
    \---pywbem
        +---irecv
        \---testsuite
            \---testmofs

then changed my buildout.cfg file from:

    [buildout]
        devel = . pywbem

to:

    [buildout]
        devel = . src/pywbem

This works, of course, because my 'src' directory is one of the items that
buildout adds to sys.path, so pywbem is naturally found there. Naturally,
'src/pywbem' is also added to the sys.path, but that addition has no effect
in this case.

I consider this a hack. If anyone has better ideas, I'd love to hear them...

Thanks,
John

--
View this message in context: http://python.6.n6.nabble.com/buildout-referencing-sub-project-with-bad-directory-form-tp4341312p4341656.html
Sent from the Python - distutils-sig mailing list archive at Nabble.com.

From me at rpatterson.net  Fri Jan 27 17:56:32 2012
From: me at rpatterson.net (Ross Patterson)
Date: Fri, 27 Jan 2012 08:56:32 -0800
Subject: [Distutils] buildout: several fold performance increases
References: <87obu3jbd1.fsf@transitory.lefae.org>
	<87fwf98ijs.fsf@transitory.lefae.org>
	<20120121150015.GA29663@fridge.pov.lt>
	<87sjj8uba3.fsf@transitory.lefae.org>
Message-ID: <87wr8d13un.fsf@transitory.lefae.org>

Ross Patterson <me at rpatterson.net> writes:

> Marius Gedminas <marius at pov.lt> writes:
>
>> On Sat, Jan 21, 2012 at 02:19:03AM -0800, Ross Patterson wrote:
>>>
>>> Also, buildout.dumppickedversions suffers the same logging hot spot
>>> as I fixed in zc.buildout r122980 and before I patched it the
>>> buildout run time was 2m13s.  Can someone cut a release of
>>> buildout.dumppickedversions?
>>
>> There's also buildout-versions [1], which is a fork/reimplementation
>> of buildout.dumppickedversions, and which I use for my buildouts.  I
>> wonder if the same speedup could be applied to it.  Perhaps not: if I
>> disable buildout-versions in my 41s do-nothing buildout, the
>> do-nothing time stays exactly the same.
>>
>>   [1] http://pypi.python.org/pypi/buildout-versions
>
> Yup, it does suffer the same:
>
> https://github.com/Simplistix/buildout-versions/blob/master/src/buildout_versions/__init__.py#L20

I just timed a production buildout with zc.buildout 1.5.2 and compared
buildout.dumppickedversions 0.5 against buildout-versions 1.7.

    buildout.dumppickedversions 0.5 = 0m33.449s
    buildout-versions 1.7           = 1m11.667s

So yeah, your package seems to suffer the same problem as I fixed in
buildout.dumppickedversions to the tune of more than a 2x increase in
run time.

HTH,
Ross


From dglick at gmail.com  Sat Jan 28 21:27:56 2012
From: dglick at gmail.com (David Glick)
Date: Sat, 28 Jan 2012 12:27:56 -0800
Subject: [Distutils] buildout: several fold performance increases
In-Reply-To: <87obu3jbd1.fsf@transitory.lefae.org>
References: <87obu3jbd1.fsf@transitory.lefae.org>
Message-ID: <4F245A4C.9050604@gmail.com>

On 1/16/12 6:45 PM, Ross Patterson wrote:
> Unfortunately, I haven't been able to get a clean test environment for
> the life of me.  I'm using a clean Python 2.7 build from source, turning
> everything in ~/.buildout/default.cfg off, and running tests in a clean
> checkout of the zc.buildout/trunk buildout.  Even under those conditions
> I get 17 failing tests before any changes.  With this environments
> cache, I see 41 failures, but I can't make sense of it.  This patch
> yields another 2-3 fold decrease to 6 seconds for the same buildout and
> is driven by profiling data, not guessing.  Can someone help me get this
> patch in?
>
>
See DEVELOPERS.txt in the zc.buildout source for how to get a working 
test setup. In particular you need to run dev.py using a Python without 
setuptools present, and you need to have python2.5 available on your path.

David

From aclark at aclark.net  Sat Jan 28 22:22:06 2012
From: aclark at aclark.net (Alex Clark)
Date: Sat, 28 Jan 2012 16:22:06 -0500
Subject: [Distutils] buildout: several fold performance increases
In-Reply-To: <20120126134508.GA13862@fridge.pov.lt>
References: <87obu3jbd1.fsf@transitory.lefae.org>
	<87fwf98ijs.fsf@transitory.lefae.org>
	<20120121150015.GA29663@fridge.pov.lt>
	<87sjj8uba3.fsf@transitory.lefae.org>
	<87r4ysoenc.fsf@transitory.lefae.org>
	<20120124143340.GA30974@fridge.pov.lt>
	<878vkv1td0.fsf@transitory.lefae.org>
	<20120126134508.GA13862@fridge.pov.lt>
Message-ID: <4F2466FE.5070703@aclark.net>

On 1/26/12 8:45 AM, Marius Gedminas wrote:
> On Wed, Jan 25, 2012 at 11:20:59AM -0800, Ross Patterson wrote:
>> Marius Gedminas<marius at pov.lt>  writes:
>>
>>> On Sat, Jan 21, 2012 at 02:48:55PM -0800, Ross Patterson wrote:
>>>> Now time with the zc.buildout checkout by overriding the version pin for
>>>> zc.buildout:
>>>>
>>>>      $ bin/buildout -N versions:zc.buildout=&&  time bin/buildout -N versions:zc.buildout=
>>>
>>> My baseline is 41s (that includes 'newest = false' in my buildout.cfg,
>>> which is what -N does here).
>>>
>>> Your modified buildout's first run was 12s.  Second run decided to
>>> recompile PIL for no reason and took 37s.  Third, fourth and fifth runs
>>> were 10s each.
>>>
>>> Awesome.  I want your branch merged.
>>
>> Was your baseline under zc.buildout 1.5?
>
> 1.5.2.
>
> Incidentally, I pinged Jim Fulton on IRC (#zope3-dev on FreeNode), but
> he's very busy currently and so does not have the time to take a look at
> your branch right now.  :(


What about Gary Poster, is he still doing any zc.buildout work? He did 
the last release IIRC. Or he was at least active with the 1.5.x series.


Alex



>
> Marius Gedminas
>
>
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig


-- 
Alex Clark ? http://pythonpackages.com


From me at rpatterson.net  Sat Jan 28 22:24:22 2012
From: me at rpatterson.net (Ross Patterson)
Date: Sat, 28 Jan 2012 13:24:22 -0800
Subject: [Distutils] buildout: several fold performance increases
References: <87obu3jbd1.fsf@transitory.lefae.org> <4F245A4C.9050604@gmail.com>
Message-ID: <87r4yj1px5.fsf@transitory.lefae.org>

David Glick <dglick at gmail.com> writes:

> On 1/16/12 6:45 PM, Ross Patterson wrote:
>> Unfortunately, I haven't been able to get a clean test environment for
>> the life of me.  I'm using a clean Python 2.7 build from source, turning
>> everything in ~/.buildout/default.cfg off, and running tests in a clean
>> checkout of the zc.buildout/trunk buildout.  Even under those conditions
>> I get 17 failing tests before any changes.  With this environments
>> cache, I see 41 failures, but I can't make sense of it.  This patch
>> yields another 2-3 fold decrease to 6 seconds for the same buildout and
>> is driven by profiling data, not guessing.  Can someone help me get this
>> patch in?
>>
>>
> See DEVELOPERS.txt in the zc.buildout source for how to get a working
> test setup. In particular you need to run dev.py using a Python
> without setuptools present, and you need to have python2.5 available
> on your path.

Thanks for this tip and, as I said on freenode, sorry for not RTFMing.
To get this working i had to do completely separate Python 2.6 and 2.5
builds from source like so:

    $ ./configure --prefix=~/src/zc.buildout && make && make install

That way my zc.buildout checkout is it's *own* python installation of
two versions.  With this and the differences in DEVELOPERS.txt, I now
have all tests passing on trunk.  Yay!  Now I'm looking into the
failures on my branch.

FYI, I also tried doing:

    $ virtualenv -p  ~/src/buildout.python/python-2.6/bin/python2.6 --no-site-packages --distribute --clear .

...for both Python 2.5 and 2.6 and then deleting the distribute egg from
lib/python2.#/site-packages but even that resulted in test failures.
Not sure if this indicates a problem with the way buildout.python builds
python.

Ross


From dglick at gmail.com  Sat Jan 28 22:28:35 2012
From: dglick at gmail.com (David Glick)
Date: Sat, 28 Jan 2012 13:28:35 -0800
Subject: [Distutils] buildout: several fold performance increases
In-Reply-To: <87r4yj1px5.fsf@transitory.lefae.org>
References: <87obu3jbd1.fsf@transitory.lefae.org> <4F245A4C.9050604@gmail.com>
	<87r4yj1px5.fsf@transitory.lefae.org>
Message-ID: <4F246883.7040509@gmail.com>

On 1/28/12 1:24 PM, Ross Patterson wrote:
> FYI, I also tried doing:
>
>      $ virtualenv -p  ~/src/buildout.python/python-2.6/bin/python2.6 --no-site-packages --distribute --clear .
>
> ...for both Python 2.5 and 2.6 and then deleting the distribute egg from
> lib/python2.#/site-packages but even that resulted in test failures.
> Not sure if this indicates a problem with the way buildout.python builds
> python.
>
>
I think this is simply because some of the zc.buildout tests run python 
-S (don't use site.py), and virtualenv relies on a modified site.py to 
add the stdlib to sys.path. IOW, the zc.buildout tests are not 
compatible with virtualenv's approach to isolation.
David

From me at rpatterson.net  Sun Jan 29 06:25:50 2012
From: me at rpatterson.net (Ross Patterson)
Date: Sat, 28 Jan 2012 21:25:50 -0800
Subject: [Distutils] buildout: several fold performance increases
References: <87obu3jbd1.fsf@transitory.lefae.org> <4F245A4C.9050604@gmail.com>
	<87r4yj1px5.fsf@transitory.lefae.org> <4F246883.7040509@gmail.com>
Message-ID: <87k44b13mp.fsf@transitory.lefae.org>

David Glick <dglick at gmail.com> writes:

> On 1/28/12 1:24 PM, Ross Patterson wrote:
>> FYI, I also tried doing:
>>
>>      $ virtualenv -p ~/src/buildout.python/python-2.6/bin/python2.6
>> --no-site-packages --distribute --clear .
>>
>> ...for both Python 2.5 and 2.6 and then deleting the distribute egg from
>> lib/python2.#/site-packages but even that resulted in test failures.
>> Not sure if this indicates a problem with the way buildout.python builds
>> python.
>>
> I think this is simply because some of the zc.buildout tests run
> python -S (don't use site.py), and virtualenv relies on a modified
> site.py to add the stdlib to sys.path. IOW, the zc.buildout tests are
> not compatible with virtualenv's approach to isolation.

Ah, I didn't know that's how virtualenv did things.  Thanks for the
explanation.

BTW, thanks to your help, I have properly running tests and have gotten
the env-cache branch to pass all tests.  Yay!

So, now that all tests are passing, can someone review this branch and
give me feedback?

Ross


From aclark at aclark.net  Sun Jan 29 17:28:06 2012
From: aclark at aclark.net (Alex Clark)
Date: Sun, 29 Jan 2012 11:28:06 -0500
Subject: [Distutils] Is the Packaging Guide dead?
In-Reply-To: <8e5f4c2bb0c03d9257605dfca18df100@netwok.org>
References: <CAPkN8x+xkdWcZomK7S3x+edkoniFrg3hZ4U4fpA=nyiH9iRQsg@mail.gmail.com>
	<8e5f4c2bb0c03d9257605dfca18df100@netwok.org>
Message-ID: <jg3s2n$urn$1@dough.gmane.org>

On 1/17/12 12:25 PM, ?ric Araujo wrote:
>> The last commit to Packaging Guide was 2010-07-06.
>> https://bitbucket.org/tarek/hitchhiker-guide-packaging/
>>
>> The last message in the mailing list was 2010-05-05
>> https://groups.google.com/forum/#!forum/packaging-guide
>>
>> Is the project dead or moved?
>
> Dead. Tarek?s weblog posts in 2011 explain how distutils2/packaging
> will replace distribute in the long run. I?m still working on the
> packaging docs, and maybe when this first set of doc is in a good shape
> I?ll work on updating the distribute packaging guide.

I wouldn't declare it dead based on that info (i.e. that 
distutils2/packaging will replace distribute.)

Distribute is still the guiding force in getting everyone to 
distutils/packaging IIUC.

So, the documentation has not been updated recently, but everything in 
the guide is still relevant AFAIK.

Is this a fair/accurate assessment?


>
> Regards
> _______________________________________________
> Distutils-SIG maillist - Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig


-- 
Alex Clark ? http://pythonpackages.com