From gora.khargosh at gmail.com Thu Dec 2 11:35:11 2010 From: gora.khargosh at gmail.com (Gora Khargosh) Date: Thu, 2 Dec 2010 16:05:11 +0530 Subject: [Distutils] Problem with "custom" version nosetests Message-ID: Greetings, I'm trying to get nosetests to run doctests within my source tree, but I get this instead. $ nosetests --with-doctest Usage: This is a custom version of the zc.buildout nosetests script. It is intended to meet a temporary need if you encounter problems with the zc.buildout 1.5 release. nosetests: error: no such option: --with-doctest Can someone tell me why nose would be "replaced" with a custom version and how do I get nosetests to run properly by removing this custom version? Thank you. =) Cheers. -- Gora Khargosh ? ???? ??????? -------------- next part -------------- An HTML attachment was scrubbed... URL: From gora.khargosh at gmail.com Thu Dec 2 16:24:20 2010 From: gora.khargosh at gmail.com (Gora Khargosh) Date: Thu, 2 Dec 2010 20:54:20 +0530 Subject: [Distutils] Problem with "custom" version nosetests In-Reply-To: References: Message-ID: On Thu, Dec 2, 2010 at 4:05 PM, Gora Khargosh wrote: > Greetings, > > I'm trying to get nosetests to run doctests within my source tree, but I > get this instead. > > $ nosetests --with-doctest > Usage: This is a custom version of the zc.buildout nosetests script. It is > intended to meet a temporary need if you encounter problems with the > zc.buildout 1.5 release. > > nosetests: error: no such option: --with-doctest > > Can someone tell me why nose would be "replaced" with a custom version > and how do I get nosetests to run properly by removing this custom version? > > Thank you. =) > > Cheers. > -- > Gora Khargosh ? ???? ??????? > > I have found the problem. I have a bootstrap.py in my project directory which nosetests scans for tests and ends up being borked. Giving the bootstrap.py file executable permissions makes nosetests ignore it and run correctly as expected. =) HTH anyone who faces the same problem. Cheers. -- Gora Khargosh ? ???? ??????? -------------- next part -------------- An HTML attachment was scrubbed... URL: From devyan.parmar at gmail.com Fri Dec 3 11:55:52 2010 From: devyan.parmar at gmail.com (devyan parmar) Date: Fri, 3 Dec 2010 16:25:52 +0530 Subject: [Distutils] Unable to install packages after specifying package version Message-ID: Hello, i am trying to install package from my packageserver using PIP. for example: m trying /bin/pip install --extra-index-url=http://mypkgserver/devpackage_name==2.4.12b "2.4.12b" is the version of my package but in package server it is going as "2.4.12b_1" extra part in version is nothing but a build number which getting generated from hudson JOB. is setuptool avoids extra build number which goes with my pcakage version ?? coz if i try to install package it's gives me message "No distributions at all found ". is there anything wrong am doing in package version.?? Thanks Regards Devyan Parmar -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl at meyerloewen.net Fri Dec 3 13:35:54 2010 From: carl at meyerloewen.net (Carl Meyer) Date: Fri, 03 Dec 2010 07:35:54 -0500 Subject: [Distutils] Unable to install packages after specifying package version In-Reply-To: References: Message-ID: <4CF8E42A.5090005@meyerloewen.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, On 12/03/2010 05:55 AM, devyan parmar wrote: > i am trying to install package from my packageserver using PIP. > > for example: > m trying > /bin/pip install --extra-index-url=http://mypkgserver/dev > package_name==2.4.12b > > "2.4.12b" is the version of my package but in package server it is going > as "2.4.12b_1" extra part in version is nothing but a build number which > getting generated from hudson JOB. > is setuptool avoids extra build number which goes with my pcakage > version ?? coz if i try to install package it's gives me message "No > distributions at all found ". No, neither setuptools nor pip will arbitrarily ignore parts of your version number. You need to make sure the package appears in your index with the same version numbering scheme you plan to use for installing it. Carl -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkz45CoACgkQFRcxmeyPUXJx6QCcDUt6PIf6khe+rA35sihP7vQ5 aI8An3mhLopQQfBUhsGxsPLYTMqp2hpD =bEKu -----END PGP SIGNATURE----- From mailing at franzoni.eu Fri Dec 3 15:14:16 2010 From: mailing at franzoni.eu (Alan Franzoni) Date: Fri, 3 Dec 2010 15:14:16 +0100 Subject: [Distutils] specifying dependencies to latest major/minor Message-ID: Hello, I've got some issues with dependency specification; this seems to apply both to setuptools and distribute. Let's suppose I want to have a dependency on a certain library major.minor , because I want to depend on a certain API - which is supposed to be stable for that lib unless at least the minor changes - but I want to pick in any bugfix release they might end up with. I'd like to do something like "pick up the latest 1.1 release of mylib" The only way to do it seems to be this: install_requires = [ "mylib>=1.1, <1.1.999" ] if I use just == 1.1 , it picks 1.1.0 . If I use >= 1.1 and <1.2.0, it will pick beta releases for 1.2 as well. That's quite clumsy, since it's something I do often and it's quite a common practice (I suppose); isn't there any regex/wildcald character support? Would any patch about that be accepted? -- Alan Franzoni -- contact me at public@[mysurname].eu From pje at telecommunity.com Fri Dec 3 16:13:22 2010 From: pje at telecommunity.com (P.J. Eby) Date: Fri, 03 Dec 2010 10:13:22 -0500 Subject: [Distutils] specifying dependencies to latest major/minor In-Reply-To: References: Message-ID: <20101203151342.D2EA13A4092@sparrow.telecommunity.com> At 03:14 PM 12/3/2010 +0100, Alan Franzoni wrote: >Hello, >I've got some issues with dependency specification; this seems to >apply both to setuptools and distribute. > >Let's suppose I want to have a dependency on a certain library >major.minor , because I want to depend on a certain API - which is >supposed to be stable for that lib unless at least the minor changes - >but I want to pick in any bugfix release they might end up with. > >I'd like to do something like "pick up the latest 1.1 release of mylib" > >The only way to do it seems to be this: > >install_requires = [ "mylib>=1.1, <1.1.999" ] > >if I use just == 1.1 , it picks 1.1.0 . If I use >= 1.1 and <1.2.0, it >will pick beta releases for 1.2 as well. Use ">=1.1,<1.2dev". From pje at telecommunity.com Fri Dec 3 17:44:31 2010 From: pje at telecommunity.com (P.J. Eby) Date: Fri, 03 Dec 2010 11:44:31 -0500 Subject: [Distutils] Unable to install packages after specifying package version In-Reply-To: References: Message-ID: <20101203164450.EA4223A4092@sparrow.telecommunity.com> At 04:25 PM 12/3/2010 +0530, devyan parmar wrote: >Hello, > >i am trying to install package from my packageserver using PIP. > >for example: >m trying >/bin/pip install >--extra-index-url=http://mypkgserver/dev >package_name==2.4.12b > >"2.4.12b" is the version of my package but in package server it is >going as "2.4.12b_1" extra part in version is nothing but a build >number which getting generated from hudson JOB. >is setuptool avoids extra build number which goes with my pcakage version ?? No. That's why it's not being found. Ask for "package_name==2.4.12b-1" instead. From mailing at franzoni.eu Fri Dec 3 18:28:47 2010 From: mailing at franzoni.eu (Alan Franzoni) Date: Fri, 3 Dec 2010 18:28:47 +0100 Subject: [Distutils] zc.buildout fails to use system-installed dep? Message-ID: Hello, there's a problem with zc.buildout that I can reproduce very easily on Ubuntu 10.10 (64 bit) it seems that zc.buildout does not download the dependency from pypi if the very same version is already available in the host system, but then for some reason does not add it to sys.path and/or is unable to use it for some reason; zope.testrunner hence fails on importing zope.interface python-zope.interface is installed at version 3.6.1, and zope.testrunner seems to use that very same dependency. Any workaround? I'll open an issue on launchpad in the weekend. Do this in order to reproduce the problem: # start cd /tmp mkdir -p MyProj/myproj cd MyProj cat < buildout.cfg [buildout] parts = test develop = myproj [test] recipe = zc.recipe.testrunner eggs = myproj EOM cat < myproj/setup.py from setuptools import setup setup( name="myproj", version="1.0.0", description="myproj" ) EOM wget http://svn.zope.org/*checkout*/zc.buildout/trunk/bootstrap/bootstrap.py python bootstrap.py --distribute && bin/buildout -vvv bin/test #end output from buildout run: alan at afra:/tmp/MyProj$ python bootstrap.py --distribute && bin/buildout Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.14.tar.gz Extracting in /tmp/tmpUAAumj Now working in /tmp/tmpUAAumj/distribute-0.6.14 Building a Distribute egg in /tmp/tmphpzBsD /tmp/tmphpzBsD/distribute-0.6.14-py2.6.egg install_dir /tmp/tmphpzBsD Creating directory '/tmp/MyProj/bin'. Creating directory '/tmp/MyProj/parts'. Creating directory '/tmp/MyProj/eggs'. Creating directory '/tmp/MyProj/develop-eggs'. Generated script '/tmp/MyProj/bin/buildout'. Upgraded: distribute version 0.6.14; restarting. Develop: '/tmp/MyProj/myproj' install_dir /tmp/MyProj/develop-eggs/tmpoKKOZAbuild Getting distribution for 'zc.recipe.testrunner'. install_dir /tmp/MyProj/eggs/tmpQvIfIg zip_safe flag not set; analyzing archive contents... Got zc.recipe.testrunner 1.4.0. Getting distribution for 'z3c.recipe.scripts>=1.0.0'. install_dir /tmp/MyProj/eggs/tmpQPz2RH Got z3c.recipe.scripts 1.0.1. Getting distribution for 'zope.testrunner'. install_dir /tmp/MyProj/eggs/tmpGvi6JB package init file 'src/zope/testrunner/testrunner-ex/__init__.py' not found (or not a regular file) Creating missing __init__.py for zope.testrunner.testrunner-ex Got zope.testrunner 4.0.0. Getting distribution for 'zc.recipe.egg>=1.3.0'. install_dir /tmp/MyProj/eggs/tmp_pJ4gb Got zc.recipe.egg 1.3.2. Getting distribution for 'zope.exceptions'. install_dir /tmp/MyProj/eggs/tmpxRSjc1 Got zope.exceptions 3.6.1. Installing test. Generated script '/tmp/MyProj/bin/test'. alan at afra:/tmp/MyProj$ bin/test Traceback (most recent call last): File "bin/test", line 20, in import zope.testrunner File "/tmp/MyProj/eggs/zope.testrunner-4.0.0-py2.6.egg/zope/testrunner/__init__.py", line 21, in import zope.testrunner.interfaces File "/tmp/MyProj/eggs/zope.testrunner-4.0.0-py2.6.egg/zope/testrunner/interfaces.py", line 21, in import zope.interface ImportError: No module named interface -- Alan Franzoni -- contact me at public@[mysurname].eu From attilaolah at gmail.com Fri Dec 3 18:53:25 2010 From: attilaolah at gmail.com (=?UTF-8?B?QXR0aWxhIE9sw6Fo?=) Date: Fri, 3 Dec 2010 18:53:25 +0100 Subject: [Distutils] zc.buildout fails to use system-installed dep? In-Reply-To: References: Message-ID: Hi, On Fri, Dec 3, 2010 at 18:28, Alan Franzoni wrote: > Hello, > there's a problem with zc.buildout that I can reproduce very easily on > Ubuntu 10.10 (64 bit) > > it seems that zc.buildout does not download the dependency from pypi > if the very same version is already available in the host system, but > then for some reason does not add it to sys.path and/or is unable to > use it for some reason; zope.testrunner hence fails on importing > zope.interface > > python-zope.interface is installed at version 3.6.1, and > zope.testrunner seems to use that very same dependency. > > Any workaround? I'll open an issue on launchpad in the weekend. > > Do this in order to reproduce the problem: I ran your script, and the result ended with: Total: 0 tests, 0 failures, 0 errors in 0.000 seconds. I also have zope.interface 3.6.1 on my system. Gentoo, Python 2.7.1+. $ eix zope.interface [D] net-zope/zope-interface Available versions: 3.5.1 3.5.2 (~)3.5.3 Installed versions: 3.6.1(06:23:08 10/23/10) Homepage: http://pypi.python.org/pypi/zope.interface Description: Interfaces for Python Perhaps try adding this to the buildout config: include-site-packages = true allowed-eggs-from-site-packages = zope.interface Attila From mailing at franzoni.eu Fri Dec 3 20:35:14 2010 From: mailing at franzoni.eu (Alan Franzoni) Date: Fri, 3 Dec 2010 20:35:14 +0100 Subject: [Distutils] zc.buildout fails to use system-installed dep? In-Reply-To: References: Message-ID: I had only linux to test it with, and I could reproduce the bug on two distinct Ubuntu 10.10 64 bit workstations. It doesn't happen on Ubuntu 10.04, but that distro is got zope.interface 3.5.something and I can see zope.interface gets downloaded again. It doesn't happen on mac os x with python 2.7 either, even though zope.interface was installed through easy_install. I forgot to specify that Ubuntu 10.10 is got Python 2.6.5. I suspect it's something ubuntu-specific, maybe related to the fact that it's not an egg-installed package, but a .deb installed one - some interference between the two install systems is not unlikely. Just some questions for you, Attila: - was your zope.interface installed through portage or via easy_install/pip? - can you see zope.interface 3.6.1 being downloaded during the buildout launch phase? -- Alan Franzoni -- contact me at public@[mysurname].eu From attilaolah at gmail.com Fri Dec 3 20:45:49 2010 From: attilaolah at gmail.com (=?UTF-8?B?QXR0aWxhIE9sw6Fo?=) Date: Fri, 3 Dec 2010 20:45:49 +0100 Subject: [Distutils] zc.buildout fails to use system-installed dep? In-Reply-To: References: Message-ID: Hi, On Fri, Dec 3, 2010 at 20:35, Alan Franzoni wrote: > - was your zope.interface installed through portage or via easy_install/pip? Portage. This is what I get from IPython though: In [1]: import zope In [2]: zope Out[2]: In [3]: import zope.interface In [4]: zope.interface Out[4]: aatiis at aiur ~ $ ls /usr/lib/python2.7/site-packages/zope* /usr/lib/python2.7/site-packages/zope.interface-3.6.1-py2.7-nspkg.pth /usr/lib/python2.7/site-packages/zope: interface /usr/lib/python2.7/site-packages/zope.interface-3.6.1-py2.7.egg-info: PKG-INFO SOURCES.txt dependency_links.txt namespace_packages.txt not-zip-safe requires.txt top_level.txt > - can you see zope.interface 3.6.1 being downloaded during the > buildout launch phase? Nope, I see something like "Egg from site-packages: zope.interface" I guess Gentoo installs a valid egg using portage, and buildout can recognize it. Attila From skip at pobox.com Fri Dec 3 22:05:44 2010 From: skip at pobox.com (skip at pobox.com) Date: Fri, 3 Dec 2010 15:05:44 -0600 Subject: [Distutils] easy_install from source? Message-ID: <19705.23464.429019.30114@montanaro.dyndns.org> I installed the 0.39 version of python-sybase using easy_install -m. That worked fine, but that version is extremely old. Is there a way to use easy_install with the a checkout from the project's version control system? I see this in the help output: --local-snapshots-ok (-l) allow building eggs from local checkouts but how do I specify the location of the local checkout? Thanks, -- Skip Montanaro - skip at pobox.com - http://www.smontanaro.net/ From pje at telecommunity.com Sat Dec 4 02:58:59 2010 From: pje at telecommunity.com (P.J. Eby) Date: Fri, 03 Dec 2010 20:58:59 -0500 Subject: [Distutils] easy_install from source? In-Reply-To: <19705.23464.429019.30114@montanaro.dyndns.org> References: <19705.23464.429019.30114@montanaro.dyndns.org> Message-ID: <20101204015924.A97C93A40D7@sparrow.telecommunity.com> At 03:05 PM 12/3/2010 -0600, skip at pobox.com wrote: >I installed the 0.39 version of python-sybase using easy_install -m. That >worked fine, but that version is extremely old. Is there a way to use >easy_install with the a checkout from the project's version control system? >I see this in the help output: > > --local-snapshots-ok (-l) allow building eggs from local checkouts > >but how do I specify the location of the local checkout? Just "easy_install /path/to/checkout", assuming there's a /path/to/checkout/setup.py. (The -l option only enables the use of packages that were locally installed using 'setup.py develop'.) >Thanks, > >-- >Skip Montanaro - skip at pobox.com - http://www.smontanaro.net/ >_______________________________________________ >Distutils-SIG maillist - Distutils-SIG at python.org >http://mail.python.org/mailman/listinfo/distutils-sig From mailing at franzoni.eu Sun Dec 5 23:14:05 2010 From: mailing at franzoni.eu (Alan Franzoni) Date: Sun, 5 Dec 2010 23:14:05 +0100 Subject: [Distutils] zc.buildout fails to use system-installed dep? In-Reply-To: References: Message-ID: I've verified this happens all the times on Ubuntu 10.10, both 32 and 64 bit. I don't know if it's Ubuntu's or zc.buildout's fault; I've opened a ticket. -- Alan Franzoni -- contact me at public@[mysurname].eu From mailing at franzoni.eu Mon Dec 6 09:13:19 2010 From: mailing at franzoni.eu (Alan Franzoni) Date: Mon, 6 Dec 2010 09:13:19 +0100 Subject: [Distutils] zc.buildout fails to use system-installed dep? In-Reply-To: References: Message-ID: For the records, or if anybody hits this problem before it's fixed; the only way to cope with this that I found is to add include-site-packages = false in the configuration. This forces the redownload of the egg and everything works fine. -- Alan Franzoni -- contact me at public@[mysurname].eu From devyan.parmar at gmail.com Tue Dec 7 11:34:47 2010 From: devyan.parmar at gmail.com (devyan parmar) Date: Tue, 7 Dec 2010 16:04:47 +0530 Subject: [Distutils] Unable to install packages after specifying package version In-Reply-To: <20101203164450.EA4223A4092@sparrow.telecommunity.com> References: <20101203164450.EA4223A4092@sparrow.telecommunity.com> Message-ID: On Fri, Dec 3, 2010 at 10:14 PM, P.J. Eby wrote: > At 04:25 PM 12/3/2010 +0530, devyan parmar wrote: > >> Hello, >> >> i am trying to install package from my packageserver using PIP. >> >> for example: >> m trying >> /bin/pip install --extra-index-url= >> http://mypkgserver/dev package_name==2.4.12b >> >> >> "2.4.12b" is the version of my package but in package server it is going >> as "2.4.12b_1" extra part in version is nothing but a build number which >> getting generated from hudson JOB. >> is setuptool avoids extra build number which goes with my pcakage version >> ?? >> > > No. That's why it's not being found. Ask for "package_name==2.4.12b-1" > instead. > Is there any way to avoid the extra hudson build number which is coming with my package version ?? coz i need that build number to check which package build test got failed...and issue is in package/setup.py i am specifying only versions of dependent package. and pip install is looking for only those particular version package in package server, but finds hudson build number attached with package version which is mismatching with setup.py install requires . can you help me how version is getting identified from the package by setuptools.?? Thanks Regards devyan parmar -------------- next part -------------- An HTML attachment was scrubbed... URL: From sjmachin at lexicon.net Tue Dec 7 22:14:11 2010 From: sjmachin at lexicon.net (John Machin) Date: Wed, 8 Dec 2010 08:14:11 +1100 Subject: [Distutils] msi installer for 64-bit Windows? Message-ID: <29846bdc4722ccd9b1f9b73af5effc65.squirrel@webmail.lexicon.net> Hello, I'm trying to figure out how to produce a Windows installer for pure-Python packages such that it will "just run" on a 64-bit Windows platform. I'm running a 32-bit Windows (Windows 7 Pro), and the msi produced by python setup.py bdist_msi works just fine on Win 7 Pro. I don't have a win64 platform to test on. The distutils docs for Python 2.7.1 say: ""In most cases, the bdist_msi installer is a better choice than the bdist_wininst installer, because it provides better support for Win64 platforms, allows administrators to perform non-interactive installations, and allows installation through group policies.""" but give no more detail. Running python setup.py bdist_msi --help produces only one item of interest: Options for 'bdist_msi' command: --plat-name (-p) platform name to embed in generated filenames (default: win32) Does the win32 version work on win64? Is the purpose of changing the name solely in order that win64 users are not deterred from using it because the name includes "win32"? Cheers, John From martin at v.loewis.de Tue Dec 7 22:48:44 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 07 Dec 2010 22:48:44 +0100 Subject: [Distutils] msi installer for 64-bit Windows? In-Reply-To: <29846bdc4722ccd9b1f9b73af5effc65.squirrel@webmail.lexicon.net> References: <29846bdc4722ccd9b1f9b73af5effc65.squirrel@webmail.lexicon.net> Message-ID: <4CFEABBC.9000802@v.loewis.de> > I'm trying to figure out how to produce a Windows installer for > pure-Python packages such that it will "just run" on a 64-bit Windows > platform. You'll need a 64-bit Python installation to build the 64-bit installer. > The distutils docs for Python 2.7.1 say: ""In most cases, the bdist_msi > installer is a better choice than the bdist_wininst installer, because it > provides better support for Win64 platforms, allows administrators to > perform non-interactive installations, and allows installation through > group policies.""" but give no more detail. The Win64 part may be outdated; bdist_wininst can now create Win64 installers (for AMD64) as well. > Does the win32 version work on win64? Yes, but it will require a 32-bit Python to be installed on the target system. > Is the purpose of changing the name > solely in order that win64 users are not deterred from using it because > the name includes "win32"? Yes, it only affects the resulting filename. Regards, Martin From pje at telecommunity.com Wed Dec 8 05:02:06 2010 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 07 Dec 2010 23:02:06 -0500 Subject: [Distutils] Unable to install packages after specifying package version In-Reply-To: References: <20101203164450.EA4223A4092@sparrow.telecommunity.com> Message-ID: <20101208040221.0E2DF3A408B@sparrow.telecommunity.com> At 04:04 PM 12/7/2010 +0530, devyan parmar wrote: >On Fri, Dec 3, 2010 at 10:14 PM, P.J. Eby ><pje at telecommunity.com> wrote: >At 04:25 PM 12/3/2010 +0530, devyan parmar wrote: >Hello, > >i am trying to install package from my packageserver using PIP. > >for example: >m trying >/bin/pip install >--extra-index-url=<http://mypkgserver/dev>http://mypkgserver/dev >package_name==2.4.12b > > >"2.4.12b" is the version of my package but in package server it is >going as "2.4.12b_1" extra part in version is nothing but a build >number which getting generated from hudson JOB. >is setuptool avoids extra build number which goes with my pcakage version ?? > > > No. That's why it's not being found. Ask for > "package_name==2.4.12b-1" instead. > > >Is there any way to avoid the extra hudson build number which is >coming with my package version ?? Er, don't put it in there? > coz i need that build number to check which package build test got > failed...and issue is in package/setup.py i am specifying only > versions of dependent package. Don't use '==', then, use '>='. >and pip install is looking for only those particular version package >in package server, but finds hudson build number attached with >package version which is mismatching with setup.py install requires . > >can you help me how version is getting identified from the package >by setuptools.?? > >Thanks Regards >devyan parmar From martin.raspaud at smhi.se Wed Dec 8 09:52:15 2010 From: martin.raspaud at smhi.se (Martin Raspaud) Date: Wed, 08 Dec 2010 09:52:15 +0100 Subject: [Distutils] setuptools: zip_safe as command line option ? Message-ID: <4CFF473F.409@smhi.se> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, Is it possible to tell setuptools to not zip the egg when installing ? I can't seem to find any option regarding this in the command line help... Thanks, Martin -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Remi - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJM/0c/AAoJEBdvyODiyJI4EXAH/2EVqqAFIBqmiZLMPyQ6Ibd1 CBLZlT12lm9oQ3+HvGIpow+UlWJCC+IRq/MsbUrrlLk1lSQ1lHt+gnjtiinixi5s XfH81ykaEhD0biWKVPHamU6LKjzkUl9UVIji3kUcznkl5clhXXuim7sCYfHYUVrH PyhJgWygK4r7c3bmrpd8VRDPmpRDLb2kFokhNQ+oe3JiX6UGWzVSLG2ohzy7CeDj AIWdadYc6i5QIHXpra5i8TkX2XLRvtYV5Yb5P3znArjc61Gjd7RRQUm83nug9/xV ZMsvPE8KtQsbHXKQHQ9npan+YF2p4LJW/4t/7SZw3pY3UHroTFa6Tyb6tXoSjsE= =GxiC -----END PGP SIGNATURE----- -------------- next part -------------- A non-text attachment was scrubbed... Name: martin_raspaud.vcf Type: text/x-vcard Size: 259 bytes Desc: not available URL: From reinout at vanrees.org Wed Dec 8 12:21:55 2010 From: reinout at vanrees.org (Reinout van Rees) Date: Wed, 08 Dec 2010 12:21:55 +0100 Subject: [Distutils] zc.buildout fails to use system-installed dep? In-Reply-To: References: Message-ID: On 12/05/2010 11:14 PM, Alan Franzoni wrote: > I've verified this happens all the times on Ubuntu 10.10, both 32 and > 64 bit. I don't know if it's Ubuntu's or zc.buildout's fault; I've > opened a ticket. I have the same problem on ubuntu. I never managed to find out what the real reason is, but my guess is that it's ubuntu's way of installing python packages for multiple python versions. You have a /usr/lib/python2.6/dist-packages/ directory, but also a /usr/lib/pyshared/python2.6/, for instance. Something somewhere is doing some funny sys.path mangling, probably ubuntu. The end result is that I still haven't gotten buildout 1.5.x to work on ubuntu with the system packages that I need. I'm still at 1.4.4 with the custom 1.4-x-only bootstrap, in combination with osc.recipe.sysegg I hope to have some debugging-time in January, as it is pretty irritating not to be able to use buildout 1.5.x ;-) Reinout -- Reinout van Rees - reinout at vanrees.org - http://reinout.vanrees.org Collega's gezocht! Django/python vacature in Utrecht: http://tinyurl.com/35v34f9 From mailing at franzoni.eu Wed Dec 8 13:38:19 2010 From: mailing at franzoni.eu (Alan Franzoni) Date: Wed, 8 Dec 2010 13:38:19 +0100 Subject: [Distutils] zc.buildout fails to use system-installed dep? In-Reply-To: References: Message-ID: On Wed, Dec 8, 2010 at 12:21 PM, Reinout van Rees wrote: > I have the same problem on ubuntu. ?I never managed to find out what the > real reason is, but my guess is that it's ubuntu's way of installing python > packages for multiple python versions. I have filed a ticket on zc.buildout bugtracker, you could use the "this affects me" feature. https://bugs.launchpad.net/zc.buildout/+bug/685638 >?You have a > /usr/lib/python2.6/dist-packages/ directory, but also a > /usr/lib/pyshared/python2.6/, for instance. ?Something somewhere is doing > some funny sys.path mangling, probably ubuntu. Yes, I suppose it's a kind of interference. I don't really know who's causing the problems, I think we'd need some feedback from both zc.buildout and ubuntu developers, but I can see Ubuntu still packages 1.4.3 even in Natty - there might be a reason for that. -- Alan Franzoni -- contact me at public@[mysurname].eu From marius at pov.lt Wed Dec 8 17:34:04 2010 From: marius at pov.lt (Marius Gedminas) Date: Wed, 8 Dec 2010 18:34:04 +0200 Subject: [Distutils] setuptools: zip_safe as command line option ? In-Reply-To: <4CFF473F.409@smhi.se> References: <4CFF473F.409@smhi.se> Message-ID: <20101208163404.GD29450@fridge.pov.lt> On Wed, Dec 08, 2010 at 09:52:15AM +0100, Martin Raspaud wrote: > Is it possible to tell setuptools to not zip the egg when installing ? I can't > seem to find any option regarding this in the command line help... Setuptools has no command-line. If you mean easy_install, it's -Z (--always-unzip). If you use pip, I believe it does the sane thing and never zips. I believe there's a config file to tell setuptools to never zip eggs; I don't remember the specifics. Have you tried google? Marius Gedminas -- Look! Before our very eyes, the future is becoming the past. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From benji at benjiyork.com Wed Dec 8 17:40:26 2010 From: benji at benjiyork.com (Benji York) Date: Wed, 8 Dec 2010 11:40:26 -0500 Subject: [Distutils] setuptools: zip_safe as command line option ? In-Reply-To: <20101208163404.GD29450@fridge.pov.lt> References: <4CFF473F.409@smhi.se> <20101208163404.GD29450@fridge.pov.lt> Message-ID: On Wed, Dec 8, 2010 at 11:34 AM, Marius Gedminas wrote: > I believe there's a config file to tell setuptools to never zip eggs; I > don't remember the specifics. ?Have you tried google? In distutils.cfg: [easy_install] zip_ok = 0 -- Benji York From marius at pov.lt Wed Dec 8 17:32:03 2010 From: marius at pov.lt (Marius Gedminas) Date: Wed, 8 Dec 2010 18:32:03 +0200 Subject: [Distutils] zc.buildout fails to use system-installed dep? In-Reply-To: References: Message-ID: <20101208163203.GC29450@fridge.pov.lt> On Wed, Dec 08, 2010 at 12:21:55PM +0100, Reinout van Rees wrote: > On 12/05/2010 11:14 PM, Alan Franzoni wrote: > >I've verified this happens all the times on Ubuntu 10.10, both 32 and > >64 bit. I don't know if it's Ubuntu's or zc.buildout's fault; I've > >opened a ticket. > > I have the same problem on ubuntu. I never managed to find out what > the real reason is, but my guess is that it's ubuntu's way of > installing python packages for multiple python versions. You have a > /usr/lib/python2.6/dist-packages/ directory, which is on sys.path > but also a > /usr/lib/pyshared/python2.6/, which isn't. But then there's /usr/lib/pymodules/python2.6, which is on sys.path, and I've no idea what it's about. dist-packages/python-support.pth adds this one. > for instance. Something somewhere is > doing some funny sys.path mangling, probably ubuntu. Debian and Ubuntu use dist-packages instead of site-packages to avoid conflicts with manual installations (I'm not sure if they mean manual as in sudo easy_install foo, or if they mean manual as in self-compiled Pythons. Probably the latter.). If zc.buildout assumes it can find system packages in site-packages instead of dist-packages, then it would fail. I've no idea if that's actually the case. /usr/share/doc/python/python-policy.txt.gz has more information about Debian and Ubuntu specifics. > The end result is that I still haven't gotten buildout 1.5.x to work > on ubuntu with the system packages that I need. I'm still at 1.4.4 > with the custom 1.4-x-only bootstrap, in combination with > osc.recipe.sysegg IIRC you can use the new bootstrap and ask it to get a specific version of zc.buildout. I never needed recipes to use system Python packages in 1.4 buildouts, but that was before Python 2.6 and dist-packages -- Debian's Python 2.5 still uses site-packages. > I hope to have some debugging-time in January, as it is pretty > irritating not to be able to use buildout 1.5.x ;-) Marius Gedminas -- "... one of the main causes of the fall of the Roman Empire was that, lacking zero, they had no way to indicate successful termination of their C programs." -- Robert Firth -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From barry at python.org Wed Dec 8 20:10:02 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 8 Dec 2010 14:10:02 -0500 Subject: [Distutils] zc.buildout fails to use system-installed dep? In-Reply-To: <20101208163203.GC29450@fridge.pov.lt> References: <20101208163203.GC29450@fridge.pov.lt> Message-ID: <20101208141002.67d43e42@mission> To start with, the Debian/Ubuntu arrangement is made even more confusing for there being three competing "build systems" for Python. There are the traditional python-central and python-support systems, which most packages these days use. python-support is probably more popular than python-central. Both however are being obsoleted for the new goodness, dh_python2. The biggest benefit to dh_python2 is that the symlinks are created at build-time and appear in the package, instead of being created at install-time and managed by what can be flaky post-scripts. The conversion process will begin happening in earnest once Debian squeeze is released. Why do we have symlinks in the first place? It's because Debian and Ubuntu support multiple active versions of Python at the same time. Once Python 2 is killed off and we're all using >= Python 3.2, we can get rid of even this cruft. That's the major reason why I worked on PEPs 3147 and 3149. On Dec 08, 2010, at 06:32 PM, Marius Gedminas wrote: >On Wed, Dec 08, 2010 at 12:21:55PM +0100, Reinout van Rees wrote: >> On 12/05/2010 11:14 PM, Alan Franzoni wrote: >> >I've verified this happens all the times on Ubuntu 10.10, both 32 and >> >64 bit. I don't know if it's Ubuntu's or zc.buildout's fault; I've >> >opened a ticket. >> >> I have the same problem on ubuntu. I never managed to find out what >> the real reason is, but my guess is that it's ubuntu's way of >> installing python packages for multiple python versions. You have a >> /usr/lib/python2.6/dist-packages/ directory, > >which is on sys.path > >> but also a >> /usr/lib/pyshared/python2.6/, > >which isn't. > >But then there's /usr/lib/pymodules/python2.6, which is on sys.path, >and I've no idea what it's about. dist-packages/python-support.pth adds >this one. All roads point to pyshared, which is where the files are *actually* installed. pymodules and dist-packages point into pyshared and are there to handle the multiple-installed-python arrangement. I think the difference comes about depending on whether python-central or python-support is being used. >> for instance. Something somewhere is >> doing some funny sys.path mangling, probably ubuntu. > >Debian and Ubuntu use dist-packages instead of site-packages to avoid >conflicts with manual installations (I'm not sure if they mean manual as >in sudo easy_install foo, or if they mean manual as in self-compiled >Pythons. Probably the latter.). Yes. A from-source build and install of Python would use /usr/local/lib/pythonX.Y/site-packages, but Debian's interpretation of the FHS puts that directory under the domain of the system administrator. Which means, it was possible to install third party packages into a from-source build of Python (with default configure options and 'make altinstall') and break your system Python, or vice versa. dist-packages is a compromise to the decade's tradition of Python source default locations. (Arguably, a from-source build of Python with the default options should install into /opt.) >If zc.buildout assumes it can find system packages in site-packages >instead of dist-packages, then it would fail. I've no idea if that's >actually the case. I don't remember specifically for zc.buildout, but distribute for example has been modified to use dist-packages. >/usr/share/doc/python/python-policy.txt.gz has more information about >Debian and Ubuntu specifics. > >> The end result is that I still haven't gotten buildout 1.5.x to work >> on ubuntu with the system packages that I need. I'm still at 1.4.4 >> with the custom 1.4-x-only bootstrap, in combination with >> osc.recipe.sysegg > >IIRC you can use the new bootstrap and ask it to get a specific version >of zc.buildout. I never needed recipes to use system Python packages in >1.4 buildouts, but that was before Python 2.6 and dist-packages -- >Debian's Python 2.5 still uses site-packages. I'm using zc.buildout 1.5.2 on Ubuntu 10.10 for Mailman 3. There is a known bug related to namespace packages, which force you to use include-site-packages=false in your buildout.cfg. Here's the bug report: https://bugs.launchpad.net/zc.buildout/+bug/659231 Gary is assigned to the bug but there is as yet no resolution. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From martin.raspaud at smhi.se Wed Dec 8 20:54:21 2010 From: martin.raspaud at smhi.se (Martin Raspaud) Date: Wed, 08 Dec 2010 20:54:21 +0100 Subject: [Distutils] setuptools: zip_safe as command line option ? In-Reply-To: <20101208163404.GD29450@fridge.pov.lt> References: <4CFF473F.409@smhi.se> <20101208163404.GD29450@fridge.pov.lt> Message-ID: <4CFFE26D.60406@smhi.se> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 2010-12-08 17:34, Marius Gedminas skrev: > On Wed, Dec 08, 2010 at 09:52:15AM +0100, Martin Raspaud wrote: >> Is it possible to tell setuptools to not zip the egg when installing ? I can't >> seem to find any option regarding this in the command line help... > > Setuptools has no command-line. > > If you mean easy_install, it's -Z (--always-unzip). Ok, I meant on the "python setup.py install" line. But seeing your answer and the next one, I guess setuptools is not the best tool to use anymore ? Thanks, Martin -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Remi - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJM/+JsAAoJEBdvyODiyJI44lgIAJ/coGpoQrXQxRSI2kjnt4Pf XUplHGUn8KcSZf2YJRl6FWgD6T8C58Eja82p84PL+xJEZMgk5emNTw0MeTtvkZce wa40694Y9sPiNAE8xjAfKGLpdn1jqdOluIGfSayjkuqrLRODY10g1TuKT3da78Dm QA+C0VUyLFJBG4P2JFWG5PvxQN/Sb5fy86f7xz5Src/jsEAUO8+HhKsYe2FKpRU9 qJ66TZ0D8xskUQXJwpT1yXDZcAZdLmUAg2b+olarKz2N68hVXfAXWXhxI6/VuAKb /TJnoEO4MTWijTcDVHKaN4AlCgje40ZtmzB5N0xmhfqvx0vcqlZFfZ5QykJWMPA= =PsGG -----END PGP SIGNATURE----- -------------- next part -------------- A non-text attachment was scrubbed... Name: martin_raspaud.vcf Type: text/x-vcard Size: 259 bytes Desc: not available URL: From pje at telecommunity.com Wed Dec 8 21:43:55 2010 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 08 Dec 2010 15:43:55 -0500 Subject: [Distutils] setuptools: zip_safe as command line option ? In-Reply-To: <4CFFE26D.60406@smhi.se> References: <4CFF473F.409@smhi.se> <20101208163404.GD29450@fridge.pov.lt> <4CFFE26D.60406@smhi.se> Message-ID: <20101208204406.027DD3A40D7@sparrow.telecommunity.com> At 08:54 PM 12/8/2010 +0100, Martin Raspaud wrote: >-----BEGIN PGP SIGNED MESSAGE----- >Hash: SHA1 > >2010-12-08 17:34, Marius Gedminas skrev: > > On Wed, Dec 08, 2010 at 09:52:15AM +0100, Martin Raspaud wrote: > >> Is it possible to tell setuptools to not zip the egg when > installing ? I can't > >> seem to find any option regarding this in the command line help... > > > > Setuptools has no command-line. > > > > If you mean easy_install, it's -Z (--always-unzip). > >Ok, I meant on the >"python setup.py install" >line. If you're using "python setup.py install" and want it to install in the old-style (distutils) way, you can use "setup.py install --single-version-externally-managed". This will install it in a way similar to pip, but without the uninstall capability. (You should not use this unless that's *really* what you want, and it probably isn't!) However, if you want to do the equivalent of "easy_install -Z" from a setup.py, use: python setup.py easy_install -Z . Which is roughly like "python setup.py install", but with .egg directory installation in place of a zipped .egg file. Also, as Benji pointed out, you can put this: [easy_install] zip_ok = 0 in your project's setup.cfg or your user or global distutils.cfg file to force this to happen automatically, even if you use "python setup.py install". For that matter, you can put 'zip_safe=False' in your setup() call in your setup.py to make that one specific project always install unzipped, even when somebody else installs it from an .egg file. There are many ways to do it -- so use whichever is most convenient for what you are trying to accomplish. (All of the above answers are valid for setuptools-based projects.) From mailing at franzoni.eu Thu Dec 9 00:35:17 2010 From: mailing at franzoni.eu (Alan Franzoni) Date: Thu, 9 Dec 2010 00:35:17 +0100 Subject: [Distutils] zc.buildout fails to use system-installed dep? In-Reply-To: <20101208141002.67d43e42@mission> References: <20101208163203.GC29450@fridge.pov.lt> <20101208141002.67d43e42@mission> Message-ID: On Wed, Dec 8, 2010 at 8:10 PM, Barry Warsaw wrote: > Why do we have symlinks in the first place? ?It's because Debian and Ubuntu > support multiple active versions of Python at the same time. ?Once Python 2 is > killed off and we're all using >= Python 3.2, we can get rid of even this > cruft. ?That's the major reason why I worked on PEPs 3147 and 3149. It might be a long time indeed. Python 2.7 will probably keep up for several years. > I'm using zc.buildout 1.5.2 on Ubuntu 10.10 for Mailman 3. ?There is a known > bug related to namespace packages, which force you to use > include-site-packages=false in your buildout.cfg. ?Here's the bug report: > > https://bugs.launchpad.net/zc.buildout/+bug/659231 > > Gary is assigned to the bug but there is as yet no resolution. It's the very same error and the same workaround is suggested, but I don't see enough proof that the reason behind such error lies within namespace_packages. I'll make some tests in the next days and I'll check whether it's the case. -- Alan Franzoni -- contact me at public@[mysurname].eu From barry at python.org Thu Dec 9 00:39:28 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 8 Dec 2010 18:39:28 -0500 Subject: [Distutils] zc.buildout fails to use system-installed dep? In-Reply-To: References: <20101208163203.GC29450@fridge.pov.lt> <20101208141002.67d43e42@mission> Message-ID: <20101208183928.7987f68f@mission> On Dec 09, 2010, at 12:35 AM, Alan Franzoni wrote: >On Wed, Dec 8, 2010 at 8:10 PM, Barry Warsaw wrote: >> Why do we have symlinks in the first place? ?It's because Debian and Ubuntu >> support multiple active versions of Python at the same time. ?Once Python 2 is >> killed off and we're all using >= Python 3.2, we can get rid of even this >> cruft. ?That's the major reason why I worked on PEPs 3147 and 3149. > >It might be a long time indeed. Python 2.7 will probably keep up for >several years. Indeed. I'm glad I'm not the 2.7 release manager. :) >> I'm using zc.buildout 1.5.2 on Ubuntu 10.10 for Mailman 3. ?There is a known >> bug related to namespace packages, which force you to use >> include-site-packages=false in your buildout.cfg. ?Here's the bug report: >> >> https://bugs.launchpad.net/zc.buildout/+bug/659231 >> >> Gary is assigned to the bug but there is as yet no resolution. > >It's the very same error and the same workaround is suggested, but I >don't see enough proof that the reason behind such error lies within >namespace_packages. I'll make some tests in the next days and I'll >check whether it's the case. Please do. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From mailing at franzoni.eu Thu Dec 9 23:25:30 2010 From: mailing at franzoni.eu (Alan Franzoni) Date: Thu, 9 Dec 2010 23:25:30 +0100 Subject: [Distutils] zc.buildout fails to use system-installed dep? In-Reply-To: <20101208183928.7987f68f@mission> References: <20101208163203.GC29450@fridge.pov.lt> <20101208141002.67d43e42@mission> <20101208183928.7987f68f@mission> Message-ID: On Thu, Dec 9, 2010 at 12:39 AM, Barry Warsaw wrote: [cut] After taking a look, it seems that namespace packages are involved indeed, even though I'm not sure how yet, and there seems to be another related bug where you commented as well: https://bugs.launchpad.net/ubuntu/+source/python2.6/+bug/621348 I admit I'm not extremely expert at import matters, and I'm getting mad at understanding how many levels of indirection are involved when resolving a python import! I tinkered around a bit trying to check whether the behaviour could be reproduced for other modules. It seems it cannot! I've tried installing zc.lockfile from Ubuntu packages, then i added the "zc.dict" egg to my test recipe. I expected to be unable to import zc.lockfile, but that wasn't the case. nspkg.pth files exists both for zc.lockfile and zope.interface, but while the dist-packages/zope dir has got a __init__.py file with declare_namespace(__name__) in it, the dist-packages/zc dir hasn't. I removed the __init__.py file in dist-packages/zope and now I'm able to run my script without a flaw. I must admit I don't fully understand nspkg.pth files (where can I find more info on them? Who's creating them, setuptools/distribute?), but their usage and the declare_namespace method seems mutually exclusive by the way; was that confusing zc.buildout, I suppose? Should I open a ticket on ubuntu zope.interface package? -- Alan Franzoni -- contact me at public@[mysurname].eu From mailing at franzoni.eu Fri Dec 10 00:17:02 2010 From: mailing at franzoni.eu (Alan Franzoni) Date: Fri, 10 Dec 2010 00:17:02 +0100 Subject: [Distutils] zc.buildout fails to use system-installed dep? In-Reply-To: References: <20101208163203.GC29450@fridge.pov.lt> <20101208141002.67d43e42@mission> <20101208183928.7987f68f@mission> Message-ID: On Thu, Dec 9, 2010 at 11:25 PM, Alan Franzoni wrote: [cut] It appears that the zope.interface mantainer from Debian manually copies the unnecessary - and confusing - __init__.py files: http://svn.debian.org/viewsvn/pkg-zope/zope.interface/trunk/debian/rules?revision=2074&view=markup It seems the only package in the python-zope.* hierarchy doing that; I don't know why. I have opened a ticket in Ubuntu: https://bugs.launchpad.net/ubuntu/+source/zope.interface/+bug/688335 I'm unable to do the same in Debian although it's probably there, because they seem to have no web interface and I don't have a Debian machine available right now. If anybody could check it and do so it would be great. -- Alan Franzoni -- contact me at public@[mysurname].eu From brian at vanguardistas.net Fri Dec 10 09:36:32 2010 From: brian at vanguardistas.net (Brian Sutherland) Date: Fri, 10 Dec 2010 09:36:32 +0100 Subject: [Distutils] zc.buildout fails to use system-installed dep? In-Reply-To: References: <20101208163203.GC29450@fridge.pov.lt> <20101208141002.67d43e42@mission> <20101208183928.7987f68f@mission> Message-ID: <20101210083632.GA80263@Boo.local> On Fri, Dec 10, 2010 at 12:17:02AM +0100, Alan Franzoni wrote: > On Thu, Dec 9, 2010 at 11:25 PM, Alan Franzoni wrote: > [cut] > > It appears that the zope.interface mantainer from Debian manually > copies the unnecessary - and confusing - __init__.py files: > > http://svn.debian.org/viewsvn/pkg-zope/zope.interface/trunk/debian/rules?revision=2074&view=markup > > It seems the only package in the python-zope.* hierarchy doing that; I > don't know why. > > I have opened a ticket in Ubuntu: > > https://bugs.launchpad.net/ubuntu/+source/zope.interface/+bug/688335 > > I'm unable to do the same in Debian although it's probably there, > because they seem to have no web interface and I don't have a Debian > machine available right now. If anybody could check it and do so it > would be great. Please don't, this has been reported various times already. Removing the zope/__init__.py causes breakage in other places, as I detail in my response to your bug. I would guess that the real bug is at a deeper level. -- Brian Sutherland From mailing at franzoni.eu Fri Dec 10 10:34:57 2010 From: mailing at franzoni.eu (Alan Franzoni) Date: Fri, 10 Dec 2010 10:34:57 +0100 Subject: [Distutils] zc.buildout fails to use system-installed dep? In-Reply-To: <20101210083632.GA80263@Boo.local> References: <20101208163203.GC29450@fridge.pov.lt> <20101208141002.67d43e42@mission> <20101208183928.7987f68f@mission> <20101210083632.GA80263@Boo.local> Message-ID: On Fri, Dec 10, 2010 at 9:36 AM, Brian Sutherland wrote: > > Please don't, this has been reported various times already. > > Removing the zope/__init__.py causes breakage in other places, as I > detail in my response to your bug. Sigh, I guess we'd need a Ubuntu/Debian python packaging FAQ? If it has been reported many times but keeps popping up, it's probably unpredictable enough. It took me 3 hours just on yesterday evening to find a workaround to that, and I'd have gladly spared that :-( -- Alan Franzoni -- contact me at public@[mysurname].eu From brian at vanguardistas.net Fri Dec 10 11:17:14 2010 From: brian at vanguardistas.net (Brian Sutherland) Date: Fri, 10 Dec 2010 11:17:14 +0100 Subject: [Distutils] zc.buildout fails to use system-installed dep? In-Reply-To: References: <20101208163203.GC29450@fridge.pov.lt> <20101208141002.67d43e42@mission> <20101208183928.7987f68f@mission> <20101210083632.GA80263@Boo.local> Message-ID: <20101210101714.GB80263@Boo.local> On Fri, Dec 10, 2010 at 10:34:57AM +0100, Alan Franzoni wrote: > On Fri, Dec 10, 2010 at 9:36 AM, Brian Sutherland > wrote: > > > > Please don't, this has been reported various times already. > > > > Removing the zope/__init__.py causes breakage in other places, as I > > detail in my response to your bug. > > Sigh, I guess we'd need a Ubuntu/Debian python packaging FAQ? > > If it has been reported many times but keeps popping up, it's probably > unpredictable enough. It took me 3 hours just on yesterday evening to > find a workaround to that, and I'd have gladly spared that :-( Sorry, if I'd realized sooner that your issues with buildout were related, I would have spoken up sooner. I've also lost many hours of my life due to this one :( I'll mark your launchpad bug for now as Wontfix and hope that the deeper bug gets resolved sometime. Unfortunately even figuring out what that is (or what causes it) is beyond my ken. -- Brian Sutherland From barry at python.org Fri Dec 10 13:17:16 2010 From: barry at python.org (Barry Warsaw) Date: Fri, 10 Dec 2010 07:17:16 -0500 Subject: [Distutils] zc.buildout fails to use system-installed dep? In-Reply-To: <20101210083632.GA80263@Boo.local> References: <20101208163203.GC29450@fridge.pov.lt> <20101208141002.67d43e42@mission> <20101208183928.7987f68f@mission> <20101210083632.GA80263@Boo.local> Message-ID: <20101210071716.26c8930d@snowdog> On Dec 10, 2010, at 09:36 AM, Brian Sutherland wrote: >Please don't, this has been reported various times already. > >Removing the zope/__init__.py causes breakage in other places, as I >detail in my response to your bug. > >I would guess that the real bug is at a deeper level. Just a couple of things: The way zope.interface "owns" the zope namespace is not correct. Ideally, there would be a separate binary package that owns zope/__init__.py and on which all other zope subpackages depends. This could of course be built from the zope.interface source package. Second, we're going to make a big push after Squeeze is released to convert packaging to use dh_python2, the new goodness in Debian Python packaging. This should make things more consistent, and, while I have not yet tracked down specific details yet, it should make handling namespace packages much more robust. For example, with my flufl packages, there is no binary package owning flufl/__init__.py. That file does not show up in the binary package and yet it still gets created properly when any of the subpackages get installed. Sadly PEP 382 was not complete in time for Python 3.2. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From brian at vanguardistas.net Fri Dec 10 13:42:19 2010 From: brian at vanguardistas.net (Brian Sutherland) Date: Fri, 10 Dec 2010 13:42:19 +0100 Subject: [Distutils] zc.buildout fails to use system-installed dep? In-Reply-To: <20101210071716.26c8930d@snowdog> References: <20101208163203.GC29450@fridge.pov.lt> <20101208141002.67d43e42@mission> <20101208183928.7987f68f@mission> <20101210083632.GA80263@Boo.local> <20101210071716.26c8930d@snowdog> Message-ID: <20101210124219.GC80263@Boo.local> On Fri, Dec 10, 2010 at 07:17:16AM -0500, Barry Warsaw wrote: > On Dec 10, 2010, at 09:36 AM, Brian Sutherland wrote: > > >Please don't, this has been reported various times already. > > > >Removing the zope/__init__.py causes breakage in other places, as I > >detail in my response to your bug. > > > >I would guess that the real bug is at a deeper level. > > Just a couple of things: > > The way zope.interface "owns" the zope namespace is not correct. Ideally, > there would be a separate binary package that owns zope/__init__.py and on > which all other zope subpackages depends. This could of course be built from > the zope.interface source package. I originally had it that way, but was strongly advised to change it to the current method to be able to pass the Debian FTP Master gauntlet. The current method using dpkg-divert is not too bad, more packages than python-zope.interface could include that file as well. So you don't force installation of python-zope.interface. It also uses standard dpkg functionality, which is a robustness bonus. > Second, we're going to make a big push after Squeeze is released to convert > packaging to use dh_python2, the new goodness in Debian Python packaging. I've had a brief look at it already and will have a much deeper look once squeeze is released. AFAIKR the only feature it was missing to completely cover my usecase was good handling of setuptools extras. I'm hoping to be able to completely replace the custom tools we use in python-zope.* packages with it at some point. > This should make things more consistent, and, while I have not yet tracked > down specific details yet, it should make handling namespace packages much > more robust. For example, with my flufl packages, there is no binary package > owning flufl/__init__.py. That file does not show up in the binary package > and yet it still gets created properly when any of the subpackages get > installed. Any idea as to the mechanism? What about 2nd level namespace packages (horror: zope.app.foo)? > Sadly PEP 382 was not complete in time for Python 3.2. Yeah, there are a lot of packaging related PEPs coming out lately. It's really great to see attention being paid to these dusty corners:) -- Brian Sutherland From barry at python.org Fri Dec 10 14:19:32 2010 From: barry at python.org (Barry Warsaw) Date: Fri, 10 Dec 2010 08:19:32 -0500 Subject: [Distutils] zc.buildout fails to use system-installed dep? In-Reply-To: <20101210124219.GC80263@Boo.local> References: <20101208163203.GC29450@fridge.pov.lt> <20101208141002.67d43e42@mission> <20101208183928.7987f68f@mission> <20101210083632.GA80263@Boo.local> <20101210071716.26c8930d@snowdog> <20101210124219.GC80263@Boo.local> Message-ID: <20101210081932.5115ff2e@mission> On Dec 10, 2010, at 01:42 PM, Brian Sutherland wrote: >On Fri, Dec 10, 2010 at 07:17:16AM -0500, Barry Warsaw wrote: >> The way zope.interface "owns" the zope namespace is not correct. Ideally, >> there would be a separate binary package that owns zope/__init__.py and on >> which all other zope subpackages depends. This could of course be built >> from the zope.interface source package. > >I originally had it that way, but was strongly advised to change it to >the current method to be able to pass the Debian FTP Master gauntlet. Just goes to illustrate the myth of TOOWTDI. :) AFAICT, from my discussions with various folks on debian-python, it should now[*] be done with the separate binary package. [*] At least until dh_python2, if we can ensure it always DTRT. >The current method using dpkg-divert is not too bad, more packages than >python-zope.interface could include that file as well. So you don't >force installation of python-zope.interface. Doing the diversion means it's harder for someone to explicitly determine which package owns the file. Better (I think!) to have none of them own the file. >It also uses standard dpkg functionality, which is a robustness bonus. True. >> Second, we're going to make a big push after Squeeze is released to convert >> packaging to use dh_python2, the new goodness in Debian Python packaging. > >I've had a brief look at it already and will have a much deeper look >once squeeze is released. AFAIKR the only feature it was missing to >completely cover my usecase was good handling of setuptools extras. Can you be more specific? Before I got swamped with the Python 2.7 transition (it will be the default in Ubuntu 11.04), I began looking at dh_python2, adding unit tests, etc. I do plan to get back to it once we have the archive more happily on Python 2.7[*]. [*] https://bugs.launchpad.net/ubuntu/+bugs?field.tag=python27 >I'm hoping to be able to completely replace the custom tools we use in >python-zope.* packages with it at some point. +1. We hope to get rid of python-support and python-central. >> This should make things more consistent, and, while I have not yet tracked >> down specific details yet, it should make handling namespace packages much >> more robust. For example, with my flufl packages, there is no binary >> package owning flufl/__init__.py. That file does not show up in the binary >> package and yet it still gets created properly when any of the subpackages >> get installed. > >Any idea as to the mechanism? Hints, but nothing definite. I'm not sure which part of the tool chain is suppressing the neamespace package's __init__.py, and which part is laying it down when the package gets installed. It *is* nice that the packager doesn't have to worry about it though. I don't think you should have to do the tricks zope.interface is doing, or even do the extra binary package. IOW, it should Just Work. >What about 2nd level namespace packages (horror: zope.app.foo)? Haven't tried that yet. >> Sadly PEP 382 was not complete in time for Python 3.2. > >Yeah, there are a lot of packaging related PEPs coming out lately. It's >really great to see attention being paid to these dusty corners:) Indeed! I think we all owe Tarek and the other folks an unlimited supply of beers at the next Pycon. :) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From mailing at franzoni.eu Fri Dec 10 14:37:28 2010 From: mailing at franzoni.eu (Alan Franzoni) Date: Fri, 10 Dec 2010 14:37:28 +0100 Subject: [Distutils] zc.buildout fails to use system-installed dep? In-Reply-To: <20101210081932.5115ff2e@mission> References: <20101208163203.GC29450@fridge.pov.lt> <20101208141002.67d43e42@mission> <20101208183928.7987f68f@mission> <20101210083632.GA80263@Boo.local> <20101210071716.26c8930d@snowdog> <20101210124219.GC80263@Boo.local> <20101210081932.5115ff2e@mission> Message-ID: On Fri, Dec 10, 2010 at 2:19 PM, Barry Warsaw wrote: > On Dec 10, 2010, at 01:42 PM, Brian Sutherland wrote: [cut] A new, interesting twist! A zc.buildout developer has answered to my original ticket: https://bugs.launchpad.net/zc.buildout/+bug/685638 and suggested I took a look at another ticket he reported for Ubuntu: https://bugs.launchpad.net/ubuntu/+source/distribute/+bug/576434 Following the instruction of the latter I was able to use zope.interface without an issue, even though __init__.py it's inside it. By the way: things are getting way, way too complicated. I can imagine nobody has really a good grasp on what's going on "under the hood". I'm looking forward to help with packaging tools before it's too late :-) -- Alan Franzoni -- contact me at public@[mysurname].eu From brian at vanguardistas.net Fri Dec 10 15:06:53 2010 From: brian at vanguardistas.net (Brian Sutherland) Date: Fri, 10 Dec 2010 15:06:53 +0100 Subject: [Distutils] zc.buildout fails to use system-installed dep? In-Reply-To: <20101210081932.5115ff2e@mission> References: <20101208163203.GC29450@fridge.pov.lt> <20101208141002.67d43e42@mission> <20101208183928.7987f68f@mission> <20101210083632.GA80263@Boo.local> <20101210071716.26c8930d@snowdog> <20101210124219.GC80263@Boo.local> <20101210081932.5115ff2e@mission> Message-ID: <20101210140653.GA84976@Boo.local> On Fri, Dec 10, 2010 at 08:19:32AM -0500, Barry Warsaw wrote: > On Dec 10, 2010, at 01:42 PM, Brian Sutherland wrote: > > >On Fri, Dec 10, 2010 at 07:17:16AM -0500, Barry Warsaw wrote: > > >> The way zope.interface "owns" the zope namespace is not correct. Ideally, > >> there would be a separate binary package that owns zope/__init__.py and on > >> which all other zope subpackages depends. This could of course be built > >> from the zope.interface source package. > > > >I originally had it that way, but was strongly advised to change it to > >the current method to be able to pass the Debian FTP Master gauntlet. > > Just goes to illustrate the myth of TOOWTDI. :) AFAICT, from my discussions > with various folks on debian-python, it should now[*] be done with the > separate binary package. > > [*] At least until dh_python2, if we can ensure it always DTRT. I'll just till dh_python2 then before changing anything ;) hmm, perhaps I should join debina-python again... > >> Second, we're going to make a big push after Squeeze is released to convert > >> packaging to use dh_python2, the new goodness in Debian Python packaging. > > > >I've had a brief look at it already and will have a much deeper look > >once squeeze is released. AFAIKR the only feature it was missing to > >completely cover my usecase was good handling of setuptools extras. > > Can you be more specific? Before I got swamped with the Python 2.7 transition > (it will be the default in Ubuntu 11.04), I began looking at dh_python2, > adding unit tests, etc. I do plan to get back to it once we have the archive > more happily on Python 2.7[*]. > > [*] https://bugs.launchpad.net/ubuntu/+bugs?field.tag=python27 I always use zope.component as an example because it has the worst case use of setuptools extras which a lot of other packages depend on. In 3.9.1 this is: extras_require = dict( hook = ['zope.hookable'], persistentregistry = ['ZODB3'], zcml = ['zope.configuration', 'zope.i18nmessageid', ], test = ['ZODB3', 'zope.testing', 'zope.hookable', 'zope.location', 'zope.proxy', 'zope.security', ], docs = ['z3c.recipe.sphinxdoc'], ), The way we treat these extras in the python-zope.* packages is as if they were extra binary packages with names like python-${distribution_name}-${extra_name}. So zope.component is actually 6 binary packages: python-zope.component python-zope.component-hook python-zope.component-persistentregistry python-zope.component-zcml python-zope.component-test python-zope.component-docs But, we can't make 6 binary packages or the FTP Masters will kill us. So we do a few things with extras: * Have Ignore them and their dependencies. * Provide the extra package from the main package and depend on it's dependencies. * Have their dependencies as "suggests" on the main package * Break them out into a binary metapackage. python-zope.component does all of these for it's extras as appropriate in each case. I wrote a plugin to debhelper 7 so this can be specified in an easy way using environment variables in the rules file: http://svn.debian.org/viewsvn/pkg-zope/zope.component/trunk/debian/rules?revision=2053&view=markup The code to do the work is in the python-van.pydeb package. One of the things I had hoped to do after squeeze was port the van.pydep tests to dh_python2 and then bother someone else to fix them. python-zope.security is also another tough case. In that case, Ubuntu needs to carry a diff against Debian to cope with the "untrustedpython" extra. > >> This should make things more consistent, and, while I have not yet tracked > >> down specific details yet, it should make handling namespace packages much > >> more robust. For example, with my flufl packages, there is no binary > >> package owning flufl/__init__.py. That file does not show up in the binary > >> package and yet it still gets created properly when any of the subpackages > >> get installed. > > > >Any idea as to the mechanism? > > Hints, but nothing definite. I'm not sure which part of the tool chain is > suppressing the neamespace package's __init__.py, and which part is laying it > down when the package gets installed. It *is* nice that the packager doesn't > have to worry about it though. I don't think you should have to do the tricks > zope.interface is doing, or even do the extra binary package. IOW, it should > Just Work. +100 to just working :) > >What about 2nd level namespace packages (horror: zope.app.foo)? > > Haven't tried that yet. Well, cross that bridge when we come to it... > >> Sadly PEP 382 was not complete in time for Python 3.2. > > > >Yeah, there are a lot of packaging related PEPs coming out lately. It's > >really great to see attention being paid to these dusty corners:) > > Indeed! I think we all owe Tarek and the other folks an unlimited supply of > beers at the next Pycon. :) Yep :) -- Brian Sutherland From marius at pov.lt Fri Dec 10 18:56:37 2010 From: marius at pov.lt (Marius Gedminas) Date: Fri, 10 Dec 2010 19:56:37 +0200 Subject: [Distutils] zc.buildout fails to use system-installed dep? In-Reply-To: <20101210081932.5115ff2e@mission> References: <20101208163203.GC29450@fridge.pov.lt> <20101208141002.67d43e42@mission> <20101208183928.7987f68f@mission> <20101210083632.GA80263@Boo.local> <20101210071716.26c8930d@snowdog> <20101210124219.GC80263@Boo.local> <20101210081932.5115ff2e@mission> Message-ID: <20101210175637.GA6803@fridge.pov.lt> On Fri, Dec 10, 2010 at 08:19:32AM -0500, Barry Warsaw wrote: > >Yeah, there are a lot of packaging related PEPs coming out lately. It's > >really great to see attention being paid to these dusty corners:) > > Indeed! I think we all owe Tarek and the other folks an unlimited supply of > beers at the next Pycon. :) Oh yes. And PJE too, for kickstarting the whole thing when he came up with setuptools. Marius Gedminas -- Anybody who doesn't cut his speed at the sight of a police car is probably parked. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From copsclubuk at gmail.com Sun Dec 12 18:49:33 2010 From: copsclubuk at gmail.com (Cops Club) Date: Sun, 12 Dec 2010 17:49:33 +0000 Subject: [Distutils] Newbie - Buildout bootstrap error pkg_resources.DistributionNotFound: zc.buildout==1.5.2 Message-ID: I'm a newbie to Python, so please be patient with me.... 1) On Ubunto 10.10 I used sudo to install python-setuptools then used easy_install to install zc.buildout. 2) I then logged on as a different user (who is not able to sudo) to create my first buildout project. plonesvc at ubuntu:~$ mkdir exercise1 plonesvc at ubuntu:~$ cd exercise1/ plonesvc at ubuntu:~/exercise1$ buildout init Creating '/home/plonesvc/exercise1/buildout.cfg'. Creating directory '/home/plonesvc/exercise1/bin'. Creating directory '/home/plonesvc/exercise1/parts'. Creating directory '/home/plonesvc/exercise1/eggs'. Creating directory '/home/plonesvc/exercise1/develop-eggs'. Generated script '/home/plonesvc/exercise1/bin/buildout'. 3) I downloaded the bootstrap.py from svn plonesvc at ubuntu:~$ wget \ http://svn.zope.org/*checkout*/zc.buildout/trunk/bootstrap/bootstrap.py 4)And tried to run the bootstrap - which failed: plonesvc at ubuntu:~/exercise1$ python bootstrap.py Downloading http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg Traceback (most recent call last): File "bootstrap.py", line 256, in ws.require(requirement) File "/tmp/tmpCfYQ3c/setuptools-0.6c11-py2.6.egg/pkg_resources.py", line 666, in require File "/tmp/tmpCfYQ3c/setuptools-0.6c11-py2.6.egg/pkg_resources.py", line 565, in resolve pkg_resources.DistributionNotFound: zc.buildout==1.5.2 5) I've tried searching for the error via google without success. 6) I have subsequently installed virtual env and setup a new buildout project and run the bootstrap.py successfully. Which, is great but I want to know why I wasn't able to do this without virtual env? I was wondering if someone could explain why step 4 above failed. What was the reason for this? Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From regebro at gmail.com Mon Dec 13 12:41:39 2010 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 13 Dec 2010 12:41:39 +0100 Subject: [Distutils] Distribute name for the setup() options. Message-ID: A quick look through the Distribute docs shows that we call the settings we give to setup py various things. Both "options" and "arguments" and "parameters" (that last is my fault). We should probably decide on one thing, and go through the docs to make sure it's consistent. I prefer "option", although strictly speaking they *are* keyword arguments. Opinions? //Lennart From doko at ubuntu.com Mon Dec 13 20:06:08 2010 From: doko at ubuntu.com (Matthias Klose) Date: Mon, 13 Dec 2010 20:06:08 +0100 Subject: [Distutils] zc.buildout fails to use system-installed dep? In-Reply-To: <20101210124219.GC80263@Boo.local> References: <20101208163203.GC29450@fridge.pov.lt> <20101208141002.67d43e42@mission> <20101208183928.7987f68f@mission> <20101210083632.GA80263@Boo.local> <20101210071716.26c8930d@snowdog> <20101210124219.GC80263@Boo.local> Message-ID: <4D066EA0.6030801@ubuntu.com> On 10.12.2010 13:42, Brian Sutherland wrote: > The current method using dpkg-divert is not too bad, more packages than > python-zope.interface could include that file as well. So you don't > force installation of python-zope.interface. > > It also uses standard dpkg functionality, which is a robustness bonus. A diversion only works if not more than one package diverts the file. You would have to use alternatives to handle the file for more than two packages, although you would most likely point to instances of the same file. Matthias From bradallen137 at gmail.com Tue Dec 14 00:11:44 2010 From: bradallen137 at gmail.com (Brad Allen) Date: Mon, 13 Dec 2010 17:11:44 -0600 Subject: [Distutils] namespace packages ok to contain code? Message-ID: Where I work we're considering a change to moving a group of packages to using namespaces. We want the namespace paths to reflect a dependency hierarchy, and the current plan requires putting some source code into actual module distributions at the level of some of the namespace packages. Example: toplevel.midlevel toplevel.midlevel.a toplevel.midlevel.b Each of these three is a separate module distribution We want to put core library code in the midlevel. Some have warned this is not usually done, but no specific problems with this approach have been identified. Any guidance on this would be appreciated. Thanks! From bradallen137 at gmail.com Tue Dec 14 00:11:44 2010 From: bradallen137 at gmail.com (Brad Allen) Date: Mon, 13 Dec 2010 17:11:44 -0600 Subject: [Distutils] namespace packages ok to contain code? Message-ID: Where I work we're considering a change to moving a group of packages to using namespaces. We want the namespace paths to reflect a dependency hierarchy, and the current plan requires putting some source code into actual module distributions at the level of some of the namespace packages. Example: toplevel.midlevel toplevel.midlevel.a toplevel.midlevel.b Each of these three is a separate module distribution We want to put core library code in the midlevel. Some have warned this is not usually done, but no specific problems with this approach have been identified. Any guidance on this would be appreciated. Thanks! From martin at v.loewis.de Tue Dec 14 00:57:41 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 14 Dec 2010 00:57:41 +0100 Subject: [Distutils] namespace packages ok to contain code? In-Reply-To: References: Message-ID: <4D06B2F5.9070502@v.loewis.de> Am 14.12.2010 00:11, schrieb Brad Allen: > Where I work we're considering a change to moving a group of packages > to using namespaces. We want the namespace paths to reflect a > dependency hierarchy, and the current plan requires putting some > source code into actual module distributions at the level of some of > the namespace packages. > > Example: > > toplevel.midlevel > toplevel.midlevel.a > toplevel.midlevel.b > > Each of these three is a separate module distribution We want to put > core library code in the midlevel. This I don't understand. What would be the module name of the core library? > Some have warned this is not usually done, but no specific problems > with this approach have been identified. Any guidance on this would be > appreciated. You can't put code into __init__.py of any of the namespace packages. Otherwise, having additional modules in any of these packages is fine. Regards, Martin From bradallen137 at gmail.com Tue Dec 14 01:49:24 2010 From: bradallen137 at gmail.com (Brad Allen) Date: Mon, 13 Dec 2010 18:49:24 -0600 Subject: [Distutils] namespace packages ok to contain code? In-Reply-To: <4D06B2F5.9070502@v.loewis.de> References: <4D06B2F5.9070502@v.loewis.de> Message-ID: On Mon, Dec 13, 2010 at 5:57 PM, "Martin v. L?wis" wrote: > You can't put code into __init__.py of any of the namespace packages. > Otherwise, having additional modules in any of these packages is fine. Ok, that answers my question. Thanks! From adso.lists at gmail.com Tue Dec 14 01:54:35 2010 From: adso.lists at gmail.com (adso 0000) Date: Tue, 14 Dec 2010 01:54:35 +0100 Subject: [Distutils] custom site_dirs location Message-ID: Hi, i am doing something wrong probably, but when i try to get a custom site-packages location in Debian, i need to set the contents of altinstall.pth just several line with the absolute paths. If i use the usual python code with site.addsitedir(....) in altinstall.pth the directories are not added to sys.path. In sitecustomize.py, by other hand, the code works just fine. btw, i've not found any other manner to set a list for site-dirs in distutils.cfg than this: site_dirs = /path/to/dir1,/path/to/dir2 . Any space, tab or quote makes it fail for me :-S Any clue ? Thank you. --adso this works for me, though --- /home/user/opt/lib/python2.6/site-packages/altinstall.pth -- /home/user/opt/lib/python2.6/site-packages-testing3 /home/user/opt/lib/python2.6/site-packages-testing2 --- /home/user/opt/lib/python2.6/distutils/distutils.cfg --- [easy_install] site_dirs =/home/user/opt/lib/python2.6/site-packages-testing2,/home/user/opt/lib/python2.6/site-packages-testing3 From pje at telecommunity.com Tue Dec 14 02:21:38 2010 From: pje at telecommunity.com (P.J. Eby) Date: Mon, 13 Dec 2010 20:21:38 -0500 Subject: [Distutils] namespace packages ok to contain code? In-Reply-To: References: Message-ID: <20101214012139.BCA6C3A4076@sparrow.telecommunity.com> At 05:11 PM 12/13/2010 -0600, Brad Allen wrote: >Where I work we're considering a change to moving a group of packages >to using namespaces. We want the namespace paths to reflect a >dependency hierarchy, Note that you don't *need* to use namespace paths to reflect dependencies; the main purpose of namespace packages is to allow you to separately distribute parts of a larger package, and to avoid naming conflicts with code produced by other organizations. (In other words, it's a rare use case that really needs even *two* levels of namespace package; the common use is simply to make one parent namespace package for your organization. In general, the "flat is better than nested" rule of Python still applies. ;-) ) >Some have warned this is not usually done, but no specific problems >with this approach have been identified. Any guidance on this would be >appreciated. As Martin said, don't put code in the __init__.py of a namespace package. (And parent packages of namespace packages count as namespace packages.) From pje at telecommunity.com Tue Dec 14 02:23:32 2010 From: pje at telecommunity.com (P.J. Eby) Date: Mon, 13 Dec 2010 20:23:32 -0500 Subject: [Distutils] custom site_dirs location In-Reply-To: References: Message-ID: <20101214012328.16EB13A4076@sparrow.telecommunity.com> At 01:54 AM 12/14/2010 +0100, adso 0000 wrote: >Hi, >i am doing something wrong probably, but when i try to get a custom >site-packages location in Debian, i need to set the contents of >altinstall.pth just several line with the absolute paths. > >If i use the usual python code with site.addsitedir(....) in Are you using "import site; site.addsitedir(...)", or just "site.addsitedir(...)"? From adso.lists at gmail.com Tue Dec 14 02:30:16 2010 From: adso.lists at gmail.com (adso 0000) Date: Tue, 14 Dec 2010 02:30:16 +0100 Subject: [Distutils] custom site_dirs location. Solved Message-ID: Yeah. It was that. In this mail this become clear (http://www.mail-archive.com/distutils-sig at python.org/msg00439.html ) it seems the semi-colons are needed in .pth files, and i was using indentation as in python files. Thank you. --adso On Tue, Dec 14, 2010 at 2:23 AM, P.J. Eby wrote: > At 01:54 AM 12/14/2010 +0100, adso 0000 wrote: >> >> Hi, >> i am doing something wrong probably, but when i try to get a custom >> site-packages location in Debian, i need to set the contents of >> altinstall.pth just several line with the absolute paths. >> >> If i use the usual python code with site.addsitedir(....) in > > Are you using "import site; site.addsitedir(...)", or just > "site.addsitedir(...)"? > > From arve.knudsen at gmail.com Tue Dec 14 13:42:49 2010 From: arve.knudsen at gmail.com (Arve Knudsen) Date: Tue, 14 Dec 2010 13:42:49 +0100 Subject: [Distutils] How to best install pkg_resources? Message-ID: Hi Using Distribute in my setup.py, how do I best ensure that pkg_resources gets installed along with my package? I've tried 'install_requires=["setuptools>=0.6"]' as an argument to setup(), but setuptools didn't get installed. Thanks, Arve -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Tue Dec 14 14:24:45 2010 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 14 Dec 2010 08:24:45 -0500 Subject: [Distutils] How to best install pkg_resources? In-Reply-To: References: Message-ID: <20101214132445.3C7BE3A4076@sparrow.telecommunity.com> At 01:42 PM 12/14/2010 +0100, Arve Knudsen wrote: >Hi > >Using Distribute in my setup.py, how do I best ensure that >pkg_resources gets installed along with my package? I've tried >'install_requires=["setuptools>=0.6"]' as an argument to setup(), >but setuptools didn't get installed. I don't know about Distribute, but it should definitely get installed if you're using setuptools. Under what installation scenario(s) did setuptools *not* get installed? From arve.knudsen at gmail.com Tue Dec 14 14:40:54 2010 From: arve.knudsen at gmail.com (Arve Knudsen) Date: Tue, 14 Dec 2010 14:40:54 +0100 Subject: [Distutils] How to best install pkg_resources? In-Reply-To: <20101214132445.3C7BE3A4076@sparrow.telecommunity.com> References: <20101214132445.3C7BE3A4076@sparrow.telecommunity.com> Message-ID: On Tue, Dec 14, 2010 at 2:24 PM, P.J. Eby wrote: > At 01:42 PM 12/14/2010 +0100, Arve Knudsen wrote: > >> Hi >> >> Using Distribute in my setup.py, how do I best ensure that pkg_resources >> gets installed along with my package? I've tried >> 'install_requires=["setuptools>=0.6"]' as an argument to setup(), but >> setuptools didn't get installed. >> > > > I don't know about Distribute, but it should definitely get installed if > you're using setuptools. Under what installation scenario(s) did setuptools > *not* get installed? > Basically, my project bundles Distribute through distribute_setup.py, so Distribute doesn't need to be preinstalled. When I try to install my project (through 'python setup.py install') on a machine without setuptools/pkg_resources installed, however, setuptools does not get installed. I'll try to reduce this to a simple case, i.e. a dummy project that requires setuptools. Arve -------------- next part -------------- An HTML attachment was scrubbed... URL: From arve.knudsen at gmail.com Tue Dec 14 14:58:59 2010 From: arve.knudsen at gmail.com (Arve Knudsen) Date: Tue, 14 Dec 2010 14:58:59 +0100 Subject: [Distutils] How to best install pkg_resources? In-Reply-To: References: <20101214132445.3C7BE3A4076@sparrow.telecommunity.com> Message-ID: On Tue, Dec 14, 2010 at 2:40 PM, Arve Knudsen wrote: > On Tue, Dec 14, 2010 at 2:24 PM, P.J. Eby wrote: > >> At 01:42 PM 12/14/2010 +0100, Arve Knudsen wrote: >> >>> Hi >>> >>> Using Distribute in my setup.py, how do I best ensure that pkg_resources >>> gets installed along with my package? I've tried >>> 'install_requires=["setuptools>=0.6"]' as an argument to setup(), but >>> setuptools didn't get installed. >>> >> >> >> I don't know about Distribute, but it should definitely get installed if >> you're using setuptools. Under what installation scenario(s) did setuptools >> *not* get installed? >> > > Basically, my project bundles Distribute through distribute_setup.py, so > Distribute doesn't need to be preinstalled. When I try to install my project > (through 'python setup.py install') on a machine without > setuptools/pkg_resources installed, however, setuptools does not get > installed. > > I'll try to reduce this to a simple case, i.e. a dummy project that > requires setuptools. > This is strange, with a super-simple case (see attachment), Distribute gets installed without even having to state install_requires. Apparently, Distribute is supposed to install itself automatically. I wonder what goes wrong in my more elaborate project.. Arve -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test.zip Type: application/zip Size: 386 bytes Desc: not available URL: From arve.knudsen at gmail.com Tue Dec 14 15:14:40 2010 From: arve.knudsen at gmail.com (Arve Knudsen) Date: Tue, 14 Dec 2010 15:14:40 +0100 Subject: [Distutils] How to best install pkg_resources? In-Reply-To: References: <20101214132445.3C7BE3A4076@sparrow.telecommunity.com> Message-ID: I was finally able to determine the reason. I am installing a specialized 'install' command with the argument 'cmdclass={"install": install}' to setup(). I've attached a revised case which demonstrates the problem. In my specialized 'install' class I try to wrap the normal installation command by calling setuptools.command.install.install, but apparently that's not enough. What should I call to invoke the normal installation command? Thanks, Arve On Tue, Dec 14, 2010 at 2:58 PM, Arve Knudsen wrote: > On Tue, Dec 14, 2010 at 2:40 PM, Arve Knudsen wrote: > >> On Tue, Dec 14, 2010 at 2:24 PM, P.J. Eby wrote: >> >>> At 01:42 PM 12/14/2010 +0100, Arve Knudsen wrote: >>> >>>> Hi >>>> >>>> Using Distribute in my setup.py, how do I best ensure that pkg_resources >>>> gets installed along with my package? I've tried >>>> 'install_requires=["setuptools>=0.6"]' as an argument to setup(), but >>>> setuptools didn't get installed. >>>> >>> >>> >>> I don't know about Distribute, but it should definitely get installed if >>> you're using setuptools. Under what installation scenario(s) did setuptools >>> *not* get installed? >>> >> >> Basically, my project bundles Distribute through distribute_setup.py, so >> Distribute doesn't need to be preinstalled. When I try to install my project >> (through 'python setup.py install') on a machine without >> setuptools/pkg_resources installed, however, setuptools does not get >> installed. >> >> I'll try to reduce this to a simple case, i.e. a dummy project that >> requires setuptools. >> > > This is strange, with a super-simple case (see attachment), Distribute gets > installed without even having to state install_requires. Apparently, > Distribute is supposed to install itself automatically. I wonder what goes > wrong in my more elaborate project.. > > Arve > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test.zip Type: application/zip Size: 513 bytes Desc: not available URL: From arve.knudsen at gmail.com Tue Dec 14 15:33:22 2010 From: arve.knudsen at gmail.com (Arve Knudsen) Date: Tue, 14 Dec 2010 15:33:22 +0100 Subject: [Distutils] How to best install pkg_resources? In-Reply-To: References: <20101214132445.3C7BE3A4076@sparrow.telecommunity.com> Message-ID: I found out how to solve it; I need to call setuptools.command.install.install.do_egg_install, since setuptools.command.install.install.run will emulate the standard distutils behaviour unless it's run directly from setup.py :( This behaviour was hardly obvious I have to say :( Arve On Tue, Dec 14, 2010 at 3:14 PM, Arve Knudsen wrote: > I was finally able to determine the reason. I am installing a specialized > 'install' command with the argument 'cmdclass={"install": install}' to > setup(). I've attached a revised case which demonstrates the problem. > > In my specialized 'install' class I try to wrap the normal installation > command by calling setuptools.command.install.install, but apparently that's > not enough. What should I call to invoke the normal installation command? > > Thanks, > Arve > > > On Tue, Dec 14, 2010 at 2:58 PM, Arve Knudsen wrote: > >> On Tue, Dec 14, 2010 at 2:40 PM, Arve Knudsen wrote: >> >>> On Tue, Dec 14, 2010 at 2:24 PM, P.J. Eby wrote: >>> >>>> At 01:42 PM 12/14/2010 +0100, Arve Knudsen wrote: >>>> >>>>> Hi >>>>> >>>>> Using Distribute in my setup.py, how do I best ensure that >>>>> pkg_resources gets installed along with my package? I've tried >>>>> 'install_requires=["setuptools>=0.6"]' as an argument to setup(), but >>>>> setuptools didn't get installed. >>>>> >>>> >>>> >>>> I don't know about Distribute, but it should definitely get installed if >>>> you're using setuptools. Under what installation scenario(s) did setuptools >>>> *not* get installed? >>>> >>> >>> Basically, my project bundles Distribute through distribute_setup.py, so >>> Distribute doesn't need to be preinstalled. When I try to install my project >>> (through 'python setup.py install') on a machine without >>> setuptools/pkg_resources installed, however, setuptools does not get >>> installed. >>> >>> I'll try to reduce this to a simple case, i.e. a dummy project that >>> requires setuptools. >>> >> >> This is strange, with a super-simple case (see attachment), Distribute >> gets installed without even having to state install_requires. Apparently, >> Distribute is supposed to install itself automatically. I wonder what goes >> wrong in my more elaborate project.. >> >> Arve >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Tue Dec 14 18:15:19 2010 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 14 Dec 2010 12:15:19 -0500 Subject: [Distutils] How to best install pkg_resources? In-Reply-To: References: <20101214132445.3C7BE3A4076@sparrow.telecommunity.com> Message-ID: <20101214171526.6D2FD3A4076@sparrow.telecommunity.com> At 03:33 PM 12/14/2010 +0100, Arve Knudsen wrote: >I found out how to solve it; I need to call >setuptools.command.install.install.do_egg_install, since >setuptools.command.install.install.run will emulate the standard >distutils behaviour unless it's run directly from setup.py :( This >behaviour was hardly obvious I have to say :( Worse news: unless you emulate that same behavior in *your* install class, you will break bdist_rpm, bdist_wininst, bdist_dumb, etc. In other words, all that non-obvious behavior is there for backward compatibility with other distutils commands that invoke "install" as a subcommand. If you test your now-"fixed" setup.py with them, you will find that your project does terribly broken things. (i.e., try "bdist_rpm" or "bdist_wininst" on a simple setup and your more complex project, and compare the results.) From arve.knudsen at gmail.com Tue Dec 14 18:30:10 2010 From: arve.knudsen at gmail.com (Arve Knudsen) Date: Tue, 14 Dec 2010 18:30:10 +0100 Subject: [Distutils] How to best install pkg_resources? In-Reply-To: <20101214171526.6D2FD3A4076@sparrow.telecommunity.com> References: <20101214132445.3C7BE3A4076@sparrow.telecommunity.com> <20101214171526.6D2FD3A4076@sparrow.telecommunity.com> Message-ID: On Tue, Dec 14, 2010 at 6:15 PM, P.J. Eby wrote: > At 03:33 PM 12/14/2010 +0100, Arve Knudsen wrote: > >> I found out how to solve it; I need to call >> setuptools.command.install.install.do_egg_install, since >> setuptools.command.install.install.run will emulate the standard distutils >> behaviour unless it's run directly from setup.py :( This behaviour was >> hardly obvious I have to say :( >> > > Worse news: unless you emulate that same behavior in *your* install class, > you will break bdist_rpm, bdist_wininst, bdist_dumb, etc. > > In other words, all that non-obvious behavior is there for backward > compatibility with other distutils commands that invoke "install" as a > subcommand. If you test your now-"fixed" setup.py with them, you will find > that your project does terribly broken things. > > (i.e., try "bdist_rpm" or "bdist_wininst" on a simple setup and your more > complex project, and compare the results.) > Argh :( Can you describe some other way of customizing the install command that works better with setuptools? Arve -------------- next part -------------- An HTML attachment was scrubbed... URL: From l at t-0.be Tue Dec 14 19:07:19 2010 From: l at t-0.be (Devin Bayer) Date: Tue, 14 Dec 2010 19:07:19 +0100 Subject: [Distutils] egg-links are added to end of sys.path? Message-ID: Hello. I have been struggling a bit with using buildout for developing eggs and would appreciate some advice. "." is added to the end of sys.path, so the older version of the package (installed in eggs/) is used instead. If I add the following code, it works. But I feel like I'm missing something and this should not be needed: ====================== [python] initialization = import sys sys.path.insert(0, '.') ====================== Any ideas? Thanks, Devin === my buildout.cfg === [buildout] parts = python develop = . [python] recipe = z3c.recipe.scripts eggs = myegg ====================== From pje at telecommunity.com Tue Dec 14 19:22:45 2010 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 14 Dec 2010 13:22:45 -0500 Subject: [Distutils] How to best install pkg_resources? In-Reply-To: References: <20101214132445.3C7BE3A4076@sparrow.telecommunity.com> <20101214171526.6D2FD3A4076@sparrow.telecommunity.com> Message-ID: <20101214182245.C74793A4076@sparrow.telecommunity.com> At 06:30 PM 12/14/2010 +0100, Arve Knudsen wrote: >Argh :( Can you describe some other way of customizing the install >command that works better with setuptools? That depends on what you're trying to do; but probably it will involve subclassing the install command rather than simply invoking it. What are you trying to customize, anyway? There really aren't many user-serviceable parts inside; you'd probably be better off customizing subcommands (like install_lib or some of the build_* commands). From arve.knudsen at gmail.com Tue Dec 14 19:30:23 2010 From: arve.knudsen at gmail.com (Arve Knudsen) Date: Tue, 14 Dec 2010 19:30:23 +0100 Subject: [Distutils] How to best install pkg_resources? In-Reply-To: <20101214182245.C74793A4076@sparrow.telecommunity.com> References: <20101214132445.3C7BE3A4076@sparrow.telecommunity.com> <20101214171526.6D2FD3A4076@sparrow.telecommunity.com> <20101214182245.C74793A4076@sparrow.telecommunity.com> Message-ID: On Tue, Dec 14, 2010 at 7:22 PM, P.J. Eby wrote: > At 06:30 PM 12/14/2010 +0100, Arve Knudsen wrote: > >> Argh :( Can you describe some other way of customizing the install command >> that works better with setuptools? >> > > That depends on what you're trying to do; but probably it will involve > subclassing the install command rather than simply invoking it. > > What are you trying to customize, anyway? There really aren't many > user-serviceable parts inside; you'd probably be better off customizing > subcommands (like install_lib or some of the build_* commands). > I don't have the project in front of me right now, but what I needed to do was to install something non-Python. That is, if the package gets installed under /lib, the extra data gets installed under /data. The data is part of the project, but not part of the Python package(s). Arve -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Tue Dec 14 19:39:03 2010 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 14 Dec 2010 13:39:03 -0500 Subject: [Distutils] How to best install pkg_resources? In-Reply-To: References: <20101214132445.3C7BE3A4076@sparrow.telecommunity.com> <20101214171526.6D2FD3A4076@sparrow.telecommunity.com> <20101214182245.C74793A4076@sparrow.telecommunity.com> Message-ID: <20101214183901.CDCC33A4076@sparrow.telecommunity.com> At 07:30 PM 12/14/2010 +0100, Arve Knudsen wrote: >On Tue, Dec 14, 2010 at 7:22 PM, P.J. Eby ><pje at telecommunity.com> wrote: >At 06:30 PM 12/14/2010 +0100, Arve Knudsen wrote: >Argh :( Can you describe some other way of customizing the install >command that works better with setuptools? > > >That depends on what you're trying to do; but probably it will >involve subclassing the install command rather than simply invoking it. > >What are you trying to customize, anyway? There really aren't many >user-serviceable parts inside; you'd probably be better off >customizing subcommands (like install_lib or some of the build_* commands). > > >I don't have the project in front of me right now, but what I needed >to do was to install something non-Python. That is, if the package >gets installed under /lib, the extra data gets installed >under /data. The data is part of the project, but not part >of the Python package(s). That's not going to work except in the old-style installation mode. With a setuptools or distribute project, your only places to store data are inside the package or in the metadata (e.g. a subdirectory inside .egg-info). From arve.knudsen at gmail.com Tue Dec 14 22:09:05 2010 From: arve.knudsen at gmail.com (Arve Knudsen) Date: Tue, 14 Dec 2010 22:09:05 +0100 Subject: [Distutils] How to best install pkg_resources? In-Reply-To: <20101214183901.CDCC33A4076@sparrow.telecommunity.com> References: <20101214132445.3C7BE3A4076@sparrow.telecommunity.com> <20101214171526.6D2FD3A4076@sparrow.telecommunity.com> <20101214182245.C74793A4076@sparrow.telecommunity.com> <20101214183901.CDCC33A4076@sparrow.telecommunity.com> Message-ID: On Tue, Dec 14, 2010 at 7:39 PM, P.J. Eby wrote: > At 07:30 PM 12/14/2010 +0100, Arve Knudsen wrote: > > On Tue, Dec 14, 2010 at 7:22 PM, P.J. Eby < >> pje at telecommunity.com> wrote: >> At 06:30 PM 12/14/2010 +0100, Arve Knudsen wrote: >> Argh :( Can you describe some other way of customizing the install command >> that works better with setuptools? >> >> >> That depends on what you're trying to do; but probably it will involve >> subclassing the install command rather than simply invoking it. >> >> What are you trying to customize, anyway? There really aren't many >> user-serviceable parts inside; you'd probably be better off customizing >> subcommands (like install_lib or some of the build_* commands). >> >> >> I don't have the project in front of me right now, but what I needed to do >> was to install something non-Python. That is, if the package gets installed >> under /lib, the extra data gets installed under /data. The >> data is part of the project, but not part of the Python package(s). >> > > That's not going to work except in the old-style installation mode. With a > setuptools or distribute project, your only places to store data are inside > the package or in the metadata (e.g. a subdirectory inside .egg-info). > I'm aware of that issue, but it doesn't matter in this case, since the data are really orthogonal to the Python distribution (and the project is proprietary). Arve -------------- next part -------------- An HTML attachment was scrubbed... URL: From wichert at wiggy.net Tue Dec 14 22:44:59 2010 From: wichert at wiggy.net (Wichert Akkerman) Date: Tue, 14 Dec 2010 22:44:59 +0100 Subject: [Distutils] Including symlinked data files in an egg Message-ID: <4D07E55B.3090700@wiggy.net> I have a package with a source that looks like this: setup.py MANIFEST.in Prototype/libraries/jquery.js Prototype/style/main.css my/ my/package/ my/package/templates/libraries -> ../../../Prototype/libraries my/package/templates/style -> ../../../Prototype/style what I would like to accomplish is that when a bdist is made either Prototype/ is installed somewhere and the symlinks point to it, or the symlinks are replaced with a copy of the data they point to. For a package maintained in subverion this works: when the sdist is created the symlink is replaced with a copy of the thing it points to, which is then included in the bdist. For a package maintained in git this does not work: the symlink always appears to be ignored. Is there a way to accomplish this? Wichert. -- Wichert Akkerman It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. From lac at openend.se Wed Dec 15 01:50:01 2010 From: lac at openend.se (Laura Creighton) Date: Wed, 15 Dec 2010 01:50:01 +0100 Subject: [Distutils] How to best install pkg_resources? Message-ID: <201012150050.oBF0o17P022161@theraft.openend.se> Arve: You may find that Bento already does what you want. http://cournape.github.com/Bento/ Laura Creighton From bradallen137 at gmail.com Wed Dec 15 04:21:01 2010 From: bradallen137 at gmail.com (Brad Allen) Date: Tue, 14 Dec 2010 21:21:01 -0600 Subject: [Distutils] namespace packages ok to contain code? In-Reply-To: <20101214012139.BCA6C3A4076@sparrow.telecommunity.com> References: <20101214012139.BCA6C3A4076@sparrow.telecommunity.com> Message-ID: On Mon, Dec 13, 2010 at 7:21 PM, P.J. Eby wrote: > At 05:11 PM 12/13/2010 -0600, Brad Allen wrote: >> >> Where I work we're considering a change to moving a group of packages >> to using namespaces. We want the namespace paths to reflect a >> dependency hierarchy, > > Note that you don't *need* to use namespace paths to reflect dependencies; > the main purpose of namespace packages is to allow you to separately > distribute parts of a larger package, and to avoid naming conflicts with > code produced by other organizations. > (In other words, it's a rare use case that really needs even *two* levels of > namespace package; the common use is simply to make one parent namespace > package for your organization. In general, the "flat is better than nested" > rule of Python still applies. ?;-) ) Thanks for the guidance. In our case, after thinking through the component design for a large application, we decided not to use the namespace package part of the component name to describe dependencies...that will be done by a different part of the component name. Instead, we can use namespace packages for grouping components with related functionality even when they exist at different levels within the architecture (web components, model components, integration components). There are some functional areas where we really can't put those functionally related components into the same package, so we really will end up with some two-level namespace packages (productname.functionalgrouping.model, productname.functionalgrouping.web, etc). It's important to be able to independently release those kinds of components to customers without necessarily requiring an upgrade of the other related component. This will introduce some manageability challenges with a large number of module distributions but I think we can avoid dependency hell with strict semantic versioning (per semver.org), good dependency declarations in setup.py, comprehensive test suites, and use of continuous integration to run tests and automate builds. This can also be made manageable for developers by using buildout + mr.developer + omelette (which I just learned about, and can't wait to try out). See http://pypi.python.org/pypi/collective.recipe.omelette Feel free to rain on my parade with a reality check :-) At this point it feels a bit like going off the deep end, but this migration will be done piecewise, not with one big cutover. It will be an interesting exercise in seeing what the maturing Python build tools ecosystem has made possible. From arve.knudsen at gmail.com Wed Dec 15 07:53:21 2010 From: arve.knudsen at gmail.com (Arve Knudsen) Date: Wed, 15 Dec 2010 07:53:21 +0100 Subject: [Distutils] How to best install pkg_resources? In-Reply-To: <201012150050.oBF0o17P022161@theraft.openend.se> References: <201012150050.oBF0o17P022161@theraft.openend.se> Message-ID: Hi Laura and thanks for the tip. I'd never heard of Bento :) Arve On Wed, Dec 15, 2010 at 1:50 AM, Laura Creighton wrote: > Arve: > > You may find that Bento already does what you want. > http://cournape.github.com/Bento/ > > Laura Creighton > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From reinout at vanrees.org Wed Dec 15 12:37:19 2010 From: reinout at vanrees.org (Reinout van Rees) Date: Wed, 15 Dec 2010 12:37:19 +0100 Subject: [Distutils] Newbie - Buildout bootstrap error pkg_resources.DistributionNotFound: zc.buildout==1.5.2 In-Reply-To: References: Message-ID: On 12/12/2010 06:49 PM, Cops Club wrote: > > plonesvc at ubuntu:~/exercise1$ buildout init Hey, this means you already installed buildout globally somehow. ("easy_install zc.buildout", for instance?) > 3) I downloaded the bootstrap.py from svn > plonesvc at ubuntu:~$ wget \ > > http://svn.zope.org/*checkout*/zc.buildout/trunk/bootstrap/bootstrap.py > > > 4)And tried to run the bootstrap - which failed: That bootstrap script downloads and installs buildout local to that buildout directory. But you've already installed buildout globally. Apparently, that bites. You mention that it *does* work in a virtualenv: yep, there you're isolated from the global buildout. => Best you remove the globally installed zc.buildout. Alternatively you can try "python -S bootstrap.py" to run python without system packages. Reinout -- Reinout van Rees - reinout at vanrees.org - http://reinout.vanrees.org Collega's gezocht! Django/python vacature in Utrecht: http://tinyurl.com/35v34f9 From pje at telecommunity.com Wed Dec 15 16:15:59 2010 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 15 Dec 2010 10:15:59 -0500 Subject: [Distutils] Including symlinked data files in an egg In-Reply-To: <4D07E55B.3090700@wiggy.net> References: <4D07E55B.3090700@wiggy.net> Message-ID: <20101215151601.0BB8B3A4076@sparrow.telecommunity.com> At 10:44 PM 12/14/2010 +0100, Wichert Akkerman wrote: >I have a package with a source that looks like this: > > setup.py > MANIFEST.in > Prototype/libraries/jquery.js > Prototype/style/main.css > my/ > my/package/ > my/package/templates/libraries -> ../../../Prototype/libraries > my/package/templates/style -> ../../../Prototype/style > >what I would like to accomplish is that when a bdist is made either >Prototype/ is installed somewhere and the symlinks point to it, or >the symlinks are replaced with a copy of the data they point to. > >For a package maintained in subverion this works: when the sdist is >created the symlink is replaced with a copy of the thing it points >to, which is then included in the bdist. For a package maintained in >git this does not work: the symlink always appears to be ignored. Is there >a way to accomplish this? Most likely, the problem is with the file finder being used. Are you using a git plugin for setuptools, or simply listing everything in MANIFEST.in? If the former, it may be that the git plugin you're using simply doesn't support symlinks. If you're not using a git plugin, perhaps there is something amiss in the MANIFEST.in, and setuptools' built-in support for subversion is hiding the problem by picking up the symlinks. >Wichert. > >-- >Wichert Akkerman It is simple to make things. >http://www.wiggy.net/ It is hard to make things simple. >_______________________________________________ >Distutils-SIG maillist - Distutils-SIG at python.org >http://mail.python.org/mailman/listinfo/distutils-sig From wichert at wiggy.net Wed Dec 15 16:18:27 2010 From: wichert at wiggy.net (Wichert Akkerman) Date: Wed, 15 Dec 2010 16:18:27 +0100 Subject: [Distutils] Including symlinked data files in an egg In-Reply-To: <20101215151601.0BB8B3A4076@sparrow.telecommunity.com> References: <4D07E55B.3090700@wiggy.net> <20101215151601.0BB8B3A4076@sparrow.telecommunity.com> Message-ID: <4D08DC43.7040003@wiggy.net> On 12/15/10 16:15 , P.J. Eby wrote: > Most likely, the problem is with the file finder being used. Are you > using a git plugin for setuptools, or simply listing everything in > MANIFEST.in? If the former, it may be that the git plugin you're using > simply doesn't support symlinks. If you're not using a git plugin, > perhaps there is something amiss in the MANIFEST.in, and setuptools' > built-in support for subversion is hiding the problem by picking up the > symlinks. I am not using a git (or other) plugin. I tried listing things in MANIFEST.in, for example using: graft my/package/templates/ graft my/package/templates/libraries unfortunately that had no effect: libraries was not included, not as symlink nor as a copy of the symlink target. Wichert. From arve.knudsen at gmail.com Wed Dec 15 17:49:56 2010 From: arve.knudsen at gmail.com (Arve Knudsen) Date: Wed, 15 Dec 2010 17:49:56 +0100 Subject: [Distutils] Obtuse warning from pkg_resources when egg and non-egg distributions clash Message-ID: Hello I've just been able to properly identify why pkg_resources at times issues an obtuse (to me at least) warning along these lines: "Module was already imported from , but is being added to sys.path". It turns out that really that a non-egg distribution is on the module search path before a corresponding egg distribution. If you run bin/test.py in the attached case, you should see a demonstration of the issue. Right now, it bit me when testing under a continuous integration slave on a machine where the tested project is also installed as an egg. I couldn't for the life of me understand why this warning was being issued, when I couldn't see it on my development machine. Not before painstakingly debugging pkg_resources, did it dawn on me that pkg_resources only complained on the test machine since there was no egg metadata in the source tree (whereas in my development source tree there is). Until debugging pkg_resources I actually didn't know that a directory (i.e., with Python packages) is considered an egg if it has a .egg-info file (with egg metadata) in it. I just hadn't had the motivation yet to learn the egg technology that well. Considering that pkg_resources' rationale for issuing this warning is so difficult to derive from the message, may I suggest please that we modify the message somewhat for added clarity? If it were more explicit, it should be much easier to rectify the warning's cause. My suggestion is a message on this form: "'' is being added to sys.path, but contained module '' was already imported from ''. Could this be a non-egg distribution?". From such a warning, it should be possible to derive that the module may have been imported from what is not considered by pkg_resources an egg distribution (such as a clean source tree). Thanks, Arve -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test.zip Type: application/zip Size: 6458 bytes Desc: not available URL: From sean.m.ochoa at gmail.com Wed Dec 15 19:05:48 2010 From: sean.m.ochoa at gmail.com (Sean Ochoa) Date: Wed, 15 Dec 2010 10:05:48 -0800 Subject: [Distutils] stdeb development? Message-ID: Hello all. I'm wondering if there is still any effort behind the stdeb package development. I find it super handy for what I do at work, and I'm wondering if there's going to be canonical support and/or further package development done in this area. - Sean -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Wed Dec 15 21:58:07 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 15 Dec 2010 15:58:07 -0500 Subject: [Distutils] stdeb development? In-Reply-To: References: Message-ID: <20101215155807.32c93b97@mission> On Dec 15, 2010, at 10:05 AM, Sean Ochoa wrote: >Hello all. I'm wondering if there is still any effort behind the stdeb >package development. I find it super handy for what I do at work, and I'm >wondering if there's going to be canonical support and/or further package >development done in this area. - Sean As awesome as stdeb is, you should take a look at pkgme: https://launchpad.net/pkgme This is a generic tool that generates packaging, not just for Python, but for all kinds of simple projects. It supports a set of backends for language/project specific details and handles all the common stuff so individual backends don't need to. It was discussed at the last UDS, and James Westby is doing most of the driving right now. I'm trying to contribute a few small things, and will be spending any future related effort on pkgme. Python is one of the built-in backends. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From pje at telecommunity.com Thu Dec 16 00:52:11 2010 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 15 Dec 2010 18:52:11 -0500 Subject: [Distutils] Including symlinked data files in an egg In-Reply-To: <4D08DC43.7040003@wiggy.net> References: <4D07E55B.3090700@wiggy.net> <20101215151601.0BB8B3A4076@sparrow.telecommunity.com> <4D08DC43.7040003@wiggy.net> Message-ID: <20101215235214.19D923A408B@sparrow.telecommunity.com> At 04:18 PM 12/15/2010 +0100, Wichert Akkerman wrote: >On 12/15/10 16:15 , P.J. Eby wrote: >>Most likely, the problem is with the file finder being used. Are you >>using a git plugin for setuptools, or simply listing everything in >>MANIFEST.in? If the former, it may be that the git plugin you're using >>simply doesn't support symlinks. If you're not using a git plugin, >>perhaps there is something amiss in the MANIFEST.in, and setuptools' >>built-in support for subversion is hiding the problem by picking up the >>symlinks. > >I am not using a git (or other) plugin. I tried listing things in >MANIFEST.in, for example using: > > graft my/package/templates/ > graft my/package/templates/libraries > >unfortunately that had no effect: libraries was not included, not as >symlink nor as a copy of the symlink target. I don't know whether MANIFEST.in is supposed to support symlinks or not, but the fact that it's working with subversion is strongly indicative that setuptools' file finder is picking up stuff that MANIFEST.in is not. I don't know *why* MANIFEST.in wouldn't pick up symlinks, though. Perhaps you could try boiling this down to a simpler test case (e.g., subversion and git versions of the same minimal package containing only a symlink referred to by MANIFEST.in) so we can narrow it down some more? >Wichert. From ozarow at gmail.com Thu Dec 16 11:49:16 2010 From: ozarow at gmail.com (Piotr Ozarowski) Date: Thu, 16 Dec 2010 11:49:16 +0100 Subject: [Distutils] stdeb development? In-Reply-To: References: Message-ID: <20101216104916.GV22258@piotro.eu> [Sean Ochoa, 2010-12-15] > Hello all. I'm wondering if there is still any effort behind the stdeb > package development. I find it super handy for what I do at work, and I'm > wondering if there's going to be canonical support and/or further package > development done in this area. - Sean I made few changes? in python-stdeb Debian package (and I plan to add more) - hopefully Andrew will merge them soon... ? http://svn.debian.org/viewsvn/python-modules/packages/stdeb/trunk/debian/patches/ -- Piotr O?arowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645 From mailing at franzoni.eu Fri Dec 17 16:05:25 2010 From: mailing at franzoni.eu (Alan Franzoni) Date: Fri, 17 Dec 2010 16:05:25 +0100 Subject: [Distutils] Buildout or distribute raise unnecessary version conflict? Message-ID: Hello, I've got a strange issue with version resolution when using zc.buildout; I don't know if it's zc.buildout specific or it's caused by something which is done by distribute. The example code can be fetched from there: hg clone https://Alan at bitbucket.org/Alan/buildoutversionissue End of output result: Installing test. While: Installing test. Error: There is a version conflict. We already have: zope.testrunner 4.0.0 but asd 0.1.0 requires 'zope.testrunner==4.0.0b1'. If you check within the downloaded eggs, you'll see there's no egg requiring zope.testrunner 4.0.0; there's just zc.recipe.testrunner requiring "zope.testrunner", without any version. if egg A requires "foolib" while egg B requires "foolib==1.0.0", and the latest available version of foolib is "1.0.1", then a version conflict error will arise; it's just as if a "naked" egg dependency is pinned to the latest available version of a package. Is that an intended behaviour? I don't seem to find any evidence of that in distribute or distutils doc. Although specifying a dependency without a version shouldn't be a good practice, it seems quite common - mostly because many people have got full control of the pypi section on their own repo and they can decide what's offered there, so they just specify "mydependency" on their other projects. I think it's quite a wide assumption, by the way. Also, this is not the usualy way dependencies are resolved in most package managers. When using yum (and apt should be quite similar), something like that happens: Scenario: two versions of "mydep" are available: mydep 1.0.0 and 1.0.1 "foo" depends just on mydep "bar" depends on mydep 1.0.0 If I just want to install "foo", I'll get foo and mydep 1.0.1 on my system. But if I try installing both, I'll get foo, bar and mydep 1.0.0, because that will be resolved as the best way to handle all dependencies. Any comment? -- Alan Franzoni -- contact me at public@[mysurname].eu From jim at zope.com Fri Dec 17 16:48:23 2010 From: jim at zope.com (Jim Fulton) Date: Fri, 17 Dec 2010 10:48:23 -0500 Subject: [Distutils] Buildout or distribute raise unnecessary version conflict? In-Reply-To: References: Message-ID: On Fri, Dec 17, 2010 at 10:05 AM, Alan Franzoni wrote: > Hello, > > I've got a strange issue with version resolution when using > zc.buildout; I don't know if it's zc.buildout specific or it's caused > by something which is done by distribute. > > The example code can be fetched from there: > > hg clone https://Alan at bitbucket.org/Alan/buildoutversionissue > > End of output result: > > Installing test. > While: > ?Installing test. > Error: There is a version conflict. > We already have: zope.testrunner 4.0.0 > but asd 0.1.0 requires 'zope.testrunner==4.0.0b1'. > > If you check within the downloaded eggs, you'll see there's no egg > requiring zope.testrunner 4.0.0; there's just zc.recipe.testrunner > requiring "zope.testrunner", without any version. > > if egg A requires "foolib" while egg B requires "foolib==1.0.0", and > the latest available version of foolib is "1.0.1", then a version > conflict error will arise; it's just as if a "naked" egg dependency is > pinned to the latest available version of a package. > > Is that an intended behaviour? Sort of. buildout (and setuptools and distrtibute, I assume) will, when fetching a distribution, always fetch the most recent that meets the given requirements. Buildout is a bit different in that it checks for new versions any time it runs. Now, if you have project A that requires B and C, which each require D, then when installing B, buildout will retrieve the newest distribution of D that satisfies B's requirements. Later, when it installs C, if C requires a different version of D, then there is a conflict. The same thing would happen afaik, using easy_install if none of the projects are installed at the start. It would be better if this situation was handled more cleanly. In the past I've considered backtracking schemes to handle situations like this, which seemed scary. :) It occurs to me writing this that a breadth first, which wouldn't be hard to implement, might do much better than the current depth first approach. > Although specifying a dependency > without a version shouldn't be a good practice, Actually, it's best practice. > it seems quite common > - mostly because many people have got full control of the pypi section > on their own repo and they can decide what's offered there, so they > just specify "mydependency" on their other projects. I think it's > quite a wide assumption, by the way. It's best practice. Requiring a specific version is extremely inflexible. If everyone did it, you'd never be able to assemble anything of any size. > > Also, this is not the usualy way dependencies are resolved in most > package managers. When using yum (and apt should be quite similar), > something like that happens: > > Scenario: > two versions of "mydep" are available: mydep 1.0.0 and 1.0.1 > "foo" depends just on mydep > "bar" depends on mydep 1.0.0 > > If I just want to install "foo", I'll get foo and mydep 1.0.1 on my > system. But if I try installing both, I'll get foo, bar and mydep > 1.0.0, because that will be resolved as the best way to handle all > dependencies. > > Any comment? Maybe system packing tools use better dependency-resolution mechanisms. It could happen! I have certainly been stymied by version conflicts in system packaging systems, so I don't think their algorithms are that great. The way to work around this with buildout is to use a buildout versions section: http://pypi.python.org/pypi/zc.buildout#repeatable-buildouts-controlling-eggs-used It would be interesting to see if a breadth first strategy would provide better behavior. Jim -- Jim Fulton From benji at benjiyork.com Fri Dec 17 17:01:41 2010 From: benji at benjiyork.com (Benji York) Date: Fri, 17 Dec 2010 11:01:41 -0500 Subject: [Distutils] Buildout or distribute raise unnecessary version conflict? In-Reply-To: References: Message-ID: On Fri, Dec 17, 2010 at 10:05 AM, Alan Franzoni wrote: > if egg A requires "foolib" while egg B requires "foolib==1.0.0", and > the latest available version of foolib is "1.0.1", then a version > conflict error will arise; it's just as if a "naked" egg dependency is > pinned to the latest available version of a package. I seem to recall that this is a result of the non-backtracking behavior of setuptools version resolution. It's not a bug per se, just that it doesn't go the extra mile. If I'm wrong I'm sure someone will correct me. > If I just want to install "foo", I'll get foo and mydep 1.0.1 on my > system. But if I try installing both, I'll get foo, bar and mydep > 1.0.0, because that will be resolved as the best way to handle all > dependencies. Fancier version resolution would be nice, however a library-like package (as opposed to an app) that requires one and only one version of a dependency is being antisocial. You can use zc.buildout to enforce the use of the 1.0.1 version for you as a work-around: http://pypi.python.org/pypi/zc.buildout#specifying-version-information-independent-of-requirements -- Benji York From mailing at franzoni.eu Fri Dec 17 17:05:58 2010 From: mailing at franzoni.eu (Alan Franzoni) Date: Fri, 17 Dec 2010 17:05:58 +0100 Subject: [Distutils] Buildout or distribute raise unnecessary version conflict? In-Reply-To: References: Message-ID: On Fri, Dec 17, 2010 at 4:48 PM, Jim Fulton wrote: >> Although specifying a dependency >> without a version shouldn't be a good practice, > > Actually, it's best practice. I don't think so. It's what people does, removing packages from pypi it's common, but it's awful. As a library mantainer, I can never break an API, not even if I release 2.0 after working on 1.0, because projects depending on my lib might just break. Hence we get "project" "project2" "project3", e.g. versioning through project naming. Is that a good idea? Consider the Java world, where Maven is a de-facto standard. Once a package with a certain version is uploaded, it's usually never removed. You can decide to depend on the latest 3.x.x, on the latest 3.5.x, on version ranges, or on a very specific versions. This makes it possibile for developers to depend on the versions they really need. Dependency problems may arise whatsoever and must be usually manually fixed, but in this case the Python way seems to say "hey, let's pretend the problem doesn't exist and the API never changes in a backwards incompatible way". >> it seems quite common >> - mostly because many people have got full control of the pypi section >> on their own repo and they can decide what's offered there, so they >> just specify "mydependency" on their other projects. I think it's >> quite a wide assumption, by the way. > > It's best practice. ?Requiring a specific version is extremely inflexible. > If everyone did it, you'd never be able to assemble anything of any size. You might require a version range - I think I asked something about version specification, which is quite fuzzy in setuptools/distribute, some weeks ago. BTW if your require the API exposed by a project at version, let's say, 1.4 (or "at least 1.4 but less than 2.0") it won't do any good not to specify your version. You're effectively depending on the state of an external entity which is not under your control. Scenario: You project A today works 100% with a certain version of lib FOO, which is, let's say, 1.0.0, but you don't explicitly write that version requirement. In a year, you get back to project A, run the tests, and everything fails, because today's lib FOO 1.2.0 API changed. Of course you could use a private repository as a backend instead of pypi, but this means you'd require an additional, stateful indirection level for what could be declaratively expressed just by version numbers. > Maybe system packing tools use better dependency-resolution > mechanisms. ?It could happen! I have certainly been stymied by > version conflicts in system packaging systems, so I don't think their > algorithms are that great. Version conflicts of course can arise. If there's no possible dependency path, nor yum neither apt will be able to do anything. > > The way to work around this with buildout is to use a buildout versions > section: > > ?http://pypi.python.org/pypi/zc.buildout#repeatable-buildouts-controlling-eggs-used > > It would be interesting to see if a breadth first strategy would provide better > behavior. I'll take a look at this and see what it happens. But I think we need to rethink the way versioning is handled. -- Alan Franzoni -- contact me at public@[mysurname].eu From mailing at franzoni.eu Fri Dec 17 17:15:44 2010 From: mailing at franzoni.eu (Alan Franzoni) Date: Fri, 17 Dec 2010 17:15:44 +0100 Subject: [Distutils] Buildout or distribute raise unnecessary version conflict? In-Reply-To: References: Message-ID: On Fri, Dec 17, 2010 at 5:01 PM, Benji York wrote: > Fancier version resolution would be nice, however a library-like package > (as opposed to an app) that requires one and only one version of a > dependency is being antisocial. Of course a version range is preferred, but whether it's a library or an app, if I depend on a certain version might be a matter of functionality. I think I've found many python packages and libraries that don't work anymore simply because an upstream dep has changed. I mean, if don't get a version conflict but then the app or lib does not work, it's not a great win, don't you think so? -- Alan Franzoni -- contact me at public@[mysurname].eu From jim at zope.com Fri Dec 17 17:41:58 2010 From: jim at zope.com (Jim Fulton) Date: Fri, 17 Dec 2010 11:41:58 -0500 Subject: [Distutils] Buildout or distribute raise unnecessary version conflict? In-Reply-To: References: Message-ID: On Fri, Dec 17, 2010 at 11:05 AM, Alan Franzoni wrote: > On Fri, Dec 17, 2010 at 4:48 PM, Jim Fulton wrote: >>> Although specifying a dependency >>> without a version shouldn't be a good practice, >> >> Actually, it's best practice. > > I don't think so. It's what people does, removing packages from pypi > it's common, but it's awful. > > As a library mantainer, I can never break an API, not even if I > release 2.0 after working on 1.0, because projects depending on my lib > might just break. Right. So you shouldn't break an API. > > Hence we get "project" "project2" "project3", e.g. versioning through > project naming. Is that a good idea? No. I don't see how that follows. In your original example, you had a distribution that depended on a specific version of a project. That was a very bad idea. In your example above, you have version 1 and version 2. Let's assume that version 2 ads some new API. A client that works with version 1 should work just fine with version 2. So it has no need to specify a version. A client written against version 2 might require at least version 2. In that case it should require a version >= 2. It should not require version 2 (==2). If all clients only require minimum versions, then you'll never get a version conflict. It's when clients specify maximum versions (of which specifying a specific version is a case) that you potentially get into trouble. This might be needed in rare cases if a library makes a breaking change, but that is an exception. > Consider the Java world, where Maven is a de-facto standard. Once a > package with a certain version is uploaded, it's usually never > removed. You can decide to depend on the latest 3.x.x, on the latest > 3.5.x, on version ranges, or on a very specific versions. This makes > it possibile for developers to depend on the versions they really > need. Yeah, and buildout works the same way. > Dependency problems may arise whatsoever and must be usually manually > fixed, but in this case the Python way seems to say "hey, let's > pretend the problem doesn't exist and the API never changes in a > backwards incompatible way". You said above that libraries shouldn't make breaking changes. Now you seem to be saying people should assume they do. I'm not sure what you're trying to say. Of course, libraries do sometimes make breaking changes. Any responsible developer tries to avoid this. A client that assumes a library makes breaking changes should switch to an alternate library if possible. When dealing with breaking changes, it's generally easier to deal with that at the level of application assembly, which is why buildout provides a number of ways to control versions. Being highly restrictive at the distribution level (unless the distribution is, itself an application) is counter productive. >>> it seems quite common >>> - mostly because many people have got full control of the pypi section >>> on their own repo and they can decide what's offered there, so they >>> just specify "mydependency" on their other projects. I think it's >>> quite a wide assumption, by the way. >> >> It's best practice. ?Requiring a specific version is extremely inflexible. >> If everyone did it, you'd never be able to assemble anything of any size. > > You might require a version range - I think I asked something about > version specification, which is quite fuzzy in setuptools/distribute, > some weeks ago. A lower bound is fine. An upper bound is sometimes necessary, but you want to be as nonrestrictive as possible, > BTW if your require the API exposed by a project at version, let's > say, 1.4 (or "at least 1.4 but less than 2.0") it won't do any good > not to specify your version. You're effectively depending on the state > of an external entity which is not under your control. > > Scenario: > You project A today works 100% with a certain version of lib FOO, > which is, let's say, 1.0.0, but you don't explicitly write that > version requirement. > > In a year, you get back to project A, run the tests, and everything > fails, because today's lib FOO 1.2.0 API changed. That happens sometimes. At that point, you should consider firing project FOO. How you approach this depends on whether project A is a top-level application or a library module. If it's an application, then nailing all of the versions is a good idea. This can be done via distribution dependencies or via a buildout. If it's a library, then nailing all of the versions will cause a lot of pain for people trying to use your library. ... >> Maybe system packing tools use better dependency-resolution >> mechanisms. ?It could happen! I have certainly been stymied by >> version conflicts in system packaging systems, so I don't think their >> algorithms are that great. > > Version conflicts of course can arise. If there's no possible > dependency path, nor yum neither apt will be able to do anything. In all the cases where I've run into version conflicts, there have been valid configurations, but the system couldn't find them without me sorting it out manually. >> The way to work around this with buildout is to use a buildout versions >> section: >> >> ?http://pypi.python.org/pypi/zc.buildout#repeatable-buildouts-controlling-eggs-used >> >> It would be interesting to see if a breadth first strategy would provide better >> behavior. > > I'll take a look at this and see what it happens. But I think we need > to rethink the way versioning is handled. This has been discussed at length in the past. The consensus, AFAIK is to be as nonrestrictive as possible in library distributions. Use lower bounds when you depend on a new feature. Use upper bounds if you know you have to, but you probably don't know this at the time you make the distribution. Be really annoyed at the library that makes you do this. Applications with high availability requirements should specify as many versions as possible. There was a proposal floated to routinely set upper bounds to dependency major versions. For example, if you use foo 1.2, you might set an upper bound to foo <= 2dev, but almost no one has done this in practice because it's too much of a pain. There's no question that a better search algorithm would be preferable. It would be awesome if buildout could always find an acceptable set of versions (assuming there is one). It would be awesomer if it could find a best one. :) This could be rather hard. Patches (to pkg_utils or a distribute equivalent) Jim -- Jim Fulton From benji at benjiyork.com Fri Dec 17 17:54:14 2010 From: benji at benjiyork.com (Benji York) Date: Fri, 17 Dec 2010 11:54:14 -0500 Subject: [Distutils] Buildout or distribute raise unnecessary version conflict? In-Reply-To: References: Message-ID: On Fri, Dec 17, 2010 at 11:15 AM, Alan Franzoni wrote: > On Fri, Dec 17, 2010 at 5:01 PM, Benji York wrote: >> Fancier version resolution would be nice, however a library-like package >> (as opposed to an app) that requires one and only one version of a >> dependency is being antisocial. > > Of course a version range is preferred, but whether it's a library or > an app, if I depend on a certain version might be a matter of > functionality. I think I've found many python packages and libraries > that don't work anymore simply because an upstream dep has changed. > > I mean, if don't get a version conflict but then the app or lib does > not work, it's not a great win, don't you think so? Agreed. The opposite is a problem as well: one dependency requires an older version and another needing a newer one conflict even though the newer version would have worked just fine. Unfortunately, the only practical thing to do at the moment is for library authors to set a minimum version when they know their package absolutely will not work with older versions and do their best with testing/documenting other version requirements. Library users then have to manage their versions as best they see fit. Some, like me, are maniacal about maintaining strict control on versions (zc.buildout's "allow-picked-versions = false" is for these people). Others don't like having to manage all that and are happy to let their dependency versions "float" -- of course they then have the go-away-and-come-back-six-months-later-and-your-app-is-broken problem you mentioned earlier. Until something changes, it's a "choose your poison" situation. -- Benji York From mailing at franzoni.eu Fri Dec 17 18:14:05 2010 From: mailing at franzoni.eu (Alan Franzoni) Date: Fri, 17 Dec 2010 18:14:05 +0100 Subject: [Distutils] Buildout or distribute raise unnecessary version conflict? In-Reply-To: References: Message-ID: On Fri, Dec 17, 2010 at 5:41 PM, Jim Fulton wrote: >> As a library mantainer, I can never break an API, not even if I >> release 2.0 after working on 1.0, because projects depending on my lib >> might just break. > > Right. So you shouldn't break an API. > >> >> Hence we get "project" "project2" "project3", e.g. versioning through >> project naming. Is that a good idea? > > No. ?I don't see how that follows. Usually versions have meaning; a typical approach to versioning - python itself works in a very similar way - is: version = MAJOR.MINOR.REVISION in a new REVISION > no functional change should be done, only bugfixes. in a new MINOR -> any backwards incompatible change should be minimized, and there should be a "grace period" , e.g. if something must be removed, if it works fine in MINOR, it should be deprecated in MINOR + 1 and then can be removed in MINOR + 2. This in a new MAJOR -> everything can be changed, APIs can be broken, etc. etc. etc. The only way to currently achieve the MAJOR change effect with current python tools and best practices is to create a new project name. Otherwise old clients might break > In your original example, you had a distribution that depended on a > specific version of a project. That was a very bad idea. Sure it was, and it was just an example in order to show the error to the ML. This is not what I usually want to do. > If all clients only require minimum versions, then you'll never get a > version conflict. ?It's when clients specify maximum versions (of > which specifying a specific version is a case) that you potentially > get into trouble. ?This might be needed in rare cases if a library > makes a breaking change, but that is an exception. It's not so rare as you may expect. I encounter it every day, and that's why I'm starting to think we might not be doing the best thing. An example: a recent change in the minor in zope.testing (3.9 to 3.10) recently broke collective.xmltestreport, since an API changed (a subpackage was removed). I find th >> Consider the Java world, where Maven is a de-facto standard. Once a >> package with a certain version is uploaded, it's usually never >> removed. You can decide to depend on the latest 3.x.x, on the latest >> 3.5.x, on version ranges, or on a very specific versions. This makes >> it possibile for developers to depend on the versions they really >> need. > > Yeah, and buildout works the same way. No. Most mavenized projects specify a dependency, maybe a SNAPSHOT one, but at least the major is specified, and often the minor is specified as well. Peeking through 1.0-SNAPSHOT deps is not uncommon while developing. Also, whenever doing a release, Maven freezes the version numbers the library uses. And it's possibile to add dependency ranges as well, but Maven won't download the latest version whatsoever - it'll try downloading the best suited dependency. Don't think I don't like zc.buildout; I do. I think it's great, and this issue is probably not zc.buildout's fault. But if there's a problem we should start thinking about fixing it before it's too late. > You said above that libraries shouldn't make breaking changes. ?Now > you seem to be saying people should assume they do. ?I'm not sure what > you're trying to say. > > Of course, libraries do sometimes make breaking changes. ?Any > responsible developer tries to avoid this. ?A client that assumes a > library makes breaking changes should switch to an alternate library > if possible. I think that there should a *policy* on breaking changes. Sometimes you must do them - Python 3 is a good example of that. But if I can't rely on the fact the clients will specify a dep at least to my MAJOR then such policies are useless. > Being highly restrictive at the distribution level (unless the > distribution is, itself an application) is counter productive. What does "restrictive" mean in this context? Dependencies are just metadata, after all. I'd like to say "hey, my lib works with FOOLIB from v1.2 till 1.4.3 and BARLIB from 2.0 till 2.2". If you are able to pick a version that satisfies both my requirements and yours, then you're just happy. Otherwise you can resolve deps in whatever way you like, but you shouldn't shout at me saying that my code doesn't work; you're on your own. Inviting people not to specify versions as a "best practice" means "less information is best". Versioning info could just be ignored or skipped, but if you want to use it I've got it. Otherwise I might just need to go crawling for a dependency version to see which one might fit the app or lib I'm trying - not exactly an exciting activity. > A lower bound is fine. An upper bound is sometimes necessary, but you > want to be as nonrestrictive as possible, If there's a policy, this could be done. If there's no policy > That happens sometimes. ?At that point, you should consider firing > project FOO. Or you could specify a auitable version number or range. This usually happens in many Java/Maven projects. > In all the cases where I've run into version conflicts, there have > been valid configurations, but the system couldn't find them without > me sorting it out manually. Yes, that could happen. But I've seen times where it actually works. > This has been discussed at length in the past. ?The consensus, AFAIK > is to be as nonrestrictive as possible in library distributions. Use > lower bounds when you depend on a new feature. ?Use upper bounds if > you know you have to, but you probably don't know this at the time you > make the distribution. ?Be really annoyed at the library that makes > you do this. ?Applications with high availability requirements should > specify as many versions as possible. Unless there's a widely recognized best practice and a clear policy, it's very hard to "be annoyed" with libraries breaking APIs. -- Alan Franzoni -- contact me at public@[mysurname].eu From barry at python.org Fri Dec 17 23:26:47 2010 From: barry at python.org (Barry Warsaw) Date: Fri, 17 Dec 2010 17:26:47 -0500 Subject: [Distutils] 'python setup.py test' in develop mode? Message-ID: <20101217172647.45fdbf57@mission> Something bugs me about virtualenv, distribute, development, and testing. I figure I'm probably doing something wrong and that y'all will be able to set me straight. I'm working on a new tool called pkgme[1]. It uses distribute and recommends using virtualenv for development. After activating the virtualenv you can do 'python setup.py develop' and do convenient in-place editing so that your changes to the source show up immediately in the running environment. So far so good. However, when I want to run the test suite, it does the equivalent of a 'python setup.py build' and then runs the tests out of the build/lib.{platform}-{version} directory. This actually screws with our tests because (at least currently) the tests are looking for shell scripts and other programs that live in our tree, but are not part of the Python packaging, so don't get built or installed by setup.py. When we do filesystem introspection to try to find the scripts to exec them, it will work when running in develop mode, but fail in the tests, since __file__ points into build/lib.* instead of the source tree. So I'm wondering if other folks have hit similar problems and what best practices can be recommended to deal with this problem. I can think of various hacks around the problem, but don't want to reinvent the wheel. Or do you think that 'python setup.py test' should use the in-place tree and this is Just A Bug? Cheers, -Barry [1] https://launchpad.net/pkgme -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From pje at telecommunity.com Sat Dec 18 00:16:52 2010 From: pje at telecommunity.com (P.J. Eby) Date: Fri, 17 Dec 2010 18:16:52 -0500 Subject: [Distutils] 'python setup.py test' in develop mode? In-Reply-To: <20101217172647.45fdbf57@mission> References: <20101217172647.45fdbf57@mission> Message-ID: <20101217231658.73FF43A4076@sparrow.telecommunity.com> At 05:26 PM 12/17/2010 -0500, Barry Warsaw wrote: >Something bugs me about virtualenv, distribute, development, and testing. I >figure I'm probably doing something wrong and that y'all will be able to set >me straight. > >I'm working on a new tool called pkgme[1]. It uses distribute and recommends >using virtualenv for development. After activating the virtualenv you can do >'python setup.py develop' and do convenient in-place editing so that your >changes to the source show up immediately in the running environment. So far >so good. > >However, when I want to run the test suite, it does the equivalent of a >'python setup.py build' and then runs the tests out of the >build/lib.{platform}-{version} directory. Really? That sounds wrong. "python setup.py test" should run the tests against the *source* tree, with in-place extension builds. From marius at pov.lt Sat Dec 18 03:24:34 2010 From: marius at pov.lt (Marius Gedminas) Date: Sat, 18 Dec 2010 04:24:34 +0200 Subject: [Distutils] 'python setup.py test' in develop mode? In-Reply-To: <20101217172647.45fdbf57@mission> References: <20101217172647.45fdbf57@mission> Message-ID: <20101218022434.GB6182@fridge.pov.lt> On Fri, Dec 17, 2010 at 05:26:47PM -0500, Barry Warsaw wrote: > Something bugs me about virtualenv, distribute, development, and testing. I > figure I'm probably doing something wrong and that y'all will be able to set > me straight. I've a set of rules that serve me well. Rule #1: $VCS checkout and 'make test' should run the tests. This means I've a Makefile that creates the virtualenv/bootstraps a buildout, with all my test dependencies, and then runs the actual test runner script (which may be setup.py test, or it may be something different). (Likewise, 'make run' should start a local copy of the application, if this is a runnable application. But that's unrelated to testing.) > Or do you think that 'python setup.py test' should use the in-place tree and > this is Just A Bug? I don't know if it's a bug or not. I always assumed it worked as designed: perhaps trying to verify that your setuptools metadata is correctly specified and no modules are going to be missing when the package is installed? Actually, isn't running the tests from the build/ directory required when you want to support both Python 2.x and 3.x via 2to3, and you want to run the tests with Python 3? (I haven't done that personally, but I think Distribute explicitly supports this.) FWIW I found this tests-against-build-dir behaviour mildly irritating, but mostly irrelevant: I always specify my own test suite loader in setup.py (for various reasons, e.g. doctests), and my test loader ignores build/ and uses the in-place tree. I tend to support 'python setup.py test' as a convenience for other people. I dislike the copious build output it always produces at the beginning, and the time wasted to do the build. This might change when I start paying more attention to Python 3. -- Bumper sticker: No radio - Already stolen. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From arfrever.fta at gmail.com Sat Dec 18 04:49:23 2010 From: arfrever.fta at gmail.com (Arfrever Frehtes Taifersar Arahesis) Date: Sat, 18 Dec 2010 04:49:23 +0100 Subject: [Distutils] 'python setup.py test' in develop mode? In-Reply-To: <20101217231658.73FF43A4076@sparrow.telecommunity.com> References: <20101217172647.45fdbf57@mission> <20101217231658.73FF43A4076@sparrow.telecommunity.com> Message-ID: <201012180449.53487.Arfrever.FTA@gmail.com> 2010-12-18 00:16:52 P.J. Eby napisa?(a): > At 05:26 PM 12/17/2010 -0500, Barry Warsaw wrote: > >Something bugs me about virtualenv, distribute, development, and testing. I > >figure I'm probably doing something wrong and that y'all will be able to set > >me straight. > > > >I'm working on a new tool called pkgme[1]. It uses distribute and recommends > >using virtualenv for development. After activating the virtualenv you can do > >'python setup.py develop' and do convenient in-place editing so that your > >changes to the source show up immediately in the running environment. So far > >so good. > > > >However, when I want to run the test suite, it does the equivalent of a > >'python setup.py build' and then runs the tests out of the > >build/lib.{platform}-{version} directory. > > Really? That sounds wrong. "python setup.py test" should run the > tests against the *source* tree, with in-place extension builds. Modules are sometimes transformed by 2to3 during building (transformed versions are placed in build/lib*). Running tests (with Python 3) against the source tree would cause syntax errors etc. -- Arfrever Frehtes Taifersar Arahesis -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part. URL: From sandro at e-den.it Sat Dec 18 14:25:16 2010 From: sandro at e-den.it (Alessandro Dentella) Date: Sat, 18 Dec 2010 14:25:16 +0100 Subject: [Distutils] demo and setup.py Message-ID: <20101218132516.GA28967@ubuntu> Hi, i'm new to this list. I hope this question is pertinent to the list, if not please address me to a more appropriate one. I'd like to understand what is the preferred way to cope with demo examples/directory while packaging. I have a classic structure sqlkit-N.n/ sqlkit/ setup.py MANIFEST.in demo/ My goal would be to have a setup so that 'python setup.py sdist' would package also the demo, so that 'python setup.py sdist upload' could be all I need to do to publish it. Is this a reasonable goal? I ask this as I wouldn't know where to place the demo in the target installation. Under Linux that I know pretty well a demo would be under /usr/(local/)/share/doc... but where sould it be placed under Windows? Is this simething that should be taken care of by setuptools? Searching in the docs I never found the 'demo' word and google was not very usefull too. Any hint is appreciated. thanks a lot sandro -- Sandro Dentella *:-) http://www.reteisi.org Soluzioni libere per le scuole http://sqlkit.argolinux.org SQLkit home page - PyGTK/python/sqlalchemy From pje at telecommunity.com Sat Dec 18 18:34:35 2010 From: pje at telecommunity.com (P.J. Eby) Date: Sat, 18 Dec 2010 12:34:35 -0500 Subject: [Distutils] 'python setup.py test' in develop mode? In-Reply-To: <201012180449.53487.Arfrever.FTA@gmail.com> References: <20101217172647.45fdbf57@mission> <20101217231658.73FF43A4076@sparrow.telecommunity.com> <201012180449.53487.Arfrever.FTA@gmail.com> Message-ID: <20101218173441.D9F8A3A4116@sparrow.telecommunity.com> At 04:49 AM 12/18/2010 +0100, Arfrever Frehtes Taifersar Arahesis wrote: >2010-12-18 00:16:52 P.J. Eby napisa??(a): > > At 05:26 PM 12/17/2010 -0500, Barry Warsaw wrote: > > >Something bugs me about virtualenv, distribute, development, and > testing. I > > >figure I'm probably doing something wrong and that y'all will be > able to set > > >me straight. > > > > > >I'm working on a new tool called pkgme[1]. It uses distribute > and recommends > > >using virtualenv for development. After activating the > virtualenv you can do > > >'python setup.py develop' and do convenient in-place editing so that your > > >changes to the source show up immediately in the running > environment. So far > > >so good. > > > > > >However, when I want to run the test suite, it does the equivalent of a > > >'python setup.py build' and then runs the tests out of the > > >build/lib.{platform}-{version} directory. > > > > Really? That sounds wrong. "python setup.py test" should run the > > tests against the *source* tree, with in-place extension builds. > >Modules are sometimes transformed by 2to3 during building >(transformed versions >are placed in build/lib*). Running tests (with Python 3) against the >source tree >would cause syntax errors etc. Ah, so this issue is specific to distribute, then, not setuptools. From pje at telecommunity.com Sat Dec 18 18:35:55 2010 From: pje at telecommunity.com (P.J. Eby) Date: Sat, 18 Dec 2010 12:35:55 -0500 Subject: [Distutils] demo and setup.py In-Reply-To: <20101218132516.GA28967@ubuntu> References: <20101218132516.GA28967@ubuntu> Message-ID: <20101218173556.E805F3A4116@sparrow.telecommunity.com> At 02:25 PM 12/18/2010 +0100, Alessandro Dentella wrote: >Hi, > >i'm new to this list. I hope this question is pertinent to the list, if not >please address me to a more appropriate one. > >I'd like to understand what is the preferred way to cope with demo >examples/directory while packaging. > >I have a classic structure > > sqlkit-N.n/ > sqlkit/ > setup.py > MANIFEST.in > demo/ > >My goal would be to have a setup so that 'python setup.py sdist' would >package also the demo, so that 'python setup.py sdist upload' could be all I >need to do to publish it. Is this a reasonable goal? Yes. Just include demo in your MANIFEST.in. > I ask this as I >wouldn't know where to place the demo in the target installation. Well, if you just want to package it in the sdist, you don't need to have a place to install it. If the user wants to run the demo, they can just run it from the unpacked sdist, no? From barry at python.org Sat Dec 18 19:08:18 2010 From: barry at python.org (Barry Warsaw) Date: Sat, 18 Dec 2010 13:08:18 -0500 Subject: [Distutils] 'python setup.py test' in develop mode? In-Reply-To: <201012180449.53487.Arfrever.FTA@gmail.com> References: <20101217172647.45fdbf57@mission> <20101217231658.73FF43A4076@sparrow.telecommunity.com> <201012180449.53487.Arfrever.FTA@gmail.com> Message-ID: <20101218130818.3cf1ccef@mission> On Dec 18, 2010, at 04:49 AM, Arfrever Frehtes Taifersar Arahesis wrote: >2010-12-18 00:16:52 P.J. Eby napisa?(a): >> At 05:26 PM 12/17/2010 -0500, Barry Warsaw wrote: >> >Something bugs me about virtualenv, distribute, development, and testing. I >> >figure I'm probably doing something wrong and that y'all will be able to set >> >me straight. >> > >> >I'm working on a new tool called pkgme[1]. It uses distribute and recommends >> >using virtualenv for development. After activating the virtualenv you can do >> >'python setup.py develop' and do convenient in-place editing so that your >> >changes to the source show up immediately in the running environment. So far >> >so good. >> > >> >However, when I want to run the test suite, it does the equivalent of a >> >'python setup.py build' and then runs the tests out of the >> >build/lib.{platform}-{version} directory. >> >> Really? That sounds wrong. "python setup.py test" should run the >> tests against the *source* tree, with in-place extension builds. > >Modules are sometimes transformed by 2to3 during building (transformed >versions are placed in build/lib*). Running tests (with Python 3) against the >source tree would cause syntax errors etc. Okay, I get that there are build-specific details that could cause problems running the tests out of the source tree. 'setup.py develop' would have similar problems, but that's much more of an isolated environment. By that I mean, it's mostly there for the developer while 'setup.py test' is a generic interface that 3rd parties would use. E.g. many Debian packages run 'setup.py test' as part of the build process (and this against multiple versions of Python, including 2.X and 3.X). Still, it would be nice if there were a 'setup.py test --develop' mode or some such. Usually when I'm working on a package, I don't care about build details, and I'm happy to do the moral equivalent of 'make clean' when I'm doing testing with 2to3 and such. In any case, thanks for pointing out that a 'setup.py test' using in-tree code is not the default case, and wouldn't make sense to be the default. I've worked around the issue in our specific case. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Sat Dec 18 19:13:53 2010 From: barry at python.org (Barry Warsaw) Date: Sat, 18 Dec 2010 13:13:53 -0500 Subject: [Distutils] 'python setup.py test' in develop mode? In-Reply-To: <20101218022434.GB6182@fridge.pov.lt> References: <20101217172647.45fdbf57@mission> <20101218022434.GB6182@fridge.pov.lt> Message-ID: <20101218131353.5ac06a03@mission> On Dec 18, 2010, at 04:24 AM, Marius Gedminas wrote: >On Fri, Dec 17, 2010 at 05:26:47PM -0500, Barry Warsaw wrote: >> Something bugs me about virtualenv, distribute, development, and testing. I >> figure I'm probably doing something wrong and that y'all will be able to set >> me straight. > >I've a set of rules that serve me well. > >Rule #1: $VCS checkout and 'make test' should run the tests. This means >I've a Makefile that creates the virtualenv/bootstraps a buildout, with >all my test dependencies, and then runs the actual test runner script >(which may be setup.py test, or it may be something different). > >(Likewise, 'make run' should start a local copy of the application, if >this is a runnable application. But that's unrelated to testing.) I think these are fine, but certainly with tests, I'd like 'make test' to be a thin wrapper around 'setup.py test' *in the general case*. Or IOW, I'd like to see 'setup.py test' be an established standard convention for Python packages. Not required of course, but I do think most simple packages can hook into it very easily, and should! >FWIW I found this tests-against-build-dir behaviour mildly irritating, >but mostly irrelevant: I always specify my own test suite loader in >setup.py (for various reasons, e.g. doctests), and my test loader >ignores build/ and uses the in-place tree. I think the fundamental problem in our case is that these extra artifacts live in-tree but are not in Python package directories. They also aren't installed, so we probably need to add rules to get these installed properly, and then the auto-discovery would be able to find them without caring about the build directories. For now, I do some path hacking to get it working, but it's an ugly solution. >I tend to support 'python setup.py test' as a convenience for other >people. I dislike the copious build output it always produces at the >beginning, and the time wasted to do the build. This might change when >I start paying more attention to Python 3. A reduced output command would be useful, but I do like the interface a lot. 'python setup.py test' is nice and consistent, so IMO should be promoted as a best practice. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From email at lnowak.com Mon Dec 20 12:02:14 2010 From: email at lnowak.com (=?UTF-8?Q?=C5=81ukasz_Nowak?=) Date: Mon, 20 Dec 2010 12:02:14 +0100 Subject: [Distutils] buildout download utility and failed checksums in cache Message-ID: Hello, Buildout download utility allows to provide md5sum to check if downloaded (or available file from cache) is correct. In case of using download-cache and having wrong file in this cache it is required to manually remove this file and redownload. Is it possible to have redownload "request" inside of download utility? In case if file exists in cache, but passed checksum does not match, download utility cloud redownload this file, re check sum and eventually succeed. Is it ok to have such change of buildout download cache behaviour? Regards, Luke -- ?ukasz Nowak IT Specialist email at lnowak.com http://lnowak.com/ Skype: Shufla jid: shufla at jabster.pl gg: 1157726 ``Use the Source, Luke...'' From jim at zope.com Mon Dec 20 16:24:45 2010 From: jim at zope.com (Jim Fulton) Date: Mon, 20 Dec 2010 10:24:45 -0500 Subject: [Distutils] buildout download utility and failed checksums in cache In-Reply-To: References: Message-ID: 2010/12/20 ?ukasz Nowak : > Hello, > > Buildout download utility allows to provide md5sum to check if > downloaded (or available file from cache) is correct. > > In case of using download-cache and having wrong file in this cache it > is required to manually remove this file and redownload. > > Is it possible to have redownload "request" inside of download utility? > > In case if file exists in cache, but passed checksum does not match, > download utility cloud redownload this file, re check sum and > eventually succeed. > > Is it ok to have such change of buildout download cache behaviour? Sound like a good idea to me. I suggest reporting a bug: https://bugs.launchpad.net/zc.buildout/+bugs Jim -- Jim Fulton From email at lnowak.com Mon Dec 20 16:46:12 2010 From: email at lnowak.com (=?UTF-8?Q?=C5=81ukasz_Nowak?=) Date: Mon, 20 Dec 2010 16:46:12 +0100 Subject: [Distutils] buildout download utility and failed checksums in cache In-Reply-To: References: Message-ID: Hello, W dniu 20 grudnia 2010 16:24 u?ytkownik Jim Fulton napisa?: > 2010/12/20 ?ukasz Nowak : (...) >> Is it ok to have such change of buildout download cache behaviour? > > Sound like a good idea to me. Thank you for confirmation. > I suggest reporting a bug: > > ?https://bugs.launchpad.net/zc.buildout/+bugs Done: https://bugs.launchpad.net/zc.buildout/+bug/692600 Merry Christmas and Happy New Year, Luke From mailing at franzoni.eu Mon Dec 20 17:37:28 2010 From: mailing at franzoni.eu (Alan Franzoni) Date: Mon, 20 Dec 2010 17:37:28 +0100 Subject: [Distutils] Buildout or distribute raise unnecessary version conflict? In-Reply-To: References: Message-ID: On Fri, Dec 17, 2010 at 5:54 PM, Benji York wrote: > Agreed. ?The opposite is a problem as well: one dependency requires an > older version and another needing a newer one conflict even though the > newer version would have worked just fine. That's the case where an external override is all right. But again, the fact that I can't know which version is possibly working for a dependency is just crazy. > Until something changes, it's a "choose your poison" situation. I think it's high time for a change. My experience at dependency management, both for systemwide package managers - dpkg/rpm - and Java software (maven-ivy-etc) says we're currently doing it wrong in Python. I'll do a research in other environments - ruby/gems, perl/cpan, Haskell - and I hope we can work to change this. -- Alan Franzoni -- contact me at public@[mysurname].eu From sridharr at activestate.com Tue Dec 21 22:41:55 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Tue, 21 Dec 2010 13:41:55 -0800 Subject: [Distutils] PEP 345: real-world examples of "virtual" projects (Provides-Release) Message-ID: <4D111F23.2040404@activestate.com> From https://bitbucket.org/ametaireau/python-peps/src/tip/pep-0345.txt """ Provides-Release (multiple use) ::::::::::::::::::::::::::::::: [...] A release may also provide a "virtual" project name, which does not correspond to any separately-distributed project: such a name might be used to indicate an abstract capability which could be supplied by one of multiple projects. E.g., multiple projects might supply RDBMS bindings for use by a given ORM: each project might declare that it provides ``ORM-bindings``, allowing other projects to depend only on having at most one of them installed. """ 1. I find this to be a strange example. If only one provided project (ORM-bindings) can be installed at any time, what if anyone wants to install more than one DB backends for ORM? 2. What is the expected behavior of 'pip install ORM-bindings'[1]? To pick one of the many projects "providing" ORM-bindings? 3. Did anyone--Alexis and Tarek, in particular--think of real-world use cases for virtual projects (and even "provides" in general) other than the Zope transaction case? If yes, what are they? 4. Personally, I have needs for "virtual" packages from a binary (not source) distribution perspective. For example, "MySQL-python" can be a virtual package "provided by" the binary distributions: mysql5.1-python, mysql5.0-python; similarly, "psycopg2" can be provided by psycopg2-pg8.4 and psycopg2-pg9.0. Or, "modwsgi" can be provided by modwsgi-apache2.2, modwsgi-apache2.4. How would PEP 345's "Provides-Release" help, if at all, in describing this scenario? -srid [1] Or substitute `pip` with any package installer (easy_install, distutils2.install, pypm, enstaller). From marius at pov.lt Wed Dec 22 02:51:14 2010 From: marius at pov.lt (Marius Gedminas) Date: Wed, 22 Dec 2010 03:51:14 +0200 Subject: [Distutils] PEP 345: real-world examples of "virtual" projects (Provides-Release) In-Reply-To: <4D111F23.2040404@activestate.com> References: <4D111F23.2040404@activestate.com> Message-ID: <20101222015114.GB7565@fridge.pov.lt> On Tue, Dec 21, 2010 at 01:41:55PM -0800, Sridhar Ratnakumar wrote: > From https://bitbucket.org/ametaireau/python-peps/src/tip/pep-0345.txt > > """ > Provides-Release (multiple use) > ::::::::::::::::::::::::::::::: > [...] > A release may also provide a "virtual" project name, which does > not correspond to any separately-distributed project: such a name > might be used to indicate an abstract capability which could be supplied > by one of multiple projects. E.g., multiple projects might supply > RDBMS bindings for use by a given ORM: each project might declare > that it provides ``ORM-bindings``, allowing other projects to depend > only on having at most one of them installed. > """ Perhaps it was supposed to say "depend only on having at *least* one of them installed"? That would be how Debian's virtual packages work, for example. Marius Gedminas -- Programs that write programs are the happiest programs in the world. -- Andrew Hume -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From setuptools at bugs.python.org Thu Dec 23 06:39:52 2010 From: setuptools at bugs.python.org (hm) Date: Thu, 23 Dec 2010 05:39:52 +0000 Subject: [Distutils] [issue119] setuptools-0.6c11.win32-py2.5 python path recognition problem In-Reply-To: <1293082792.64.0.338241583128.issue119@psf.upfronthosting.co.za> Message-ID: <1293082792.64.0.338241583128.issue119@psf.upfronthosting.co.za> New submission from hm : I have downloaded setuptools-0.6c11.win32-py2.5.exe and tried to install it on Windows 7 (64 bit). I have an existing Python installation. The setuptool recognizes the path, however, puts a quote around the path. The problem is that it also carries the quotes around the Python path in the installation directory and makes a path like: "C:\Python25\"\Lib\site-packages\ This is an invalid path and in the next step the installer complains that cannot install the files in the provided path. The setuptool also does not allow editing this path. Any help is appreciated, asap. Thank Hamid ---------- messages: 569 nosy: hm priority: bug status: unread title: setuptools-0.6c11.win32-py2.5 python path recognition problem _______________________________________________ Setuptools tracker _______________________________________________ From pje at telecommunity.com Thu Dec 23 16:39:50 2010 From: pje at telecommunity.com (P.J. Eby) Date: Thu, 23 Dec 2010 10:39:50 -0500 Subject: [Distutils] [issue119] setuptools-0.6c11.win32-py2.5 python path recognition problem In-Reply-To: <1293082792.64.0.338241583128.issue119@psf.upfronthosting.c o.za> References: <1293082792.64.0.338241583128.issue119@psf.upfronthosting.co.za> <1293082792.64.0.338241583128.issue119@psf.upfronthosting.co.za> Message-ID: <20101223154004.CFCDD3A411A@sparrow.telecommunity.com> At 05:39 AM 12/23/2010 +0000, hm wrote: >I have downloaded setuptools-0.6c11.win32-py2.5.exe and tried to >install it on Windows 7 (64 bit). I have an existing Python >installation. The setuptool recognizes the path, however, puts a >quote around the path. The problem is that it also carries the >quotes around the Python path in the installation directory and >makes a path like: > >"C:\Python25\"\Lib\site-packages\ > >This is an invalid path and in the next step the installer complains >that cannot install the files in the provided path. The setuptool >also does not allow editing this path. Any help is appreciated, asap. This sounds like a problem with the distutils installer for Windows. Have you tried installing any other Python .exe's from PyPI, and do they have the same problem? From sridharr at activestate.com Thu Dec 23 19:13:14 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Thu, 23 Dec 2010 10:13:14 -0800 Subject: [Distutils] PEP 345: real-world examples of "virtual" projects (Provides-Release) In-Reply-To: <20101222015114.GB7565@fridge.pov.lt> References: <4D111F23.2040404@activestate.com> <20101222015114.GB7565@fridge.pov.lt> Message-ID: <4D13913A.9010401@activestate.com> Marius Gedminas wrote: > On Tue, Dec 21, 2010 at 01:41:55PM -0800, Sridhar Ratnakumar wrote: >> > Fromhttps://bitbucket.org/ametaireau/python-peps/src/tip/pep-0345.txt >> > >> > """ >> > Provides-Release (multiple use) >> > ::::::::::::::::::::::::::::::: >> > [...] >> > A release may also provide a "virtual" project name, which does >> > not correspond to any separately-distributed project: such a name >> > might be used to indicate an abstract capability which could be supplied >> > by one of multiple projects. E.g., multiple projects might supply >> > RDBMS bindings for use by a given ORM: each project might declare >> > that it provides ``ORM-bindings``, allowing other projects to depend >> > only on having at most one of them installed. >> > """ > > Perhaps it was supposed to say "depend only on having at*least* one of > them installed"? > That would be how Debian's virtual packages work, for example. From http://www.debian.org/doc/FAQ/ch-pkg_basics.en.html#s-virtual "A [Debian] virtual package is a generic name that applies to any one of a group of packages, all of which provide similar basic functionality. For example, both the tin and trn programs are news readers, and should therefore satisfy any dependency of a program that required a news reader on a system, in order to work or to be useful. They are therefore both said to provide the "virtual package" called news-reader." Is there a real-world example (in Python packaging) requiring such a constraint? ... To me, only "at most one of them installed" makes sense ... and *only* for binary distributions (except 'setuptools' being provided by Distribute), viz. [quote] 4. Personally, I have needs for "virtual" packages from a binary (not source) distribution perspective. For example, "MySQL-python" can be a virtual package "provided by" the binary distributions: mysql5.1-python, mysql5.0-python; similarly, "psycopg2" can be provided by psycopg2-pg8.4 and psycopg2-pg9.0. Or, "modwsgi" can be provided by modwsgi-apache2.2, modwsgi-apache2.4. How would PEP 345's "Provides-Release" help, if at all, in describing this scenario? [endquote] Just to provide a context, I'd like to add support for virtual packages in PyPM[1] to facilitate providing such binary variants as above for some Python packages. And I'd like to understand and stick to the standards[2] as much as possible (instead of designing a proprietary metadata extension). ... This is how I understand the PEP: $ cat psycopg2-pg9.0/DIST-INFO # binary [...] Version: 2.3.1 Provides-Release: psycopg2 $ cat psycopg2/DIST-INFO # source [...] Version: 2.2.9 Provides-Release: psycopg2 $ cat pgbrowser/DIST-INFO # source Requires-Release: psycopg2 (>=2.3) This is what I asked in regards to installing such a virtual package: [quote] 2. What is the expected behavior of 'pip install ORM-bindings'? To pick *one* of the many projects "providing" ORM-bindings? [endquote] (emphasis added) How is 'pip install pgbrowser'[3] expected resolve the `psycopg2' dependency (it may not know yet that it is a virtual package)? I have an idea as to how to do this in binary package repositories (such as those of PyPM): 1. Index the `Provides-Release` field of all packages 2. If `Provides-Release` is missing, use the name of the distribution itself 3. Resolve dependencies (and requests from user) by looking at this `Provides` index, not the package *name* index. For PyPI - step 3. will require, at the least, Simple Index to expose "provides" in URLs, eg: pypi.python.org/simple/@P/psycopg2 pypi.python.org/simple/@P/psycopg2/psycopg2-pg9.0-2.3.1.win32.egg pypi.python.org/simple/@P/psycopg2/psycopg2-2.2.9.tar.gz So `pip install pgbrowser` will have to resolve the dependency of psycopg2 by looking at /simple/@P/psycopg2, but *not* /simple/psycopg2. Further, how would pip know that "psycopg2-pg9.0-2.3.1.win32.egg" is indeed the appropriate platform binary? The `Supported-Platform` field would be useless to package installers as the PEP does not specify its semantics (I wonder why it exists in first place). Did I misunderstand anything? Has anyone thought of all this in regards to PEP 345? -srid [1] http://code.activestate.com/pypm/ [2] I just incorporated "environment markers" support. [3] I use `pip` only as an example; substitute it with any package installer From cool-rr at cool-rr.com Thu Dec 23 20:00:05 2010 From: cool-rr at cool-rr.com (cool-RR) Date: Thu, 23 Dec 2010 21:00:05 +0200 Subject: [Distutils] The new MSI given by Python 2.7 Message-ID: I'm making a release of my project today, and I noticed that the MSI created by Python 2.7 when doing `bdist_msi` does not seem to be attached to a specific Python version. Is it safe to supply it for users of Python 2.5-2.7? (My package is pure-Python.) When uploading to PyPI, I'm forced to choose a single Python version for the MSI. What do I do? I don't want people to be mislead to think that it's an MSI only for 2.6 when it works for 2.5-2.7. I tried uploading the same file for all three versions, but PyPI didn't allow this, telling me I already uploaded that file. Please help, I want to make a release today. Thanks, Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Thu Dec 23 20:25:31 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 23 Dec 2010 20:25:31 +0100 Subject: [Distutils] The new MSI given by Python 2.7 In-Reply-To: References: Message-ID: <4D13A22B.2040002@v.loewis.de> Am 23.12.2010 20:00, schrieb cool-RR: > I'm making a release of my project today, and I noticed that the MSI > created by Python 2.7 when doing `bdist_msi` does not seem to be > attached to a specific Python version. > > Is it safe to supply it for users of Python 2.5-2.7? (My package is > pure-Python.) Yes, that's the intention. The feature has seen little testing so far, so feedback is welcome. > When uploading to PyPI, I'm forced to choose a single Python version for > the MSI. How so? Because it says that it's needed, or because you get an error message when you specify "any"? If so, what error message exactly? Regards, Martin From cool-rr at cool-rr.com Thu Dec 23 20:29:39 2010 From: cool-rr at cool-rr.com (cool-RR) Date: Thu, 23 Dec 2010 21:29:39 +0200 Subject: [Distutils] The new MSI given by Python 2.7 In-Reply-To: <4D13A22B.2040002@v.loewis.de> References: <4D13A22B.2040002@v.loewis.de> Message-ID: On Thu, Dec 23, 2010 at 9:25 PM, "Martin v. L?wis" wrote: > Am 23.12.2010 20:00, schrieb cool-RR: > > I'm making a release of my project today, and I noticed that the MSI > > created by Python 2.7 when doing `bdist_msi` does not seem to be > > attached to a specific Python version. > > > > Is it safe to supply it for users of Python 2.5-2.7? (My package is > > pure-Python.) > > Yes, that's the intention. The feature has seen little testing so far, > so feedback is welcome. > > > When uploading to PyPI, I'm forced to choose a single Python version for > > the MSI. > > How so? Because it says that it's needed, or because you get an error > message when you specify "any"? If so, what error message exactly? > > Regards, > Martin > I tried submitting without a Python version at all. and it gave an error. Now I did "any Python version," which worked, and I noted in the comments that it's intended for 2.5-2.7. So I guess I'm okay. Thanks for your help, Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From marius at pov.lt Fri Dec 24 01:49:40 2010 From: marius at pov.lt (Marius Gedminas) Date: Fri, 24 Dec 2010 02:49:40 +0200 Subject: [Distutils] The new MSI given by Python 2.7 In-Reply-To: References: <4D13A22B.2040002@v.loewis.de> Message-ID: <20101224004940.GA22979@fridge.pov.lt> On Thu, Dec 23, 2010 at 09:29:39PM +0200, cool-RR wrote: > On Thu, Dec 23, 2010 at 9:25 PM, "Martin v. L?wis" wrote: > > Am 23.12.2010 20:00, schrieb cool-RR: > > > I'm making a release of my project today, and I noticed that the MSI > > > created by Python 2.7 when doing `bdist_msi` does not seem to be > > > attached to a specific Python version. > > > > > > Is it safe to supply it for users of Python 2.5-2.7? (My package is > > > pure-Python.) > > > > Yes, that's the intention. The feature has seen little testing so far, > > so feedback is welcome. ... > Now I did "any Python version," which worked, and I noted in the comments > that it's intended for 2.5-2.7. So I guess I'm okay. You could add Programming Language :: Python :: 2.5 Programming Language :: Python :: 2.6 Programming Language :: Python :: 2.7 to your classifiers. (And please also upload an sdist, in case you haven't done so already.) Marius Gedminas -- Mosher's Law of Software Engineering: Don't worry if it doesn't work right. If everything did, you'd be out of a job. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From cool-rr at cool-rr.com Fri Dec 24 11:57:38 2010 From: cool-rr at cool-rr.com (cool-RR) Date: Fri, 24 Dec 2010 12:57:38 +0200 Subject: [Distutils] The new MSI given by Python 2.7 In-Reply-To: <20101224004940.GA22979@fridge.pov.lt> References: <4D13A22B.2040002@v.loewis.de> <20101224004940.GA22979@fridge.pov.lt> Message-ID: On Fri, Dec 24, 2010 at 2:49 AM, Marius Gedminas wrote: > On Thu, Dec 23, 2010 at 09:29:39PM +0200, cool-RR wrote: > > On Thu, Dec 23, 2010 at 9:25 PM, "Martin v. L?wis" >wrote: > > > Am 23.12.2010 20:00, schrieb cool-RR: > > > > I'm making a release of my project today, and I noticed that the MSI > > > > created by Python 2.7 when doing `bdist_msi` does not seem to be > > > > attached to a specific Python version. > > > > > > > > Is it safe to supply it for users of Python 2.5-2.7? (My package is > > > > pure-Python.) > > > > > > Yes, that's the intention. The feature has seen little testing so far, > > > so feedback is welcome. > ... > > Now I did "any Python version," which worked, and I noted in the comments > > that it's intended for 2.5-2.7. So I guess I'm okay. > > You could add > > Programming Language :: Python :: 2.5 > Programming Language :: Python :: 2.6 > Programming Language :: Python :: 2.7 > > to your classifiers. (And please also upload an sdist, in case you > haven't done so already.) > > Marius Gedminas > I did both these things, thanks. Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexis at notmyidea.org Fri Dec 24 12:09:33 2010 From: alexis at notmyidea.org (=?ISO-8859-1?Q?Alexis_M=E9taireau?=) Date: Fri, 24 Dec 2010 12:09:33 +0100 Subject: [Distutils] PEP 345: real-world examples of "virtual" projects (Provides-Release) In-Reply-To: <4D111F23.2040404@activestate.com> References: <4D111F23.2040404@activestate.com> Message-ID: <4D147F6D.2040605@notmyidea.org> Hi Srid, On 12/21/2010 09:41 PM, Sridhar Ratnakumar wrote: > 1. I find this to be a strange example. If only one provided project > (ORM-bindings) can be installed at any time, what if anyone wants to > install more than one DB backends for ORM? Supposing I've understood that part the right way, it's not preventing to install more than one backend for an ORM. It provides a way to have a dependency to a project providing an ORM-binding. ORM bindings are different from ORM backends in my opinion. Thus, if you do have a better example to propose, I will be very happy to discuss it here, I find the DB-related example a bit odd too (It's probably not very likely to have something like that in a real use case). > 2. What is the expected behavior of 'pip install ORM-bindings'[1]? To > pick one of the many projects "providing" ORM-bindings? That's what I have in mind, yes. But until we got a determinist algorithm, we don't have any way to select the same one on different installer systems. That kind of things should be provided by distutils2, but we did not thought about that until now. > 3. Did anyone--Alexis and Tarek, in particular--think of real-world use > cases for virtual projects (and even "provides" in general) other than > the Zope transaction case? If yes, what are they? "Provides-Release" lists the specific projects provided with this distribution. One case I can think about is having a distribution providing two different projects. A software can have a project for the "core" features and a project for the "default-plugins". It can then provides the two projects in only one distribution. Those can be two virtual projects 'core' and 'defaultplugins', which can be *provided* by different projects then. This allows to choose between the possible projects when resolving the dependencies. setuptools and distribute can be both provided by the same virtual project. > 4. Personally, I have needs for "virtual" packages from a binary (not > source) distribution perspective. For example, "MySQL-python" can be a > virtual package "provided by" the binary distributions: mysql5.1-python, > mysql5.0-python; Yep, I understand what you mean. I don't think the Provides-Release is adapted to deal with those. 'MySQL-python' is *not* a virtual project, but it is a real one. In my opinion, the 'Provides-Release' field and virtual project concept are not to resolve this kind of problems: they are useful to deal with different *projects* (Distribute and Setuptools) providing the same features (packaging facilities). > similarly, "psycopg2" can be provided by psycopg2-pg8.4 > and psycopg2-pg9.0. Or, "modwsgi" can be provided by modwsgi-apache2.2, > modwsgi-apache2.4. Each time, those use cases are related to version distributions, or to compiler options. If you are dealing with such things, I don't think the python package index and the current fields (PEP 345) will be useful to you. I'm not even sure that we need to have some fields to support that at this level. From what I understand, you have a "psycopg2" version for each of the postgre distributions. How are we supposed to deal with that on PyPI ? Arent those dependencies at the OS level rather than the python level ? > How would PEP 345's "Provides-Release" help, if at > all, in describing this scenario? I'm not sure it will help in this case, unfortunately. Hope this helps, Alex From rich at noir.com Wed Dec 29 20:42:38 2010 From: rich at noir.com (K. Richard Pixley) Date: Wed, 29 Dec 2010 11:42:38 -0800 Subject: [Distutils] Distribute comments Message-ID: <4D1B8F2E.5080506@noir.com> I've just read through the Distribute doc for the first time and I have a few comments. First, I just want to say that the state of python packaging is a sad morass. It's not easy for someone to sort out distutils/setuptools/distribute to figure out how to get a package built and released on pypi. Distutils is official and available, but the documentation is only slightly relevant with much time and space given to features that aren't clear, aren't relevant, or don't solve today's problems. Setuptools is messy, confusing, ill documented, and difficult to use. The Distribute documentation helps in this considerably but it could be better. For instance... I wish that the Distribute documentation didn't refer to itself as "setuptools". This is confusing. From where I'm sitting, Distribute is a fork. It may have started life once upon a time in a distant past that I don't care about as a fork of setuptools but as of today, it's a separate package which just happens to provide a superset of the setuptools features along with a setuptools compatible replacement interface. To say that one can use a Distribute script to install "setuptools" is a misnomer. It suggests that the original setuptools is being installed rather than installing Distribute, (which just happens to provide functional replacements for setuptools). That's not what I want. I want Distribute and Distribute alone. I'm willing to go to some effort to make sure that people who use my package never need to know about, read about, or even think about setuptools. Toward that end, I wish that the documentation would explain how to import Distribute specifically, in a non-setuptools compatible way. I don't need or want a setuptools compatible replacement. I have a new package and I'm completely willing to make my package wholly dependent on Distribute rather than setuptools. Even attempting to support setuptools at this point in history seems like a mistake to me. If I'm going to include distribute_setup.py in my package, then it seems to me that I'm already committed to Distribute, not setuptools. Leaving the illusion that an installer might be able to make setuptools work for my package seems misguided. I'd like to eliminate that thought. The section on "What Your Users Should Know" sounds like the sort of information which has traditionally been released in an INSTALL file with GNU software. Is there a reusable, sample template which explains this information in a package agnostic sort of way that I can simply include in my package? Thanks for reading. --rich -------------- next part -------------- An HTML attachment was scrubbed... URL: