From barry at python.org Mon Nov 1 20:15:15 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 1 Nov 2010 15:15:15 -0400 Subject: [Distutils] An observation on how system packagers and developers can be friends In-Reply-To: <20101030135919.GA31215@fridge.pov.lt> References: <20101028165035.7a3faa3a@mission> <20101030135919.GA31215@fridge.pov.lt> Message-ID: <20101101151515.0588420d@snowdog> On Oct 30, 2010, at 04:59 PM, Marius Gedminas wrote: >Why is it a build-dep, anyway? Python's documentation sources are now >ReStructuredText, not LaTeX -- since 2.6, I think. I don't believe >there are PDFs shipped in python2.x-doc packages. Is it an obsolete >build dependency from earlier times? Apparently so. I just took a look at the python2.6 source package for Ubuntu (likely the same for Debian), and there is no build-dep on latex. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From roban.kramer at phys.ethz.ch Wed Nov 3 09:32:22 2010 From: roban.kramer at phys.ethz.ch (Roban Hultman Kramer) Date: Wed, 3 Nov 2010 09:32:22 +0100 Subject: [Distutils] package data not installed from python .egg file Message-ID: I'm trying to include some data files in a python package using the setuptools package_data option. I'm then accessing the files with pkg_resources. This works perfectly when the python .egg file is installed as-is (i.e. still zipped). But when the egg file is unzipped during installation, the data files are not installed. In other words, if I run: python setup.py bdist_egg cd dist sudo easy_install -z EnrichPy-0.1.001-py2.6.egg then the egg file is installed (with the data safely zipped inside) and everything works. On the other hand, if I run sudo easy_install -Z EnrichPy-0.1.001-py2.6.egg then the data files are not installed. I have a directory called EnrichPy-0.1.001-py2.6.egg/enrichpy/ under dist-packages, but it contains only my source files, not my data files. Can anyone suggest what I need to do to get the package_data files to be installed when easy_install unzips the egg file? Notes: The package is available at http://github.com/roban/EnrichPy I can test it by running: import enrichpy.yields enrichpy.yields.Data_vdHG().data If that exits without errors, then pkg_resources is finding the installed data. I asked this question on stackoverflow without response: http://stackoverflow.com/questions/4043315/package-data-not-installed-from-python-egg-file -- Roban Hultman Kramer Zwicky Fellow Institute for Astronomy ETH Z?rich (Swiss Federal Institute of Technology) From techtonik at gmail.com Wed Nov 3 10:47:15 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 3 Nov 2010 11:47:15 +0200 Subject: [Distutils] Security issue with Distutils register is still actual Message-ID: Hello, Does anybody care that PyPI password are stored in a well-known location in cleartext and developers are forced to store them when they submit packages for review? http://bugs.python.org/issue9995 -- anatoly t. From ziade.tarek at gmail.com Wed Nov 3 12:35:04 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 3 Nov 2010 12:35:04 +0100 Subject: [Distutils] Security issue with Distutils register is still actual In-Reply-To: References: Message-ID: On Wed, Nov 3, 2010 at 10:47 AM, anatoly techtonik wrote: > Hello, > > Does anybody care that PyPI password are stored in a well-known > location in cleartext and developers are forced to store them when > they submit packages for review? > http://bugs.python.org/issue9995 > We have hundreds of bugs to fix for distutils. If you propose a patch + test, things will speed up. There are already tests for various register/upload scenarii, so it should not be hard to copy-paste one to create your test > -- > anatoly t. > _______________________________________________ > Distutils-SIG maillist ?- ?Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -- Tarek Ziad? | http://ziade.org From jim at zope.com Wed Nov 3 14:58:48 2010 From: jim at zope.com (Jim Fulton) Date: Wed, 3 Nov 2010 09:58:48 -0400 Subject: [Distutils] Security issue with Distutils register is still actual In-Reply-To: References: Message-ID: On Wed, Nov 3, 2010 at 7:35 AM, Tarek Ziad? wrote: > On Wed, Nov 3, 2010 at 10:47 AM, anatoly techtonik wrote: >> Hello, >> >> Does anybody care that PyPI password are stored in a well-known >> location in cleartext and developers are forced to store them when >> they submit packages for review? >> http://bugs.python.org/issue9995 >> > > We have hundreds of bugs to fix for distutils. If you propose a patch > + test, things will speed up. > There are already tests for various register/upload scenarii, so it > should not be hard to copy-paste one to create your test While that's usually a reasonable response, this isn't a bug. This is a case where we need to come up with a better way of doing things. Someone needs to propose something and folks need to weigh in. Jim From jim at zope.com Wed Nov 3 15:03:41 2010 From: jim at zope.com (Jim Fulton) Date: Wed, 3 Nov 2010 10:03:41 -0400 Subject: [Distutils] Security issue with Distutils register is still actual In-Reply-To: References: Message-ID: On Wed, Nov 3, 2010 at 9:58 AM, Jim Fulton wrote: > On Wed, Nov 3, 2010 at 7:35 AM, Tarek Ziad? wrote: >> On Wed, Nov 3, 2010 at 10:47 AM, anatoly techtonik wrote: >>> Hello, >>> >>> Does anybody care that PyPI password are stored in a well-known >>> location in cleartext and developers are forced to store them when >>> they submit packages for review? >>> http://bugs.python.org/issue9995 >>> >> >> We have hundreds of bugs to fix for distutils. If you propose a patch >> + test, things will speed up. >> There are already tests for various register/upload scenarii, so it >> should not be hard to copy-paste one to create your test > > While that's usually a reasonable response, this isn't a bug. I should have looked more carefully at the issue. The refusal to use a password without storing it *is* a fairly narrow bug. > This is a case where we need to come up with a better way of doing things. > Someone needs to propose something and folks need to weigh in. I would love to see a solution to the broader problem. I really don't want to have to enter a password every time I upload a package. I guess a good solution would be to integrate with existing password-management tools. This could be prototyped as an a separate upload tool. JIm -- Jim Fulton From ziade.tarek at gmail.com Wed Nov 3 15:07:47 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 3 Nov 2010 15:07:47 +0100 Subject: [Distutils] Security issue with Distutils register is still actual In-Reply-To: References: Message-ID: On Wed, Nov 3, 2010 at 3:03 PM, Jim Fulton wrote: > On Wed, Nov 3, 2010 at 9:58 AM, Jim Fulton wrote: >> On Wed, Nov 3, 2010 at 7:35 AM, Tarek Ziad? wrote: >>> On Wed, Nov 3, 2010 at 10:47 AM, anatoly techtonik wrote: >>>> Hello, >>>> >>>> Does anybody care that PyPI password are stored in a well-known >>>> location in cleartext and developers are forced to store them when >>>> they submit packages for review? >>>> http://bugs.python.org/issue9995 >>>> >>> >>> We have hundreds of bugs to fix for distutils. If you propose a patch >>> + test, things will speed up. >>> There are already tests for various register/upload scenarii, so it >>> should not be hard to copy-paste one to create your test >> >> While that's usually a reasonable response, this isn't a bug. > > I should have looked more carefully at the issue. The refusal to > use a password without storing it *is* a fairly narrow bug. Yes this is a bug. the password should be reused by upload. There's code for this but it seems to fails >> This is a case where we need to come up with a better way of doing things. >> Someone needs to propose something and folks need to weigh in. > > I would love to see a solution to the broader problem. > > I really don't want to have to enter a password every time I > upload a package. me neither :) > I guess a good solution would be to integrate with existing > password-management tools. This could be prototyped as an > a separate upload tool. I have mentored a project in GSOC last year exactly for this case: keyring (avialable at PyPI) It is already successfully used in Mercurial (mercurial-keyring) that suffers the same problem when doing http/https The next step was to integrate keyring in distutils/upload but was not done yet due to a lack of time. Tarek -- Tarek Ziad? | http://ziade.org From techtonik at gmail.com Wed Nov 3 20:56:56 2010 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 3 Nov 2010 21:56:56 +0200 Subject: [Distutils] Security issue with Distutils register is still actual In-Reply-To: References: Message-ID: On Wed, Nov 3, 2010 at 4:07 PM, Tarek Ziad? wrote: >> I should have looked more carefully at the issue. The refusal to >> use a password without storing it *is* a fairly narrow bug. > > Yes this is a bug. the password should be reused by upload. There's > code for this but it seems to fails Fix landed. http://bugs.python.org/issue9995 >>> This is a case where we need to come up with a better way of doing things. >>> Someone needs to propose something and folks need to weigh in. >> >> I would love to see a solution to the broader problem. >> >> I really don't want to have to enter a password every time I >> upload a package. > > me neither :) Does anybody know where is documentation on supported authentication in PyPI? >> I guess a good solution would be to integrate with existing >> password-management tools. This could be prototyped as an >> a separate upload tool. > > I have mentored a project in GSOC last year exactly for this case: > keyring (avialable at PyPI) > > It is already successfully used in Mercurial (mercurial-keyring) that > suffers the same problem when doing http/https > > The next step was to integrate keyring in distutils/upload but was not > done yet due to a lack of time. Network protection is still weak. The password is sent nearly in cleartext. -- anatoly t. From jim at zope.com Wed Nov 3 21:04:17 2010 From: jim at zope.com (Jim Fulton) Date: Wed, 3 Nov 2010 16:04:17 -0400 Subject: [Distutils] Security issue with Distutils register is still actual In-Reply-To: References: Message-ID: On Wed, Nov 3, 2010 at 3:56 PM, anatoly techtonik wrote: > On Wed, Nov 3, 2010 at 4:07 PM, Tarek Ziad? wrote: >>> I should have looked more carefully at the issue. The refusal to >>> use a password without storing it *is* a fairly narrow bug. >> >> Yes this is a bug. the password should be reused by upload. There's >> code for this but it seems to fails > > Fix landed. > http://bugs.python.org/issue9995 > >>>> This is a case where we need to come up with a better way of doing things. >>>> Someone needs to propose something and folks need to weigh in. >>> >>> I would love to see a solution to the broader problem. >>> >>> I really don't want to have to enter a password every time I >>> upload a package. >> >> me neither :) > > Does anybody know where is documentation on supported authentication in PyPI? > >>> I guess a good solution would be to integrate with existing >>> password-management tools. This could be prototyped as an >>> a separate upload tool. >> >> I have mentored a project in GSOC last year exactly for this case: >> keyring (avialable at PyPI) >> >> It is already successfully used in Mercurial (mercurial-keyring) that >> suffers the same problem when doing http/https >> >> The next step was to integrate keyring in distutils/upload but was not >> done yet due to a lack of time. > > Network protection is still weak. The password is sent nearly in cleartext. Right, we'd want to use https as well. Presumably, that's the easy part. Jim -- Jim Fulton From manlio.perillo at gmail.com Wed Nov 3 21:23:56 2010 From: manlio.perillo at gmail.com (Manlio Perillo) Date: Wed, 03 Nov 2010 21:23:56 +0100 Subject: [Distutils] [RFC] eggbuild Message-ID: <4CD1C4DC.7000401@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi. I'm pleased to announce the eggbuild package. This is a proof of concept for some ideas I have in mind for Python packages distribution. The package name, the implemented setuptools command names and the API are unstable, and may change in the future. The project is currently available at: http://hg.mperillo.ath.cx/eggbuild. The README and TODO file contain some informations, and I will try to not replicate them here. Basically, the idea is to build a setuptools distribution format that, like PyInstaller, is *self contained* (containing all required packages and all external shared libraries). However, unlike PyInstaller, there are some important differences, listed in the README file. I have tested the eggbuild command with a patched psycopg2 package, on a Linux Debian Squeeze system. Just for information, the resulting egg is 1.4 Mb big, when a normal egg is 156 Kb big. The resulting EGG-INFO\native_libs file is available here: http://paste.pocoo.org/show/285677/ The patch for psycopg2 is available here: http://paste.pocoo.org/show/285662/ The code *seems* to work. When importing the psycopg2 module, the *correct* shared libraries are loaded. At least the correct libpq.so.5 library is used. The other libraries required by libpq.so.5 are searched in the system default location, and I need to check if this can be solved without having to patch each library (adding LD_RPATH). A simple solution is to set LD_LIBRARY_PATH environment variable (but there are some issues to be solved). Now, what I want to implement is a setuptools command that will build a Debian package, with the following features: * pre-install script will create a virtual environment, using virtualenv. The virtualenv should be created in /opt/. * the package and required packages are stored in the deb data file, in egg binary format, and will be automatically copied to the virtual python installation. The main feature to implement is, however, a full MSI installer. However I don't know how MSI is supposed to work, and I will probably need some help (at least some good tutorial about MSI technology, what can be and what can not be done). I will appreciate comments and suggestions about the eggbuild design and implementation. Thanks Manlio -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzRxNwACgkQscQJ24LbaUQimACfRhi6kJUbPR8RkOz16Zcjni8T AA8AmQGqdakXHSuZTMc1oKbOaGPJKq2p =Pwqc -----END PGP SIGNATURE----- From ziade.tarek at gmail.com Wed Nov 3 23:26:39 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 3 Nov 2010 23:26:39 +0100 Subject: [Distutils] Security issue with Distutils register is still actual In-Reply-To: References: Message-ID: 2010/11/3 Jim Fulton : > On Wed, Nov 3, 2010 at 3:56 PM, anatoly techtonik wrote: >> On Wed, Nov 3, 2010 at 4:07 PM, Tarek Ziad? wrote: >>>> I should have looked more carefully at the issue. The refusal to >>>> use a password without storing it *is* a fairly narrow bug. >>> >>> Yes this is a bug. the password should be reused by upload. There's >>> code for this but it seems to fails >> >> Fix landed. >> http://bugs.python.org/issue9995 >> >>>>> This is a case where we need to come up with a better way of doing things. >>>>> Someone needs to propose something and folks need to weigh in. >>>> >>>> I would love to see a solution to the broader problem. >>>> >>>> I really don't want to have to enter a password every time I >>>> upload a package. >>> >>> me neither :) >> >> Does anybody know where is documentation on supported authentication in PyPI? >> >>>> I guess a good solution would be to integrate with existing >>>> password-management tools. This could be prototyped as an >>>> a separate upload tool. >>> >>> I have mentored a project in GSOC last year exactly for this case: >>> keyring (avialable at PyPI) >>> >>> It is already successfully used in Mercurial (mercurial-keyring) that >>> suffers the same problem when doing http/https >>> >>> The next step was to integrate keyring in distutils/upload but was not >>> done yet due to a lack of time. >> >> Network protection is still weak. The password is sent nearly in cleartext. > > Right, we'd want to use https as well. Presumably, that's the easy part. +1. > Jim > > -- > Jim Fulton > -- Tarek Ziad? | http://ziade.org From mailing at franzoni.eu Thu Nov 4 16:52:59 2010 From: mailing at franzoni.eu (Alan Franzoni) Date: Thu, 4 Nov 2010 16:52:59 +0100 Subject: [Distutils] [RFC] eggbuild In-Reply-To: <4CD1C4DC.7000401@gmail.com> References: <4CD1C4DC.7000401@gmail.com> Message-ID: On Wed, Nov 3, 2010 at 9:23 PM, Manlio Perillo wrote: > Hi. Ciao Manlio! > Basically, the idea is to build a setuptools distribution format that, > like PyInstaller, is *self contained* (containing all required packages > and all external shared libraries). > > However, unlike PyInstaller, there are some important differences, > listed in the README file. What are the key *advantages* ? From what I can see the great difference is in packaging, since it aims at providing automatic multi-format output support; but there're already some efforts to create a .deb or a .rpm from an arbitrary, well formed python project; I'm not sure about MSI installers, but this still seems to carry some problems, as you say later. Maybe an automated support for pyinstaller packaging would be more useful, as it's already tested and being used around with success; otherwise you'd be duplicating a lot of effort from that project. Also, check zc.buildout: by properly setting the paths inside a "main dir", you can get a great jar-like effect and achieve a result which is very similar to your original intent, e.g. you can have an /opt/something dir holding your code, your scripts, and all downloads & dependency eggs. And it's *really* easy to automatically create an rpm once your project is setup that way, I think zc.buildout is already got a recipe for RPMs about that. -- Alan Franzoni -- contact me at public@[mysurname].eu From manlio.perillo at gmail.com Thu Nov 4 17:53:06 2010 From: manlio.perillo at gmail.com (Manlio Perillo) Date: Thu, 04 Nov 2010 17:53:06 +0100 Subject: [Distutils] [RFC] eggbuild In-Reply-To: References: <4CD1C4DC.7000401@gmail.com> Message-ID: <4CD2E4F2.9020906@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Il 04/11/2010 16:52, Alan Franzoni ha scritto: > On Wed, Nov 3, 2010 at 9:23 PM, Manlio Perillo wrote: >> Hi. > > Ciao Manlio! > Ciao Alan! >> Basically, the idea is to build a setuptools distribution format that, >> like PyInstaller, is *self contained* (containing all required packages >> and all external shared libraries). >> >> However, unlike PyInstaller, there are some important differences, >> listed in the README file. > > What are the key *advantages* ? PyInstaller and py2exe do have some problems. For a project using Qt I was unable to use py2exe, and I was unable to understand how to use PyInstaller (!). Moreover, I don't see reasons why the Python interpreter should be included in the distribution. Another interesting feature of eggbuild is that it produces standard eggs. Not only this means that these eggs can be made available from something like PyPi, but this also means that they can be easy installed. This make distribution more easy, IMHO. > From what I can see the great > difference is in packaging, since it aims at providing automatic > multi-format output support; but there're already some efforts to > create a .deb or a .rpm from an arbitrary, well formed python project; Yes, the problem is that (assuming you are speaking about plain distutils): 1) packages dependencies are not handled 2) "external" shared libraries are not handled > [...] Thanks Manlio -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzS5PIACgkQscQJ24LbaURnYQCeI2esQYELtnsFVH8r7iEqgauc 88gAniMDD0qQc5+L21okaZrdYezcAXwV =V8St -----END PGP SIGNATURE----- From mailing at franzoni.eu Thu Nov 4 18:16:32 2010 From: mailing at franzoni.eu (Alan Franzoni) Date: Thu, 04 Nov 2010 18:16:32 +0100 Subject: [Distutils] [RFC] eggbuild In-Reply-To: <4CD2E4F2.9020906@gmail.com> References: <4CD1C4DC.7000401@gmail.com> <4CD2E4F2.9020906@gmail.com> Message-ID: <4CD2EA70.9070500@franzoni.eu> On 11/4/10 5:53 PM, Manlio Perillo wrote: > For a project using Qt I was unable to use py2exe, and I was unable to > understand how to use PyInstaller (!). That's quite a huge issue :-) but I think we've used PyInstaller internally ad it seemed to work. Have you asked for help on the pyinstaller ml and got no response? Ping Giovanni directly, then :P > Moreover, I don't see reasons why the Python interpreter should be > included in the distribution. If you're distributing an an app, the end user will probably just want it to be working without the hassle of installing it by himself - in this sense distributing the interpreter along the app is good. If you're distributing a library then you'd probably not need the eggbuild at all, since developers will very likely know their work. What specific problem are you trying to address? > Yes, the problem is that (assuming you are speaking about plain distutils): > > 1) packages dependencies are not handled I must admit I've never tried auto-building Python extensions, but I've tried zc.buildout to deliver and it works. I can post an example of how I do use it and then you can take a peek if it's suitable for your use. I'll do it tomorrow, now I'm a bit in a rush. > 2) "external" shared libraries are not handled External shared libraries might be an issue. If there're enough deps you might just end up packaging half a system shared libraries in order to distribute your packages in a self-contained way. It's quite risky, my approach with RPMs is to let the packaging system handle system-wide deps, and use zc.buildout to handle python eggs and extensions. -- Alan Franzoni contact me at public@[mysurname].eu From manlio.perillo at gmail.com Thu Nov 4 19:25:24 2010 From: manlio.perillo at gmail.com (Manlio Perillo) Date: Thu, 04 Nov 2010 19:25:24 +0100 Subject: [Distutils] [RFC] eggbuild In-Reply-To: <4CD2EA70.9070500@franzoni.eu> References: <4CD1C4DC.7000401@gmail.com> <4CD2E4F2.9020906@gmail.com> <4CD2EA70.9070500@franzoni.eu> Message-ID: <4CD2FA94.3080908@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Il 04/11/2010 18:16, Alan Franzoni ha scritto: > On 11/4/10 5:53 PM, Manlio Perillo wrote: >> For a project using Qt I was unable to use py2exe, and I was unable to >> understand how to use PyInstaller (!). > > That's quite a huge issue :-) but I think we've used PyInstaller > internally ad it seemed to work. Have you asked for help on the > pyinstaller ml and got no response? Ping Giovanni directly, then :P > I just read the documentation and tried to use it. > >> Moreover, I don't see reasons why the Python interpreter should be >> included in the distribution. > > If you're distributing an an app, the end user will probably just want > it to be working without the hassle of installing it by himself - in > this sense distributing the interpreter along the app is good. > > If you're distributing a library then you'd probably not need the > eggbuild at all, since developers will very likely know their work. > > What specific problem are you trying to address? > Distribute a Python package on Windows. Suppose package A depends on pure Python package B and on PyQT. Right now I have two options: 1) create an installer using PyInstaller or py2exe 2) have the user install Python (not a problem), PyQt (the "all included" installer) and then easy_install my application I think this is a big problem. Even if we assume that it is ok to create a PyInstaller installer, I don't see why a Windows user must not be able to do: easy_install A and have B and PyQt (with external libraries) correctly installed on the system. eggbuild tries to solve this problem. It build eggs that "embeds" all required shared libraries (excluding shared libraries already required by the Python interpreter -- this usually allow me to forget about SxS mess with Visual Studio redistributable libraries). >> Yes, the problem is that (assuming you are speaking about plain distutils): >> >> 1) packages dependencies are not handled > > > I must admit I've never tried auto-building Python extensions, but I've > tried zc.buildout to deliver and it works. I can post an example of how > I do use it and then you can take a peek if it's suitable for your use. > I have never used zc.buildout; however, as I wrote, what I'm trying to do is to solve the "external libraries" problem using standard eggs. Building an all included installer its the second step. For me it's already good if easy_install is able to install binary eggs with external libraries embedded! Thanks Manlio -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzS+pQACgkQscQJ24LbaUQ+PwCffNKkoaYfTig3uHucDaQ2mL84 GYcAn35I4PBVMAR1QnHwC/g9XXv0IM2l =kQCO -----END PGP SIGNATURE----- From mailing at franzoni.eu Thu Nov 4 22:01:36 2010 From: mailing at franzoni.eu (Alan Franzoni) Date: Thu, 04 Nov 2010 22:01:36 +0100 Subject: [Distutils] [RFC] eggbuild In-Reply-To: <4CD2FA94.3080908@gmail.com> References: <4CD1C4DC.7000401@gmail.com> <4CD2E4F2.9020906@gmail.com> <4CD2EA70.9070500@franzoni.eu> <4CD2FA94.3080908@gmail.com> Message-ID: <4CD31F30.2050201@franzoni.eu> On 11/4/10 7:25 PM, Manlio Perillo wrote: >> What specific problem are you trying to address? > > > Distribute a Python package on Windows. > > Suppose package A depends on pure Python package B and on PyQT. > Right now I have two options: > > 1) create an installer using PyInstaller or py2exe > 2) have the user install Python (not a problem), PyQt > (the "all included" installer) and then easy_install my application > > > I think this is a big problem. > Even if we assume that it is ok to create a PyInstaller installer, I > don't see why a Windows user must not be able to do: > easy_install A > > and have B and PyQt (with external libraries) correctly installed on the > system. OK, I got it. I must admit that my knowledge of Windows is not advanced enough to tell you whether this could work or if there're obvious issues. This could be a problem on Linux as well, by the way. If I try installing, let's say, lxml, and libxml2 and libxml2-dev are not installed on my system, easy_install will fail. Those are my 2c for the "weak spots": - you need to check recursively for all the shared libraries needed by python modules and extensions, and all the shared libraries needed by those shared libraries, and so on. Eggbuilds might end up just like huge chroots. If you want to, let's say, distribute something built with PyQt on Linux, you'll need PyQt, the QT Runtime, then? most probably X libs, libc and something else. Do you *really* want to distribute everything? I've just checked for your EGG-INFO/native_libs: you don't seem to recurse libraries. Just an example I spotted: libgnutls requires zlib, but no libz.so.something is included in native libs. - you need to check for dlopen() calls as well (or LoadLibrary/Ex on Windows - but I don't know whether it's easy to look for them in PE executables or DLLs); this might not be trivial and will increase the number of libraries that need to be included. - there might be legal implications when redistributing shared libraries; it might not be easy to tell which license each library has, and a single non-redistributable library might prevent to release the whole package. GPL and LGPL mandates source distribution along binary distribution, are you ready to package and make all the sources available to the user? Remember that, as you will be the redistributor, this requirement must be met by you - you can't just tell people to fetch sources from this or that environment. - reproducible builds will be almost impossible - if a dep changes on your system, you might end up without a clear understanding of what happened between one build and another - apparently identical - build. You may resort to chroot-building tools like pbuilder or mock on Linux, but this will be far more difficult on Windows. My idea is that everything looks far too complex, and would work easily only for really simple cases. Everything else will just turn overcomplicated. You're really trying to port to Python what Java does through its classpath system and, to a greater extent, with Osgi; but Python philosophy is quite different, you'll end up distributing whole OSes! -- Alan Franzoni contact me at public@[mysurname].eu From manlio.perillo at gmail.com Thu Nov 4 23:09:43 2010 From: manlio.perillo at gmail.com (Manlio Perillo) Date: Thu, 04 Nov 2010 23:09:43 +0100 Subject: [Distutils] [RFC] eggbuild In-Reply-To: <4CD31F30.2050201@franzoni.eu> References: <4CD1C4DC.7000401@gmail.com> <4CD2E4F2.9020906@gmail.com> <4CD2EA70.9070500@franzoni.eu> <4CD2FA94.3080908@gmail.com> <4CD31F30.2050201@franzoni.eu> Message-ID: <4CD32F27.8000900@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Il 04/11/2010 22:01, Alan Franzoni ha scritto: > On 11/4/10 7:25 PM, Manlio Perillo wrote: >>> What specific problem are you trying to address? >> >> >> Distribute a Python package on Windows. >> >> Suppose package A depends on pure Python package B and on PyQT. >> Right now I have two options: >> >> 1) create an installer using PyInstaller or py2exe >> 2) have the user install Python (not a problem), PyQt >> (the "all included" installer) and then easy_install my application >> >> >> I think this is a big problem. >> Even if we assume that it is ok to create a PyInstaller installer, I >> don't see why a Windows user must not be able to do: >> easy_install A >> >> and have B and PyQt (with external libraries) correctly installed on the >> system. > > > OK, I got it. > > I must admit that my knowledge of Windows is not advanced enough to tell > you whether this could work or if there're obvious issues. > I will do some tests on Windows when I find some free time. But it should work. > This could be a problem on Linux as well, by the way. If I try > installing, let's say, lxml, and libxml2 and libxml2-dev are not > installed on my system, easy_install will fail. > Yes. But on Linux systems at least we have package managers, and a compiler is almost always available, > Those are my 2c for the "weak spots": > > - you need to check recursively for all the shared libraries needed by > python modules and extensions, and all the shared libraries needed by > those shared libraries, and so on. This is what I do. > Eggbuilds might end up just like huge > chroots. > > If you want to, let's say, distribute something built with PyQt on > Linux, you'll need PyQt, the QT Runtime, then? most probably X libs, > libc and something else. Do you *really* want to distribute everything? > A simple solution is to specify, in the setup.cfg file, a list of "know" libraries that you can assume are always available on the system. > I've just checked for your EGG-INFO/native_libs: you don't seem to > recurse libraries. Just an example I spotted: libgnutls requires zlib, > but no libz.so.something is included in native libs. > The reason libz.so is not included is because it is already required by libpython. I assume that libraries required by libpython are available on the system, and I avoid to copy them in the egg. > - you need to check for dlopen() calls as well (or LoadLibrary/Ex on > Windows - but I don't know whether it's easy to look for them in PE > executables or DLLs); this might not be trivial and will increase the > number of libraries that need to be included. > I assume only few projects use dlopen/LoadLibrary/Ex. This problem, moreover, is so complex that can not even be solved by advanced package managers; why should I try to solve it? :) > > - there might be legal implications when redistributing shared > libraries; it might not be easy to tell which license each library has, > and a single non-redistributable library might prevent to release the > whole package. Yes, this is indeed a problem. It is resposibility of the egg builder to check this. > GPL and LGPL mandates source distribution along binary > distribution, are you ready to package and make all the sources > available to the user? Remember that, as you will be the redistributor, > this requirement must be met by you - you can't just tell people to > fetch sources from this or that environment. > > - reproducible builds will be almost impossible - if a dep changes on > your system, you might end up without a clear understanding of what > happened between one build and another - apparently identical - build. As far as I can tell, the same can happen with PyInstaller. And the same can happen, as an example, when riverbank builds the PyQt installer. > You may resort to chroot-building tools like pbuilder or mock on Linux, > but this will be far more difficult on Windows. > > My idea is that everything looks far too complex, and would work easily > only for really simple cases. Everything else will just turn > overcomplicated. > It is indeed a complex problem, but PyInstaller is far more complex. > You're really trying to port to Python what Java does through its > classpath system and, to a greater extent, with Osgi; but Python > philosophy is quite different, you'll end up distributing whole OSes! > > I don't know Java :). Regards Manlio -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzTLycACgkQscQJ24LbaUSdoQCeJlSzjJ+iYR8gyqLu3RwKJFfk e4IAn0VrfK97p4p0TCsNyBxje26Qu74T =7xgA -----END PGP SIGNATURE----- From pje at telecommunity.com Fri Nov 5 01:06:12 2010 From: pje at telecommunity.com (P.J. Eby) Date: Thu, 04 Nov 2010 20:06:12 -0400 Subject: [Distutils] package data not installed from python .egg file In-Reply-To: References: Message-ID: <20101105000610.1E3A23A414B@sparrow.telecommunity.com> At 09:32 AM 11/3/2010 +0100, Roban Hultman Kramer wrote: >Can anyone suggest what I need to do to get the package_data files to >be installed when easy_install unzips the egg file? The issue is the '..' in your data file names; currently setuptools.archive_util refuses to extract files with '..' in their path. I'm going to go ahead and fix this, but even after I do, please note that it will continue to be a problem for versions of easy_install that are out there "in the wild". From roban.kramer at phys.ethz.ch Fri Nov 5 13:04:09 2010 From: roban.kramer at phys.ethz.ch (Roban Hultman Kramer) Date: Fri, 5 Nov 2010 13:04:09 +0100 Subject: [Distutils] package data not installed from python .egg file In-Reply-To: <20101105000610.1E3A23A414B@sparrow.telecommunity.com> References: <20101105000610.1E3A23A414B@sparrow.telecommunity.com> Message-ID: Thanks very much! That was an easy fix on my end. Combining that with a MANIFEST.in file, and the data gets included and installed properly with bdist_egg and sdist. I really appreciate your help. -Roban On Fri, Nov 5, 2010 at 1:06 AM, P.J. Eby wrote: > At 09:32 AM 11/3/2010 +0100, Roban Hultman Kramer wrote: >> >> Can anyone suggest what I need to do to get the package_data files to >> be installed when easy_install unzips the egg file? > > The issue is the '..' in your data file names; currently > setuptools.archive_util refuses to extract files with '..' in their path. > ?I'm going to go ahead and fix this, but even after I do, please note that > it will continue to be a problem for versions of easy_install that are out > there "in the wild". > > _______________________________________________ > Distutils-SIG maillist ?- ?Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From sienkiew at stsci.edu Fri Nov 5 16:55:54 2010 From: sienkiew at stsci.edu (Mark Sienkiewicz) Date: Fri, 05 Nov 2010 11:55:54 -0400 Subject: [Distutils] An observation on how system packagers and developers can be friends In-Reply-To: <4CC9C573.5080903@v.loewis.de> References: <4CC9C573.5080903@v.loewis.de> Message-ID: <4CD4290A.9090800@stsci.edu> > In the same direction, I wish people would understand that 64-bit Python > builds just waste memory, and that they were better off with 32-bit > implementations. There are certain irrational reasons to believe that "64 bits is better than 32", but there is also some logic to it. I support an application that can't run in a 32 bit python because there literally is not enough memory available. It needs a larger address space. If the system python is only 32 bits, the users can't run the application. Most of them are not qualified to try to build their own 64 bit python. Most of them are not even qualified to try to maintain two copies of python, install certain software in each, and then choose whether they need 32 or 64 bits before they start their data analysis. Since some applications need the larger address space, it is better for python to just be 64 bits all the time. It makes the computer do a little extra work for our convenience, but that is exactly the point of using computers. Mark S. From jim at zope.com Fri Nov 5 18:29:34 2010 From: jim at zope.com (Jim Fulton) Date: Fri, 5 Nov 2010 13:29:34 -0400 Subject: [Distutils] 64-bit vs 32-bit (Re: An observation on how system packagers and developers can be friends) Message-ID: On Fri, Nov 5, 2010 at 11:55 AM, Mark Sienkiewicz wrote: > >> In the same direction, I wish people would understand that 64-bit Python >> builds just waste memory, and that they were better off with 32-bit >> implementations. > > There are certain irrational reasons to believe that "64 bits is better than > 32", but there is also some logic to it. > > I support an application that can't run in a 32 bit python because there > literally is not enough memory available. ?It needs a larger address space. Me too. I've found that on 32-bit linux systems, when programs get above around 2G, mallocs of a few tens of megs will fail. What's worse -- much much worse -- when a memory error occurs, it can leave an application in an unpredictable state. (Think memory errors in the bowels of your database.) These errors tend to lead to really odd symptoms and be be hard to diagnose. This is bad enough, but I really want to restart an app when a memory error occurs, but MemoryError is a subclass of Exception so it tends to be caught a lot. I really wish MemoryError was one of those errors you had to go way out of your way to catch. Because of the havoc that can be caused by even infrequent memory errors, we're moving most of our apps to 64 bits, even though they rarely use more than 2G. Jim -- Jim Fulton From mailing at franzoni.eu Sat Nov 6 13:41:53 2010 From: mailing at franzoni.eu (Alan Franzoni) Date: Sat, 06 Nov 2010 13:41:53 +0100 Subject: [Distutils] [RFC] eggbuild In-Reply-To: <4CD32F27.8000900@gmail.com> References: <4CD1C4DC.7000401@gmail.com> <4CD2E4F2.9020906@gmail.com> <4CD2EA70.9070500@franzoni.eu> <4CD2FA94.3080908@gmail.com> <4CD31F30.2050201@franzoni.eu> <4CD32F27.8000900@gmail.com> Message-ID: <4CD54D11.5080501@franzoni.eu> On 11/4/10 11:09 PM, Manlio Perillo wrote: Sorry for being late, I had a busy Friday. >> If you want to, let's say, distribute something built with PyQt on >> Linux, you'll need PyQt, the QT Runtime, then? most probably X libs, >> libc and something else. Do you *really* want to distribute everything? > A simple solution is to specify, in the setup.cfg file, a list of "know" > libraries that you can assume are always available on the system. But you don't know if they are really available or if the *proper* version of them is available. This might make the whole project unuseful. And the behaviour of your project while specifying "known" libraries might just end up being very similar to PyInstaller - without the embedded interpreter. Another great issue with Python extensions is usually the ucs2/ucs4 problem; an extension compiled with ucs2 builtin won't work with a ucs4 interpreter and viceversa; that's probably a good reason alone to pack an interpreter along with the libs. > The reason libz.so is not included is because it is already required by > libpython. I think I can remember it is an optional dependency. If you assume a lib is available on your interpreter and the user's interpreter has been compiled in a different way you might just end up with many issues. > I assume only few projects use dlopen/LoadLibrary/Ex. This might be an HUGE assumption. Have you verified that? You should at least look for dlopen() calls in your libraries and prevent building if that gets used. Otherwise your code might just unpredicably fail at runtime when the shared lib is opened. >> - reproducible builds will be almost impossible - if a dep changes on >> your system, you might end up without a clear understanding of what >> happened between one build and another - apparently identical - build. > > As far as I can tell, the same can happen with PyInstaller. > And the same can happen, as an example, when riverbank builds the PyQt > installer. Sure it can happen. >> My idea is that everything looks far too complex, and would work easily >> only for really simple cases. Everything else will just turn >> overcomplicated. > > > It is indeed a complex problem, but PyInstaller is far more complex. But it's already there - why should you reinvent the wheel? Maybe you can just add some patches on top of it. Maybe you could just pick a scenario ( Windows XP? Vista? 7? 32-bit only?) and try to find out a narrow-scoped, quickly-available solution instead of going for the big, definitive, cross-platform solution. You might just say "hey, lib A B C are always available on Windows environments, pack everything else along." . I would keep advising packing the interpreter as well, by the way. Very important: I'm not a PyInstaller fan (I've used it a couple of times), I'd just like to prevent you from doing some work I think is unnecessary. -- Alan Franzoni contact me at public@[mysurname].eu From geoff.bache at gmail.com Tue Nov 9 11:58:14 2010 From: geoff.bache at gmail.com (Geoff Bache) Date: Tue, 9 Nov 2010 11:58:14 +0100 Subject: [Distutils] setup.py script using python interpreter from previous run Message-ID: Hi all, I have the following setup.py script: #!/usr/bin/env python from distutils.core import setup scripts=["hello.py"] setup(scripts=scripts) I have two different python installations (using virtualenv) where I wish to install this program. So I do ~/tmp/test_setup/python1/bin/python setup.py install which creates a file at /users/geoff/tmp/test_setup/python1/bin/hello.py, that looks like this: #!/users/geoff/tmp/test_setup/python1/bin/python print "Hello" So far so good. But then I also install it somewhere else: ~/tmp/test_setup/python2/bin/python setup.py install which creates a file at /users/geoff/tmp/test_setup/python2/bin/hello.py which refers to "python1", i..e it has the same contents as the first one. Which is clearly not what I want. Is this a bug? Or have I missed something in my setup.py? It works of course if I remove the generated "build" directory by hand, but it's not so nice to have to remember to do that. (I'm using Python2.6 on Linux) Regards, Geoff Bache From fdrake at acm.org Tue Nov 9 12:04:37 2010 From: fdrake at acm.org (Fred Drake) Date: Tue, 9 Nov 2010 06:04:37 -0500 Subject: [Distutils] setup.py script using python interpreter from previous run In-Reply-To: References: Message-ID: On Tue, Nov 9, 2010 at 5:58 AM, Geoff Bache wrote: > Is this a bug? I think so. > Or have I missed something in my setup.py? It works of > course if I remove the generated "build" directory by hand, but it's > not so nice to have to remember to do that. You should be able to remove the build/ directory that was created where the package is unpacked, and then run the install command with your python2/bin/python. The problem is that "build" and "install" are traditionally two distinct steps. While setup.py takes care of both for you, it does detect that you switched Python's, so doesn't re-build for the second. ? -Fred -- Fred L. Drake, Jr.? ? "A storm broke loose in my mind."? --Albert Einstein From ziade.tarek at gmail.com Tue Nov 9 12:07:36 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 9 Nov 2010 12:07:36 +0100 Subject: [Distutils] setup.py script using python interpreter from previous run In-Reply-To: References: Message-ID: On Tue, Nov 9, 2010 at 11:58 AM, Geoff Bache wrote: > Hi all, > > I have the following setup.py script: > > #!/usr/bin/env python > from distutils.core import setup > > scripts=["hello.py"] > > setup(scripts=scripts) > > I have two different python installations (using virtualenv) where I > wish to install this program. So I do > > ~/tmp/test_setup/python1/bin/python setup.py install > > which creates a file at > /users/geoff/tmp/test_setup/python1/bin/hello.py, that looks like > this: > > #!/users/geoff/tmp/test_setup/python1/bin/python > > print "Hello" > > So far so good. But then I also install it somewhere else: > > ~/tmp/test_setup/python2/bin/python setup.py install > > which creates a file at > /users/geoff/tmp/test_setup/python2/bin/hello.py which refers to > "python1", i..e it has the same contents as the first one. Which is > clearly not what I want. > > Is this a bug? Or have I missed something in my setup.py? It works of > course if I remove the generated "build" directory by hand, but it's > not so nice to have to remember to do that. Can you add an issue in bugs.python.org please (using the "Distutils2" component and "feature request") This is a general problem we have in Distutils: update an existing build if some of its elements are out of date. We need to add some kind of replacement mechanism (not sure how yet). This won't be fixed in Distutils 1, but we can change the behavior in Distutils2. In the meantime, you should remove the build directory by hand or use the "clean" command that does it for you in your call: ~/tmp/test_setup/python2/bin/python setup.py clean -a install Sorry for the inconveniency > (I'm using Python2.6 on Linux) > > Regards, > Geoff Bache > _______________________________________________ > Distutils-SIG maillist ?- ?Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -- Tarek Ziad? | http://ziade.org From ziade.tarek at gmail.com Tue Nov 9 12:31:07 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 9 Nov 2010 12:31:07 +0100 Subject: [Distutils] setup.py script using python interpreter from previous run In-Reply-To: References: Message-ID: On Tue, Nov 9, 2010 at 12:04 PM, Fred Drake wrote: > On Tue, Nov 9, 2010 at 5:58 AM, Geoff Bache wrote: >> Is this a bug? > > I think so. > >> Or have I missed something in my setup.py? It works of >> course if I remove the generated "build" directory by hand, but it's >> not so nice to have to remember to do that. > > You should be able to remove the build/ directory that was created > where the package is unpacked, and then run the install command with > your python2/bin/python. ?The problem is that "build" and "install" > are traditionally two distinct steps. ?While setup.py takes care of > both for you, it does detect that you switched Python's, so doesn't > re-build for the second. Since we're adding the "configure" command that will let install have an access to all build options that were used, (still in Eric's branch) we should be able to know where the script is in the build tree, and check in install_scripts that its not outdated with the file timestamps + checking the shebang line. Or... keep in that configure file the path to the interpreter and just rebuild all scripts in case the interpreter has changed. > > > ? -Fred > > -- > Fred L. Drake, Jr.? ? > "A storm broke loose in my mind."? --Albert Einstein > _______________________________________________ > Distutils-SIG maillist ?- ?Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -- Tarek Ziad? | http://ziade.org From fdrake at acm.org Tue Nov 9 14:06:23 2010 From: fdrake at acm.org (Fred Drake) Date: Tue, 9 Nov 2010 08:06:23 -0500 Subject: [Distutils] setup.py script using python interpreter from previous run In-Reply-To: References: Message-ID: On Tue, Nov 9, 2010 at 6:31 AM, Tarek Ziad? wrote: > Or... keep in that configure file the path to the interpreter and just > rebuild all scripts in case the interpreter has changed. +1 to the simple approach. Given the need to do this, I guess I don't really see the value of storing scripts in the build tree. But I suppose someone actually uses them from there somehow. (I'd expect getting the imports to work would be a painful exercise, but I've seen enough tortured processes that I wouldn't put it past folks.) ? -Fred -- Fred L. Drake, Jr.? ? "A storm broke loose in my mind."? --Albert Einstein From geoff.bache at gmail.com Tue Nov 9 14:24:05 2010 From: geoff.bache at gmail.com (Geoff Bache) Date: Tue, 9 Nov 2010 14:24:05 +0100 Subject: [Distutils] setup.py script using python interpreter from previous run In-Reply-To: References: Message-ID: Hi Tarek, > Can you add an issue in bugs.python.org please (using the "Distutils2" > component and "feature request") I've added a bug, but felt I had to describe it as I saw it (i.e. a behaviour issue with Distutils). It took me quite some time to track down what was going on, and I didn't think I was doing anything complicated or unusual. It can't be that hard to fix it in distutils can it (as you've just been discussing)? Regards, Geoff From friedrichromstedt at gmail.com Wed Nov 10 12:07:35 2010 From: friedrichromstedt at gmail.com (Friedrich Romstedt) Date: Wed, 10 Nov 2010 12:07:35 +0100 Subject: [Distutils] setup.py script using python interpreter from previous run In-Reply-To: References: Message-ID: 2010/11/9 Tarek Ziad? : > Since we're adding the "configure" command that will let install have > an access to all build options that were used, (still in Eric's > branch) > we should be able to know where the script is in the build tree, and > check in install_scripts that its not outdated with the file > timestamps + checking the shebang line. > > Or... keep in that configure file the path to the interpreter and just > rebuild all scripts in case the interpreter has changed. In my opinion it might be worthy storing some combination (interpreter path, interpreter build no). Case study: For building some binaries, we are using both py2.7-macosx10.3 and py2.7-macosx10.5 on Mac OS X (obviously), which are different builds but must be installed to exactly the same default location (some technicalities). Of course only one of them is installed at a time. We just recently ran into an error (failure) where we think we used py2.7-10.3 built binaries to make a py2.7-10.5 installer (or vice versa). This came so because the build directory was reused. A solution might be to include the build number or something similar in the build/anything-something directory name. Maybe just add a hash of the tuple (installer path, build no) in the build directory's name? Obviously, then both interpreter path change as well as interpreter build number change would lead to another build directory. Might be useful to be able to use different Pythons in parallel for parallel builds. One could add whatever is needed later to the hashed tuple without breaking compatibility. (E.g., build options :-) Friedrich From atagar1 at gmail.com Tue Nov 16 20:13:34 2010 From: atagar1 at gmail.com (Damian Johnson) Date: Tue, 16 Nov 2010 11:13:34 -0800 Subject: [Distutils] Python Deb Packages Message-ID: Hi, I'm working on deb/rpm packaging for a terminal monitor of tor relays [1] and running into an issue with the egg-info distutil autogenerates. My project currently installs to the default '/usr/share/pyshared' [2] but according to the debian policy for private python modules [3] it should be installed to '/usr/share' instead (my program is an application and there's no reason for it to be importable by other modules). The problem with doing this is that the egg-info metadata is hardcoded in 'distutils/command/install.py' to be added on install, and having an egg-info file on '/usr/share' causes a lintian error when generating the deb. Also, most things I've been reading seems to say that the package belongs in '/usr/share/pyshared' and that removing the egg-info is kinda evil [4]. Help would be appreciated. Cheers! -Damian [1] http://www.atagar.com/arm/ [2] https://svn.torproject.org/svn/arm/trunk/setup.py [3] http://www.debian.org/doc/packaging-manuals/python-policy/ch-programs.html#s-current_version_progs [4] http://mail.python.org/pipermail/distutils-sig/2007-September/008225.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From bugtrack at roumenpetrov.info Tue Nov 16 23:22:21 2010 From: bugtrack at roumenpetrov.info (Roumen Petrov) Date: Wed, 17 Nov 2010 00:22:21 +0200 Subject: [Distutils] disutils2 tests and python 2.5 Message-ID: <4CE3041D.8090509@roumenpetrov.info> Hello all, I would like to ask to compare/check unit test results of distutils2 (repository) with someone. In my build environment the unit test exit without errors: ---------------- test_check_metadata (distutils2.tests.test_command_check.CheckTestCase) ... ok test_check_restructuredtext (distutils2.tests.test_command_check.CheckTestCase) ... skipped 'requires docutils' test_config (distutils2.tests.test_config.ConfigTestCase) ... ---------------- Note that test_config is without ok at the end. Did someone get same results with python 2.5 ? Roumen From merwok at netwok.org Wed Nov 17 10:25:32 2010 From: merwok at netwok.org (=?UTF-8?Q?=C3=89ric_Araujo?=) Date: Wed, 17 Nov 2010 10:25:32 +0100 Subject: [Distutils] disutils2 tests and python 2.5 In-Reply-To: <4CE3041D.8090509@roumenpetrov.info> References: <4CE3041D.8090509@roumenpetrov.info> Message-ID: Hi Roumen, A recent change in d2.run has broken test_config. I propose to revert the change. If Tarek agrees, he'll pull http://bitbucket.org/Merwok/distutils2/changeset/593109ce2549 Thank you for the testing! From piotr at debian.org Wed Nov 17 12:29:06 2010 From: piotr at debian.org (Piotr =?utf-8?Q?O=C5=BCarowski?=) Date: Wed, 17 Nov 2010 12:29:06 +0100 Subject: [Distutils] Python Deb Packages In-Reply-To: References: Message-ID: <20101117112906.GA15091@piotro.eu> [Mail-Followup-To set to debian-python at lists.debian.org] [Damian Johnson, 2010-11-16] > My project currently installs to the default '/usr/share/pyshared' [2] but /usr/share/pyshared is Debian tools' internal location and you should not use it (i.e. it's not and will never be in sys.path) > according to the debian policy for private python modules [3] it should be > installed to '/usr/share' instead (my program is an application and there's /usr/share/binary_package_name/ to be exact > no reason for it to be importable by other modules). The problem with doing > this is that the egg-info metadata is hardcoded in > 'distutils/command/install.py' to be added on install, and having an > egg-info file on '/usr/share' causes a lintian error when generating the > deb. --install-lib=/usr/share/yourpackagename/ --install-data=/usr/share/yourpackagename/ (and maybe --install-scripts=/usr/share/yourpackagename/) should do the right thing (i.e. don't install directly to /usr/share/ as it could couse namespace prioblems with other private modules) -- Piotr O?arowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645 From matt at tplus1.com Wed Nov 17 16:14:13 2010 From: matt at tplus1.com (W. Matthew Wilson) Date: Wed, 17 Nov 2010 10:14:13 -0500 Subject: [Distutils] Where should I put my tests in my package? Message-ID: Right now, I've got a project that started off looking sort of like this: pitz/ pitz/ cmdline/ webapp/ entity/ tests/ test_cmdline.py test_entity.py test_webapp.py In my setup.py file, I have this line packages=find_packages(exclude=[''tests']), so that the tests don't get installed when the code is installed. This works OK for now, but I'm adding more folders under the original folders, like this: pitz/ pitz/ cmdline/ webapp/ handlers/ __init__.py statichandler.py entity/ tests/ test_cmdline.py test_entity.py test_webapp.py test_handlers.py test_statichandler.py It is getting more difficult to figure out the links between test files and code files. I have a few questions: 1. I would like to put lots of tests/ folders next to the code they test, rather than at the top level of the project. How do I make sure the tests folders don't get installed? 2. Should I worry about not installing tests? Should I install them? 3. Is there any pattern I should follow for organizing my tests within my project? Thanks for the help. Matt -- W. Matthew Wilson matt at tplus1.com http://tplus1.com From atagar1 at gmail.com Wed Nov 17 17:25:05 2010 From: atagar1 at gmail.com (Damian Johnson) Date: Wed, 17 Nov 2010 08:25:05 -0800 Subject: [Distutils] Python Deb Packages In-Reply-To: References: Message-ID: > /usr/share/pyshared is Debian tools' internal location and you should not use it Great, thanks Piotr for the clarification. > /usr/share/binary_package_name/ to be exact > --install-lib=/usr/share/yourpackagename/ --install-data=/usr/share/yourpackagename/ > (and maybe --install-scripts=/usr/share/yourpackagename/) should do the > right thing Unfortunately not quite. Passing those flags results in: /usr/share/arm/arm/ /usr/share/arm/arm-1.3.7_dev.egg-info The egg-info metadata only has relevance if this was being installed to pyshared so ideally it should be omitted and the package contents just installed to: /usr/share/arm/ Setting "--install-lib=/usr/share" adds my package to the right location, but also includes "/usr/share/arm-1.3.7_dev.egg-info" which causes a lintian error. A couple quick questions: - If private python modules aren't supposed to be living in '/usr/share/pyshared' then shouldn't distutil include an option to omit the egg-info metadata? - Any ideas you have for how to remove the egg-info during debian builds would be much appreciated. Peter has mentioned that a simple change to the debian/rules [1] should do the trick but I'm still a bit too new to debian builds to know what. Cheers! -Damian [1] https://svn.torproject.org/svn/arm/resources/build/debian/rules On Tue, Nov 16, 2010 at 11:13 AM, Damian Johnson wrote: > Hi, I'm working on deb/rpm packaging for a terminal monitor of tor relays > [1] and running into an issue with the egg-info distutil autogenerates. > > My project currently installs to the default '/usr/share/pyshared' [2] but > according to the debian policy for private python modules [3] it should be > installed to '/usr/share' instead (my program is an application and there's > no reason for it to be importable by other modules). The problem with doing > this is that the egg-info metadata is hardcoded in > 'distutils/command/install.py' to be added on install, and having an > egg-info file on '/usr/share' causes a lintian error when generating the > deb. Also, most things I've been reading seems to say that the package > belongs in '/usr/share/pyshared' and that removing the egg-info is kinda > evil [4]. > > Help would be appreciated. Cheers! -Damian > > [1] http://www.atagar.com/arm/ > [2] https://svn.torproject.org/svn/arm/trunk/setup.py > [3] > http://www.debian.org/doc/packaging-manuals/python-policy/ch-programs.html#s-current_version_progs > [4] > http://mail.python.org/pipermail/distutils-sig/2007-September/008225.html > > From marius at pov.lt Wed Nov 17 17:26:22 2010 From: marius at pov.lt (Marius Gedminas) Date: Wed, 17 Nov 2010 18:26:22 +0200 Subject: [Distutils] Where should I put my tests in my package? In-Reply-To: References: Message-ID: <20101117162622.GE22398@fridge.pov.lt> On Wed, Nov 17, 2010 at 10:14:13AM -0500, W. Matthew Wilson wrote: > I have a few questions: > > 1. I would like to put lots of tests/ folders next to the code they > test, rather than at the top level of the project. How do I make sure > the tests folders don't get installed? > 2. Should I worry about not installing tests? Should I install them? There was a discussion on the testing-in-python list a while ago: http://lists.idyll.org/pipermail/testing-in-python/2010-July/003147.html IIRC there was no consensus reached. > 3. Is there any pattern I should follow for organizing my tests > within my project? I like to put the tests inside my packages and install them: pitz/ src/ pitz/ cmdline/ tests/ webapp/ handlers/ tests/ tests/ entity/ tests/ tests/ I also like to have one test module for each code module. Marius Gedminas -- Some people around here wouldn't recognize subtlety if it hit them on the head. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From fdrake at acm.org Wed Nov 17 18:01:47 2010 From: fdrake at acm.org (Fred Drake) Date: Wed, 17 Nov 2010 12:01:47 -0500 Subject: [Distutils] Where should I put my tests in my package? In-Reply-To: References: Message-ID: On Wed, Nov 17, 2010 at 10:14 AM, W. Matthew Wilson wrote: > 1. ?I would like to put lots of tests/ folders next to the code they > test, rather than at the top level of the project. ?How do I make sure > the tests folders don't get installed? I'm sure there's a way to prevent installation in this case, but I don't know what it is. > 2. ?Should I worry about not installing tests? ?Should I install them? Unless you're targeting storage-constrained platform, why shouldn't you install them? > 3. ?Is there any pattern I should follow for organizing my tests > within my project? There's a lot of variations here. Some people prefer your original arrangement, and others prefer embedding the tests within each package as you're describing for your new arrangement. (The later is widely used in the Zope community.) Another approach, good if you want to avoid installation of the tests, is to use a mirror hierarchy for your tests: pitz/ pitz/ cmdline/ webapp/ handlers/ __init__.py statichandler.py entity/ tests/ commandline/ test_cmdline.py webapp/ test_webapp.py handlers/ test_handlers.py test_statichandler.py entity/ test_entity.py I favor the Zopish approach these days. ? -Fred -- Fred L. Drake, Jr.? ? "A storm broke loose in my mind."? --Albert Einstein From mailing at franzoni.eu Wed Nov 17 18:18:44 2010 From: mailing at franzoni.eu (Alan Franzoni) Date: Wed, 17 Nov 2010 18:18:44 +0100 Subject: [Distutils] Retrieval of python PKG-INFO data for installed packages Message-ID: Hello, is it possible to retrieve the same info which is stored in PKG-INFO for an installed package? I've seen this project: http://pypi.python.org/pypi/pkginfo doesn't anything built in setuptools/distribute/pkg_resources achieve a similar result? e.g., while in a lib, I'd like to do something like: import mymodule get_pkg_info(mymodule).project_name ? -- Alan Franzoni -- contact me at public@[mysurname].eu From eposse at gmail.com Wed Nov 17 18:21:31 2010 From: eposse at gmail.com (Ernesto Posse) Date: Wed, 17 Nov 2010 12:21:31 -0500 Subject: [Distutils] Where should I put my tests in my package? In-Reply-To: <20101117162622.GE22398@fridge.pov.lt> References: <20101117162622.GE22398@fridge.pov.lt> Message-ID: On Wed, Nov 17, 2010 at 11:26 AM, Marius Gedminas wrote: > 1. ?I would like to put lots of tests/ folders next to the code they > test, rather than at the top level of the project. ?How do I make sure > the tests folders don't get installed? > 3. ?Is there any pattern I should follow for organizing my tests > within my project? My approach is a bit different from what has been suggested. I create a separate tests directory which mirrors the structure of the packages I want to test, and install them in a (share) data directory. So, in setup.py I have import os, os.path ... data_dir = 'share/pitz/' data_files = \ [(os.path.join(data_dir, root), [os.path.join(root, file_name) for file_name in files]) for root, dirs, files in os.walk('tests')] + other_data_files ... and in MANIFEST.in I have recursive-include tests *.py so the directory structure would look like this: pitz/ src/ pitz/ cmdline/ webapp/ handlers/ __init__.py statichandler.py entity/ tests/ pitz/ cmdline/ webapp/ handlers/ __init__.py test_statichandler.py entity/ setup.py MANIFEST.in I like this approach because it cleanly separates the tests from the actual package code, the tests go into the data directory, rather than with the package itself, and I can easily decide whether to include or not the tests with the distribution. -- Ernesto Posse Applied Formal Methods Group - Software Technology Lab School of Computing Queen's University - Kingston, Ontario, Canada From merwok at netwok.org Wed Nov 17 22:07:22 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Wed, 17 Nov 2010 22:07:22 +0100 Subject: [Distutils] disutils2 tests and python 2.5 In-Reply-To: <4CE4419B.90100@roumenpetrov.info> References: <4CE3041D.8090509@roumenpetrov.info> <4CE4419B.90100@roumenpetrov.info> Message-ID: <4CE4440A.6010901@netwok.org> > - ['README', 'setup.cfg', 'data/data1'] > + ['setup.cfg', 'README', 'data/data1'] We need to use the TestCase method that checks for items without respect to order. Regards From tseaver at palladion.com Wed Nov 17 22:30:48 2010 From: tseaver at palladion.com (Tres Seaver) Date: Wed, 17 Nov 2010 16:30:48 -0500 Subject: [Distutils] Retrieval of python PKG-INFO data for installed packages In-Reply-To: References: Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 11/17/2010 12:18 PM, Alan Franzoni wrote: > Hello, > is it possible to retrieve the same info which is stored in PKG-INFO > for an installed package? > > I've seen this project: > http://pypi.python.org/pypi/pkginfo > > doesn't anything built in setuptools/distribute/pkg_resources achieve > a similar result? pkginfo is aimed at parsing the PKG-INFO file for *any* kind of package: - - Source distribution (created via 'setup.py sdist') - - BInary distributions (created via 'setup.py bdist_egg', maybe others) - - Installed packages (works only if the 'PKG-INFO' file is created in the "right place" under / next to the package, e.g., via setuptools, distribute, or Python >= 2.6). - - "Develop" packages (installed via 'setup.py develop', therefore only with setuptools / distribute support), You query it via either the path to a distribution, the name of a package, or the installed package itself: >>> from pkginfo import get_metadata >>> get_metadata('path/to/tarball.gz') >>> import wsgiref >>> get_metadata(wsgiref) >>> _.name u'wsgiref' The package provides a same-named shell script for doing those queries at the command line: $ ../../bin/pkginfo docs/examples/mypackage-0.1.tar.gz metadata_version: 1.0 name: mypackage version: 0.1 platforms: [u'UNKNOWN'] home_page: http://pypi.python.org/pypi/pkginfo author: Tres Seaver author_email: tseaver at palladion.comginfo] $ ../../bin/pkginfo wsgiref metadata_version: 1.0 name: wsgiref version: 0.1.2 platforms: [u'UNKNOWN'] summary: WSGI (PEP 333) Reference Library author: Phillip J. Eby author_email: web-sig at python.org license: PSF or ZPL Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzkSYcACgkQ+gerLs4ltQ66RQCgrRMTO4gFvitIqd3nm6TUZXTx d5UAn1q7l7keTrs6+yYSe3BTG87cvxY3 =IuFu -----END PGP SIGNATURE----- From fdrake at acm.org Wed Nov 17 22:46:54 2010 From: fdrake at acm.org (Fred Drake) Date: Wed, 17 Nov 2010 16:46:54 -0500 Subject: [Distutils] disutils2 tests and python 2.5 In-Reply-To: <4CE4440A.6010901@netwok.org> References: <4CE3041D.8090509@roumenpetrov.info> <4CE4419B.90100@roumenpetrov.info> <4CE4440A.6010901@netwok.org> Message-ID: On Wed, Nov 17, 2010 at 4:07 PM, ?ric Araujo wrote: > We need to use the TestCase method that checks for items without respect > to order. self.assertEqual(sorted(a), sorted(b)) ? -Fred -- Fred L. Drake, Jr.? ? "A storm broke loose in my mind."? --Albert Einstein From merwok at netwok.org Wed Nov 17 23:02:33 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Wed, 17 Nov 2010 23:02:33 +0100 Subject: [Distutils] disutils2 tests and python 2.5 In-Reply-To: References: <4CE3041D.8090509@roumenpetrov.info> <4CE4419B.90100@roumenpetrov.info> <4CE4440A.6010901@netwok.org> Message-ID: <4CE450F9.8020900@netwok.org> >> We need to use the TestCase method that checks for items without respect >> to order. > > self.assertEqual(sorted(a), sorted(b)) I wanted to avoid that. I?m sure one of the new TestCase methods checks for ?same items?. Regards From leorochael at gmail.com Thu Nov 18 10:57:41 2010 From: leorochael at gmail.com (Leonardo Rochael Almeida) Date: Thu, 18 Nov 2010 10:57:41 +0100 Subject: [Distutils] M2Crypto install failing Message-ID: Hi, With this simple buildout: [buildout] parts = m2c [m2c] recipe = zc.recipe.egg eggs = M2Crypto I'm getting this failure: Installing 'M2Crypto'. We have no distributions for M2Crypto that satisfies 'M2Crypto'. Getting distribution for 'M2Crypto'. Running easy_install: /usr/bin/python "-c" "from setuptools.command.easy_install import main; main()" "-mUNxd" "/home/leo/opt/m2cryptotest/eggs/tmp5wm_oK" "-q" "/tmp/tmphof_2Jget_dist/M2Crypto-0.20.2.win32.zip" path=/home/leo/opt/m2cryptotest/eggs/setuptools-0.6c12dev_r85381-py2.6.egg error: Couldn't find a setup script in /tmp/tmphof_2Jget_dist/M2Crypto-0.20.2.win32.zip An error occured when trying to install M2Crypto 0.20.2.win32. Look above this message for any errors that were output by easy_install. While: Installing m2c. Getting distribution for 'M2Crypto'. Error: Couldn't install: M2Crypto 0.20.2.win32 AFAICT, This is due to the fact that M2Crypto win32 egg available here: http://chandlerproject.org/pub/Projects/MeTooCrypto/M2Crypto-0.20.2.win32.zip And linked from the project "Home Page" as registered on PyPI takes precedence over the package stored here: http://pypi.python.org/packages/source/M/M2Crypto/M2Crypto-0.20.2.tar.gz And linked from the PyPI page itself. If I'm not mistaken, this is also because "win32" in M2Crypto-0.20.2.win32.zip looks like part of the version, and not the architecture. Is this analysis correct? Is there a quick way to blacklist this package in buildout or setuptools? For the record, I can work-around this by downloading the right M2Crypto package, then doing: bin/buildout setup src/M2Crypto-0.20.2 bdist_egg and moving and unpacking the generated egg in the egg cache directory, but this is really ugly. Cheers, Leo From email at lnowak.com Thu Nov 18 13:15:32 2010 From: email at lnowak.com (=?UTF-8?Q?=C5=81ukasz_Nowak?=) Date: Thu, 18 Nov 2010 13:15:32 +0100 Subject: [Distutils] setuptools.archive_util.unpack_archive dies in case of bad tarball Message-ID: Hello, Please see attached tarkill.tar.gz and buildout profile. This tarball containts dead link: ls -l tarkill lrwxrwxrwx 1 luke users 4 2010-11-18 19:56 kill -> kill After running attached buildout profile and having python shell setuptools will enter infinite loop: >>> import sys >>> sys.version_info sys.version_info(major=2, minor=7, micro=0, releaselevel='final', serial=0) >>> import tempfile >>> import setuptools.archive_util >>> tempdir = tempfile.mkdtemp() >>> setuptools.archive_util.unpack_archive('tarkill.tar.gz', tempdir) ^CTraceback (most recent call last): File "", line 1, in File "/srv/home/luke/python2.7/setup/eggs/setuptools-0.6c12dev_r85381-py2.7.egg/setuptools/archive_util.py", line 67, in unpack_archive driver(filename, extract_dir, progress_filter) File "/srv/home/luke/python2.7/setup/eggs/setuptools-0.6c12dev_r85381-py2.7.egg/setuptools/archive_util.py", line 190, in unpack_tarfile member = tarobj._getmember(linkpath) File "/srv/home/luke/python2.7/parts/python2.7/lib/python2.7/tarfile.py", line 2357, in _getmember members = self.getmembers() File "/srv/home/luke/python2.7/parts/python2.7/lib/python2.7/tarfile.py", line 1801, in getmembers self._check() KeyboardInterrupt Attached workaround which breaks the loop, but I think that such deadlink shall be created, just like tar does it. Regards, Luke -- ?ukasz Nowak IT Specialist email at lnowak.com http://lnowak.com/ Skype: Shufla jid: shufla at jabster.pl gg: 1157726 ``Use the Source, Luke...'' -------------- next part -------------- A non-text attachment was scrubbed... Name: buildout.cfg Type: application/octet-stream Size: 108 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: setuptools-0.6c12dev_r85381.workaround.badtar.patch Type: text/x-patch Size: 1099 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: tarkill.tar.gz Type: application/x-gzip Size: 146 bytes Desc: not available URL: From ozarow at gmail.com Thu Nov 18 14:10:35 2010 From: ozarow at gmail.com (Piotr Ozarowski) Date: Thu, 18 Nov 2010 14:10:35 +0100 Subject: [Distutils] Python Deb Packages In-Reply-To: References: Message-ID: <20101118131035.GG15091@piotro.eu> [Damian Johnson, 2010-11-17] > > --install-lib=/usr/share/yourpackagename/ --install-data=/usr/share/yourpackagename/ > > (and maybe --install-scripts=/usr/share/yourpackagename/) should do the > > right thing > > Unfortunately not quite. Passing those flags results in: > /usr/share/arm/arm/ > /usr/share/arm/arm-1.3.7_dev.egg-info ... and that's perfectly fine (if the first "arm" is (Debian) package name) > The egg-info metadata only has relevance if this was being installed > to pyshared so ideally it should be omitted and the package contents > just installed to: > /usr/share/arm/ You can remove it in debian/rules (see below) if you want (if it's not used in the module), keeping it shouldn't harm, though > Setting "--install-lib=/usr/share" adds my package to the right > location, but also includes "/usr/share/arm-1.3.7_dev.egg-info" which > causes a lintian error. A couple quick questions: What if package foo will install private module "arm" in /usr/share/ as well? Please don't do that unless there's a really good reason to take over a namespace > - If private python modules aren't supposed to be living in > '/usr/share/pyshared' then shouldn't distutil include an option to > omit the egg-info metadata? Are you sure this data isn't used? An option to disable .egg-info file/directory generation would be nice, though. > - Any ideas you have for how to remove the egg-info during debian > builds would be much appreciated. How about something like this (I didn't test it): | #!/usr/bin/make -f | %: | dh $@ --with python2 | | override_dh_auto_install: | python setup.py install --install-lib=/usr/share/foo/ --install-data=/usr/share/foo/ --install-scripts=/usr/share/foo/ | dh_link /usr/share/foo/myscript /usr/bin/myscript | rm -rf debian/foo/usr/share/foo/arm*info replace foo with [Debian] binary package name; instead of dh_link line you can add debian/links file -- Piotr O?arowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645 From ozarow at gmail.com Thu Nov 18 14:37:38 2010 From: ozarow at gmail.com (Piotr Ozarowski) Date: Thu, 18 Nov 2010 14:37:38 +0100 Subject: [Distutils] Python Deb Packages In-Reply-To: <20101118131035.GG15091@piotro.eu> References: <20101118131035.GG15091@piotro.eu> Message-ID: <20101118133738.GH15091@piotro.eu> [Piotr Ozarowski, 2010-11-18] > | python setup.py install --install-lib=/usr/share/foo/ --install-data=/usr/share/foo/ --install-scripts=/usr/share/foo/ add "--root=debian/foo" to that line -- Piotr O?arowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645 From manlio.perillo at gmail.com Thu Nov 18 16:51:40 2010 From: manlio.perillo at gmail.com (Manlio Perillo) Date: Thu, 18 Nov 2010 16:51:40 +0100 Subject: [Distutils] setuptools vs distribute on Debian Message-ID: <4CE54B8C.7090000@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 No, this is not a flame. I'm on Debian Squeeze and I noticed this strange thing: $sudo easy_install -U setuptools install_dir /usr/local/lib/python2.6/dist-packages/ Searching for distribute Reading http://pypi.python.org/simple/distribute/ Reading http://packages.python.org/distribute Best match: distribute 0.6.14 Processing distribute-0.6.14-py2.6.egg distribute 0.6.14 is already the active version in easy-install.pth Installing easy_install script to /usr/local/bin Installing easy_install-2.6 script to /usr/local/bin Using /usr/local/lib/python2.6/dist-packages/distribute-0.6.14-py2.6.egg Processing dependencies for distribute Finished processing dependencies for distribute Now, I read that this is a Debian policy: for some reasons they force me to use distribute instead setuptools. However, I *do* want to use setuptools, and I'm unable to do what I want. I checked the "fake" setuptools source code but I can't see what I need to modify in order to get setuptools when I ask for setuptools. By the way, I suspect there is a bug in Debian setuptools: if dist.key=='distribute': # Ensure that setuptools itself never becomes unavailable! # XXX should this check for latest version? filename = os.path.join(self.install_dir,'setuptools.pth') if os.path.islink(filename): os.unlink(filename) f = open(filename, 'wt') f.write(self.pth_file.make_relative(dist.location)+'\n') f.close() However if `easy_install setuptools` is an alias for `easy_install distribute`, that code may not executed (?) Thanks Manlio -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzlS4wACgkQscQJ24LbaUQVMgCfV0oIvuUJ3xRPlNhftTJZ3HMR 0BoAn2rb8DJFaPHhRRKcDNyRNN28uUfl =Lg51 -----END PGP SIGNATURE----- From tseaver at palladion.com Thu Nov 18 17:08:45 2010 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 18 Nov 2010 11:08:45 -0500 Subject: [Distutils] setuptools vs distribute on Debian In-Reply-To: <4CE54B8C.7090000@gmail.com> References: <4CE54B8C.7090000@gmail.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 11/18/2010 10:51 AM, Manlio Perillo wrote: > No, this is not a flame. > > I'm on Debian Squeeze and I noticed this strange thing: > > $sudo easy_install -U setuptools > install_dir /usr/local/lib/python2.6/dist-packages/ > Searching for distribute > Reading http://pypi.python.org/simple/distribute/ > Reading http://packages.python.org/distribute > Best match: distribute 0.6.14 > Processing distribute-0.6.14-py2.6.egg > distribute 0.6.14 is already the active version in easy-install.pth > Installing easy_install script to /usr/local/bin > Installing easy_install-2.6 script to /usr/local/bin > > Using /usr/local/lib/python2.6/dist-packages/distribute-0.6.14-py2.6.egg > Processing dependencies for distribute > Finished processing dependencies for distribute > > > Now, I read that this is a Debian policy: for some reasons they force me > to use distribute instead setuptools. > > However, I *do* want to use setuptools, and I'm unable to do what I want. > I checked the "fake" setuptools source code but I can't see what I need > to modify in order to get setuptools when I ask for setuptools. If you use the OS tools to install from the OS repositories, you are pretty much forced to live with any of their non-optional policy choices. Alternatives would be to go "outside the box" in one way or another: - - Install an existing .deb file which somebody else who agrees with you has built. - - Build and install such a .deb file yourself, e.g. via 'apt-get source setuptools' and then hack on the build files. - - Install setuptools into the system Python manually (probably a Really Bad Idea). - - Install and use 'virtualenv' to get an environment insulated from the system Python. 'virtualenv' still uses the "real" setuptools by default. - - Install Python and setuptools from source. I prefer the last option, myself, as I know better than the OS packagers do about how to configure and build Python and related stuff for my applications, just as they know better how to build it to support other system packages. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzlT40ACgkQ+gerLs4ltQ7ZYwCfROhAZcofntZNfFvTV5CqRo9v J1oAn3AGDhRJyrSBzCSjTT1qS40MUibJ =IEjs -----END PGP SIGNATURE----- From manlio.perillo at gmail.com Thu Nov 18 17:23:56 2010 From: manlio.perillo at gmail.com (Manlio Perillo) Date: Thu, 18 Nov 2010 17:23:56 +0100 Subject: [Distutils] setuptools vs distribute on Debian In-Reply-To: References: <4CE54B8C.7090000@gmail.com> Message-ID: <4CE5531C.9090603@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Il 18/11/2010 17:08, Tres Seaver ha scritto: > On 11/18/2010 10:51 AM, Manlio Perillo wrote: >> No, this is not a flame. > >> I'm on Debian Squeeze and I noticed this strange thing: > > [setuptools is an alias for sitribute] > >> Now, I read that this is a Debian policy: for some reasons they force me >> to use distribute instead setuptools. > >> However, I *do* want to use setuptools, and I'm unable to do what I want. >> I checked the "fake" setuptools source code but I can't see what I need >> to modify in order to get setuptools when I ask for setuptools. > > If you use the OS tools to install from the OS repositories, you are > pretty much forced to live with any of their non-optional policy > choices. Alternatives would be to go "outside the box" in one way or > another: > > [...] > - Install setuptools into the system Python manually (probably a Really > Bad Idea). > What I would like to do is to edit Python setuptools source files in order to remove the setuptools -> distribute alias. Then do: $easy_install -U setuptools $aptitude purge python-setuptools $python -c "from setuptools.command import easy_install; easy_install.main(['setuptools'])" The latter is required in order to reinstall scripts. Note that this works on Debian Etch, but it does no more work on Debian Squeeze. > - Install and use 'virtualenv' to get an environment insulated from > the system Python. 'virtualenv' still uses the "real" setuptools by > default. > > - Install Python and setuptools from source. > > > I prefer the last option, myself, as I know better than the OS packagers > do about how to configure and build Python and related stuff for my > applications, just as they know better how to build it to support other > system packages. > I would like to avoid this option! I usually build Python packages by myself (and often I use virtualenv), however I do find very useful precompiled packages for something like pygtk and pyqt, that I really *don't* want to build by myself (and, as far as I know, are not even easy installable). Thanks Manlio -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzlUxwACgkQscQJ24LbaUR12QCfZZsSGPPglfY38fWAMYpxG2J6 q7AAnAqKHqRTvlihnEVvCDfZyCd8wrSo =TcrT -----END PGP SIGNATURE----- From pje at telecommunity.com Thu Nov 18 17:09:36 2010 From: pje at telecommunity.com (P.J. Eby) Date: Thu, 18 Nov 2010 11:09:36 -0500 Subject: [Distutils] setuptools vs distribute on Debian In-Reply-To: <4CE54B8C.7090000@gmail.com> References: <4CE54B8C.7090000@gmail.com> Message-ID: <20101118160942.0D5293A40B0@sparrow.telecommunity.com> At 04:51 PM 11/18/2010 +0100, Manlio Perillo wrote: >Now, I read that this is a Debian policy: for some reasons they force me >to use distribute instead setuptools. Have you tried installing setuptools from source? From manlio.perillo at gmail.com Thu Nov 18 17:33:06 2010 From: manlio.perillo at gmail.com (Manlio Perillo) Date: Thu, 18 Nov 2010 17:33:06 +0100 Subject: [Distutils] setuptools vs distribute on Debian In-Reply-To: <20101118160942.0D5293A40B0@sparrow.telecommunity.com> References: <4CE54B8C.7090000@gmail.com> <20101118160942.0D5293A40B0@sparrow.telecommunity.com> Message-ID: <4CE55542.6020206@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Il 18/11/2010 17:09, P.J. Eby ha scritto: > At 04:51 PM 11/18/2010 +0100, Manlio Perillo wrote: >> Now, I read that this is a Debian policy: for some reasons they force me >> to use distribute instead setuptools. > > Have you tried installing setuptools from source? > > Done, but still: http://paste.pocoo.org/show/293158/ This is strange, I'm probably missing something. Thanks Manlio -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzlVUIACgkQscQJ24LbaURWCQCfSpSLksFVh0CAyr11z4fBewn4 eusAn1+p5TS+fL50dZii/B+ddjf0m5Ax =NQbr -----END PGP SIGNATURE----- From barry at python.org Thu Nov 18 18:04:24 2010 From: barry at python.org (Barry Warsaw) Date: Thu, 18 Nov 2010 12:04:24 -0500 Subject: [Distutils] setuptools vs distribute on Debian In-Reply-To: <4CE5531C.9090603@gmail.com> References: <4CE54B8C.7090000@gmail.com> <4CE5531C.9090603@gmail.com> Message-ID: <20101118120424.59f5478c@mission> On Nov 18, 2010, at 05:23 PM, Manlio Perillo wrote: >I usually build Python packages by myself (and often I use virtualenv), FWIW, virtualenv on Debian has a --setuptools option (and $VIRTUALENV_USE_SETUPTOOLS environment variable) to force the virtual environment to use setuptools instead of the default distribute. In fact, the standard virtualenv --distribute option is basically a no-op. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From manlio.perillo at gmail.com Thu Nov 18 18:16:35 2010 From: manlio.perillo at gmail.com (Manlio Perillo) Date: Thu, 18 Nov 2010 18:16:35 +0100 Subject: [Distutils] setuptools vs distribute on Debian In-Reply-To: <20101118120424.59f5478c@mission> References: <4CE54B8C.7090000@gmail.com> <4CE5531C.9090603@gmail.com> <20101118120424.59f5478c@mission> Message-ID: <4CE55F73.9030808@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Il 18/11/2010 18:04, Barry Warsaw ha scritto: > On Nov 18, 2010, at 05:23 PM, Manlio Perillo wrote: > >> I usually build Python packages by myself (and often I use virtualenv), > > FWIW, virtualenv on Debian has a --setuptools option (and > $VIRTUALENV_USE_SETUPTOOLS environment variable) to force the virtual > environment to use setuptools instead of the default distribute. In fact, the > standard virtualenv --distribute option is basically a no-op. > Thanks, I was not aware of this. However, since I install virtualenv using easy_install, this is not a problem. Manlio -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzlX3MACgkQscQJ24LbaUQWZQCghGXC5BbsMnbFMYdsL3nbvaoq SIYAoIZnOB27YKRK98uNzFoUKYksZEb5 =vxFE -----END PGP SIGNATURE----- From pje at telecommunity.com Thu Nov 18 18:28:43 2010 From: pje at telecommunity.com (P.J. Eby) Date: Thu, 18 Nov 2010 12:28:43 -0500 Subject: [Distutils] setuptools vs distribute on Debian In-Reply-To: <4CE55542.6020206@gmail.com> References: <4CE54B8C.7090000@gmail.com> <20101118160942.0D5293A40B0@sparrow.telecommunity.com> <4CE55542.6020206@gmail.com> Message-ID: <20101118172848.1E70E3A40B0@sparrow.telecommunity.com> At 05:33 PM 11/18/2010 +0100, Manlio Perillo wrote: >-----BEGIN PGP SIGNED MESSAGE----- >Hash: SHA1 > >Il 18/11/2010 17:09, P.J. Eby ha scritto: > > At 04:51 PM 11/18/2010 +0100, Manlio Perillo wrote: > >> Now, I read that this is a Debian policy: for some reasons they force me > >> to use distribute instead setuptools. > > > > Have you tried installing setuptools from source? > > > > > >Done, but still: >http://paste.pocoo.org/show/293158/ > > >This is strange, I'm probably missing something. Probably there's a .pth file somewhere on sys.path that still contains distribute, so it's getting loaded first. From manlio.perillo at gmail.com Thu Nov 18 19:34:20 2010 From: manlio.perillo at gmail.com (Manlio Perillo) Date: Thu, 18 Nov 2010 19:34:20 +0100 Subject: [Distutils] setuptools vs distribute on Debian In-Reply-To: <20101118160942.0D5293A40B0@sparrow.telecommunity.com> References: <4CE54B8C.7090000@gmail.com> <20101118160942.0D5293A40B0@sparrow.telecommunity.com> Message-ID: <4CE571AC.1090309@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Il 18/11/2010 17:09, P.J. Eby ha scritto: > At 04:51 PM 11/18/2010 +0100, Manlio Perillo wrote: >> Now, I read that this is a Debian policy: for some reasons they force me >> to use distribute instead setuptools. > > Have you tried installing setuptools from source? > > I have removed distribute from my system, reinstalled setuptools from source and now it works. I'm still curious to know how setuptools was forced to be an alias for distribute... Thanks Manlio -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzlcawACgkQscQJ24LbaUQImgCeIn4nIbdPRLwU3OXWZ6Q9RrC+ ppYAnA9h2y9DNpsw2Zx7FmJpDHeffBC0 =RRgf -----END PGP SIGNATURE----- From ziade.tarek at gmail.com Thu Nov 18 21:31:26 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 18 Nov 2010 21:31:26 +0100 Subject: [Distutils] disutils2 tests and python 2.5 In-Reply-To: References: <4CE3041D.8090509@roumenpetrov.info> Message-ID: On Wed, Nov 17, 2010 at 10:25 AM, ?ric Araujo wrote: > Hi Roumen, > > A recent change in d2.run has broken test_config. ?I propose to revert the > change. > If Tarek agrees, he'll pull > http://bitbucket.org/Merwok/distutils2/changeset/593109ce2549 -1. The change was intended, as we are currently removing all command-line related things to run.py, and make dist.py do a single thing : be a Distribution object. The run.py script is becoming the future UI for distutils2, as we move from a distribution-centric (==driven by setup.py) to a global script. So, we'll fix failing tests but no revert is planned Cheers Tarek > > Thank you for the testing! > _______________________________________________ > Distutils-SIG maillist ?- ?Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -- Tarek Ziad? | http://ziade.org From bugtrack at roumenpetrov.info Wed Nov 17 21:56:59 2010 From: bugtrack at roumenpetrov.info (Roumen Petrov) Date: Wed, 17 Nov 2010 22:56:59 +0200 Subject: [Distutils] disutils2 tests and python 2.5 In-Reply-To: References: <4CE3041D.8090509@roumenpetrov.info> Message-ID: <4CE4419B.90100@roumenpetrov.info> ?ric Araujo wrote: > Hi Roumen, > > A recent change in d2.run has broken test_config. I propose to revert > the change. > If Tarek agrees, he'll pull > http://bitbucket.org/Merwok/distutils2/changeset/593109ce2549 > > Thank you for the testing! Ok thanks . Now after patch the reported issue is fixed and results is .... ====================================================================== FAIL: test_find_files (distutils2.tests.test_mkcfg.MkcfgTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "..../distutils2/trunk/distutils2/tests/test_mkcfg.py", line 40, in test_find_files ['setup.cfg', 'README', 'data/data1']) AssertionError: Lists differ: ['README', 'setup.cfg', 'data/... != ['setup.cfg', 'README', 'data/... First differing element 0: README setup.cfg - ['README', 'setup.cfg', 'data/data1'] + ['setup.cfg', 'README', 'data/data1'] ---------------------------------------------------------------------- Ran 259 tests in 15.501s FAILED (failures=1, skipped=19) Roumen From fdrake at acm.org Fri Nov 19 12:20:56 2010 From: fdrake at acm.org (Fred Drake) Date: Fri, 19 Nov 2010 06:20:56 -0500 Subject: [Distutils] disutils2 tests and python 2.5 In-Reply-To: <4CE450F9.8020900@netwok.org> References: <4CE3041D.8090509@roumenpetrov.info> <4CE4419B.90100@roumenpetrov.info> <4CE4440A.6010901@netwok.org> <4CE450F9.8020900@netwok.org> Message-ID: On Wed, Nov 17, 2010 at 5:02 PM, ?ric Araujo wrote: > I wanted to avoid that. ?I?m sure one of the new TestCase methods checks > for ?same items?. I guess that's fine for the d2 tests. I should spend a little more time looking at u2 to determine if it makes sense to start using that in tests at work (where it adds a dependency, and possibly some re-knitting of base classes). ? -Fred -- Fred L. Drake, Jr.? ? "A storm broke loose in my mind."? --Albert Einstein From rich at noir.com Sat Nov 20 00:18:24 2010 From: rich at noir.com (K. Richard Pixley) Date: Fri, 19 Nov 2010 15:18:24 -0800 Subject: [Distutils] installing to a non-standard directory Message-ID: <4CE705C0.6030301@noir.com> I'm trying to install to a non-standard directory. Something akin to "python setup.py install --prefix=`pwd`/junk" using gnu makefile standards semantics for --prefix but I can't seem to find the right arguments to install to make this happen. The closest I can seem to find is: python setup.py install --prefix=`pwd`/junk --install-layout=unix But that returns errors: distutils.errors.DistutilsError: can't create or remove files in install directory The following error occurred while trying to add or remove files in the installation directory: [Errno 2] No such file or directory: '/home/rich/projects/rcmp/junk/lib/python2.6/site-packages/test-easy-install-12743.pth' The installation directory you specified (via --install-dir, --prefix, or the distutils default setting) was: /home/rich/projects/rcmp/junk/lib/python2.6/site-packages/ This directory does not currently exist. Please create it and try again, or choose a different installation directory (using the -d or --install-dir option). Although there doesn't appear to be any --install-dir option. Can anyone explain what any of the 13 options related to this are intended to do? Or better, can anyone tell me which one I want? --rich -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Sat Nov 20 02:51:41 2010 From: pje at telecommunity.com (P.J. Eby) Date: Fri, 19 Nov 2010 20:51:41 -0500 Subject: [Distutils] installing to a non-standard directory In-Reply-To: <4CE705C0.6030301@noir.com> References: <4CE705C0.6030301@noir.com> Message-ID: <20101120015149.D457E3A40B0@sparrow.telecommunity.com> At 03:18 PM 11/19/2010 -0800, K. Richard Pixley wrote: >I'm trying to install to a non-standard directory. Why? It makes a difference as to what the best way to do it is. For example, if you're planning to install a Python application or create a restricted development environment, you might be better off using a virtualenv. Or, alternatively, you may want to configure your ~/.pydistutils.cfg to set the default installation paths rather than having to specify them on the command line for every package you install. >distutils.errors.DistutilsError: can't create or remove files in >install directory > >The following error occurred while trying to add or remove files in the >installation directory: > > [Errno 2] No such file or directory: > '/home/rich/projects/rcmp/junk/lib/python2.6/site-packages/test-easy-install-12743.pth' > >The installation directory you specified (via --install-dir, --prefix, or >the distutils default setting) was: > > /home/rich/projects/rcmp/junk/lib/python2.6/site-packages/ > >This directory does not currently exist. Please create it and try again, or >choose a different installation directory (using the -d or --install-dir >option). As the above says, the path you're trying to install to doesn't exist. (The next thing you'll find after you fix that, is that it isn't on your PYTHONPATH.) What are you trying to install, and why do you want to install it in that specific directory? From rich at noir.com Sun Nov 21 00:39:31 2010 From: rich at noir.com (K. Richard Pixley) Date: Sat, 20 Nov 2010 15:39:31 -0800 Subject: [Distutils] installing to a non-standard directory In-Reply-To: <20101120015149.D457E3A40B0@sparrow.telecommunity.com> References: <4CE705C0.6030301@noir.com> <20101120015149.D457E3A40B0@sparrow.telecommunity.com> Message-ID: <4CE85C33.8000605@noir.com> On 20101119 17:51, P.J. Eby wrote: > At 03:18 PM 11/19/2010 -0800, K. Richard Pixley wrote: >> I'm trying to install to a non-standard directory. > > Why? It makes a difference as to what the best way to do it is. > > For example, if you're planning to install a Python application or > create a restricted development environment, you might be better off > using a virtualenv. Or, alternatively, you may want to configure your > ~/.pydistutils.cfg to set the default installation paths rather than > having to specify them on the command line for every package you install. I'm aware of those and they are not relevant in this context. >> distutils.errors.DistutilsError: can't create or remove files in >> install directory >> >> The following error occurred while trying to add or remove files in the >> installation directory: >> >> [Errno 2] No such file or directory: >> '/home/rich/projects/rcmp/junk/lib/python2.6/site-packages/test-easy-install-12743.pth' >> >> The installation directory you specified (via --install-dir, >> --prefix, or >> the distutils default setting) was: >> >> /home/rich/projects/rcmp/junk/lib/python2.6/site-packages/ >> >> This directory does not currently exist. Please create it and try >> again, or >> choose a different installation directory (using the -d or --install-dir >> option). > > As the above says, the path you're trying to install to doesn't exist. Yes. Although using --root= doesn't result in this problem. So some portion of the install process knows how to create the appropriate directory tree and yet that is explicitly being skipped in this context for some mysterious reason. > (The next thing you'll find after you fix that, is that it isn't on > your PYTHONPATH.) No, that won't be a problem in this context. > What are you trying to install, and why do you want to install it in > that specific directory? I really don't think it matters, but if you insist, there are several contexts I'm looking at. Context #1: an alternate packaging mechanism. I want to use that packaging mechanism because it's the most appropriate way to package and distribute my module for this application. The packaging mechanism involves calling "install", (typically "make install prefix=/tmp/jail", but there's no reason it couldn't be "python setup.py install"), in a sort of jail. Anything installed during that encapsulation is recorded and those pieces are the ones which are picked up and packaged by the automatic packaging system. Context #2: cross development. I'm running /on/ one system producing a root file system which will be used later to boot a new system. This is another application of "make install PREFIX=/tmp/newroot" Context #3: this one is complicated, but since you think it's relevant... Bitbake is a tool written in python for managing Openembedded, (http://openembedded.org). In rough terms you can think of bitbake as a sort of make replacement which builds a collection of components into binary packages and then installs those packages into a captive root file system. The usual use of this system is cross development where the target system is woefully underpowered, (Eg, a 600Mhz arm system with a few Megabytes of memory), so we'd prefer to use our battery of x86 servers, (~64 servers, each of which are ~24 cores, 24G memory, 4 - 8 way striped disks, etc). The difference in build time here is literally the difference between about 20 minutes, (on the server farm), vs several months on a single, native arm based system. Bitbake runs as a python application using the host's python interpreter. But for code management reasons, bitbake itself and all of the bitbake input for building the system are stored as source. This allows bitbake to follow the code as code is branched and the like and allows for multiple build directories on the same server without needing to use a virtual machine, (which has a huge performance overhead vs raw metal), or needing to reload the server, (which would also make the system unwieldy). In this particular application, I want to write the application in python so that I can manage nose based tests outside of bitbake and so that I can produce a shell level cover into my tool. However, the primary use of the tool will be from within bitbake. The code of my tool will need to follow the source code of the system, not bitbake, so putting the tool into bitbake directly isn't appropriate nor is it appropriate to install the tool globally on the build machine(s). Similarly, a python virtual environment isn't relevant or appropriate here either as the top level control of the system is already bitbake so our interpreter is running before we have a chance to virtualize it. And there's no good reason to change that architecture. What I want to do is to manage and build my tool in python, including nose tests outside of bitbake, as a component within openembedded. This means that bitbake will build the component and install the component into a staging area which is unique to this particular build directory. It can then import the tool from the local staging area and use it. What I want to do is the moral equivalent of "make install prefix=foo" or "make install DESTDIR=foo". It's a common task with numerous applications. This shouldn't be difficult and I really can't believe that I'm the first person using python to want to do this. It's a 10 character command line option. There should be something comparably simple in distutils/setuptools/whatever, no? --rich -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Sun Nov 21 19:33:37 2010 From: pje at telecommunity.com (P.J. Eby) Date: Sun, 21 Nov 2010 13:33:37 -0500 Subject: [Distutils] installing to a non-standard directory In-Reply-To: <4CE85C33.8000605@noir.com> References: <4CE705C0.6030301@noir.com> <20101120015149.D457E3A40B0@sparrow.telecommunity.com> <4CE85C33.8000605@noir.com> Message-ID: <20101121183358.186ED3A407F@sparrow.telecommunity.com> At 03:39 PM 11/20/2010 -0800, K. Richard Pixley wrote: >Context #1: an alternate packaging mechanism. I want to use that >packaging mechanism because it's the most appropriate way to package >and distribute my module for this application. The packaging >mechanism involves calling "install", (typically "make install >prefix=/tmp/jail", but there's no reason it couldn't be "python >setup.py install"), in a sort of jail. Anything installed during >that encapsulation is recorded and those pieces are the ones which >are picked up and packaged by the automatic packaging system. Use --root, then. That's what it's for. >Context #2: cross development. I'm running on one system producing >a root file system which will be used later to boot a new >system. This is another application of "make install PREFIX=/tmp/newroot" Likewise, this is what --root is for. In both of the above cases, you want the prefix of anything you build to be set up for the prefix where it's really going to *end up* installed, not to the temporary directory where you're mock-installing it to. That's why all distutils' internal commands use --root when they call an install operation for packaging. >What I want to do is to manage and build my tool in python, >including nose tests outside of bitbake, as a component within >openembedded. This means that bitbake will build the component and >install the component into a staging area which is unique to this >particular build directory. It can then import the tool from the >local staging area and use it. > >What I want to do is the moral equivalent of "make install >prefix=foo" or "make install DESTDIR=foo". It's a common task with >numerous applications. This shouldn't be difficult and I really >can't believe that I'm the first person using python to want to do >this. It's a 10 character command line option. There should be >something comparably simple in distutils/setuptools/whatever, no? Yes. It's --root. ;-) From hltbra at gmail.com Sun Nov 21 21:35:44 2010 From: hltbra at gmail.com (Hugo Lopes Tavares) Date: Sun, 21 Nov 2010 18:35:44 -0200 Subject: [Distutils] mercurial update problem with pip In-Reply-To: <20101019131954.152755vldvs25log@www.rtfs.org> References: <20101019131954.152755vldvs25log@www.rtfs.org> Message-ID: Hi Fabian, I see you have a too old pip version. Sorry for not giving you a faster reply, but just do: $ pip install --upgrade pip And you will get the latest version from PyPI (0.8.1 by now). I tried to install the package you did, and it worked here. http://paste.pocoo.org/show/294334/ Just update it! Hugo. On Tue, Oct 19, 2010 at 9:19 AM, Fabian Sturm wrote: > Hi, > > I am using pip version 0.3.1 on Ubuntu Linux 9.10 to install django-piston > in a virtualenv. > > Since I need a specific version of django-piston I have the following entry > in the requirements.txt file: > > -e > hg+http://bitbucket.org/jespern/django-piston at dc0ee00d3bfc#egg=django-piston > > The first time the install works fine with pip but on the second time I get > the following error: > > Command hg fetch -q failed with error code 255 > Exception information: > Traceback (most recent call last): > ?File "/usr/lib/python2.6/dist-packages/pip.py", line 252, in main > ? ?self.run(options, args) > ?File "/usr/lib/python2.6/dist-packages/pip.py", line 408, in run > ? ?requirement_set.install_files(finder, force_root_egg_info=self.bundle) > ?File "/usr/lib/python2.6/dist-packages/pip.py", line 1741, in install_files > ? ?req_to_install.update_editable() > ?File "/usr/lib/python2.6/dist-packages/pip.py", line 1486, in > update_editable > ? ?version_control(self.url).obtain(self.source_dir) > ?File "/usr/lib/python2.6/dist-packages/pip.py", line 2942, in obtain > ? ?call_subprocess(['hg', 'fetch', '-q'], cwd=dest) > ?File "/usr/lib/python2.6/dist-packages/pip.py", line 3543, in > call_subprocess > ? ?% (command_desc, proc.returncode)) > InstallationError: Command hg fetch -q failed with error code 255 > > > If I run "hg fetch -q" manually in the django-piston repository I get: > > abort: working dir not at branch tip (use "hg update" to check out branch > tip) > > mercurial is at version 1.3.1. > > Any ideas what is wrong and how I get it to work? > > Thanks a lot, > Fabian > > > > > ---------------------------------------------------------------- > This message was sent using IMP, the Internet Messaging Program. > _______________________________________________ > Distutils-SIG maillist ?- ?Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From kverdecia at uci.cu Mon Nov 22 07:55:17 2010 From: kverdecia at uci.cu (Karel Antonio Verdecia Ortiz) Date: Mon, 22 Nov 2010 01:55:17 -0500 Subject: [Distutils] distributing only .pyc with stdeb Message-ID: <4CEA13D5.9010003@uci.cu> Hi, I'm using stdb to generate .deb packages for ubuntu but I need this packages to contain only the compiled modules (.pyc) and not the source modules (.py). ?What parameters have I to use to get only the .pyc files? Thanks. -------------- next part -------------- A non-text attachment was scrubbed... Name: kverdecia.vcf Type: text/x-vcard Size: 130 bytes Desc: not available URL: From barry at python.org Mon Nov 22 21:13:43 2010 From: barry at python.org (Barry Warsaw) Date: Mon, 22 Nov 2010 15:13:43 -0500 Subject: [Distutils] distributing only .pyc with stdeb In-Reply-To: <4CEA13D5.9010003@uci.cu> References: <4CEA13D5.9010003@uci.cu> Message-ID: <20101122151343.5a68a76e@mission> On Nov 22, 2010, at 01:55 AM, Karel Antonio Verdecia Ortiz wrote: >I'm using stdb to generate .deb packages for ubuntu but I need this packages >to contain only the compiled modules (.pyc) and not the source modules >(.py). ?What parameters have I to use to get only the .pyc files? Including only the .pyc files would mean that your package was not free software, and so it would violate the Debian Free Software Guideline. Thus I'd be surprised if the tool supported that use case easily or directly. Not that it can't be done, but you'll probably have to write some custom debian/rules file to handle that case. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ozarow at gmail.com Tue Nov 23 15:42:29 2010 From: ozarow at gmail.com (Piotr Ozarowski) Date: Tue, 23 Nov 2010 15:42:29 +0100 Subject: [Distutils] distributing only .pyc with stdeb In-Reply-To: <20101122151343.5a68a76e@mission> References: <4CEA13D5.9010003@uci.cu> <20101122151343.5a68a76e@mission> Message-ID: <20101123144228.GI30169@piotro.eu> [Barry Warsaw, 2010-11-22] > Not that it can't be done, but you'll probably have to write some custom > debian/rules file to handle that case. |#!/usr/bin/make -f |%: | dh $@ --with python2 | |override_dh_python2: | dh_python2 | pycompile debian/python-foo/usr/lib/ | # if you have different files for python2.X and python2.Y (i.e. pyshared is not used) | find debian/python-foo/usr/lib/ -name *.py -delete | rm -rf debian/python-foo/usr/share/pyshared/ should do the trick (not tested) -- Piotr O?arowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645 From ninja.gofer at gmail.com Sun Nov 28 13:44:55 2010 From: ninja.gofer at gmail.com (Oisin O'Maley) Date: Sun, 28 Nov 2010 12:44:55 +0000 Subject: [Distutils] Distutils fails to build Message-ID: Hi, I have been trying to reinstall distutils-0.6.14 on my gentoo system after some file corruption and now distutils fails to build. When manually building from source it throws the following error: *python2.6 setup.py build* Traceback (most recent call last): File "setup.py", line 211, in scripts = scripts, File "/usr/lib64/python2.6/distutils/core.py", line 113, in setup _setup_distribution = dist = klass(attrs) File "/home/oisin/source/distribute-0.6.14/setuptools/dist.py", line 222, in __init__ for ep in pkg_resources.iter_entry_points('distutils.setup_keywords'): File "/home/oisin/source/distribute-0.6.14/pkg_resources.py", line 454, in iter_entry_points entries = dist.get_entry_map(group) File "/home/oisin/source/distribute-0.6.14/pkg_resources.py", line 2252, in get_entry_map self._get_metadata('entry_points.txt'), self File "/home/oisin/source/distribute-0.6.14/pkg_resources.py", line 2038, in parse_map raise ValueError("Entry points must be listed in groups") ValueError: Entry points must be listed in groups Anyone got any idea what is causing this? This is preventing me from reinstalling allot of packages on my system, so any help would be greatly appreciated. I have a tread on the gentoo forums with more info on the problem if anyone is interested; http://forums.gentoo.org/viewtopic-t-854323.html Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ziade.tarek at gmail.com Sun Nov 28 14:31:20 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 28 Nov 2010 14:31:20 +0100 Subject: [Distutils] Distutils fails to build In-Reply-To: References: Message-ID: On Sun, Nov 28, 2010 at 1:44 PM, Oisin O'Maley wrote: > Hi, I have been trying to reinstall distutils-0.6.14 on my gentoo system > after some file corruption and now distutils fails to build. > > When manually building from source it throws the following error: Looks like a formatting issue in entry_points...like a missing [ or ] Was this from the source at PyPI or a source from Gentoo ? Try this: >>> import pkg_resources >>> list(pkg_resources.split_sections(open('distribute.egg-info/entry_points.txt').read())) If it fails on Gentoo but works from the PyPI, you likely to have a typo there. Note: the egg-info dir is created with "python setup.py egg_info" -- Tarek Ziad? | http://ziade.org From missive at hotmail.com Sun Nov 28 17:24:23 2010 From: missive at hotmail.com (Lee Harr) Date: Sun, 28 Nov 2010 20:54:23 +0430 Subject: [Distutils] stdeb gzipping some data_files Message-ID: I am using stdeb (https://github.com/astraw/stdeb) to create debian .deb packages from a python project. (That page says this is the place to ask questions...) stdeb is working quite well, but some of the files that I mention in the data_files part of the setup.py are being gzipped at some point in the process. Is there any way to prevent (hopefully from setup.py, setup.cfg, or stdeb.cfg) the alteration of those files? Thanks for any pointers. From ben+python at benfinney.id.au Sun Nov 28 22:27:00 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 29 Nov 2010 08:27:00 +1100 Subject: [Distutils] stdeb gzipping some data_files References: Message-ID: <87pqtp5ecb.fsf@benfinney.id.au> Lee Harr writes: > stdeb is working quite well, but some of the files that I mention in > the data_files part of the setup.py are being gzipped at some point in > the process. Yes. The Debian packaging policy mandates some files are gzipped; see the policy document for more, otherwise leave the tool to do its work. > Is there any way to prevent (hopefully from setup.py, setup.cfg, or > stdeb.cfg) the alteration of those files? Consider it an opportunity to learn about the Debian packaging policy, if you're interested in why :-) -- \ ?First they came for the verbs, and I said nothing, for verbing | `\ weirds language. Then, they arrival for the nouns and I speech | _o__) nothing, for I no verbs.? ?Peter Ellis | Ben Finney From aljosa.mohorovic at gmail.com Mon Nov 29 02:59:53 2010 From: aljosa.mohorovic at gmail.com (=?UTF-8?B?QWxqb8WhYSBNb2hvcm92acSH?=) Date: Mon, 29 Nov 2010 02:59:53 +0100 Subject: [Distutils] something like pkg==dev in distutils2? Message-ID: pip enables "pip install mypackage==dev" if link (with #egg=mypackage-dev) exists in long_description for pypi. are there some plans for an official way to do this? Aljosa Mohorovic From ziade.tarek at gmail.com Mon Nov 29 10:08:38 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 29 Nov 2010 10:08:38 +0100 Subject: [Distutils] Distutils fails to build In-Reply-To: References: Message-ID: On Mon, Nov 29, 2010 at 12:29 AM, Oisin O'Maley wrote: > There doesn't appear to be anything anything wrong with > "distribute.egg-info/entry_points.txt" It's the same file indeed. .. > The error is thrown on both the Gentoo and PyPi version. ... Ok. I cannot reproduce this. >> >>> import pkg_resources >> >>> >> >>> list(pkg_resources.split_sections(open('distribute.egg-info/entry_points.txt').read())) > > This seems to run fine on both Gentoo and PyPi versions. But running "python > setup.py egg_info" throws the same error as I mentioned above. Can you show us the output ? The error happens because split_sections() returns at some point a (group, lines) where group is None. >>> import pkg_resources >>> for group, lines in pkg_resources.split_sections(open('distribute.egg-info/entry_points.txt').read()): ... if group is None and lines is not None: ... print 'ha!' ... print lines ... You can also add a pdb in the code just before the raise, and call egg_info -- Tarek Ziad? | http://ziade.org From ziade.tarek at gmail.com Mon Nov 29 10:26:37 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 29 Nov 2010 10:26:37 +0100 Subject: [Distutils] something like pkg==dev in distutils2? In-Reply-To: References: Message-ID: On Mon, Nov 29, 2010 at 2:59 AM, Aljo?a Mohorovi? wrote: > pip enables "pip install mypackage==dev" if link (with > #egg=mypackage-dev) exists in long_description for pypi. > are there some plans for an official way to do this? No plans, but if you want this you could work on a proposal for d2 + PEP 345. I didn't look at the pip code for this, but if it works like easy_install, it looks for specific links in the (long_)description of the package that have an #egg=project-dev fragment. If found, it's used to download the dev version and install it. In Distutils2, I would rather have a specific metadata for this, rather than links to look for in the project description. I am not sure about the name, an initial thought would be: Development-Download-URL: A string containing the URL from which the development version of the distribution can be downloaded. (This means that the URL can't be something like ".../BeagleVote-latest.tgz", but instead must be ".../BeagleVote-0.45.tgz".) Note: the development version must be a source distribution. Regards Tarek > > Aljosa Mohorovic > _______________________________________________ > Distutils-SIG maillist ?- ?Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -- Tarek Ziad? | http://ziade.org From ziade.tarek at gmail.com Mon Nov 29 10:31:54 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 29 Nov 2010 10:31:54 +0100 Subject: [Distutils] something like pkg==dev in distutils2? In-Reply-To: References: Message-ID: 2010/11/29 Tarek Ziad? : skip this (bad copy/paste): > ?(This means that the URL can't be something like > ? ".../BeagleVote-latest.tgz", but instead must be ".../BeagleVote-0.45.tgz".) From ozarow at gmail.com Mon Nov 29 11:04:15 2010 From: ozarow at gmail.com (Piotr Ozarowski) Date: Mon, 29 Nov 2010 11:04:15 +0100 Subject: [Distutils] stdeb gzipping some data_files In-Reply-To: References: Message-ID: <20101129100415.GY30169@piotro.eu> [Lee Harr, 2010-11-28] > stdeb is working quite well, but some of the files that I mention > in the data_files part of the setup.py are being gzipped at some > point in the process. add to debian/rules: | override_dh_compress: | dh_compress -X.txt -X.rst -X.foo -- Piotr O?arowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645 From ziade.tarek at gmail.com Mon Nov 29 16:06:10 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 29 Nov 2010 16:06:10 +0100 Subject: [Distutils] Distutils fails to build In-Reply-To: References: Message-ID: > On Mon, Nov 29, 2010 at 1:43 PM, Oisin O'Maley wrote: ... > > ?I would appear distribute.egg-info/entry_points.txt is being read > correctly. I fail to see the problem in Distribute original source then. Did you try that on the original source ? If yes, I fail to understand what's wrong. You need to trace the code using PDB, to find out why this error is raised/ Please keep Distutils-SIG so everyone can participate to the thread Regards Tarek -- Tarek Ziad? | http://ziade.org From chris at simplistix.co.uk Mon Nov 29 21:09:13 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 29 Nov 2010 20:09:13 +0000 Subject: [Distutils] Building Python (Re: An observation on how system packagers and developers can be friends) In-Reply-To: References: Message-ID: <4CF40869.1040305@simplistix.co.uk> On 29/10/2010 11:44, Jim Fulton wrote: > It would be nice if Python's standard build required more dependencies and > had options to turn them off is desired, rather than sniffing to tell > whether they > are there and silently building badly. + lots Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Mon Nov 29 21:16:45 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 29 Nov 2010 20:16:45 +0000 Subject: [Distutils] 64-bit vs 32-bit (Re: An observation on how system packagers and developers can be friends) In-Reply-To: References: Message-ID: <4CF40A2D.6020602@simplistix.co.uk> On 05/11/2010 17:29, Jim Fulton wrote: > This is bad enough, but I really want to restart an app when a memory > error occurs, but MemoryError is a subclass of Exception so it tends > to be caught a lot. I really wish MemoryError was one of those errors > you had to go way out of your way to catch. Another +1 from me ;-) Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Mon Nov 29 21:25:51 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 29 Nov 2010 20:25:51 +0000 Subject: [Distutils] distributing only .pyc with stdeb In-Reply-To: <20101122151343.5a68a76e@mission> References: <4CEA13D5.9010003@uci.cu> <20101122151343.5a68a76e@mission> Message-ID: <4CF40C4F.8080400@simplistix.co.uk> On 22/11/2010 20:13, Barry Warsaw wrote: > On Nov 22, 2010, at 01:55 AM, Karel Antonio Verdecia Ortiz wrote: > >> I'm using stdb to generate .deb packages for ubuntu but I need this packages >> to contain only the compiled modules (.pyc) and not the source modules >> (.py). ?What parameters have I to use to get only the .pyc files? > > Including only the .pyc files would mean that your package was not free > software, and so it would violate the Debian Free Software Guideline. Thus > I'd be surprised if the tool supported that use case easily or directly. Nevermind that .pyc's are but a sneeze away from the actual source anyway ;-) Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From jamesql at mail.com Mon Nov 29 09:37:30 2010 From: jamesql at mail.com (Garciahh) Date: Mon, 29 Nov 2010 00:37:30 -0800 (PST) Subject: [Distutils] disutils2 tests and python 2.5 In-Reply-To: <4CE3041D.8090509@roumenpetrov.info> References: <4CE3041D.8090509@roumenpetrov.info> Message-ID: <30328471.post@talk.nabble.com> Roumen Petrov-3 wrote: > > Hello all, > > I would like to ask to compare/check unit test results of distutils2 > (repository) with someone. > In my build environment the unit test exit without errors: > ---------------- > test_check_metadata (distutils2.tests.test_command_check.CheckTestCase) > ... ok > test_check_restructuredtext > (distutils2.tests.test_command_check.CheckTestCase) ... skipped > 'requires docutils' > test_config (distutils2.tests.test_config.ConfigTestCase) ... > ---------------- > Note that test_config is without ok at the end. > > Did someone get same results with python 2.5 ? > > Roumen > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > Yes, the module testing must be checked up carefully. I also am often engaged in this. ----- http://uk.bestessays.com/ uk essays -- View this message in context: http://old.nabble.com/disutils2-tests-and-python-2.5-tp30234750p30328471.html Sent from the Python - distutils-sig mailing list archive at Nabble.com. From jamesql at mail.com Mon Nov 29 09:41:37 2010 From: jamesql at mail.com (Garciahh) Date: Mon, 29 Nov 2010 00:41:37 -0800 (PST) Subject: [Distutils] What alternatives DJV? Message-ID: <30328499.post@talk.nabble.com> What alternatives For me problemma. Komp of not want to perceive the program DJV. Whether there are alike, which can facilitate work with pictures in DJV? ----- http://uk.bestessays.com/ uk essays -- View this message in context: http://old.nabble.com/What-alternatives-DJV--tp30328499p30328499.html Sent from the Python - distutils-sig mailing list archive at Nabble.com. From jamesql at mail.com Mon Nov 29 09:36:17 2010 From: jamesql at mail.com (Garciahh) Date: Mon, 29 Nov 2010 00:36:17 -0800 (PST) Subject: [Distutils] disutils2 tests and python 2.5 In-Reply-To: <4CE3041D.8090509@roumenpetrov.info> References: <4CE3041D.8090509@roumenpetrov.info> Message-ID: <30328469.post@talk.nabble.com> Yes, the module testing must be checked up carefully. I also am often engaged in this. ----- http://uk.bestessays.com/ uk essays -- View this message in context: http://old.nabble.com/disutils2-tests-and-python-2.5-tp30234750p30328469.html Sent from the Python - distutils-sig mailing list archive at Nabble.com.