From gentoodev at gmail.com Mon Jun 4 19:43:24 2007 From: gentoodev at gmail.com (Rob) Date: Mon, 4 Jun 2007 10:43:24 -0700 Subject: [Distutils] Parsing setup() Message-ID: Hi, I'm writing a dependency resolver and I need to parse the setup() function in setup.py files and get all the packages in the "install_requires = [...]" list. Reading setup.py and using regular expressions work ok but not 100% of the time because people pass a variable sometimes etc. Any hints on how I can go about parsing the setup function? Thanks, Rob From ianb at colorstudy.com Mon Jun 4 19:59:56 2007 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 04 Jun 2007 12:59:56 -0500 Subject: [Distutils] Parsing setup() In-Reply-To: References: Message-ID: <4664531C.5000305@colorstudy.com> Rob wrote: > Hi, > > I'm writing a dependency resolver and I need to parse the setup() function > in setup.py files and get all the packages in the "install_requires = > [...]" list. > > Reading setup.py and using regular expressions work ok but not 100% of > the time because people pass a variable sometimes etc. > > Any hints on how I can go about parsing the setup function? If you have a built egg (which you can do with python setup.py egg_info, without actually installing anything) you can read the requires.txt file with pkg_resources. -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org | Write code, do good | http://topp.openplans.org/careers From gentoodev at gmail.com Tue Jun 5 00:58:15 2007 From: gentoodev at gmail.com (Rob) Date: Mon, 4 Jun 2007 15:58:15 -0700 Subject: [Distutils] Parsing setup() Message-ID: Ian Bicking wrote: > If you have a built egg (which you can do with python setup.py egg_info, without actually installing > anything) you can read the requires.txt file with pkg_resources. I had tried that but I was having problems with un-installed packages. I have this so far, which works with *nux, but I'll have to figure out how to find the correct Python executable in Windows (I think using different Python versions could yield different requirements?). pkg_name = "pycopia-net" os.chdir(setup_dir) os.system("/usr/bin/env python %s egg_info" % setup_file) ws = WorkingSet(setup_dir) env = Environment() dist = env.best_match(Requirement.parse(pkg_name), ws) print dist.get_metadata("requires.txt").split() ['pycopia-process>=0.9', 'pycopia-utils>=0.9'] Thanks, Ian. From robert.kern at gmail.com Tue Jun 5 01:19:06 2007 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 04 Jun 2007 18:19:06 -0500 Subject: [Distutils] Parsing setup() In-Reply-To: References: Message-ID: Rob wrote: > Ian Bicking wrote: > >> If you have a built egg (which you can do with python setup.py egg_info, without actually installing >> anything) you can read the requires.txt file with pkg_resources. > > I had tried that but I was having problems with un-installed packages. > > I have this so far, which works with *nux, but I'll have to figure out > how to find the correct Python > executable in Windows (I think using different Python versions could > yield different requirements?). > > > pkg_name = "pycopia-net" > os.chdir(setup_dir) > os.system("/usr/bin/env python %s egg_info" % setup_file) You probably want to use sys.executable (and the subprocess module if you have Python >= 2.4): import subprocess import sys subprocess.call([sys.executable, 'setup.py', 'egg_info']) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From pje at telecommunity.com Tue Jun 5 01:22:11 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 04 Jun 2007 19:22:11 -0400 Subject: [Distutils] Parsing setup() In-Reply-To: References: Message-ID: <20070604232137.BAAFF3A407F@sparrow.telecommunity.com> At 03:58 PM 6/4/2007 -0700, Rob wrote: >Ian Bicking wrote: > > > If you have a built egg (which you can do with python setup.py > egg_info, without actually installing > > anything) you can read the requires.txt file with pkg_resources. > >I had tried that but I was having problems with un-installed packages. > >I have this so far, which works with *nux, but I'll have to figure out >how to find the correct Python >executable in Windows (I think using different Python versions could >yield different requirements?). > > >pkg_name = "pycopia-net" >os.chdir(setup_dir) >os.system("/usr/bin/env python %s egg_info" % setup_file) If you use "-e sometmpdir" as an option, you can have the .egg-info put in a temporary directory with a known location. You need this because some package layouts don't have the .egg-info in the package root, so the rest of your stuff won't work. To be honest, though, I think you're going to way too much trouble here. Just what exactly is the use case here? Couldn't you just easy_install the package to a temporary directory with --no-deps? Why do you think you need to inspect the setup.py in the first place? By the way, your code here is broken even on *nux systems if the user runs your tool with a different Python than the system one. sys.executable gives you the full path to the currently-running version of Python, irrespective of platform. (You might also want to consider using the setuptools.sandbox.run_setup() function instead of os.system.) >ws = WorkingSet(setup_dir) >env = Environment() >dist = env.best_match(Requirement.parse(pkg_name), ws) >print dist.get_metadata("requires.txt").split() > >['pycopia-process>=0.9', 'pycopia-utils>=0.9'] Just use dist.requires() here, and then you won't be dependent on internal details of the format. In general, I would suggest that trying to work your way around setuptools' APIs is a very bad idea. More often than not, you will miss one of the nine jillion supported configurations and thus will end up with a tool that only works on your system... *sometimes*. Case in point: your code above depends on the .egg-info being in the setup directory, which is often not the case. Zope packages, for example, usually have a src/ or lib/ subdirectory that will be where the egg-info is. Other packages may have it in other places, perhaps even more deeply nested. Pretty much *any* place where you are reading a file format directly or looking at directory contents yourself, you're probably breaking something that setuptools has encapsulated for a Very Good Reason. When in doubt, ask. :) From gentoodev at gmail.com Tue Jun 5 02:07:05 2007 From: gentoodev at gmail.com (Rob) Date: Mon, 4 Jun 2007 17:07:05 -0700 Subject: [Distutils] Parsing setup() Message-ID: Phillip J. Eby wrote: > If you use "-e sometmpdir" as an option, you can have the .egg-info put > in a temporary directory with a known location. You need this because > some package layouts don't have the .egg-info in the package root, so > the rest of your stuff won't work. > > To be honest, though, I think you're going to way too much trouble > here. Just what exactly is the use case here? Couldn't you just > easy_install the package to a temporary directory with --no-deps? Why > do you think you need to inspect the setup.py in the first place? > Sorry, I should have explained what I was attempting. The code is going to be used for g-pypi[1], a tool I wrote specifically for creating ebuilds for Gentoo's portage package manager. It creates ebuilds by using setuptools and querying The Cheese Shop. We still use setuptools/distutils normally, I'm just grabbing the dependency info to help create the ebuilds which still have to have all the dependency info in them (portage doesn't use easy_install). > (You might also want to consider using the > setuptools.sandbox.run_setup() function instead of os.system.) > That looks much better, I'll check that out. > In general, I would suggest that trying to work your way around > setuptools' APIs is a very bad idea. More often than not, you will miss > one of the nine jillion supported configurations and thus will end up > with a tool that only works on your system... *sometimes*. > > Case in point: your code above depends on the .egg-info being in the > setup directory, which is often not the case. Zope packages, for > example, usually have a src/ or lib/ subdirectory that will be where the > egg-info is. Other packages may have it in other places, perhaps even > more deeply nested. > > Pretty much *any* place where you are reading a file format directly or > looking at directory contents yourself, you're probably breaking > something that setuptools has encapsulated for a Very Good Reason. When > in doubt, ask. :) > I'll keep all that in mind. I was thinking of making the code more general in the future in case someone else wanted to use it for a different package manager and needed to find out all the dependencies before creating rpm's or debs etc. So far it's working good - it's better than downloading, unpacking and reading all the setup.py's by hand, which we'd been doing ;) Thanks, Phillip! [1] g-pypi: http://tools.assembla.com/g-pypi From pje at telecommunity.com Tue Jun 5 02:58:45 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 04 Jun 2007 20:58:45 -0400 Subject: [Distutils] Parsing setup() In-Reply-To: References: Message-ID: <20070605011054.0080B3A4060@sparrow.telecommunity.com> At 05:07 PM 6/4/2007 -0700, Rob wrote: >Phillip J. Eby wrote: > > If you use "-e sometmpdir" as an option, you can have the .egg-info put > > in a temporary directory with a known location. You need this because > > some package layouts don't have the .egg-info in the package root, so > > the rest of your stuff won't work. > > > > To be honest, though, I think you're going to way too much trouble > > here. Just what exactly is the use case here? Couldn't you just > > easy_install the package to a temporary directory with --no-deps? Why > > do you think you need to inspect the setup.py in the first place? > > > >Sorry, I should have explained what I was attempting. > >The code is going to be used for g-pypi[1], a tool I wrote specifically for >creating ebuilds for Gentoo's portage package manager. It creates ebuilds >by using setuptools and querying The Cheese Shop. In that case, using "easy_install -mNd atempdir sourcedir" is probably your best recipe. After easy_install runs, "atempdir" will contain an egg built from "sourcedir", and you can then query it via the standard API. You don't even need to run it with system(); here's your recipe: from setuptools.command.easy_install import main as easy from pkg_resources import find_distributions easy(['-Nmxd', atempdir, sourcedir]) for dist in find_distributions(atempdir): requirements = dist.requires() # ...and whatever atempdir should be an empty directory prior to the easy-install invocation, of course. The -N means no dependencies, -m means multi-version (no .pth file), -x excludes scripts, and -d sets the target directory. From ryanjoneil at gmail.com Tue Jun 5 19:03:57 2007 From: ryanjoneil at gmail.com (Ryan O'Neil) Date: Tue, 5 Jun 2007 13:03:57 -0400 Subject: [Distutils] requiring python 2.5 Message-ID: My apologies if this was already covered; I couldn't find appropriate mention of the issue. What is the best way to require Python 2.5 in setuptools? In PyGEP 0.1.2 I did the following: setup( # --- snip --- install_requires = ['python>=2.5'], # --- snip --- ) This worked wonderfully on Linux using setuptools 0.6c5, but on Windows (also with Python 2.5 and setuptools 0.6c5), it couldn't recognize the installed version of Python and tried to download and compile it (python) from source. Granted this seems like the wrong thing to do. I was able to get things mostly working by getting rid of the install_requires line and adding this code to my setup.py: import sys if sys.version_info[:2] < (2, 5): raise Exception('PyGEP requires Python 2.5') However, easy_install will now download PyGEP for Python 2.4 and fail during setup, which is less than ideal (and ends up with cheesecaking slandering its installability ;). Am I just doing everything completely wrong? There is some more information about the error here: http://code.google.com/p/pygep/issues/detail?id=7 Thanks for your help and thanks for setuptools! -r From ianb at colorstudy.com Tue Jun 5 19:35:20 2007 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 05 Jun 2007 12:35:20 -0500 Subject: [Distutils] Setuptools bug with tag_svn_revision Message-ID: <46659ED8.6040001@colorstudy.com> If you try to do egg_info in a package that is in svn, but where any directory or subdirectory in the package has been added but not committed to svn, you'll get this error: Traceback (most recent call last): File "setup.py", line 25, in ? entry_points=""" File "distutils/core.py", line 149, in setup File "/usr/lib/python2.4/distutils/dist.py", line 946, in run_commands self.run_command(cmd) File "/usr/lib/python2.4/distutils/dist.py", line 965, in run_command cmd_obj.ensure_finalized() File "/usr/lib/python2.4/cmd.py", line 117, in ensure_finalized pass File "/home/ianb/env/webdev/lib/python2.4/setuptools-0.6c6-py2.4.egg/setuptools/command/egg_info.py", line 85, in finalize_options self.vtags = self.tags() File "/home/ianb/env/webdev/lib/python2.4/setuptools-0.6c6-py2.4.egg/setuptools/command/egg_info.py", line 179, in tags ): version += '-r%s' % self.get_svn_revision() File "/home/ianb/env/webdev/lib/python2.4/setuptools-0.6c6-py2.4.egg/setuptools/command/egg_info.py", line 227, in get_svn_revision localrev = max([int(m.group(1)) for m in revre.finditer(data)]) ValueError: max() arg is an empty sequence -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org | Write code, do good | http://topp.openplans.org/careers From pje at telecommunity.com Tue Jun 5 22:58:21 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 05 Jun 2007 16:58:21 -0400 Subject: [Distutils] requiring python 2.5 In-Reply-To: References: Message-ID: <20070605205625.77F4D3A4060@sparrow.telecommunity.com> At 01:03 PM 6/5/2007 -0400, Ryan O'Neil wrote: >My apologies if this was already covered; I couldn't find appropriate >mention of the issue. > >What is the best way to require Python 2.5 in setuptools? In PyGEP >0.1.2 I did the following: > > setup( > # --- snip --- > install_requires = ['python>=2.5'], > # --- snip --- > ) > >This worked wonderfully on Linux using setuptools 0.6c5, but on >Windows (also with Python 2.5 and setuptools 0.6c5), it couldn't >recognize the installed version of Python and tried to download and >compile it (python) from source. Interesting; it sounds like there might be a problem with what files are being installed on Windows; Python 2.5 is supposed to include a Python.egg-info file that indicates Python 2.5 is installed. However, it's probably the case that EasyInstall should not try to install Python from source, regardless. :) >I was able to get things mostly working by getting rid of the >install_requires line and adding this code to my setup.py: > > import sys > > if sys.version_info[:2] < (2, 5): > raise Exception('PyGEP requires Python 2.5') I'd recommend using either a DistutilsError subclass or SystemExit; both will be handled more gracefully by easy_install. >However, easy_install will now download PyGEP for Python 2.4 and fail >during setup, which is less than ideal (and ends up with cheesecaking >slandering its installability ;). Am I just doing everything >completely wrong? No; at the moment the infrastructure for requiring a particular version of Python just isn't that great. From peter.mowry at amd.com Thu Jun 7 01:28:38 2007 From: peter.mowry at amd.com (Mowry, Peter) Date: Wed, 6 Jun 2007 18:28:38 -0500 Subject: [Distutils] I load entry point ".so", then it only loads 60 / 63 ".so" files (3 fail?) :-( Message-ID: <475F4CA05E4E1B46A7213674E70EFA6202823CC2@SAUSEXMB2.amd.com> I am using ctypes to control a computer hardware simulation via a single shared library entry point, but having trouble getting it to fully load all the .so files on Linux :-( ctypes directly loads my entry point ".so" shared library: self.mainentrylib = ctypes.CDLL(strPathAbs, mode=ctypes.RTLD_GLOBAL) # load main entry point # I don't know why I have to use RTLD_GLOBAL, but when I do 60 / 63 work Then I call an exported function on self.mainentrylib, which causes the entry point shared library to load a bunch of model pieces (from the C++ code) (from a "libs" dir). On Windows, all 63 pieces (".dll" files) load without problem. On Linux, only 60 of the 63 pieces (".so" files) load, and 3 of them give me an error trying to load. I modified the C++ code (of the main entry point shared library) to print GetLastError(), which reported 2, which translates via strerror(2) to "No such file or directory". But this error message can't be true b/c the file does exist (and it's in the same dir as the other 60). I didn't think anything was special about these 3 shared libraries, but something must be. Anyone run into something similar or have any ideas? Any responses appreciated; thanks! From rlratzel at enthought.com Sat Jun 9 04:26:18 2007 From: rlratzel at enthought.com (Rick Ratzel) Date: Fri, 8 Jun 2007 21:26:18 -0500 (CDT) Subject: [Distutils] requiring python 2.5 In-Reply-To: <20070605205625.77F4D3A4060@sparrow.telecommunity.com> (pje@telecommunity.com) References: <20070605205625.77F4D3A4060@sparrow.telecommunity.com> Message-ID: <20070609022618.5C1441DF4FD@mail.enthought.com> Is there a way to have my package tell setuptools to "ignore" certain other packages? I'm working on a project called Enstaller, which basically sits on top of setuptools and adds functionality like query options, a GUI, etc. I'm attempting to make it a standalone "bundled" egg, where all of it's dependencies are included in 2 eggs: the "core" egg, and the optional GUI egg. Prior to this, it was distributed as a single egg with several dependencies specified in the install_requires line which were automatically installed. The problem with that was that these packages were used for many other related projects, and when users would upgrade, downgrade, or unknowingly remove them, the Enstaller app would be left broken. The "bundled" approach seemed to work great: it was installed with -m so other projects would not import the possibly incompatible packages in enstaller, and only one or two eggs were installed instead of the intimidating dozen or so. A console_script entry point allowed users to type "enstaller" to start the app. The bundled egg is structured like this: enstaller.egg/ enthought/ enstaller/ traits/ Here, I've "bundled" in the enthought.traits package. The problem is, when I install the enthought.traits egg separately, python finds the traits package in the enthought.traits egg instead of the the one in the enstaller egg. I've tried removing all the "enthought." eggs from sys.path before anything else gets imported, but even that doesn't work (and it just felt wrong, and probably is). I looked at the docs for manipulating the WorkingSet, but made no progress there either...python always managed to find the incompatible enthought.traits code in the enthought.traits egg. When I remove the incompatible enthought.traits, all works fine. What I'd like to do is simply say "do not use any enthought.* eggs". Is this possible? Thanks in advance! -- Rick Ratzel - Enthought, Inc. 515 Congress Avenue, Suite 2100 - Austin, Texas 78701 512-536-1057 x229 - Fax: 512-536-1059 http://www.enthought.com From pje at telecommunity.com Sat Jun 9 22:04:45 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 09 Jun 2007 16:04:45 -0400 Subject: [Distutils] requiring python 2.5 In-Reply-To: <20070609022618.5C1441DF4FD@mail.enthought.com> References: <20070605205625.77F4D3A4060@sparrow.telecommunity.com> <20070609022618.5C1441DF4FD@mail.enthought.com> Message-ID: <20070609201545.977523A4060@sparrow.telecommunity.com> At 09:26 PM 6/8/2007 -0500, Rick Ratzel wrote: > Is there a way to have my package tell setuptools to "ignore" > certain other >packages? Only through version specifications; i.e., request the exact versions you want. > I've tried removing all the "enthought." eggs from sys.path > before anything >else gets imported, but even that doesn't work (and it just felt wrong, and >probably is). If "enthought" is a namespace package, you would need to either be using an 0.7-development version of setuptools (i.e., a trunk SVN version), or else you'd need to remove those eggs from the path *before* pkg_resources is first imported. Otherwise, it would indeed not work. >I looked at the docs for manipulating the WorkingSet, but made no >progress there either...python always managed to find the incompatible >enthought.traits code in the enthought.traits egg. When I remove the >incompatible enthought.traits, all works fine. > > What I'd like to do is simply say "do not use any enthought.* > eggs". Is this >possible? Thanks in advance! Not without using an 0.7 version of setuptools, or manipulating enthought.__path__. The straightforward way to do it would be for enstaller's __init__.py to do this: import enthought, os enthought.__path__ = os.path.dirname(os.path.dirname(__file__)) This will ensure that all enthought.* modules imported from then on will only be from within the parent directory of enstaller. I don't think I'd recommend this to anyone in the general case, but this might be a reasonable workaround if you truly want to "de-namespace" the package. (It will *not*, however, prevent new eggs from being added to __path__ if you add anything to sys.path or the working set afterward.) From rlratzel at enthought.com Sun Jun 10 08:45:32 2007 From: rlratzel at enthought.com (Rick Ratzel) Date: Sun, 10 Jun 2007 01:45:32 -0500 (CDT) Subject: [Distutils] "ignoring" eggs (was: Re: requiring python 2.5) In-Reply-To: <20070609201545.977523A4060@sparrow.telecommunity.com> (pje@telecommunity.com) References: <20070605205625.77F4D3A4060@sparrow.telecommunity.com> <20070609022618.5C1441DF4FD@mail.enthought.com> <20070609201545.977523A4060@sparrow.telecommunity.com> Message-ID: <20070610064532.498491DF52A@mail.enthought.com> (sorry about the subject line...I meant to start a new thread) Phillip, Thanks for the response. I implemented your recommendation and it did exactly what I needed it to do. More comments inline: > Date: Sat, 09 Jun 2007 16:04:45 -0400 > From: "Phillip J. Eby" > > At 09:26 PM 6/8/2007 -0500, Rick Ratzel wrote: > > > Is there a way to have my package tell setuptools to "ignore" > > certain other > >packages? > > Only through version specifications; i.e., request the exact versions you want. Yep, that's initially what I wanted to do since it was the "standard" way to do it. Unfortunatley, the packages that enstaller required were just recently "egg'd", and most of them had broken dependencies and stale version numbers. Many of the interdependencies did not even specify versions, so an old, incompatible version that was already installed satisfied the dependency but left enstaller broken. Many of the packages are now fixed (we've been undergoing a huge effort to convert our OSS repository to an egg-based deployment), but I was later convinced that it might be a good idea to keep enstaller isolated from the other eggs. Plus, enstaller[gui] requires about 13 eggs, almost all of which are used by other projects too. Users would "uninstall" their eggs and not realize that they were breaking enstaller. The monolithic egg approach seems to have solved all these problems. Unfortunately the eggs are obviously much bigger and downloading will be more expensive, especially for the enstaller.gui egg, which is installed by default by a bootstrapping script. > > I've tried removing all the "enthought." eggs from sys.path > > before anything > >else gets imported, but even that doesn't work (and it just felt wrong, and > >probably is). > > If "enthought" is a namespace package, you would need to either be > using an 0.7-development version of setuptools (i.e., a trunk SVN > version), or else you'd need to remove those eggs from the path > *before* pkg_resources is first imported. Otherwise, it would indeed not work. > > > >I looked at the docs for manipulating the WorkingSet, but made no > >progress there either...python always managed to find the incompatible > >enthought.traits code in the enthought.traits egg. When I remove the > >incompatible enthought.traits, all works fine. > > > > What I'd like to do is simply say "do not use any enthought.* > > eggs". Is this > >possible? Thanks in advance! > > Not without using an 0.7 version of setuptools, or manipulating > enthought.__path__. The straightforward way to do it would be for > enstaller's __init__.py to do this: > > import enthought, os > enthought.__path__ = os.path.dirname(os.path.dirname(__file__)) > > This will ensure that all enthought.* modules imported from then on > will only be from within the parent directory of enstaller. I ended up setting enthought.__path__ like you suggested, and will probably stick with that rather than force users to upgrade to 0.7 right now. However, I'm curious about how this is addressed in 0.7. Is there a new API for doing this? > I don't think I'd recommend this to anyone in the general case, but > this might be a reasonable workaround if you truly want to > "de-namespace" the package. (It will *not*, however, prevent new > eggs from being added to __path__ if you add anything to sys.path or > the working set afterward.) Right. Fortunately, all the path "magic" is contained to this change, so I should be safe. Thanks again, -- Rick Ratzel - Enthought, Inc. 515 Congress Avenue, Suite 2100 - Austin, Texas 78701 512-536-1057 x229 - Fax: 512-536-1059 http://www.enthought.com From pje at telecommunity.com Sun Jun 10 20:21:55 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 10 Jun 2007 14:21:55 -0400 Subject: [Distutils] "ignoring" eggs (was: Re: requiring python 2.5) In-Reply-To: <20070610064532.498491DF52A@mail.enthought.com> References: <20070605205625.77F4D3A4060@sparrow.telecommunity.com> <20070609022618.5C1441DF4FD@mail.enthought.com> <20070609201545.977523A4060@sparrow.telecommunity.com> <20070610064532.498491DF52A@mail.enthought.com> Message-ID: <20070610181954.EC6AF3A407F@sparrow.telecommunity.com> At 01:45 AM 6/10/2007 -0500, Rick Ratzel wrote: > I ended up setting enthought.__path__ like you suggested, and > will probably >stick with that rather than force users to upgrade to 0.7 right now. However, >I'm curious about how this is addressed in 0.7. Is there a new API for doing >this? No; in 0.7 namespace packages are not automatically imported/created when pkg_resources is imported. This means you would have the chance to manipulate sys.path and/or the working_set before the other eggs could get added to enthought.__path__. From alet at librelogiciel.com Mon Jun 11 19:12:15 2007 From: alet at librelogiciel.com (Jerome Alet) Date: Mon, 11 Jun 2007 19:12:15 +0200 Subject: [Distutils] real vs effective user id Message-ID: <20070611171215.GA20770@mail.librelogiciel.com> Hi there, I've got an application which runs as root This application calls seteuid to execute part of its code as a lower priviledged user temporarily. after the seteuid call, the MySQLdb module is imported, which was installed with setuptools 0.6c6. then setuptools tries to play with root's .python-eggs directory instead of ~effective_user/.python-eggs/, because of the way os.path.expanduser() works (line 1039 in pkg_resource.py) is this a bug in setuptools, in Python, or more probably in my own application ? shouldn't Python's os.path.expanduser() call use the effective user id instead of the real one ? thanks in advance for any hint. bye Jerome Alet From pje at telecommunity.com Mon Jun 11 19:57:38 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 11 Jun 2007 13:57:38 -0400 Subject: [Distutils] real vs effective user id In-Reply-To: <20070611171215.GA20770@mail.librelogiciel.com> References: <20070611171215.GA20770@mail.librelogiciel.com> Message-ID: <20070611175747.246683A4080@sparrow.telecommunity.com> At 07:12 PM 6/11/2007 +0200, Jerome Alet wrote: >Hi there, > >I've got an application which runs as root > >This application calls seteuid to execute part of its code as a lower >priviledged user temporarily. > >after the seteuid call, the MySQLdb module is imported, which was installed >with setuptools 0.6c6. > >then setuptools tries to play with root's .python-eggs directory instead >of ~effective_user/.python-eggs/, because of the way os.path.expanduser() >works (line 1039 in pkg_resource.py) In these types of situations, it's best to set the PYTHON_EGG_CACHE path explicitly, either via the environment variable or code. From alet at librelogiciel.com Mon Jun 11 21:09:36 2007 From: alet at librelogiciel.com (Jerome Alet) Date: Mon, 11 Jun 2007 21:09:36 +0200 Subject: [Distutils] [jerome: Re: real vs effective user id] Message-ID: <20070611190936.GA24748@mail.librelogiciel.com> Sorry, I forgot to CC the list. -------------- next part -------------- An embedded message was scrubbed... From: unknown sender Subject: Re: [Distutils] real vs effective user id Date: Mon, 11 Jun 2007 20:45:05 +0200 Size: 1879 Url: http://mail.python.org/pipermail/distutils-sig/attachments/20070611/23dfd2bb/attachment.mht From pje at telecommunity.com Mon Jun 11 22:00:34 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 11 Jun 2007 16:00:34 -0400 Subject: [Distutils] [jerome: Re: real vs effective user id] In-Reply-To: <20070611190936.GA24748@mail.librelogiciel.com> References: <20070611190936.GA24748@mail.librelogiciel.com> Message-ID: <20070611195834.3B4AF3A407F@sparrow.telecommunity.com> At 09:09 PM 6/11/2007 +0200, Jerome Alet wrote: >On Mon, Jun 11, 2007 at 01:57:38PM -0400, Phillip J. Eby wrote: > > > > > >then setuptools tries to play with root's .python-eggs directory instead > > >of ~effective_user/.python-eggs/, because of the way os.path.expanduser() > > >works (line 1039 in pkg_resource.py) > > > > In these types of situations, it's best to set the PYTHON_EGG_CACHE > > path explicitly, either via the environment variable or code. > >This is effectively what I had to do to fix the problem for my >client, however this was completely unexpected because I can't >control how different people install third party modules like this >one. Considering that some imports will be done before the seteuid() >call and some after, setting the environment variable would have to >be done twice (for example), otherwise you might end up with root >owned directories into an otherwise user-owned directory, and this >might introduce further permission problems on different apps >run by the same unpriviledged user but requiring the same modules. > >Wouldn't it be better to use the effective user id when expanding >the user's home directory in setuptools or in Python ? That wouldn't fix your problem. The egg cache location is determined exactly *once* for the life of the process, unless you explicitly create and use ResourceManagers -- and unfortunately the process of importing C code from a zipped egg uses only the default ResourceManager. From alet at librelogiciel.com Mon Jun 11 22:18:54 2007 From: alet at librelogiciel.com (Jerome Alet) Date: Mon, 11 Jun 2007 22:18:54 +0200 Subject: [Distutils] [jerome: Re: real vs effective user id] In-Reply-To: <20070611195834.3B4AF3A407F@sparrow.telecommunity.com> References: <20070611190936.GA24748@mail.librelogiciel.com> <20070611195834.3B4AF3A407F@sparrow.telecommunity.com> Message-ID: <20070611201854.GA24894@mail.librelogiciel.com> On Mon, Jun 11, 2007 at 04:00:34PM -0400, Phillip J. Eby wrote: > > > >Wouldn't it be better to use the effective user id when expanding > >the user's home directory in setuptools or in Python ? > > That wouldn't fix your problem. The egg cache location is determined > exactly *once* for the life of the process, unless you explicitly > create and use ResourceManagers -- and unfortunately the process of > importing C code from a zipped egg uses only the default ResourceManager. So you mean that if for example another (different) module installed with setuptools was imported *before* the seteuid() call, then the mysqldb module import (done *after* the seteuid() call) would succeed ? Not sure if it was clear in my previous message, but the problem is that the import statement fails, while the very same import done when the very same module was installed the 'classic' (not through setuptools) way succeeds, because a 'classic' install import doesn't try to write to an user's home directory. How do you want developers to plan for this sort of things ? Then maybe setuptools should continue to work without creating/writing any file in an user's .python-eggs directory if this is not allowed to do so because of permissions (or different effective user). Instead it fails hard. Any plan to improve this ? NB : I don't know if this is possible or not, I just ask. bye Jerome Alet From pje at telecommunity.com Mon Jun 11 23:02:23 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 11 Jun 2007 17:02:23 -0400 Subject: [Distutils] [jerome: Re: real vs effective user id] In-Reply-To: <20070611201854.GA24894@mail.librelogiciel.com> References: <20070611190936.GA24748@mail.librelogiciel.com> <20070611195834.3B4AF3A407F@sparrow.telecommunity.com> <20070611201854.GA24894@mail.librelogiciel.com> Message-ID: <20070611210022.EED483A407F@sparrow.telecommunity.com> At 10:18 PM 6/11/2007 +0200, Jerome Alet wrote: >On Mon, Jun 11, 2007 at 04:00:34PM -0400, Phillip J. Eby wrote: > > > > > >Wouldn't it be better to use the effective user id when expanding > > >the user's home directory in setuptools or in Python ? > > > > That wouldn't fix your problem. The egg cache location is determined > > exactly *once* for the life of the process, unless you explicitly > > create and use ResourceManagers -- and unfortunately the process of > > importing C code from a zipped egg uses only the default ResourceManager. > >So you mean that if for example another (different) module installed >with setuptools was imported *before* the seteuid() call, then the >mysqldb module import (done *after* the seteuid() call) would >succeed ? No. I'm saying that the directory for the default ResourceManager's cache is determined exactly once, and can't be changed thereafter for the effective life of the process. Thus, you must choose a location that is readable and writable by every user id that the process will be executing as, whether real or effective. >Then maybe setuptools should continue to work without creating/writing >any file in an user's .python-eggs directory if this is not allowed >to do so because of permissions (or different effective user). > >Instead it fails hard. Any plan to improve this ? You have several options. First, setuptools can build RPMs which install files in the usual way. Second, you can install eggs containing C code as directories instead of files (using easy_install --always-unzip), which then avoids the need for the use of the egg cache. Third, you can select an appropriate cache directory, either with the environment variable, or programmatically. From alet at librelogiciel.com Tue Jun 12 01:33:38 2007 From: alet at librelogiciel.com (Jerome Alet) Date: Tue, 12 Jun 2007 01:33:38 +0200 Subject: [Distutils] [jerome: Re: real vs effective user id] In-Reply-To: <20070611210022.EED483A407F@sparrow.telecommunity.com> References: <20070611190936.GA24748@mail.librelogiciel.com> <20070611195834.3B4AF3A407F@sparrow.telecommunity.com> <20070611201854.GA24894@mail.librelogiciel.com> <20070611210022.EED483A407F@sparrow.telecommunity.com> Message-ID: <20070611233338.GA29997@mail.librelogiciel.com> On Mon, Jun 11, 2007 at 05:02:23PM -0400, Phillip J. Eby wrote: > > > >So you mean that if for example another (different) module installed > >with setuptools was imported *before* the seteuid() call, then the > >mysqldb module import (done *after* the seteuid() call) would > >succeed ? > > No. I'm saying that the directory for the default ResourceManager's > cache is determined exactly once, and can't be changed thereafter for > the effective life of the process. Thus, you must choose a location > that is readable and writable by every user id that the process will > be executing as, whether real or effective. The problem is that the setuptools code doesn't fail because the expanduser() call fails, it fails (during my import of the mysqldb this is setuptools' code which fails) because it tries to read from or write to a protected directory. It's IMHO a design problem with setuptools, which should do its best when permissions prevent it to run as expected. For example "standard" Python .py files are only compiled onto the harddisk as .pyc if permissions permit. If the process is started by root, there's no way I can tell the users of my software to make ~root/.python-eggs writable (or even readable) by the user this app will do a seteuid call to. It's basic security stuff. (and yes there are valid reasons why this software needs to be started as root) > >Then maybe setuptools should continue to work without creating/writing > >any file in an user's .python-eggs directory if this is not allowed > >to do so because of permissions (or different effective user). > > > >Instead it fails hard. Any plan to improve this ? > > You have several options. First, setuptools can build RPMs which > install files in the usual way. Second, you can install eggs > containing C code as directories instead of files (using easy_install > --always-unzip), which then avoids the need for the use of the egg > cache. Third, you can select an appropriate cache directory, either > with the environment variable, or programmatically. I think you misunderstand the problem : 1 - I don't use setuptools as part of the installation of my own software. 2 - My own software relies on third party modules which may, or may not, be installed through setuptools. I can't know if they are, or which options (in the ones you list above) will be chosen by the administrators who install these third party modules. I'm in no position to dictate what admins have to do with regard to software I didn't wrote (mysqldb for example) or that I don't even use (setuptools). 3 - My own software, which runs perfectly for hundreds of customers, failed miserably for one of them because he chose to install a database driver with setuptools, for the reasons explained previously. It could have been a different module, and happen at a different import statement in my code (the app is made of several command line tools). 4 - My code basically does : ... groupid = 25 userid = 33 os.setegid(groupid) os.seteuid(userid) ... and so can revert to being root again when needed by calling : os.seteuid(0) os.setegid(0) I know this is not entirely safe from a security point of view, but this is to prevent accidents, not crackers. Is this a bug in my code ? I don't think so, it's valid code. 5 - Why would I need to put ugly hacks (environment variable) in my own code ? As seen earlier this probably can't be done safely since the directory will be evaluated only once (I could always set it to /tmp/username/.python-eggs to mitigate the problem, but this sucks big time IMHO). 6 - Besides not having to rewrite the expanduser() method, is there a valid reason why setuptools checks with the real user's home directory instead of with the effective one ? If not, then I propose to write the patch. If yes, then which one ? Any comment ? TIA Jerome Alet From pje at telecommunity.com Tue Jun 12 01:51:27 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 11 Jun 2007 19:51:27 -0400 Subject: [Distutils] [jerome: Re: real vs effective user id] In-Reply-To: <20070611233338.GA29997@mail.librelogiciel.com> References: <20070611190936.GA24748@mail.librelogiciel.com> <20070611195834.3B4AF3A407F@sparrow.telecommunity.com> <20070611201854.GA24894@mail.librelogiciel.com> <20070611210022.EED483A407F@sparrow.telecommunity.com> <20070611233338.GA29997@mail.librelogiciel.com> Message-ID: <20070611234926.CE2053A407F@sparrow.telecommunity.com> At 01:33 AM 6/12/2007 +0200, Jerome Alet wrote: >On Mon, Jun 11, 2007 at 05:02:23PM -0400, Phillip J. Eby wrote: > > > > > >So you mean that if for example another (different) module installed > > >with setuptools was imported *before* the seteuid() call, then the > > >mysqldb module import (done *after* the seteuid() call) would > > >succeed ? > > > > No. I'm saying that the directory for the default ResourceManager's > > cache is determined exactly once, and can't be changed thereafter for > > the effective life of the process. Thus, you must choose a location > > that is readable and writable by every user id that the process will > > be executing as, whether real or effective. > >The problem is that the setuptools code doesn't fail because the >expanduser() call fails, it fails (during my import of the mysqldb >this is setuptools' code which fails) because it tries to read from or >write to a protected directory. ...which is why the cache location has to be readable and writable by every user the process will be executing as. >It's IMHO a design problem with setuptools, which should do its best >when permissions prevent it to run as expected. When it doesn't have permission to write to a cache directory, it's game over. Currently, there is no other way to execute a .so, .dll, .pyd, or other dynamic library from inside a zipfile, except for some GPL code that can do it on Windows (and which I can't mix with setuptools). >For example "standard" >Python .py files are only compiled onto the harddisk as .pyc if permissions >permit. This isn't an analagous situation; without unpacking the .so file, the driver isn't going to be loadable. > 6 - Besides not having to rewrite the expanduser() method, is > there a valid reason why setuptools checks with the real > user's home directory instead of with the effective one ? > If not, then I propose to write the patch. If yes, then > which one ? A patch to replace expanduser would be fine; please make sure, however, that it falls back to use of expanduser in the event of an error. From alet at librelogiciel.com Tue Jun 12 09:46:40 2007 From: alet at librelogiciel.com (Jerome Alet) Date: Tue, 12 Jun 2007 09:46:40 +0200 Subject: [Distutils] [jerome: Re: real vs effective user id] In-Reply-To: <20070611234926.CE2053A407F@sparrow.telecommunity.com> References: <20070611190936.GA24748@mail.librelogiciel.com> <20070611195834.3B4AF3A407F@sparrow.telecommunity.com> <20070611201854.GA24894@mail.librelogiciel.com> <20070611210022.EED483A407F@sparrow.telecommunity.com> <20070611233338.GA29997@mail.librelogiciel.com> <20070611234926.CE2053A407F@sparrow.telecommunity.com> Message-ID: <20070612074640.GA8347@mail.librelogiciel.com> Hello, On Mon, Jun 11, 2007 at 07:51:27PM -0400, Phillip J. Eby wrote: > > 6 - Besides not having to rewrite the expanduser() method, is > > there a valid reason why setuptools checks with the real > > user's home directory instead of with the effective one ? > > If not, then I propose to write the patch. If yes, then > > which one ? > > A patch to replace expanduser would be fine; please make sure, > however, that it falls back to use of expanduser in the event of an error. Attached to this message you'll find the patch. It only uses the effective user id to check for the .python-eggs directory if the one returned by os.path.expanduser() is not writeable, so for most people the actual behavior remains, and the impact is limited to applications which change the effective user id. Here's what it does : ad-port53-2:/home/jerome/setuptools-0.6c6# python Python 2.4.4 (#2, Apr 26 2007, 00:02:45) [GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import os >>> os.getuid(), os.geteuid() (0, 0) >>> from pkg_resources import get_default_cache >>> get_default_cache() '/root/.python-eggs' >>> os.setegid(500) >>> os.seteuid(500) >>> get_default_cache() '/home/jerome/.python-eggs' >>> I understand that this patch doesn't always improve the situation : since you've said the directory is computed only once for a particular process, importing a setuptools-installed module as root before doing the seteuid call and then importing another setuptools-installed directory after the seteuid call, would probably still give the same import error as before, but at least it should help in *some* situations. And finally it can probably serve as a basis for discussion. hoping this helps anyway. bye Jerome Alet -------------- next part -------------- A non-text attachment was scrubbed... Name: pkg_resources.py.patch Type: text/x-diff Size: 1396 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070612/7617dd73/attachment.bin From lists at zopyx.com Wed Jun 13 21:09:50 2007 From: lists at zopyx.com (Andreas Jung) Date: Wed, 13 Jun 2007 21:09:50 +0200 Subject: [Distutils] Conflict between egg and standard package with same namespace Message-ID: <6D5A623E46DCEA6C0C0911DC@suxmac2.local> In my setup I have a zopyx.slimp egg inside site-package and a package zopyx.txng3. With this setup I can not longer import files from the zopyx.txng3 package. ajung at galactica:~> ls /opt/python-2.4.3/lib/python2.4/site-packages/zopyx* /opt/python-2.4.3/lib/python2.4/site-packages/zopyx.slimp-0.2.0-py2.4.egg /opt/python-2.4.3/lib/python2.4/site-packages/zopyx: __init__.py __init__.pyc txng3 ajung at galactica:~> python2.4 Python 2.4.4 (#4, Oct 19 2006, 21:09:46) [GCC 4.1.0 (SUSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import zopyx.txng3 Traceback (most recent call last): File "", line 1, in ? ImportError: No module named txng3 >>> import zopyx.slimp >>> After removing the egg it is possible to import zopyx.txng3 again. Is this a bug or setuptools misconception? Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 186 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070613/c1b1cb51/attachment.pgp From pje at telecommunity.com Wed Jun 13 22:43:42 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 13 Jun 2007 16:43:42 -0400 Subject: [Distutils] Conflict between egg and standard package with same namespace In-Reply-To: <6D5A623E46DCEA6C0C0911DC@suxmac2.local> References: <6D5A623E46DCEA6C0C0911DC@suxmac2.local> Message-ID: <20070613204141.897FB3A407F@sparrow.telecommunity.com> At 09:09 PM 6/13/2007 +0200, Andreas Jung wrote: >In my setup I have a zopyx.slimp egg inside site-package and a package >zopyx.txng3. With this setup I can not longer import files from the >zopyx.txng3 package. > >ajung at galactica:~> ls /opt/python-2.4.3/lib/python2.4/site-packages/zopyx* >/opt/python-2.4.3/lib/python2.4/site-packages/zopyx.slimp-0.2.0-py2.4.egg > >/opt/python-2.4.3/lib/python2.4/site-packages/zopyx: >__init__.py __init__.pyc txng3 What's inside of __init__.py here? If it's not a namespace package declaration, it's not going to work. Specifically, both projects must declare 'zopyx' as a namespace package, in order to distribute subpackages of it separately. From pjenvey at groovie.org Wed Jun 13 23:13:58 2007 From: pjenvey at groovie.org (Philip Jenvey) Date: Wed, 13 Jun 2007 14:13:58 -0700 Subject: [Distutils] Setuptools bug with tag_svn_revision In-Reply-To: <46659ED8.6040001@colorstudy.com> References: <46659ED8.6040001@colorstudy.com> Message-ID: On Jun 5, 2007, at 10:35 AM, Ian Bicking wrote: > If you try to do egg_info in a package that is in svn, but where any > directory or subdirectory in the package has been added but not > committed to svn, you'll get this error: > > Traceback (most recent call last): > File "setup.py", line 25, in ? > entry_points=""" > File "distutils/core.py", line 149, in setup > File "/usr/lib/python2.4/distutils/dist.py", line 946, in > run_commands > self.run_command(cmd) > File "/usr/lib/python2.4/distutils/dist.py", line 965, in > run_command > cmd_obj.ensure_finalized() > File "/usr/lib/python2.4/cmd.py", line 117, in ensure_finalized > pass > File > "/home/ianb/env/webdev/lib/python2.4/setuptools-0.6c6-py2.4.egg/ > setuptools/command/egg_info.py", > line 85, in finalize_options > self.vtags = self.tags() > File > "/home/ianb/env/webdev/lib/python2.4/setuptools-0.6c6-py2.4.egg/ > setuptools/command/egg_info.py", > line 179, in tags > ): version += '-r%s' % self.get_svn_revision() > File > "/home/ianb/env/webdev/lib/python2.4/setuptools-0.6c6-py2.4.egg/ > setuptools/command/egg_info.py", > line 227, in get_svn_revision > localrev = max([int(m.group(1)) for m in revre.finditer(data)]) > ValueError: max() arg is an empty sequence I've also ran into this under the same circumstances (with an svn add'ed directory), however it manifested on line 224: File "/usr/local/lib/python2.4/site-packages/setuptools-0.6c6- py2.4.egg/setuptools/command/egg_info.py", line 179, in tags ): version += '-r%s' % self.get_svn_revision() File "/usr/local/lib/python2.4/site-packages/setuptools-0.6c6- py2.4.egg/setuptools/command/egg_info.py", line 224, in get_svn_revision localrev = max([int(d[9]) for d in data if len(d)>9 and d[9]]) ValueError: max() arg is an empty sequence Uncaught exception. Entering post mortem debugging Running 'cont' or 'step' will restart the program > /usr/local/lib/python2.4/site-packages/setuptools-0.6c6-py2.4.egg/ setuptools/command/egg_info.py(224)get_svn_revision() -> localrev = max([int(d[9]) for d in data if len(d)>9 and d[9]]) (Pdb) p data [['', 'dir', '0', 'http://svn.makotemplates.org/mako/trunk/test/ templates', 'http://svn.makotemplates.org', 'add', '', '', '', '', '', '', '', 'svn:special svn:externals svn:needs-lock'], ['l10n.mako', 'file', '', '', '', 'add'], []] (Pdb) [int(d[9]) for d in data if len(d)>9 and d[9]] [] -- Philip Jenvey From crocha at dc.uba.ar Thu Jun 14 17:22:20 2007 From: crocha at dc.uba.ar (Cristian S. Rocha) Date: Thu, 14 Jun 2007 12:22:20 -0300 Subject: [Distutils] Compiling CUDA code Message-ID: <1181834540.10867.16.camel@gator.qtp.ufl.edu> Hello, I would like to use distutils to compile CUDA code. CUDA is an extension of C and C++ to use GPUs capabilities. It need replace the gcc compiler by nvcc command line in unix, and append/modify some others parameters. I use the following command to generate my VolumeCUDA module before link it to a shared object. /usr/local/share/cuda/bin/nvcc -Xcompiler -fno-strict-aliasing,-fPIC -o VolumeCUDA.o -c src/VolumeCUDA.cu -I. -I/usr/local/share/cuda/inc -I/usr/include/python2.5 -O3 Exists any easy way to do that? Thanks, Cristian. -- Lic. Cristian S. Rocha [http://www.dc.uba.ar/~crocha] + | Grupo de Modelado Molecular (http://gmm.qi.fcen.uba.ar/) | Departamento de Computaci?n | Departamento de Qu?mica Inorg?nica y F?sica Qu?mica | Facultad de Ciencias Exactas y Naturales, | Universidad de Buenos Aires. + | Pabell?n I - Ciudad Universitaria | (C1428EGA) - Buenos Aires - Argentina | Tel./Fax:+54-11-4576-3359 - | Conmutador: +54-11-4576-3390 al 96, int. 701/702 + From rlratzel at enthought.com Fri Jun 15 01:25:50 2007 From: rlratzel at enthought.com (Rick Ratzel) Date: Thu, 14 Jun 2007 18:25:50 -0500 (CDT) Subject: [Distutils] requiring python 2.5 In-Reply-To: <20070609201545.977523A4060@sparrow.telecommunity.com> (pje@telecommunity.com) References: <20070605205625.77F4D3A4060@sparrow.telecommunity.com> <20070609022618.5C1441DF4FD@mail.enthought.com> <20070609201545.977523A4060@sparrow.telecommunity.com> Message-ID: <20070614232550.420161DF506@mail.enthought.com> Arg...I thought I had this working, but it turns out I had a setuptools 0.7 dev release installed. When I go to a setuptools 0.6 release, I get the same old problem. Here are more details... Enstaller is started with a script generated when easy_install installs the egg, so right away pkg_resources is imported and I have no chance to do anything before. I put the following in enstaller's __init__.py as recommended: from os import path import enthought enthought.__path__ = [path.dirname( path.dirname( __file__ ) )] and confirmed that it is indeed setting the __path__ for enthought correctly with a "print enthought.__path__" in another enstaller module imported later, which produced: ['c:\\python25\\lib\\site-packages\\enstaller-2.1.0b1-py2.5-win32.egg\\enthought'] This is compared to the two screenfuls of 'enthought.' paths without that code. As mentioned, this works great for setuptools 0.7, but as soon as I switch out 0.7 for 0.6, I get ImportErrors from package mismatches as python finds other incompatible 'enthought.' packages on the system instead of the ones bundled in the enstaller egg. If I leave the above code out completely, neither 0.6 or 0.7 work (as expected). Can you point out what, if anything, I'm doing wrong, since I thought setting enthought.__path__ is supposed to work with 0.6? I think I have three options at this point: 1.) have enstaller require setuptools>=0.7 (which enthought is hosting and will be found automatically by the enstaller install script) 2.) write a function in the enstaller egg build script that bundles all the packages into a namespace other than 'enthought.' inside of the egg, say 'bundle.' for example, and does a global search-and-replace of 'enthought.' with 'bundle.' Assuming the script gets everything, this seems like it would be the most straightforward and would not require me to fiddle with the python/setuptools import machinery. 3.) find the silly mistake I made which is causing the initial fix to break for 0.6, if possible. Thanks in advance! > Date: Sat, 09 Jun 2007 16:04:45 -0400 > From: "Phillip J. Eby" > > At 09:26 PM 6/8/2007 -0500, Rick Ratzel wrote: > > > Is there a way to have my package tell setuptools to "ignore" > > certain other > >packages? > > Only through version specifications; i.e., request the exact versions you want. > > > > I've tried removing all the "enthought." eggs from sys.path > > before anything > >else gets imported, but even that doesn't work (and it just felt wrong, and > >probably is). > > If "enthought" is a namespace package, you would need to either be > using an 0.7-development version of setuptools (i.e., a trunk SVN > version), or else you'd need to remove those eggs from the path > *before* pkg_resources is first imported. Otherwise, it would indeed not work. > > > >I looked at the docs for manipulating the WorkingSet, but made no > >progress there either...python always managed to find the incompatible > >enthought.traits code in the enthought.traits egg. When I remove the > >incompatible enthought.traits, all works fine. > > > > What I'd like to do is simply say "do not use any enthought.* > > eggs". Is this > >possible? Thanks in advance! > > Not without using an 0.7 version of setuptools, or manipulating > enthought.__path__. The straightforward way to do it would be for > enstaller's __init__.py to do this: > > import enthought, os > enthought.__path__ = os.path.dirname(os.path.dirname(__file__)) > > This will ensure that all enthought.* modules imported from then on > will only be from within the parent directory of enstaller. > > I don't think I'd recommend this to anyone in the general case, but > this might be a reasonable workaround if you truly want to > "de-namespace" the package. (It will *not*, however, prevent new > eggs from being added to __path__ if you add anything to sys.path or > the working set afterward.) > > -- Rick Ratzel - Enthought, Inc. 515 Congress Avenue, Suite 2100 - Austin, Texas 78701 512-536-1057 x229 - Fax: 512-536-1059 http://www.enthought.com From bhendrix at enthought.com Fri Jun 15 01:28:22 2007 From: bhendrix at enthought.com (Bryce Hendrix) Date: Thu, 14 Jun 2007 18:28:22 -0500 Subject: [Distutils] requiring python 2.5 In-Reply-To: <20070614232550.420161DF506@mail.enthought.com> References: <20070605205625.77F4D3A4060@sparrow.telecommunity.com> <20070609022618.5C1441DF4FD@mail.enthought.com> <20070609201545.977523A4060@sparrow.telecommunity.com> <20070614232550.420161DF506@mail.enthought.com> Message-ID: <4671CF16.6000500@enthought.com> Rick, We require setuptools >=0.7.0 for windows anyway, if that makes any difference... Bryce Rick Ratzel wrote: > Arg...I thought I had this working, but it turns out I had a setuptools 0.7 > dev release installed. When I go to a setuptools 0.6 release, I get the same > old problem. Here are more details... > > Enstaller is started with a script generated when easy_install installs the > egg, so right away pkg_resources is imported and I have no chance to do anything > before. I put the following in enstaller's __init__.py as recommended: > > from os import path > import enthought > enthought.__path__ = [path.dirname( path.dirname( __file__ ) )] > > and confirmed that it is indeed setting the __path__ for enthought correctly > with a "print enthought.__path__" in another enstaller module imported later, > which produced: > > ['c:\\python25\\lib\\site-packages\\enstaller-2.1.0b1-py2.5-win32.egg\\enthought'] > > This is compared to the two screenfuls of 'enthought.' paths without that > code. As mentioned, this works great for setuptools 0.7, but as soon as I > switch out 0.7 for 0.6, I get ImportErrors from package mismatches as python > finds other incompatible 'enthought.' packages on the system instead of the ones > bundled in the enstaller egg. If I leave the above code out completely, neither > 0.6 or 0.7 work (as expected). > > Can you point out what, if anything, I'm doing wrong, since I thought setting > enthought.__path__ is supposed to work with 0.6? > > I think I have three options at this point: > > 1.) have enstaller require setuptools>=0.7 (which enthought is hosting and will > be found automatically by the enstaller install script) > > 2.) write a function in the enstaller egg build script that bundles all the > packages into a namespace other than 'enthought.' inside of the egg, say > 'bundle.' for example, and does a global search-and-replace of 'enthought.' > with 'bundle.' Assuming the script gets everything, this seems like it would > be the most straightforward and would not require me to fiddle with the > python/setuptools import machinery. > > 3.) find the silly mistake I made which is causing the initial fix to break for > 0.6, if possible. > > Thanks in advance! > > > >> Date: Sat, 09 Jun 2007 16:04:45 -0400 >> From: "Phillip J. Eby" >> >> At 09:26 PM 6/8/2007 -0500, Rick Ratzel wrote: >> >> > Is there a way to have my package tell setuptools to "ignore" >> > certain other >> >packages? >> >> Only through version specifications; i.e., request the exact versions you want. >> >> >> > I've tried removing all the "enthought." eggs from sys.path >> > before anything >> >else gets imported, but even that doesn't work (and it just felt wrong, and >> >probably is). >> >> If "enthought" is a namespace package, you would need to either be >> using an 0.7-development version of setuptools (i.e., a trunk SVN >> version), or else you'd need to remove those eggs from the path >> *before* pkg_resources is first imported. Otherwise, it would indeed not work. >> >> >> >I looked at the docs for manipulating the WorkingSet, but made no >> >progress there either...python always managed to find the incompatible >> >enthought.traits code in the enthought.traits egg. When I remove the >> >incompatible enthought.traits, all works fine. >> > >> > What I'd like to do is simply say "do not use any enthought.* >> > eggs". Is this >> >possible? Thanks in advance! >> >> Not without using an 0.7 version of setuptools, or manipulating >> enthought.__path__. The straightforward way to do it would be for >> enstaller's __init__.py to do this: >> >> import enthought, os >> enthought.__path__ = os.path.dirname(os.path.dirname(__file__)) >> >> This will ensure that all enthought.* modules imported from then on >> will only be from within the parent directory of enstaller. >> >> I don't think I'd recommend this to anyone in the general case, but >> this might be a reasonable workaround if you truly want to >> "de-namespace" the package. (It will *not*, however, prevent new >> eggs from being added to __path__ if you add anything to sys.path or >> the working set afterward.) >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070614/feef40c3/attachment.html From pje at telecommunity.com Fri Jun 15 04:05:43 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 14 Jun 2007 22:05:43 -0400 Subject: [Distutils] requiring python 2.5 In-Reply-To: <20070614232550.420161DF506@mail.enthought.com> References: <20070605205625.77F4D3A4060@sparrow.telecommunity.com> <20070609022618.5C1441DF4FD@mail.enthought.com> <20070609201545.977523A4060@sparrow.telecommunity.com> <20070614232550.420161DF506@mail.enthought.com> Message-ID: <20070615020341.C11613A4080@sparrow.telecommunity.com> At 06:25 PM 6/14/2007 -0500, Rick Ratzel wrote: > Arg...I thought I had this working, but it turns out I had a > setuptools 0.7 >dev release installed. When I go to a setuptools 0.6 release, I get the same >old problem. Here are more details... > > Enstaller is started with a script generated when easy_install > installs the >egg, so right away pkg_resources is imported and I have no chance to >do anything >before. I put the following in enstaller's __init__.py as recommended: > >from os import path >import enthought >enthought.__path__ = [path.dirname( path.dirname( __file__ ) )] > > and confirmed that it is indeed setting the __path__ for > enthought correctly >with a "print enthought.__path__" in another enstaller module imported later, >which produced: > >['c:\\python25\\lib\\site-packages\\enstaller-2.1.0b1-py2.5-win32.egg\\enthought'] > > This is compared to the two screenfuls of 'enthought.' paths without that >code. As mentioned, this works great for setuptools 0.7, but as soon as I >switch out 0.7 for 0.6, I get ImportErrors from package mismatches as python >finds other incompatible 'enthought.' packages on the system instead >of the ones >bundled in the enstaller egg. It sounds like the other paths are being added (again) to __path__ later. This should only happen if new eggs get require()'d or otherwise added to the working set. Can you narrow down where this is happening? Is it possible, for example, that enstaller is being imported while pkg_resources is being imported? If you could raise an exception at the point of setting __path__, what is the full traceback? Likewise, what is the full traceback of the import errors you're getting? (Also, perhaps you could try setting __path__ to a tuple instead of a list? I'm not sure if that would work, but if it does, then an exception would occur when any other code modifies the __path__, so that might be another way to track it down. Full traceback, please, if this works...) From rlratzel at enthought.com Fri Jun 15 07:50:25 2007 From: rlratzel at enthought.com (Rick Ratzel) Date: Fri, 15 Jun 2007 00:50:25 -0500 (CDT) Subject: [Distutils] requiring python 2.5 In-Reply-To: <20070615020341.C11613A4080@sparrow.telecommunity.com> (pje@telecommunity.com) References: <20070605205625.77F4D3A4060@sparrow.telecommunity.com> <20070609022618.5C1441DF4FD@mail.enthought.com> <20070609201545.977523A4060@sparrow.telecommunity.com> <20070614232550.420161DF506@mail.enthought.com> <20070615020341.C11613A4080@sparrow.telecommunity.com> Message-ID: <20070615055025.1F9EB1DF4F8@mail.enthought.com> > Date: Thu, 14 Jun 2007 22:05:43 -0400 > From: "Phillip J. Eby" > > It sounds like the other paths are being added (again) to __path__ > later. This should only happen if new eggs get require()'d or > otherwise added to the working set. Can you narrow down where this > is happening? > > Is it possible, for example, that enstaller is being imported while > pkg_resources is being imported? If you could raise an exception at > the point of setting __path__, what is the full traceback? Likewise, > what is the full traceback of the import errors you're getting? > > (Also, perhaps you could try setting __path__ to a tuple instead of a > list? I'm not sure if that would work, but if it does, then an > exception would occur when any other code modifies the __path__, so > that might be another way to track it down. Full traceback, please, > if this works...) > Well, I do make one more require() call, but it's to add the enstaller.gui egg. enstaller is the command-line tool, and enstaller.gui adds the GUI. Both are "bundled" eggs, where enstaller has a few packages and enstaller.gui has many more. When I print the value of enthought.__path__ after that last require though, it only contains the path to the enstaller egg, and the path to enstaller.gui. I hope that's not the problem... So I tried to get an exception raised when enthought.__path__ is modified, but I could not. It appeared that it never did. Here is the traceback from the ImportError when using setuptools 0.6: Error: No module named app_data_locator.api File c:\python25\lib\site-packages\enstaller-2.1.0b1-py2.5-win32.egg\enthought\enstaller\launcher.py, line 72, in launch from enthought.enstaller.main import main File c:\python25\lib\site-packages\enstaller-2.1.0b1-py2.5-win32.egg\enthought\enstaller\main.py, line 65, in from enthought.enstaller.session import \ File c:\python25\lib\site-packages\enstaller-2.1.0b1-py2.5-win32.egg\enthought\enstaller\session.py, line 33, in from enthought.traits.api import \ File c:\python25\lib\site-packages\enthought.traits-2.0b2.dev_r12224-py2.5-win32.egg\enthought\traits\api.py, line 20, in from trait_base \ File c:\python25\lib\site-packages\enthought.traits-2.0b2.dev_r12224-py2.5-win32.egg\enthought\traits\trait_base.py, line 38, in from enthought.app_data_locator.api import AppDataLocator Not terribly helpful, but you can see that in enstaller\session.py line 33, it tries to import enthought.traits.api. Next line shows enthought.traits.api coming from the enthought.traits egg when it should be coming from the traits package in the enstaller egg. So, I put a breakpoint in session.py at line 32 to inspect, and here is what that looked like: Z:\>enstaller -l setuptools IN INIT: ('c:\\python25\\lib\\site-packages\\enstaller-2.1.0b1-py2.5-win32.egg\\enthought',) > c:\python25\lib\site-packages\enstaller-2.1.0b1-py2.5-win32.egg\enthought\enstaller\session.py(33)() -> from enthought.traits.api import \ (Pdb) l 28 import xmlrpclib 29 30 import pdb 31 pdb.set_trace() 32 33 -> from enthought.traits.api import \ 34 Trait, HasTraits, Str, List, Instance, Property 35 36 from enthought.enstaller.enstaller_traits import \ 37 CreatableDir, Url 38 from enthought.enstaller.api import \ (Pdb) import enthought (Pdb) p enthought.__path__ ('c:\\python25\\lib\\site-packages\\enstaller-2.1.0b1-py2.5-win32.egg\\enthought',) (Pdb) s --Call-- > c:\python25\lib\site-packages\enthought.traits-2.0b2.dev_r12224-py2.5-win32.egg\enthought\traits\api.py(16)() -> """ (Pdb) import enthought (Pdb) p enthought.__path__ ('c:\\python25\\lib\\site-packages\\enstaller-2.1.0b1-py2.5-win32.egg\\enthought',) (Pdb) import setuptools (Pdb) setuptools (Pdb) c The Enstaller package could not be imported...this means Enstaller is not installed or is broken or missing. Internal Error: : No module named app_data_locator.api Please submit the following postmortem file to the authors: Z:\ENSTALLER_POSTMORTEM.txt I break right before the import of enthought.traits.api and print out the value of enthought.__path__. I set it to a tuple early on like you suggested to hopefully get an exception on modification, and you can see that it's still set that way. The value is still correct with only the single path to the enthought package in the enstaller egg. I step into the next line which is the import, and you can see that I'm somehow taken to the enthought.traits egg, and it's all downhill from there. I also print out where python is finding setuptools, which is in the 0.6 egg. Here is where things get interesting...the same pdb session when I use setuptools 0.7: Z:\>enstaller -l setuptools IN INIT: ('c:\\python25\\lib\\site-packages\\enstaller-2.1.0b1-py2.5-win32.egg\\enthought',) > c:\python25\lib\site-packages\enstaller-2.1.0b1-py2.5-win32.egg\enthought\enstaller\session.py(33)() -> from enthought.traits.api import \ (Pdb) import enthought (Pdb) p enthought.__path__ ('c:\\python25\\lib\\site-packages\\enstaller-2.1.0b1-py2.5-win32.egg\\enthought',) (Pdb) s ImportError: 'No module named traits.api' > c:\python25\lib\site-packages\enstaller-2.1.0b1-py2.5-win32.egg\enthought\enstaller\session.py(33)() -> from enthought.traits.api import \ I get a different ImportError now. Now here is the pdb session when I set enthought.__path__ back to a list instead of a tuple using 0.7: Z:\>enstaller -l setuptools IN INIT: ['c:\\python25\\lib\\site-packages\\enstaller-2.1.0b1-py2.5-win32.egg\\enthought'] > c:\python25\lib\site-packages\enstaller-2.1.0b1-py2.5-win32.egg\enthought\enstaller\session.py(33)() -> from enthought.traits.api import \ (Pdb) import enthought (Pdb) p enthought.__path__ ['c:\\python25\\lib\\site-packages\\enstaller-2.1.0b1-py2.5-win32.egg\\enthought'] (Pdb) s --Call-- > c:\python25\lib\site-packages\enstaller-2.1.0b1-py2.5-win32.egg\enthought\traits\__init__.py(22)() -> try: (Pdb) import enthought (Pdb) p enthought.__path__ ['c:\\python25\\lib\\site-packages\\enstaller-2.1.0b1-py2.5-win32.egg\\enthought'] (Pdb) import setuptools (Pdb) setuptools (Pdb) c ------------------------------------------------------------ name | version | act | location ------------------------------------------------------------ setuptools | 0.6c6 | n | c:\python25\lib\site-packages setuptools | 0.7a1dev_r53614 | Y | c:\python25\lib\site-packages Works fine when using setuptools 0.7, and enthought.__path__ appears correct all the way through. Here's the code where __path__ gets set and the only other require() is: ------------------------------------------------------------------------------ from os import path # # Set the path to enthought to be this standalone enstaller egg. This # prevents any other eggs which contribute to the enthought namespace from # being found instead of the enthought packages bundled in this egg, which # are known to be compatible. # import enthought enthought.__path__ = [path.dirname( path.dirname( __file__ ) )] HAVE_GUI = True # # Try to activate the GUI...if found, this will enable GUI features. # try : require( "enstaller.gui" ) except : HAVE_GUI = False print "IN INIT: ", enthought.__path__ ------------------------------------------------------------------------------ I'd be happy to provide any additional details...thanks again, -- Rick Ratzel - Enthought, Inc. 515 Congress Avenue, Suite 2100 - Austin, Texas 78701 512-536-1057 x229 - Fax: 512-536-1059 http://www.enthought.com From pje at telecommunity.com Fri Jun 15 17:48:31 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 15 Jun 2007 11:48:31 -0400 Subject: [Distutils] requiring python 2.5 In-Reply-To: <20070615055025.1F9EB1DF4F8@mail.enthought.com> References: <20070605205625.77F4D3A4060@sparrow.telecommunity.com> <20070609022618.5C1441DF4FD@mail.enthought.com> <20070609201545.977523A4060@sparrow.telecommunity.com> <20070614232550.420161DF506@mail.enthought.com> <20070615020341.C11613A4080@sparrow.telecommunity.com> <20070615055025.1F9EB1DF4F8@mail.enthought.com> Message-ID: <20070615154630.969B43A4080@sparrow.telecommunity.com> At 12:50 AM 6/15/2007 -0500, Rick Ratzel wrote: >(Pdb) l > 28 import xmlrpclib > 29 > 30 import pdb > 31 pdb.set_trace() > 32 > 33 -> from enthought.traits.api import \ > 34 Trait, HasTraits, Str, List, Instance, Property > 35 > 36 from enthought.enstaller.enstaller_traits import \ > 37 CreatableDir, Url > 38 from enthought.enstaller.api import \ > >(Pdb) import enthought > >(Pdb) p enthought.__path__ >('c:\\python25\\lib\\site-packages\\enstaller-2.1.0b1-py2.5-win32.egg\\enthought',) > >(Pdb) s >--Call-- > > > c:\python25\lib\site-packages\enthought.traits-2.0b2.dev_r12224-py2.5-win32.egg\enthought\traits\api.py(16)() >-> """ Okay, so the problem here is that enthought.traits got imported *before* the __path__ was set. The __path__ needs to get set before any enthought.* imports occur. >(Pdb) p enthought.__path__ >('c:\\python25\\lib\\site-packages\\enstaller-2.1.0b1-py2.5-win32.egg\\enthought',) > >(Pdb) s >ImportError: 'No module named traits.api' > > > c:\python25\lib\site-packages\enstaller-2.1.0b1-py2.5-win32.egg\enthought\enstaller\session.py(33)() >-> from enthought.traits.api import \ > > > I get a different ImportError now. It appears that enthought.traits is not available in the enstaller egg; have you checked that? From rlratzel at enthought.com Fri Jun 15 19:59:15 2007 From: rlratzel at enthought.com (Rick Ratzel) Date: Fri, 15 Jun 2007 12:59:15 -0500 (CDT) Subject: [Distutils] requiring python 2.5 In-Reply-To: <20070615154630.969B43A4080@sparrow.telecommunity.com> (pje@telecommunity.com) References: <20070605205625.77F4D3A4060@sparrow.telecommunity.com> <20070609022618.5C1441DF4FD@mail.enthought.com> <20070609201545.977523A4060@sparrow.telecommunity.com> <20070614232550.420161DF506@mail.enthought.com> <20070615020341.C11613A4080@sparrow.telecommunity.com> <20070615055025.1F9EB1DF4F8@mail.enthought.com> <20070615154630.969B43A4080@sparrow.telecommunity.com> Message-ID: <20070615175915.74E7D1DF4F4@mail.enthought.com> > Date: Fri, 15 Jun 2007 11:48:31 -0400 > From: "Phillip J. Eby" > > At 12:50 AM 6/15/2007 -0500, Rick Ratzel wrote: > >(Pdb) l > > 28 import xmlrpclib > > 29 > > 30 import pdb > > 31 pdb.set_trace() > > 32 > > 33 -> from enthought.traits.api import \ > > 34 Trait, HasTraits, Str, List, Instance, Property > > 35 > > 36 from enthought.enstaller.enstaller_traits import \ > > 37 CreatableDir, Url > > 38 from enthought.enstaller.api import \ > > > >(Pdb) import enthought > > > >(Pdb) p enthought.__path__ > >('c:\\python25\\lib\\site-packages\\enstaller-2.1.0b1-py2.5-win32.egg\\enthought',) > > > >(Pdb) s > >--Call-- > > > > > c:\python25\lib\site-packages\enthought.traits-2.0b2.dev_r12224-py2.5-win32.egg\enthought\traits\api.py(16)() > >-> """ > > Okay, so the problem here is that enthought.traits got imported > *before* the __path__ was set. The __path__ needs to get set before > any enthought.* imports occur. Yes, I see now that it does get imported, but unfortunately not by my code. It's imported as a result of importing pkg_resources, since enthought.traits is a namespace package. Here's a little more detail: enstaller | - enthought.traits enstaller.gui | - enthought.traits.ui.wx And here's more pdb inspection: Z:\>python -m pdb c:\Python25\Scripts\enstaller-script.py > c:\python25\scripts\enstaller-script.py(3)() -> __requires__ = 'enstaller==2.1.0b1' (Pdb) n > c:\python25\scripts\enstaller-script.py(4)() -> import sys (Pdb) n > c:\python25\scripts\enstaller-script.py(5)() -> from pkg_resources import load_entry_point (Pdb) "enthought" in "".join(sys.modules.keys()) False (Pdb) n > c:\python25\scripts\enstaller-script.py(7)() -> sys.exit( (Pdb) "enthought" in "".join(sys.modules.keys()) True (Pdb) [sys.modules[p] for p in sys.modules.keys() if p.startswith("enthought.")] [, , ] Grasping at straws, I set enthought.__path__ in enstaller-script.py immediately before and after pkg_resources is imported, only to get the same results both times. But when I switch to setuptools 0.7 there are no enthought.traits modules loaded at all (in fact, the only enthought. modules loaded are the enstaller ones)...this must have been one of the changes you mentioned. When I step into the enthought.tratis.api import statement, it finds the traits pkg in the enstaller egg: Z:\>enstaller -l setuptools > c:\python25\lib\site-packages\enstaller-2.1.0b1-py2.5-win32.egg\enthought\enst aller\session.py(33)() -> from enthought.traits.api import \ (Pdb) l 28 import xmlrpclib 29 30 import pdb 31 pdb.set_trace() 32 33 -> from enthought.traits.api import \ 34 Trait, HasTraits, Str, List, Instance, Property 35 36 from enthought.enstaller.enstaller_traits import \ 37 CreatableDir, Url 38 from enthought.enstaller.api import \ (Pdb) [sys.modules[p] for p in sys.modules.keys() if p.startswith("enthought.tra its")] [] (Pdb) for p in [(p, sys.modules[p]) for p in sys.modules.keys() if p.startswith("enthought.")] : print p ('enthought.enstaller.optparse', None) ('enthought.enstaller.re', None) ('enthought.enstaller.session', ) ('enthought.enstaller.launcher', ) ('enthought.enstaller.traceback', None) ('enthought.enstaller.url_util', ) ('enthought.enstaller.distutils', None) ('enthought.enstaller.xmlrpclib', None) ('enthought.enstaller.enthought', None) ('enthought.enstaller.getpass', None) ('enthought.enstaller.pkg_resources', None) ('enthought.enstaller.pdb', None) ('enthought.enstaller.types', None) ('enthought.enstaller.time', None) ('enthought.enstaller.tempfile', None) ('enthought.enstaller.os', None) ('enthought.enstaller', ) ('enthought.enstaller.sys', None) ('enthought.enstaller.platform', None) ('enthought.enstaller.api', ) ('enthought.enstaller.site', None) ('enthought.enstaller.main', ) ('enthought.enstaller.urllib', None) ('enthought.enstaller.downloader', ) ('enthought.enstaller.urlparse', None) ('enthought.enstaller.text_io', ) (Pdb) s --Call-- > c:\python25\lib\site-packages\enstaller-2.1.0b1-py2.5-win32.egg\enthought\traits\__init__.py(22)() -> try: (Pdb) c ------------------------------------------------------------ name | version | act | location ------------------------------------------------------------ setuptools | 0.6c6 | n | c:\python25\lib\site-packages setuptools | 0.7a1dev_r53614 | Y | c:\python25\lib\site-packages > >(Pdb) p enthought.__path__ > >('c:\\python25\\lib\\site-packages\\enstaller-2.1.0b1-py2.5-win32.egg\\enthought',) > > > >(Pdb) s > >ImportError: 'No module named traits.api' > > > > > c:\python25\lib\site-packages\enstaller-2.1.0b1-py2.5-win32.egg\enthought\enstaller\session.py(33)() > >-> from enthought.traits.api import \ > > > > > > I get a different ImportError now. > > It appears that enthought.traits is not available in the enstaller > egg; have you checked that? I wish! Unfortunately it is. This is how Enstaller is able to work when no other enthought eggs are installed, or when I use setuptools 0.7. You can see the path to it in the pdb session above. Phillip, I really appreciate the time you're taking to look at this. I'm going to release a version which simply requires setuptools 0.7...unless you think that's a terrible idea, or you discover that I'm doing something wrong that I can fix. -- Rick Ratzel - Enthought, Inc. 515 Congress Avenue, Suite 2100 - Austin, Texas 78701 512-536-1057 x229 - Fax: 512-536-1059 http://www.enthought.com From pje at telecommunity.com Fri Jun 15 20:57:17 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 15 Jun 2007 14:57:17 -0400 Subject: [Distutils] requiring python 2.5 In-Reply-To: <20070615175915.74E7D1DF4F4@mail.enthought.com> References: <20070605205625.77F4D3A4060@sparrow.telecommunity.com> <20070609022618.5C1441DF4FD@mail.enthought.com> <20070609201545.977523A4060@sparrow.telecommunity.com> <20070614232550.420161DF506@mail.enthought.com> <20070615020341.C11613A4080@sparrow.telecommunity.com> <20070615055025.1F9EB1DF4F8@mail.enthought.com> <20070615154630.969B43A4080@sparrow.telecommunity.com> <20070615175915.74E7D1DF4F4@mail.enthought.com> Message-ID: <20070615185515.C371D3A4080@sparrow.telecommunity.com> At 12:59 PM 6/15/2007 -0500, Rick Ratzel wrote: > Yes, I see now that it does get imported, but unfortunately not > by my code. >It's imported as a result of importing pkg_resources, since >enthought.traits is >a namespace package. Does it *need* to be a namespace package? > Grasping at straws, I set enthought.__path__ in enstaller-script.py >immediately before and after pkg_resources is imported, only to get the same >results both times. > > But when I switch to setuptools 0.7 there are no enthought.traits modules >loaded at all (in fact, the only enthought. modules loaded are the enstaller >ones)...this must have been one of the changes you mentioned. Yes, that's exactly the change I'm talking about; in 0.7, namespace packages are always loaded lazily. > Phillip, I really appreciate the time you're taking to look at this. I'm >going to release a version which simply requires setuptools 0.7...unless you >think that's a terrible idea, or you discover that I'm doing something wrong >that I can fix. The only things I can think of would be removing traits as a namespace package, or manually setting its __path__ also. That is, first set enthought.__path__, then enthought.traits.__path__. However, this will only work right if no traits.__init__ module does anything but declare the namespace. From rlratzel at enthought.com Fri Jun 15 23:49:46 2007 From: rlratzel at enthought.com (Rick Ratzel) Date: Fri, 15 Jun 2007 16:49:46 -0500 (CDT) Subject: [Distutils] requiring python 2.5 In-Reply-To: <20070615185515.C371D3A4080@sparrow.telecommunity.com> (pje@telecommunity.com) References: <20070605205625.77F4D3A4060@sparrow.telecommunity.com> <20070609022618.5C1441DF4FD@mail.enthought.com> <20070609201545.977523A4060@sparrow.telecommunity.com> <20070614232550.420161DF506@mail.enthought.com> <20070615020341.C11613A4080@sparrow.telecommunity.com> <20070615055025.1F9EB1DF4F8@mail.enthought.com> <20070615154630.969B43A4080@sparrow.telecommunity.com> <20070615175915.74E7D1DF4F4@mail.enthought.com> <20070615185515.C371D3A4080@sparrow.telecommunity.com> Message-ID: <20070615214946.278201DF4F1@mail.enthought.com> > Date: Fri, 15 Jun 2007 14:57:17 -0400 > From: "Phillip J. Eby" > > At 12:59 PM 6/15/2007 -0500, Rick Ratzel wrote: > > Yes, I see now that it does get imported, but unfortunately not > > by my code. > >It's imported as a result of importing pkg_resources, since > >enthought.traits is > >a namespace package. > > Does it *need* to be a namespace package? Yes, I think it does. enthought.traits is our package for adding manifestly typed attributes to Python classes, and enthought.traits.ui.wx contributes to the enthought.traits.ui namespace by adding a wxPython backend for creating UIs out of classes with traits using wxPython. We don't want to require wxPython (we have a wxPython egg) when users may never use the UI features of traits, so we broke it out as an extra. So naturally, enthought.traits is bundled in the enstaller egg, and enthought.traits.ui is in the enstaller.gui egg. > > Grasping at straws, I set enthought.__path__ in enstaller-script.py > >immediately before and after pkg_resources is imported, only to get the same > >results both times. > > > > But when I switch to setuptools 0.7 there are no enthought.traits modules > >loaded at all (in fact, the only enthought. modules loaded are the enstaller > >ones)...this must have been one of the changes you mentioned. > > Yes, that's exactly the change I'm talking about; in 0.7, namespace > packages are always loaded lazily. > > > > Phillip, I really appreciate the time you're taking to look at this. I'm > >going to release a version which simply requires setuptools 0.7...unless you > >think that's a terrible idea, or you discover that I'm doing something wrong > >that I can fix. > > The only things I can think of would be removing traits as a > namespace package, or manually setting its __path__ also. That is, > first set enthought.__path__, then > enthought.traits.__path__. However, this will only work right if no > traits.__init__ module does anything but declare the namespace. Ah, of course...enthought.traits being a namespace package means it needs the same treatment. I fixed up the enthought.traits and enthought.traits.ui __path__ vars and it seems to be working with both versions of setuptools, with and without various conflicting enthought.* eggs installed. This should be OK since we've made an effort to move all code (except for namespace decls) from __init__.py to api.py for all of our packages. I'm going to do a formal build which "re-bundles" the enstaller egg and install it again for a more thorough test, but I think I'm in the clear (famous last words). My code looks like this now (it's actually in a "main" file which is basically imported first, and not __init__.py): enthought_path = path.dirname( path.dirname( __file__ ) ) import enthought enthought.__path__ = [enthought_path] enthought_traits_path = path.join( enthought_path, "traits" ) import enthought.traits enthought.traits.__path__ = [enthought_traits_path] import enthought.traits.ui enthought.traits.ui.__path__ = [path.join( enthought_traits_path, "ui" )] Basically, I have to keep an eye out for any other namespace packages that I use and make sure I do the same for them. Thanks again for all your help, -- Rick Ratzel - Enthought, Inc. 515 Congress Avenue, Suite 2100 - Austin, Texas 78701 512-536-1057 x229 - Fax: 512-536-1059 http://www.enthought.com From me at lbruno.org Sat Jun 16 12:41:46 2007 From: me at lbruno.org (=?UTF-8?Q?Lu=C3=ADs_Bruno?=) Date: Sat, 16 Jun 2007 11:41:46 +0100 Subject: [Distutils] Compiling CUDA code In-Reply-To: <1181834540.10867.16.camel@gator.qtp.ufl.edu> References: <1181834540.10867.16.camel@gator.qtp.ufl.edu> Message-ID: <7555ca2e0706160341o2cfa45f3l4c46cb761c250b41@mail.gmail.com> Hello, Cristian S. Rocha wrote: > I would like to use distutils to compile CUDA code. CUDA is an > extension of C and C++ to use GPUs capabilities. It need replace > the gcc compiler by nvcc command line in unix, and append/modify > some others parameters. I haven't tried this, but see psycopg2's setup.py: class psycopg_build_ext(distutils.command.build_ext.build_ext): ... def get_compiler(self): return self.compiler or get_default_compiler() Maybe you can use distutils.ccompiler.new_compiler() or subclass distutils.unixccompiler.UnixCCompiler. Then, return that instance from psycopg_build_ext.get_compiler() and use this in setup.py: setup( ... cmdclass=dict(build_ext=psycopg_build_ext), ... ) Again, I haven't tried this; just some pointers I stumbled upon. Please tell me if it served your purposes! http://docs.python.org/dist/dist.html http://docs.python.org/dist/module-distutils.ccompiler.html G'luck, -- Luis Bruno Python charmer, trapped inside a Windows sysadmin body. Help! From philipp at weitershausen.de Sun Jun 17 20:01:38 2007 From: philipp at weitershausen.de (Philipp von Weitershausen) Date: Sun, 17 Jun 2007 20:01:38 +0200 Subject: [Distutils] slow Python startup with a couple of easy_install-ed packages In-Reply-To: <5.1.1.6.0.20070223153228.04dc89f0@sparrow.telecommunity.com> References: <5.1.1.6.0.20070223153228.04dc89f0@sparrow.telecommunity.com> Message-ID: <46757702.5070800@weitershausen.de> Phillip J. Eby wrote: > Meanwhile, in setuptools 0.7 there will be a completely different way of > managing installation locations, and it will not depend on .pth files or on > putting eggs first. Instead, it will use direct old-style installation, > combined with .egg-info directories for metadata and for installation > manifests to support automated uninstallation, upgrades, and integrity > verification. Sorry for following up to a very old thread, but I'm intrigued by the above. I couldn't find anything in the setuptools trunk, so I assume this isn't implemented yet. Are there any docs that I could read to find out what exactly you guys have planned there? -- http://worldcookery.com -- Professional Zope documentation and training From pje at telecommunity.com Sun Jun 17 21:17:02 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 17 Jun 2007 15:17:02 -0400 Subject: [Distutils] slow Python startup with a couple of easy_install-ed packages In-Reply-To: <46757702.5070800@weitershausen.de> References: <5.1.1.6.0.20070223153228.04dc89f0@sparrow.telecommunity.com> <46757702.5070800@weitershausen.de> Message-ID: <20070617191937.C68333A40B2@sparrow.telecommunity.com> At 08:01 PM 6/17/2007 +0200, Philipp von Weitershausen wrote: >Phillip J. Eby wrote: >>Meanwhile, in setuptools 0.7 there will be a completely different >>way of managing installation locations, and it will not depend on >>.pth files or on putting eggs first. Instead, it will use direct >>old-style installation, combined with .egg-info directories for >>metadata and for installation manifests to support automated >>uninstallation, upgrades, and integrity verification. > >Sorry for following up to a very old thread, but I'm intrigued by >the above. I couldn't find anything in the setuptools trunk, so I >assume this isn't implemented yet. Nope. >Are there any docs that I could read to find out what exactly you >guys have planned there? Also nope, although googling "setuptools nest" (without the quotes) will probably find you some of my previous posts on the subject. From optilude at gmx.net Tue Jun 19 15:51:43 2007 From: optilude at gmx.net (Martin Aspeli) Date: Tue, 19 Jun 2007 13:51:43 +0000 (UTC) Subject: [Distutils] buildout/setuptools - Script generation on Windows breaks? Message-ID: Hi guys, I'm using buildout, with the bootstrap.py from svn (http://svn.zope.org/zc.buildout/trunk/bootstrap/bootstrap.py?rev=75593&view=auto) It's downloaded setuptools-0.6c6-py2.4.egg. It works fine on OS X, but on Windows, it generates console scripts funny. I have bin\buildout-script.py and bin\buildout-script.py.exe, where I was expecting just bin\buildout.exe. When I run the latter, I get: "Cannot open D:\development\plone3\bin\buildout-script.py-script.py" I'm pretty sure this used to work. Any ideas what's going on? From jim at zope.com Tue Jun 19 17:04:26 2007 From: jim at zope.com (Jim Fulton) Date: Tue, 19 Jun 2007 11:04:26 -0400 Subject: [Distutils] buildout/setuptools - Script generation on Windows breaks? In-Reply-To: References: Message-ID: On Jun 19, 2007, at 9:51 AM, Martin Aspeli wrote: > Hi guys, > > I'm using buildout, with the bootstrap.py from svn > (http://svn.zope.org/zc.buildout/trunk/bootstrap/bootstrap.py? > rev=75593&view=auto) > > It's downloaded setuptools-0.6c6-py2.4.egg. > > It works fine on OS X, but on Windows, it generates console scripts > funny. > > I have bin\buildout-script.py and bin\buildout-script.py.exe, where > I was > expecting just bin\buildout.exe. When I run the latter, I get: > > "Cannot open D:\development\plone3\bin\buildout-script.py-script.py" > > I'm pretty sure this used to work. > > Any ideas what's going on? I recently screwed up buildout on Windows rather badly. :( I'll try to get this fixed in the next day or 2. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From optilude at gmx.net Tue Jun 19 17:17:53 2007 From: optilude at gmx.net (Martin Aspeli) Date: Tue, 19 Jun 2007 15:17:53 +0000 (UTC) Subject: [Distutils] =?utf-8?q?buildout/setuptools_-_Script_generation_on_?= =?utf-8?q?Windows=09breaks=3F?= References: Message-ID: Jim Fulton zope.com> writes: > I recently screwed up buildout on Windows rather badly. :( > > I'll try to get this fixed in the next day or 2. Thanks - that would be great! I'm going to be doing some more testing of buildout on Windows, as well as of Zope, ZEO and possibly lxml/Deliverance. The Plone project also has a couple of Summer of Code students using Windows, who would benefit from having a buildout to play with. If you let me know when you're done, I can help you retest. Martin From edreamleo at charter.net Tue Jun 19 16:45:53 2007 From: edreamleo at charter.net (Edward Ream) Date: Tue, 19 Jun 2007 09:45:53 -0500 Subject: [Distutils] Newbie questions about setuptools Message-ID: <002c01c7b280$89df76b0$6a00a8c0@your4dacd0ea75> I have some questions and comments about setuptools and easy_install. What I like best about setuptools is that setup eliminates the need to maintain manifest.in and MANIFEST files. Assuming that everything in the cvs tree is part of the distribution is a clever idea, and it works for me. This is reason enough for me to want to use setuptools. However, I have the following problems with setuptools. These are serious enough so that for now I just use setuptools to create .zip files. 1. On both XP and Ubuntu, running python setup.py bdist_egg gives a .egg file with a suffix of py2.5.egg, even though the version arg to setup is '4.4.3preview9' I was expecting a suffix of all.egg, because platforms = ['all',] Can anyone explain what is going on? Here is the complete call to setup: setuptools.setup ( name='leo', version='4.4.3preview9', author='Edward K. Ream', author_email='edreamleo at charter.net', url='http://webpages.charter.net/edreamleo/front.html', download_url='http://downloads.sourceforge.net/leo/leo-4.4.3preview9.egg', packages = setuptools.find_packages(), include_package_data = True, exclude_package_data = { '': ['*.pyc','*.pyo']}, zip_safe=False, install_requires=[], description = 'Leo: Literate Editor with Outlines', license='Python', platforms=['all',], long_description = long_description, keywords = 'outline, outliner, ide, editor, literate programming', ) 2. The resulting egg file is much larger than the .zip file created by 'python setup.py sdist' in spite of exclude_package_data = { '': ['*.pyc','*.pyo']}, Is there any way to exclude .pyc and .pyo files? 3. 'python setup.py register' works fine, but 'python setup.py upload' fails with an 'invalid request' returned from the server. Maybe because the egg is over 8Meg? 4. 'easy_install leo' is supposed to use the download_url link arg. This is a great idea. Alas, for me it is spoiled by the setuptools pre/post release naming convention. The problem is that this convention is applied to all the links on the download_url page. Because I can't upload to the Python Cheese Shop, I used Leo's SourceForge download page for the main Leo project. The setup.py script is supposed to find the latest release on the page, but setup.py thinks that the latest match is something like Leo-4.2.1.zip (!!) This in spite of the fact that Leo-4.4.3rc1.zip exists on the page. I can't change the filenames on Leo's SourceForge page, and I can't change setuptools's naming conventions. How is 'easy_install leo' going to find the proper version? 5. setuptools is advertised as an 0.x release. This post appeared over a year ago, http://bitworking.org/news/Please_stop_using_setuptools__at_least_exclusively__for_now____ This post may be obsolete, but it troubles me. I suppose I wouldn't worry so much were it not for these other issues. Any help would be greatly appreciated. Thanks. Edward -------------------------------------------------------------------- Edward K. Ream email: edreamleo at charter.net Leo: http://webpages.charter.net/edreamleo/front.html -------------------------------------------------------------------- From pje at telecommunity.com Tue Jun 19 18:33:10 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 19 Jun 2007 12:33:10 -0400 Subject: [Distutils] Newbie questions about setuptools In-Reply-To: <002c01c7b280$89df76b0$6a00a8c0@your4dacd0ea75> References: <002c01c7b280$89df76b0$6a00a8c0@your4dacd0ea75> Message-ID: <20070619163156.CB7023A40A5@sparrow.telecommunity.com> At 09:45 AM 6/19/2007 -0500, Edward Ream wrote: >1. On both XP and Ubuntu, running > >python setup.py bdist_egg > >gives a .egg file with a suffix of py2.5.egg, even though the version arg to >setup is '4.4.3preview9' I was expecting a suffix of all.egg, because >platforms = ['all',] Can anyone explain what is going on? Eggs are named in the form "projectname-version-pythonversion[-optionalplatform].egg"; if you do not include any platform-specific code, the platform suffix is omitted. The Python version is included because .pyc and .pyo files are specific to a particular version of Python. (Also, the 'platforms' argument to setup() has no effect on the platform suffix or whether there is one. It's a recently introduced feature in newer Pythons that only affects the generation of PKG-INFO files.) >2. The resulting egg file is much larger than the .zip file created by >'python setup.py sdist' in spite of exclude_package_data = { '': >['*.pyc','*.pyo']}, > >Is there any way to exclude .pyc and .pyo files? No; you can exclude the source if you like, though, with --exclude-source-files. Eggs are a binary distribution format, originally developed to support user-installed plugins for systems like Chandler, Zope, etc. They aren't a source distribution format; sdist works well enough for that and for easy_install if you have a pure-Python package (or your users have C compilers). >3. 'python setup.py register' works fine, but 'python setup.py upload' fails >with an 'invalid request' returned from the server. Maybe because the egg >is over 8Meg? I have no idea; I'd suggest using the --show-response option so you can see what's happening. >4. 'easy_install leo' is supposed to use the download_url link arg. This is >a great idea. Alas, for me it is spoiled by the setuptools pre/post release >naming convention. The problem is that this convention is applied to all the >links on the download_url page. Because I can't upload to the Python Cheese >Shop, I used Leo's SourceForge download page for the main Leo project. The >setup.py script is supposed to find the latest release on the page, but >setup.py thinks that the latest match is something like Leo-4.2.1.zip (!!) >This in spite of the fact that Leo-4.4.3rc1.zip exists on the page. I don't follow you; here's the output from my attempt at downloading from the SF page: $ easy_install -nvf http://sourceforge.net/project/showfiles.php?group_id=3458 leo Searching for leo Reading http://sourceforge.net/project/showfiles.php?group_id=3458 Best match: leo 4.4.3preview8 Downloading http://downloads.sourceforge.net/leo/leo-4.4.3preview8.zip?modtime=1182088519&big_mirror=0 Perhaps you've changed something else? >5. setuptools is advertised as an 0.x release. This post appeared over a >year ago, > >http://bitworking.org/news/Please_stop_using_setuptools__at_least_exclusively__for_now____ > >This post may be obsolete, but it troubles me. I suppose I wouldn't worry >so much were it not for these other issues. The issues reported in that post were fixed within days of the blog posting; they'd have been fixed even faster if its author had actually asked about the issues here. Setuptools is currently at 0.6c6. There are a handful of known issues and quirks remaining, but it is otherwise pretty darn stable. From edreamleo at charter.net Tue Jun 19 20:56:09 2007 From: edreamleo at charter.net (Edward Ream) Date: Tue, 19 Jun 2007 13:56:09 -0500 Subject: [Distutils] Newbie questions about setuptools References: <002c01c7b280$89df76b0$6a00a8c0@your4dacd0ea75> <20070619163156.CB7023A40A5@sparrow.telecommunity.com> Message-ID: <001101c7b2a3$7fd13960$6a00a8c0@your4dacd0ea75> Many thanks, Phillip, for your quick and generous response. > Eggs are a binary distribution format, originally developed to support > user-installed plugins for systems like Chandler, Zope, etc. They aren't > a source distribution format; sdist works well enough for that and for > easy_install if you have a pure-Python package (or your users have C > compilers). Oh joy. sdist is working fine--Leo is indeed a pure python program. > I'd suggest using the --show-response option so you can see what's > happening. Will do, assuming there is still a problem with uploading the .zip file. > here's the output from my attempt at downloading from the SF page: > > $ easy_install -nvf > http://sourceforge.net/project/showfiles.php?group_id=3458 leo Excellent. I'll try again with the options you show. > Setuptools is currently at 0.6c6. There are a handful of known issues > and quirks remaining, but it is otherwise pretty darn stable. I'll consign the referenced post to the dustbin of history :-) Thanks again. I have lots of hope now. Edward -------------------------------------------------------------------- Edward K. Ream email: edreamleo at charter.net Leo: http://webpages.charter.net/edreamleo/front.html -------------------------------------------------------------------- From pje at telecommunity.com Tue Jun 19 22:15:21 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 19 Jun 2007 16:15:21 -0400 Subject: [Distutils] Newbie questions about setuptools In-Reply-To: <001101c7b2a3$7fd13960$6a00a8c0@your4dacd0ea75> References: <002c01c7b280$89df76b0$6a00a8c0@your4dacd0ea75> <20070619163156.CB7023A40A5@sparrow.telecommunity.com> <001101c7b2a3$7fd13960$6a00a8c0@your4dacd0ea75> Message-ID: <20070619201316.A88C13A409B@sparrow.telecommunity.com> At 01:56 PM 6/19/2007 -0500, Edward Ream wrote: >Many thanks, Phillip, for your quick and generous response. > > > Eggs are a binary distribution format, originally developed to support > > user-installed plugins for systems like Chandler, Zope, etc. They aren't > > a source distribution format; sdist works well enough for that and for > > easy_install if you have a pure-Python package (or your users have C > > compilers). > >Oh joy. sdist is working fine--Leo is indeed a pure python program. > > > I'd suggest using the --show-response option so you can see what's > > happening. > >Will do, assuming there is still a problem with uploading the .zip file. > > > here's the output from my attempt at downloading from the SF page: > > > > $ easy_install -nvf > > http://sourceforge.net/project/showfiles.php?group_id=3458 leo > >Excellent. I'll try again with the options you show. Note that this means you can use the above URL as your "Download URL" for PyPI; I just specified it using -f because that's not what you currently have up as the link on PyPI. The -n is because I didn't want to *actually* install leo, and the -v was for verbosity. If you update your "Download URL" you should be able to simply "easy_install leo" and have it work. From edreamleo at charter.net Wed Jun 20 16:43:10 2007 From: edreamleo at charter.net (Edward Ream) Date: Wed, 20 Jun 2007 09:43:10 -0500 Subject: [Distutils] Newbie questions about setuptools References: <002c01c7b280$89df76b0$6a00a8c0@your4dacd0ea75> <20070619163156.CB7023A40A5@sparrow.telecommunity.com> <001101c7b2a3$7fd13960$6a00a8c0@your4dacd0ea75> <20070619201316.A88C13A409B@sparrow.telecommunity.com> Message-ID: <000401c7b349$532f9400$6a00a8c0@your4dacd0ea75> >> http://sourceforge.net/project/showfiles.php?group_id=3458 leo > ...you can use the above URL as your "Download URL" Success, or close to it. Here is the present status. 1. setup.py register works. 2. setup.py sdist upload --show-response fails as follows: [big snip: creating files] removing 'leo-4.4.3preview10' (and everything under it) running upload Submitting dist\leo-4.4.3preview10.zip to http://www.python.org/pypi Upload failed (400): Bad Request --------------------------------------------------------------------------- Error processing form invalid distribution file Not a big deal. Any idea what might be happening? (see the P.S. for the call to setup.py) 3. After uploading 'leo-4.4.3preview10.zip to SourceForge, easy_install leo -v works (I think), but I was surprised that easy_install builds an egg: [big snip: copying, compiling and building the egg] Installed c:\python25\lib\site-packages\leo-4.4.3preview10-py2.5.egg Processing dependencies for leo Finished processing dependencies for leo I'm also a surprised that this new egg did not become the default version of Leo. C:\Documents and Settings\HP_Administrator>python Python 2.5 (r25:51908, Sep 19 2006, 09:52:17) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import leo >>> leo >>> My sitecustomize.py file is: import sys sys.path.append(r'c:\prog\tigris-cvs\leo\src') Surely this is a nit, but it will probably confuse others besides myself. Anyway, it appears that the download url you suggested is a big success. Hurray! Thanks for your help. Edward P.S. Here is the present call to setup.py: setuptools.setup ( name='leo', version='4.4.3preview10', # No spaces! # pre-release tags: 4.4.3b1 or 4.4.3rc1 or 4.4.3preview1 # Do not use post-release-tags: 4.4.3-whatever. # final release: 4.4.3final or just 4.4.3. author='Edward K. Ream', author_email='edreamleo at charter.net', url='http://webpages.charter.net/edreamleo/front.html', download_url='http://sourceforge.net/project/showfiles.php?group_id=3458', #'http://downloads.sourceforge.net/leo/leo-4.4.3preview9.egg', #'http://sourceforge.net/project/showfiles.php?group_id=3458&package_id=29106', # py_modules=[], # The manifest specifies everything. packages = setuptools.find_packages(), include_package_data = True, # Required, e.g. for Pmw.def exclude_package_data = { '': ['*.pyc','*.pyo']}, zip_safe=False, # Never run Leo from a zip file. install_requires=[], #'python>=2.2.1',], description = 'Leo: Literate Editor with Outlines', license='Python', # licence [sic] changed to license in Python 2.3 platforms=['all',], long_description = long_description, keywords = 'outline, outliner, ide, editor, literate programming', ) EKR -------------------------------------------------------------------- Edward K. Ream email: edreamleo at charter.net Leo: http://webpages.charter.net/edreamleo/front.html -------------------------------------------------------------------- From pje at telecommunity.com Wed Jun 20 20:10:49 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 20 Jun 2007 14:10:49 -0400 Subject: [Distutils] Newbie questions about setuptools In-Reply-To: <000401c7b349$532f9400$6a00a8c0@your4dacd0ea75> References: <002c01c7b280$89df76b0$6a00a8c0@your4dacd0ea75> <20070619163156.CB7023A40A5@sparrow.telecommunity.com> <001101c7b2a3$7fd13960$6a00a8c0@your4dacd0ea75> <20070619201316.A88C13A409B@sparrow.telecommunity.com> <000401c7b349$532f9400$6a00a8c0@your4dacd0ea75> Message-ID: <20070620180943.099673A409B@sparrow.telecommunity.com> At 09:43 AM 6/20/2007 -0500, Edward Ream wrote: >>>http://sourceforge.net/project/showfiles.php?group_id=3458 leo >>...you can use the above URL as your "Download URL" > >Success, or close to it. Here is the present status. > >1. setup.py register works. > >2. setup.py sdist upload --show-response fails as follows: > >[big snip: creating files] >removing 'leo-4.4.3preview10' (and everything under it) >running upload >Submitting dist\leo-4.4.3preview10.zip to http://www.python.org/pypi >Upload failed (400): Bad Request >--------------------------------------------------------------------------- >Error processing form >invalid distribution file > >Not a big deal. Any idea what might be happening? (see the P.S. >for the call to setup.py) No. I'd suggest following up with the Catalog-SIG or the PyPI support tracker. >3. After uploading 'leo-4.4.3preview10.zip to SourceForge, >easy_install leo -v works (I think), but I was surprised that >easy_install builds an egg: > >[big snip: copying, compiling and building the egg] >Installed c:\python25\lib\site-packages\leo-4.4.3preview10-py2.5.egg >Processing dependencies for leo >Finished processing dependencies for leo > >I'm also a surprised that this new egg did not become the default >version of Leo. It did; your setup() is broken. See below. >setuptools.setup ( > name='leo', > version='4.4.3preview10', # No spaces! > # pre-release tags: 4.4.3b1 or 4.4.3rc1 or 4.4.3preview1 > # Do not use post-release-tags: 4.4.3-whatever. > # final release: 4.4.3final or just 4.4.3. > author='Edward K. Ream', > author_email='edreamleo at charter.net', > url='http://webpages.charter.net/edreamleo/front.html', > download_url='http://sourceforge.net/project/showfiles.php?group_id=3458', > #'http://downloads.sourceforge.net/leo/leo-4.4.3preview9.egg', > >#'http://sourceforge.net/project/showfiles.php?group_id=3458&package_id=29106', > > # py_modules=[], # The manifest specifies everything. Sorry, you can't not list the modules and still install correctly or build correct binary distributions. py_modules isn't optional. That's as true for setuptools as it is for distutils. However, it's not clear to me whether you're intending to distribute a bunch of modules under 'src', or a package called 'leo'... Your sitecustomize implies that you have a bunch of modules under 'src' -- in which case you should list them in py_modules, and include a "package_dir={'': 'src'}" along with a bunch of other tricky-to-get-right things. However, the next line of your setup() implies that you intend to install some packages instead... > packages = setuptools.find_packages(), This line will find a bunch of packages named "plugins", "scripts", "src", "modes", "Icons", "config", "dist", "doc", "test", "www", "extensions", "extensions.Pmw", etc. This looks completely hosed, because I assume these should all be subpackages of "leo", not top-level packages. And you probably don't want a package called "src"! So, your layout and setup are quite hosed, regardless of whether you're using distutils or setuptools. What is it that you intend to distribute? Is it one package called 'leo' with a bunch of subpackages? Or what? Is everything in there really a package? (e.g. why is 'doc' a package?) > exclude_package_data = { '': ['*.pyc','*.pyo']}, This line is unnecessary; setuptools doesn't consider those files to be package data anyway, and it won't stop them from being put into any eggs built by easy_install. From edreamleo at charter.net Wed Jun 20 20:35:17 2007 From: edreamleo at charter.net (Edward Ream) Date: Wed, 20 Jun 2007 13:35:17 -0500 Subject: [Distutils] Newbie questions about setuptools References: <002c01c7b280$89df76b0$6a00a8c0@your4dacd0ea75> <20070619163156.CB7023A40A5@sparrow.telecommunity.com> <001101c7b2a3$7fd13960$6a00a8c0@your4dacd0ea75> <20070619201316.A88C13A409B@sparrow.telecommunity.com> <000401c7b349$532f9400$6a00a8c0@your4dacd0ea75> <20070620180943.099673A409B@sparrow.telecommunity.com> Message-ID: <000401c7b369$c03698d0$6a00a8c0@your4dacd0ea75> > I'd suggest following up with the Catalog-SIG or the PyPI support tracker. Perhaps the multiple mistakes you mention below contributed. I'll fix setup, etc. and then see what happens... > your setup() is broken That's good news :-) > Sorry, you can't not list the modules and still install correctly or build > correct binary distributions. Ok. I'm using sdist, not bdist_x. I assume that doesn't matter? > What is it that you intend to distribute? Is it one package called 'leo' > with a bunch of subpackages? Or what? Is everything in there really a > package? (e.g. why is 'doc' a package?) It seems my cluelessness has managed to confuse even you. As you no doubt have guessed, this is not easy for me. I'm not sure of the terminology, so please be patient. What I want to do is distribute Leo (a pure Python application, packaged as a single entity--a package?) , consisting of *all* the files under cvs management in leo directory and subdirectories of the leo directory that contain a cvs folder. I stumbled upon the way I chose because it seemed to work. I added some __init__.py files to some folders so they would be included in the list of packages, but I would be happy to list individual .py files if that is required. In fact, I would be positively glad to remove the recent __init__.py files, as they merely are confusing. > Your sitecustomize implies that you have a bunch of modules under 'src' -- > in which case you should list them in py_modules, and include a > "package_dir={'': 'src'}" along with a bunch of other tricky-to-get-right > things. Ok. I can do that. Are you implying that this is a less-than-recommended approach? >> packages = setuptools.find_packages(), > This looks completely hosed, because I assume these should all be > subpackages of "leo", not top-level packages. And you probably don't want > a package called "src"! Correct. I'll remove this line. >> exclude_package_data = { '': ['*.pyc','*.pyo']}, > > This line is unnecessary; setuptools doesn't consider those files to be > package data anyway, and it won't stop them from being put into any eggs > built by easy_install. Thanks. I'll remove this line. Ok. I've got some work to do. Thank for pointing me in a better direction. I'll work on producing something halfway decent, and we can discuss finer points later. Edward -------------------------------------------------------------------- Edward K. Ream email: edreamleo at charter.net Leo: http://webpages.charter.net/edreamleo/front.html -------------------------------------------------------------------- From jim at zope.com Wed Jun 20 21:17:06 2007 From: jim at zope.com (Jim Fulton) Date: Wed, 20 Jun 2007 15:17:06 -0400 Subject: [Distutils] buildout/setuptools - Script generation on Windows breaks? In-Reply-To: References: Message-ID: On Jun 19, 2007, at 11:17 AM, Martin Aspeli wrote: > Jim Fulton zope.com> writes: > >> I recently screwed up buildout on Windows rather badly. :( >> >> I'll try to get this fixed in the next day or 2. > > Thanks - that would be great! I'm going to be doing some more > testing of > buildout on Windows, as well as of Zope, ZEO and possibly lxml/ > Deliverance. The > Plone project also has a couple of Summer of Code students using > Windows, who > would benefit from having a buildout to play with. > > If you let me know when you're done, I can help you retest. I've just made a release that fixes the script-generation bug. There are a few somewhat obscure bugs on Windows that I still need to deal with, but I think the current release should be fine for most applications. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Wed Jun 20 22:58:08 2007 From: jim at zope.com (Jim Fulton) Date: Wed, 20 Jun 2007 16:58:08 -0400 Subject: [Distutils] buildout/setuptools - Script generation on Windows breaks? In-Reply-To: References: Message-ID: <680B353F-CBD7-410F-8234-C2141A2A43DE@zope.com> Someone pointed out to me that you may need to rerun bootstrap.py on buildouts that had been affected by the script generation bug to get proper buildout scripts. Jim On Jun 20, 2007, at 3:17 PM, Jim Fulton wrote: > > On Jun 19, 2007, at 11:17 AM, Martin Aspeli wrote: > >> Jim Fulton zope.com> writes: >> >>> I recently screwed up buildout on Windows rather badly. :( >>> >>> I'll try to get this fixed in the next day or 2. >> >> Thanks - that would be great! I'm going to be doing some more >> testing of >> buildout on Windows, as well as of Zope, ZEO and possibly lxml/ >> Deliverance. The >> Plone project also has a couple of Summer of Code students using >> Windows, who >> would benefit from having a buildout to play with. >> >> If you let me know when you're done, I can help you retest. > > I've just made a release that fixes the script-generation bug. > > There are a few somewhat obscure bugs on Windows that I still need to > deal with, but I think the current release should be fine for most > applications. > > Jim > > -- > Jim Fulton mailto:jim at zope.com Python Powered! > CTO (540) 361-1714 http://www.python.org > Zope Corporation http://www.zope.com http://www.zope.org > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Wed Jun 20 23:56:11 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 20 Jun 2007 17:56:11 -0400 Subject: [Distutils] Newbie questions about setuptools In-Reply-To: <000401c7b369$c03698d0$6a00a8c0@your4dacd0ea75> References: <002c01c7b280$89df76b0$6a00a8c0@your4dacd0ea75> <20070619163156.CB7023A40A5@sparrow.telecommunity.com> <001101c7b2a3$7fd13960$6a00a8c0@your4dacd0ea75> <20070619201316.A88C13A409B@sparrow.telecommunity.com> <000401c7b349$532f9400$6a00a8c0@your4dacd0ea75> <20070620180943.099673A409B@sparrow.telecommunity.com> <000401c7b369$c03698d0$6a00a8c0@your4dacd0ea75> Message-ID: <20070620215406.6F0893A402D@sparrow.telecommunity.com> At 01:35 PM 6/20/2007 -0500, Edward Ream wrote: >>Sorry, you can't not list the modules and still install correctly >>or build correct binary distributions. > >Ok. I'm using sdist, not bdist_x. I assume that doesn't matter? It matters if you want someone to be able to *install* your package. Most bdist_* commands are based on "build" and "install", which both need to know the modules, if you're installing any standalone modules. They also need to know about the packages, but it appears that you don't actually *have* any packages. >>What is it that you intend to distribute? Is it one package called >>'leo' with a bunch of subpackages? Or what? Is everything in >>there really a package? (e.g. why is 'doc' a package?) > >It seems my cluelessness has managed to confuse even you. As you no >doubt have guessed, this is not easy for me. > >I'm not sure of the terminology, so please be patient. What I want >to do is distribute Leo (a pure Python application, packaged as a >single entity--a package?) , A packaged entity is a "project" - the physical distribution of that entity is a "distribution". A "package" is a directory with an __init__.py and zero or more other .py files. Currently, for example, you have a package called 'src', containing various modules. If you had tried "from src import leo", or "import src.leo", for example, your attempt at using your easy_install-ed leo would have worked. (Of course, everything after that point would likely have failed.) >consisting of *all* the files under cvs management in leo directory >and subdirectories of the leo directory that contain a cvs folder. The distutils and setuptools do not install "files" - they install Python modules, scripts, extensions, and packages. It appears to me as though you do not have a project that can be meaningfully installed or distributed using the distutils or setuptools, as you have a collection of directories containing files, along with some directories with modules and scripts. In other words, more of an "application" than a "distribution". The distutils and setuptools can only support "applications" that are composed solely of Python modules, scripts, extensions, and packages (including *read-only* data files inside the packages). They can't install arbitrary trees of files, which unfortunately is what you have. You would need to reorganize your code in such a way that it does not contain any directories that you expect the user to modify, for example. Sample configuration files can still be included as package data, but your program would have to explicitly read them and copy them to a runtime configuration location. To put it another way, disutils and setuptools are designed to install *immutable* collections of code and data. At runtime, your program can read from the immutable code or data, but any writable runtime data must be placed somewhere else. Otherwise, you can't install a single copy of the project for use by all users on a system, nor is there any way to safely uninstall or upgrade it. >>Your sitecustomize implies that you have a bunch of modules under 'src' -- >>in which case you should list them in py_modules, and include a >>"package_dir={'': 'src'}" along with a bunch of other >>tricky-to-get-right things. > >Ok. I can do that. Are you implying that this is a >less-than-recommended approach? Yes. From edreamleo at charter.net Thu Jun 21 14:33:56 2007 From: edreamleo at charter.net (Edward Ream) Date: Thu, 21 Jun 2007 07:33:56 -0500 Subject: [Distutils] Newbie questions about setuptools References: <002c01c7b280$89df76b0$6a00a8c0@your4dacd0ea75> <20070619163156.CB7023A40A5@sparrow.telecommunity.com> <001101c7b2a3$7fd13960$6a00a8c0@your4dacd0ea75> <20070619201316.A88C13A409B@sparrow.telecommunity.com> <000401c7b349$532f9400$6a00a8c0@your4dacd0ea75> <20070620180943.099673A409B@sparrow.telecommunity.com> <000401c7b369$c03698d0$6a00a8c0@your4dacd0ea75> <20070620215406.6F0893A402D@sparrow.telecommunity.com> Message-ID: <000c01c7b400$6f661830$6a00a8c0@your4dacd0ea75> > It appears to me as though you do not have a project that can be > meaningfully installed or distributed using the distutils or setuptools, Doesn't this say more about distutils and setuptools than about Leo? I am in a state of shock. > > To put it another way, disutils and setuptools are designed to install > > *immutable* collections of code and data. The distinction between mutable and immutable data makes some (but not complete) sense on Linux. It is irrelevant on Windows. True, one can't install mutable data in /usr, but many Leo users would be happy to have easy_install unpack and install Leo's zip file in the user's home directory, for instance. Here is what I particularly like about setuptools: setup.py sdist register upload easy_install leo cleverly uses a generic upload_url setuptools.set can (at least sometimes) find all files managed by cvs. Why do any of the above require any particular organization of files? We in the Leo community never argue over user preferences because my guiding design principle is this: if there are two or more even halfway reasonable ways of doing something, Leo provides an option that allows users to choose what works for them, and that's is the end of the discusssion. It seems that allowing me to distribute Leo as I and Leo's users prefer would be a useful addition to setuptools. Edward -------------------------------------------------------------------- Edward K. Ream email: edreamleo at charter.net Leo: http://webpages.charter.net/edreamleo/front.html -------------------------------------------------------------------- From pje at telecommunity.com Thu Jun 21 15:09:55 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 21 Jun 2007 09:09:55 -0400 Subject: [Distutils] Newbie questions about setuptools In-Reply-To: <000c01c7b400$6f661830$6a00a8c0@your4dacd0ea75> References: <002c01c7b280$89df76b0$6a00a8c0@your4dacd0ea75> <20070619163156.CB7023A40A5@sparrow.telecommunity.com> <001101c7b2a3$7fd13960$6a00a8c0@your4dacd0ea75> <20070619201316.A88C13A409B@sparrow.telecommunity.com> <000401c7b349$532f9400$6a00a8c0@your4dacd0ea75> <20070620180943.099673A409B@sparrow.telecommunity.com> <000401c7b369$c03698d0$6a00a8c0@your4dacd0ea75> <20070620215406.6F0893A402D@sparrow.telecommunity.com> <000c01c7b400$6f661830$6a00a8c0@your4dacd0ea75> Message-ID: <20070621130759.1B0383A402D@sparrow.telecommunity.com> At 07:33 AM 6/21/2007 -0500, Edward Ream wrote: >>It appears to me as though you do not have a project that can be >>meaningfully installed or distributed using the distutils or setuptools, > >Doesn't this say more about distutils and setuptools than about Leo? Depends on what you mean by "say". :) Per http://www.python.org/doc/2.1/dist/ , the distutils were created "to make Python modules and extensions easily available to a wider audience with very little overhead for build/release/install mechanics." They aren't for installing arbitrary applications. It just happens that, if your "application" consists solely of scripts that tag along with some libraries and immutable data, then they also work. It's not reasonable to expect the distutils to be a general purpose software installation toolkit; they are, after all, for sharing Python libraries. >> > To put it another way, disutils and setuptools are designed to >> install > *immutable* collections of code and data. > >The distinction between mutable and immutable data makes some (but >not complete) sense on Linux. It is irrelevant on Windows. Actually, Microsoft has been trying to get people to stop putting mutable data in the \Program Files tree for many years now. That's why user-specific $APPDATA directories exist there. >Here is what I particularly like about setuptools: > >setup.py sdist register upload >easy_install leo cleverly uses a generic upload_url >setuptools.set can (at least sometimes) find all files managed by cvs. > >Why do any of the above require any particular organization of files? They don't -- but a LOT of other features of easy_install *do* depend on them. Notably, easy_install is intended to support (relatively) clean uninstallation, and the simultaneous installation of multiple versions of a project along the same PYTHONPATH. Aside from locating files in the first place, most of easy_install's complexity comes from either 1) supporting near-arbitrary distutils-based projects or 2) managing sys.path. Your project doesn't use either of these features, but they're absolutely vital for setuptools' *real* goal, which is to make widespread library sharing and reuse practical. The distutils do fine for distributing individual projects, but does badly for inter-project dependencies. This makes people who want to depend on other libraries either bundle them inside their project, or write their own replacement just to avoid the hassle of updating their bundled versions of things. Then, you either end up with multiple copies of some library installed anyway, or version skew/stomp. Setuptools was intended to fix this by making it easy to say, "my library uses these other libraries" -- and have them automatically installed, with version skew problems handled. This is a delicate enough problem to solve without throwing mutable files into the mix. >We in the Leo community never argue over user preferences because my >guiding design principle is this: if there are two or more even >halfway reasonable ways of doing something, Leo provides an option >that allows users to choose what works for them, and that's is the >end of the discusssion. It seems that allowing me to distribute Leo >as I and Leo's users prefer would be a useful addition to setuptools. Projects that don't consist of packages, immutable data, and scripts, don't really support library reuse very well. In fact, projects that include mutable files, would be in direct *opposition* to easy_install's goals for package management, so it really shouldn't be surprising that it doesn't support them very well. :) If you'd like to use the distutils or setuptools to distribute leo, I would suggest restructuring it as a set of Python packages rather than as a collection of modules and directories. (E.g., "import leo.config" rather than "import leoConfig".) For plugin support, I would suggest you investigate setuptools "entry points" system, which many packages and applications use to automatically discover separately-installed plugins: http://peak.telecommunity.com/DevCenter/setuptools#extensible-applications-and-frameworks http://peak.telecommunity.com/DevCenter/PkgResources#entry-points This method has the advantage that people can distribute plugins as setuptools-based projects, and users can install them with "easy_install pluginprojectname". (See, for example, all the various TurboGears widget plugin projects on the Cheeseshop, the TG/Buffet templating engines, Chandler plugins, etc., for examples of plugin systems built this way.) I'm not saying this would be easy or simple; many existing projects with other plugin or extension systems or using other distribution mechanisms have taken some time to migrate or are still migrating to this approach, in order to take advantage of the benefits. And truthfully, most projects using the full spectrum of setuptools' plugin abilities are relatively new, written after the features were available. Certainly, you could also retain older plugin mechanisms, and scan configured directories for plugins in addition to any "built-in" plugins that you provide in your main distribution project, so that you keep continuity for your existing users. Of course, you may also decide that the cost/benefit isn't there for you, and that's understandable. In that case, you may wish to use some other distribution mechanism, or create your own; setuptools unfortunately can't both achieve its own goals, and support installing non-importable things (aside from scripts and immutable package data) or arbitrary installation locations for project contents. From edreamleo at charter.net Thu Jun 21 15:55:28 2007 From: edreamleo at charter.net (Edward Ream) Date: Thu, 21 Jun 2007 08:55:28 -0500 Subject: [Distutils] Newbie questions about setuptools References: <002c01c7b280$89df76b0$6a00a8c0@your4dacd0ea75> <20070619163156.CB7023A40A5@sparrow.telecommunity.com> <001101c7b2a3$7fd13960$6a00a8c0@your4dacd0ea75> <20070619201316.A88C13A409B@sparrow.telecommunity.com> <000401c7b349$532f9400$6a00a8c0@your4dacd0ea75> <20070620180943.099673A409B@sparrow.telecommunity.com> <000401c7b369$c03698d0$6a00a8c0@your4dacd0ea75> <20070620215406.6F0893A402D@sparrow.telecommunity.com> <000c01c7b400$6f661830$6a00a8c0@your4dacd0ea75> <20070621130759.1B0383A402D@sparrow.telecommunity.com> Message-ID: <000801c7b40b$d3506610$6a00a8c0@your4dacd0ea75> > If you'd like to use the distutils or setuptools to distribute leo, I > would suggest restructuring it as a set of Python packages rather than as > a collection of modules and directories. ... > Of course, you may also decide that the cost/benefit isn't there for you, > and that's understandable. Many thanks, Phillip, for this long and graceful reply. At this stage in Leo's release cycle major changes are not possible, but I'll consider these ideas after Leo 4.4.3 final goes out the door. Edward -------------------------------------------------------------------- Edward K. Ream email: edreamleo at charter.net Leo: http://webpages.charter.net/edreamleo/front.html -------------------------------------------------------------------- From edreamleo at charter.net Thu Jun 21 16:01:10 2007 From: edreamleo at charter.net (Edward Ream) Date: Thu, 21 Jun 2007 09:01:10 -0500 Subject: [Distutils] Docs re debian distributions for pure Python programs? References: <002c01c7b280$89df76b0$6a00a8c0@your4dacd0ea75> <20070619163156.CB7023A40A5@sparrow.telecommunity.com> <001101c7b2a3$7fd13960$6a00a8c0@your4dacd0ea75> <20070619201316.A88C13A409B@sparrow.telecommunity.com> <000401c7b349$532f9400$6a00a8c0@your4dacd0ea75> <20070620180943.099673A409B@sparrow.telecommunity.com> <000401c7b369$c03698d0$6a00a8c0@your4dacd0ea75> <20070620215406.6F0893A402D@sparrow.telecommunity.com> <000c01c7b400$6f661830$6a00a8c0@your4dacd0ea75> <20070621130759.1B0383A402D@sparrow.telecommunity.com> Message-ID: <000b01c7b40c$9f603c80$6a00a8c0@your4dacd0ea75> Does anyone know of documentation for creating debian packags for pure Python projects? In particular, I would like to bypass the make-centric build process and the debian/rules file. Googling has turned up pages like http://blog.mypapit.net/2006/02/create-you-own-debianubuntu-deb-package.html which contains other links, but I am wondering whether more Python-specific pages exist. Yes, this is slightly OT, but I'm thinking that creating a debian package for Leo might lead me in a direction more compatible with distutils and easy_install. Thanks! Edward -------------------------------------------------------------------- Edward K. Ream email: edreamleo at charter.net Leo: http://webpages.charter.net/edreamleo/front.html -------------------------------------------------------------------- From strawman at astraw.com Thu Jun 21 21:12:13 2007 From: strawman at astraw.com (Andrew Straw) Date: Thu, 21 Jun 2007 12:12:13 -0700 Subject: [Distutils] Docs re debian distributions for pure Python programs? In-Reply-To: <000b01c7b40c$9f603c80$6a00a8c0@your4dacd0ea75> References: <002c01c7b280$89df76b0$6a00a8c0@your4dacd0ea75> <20070619163156.CB7023A40A5@sparrow.telecommunity.com> <001101c7b2a3$7fd13960$6a00a8c0@your4dacd0ea75> <20070619201316.A88C13A409B@sparrow.telecommunity.com> <000401c7b349$532f9400$6a00a8c0@your4dacd0ea75> <20070620180943.099673A409B@sparrow.telecommunity.com> <000401c7b369$c03698d0$6a00a8c0@your4dacd0ea75> <20070620215406.6F0893A402D@sparrow.telecommunity.com> <000c01c7b400$6f661830$6a00a8c0@your4dacd0ea75> <20070621130759.1B0383A402D@sparrow.telecommunity.com> <000b01c7b40c$9f603c80$6a00a8c0@your4dacd0ea75> Message-ID: <467ACD8D.8010307@astraw.com> Have you seen http://stdeb.python-hosting.com/ ? Edward Ream wrote: > Does anyone know of documentation for creating debian packags for pure > Python projects? From robert.kern at gmail.com Thu Jun 21 22:04:30 2007 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 21 Jun 2007 15:04:30 -0500 Subject: [Distutils] Docs re debian distributions for pure Python programs? In-Reply-To: <000b01c7b40c$9f603c80$6a00a8c0@your4dacd0ea75> References: <002c01c7b280$89df76b0$6a00a8c0@your4dacd0ea75> <20070619163156.CB7023A40A5@sparrow.telecommunity.com> <001101c7b2a3$7fd13960$6a00a8c0@your4dacd0ea75> <20070619201316.A88C13A409B@sparrow.telecommunity.com> <000401c7b349$532f9400$6a00a8c0@your4dacd0ea75> <20070620180943.099673A409B@sparrow.telecommunity.com> <000401c7b369$c03698d0$6a00a8c0@your4dacd0ea75> <20070620215406.6F0893A402D@sparrow.telecommunity.com> <000c01c7b400$6f661830$6a00a8c0@your4dacd0ea75> <20070621130759.1B0383A402D@sparrow.telecommunity.com> <000b01c7b40c$9f603c80$6a00a8c0@your4dacd0ea75> Message-ID: Edward Ream wrote: > Does anyone know of documentation for creating debian packags for pure > Python projects? In particular, I would like to bypass the make-centric > build process and the debian/rules file. The official Debian Python Policy document is the best starting point. http://www.debian.org/doc/packaging-manuals/python-policy/ The Wiki seems to have some more information, too. http://wiki.debian.org/DebianPython/NewPolicy You cannot really get around debian/rules, though. The debian-python mailing list would be the place to ask further questions along this line. http://lists.debian.org/debian-python/ -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From strawman at astraw.com Mon Jun 25 21:23:00 2007 From: strawman at astraw.com (Andrew Straw) Date: Mon, 25 Jun 2007 12:23:00 -0700 Subject: [Distutils] setuptools bug: broken simlinks break setup() Message-ID: <46801614.6080000@astraw.com> In setuptools 0.6c5, a broken symlink in the package tree will cause 'setup develop' to fail, even in the file is inconsequential for the package. I think this is a bug. Here is the output of running "python setup.py develop" in a package called flydra with a broken symlink called broken.symlink: running develop running egg_info writing flydra.egg-info/PKG-INFO writing top-level names to flydra.egg-info/top_level.txt writing dependency_links to flydra.egg-info/dependency_links.txt writing entry points to flydra.egg-info/entry_points.txt error: broken.symlink: No such file or directory (I would try this using setuptools==dev, but it seems svn.python.org is down, or at least my connection to it.) -Andrew From pje at telecommunity.com Mon Jun 25 22:20:52 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 25 Jun 2007 16:20:52 -0400 Subject: [Distutils] setuptools bug: broken simlinks break setup() In-Reply-To: <46801614.6080000@astraw.com> References: <46801614.6080000@astraw.com> Message-ID: <20070625201845.316A73A409B@sparrow.telecommunity.com> At 12:23 PM 6/25/2007 -0700, Andrew Straw wrote: >In setuptools 0.6c5, a broken symlink in the package tree will cause >'setup develop' to fail, even in the file is inconsequential for the >package. I think this is a bug. > >Here is the output of running "python setup.py develop" in a package >called flydra with a broken symlink called broken.symlink: > >running develop >running egg_info >writing flydra.egg-info/PKG-INFO >writing top-level names to flydra.egg-info/top_level.txt >writing dependency_links to flydra.egg-info/dependency_links.txt >writing entry points to flydra.egg-info/entry_points.txt >error: broken.symlink: No such file or directory > >(I would try this using setuptools==dev, but it seems svn.python.org is >down, or at least my connection to it.) Uh, where is this broken symlink exactly, and what does it point to? What is the contents of your setup.py? IOW, a minimal data set for reproducing the error would be helpful, as right now I don't have any idea where to begin looking for this. From strawman at astraw.com Mon Jun 25 22:51:03 2007 From: strawman at astraw.com (Andrew Straw) Date: Mon, 25 Jun 2007 13:51:03 -0700 Subject: [Distutils] setuptools bug: broken simlinks break setup() In-Reply-To: <20070625201845.316A73A409B@sparrow.telecommunity.com> References: <46801614.6080000@astraw.com> <20070625201845.316A73A409B@sparrow.telecommunity.com> Message-ID: <46802AB7.6000300@astraw.com> OK, I attach a fairly minimal python package. It builds fine normally. Now, if you add a broken symlink in the source tree, it fails to build. For example, "ln -s /broken broken.symlink" in the same directory as setup.py. Phillip J. Eby wrote: > At 12:23 PM 6/25/2007 -0700, Andrew Straw wrote: >> In setuptools 0.6c5, a broken symlink in the package tree will cause >> 'setup develop' to fail, even in the file is inconsequential for the >> package. I think this is a bug. >> >> Here is the output of running "python setup.py develop" in a package >> called flydra with a broken symlink called broken.symlink: >> >> running develop >> running egg_info >> writing flydra.egg-info/PKG-INFO >> writing top-level names to flydra.egg-info/top_level.txt >> writing dependency_links to flydra.egg-info/dependency_links.txt >> writing entry points to flydra.egg-info/entry_points.txt >> error: broken.symlink: No such file or directory >> >> (I would try this using setuptools==dev, but it seems svn.python.org is >> down, or at least my connection to it.) > > Uh, where is this broken symlink exactly, and what does it point to? > What is the contents of your setup.py? IOW, a minimal data set for > reproducing the error would be helpful, as right now I don't have any > idea where to begin looking for this. > -------------- next part -------------- A non-text attachment was scrubbed... Name: simplepack.tar.gz Type: application/x-gzip Size: 359 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070625/3577b032/attachment.bin From pje at telecommunity.com Tue Jun 26 00:16:43 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 25 Jun 2007 18:16:43 -0400 Subject: [Distutils] setuptools bug: broken simlinks break setup() In-Reply-To: <46802AB7.6000300@astraw.com> References: <46801614.6080000@astraw.com> <20070625201845.316A73A409B@sparrow.telecommunity.com> <46802AB7.6000300@astraw.com> Message-ID: <20070625221436.5BEC63A409B@sparrow.telecommunity.com> At 01:51 PM 6/25/2007 -0700, Andrew Straw wrote: >OK, I attach a fairly minimal python package. It builds fine normally. >Now, if you add a broken symlink in the source tree, it fails to build. >For example, "ln -s /broken broken.symlink" in the same directory as >setup.py. Tried it on Cygwin w/setuptools 0.6c6, nothing different happens with the symlink. What platform are you on? From ben at groovie.org Tue Jun 26 02:45:44 2007 From: ben at groovie.org (Ben Bangert) Date: Mon, 25 Jun 2007 17:45:44 -0700 Subject: [Distutils] Using setuptools entry points at Google (and other places...) Message-ID: <102E26C2-C6E7-42DA-A28D-61D865E5BEF2@groovie.org> Disclaimer: I don't work at Google, just talk to people who do. I know some people at Google that would like to use Pylons there, unfortunately this isn't quite possible as several parts of Pylons require setuptools entry points. While setuptools can be installed, due to Google's packaging system the run-time setuptools environment has no entry points present, thus it falls down. I'm not sure how many other companies might also have their own packaging systems that also incur this problem, but I'm wondering if this can be remedied somehow. Is there a script that could be run manually before applications start that require entry points? Or some way setuptools itself can have some intelligence in it so that it can detect when its not in charge of the layout and read a manually configured file for entry point information instead? Thanks, Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2472 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070625/17d04c7e/attachment.bin From pje at telecommunity.com Tue Jun 26 03:27:55 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 25 Jun 2007 21:27:55 -0400 Subject: [Distutils] Using setuptools entry points at Google (and other places...) In-Reply-To: <102E26C2-C6E7-42DA-A28D-61D865E5BEF2@groovie.org> References: <102E26C2-C6E7-42DA-A28D-61D865E5BEF2@groovie.org> Message-ID: <20070626012550.AD8FE3A409B@sparrow.telecommunity.com> At 05:45 PM 6/25/2007 -0700, Ben Bangert wrote: >Disclaimer: I don't work at Google, just talk to people who do. > >I know some people at Google that would like to use Pylons there, >unfortunately this isn't quite possible as several parts of Pylons >require setuptools entry points. While setuptools can be installed, >due to Google's packaging system the run-time setuptools environment >has no entry points present, thus it falls down. I'm not sure how >many other companies might also have their own packaging systems that >also incur this problem, but I'm wondering if this can be remedied >somehow. Just make sure that packages' .egg-info directory is installed alongside the code; setuptools will do the rest. See also: http://peak.telecommunity.com/DevCenter/EggFormats#eggs-and-their-formats """The .egg-info format, on the other hand, was created to support backward-compatibility, performance, and ease of installation for system packaging tools that expect to install all projects' code and resources to a single directory (e.g. site-packages). Placing the metadata in that same directory simplifies the installation process, since it isn't necessary to create .pth files or otherwise modify sys.path to include each installed egg.""" From strawman at astraw.com Tue Jun 26 04:45:31 2007 From: strawman at astraw.com (Andrew Straw) Date: Mon, 25 Jun 2007 19:45:31 -0700 Subject: [Distutils] setuptools bug: broken simlinks break setup() In-Reply-To: <20070625221436.5BEC63A409B@sparrow.telecommunity.com> References: <46801614.6080000@astraw.com> <20070625201845.316A73A409B@sparrow.telecommunity.com> <46802AB7.6000300@astraw.com> <20070625221436.5BEC63A409B@sparrow.telecommunity.com> Message-ID: <46807DCB.50100@astraw.com> Phillip J. Eby wrote: > What platform are you on? Ubuntu GNU/Linux: both 6.10 x86_64 Python 2.4 as well as 7.04 i686 Python 2.5. From primco at gmail.com Tue Jun 26 08:24:43 2007 From: primco at gmail.com (primco) Date: Mon, 25 Jun 2007 23:24:43 -0700 (PDT) Subject: [Distutils] Using setuptools entry points at Google (and other places...) In-Reply-To: <20070626012550.AD8FE3A409B@sparrow.telecommunity.com> References: <102E26C2-C6E7-42DA-A28D-61D865E5BEF2@groovie.org> <20070626012550.AD8FE3A409B@sparrow.telecommunity.com> Message-ID: <11300508.post@talk.nabble.com> Phillip J. Eby wrote: > > At 05:45 PM 6/25/2007 -0700, Ben Bangert wrote: >>Disclaimer: I don't work at Google, just talk to people who do. >> >>I know some people at Google that would like to use Pylons there, >>unfortunately this isn't quite possible as several parts of Pylons >>require setuptools entry points. While setuptools can be installed, >>due to Google's packaging system the run-time setuptools environment >>has no entry points present, thus it falls down. I'm not sure how >>many other companies might also have their own packaging systems that >>also incur this problem, but I'm wondering if this can be remedied >>somehow. > > Just make sure that packages' .egg-info directory is installed > alongside the code; setuptools will do the rest. See also: > > Thanks Phillip. What's not clear to me is if there is also some versioning mechanism. There seems to be two options for naming your egg-info. Let's take the package Mako version 0.1.8, as it has a short name. If I do setup.py install with the --root option, it will install the package Mako into the mako directory in my specified location and along side, it will create a Mako-0.1.8-py2.4.egg-info directory. I'm assuming I can have that directory simply named mako.egg-info and remove the version info from the filename. Now, in Mako-0.1.8-py2.4.egg-info/top_level.txt is the line 'mako'. Is this some sort of pointer to the package? from the peak website: "....top_level.txt ... lists all top-level modules and packages in the distribution. This is used by the easy_install command to find possibly-conflicting "unmanaged" packages when installing the distribution." Does this mean that I have a rough version control system where I could have, in addition to this plain 'mako' folder that held version 0.1.8, I could have a mako-0.1.7 distribution directory with a Mako-0.1.7-py2.4.egg-info directory that had the pointer to mako-0.1.7 in top_level.txt and setuptools would use this info to find the right version of mako? In this case, if you did 'import mako' in python you'd get mako-0.1.8 but if you used setuptools to find the package you could pick from 0.1.8 or 0.1.7? thanks. davep -- View this message in context: http://www.nabble.com/Using-setuptools-entry-points-at-Google-%28and-other-places...%29-tf3979948.html#a11300508 Sent from the Python - distutils-sig mailing list archive at Nabble.com. From nathan at yergler.net Tue Jun 26 16:26:32 2007 From: nathan at yergler.net (Nathan R. Yergler) Date: Tue, 26 Jun 2007 07:26:32 -0700 Subject: [Distutils] setuptools bug: broken simlinks break setup() In-Reply-To: <46807DCB.50100@astraw.com> References: <46801614.6080000@astraw.com> <20070625201845.316A73A409B@sparrow.telecommunity.com> <46802AB7.6000300@astraw.com> <20070625221436.5BEC63A409B@sparrow.telecommunity.com> <46807DCB.50100@astraw.com> Message-ID: I just tried it and saw the error as Andrew described. On 6/25/07, Andrew Straw wrote: > Phillip J. Eby wrote: > > > What platform are you on? > > Ubuntu GNU/Linux: both 6.10 x86_64 Python 2.4 as well as 7.04 i686 > Python 2.5. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From jim at zope.com Tue Jun 26 16:43:02 2007 From: jim at zope.com (Jim Fulton) Date: Tue, 26 Jun 2007 10:43:02 -0400 Subject: [Distutils] setuptools bug: broken simlinks break setup() In-Reply-To: References: <46801614.6080000@astraw.com> <20070625201845.316A73A409B@sparrow.telecommunity.com> <46802AB7.6000300@astraw.com> <20070625221436.5BEC63A409B@sparrow.telecommunity.com> <46807DCB.50100@astraw.com> Message-ID: <3F6A0338-C0D8-4652-BC00-423B40677182@zope.com> This is an ongoing problem for me with emacs, which intentionally uses broken links to implement lock files. Arguably, this is emacs' fault, but I'd love to see a change to setuptools that made it ignore broken links. Jim On Jun 26, 2007, at 10:26 AM, Nathan R. Yergler wrote: > I just tried it and saw the error as Andrew described. > > On 6/25/07, Andrew Straw wrote: >> Phillip J. Eby wrote: >> >>> What platform are you on? >> >> Ubuntu GNU/Linux: both 6.10 x86_64 Python 2.4 as well as 7.04 i686 >> Python 2.5. >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig >> > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Tue Jun 26 23:47:29 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 26 Jun 2007 17:47:29 -0400 Subject: [Distutils] Using setuptools entry points at Google (and other places...) In-Reply-To: <11300508.post@talk.nabble.com> References: <102E26C2-C6E7-42DA-A28D-61D865E5BEF2@groovie.org> <20070626012550.AD8FE3A409B@sparrow.telecommunity.com> <11300508.post@talk.nabble.com> Message-ID: <20070626214523.CDD613A40AA@sparrow.telecommunity.com> At 11:24 PM 6/25/2007 -0700, primco wrote: >Phillip J. Eby wrote: > > > > At 05:45 PM 6/25/2007 -0700, Ben Bangert wrote: > >>Disclaimer: I don't work at Google, just talk to people who do. > >> > >>I know some people at Google that would like to use Pylons there, > >>unfortunately this isn't quite possible as several parts of Pylons > >>require setuptools entry points. While setuptools can be installed, > >>due to Google's packaging system the run-time setuptools environment > >>has no entry points present, thus it falls down. I'm not sure how > >>many other companies might also have their own packaging systems that > >>also incur this problem, but I'm wondering if this can be remedied > >>somehow. > > > > Just make sure that packages' .egg-info directory is installed > > alongside the code; setuptools will do the rest. See also: > > > >Thanks Phillip. What's not clear to me is if there is also some versioning >mechanism. There seems to be two options for naming your egg-info. Let's >take the package Mako version 0.1.8, as it has a short name. > >If I do setup.py install with the --root option, it will install the package >Mako into the mako directory in my specified location and along side, it >will create a Mako-0.1.8-py2.4.egg-info directory. > >I'm assuming I can have that directory simply named mako.egg-info and remove >the version info from the filename. That's correct, but for performance reasons it is NOT recommended. Without the other information in the directory name, pkg_resources will have to read the PKG-INFO file at runtime to ascertain the package version, and this is then a lot slower than just doing a single listdir to get version info for all the projects in that directory. I therefore strongly recommend leaving setuptools' naming alone for installed packages. (For packages in development, the performance tradeoff is usually acceptable.) >Now, in Mako-0.1.8-py2.4.egg-info/top_level.txt is the line 'mako'. Is this >some sort of pointer to the package? > >from the peak website: "....top_level.txt ... lists all top-level modules >and packages in the distribution. This is used by the easy_install command >to find possibly-conflicting "unmanaged" packages when installing the >distribution." > >Does this mean that I have a rough version control system where I could >have, in addition to this plain 'mako' folder that held version 0.1.8, I >could have a mako-0.1.7 distribution directory with a >Mako-0.1.7-py2.4.egg-info directory that had the pointer to mako-0.1.7 in >top_level.txt and setuptools would use this info to find the right version >of mako? > >In this case, if you did 'import mako' in python you'd get mako-0.1.8 but if >you used setuptools to find the package you could pick from 0.1.8 or 0.1.7? No. top_level.txt has only ever been used to check for conflicts with non-egg packages, and it's not even used for that any more. It doesn't magically make multi-versioning possible; for multi-versioning in a single sys.path directory you have to use .egg format (either the zipfile or directory versions). But if you use the .egg format then you can indeed get your pick by using pkg_resources to request your desired version(s). The .egg-info format trades off multi-versioning and some package management features in order to provide backward compatibility and/or startup performance, which is why it's called "single version, externally managed". You need some sort of package manager to manage it, and you don't get multiple versions. From primco at gmail.com Wed Jun 27 07:06:13 2007 From: primco at gmail.com (primco) Date: Tue, 26 Jun 2007 22:06:13 -0700 (PDT) Subject: [Distutils] Using setuptools entry points at Google (and other places...) In-Reply-To: <20070626214523.CDD613A40AA@sparrow.telecommunity.com> References: <102E26C2-C6E7-42DA-A28D-61D865E5BEF2@groovie.org> <20070626012550.AD8FE3A409B@sparrow.telecommunity.com> <11300508.post@talk.nabble.com> <20070626214523.CDD613A40AA@sparrow.telecommunity.com> Message-ID: <11318442.post@talk.nabble.com> Phillip J. Eby wrote: > > At 11:24 PM 6/25/2007 -0700, primco wrote: >>Phillip J. Eby wrote: >> > >> > At 05:45 PM 6/25/2007 -0700, Ben Bangert wrote: >> >>Disclaimer: I don't work at Google, just talk to people who do. >> >> >> >>I know some people at Google that would like to use Pylons there, >> >>unfortunately this isn't quite possible as several parts of Pylons >> >>require setuptools entry points. While setuptools can be installed, >> >>due to Google's packaging system the run-time setuptools environment >> >>has no entry points present, thus it falls down. I'm not sure how >> >>many other companies might also have their own packaging systems that >> >>also incur this problem, but I'm wondering if this can be remedied >> >>somehow. >> > >> > Just make sure that packages' .egg-info directory is installed >> > alongside the code; setuptools will do the rest. See also: >> > > Thanks for the answers. That's what my messing around told me but I wanted to check. On a related note, and to finish of the dismantling of setuptools package management, how do I give the setuptools package itself this treatment? Some of the things I'd like to do: not use any .pth files for any packages not change site.py or sys.path (PYTHONPATH is ok) at runtime Avoid the included site.py that is generated by installing setuptools itself as single version externally managed. It has a "__boot()" function that does some path management. What I've done is "setup.py install --root..." setuptools from the source. This made a site.py for me which I just left out of site-packages and it all seems to work. I was able to run Pylons (a pretty good setuptools workout I think) with all dependencies installed in the same unmanaged manner. Did I just get lucky or should I assume that I've safely disabled the package management features of setuptools while still keeping some dynamic things like entrypoints and find_packages working? davep -- View this message in context: http://www.nabble.com/Using-setuptools-entry-points-at-Google-%28and-other-places...%29-tf3979948.html#a11318442 Sent from the Python - distutils-sig mailing list archive at Nabble.com. From pje at telecommunity.com Wed Jun 27 17:17:40 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 27 Jun 2007 11:17:40 -0400 Subject: [Distutils] Using setuptools entry points at Google (and other places...) In-Reply-To: <11318442.post@talk.nabble.com> References: <102E26C2-C6E7-42DA-A28D-61D865E5BEF2@groovie.org> <20070626012550.AD8FE3A409B@sparrow.telecommunity.com> <11300508.post@talk.nabble.com> <20070626214523.CDD613A40AA@sparrow.telecommunity.com> <11318442.post@talk.nabble.com> Message-ID: <20070627151533.AF03E3A40A8@sparrow.telecommunity.com> At 10:06 PM 6/26/2007 -0700, primco wrote: >On a related note, and to finish of the dismantling of setuptools package >management, how do I give the setuptools package itself this treatment? > >Some of the things I'd like to do: >not use any .pth files for any packages >not change site.py or sys.path (PYTHONPATH is ok) at runtime >Avoid the included site.py that is generated by installing setuptools itself >as single version externally managed. It has a "__boot()" function that does >some path management. > >What I've done is "setup.py install --root..." setuptools from the source. >This made a site.py for me which I just left out of site-packages and it all >seems to work. I was able to run Pylons (a pretty good setuptools workout I >think) with all dependencies installed in the same unmanaged manner. Did I >just get lucky or should I assume that I've safely disabled the package >management features of setuptools while still keeping some dynamic things >like entrypoints and find_packages working? Sounds correct to me. The site.py in site-packages wouldn't harm anything, btw, as it would never get imported. I suspect removing it breaks any attempt to use easy_install to install eggs to $PYTHONPATH (probably with a traceback and no sane error message) but perhaps that's what you intend. From dpeterson at enthought.com Thu Jun 28 17:47:43 2007 From: dpeterson at enthought.com (Dave Peterson) Date: Thu, 28 Jun 2007 10:47:43 -0500 Subject: [Distutils] Performance implications of having large numbers of eggs? Message-ID: <4683D81F.9030606@enthought.com> Has anyone done any investigation into the performance implications of having large numbers of eggs installed? Is there any sort of performance hit? It seems to me that having a really large path might slow down imports a bit, though I suspect this is in C code so probably not a significant problem. It also seems like there might be some startup penalties due to the overhead of setting up the path when using eggs, but this is a one-time cost during python startup, so probably not too bad either. I'm asking because we're in the process to switching our open-source Enthought Tool Suite library to a distribution of components via eggs and we're having some internal debate as to whether we need to minimize the number of eggs or not. It definitely seems nice to have smaller subsets of functionality -- from the point of being able to make things stable, managing their APIs, managing cross-component dependencies, and from the user update size viewpoint. But are we paying a performance penalty for going too small in scope with our eggs? -- Dave From pje at telecommunity.com Thu Jun 28 18:28:13 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 28 Jun 2007 12:28:13 -0400 Subject: [Distutils] Performance implications of having large numbers of eggs? In-Reply-To: <4683D81F.9030606@enthought.com> References: <4683D81F.9030606@enthought.com> Message-ID: <20070628162620.56C003A40D7@sparrow.telecommunity.com> At 10:47 AM 6/28/2007 -0500, Dave Peterson wrote: >Has anyone done any investigation into the performance implications of >having large numbers of eggs installed? Is there any sort of >performance hit? > > >It seems to me that having a really large path might slow down imports a >bit, though I suspect this is in C code so probably not a significant >problem. If the eggs are zipped, the performance overhead for imports is negligible, although there is a small startup cost to read the zipfile indexes. Python caches zipfile indexes in memory, so checking whether a module is present is just a dictionary lookup and is much faster than having a directory on sys.path. If the eggs are *not* zipped, however, the performance impact on individual imports is much higher. That's why easy_install installs eggs zipped by default. > It also seems like there might be some startup penalties due >to the overhead of setting up the path when using eggs, but this is a >one-time cost during python startup, so probably not too bad either. > >I'm asking because we're in the process to switching our open-source >Enthought Tool Suite library to a distribution of components via eggs >and we're having some internal debate as to whether we need to minimize >the number of eggs or not. It definitely seems nice to have smaller >subsets of functionality -- from the point of being able to make things >stable, managing their APIs, managing cross-component dependencies, and >from the user update size viewpoint. But are we paying a performance >penalty for going too small in scope with our eggs? I suggest you measure what you're concerned about. At one point, I did some timing tests that suggested that if you put the entire Cheeseshop on sys.path as zipped eggs, you might increase Python's startup time by a second or two. But your mileage may vary, and the Cheeseshop has increased a lot in size since then. ;) By the way, the long term plan for setuptools is that it should be able to install things the "old fashioned" way and still be able to manage them, using an installation manifest inside the .egg-info directories. In that way, you'd have all the benefits of separate distribution and a managed installation, as well as the benefits of having only one directory on sys.path. But I haven't done any work on implementing this yet. (Actually, now that I think of it, somebody could probably create a zc.buildout recipe to install eggs in an unpacked fashion (and let buildout handle uninstallation). The tricky part would be namespace package __init__ modules, since multiple eggs can share responsibility for the __init__, and this might confuse zc.buildout.) From pje at telecommunity.com Thu Jun 28 19:53:10 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 28 Jun 2007 13:53:10 -0400 Subject: [Distutils] setuptools bug: broken simlinks break setup() In-Reply-To: <3F6A0338-C0D8-4652-BC00-423B40677182@zope.com> References: <46801614.6080000@astraw.com> <20070625201845.316A73A409B@sparrow.telecommunity.com> <46802AB7.6000300@astraw.com> <20070625221436.5BEC63A409B@sparrow.telecommunity.com> <46807DCB.50100@astraw.com> <3F6A0338-C0D8-4652-BC00-423B40677182@zope.com> Message-ID: <20070628175103.DDE083A40AF@sparrow.telecommunity.com> At 10:43 AM 6/26/2007 -0400, Jim Fulton wrote: >This is an ongoing problem for me with emacs, which intentionally >uses broken links to implement lock files. Arguably, this is emacs' >fault, but I'd love to see a change to setuptools that made it ignore >broken links. > >Jim Actually, as it turns out, it's distutils fault (well, the error is, anyway): $ DISTUTILS_DEBUG=y python2.3 setup.py develop ... Traceback (most recent call last): ... File "/usr/lib/python2.3/distutils/filelist.py", line 48, in findall self.allfiles = findall(dir) File "/usr/lib/python2.3/distutils/filelist.py", line 298, in findall stat = os.stat(fullname) OSError: [Errno 2] No such file or directory: 'broken.symlink' I'll add a workaround in setuptools, but somebody should fix the distutils, too. It looks like the 'findall()' function should be reimplemented with 'os.walk()'. From pje at telecommunity.com Thu Jun 28 20:06:24 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 28 Jun 2007 14:06:24 -0400 Subject: [Distutils] setuptools bug: broken simlinks break setup() In-Reply-To: <46801614.6080000@astraw.com> References: <46801614.6080000@astraw.com> Message-ID: <20070628180446.6D2D83A40A8@sparrow.telecommunity.com> At 12:23 PM 6/25/2007 -0700, Andrew Straw wrote: >In setuptools 0.6c5, a broken symlink in the package tree will cause >'setup develop' to fail, even in the file is inconsequential for the >package. I think this is a bug. > >Here is the output of running "python setup.py develop" in a package >called flydra with a broken symlink called broken.symlink: > >running develop >running egg_info >writing flydra.egg-info/PKG-INFO >writing top-level names to flydra.egg-info/top_level.txt >writing dependency_links to flydra.egg-info/dependency_links.txt >writing entry points to flydra.egg-info/entry_points.txt >error: broken.symlink: No such file or directory > >(I would try this using setuptools==dev, but it seems svn.python.org is >down, or at least my connection to it.) Okay, I've confirmed it's a distutils bug, and the following patch implements a workaround for it in setuptools. It'll be included in setuptools 0.6c7: Index: setuptools/__init__.py =================================================================== --- setuptools/__init__.py (revision 55712) +++ setuptools/__init__.py (working copy) @@ -65,18 +65,17 @@ +def findall(dir = os.curdir): + """Find all files under 'dir' and return the list of full filenames + (relative to 'dir'). + """ + all_files = [] + for base, dirs, files in os.walk(dir): + if base!=os.curdir: + files = [os.path.join(base, f) for f in files] + all_files.extend(filter(os.path.isfile, files)) + return all_files +import distutils.filelist +distutils.filelist.findall = findall # fix findall bug in distutils. - - - - - - - - - - - - - From rlratzel at enthought.com Thu Jun 28 18:05:03 2007 From: rlratzel at enthought.com (Rick Ratzel) Date: Thu, 28 Jun 2007 11:05:03 -0500 (CDT) Subject: [Distutils] Performance implications of having large numbers of eggs? In-Reply-To: <4683D81F.9030606@enthought.com> (message from Dave Peterson on Thu, 28 Jun 2007 10:47:43 -0500) References: <4683D81F.9030606@enthought.com> Message-ID: <20070628160503.64AC21DF4F5@mail.enthought.com> > Date: Thu, 28 Jun 2007 10:47:43 -0500 > From: Dave Peterson > > Has anyone done any investigation into the performance implications of > having large numbers of eggs installed? Is there any sort of > performance hit? > > > It seems to me that having a really large path might slow down imports a > bit, though I suspect this is in C code so probably not a significant > problem. It also seems like there might be some startup penalties due > to the overhead of setting up the path when using eggs, but this is a > one-time cost during python startup, so probably not too bad either. Another option to avoid a startup penalty is to have all eggs installed with -m (not in the easy-install.pth file, or "deactivated") and have code require() the specific dependency. This has the obvious disadvantage of having to change a fair amount of code. But, the advantages are only adding the eggs that are needed to the path when they're needed (instead of every egg in every PYTHONPATH dir), and your code will be sure that it's using the version that it's compatible with. I should mention that I don't have any metrics on the startup penalty, so a change like this may not be worth it if you're only trying to improve that. > > I'm asking because we're in the process to switching our open-source > Enthought Tool Suite library to a distribution of components via eggs > and we're having some internal debate as to whether we need to minimize > the number of eggs or not. It definitely seems nice to have smaller > subsets of functionality -- from the point of being able to make things > stable, managing their APIs, managing cross-component dependencies, and > from the user update size viewpoint. But are we paying a performance > penalty for going too small in scope with our eggs? > > > -- Dave > -- Rick Ratzel - Enthought, Inc. 515 Congress Avenue, Suite 2100 - Austin, Texas 78701 512-536-1057 x229 - Fax: 512-536-1059 http://www.enthought.com From jkp at kirkconsulting.co.uk Thu Jun 28 20:58:23 2007 From: jkp at kirkconsulting.co.uk (Jamie Kirkpatrick) Date: Thu, 28 Jun 2007 19:58:23 +0100 Subject: [Distutils] Missing options for setuptools develop Message-ID: Couple of things. Currently there is no way to pass custom options through setuptools "develop" to distutils build_ext. What this means in practice is that you cannot do the following things: ? pass the --debug flag to build_ext so as to have your module build against debug libraries. ? pass any of the SWIG options through. ? I'm sure there are other limitations. Because of this I ended up abandoning the swig functionality of distutils and running that as a pre distuils phase of my build scripts - it would be better If I could use it through distutils though. The show stopper is the fact I can't build against debug libraries - this means I cannot mix the --debug type builds and the develop functionality, something that seems like a logical thing to do. Is there any chance that someone could find a way to allow options to be passed through to build_ext as it is run? I'm sure its just a case of adding another option to setuptools develop and then passing it through at the relevant point. If noone wants to take it on I'm happy to write a patch if we all agree its the way to go. Thanks in advance Jamie From pje at telecommunity.com Thu Jun 28 23:14:34 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 28 Jun 2007 17:14:34 -0400 Subject: [Distutils] Missing options for setuptools develop In-Reply-To: References: Message-ID: <20070628211225.11F383A40A8@sparrow.telecommunity.com> At 07:58 PM 6/28/2007 +0100, Jamie Kirkpatrick wrote: >Currently there is no way to pass custom options through setuptools >"develop" to distutils build_ext. Did you try this: setup.py build_ext --debug develop setuptools' build_ext command doesn't rebuild an existing extension built in the build/ directory, even when doing an --inplace build (as it does for "develop"). From pje at telecommunity.com Fri Jun 29 21:36:45 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 29 Jun 2007 15:36:45 -0400 Subject: [Distutils] setuptools http auth In-Reply-To: <23758615.1109031183143417333.JavaMail.nabble@isper.nabble. com> References: <23758615.1109031183143417333.JavaMail.nabble@isper.nabble.com> Message-ID: <20070629193439.8FF2D3A40A8@sparrow.telecommunity.com> At 11:56 AM 6/29/2007 -0700, Matt.Callaghan at cbsa-asfc.gc.ca wrote: >Hello Phillip; > >as per the following discusison: >http://www.nabble.com/setuptools-http-auth-t3629554.html > >Did you ever implement the more complete update? > >I'm using: >http_proxy=http://username:password at my_proxy:port_number > >When I try >~> sudo easy_install Pexpect >root's password: ******* >Searching for Pexpect >Reading http://cheeseshop.python.org/pypi/Pexpect/ >error: Download error: (-2, 'Name or service not known') > >I believe this is the same issue as discussed in the above thread. > >Could you please let me know if this has been fixed? Thanks so much The HTML-encoding issue was fixed in 0.6c6, as far as I know. From jorge.vargas at gmail.com Sat Jun 30 10:47:45 2007 From: jorge.vargas at gmail.com (Jorge Vargas) Date: Sat, 30 Jun 2007 04:47:45 -0400 Subject: [Distutils] converting a distutil command to setuptools Message-ID: <32822fe60706300147r598b8cdcg9009ac61205334dc@mail.gmail.com> hi I'm porting a distutils package to setuptools they have a Command that takes the po files and compiles them and then they are feeding a cmd Class to distutils, I read in the docs that the way of doing this in setuptools is with an entry point and that the entry point should be provided by a preinstalled package, the problem I see with that is that we'll need 2 packages to install one so is there a better way to do the following? from distutils.command.build import build as _build import msgfmt #this is the compiler class build_trans(cmd.Command): description = 'Compile .po files into .mo files' def initialize_options(self): pass def finalize_options(self): pass def run(self): po_dir = os.path.join(os.path.dirname(__file__), 'po') for path, names, filenames in os.walk(po_dir): for f in filenames: if f.endswith('.po'): lang = f[:len(f) - 3] src = os.path.join(path, f) dest_path = os.path.join('build', 'locale', lang, 'LC_MESSAGES') dest = os.path.join(dest_path, 'deluge.mo') if not os.path.exists(dest_path): os.makedirs(dest_path) if not os.path.exists(dest): print 'Compiling %s' % src msgfmt.make(src, dest) else: src_mtime = os.stat(src)[8] dest_mtime = os.stat(dest)[8] if src_mtime > dest_mtime: print 'Compiling %s' % src msgfmt.make(src, dest) class build(_build): sub_commands = _build.sub_commands + [('build_trans', None)] def run(self): _build.run(self) Also is there any setuptools enhancement to Extension, to handle C++ compilation and dependencies check? From fccoelho at fiocruz.br Sat Jun 30 14:22:32 2007 From: fccoelho at fiocruz.br (Flavio Codeco Coelho) Date: Sat, 30 Jun 2007 09:22:32 -0300 Subject: [Distutils] upgrade functionality Message-ID: <46864B08.1040408@fiocruz.br> Hello, Does setuptools have any mechanism for upgrading all packages installed with easy_install? Something similar to an "apt-get upgrade"? I have dozens of packages that have been installed with easy_install, to the point that I can't even recall the whole list. I worry that they are getting outdated... Naturally, the packages I Use more often get upgrade as I see fit but the others... Thanks, Fl?vio From pje at telecommunity.com Sat Jun 30 16:51:50 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 30 Jun 2007 10:51:50 -0400 Subject: [Distutils] converting a distutil command to setuptools In-Reply-To: <32822fe60706300147r598b8cdcg9009ac61205334dc@mail.gmail.co m> References: <32822fe60706300147r598b8cdcg9009ac61205334dc@mail.gmail.com> Message-ID: <20070630144945.AEF823A40B2@sparrow.telecommunity.com> At 04:47 AM 6/30/2007 -0400, Jorge Vargas wrote: >hi > >I'm porting a distutils package to setuptools they have a Command that >takes the po files and compiles them and then they are feeding a cmd >Class to distutils, I read in the docs that the way of doing this in >setuptools is with an entry point and that the entry point should be >provided by a preinstalled package, That's if you want the command to be globally available. If you only need to have the command for one project, the one that's being set up, then the 'cmdclass' method still works just fine. >Also is there any setuptools enhancement to Extension, to handle C++ >compilation and dependencies check? setuptools.extension.Extension only adds Pyrex support; other than that, it's the same as the distutils. From pje at telecommunity.com Sat Jun 30 16:56:28 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 30 Jun 2007 10:56:28 -0400 Subject: [Distutils] upgrade functionality In-Reply-To: <46864B08.1040408@fiocruz.br> References: <46864B08.1040408@fiocruz.br> Message-ID: <20070630145418.932CF3A40B2@sparrow.telecommunity.com> At 09:22 AM 6/30/2007 -0300, Flavio Codeco Coelho wrote: >Hello, > >Does setuptools have any mechanism for upgrading all packages installed >with easy_install? Something similar to an "apt-get upgrade"? Well, you can do something like from setuptools.command.easy_install import main as install from pkg_resources import Environment install(['-U'] + list(Environment())) That would be the equivalent of running "easy_install -U names of packages" for all installed packages. >I have dozens of packages that have been installed with easy_install, to >the point that I can't even recall the whole list. A quick recipe for listing active installed packages: python -c "for dist in __import__('pkg_resources').working_set: print dist" This won't list eggs that are installed but not active on sys.path. (e.g. older versions of packages, or ones installed in --multi-version mode). From rlratzel at enthought.com Sat Jun 30 18:35:52 2007 From: rlratzel at enthought.com (Rick Ratzel) Date: Sat, 30 Jun 2007 11:35:52 -0500 (CDT) Subject: [Distutils] upgrade functionality In-Reply-To: <46864B08.1040408@fiocruz.br> (message from Flavio Codeco Coelho on Sat, 30 Jun 2007 09:22:32 -0300) References: <46864B08.1040408@fiocruz.br> Message-ID: <20070630163552.79E521DF520@mail.enthought.com> > Date: Sat, 30 Jun 2007 09:22:32 -0300 > From: Flavio Codeco Coelho > > Hello, > > Does setuptools have any mechanism for upgrading all packages installed > with easy_install? Something similar to an "apt-get upgrade"? > > I have dozens of packages that have been installed with easy_install, to > the point that I can't even recall the whole list. I worry that they are > getting outdated... > > Naturally, the packages I Use more often get upgrade as I see fit but > the others... > > Thanks, > > Fl=E1vio With Enthought's Enstaller, you do the equivalent of "apt-get upgrade" by passing no args to -U: enstaller -U You can also list if there are upgrades available without installing them with -u (lowercase), and simply list what you have installed, the location, and if they're "active" or not with -l. Here's some additional documentation from the Enstaller wiki page, if you're interested: https://svn.enthought.com/enthought/wiki/Enstaller#Examples Because Enstaller also includes a fairly platform-specific GUI, only Windows XP and RedHat 3 64-bit are available at the moment. I will be making a change to the setup script so we can deploy only the command-line version for the remaining platforms while we port the GUI to those platforms. -- Rick Ratzel - Enthought, Inc. 515 Congress Avenue, Suite 2100 - Austin, Texas 78701 512-536-1057 x229 - Fax: 512-536-1059 http://www.enthought.com From jorge.vargas at gmail.com Sat Jun 30 20:02:39 2007 From: jorge.vargas at gmail.com (Jorge Vargas) Date: Sat, 30 Jun 2007 14:02:39 -0400 Subject: [Distutils] setuptools, how to read metadata programatically? Message-ID: <32822fe60706301102r63188b9dvf331c4b054453e5d@mail.gmail.com> hi I'm building a plugin manager on top of setuptools and I'll like to know how could I read the package metadata if I got it's EntryPoint. basically what I need is get the author, version,etc. I found a pkg_resources.EggMetadata but i'm not sure how to instantiate it because I don't know what a "importer" means in this context. From pje at telecommunity.com Sat Jun 30 20:24:12 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 30 Jun 2007 14:24:12 -0400 Subject: [Distutils] setuptools, how to read metadata programatically? In-Reply-To: <32822fe60706301102r63188b9dvf331c4b054453e5d@mail.gmail.co m> References: <32822fe60706301102r63188b9dvf331c4b054453e5d@mail.gmail.com> Message-ID: <20070630183033.0148C3A40A3@sparrow.telecommunity.com> At 02:02 PM 6/30/2007 -0400, Jorge Vargas wrote: >hi > >I'm building a plugin manager on top of setuptools and I'll like to >know how could I read the package metadata if I got it's EntryPoint. Use entrypoint.dist.get_metadata('PKG-INFO') to get the PKG-INFO file as a string. See the PKG-INFO format PEPs to understand what's in there. Use entrypoint.dist.version to get the version, without needing to parse PKG-INFO. (Well, it may get parsed for you, if it's a "develop" egg. But the point is, you don't need to explicitly mess with PKG-INFO unless you want to get other data than the version.) >basically what I need is get the author, version,etc. I found a >pkg_resources.EggMetadata but i'm not sure how to instantiate it >because I don't know what a "importer" means in this context. You don't need to instantiate those metadata classes; that's done for you automatically by functions like 'find_distributions()'. The entrypoint.dist attribute will give you a distribution with the right type of metadata already attached. Likewise, distribution objects obtained from a WorkingSet or Environment will be properly initialized as well. From jorge.vargas at gmail.com Sat Jun 30 22:31:37 2007 From: jorge.vargas at gmail.com (Jorge Vargas) Date: Sat, 30 Jun 2007 16:31:37 -0400 Subject: [Distutils] setuptools, how to read metadata programatically? In-Reply-To: <20070630183033.0148C3A40A3@sparrow.telecommunity.com> References: <32822fe60706301102r63188b9dvf331c4b054453e5d@mail.gmail.com> <20070630183033.0148C3A40A3@sparrow.telecommunity.com> Message-ID: <32822fe60706301331r296c8fd5k90e27a9421790a90@mail.gmail.com> On 6/30/07, Phillip J. Eby wrote: > At 02:02 PM 6/30/2007 -0400, Jorge Vargas wrote: > >hi > > > >I'm building a plugin manager on top of setuptools and I'll like to > >know how could I read the package metadata if I got it's EntryPoint. > > Use entrypoint.dist.get_metadata('PKG-INFO') to get the PKG-INFO file > as a string. See the PKG-INFO format PEPs to understand what's in there. > > Use entrypoint.dist.version to get the version, without needing to > parse PKG-INFO. (Well, it may get parsed for you, if it's a > "develop" egg. But the point is, you don't need to explicitly mess > with PKG-INFO unless you want to get other data than the version.) > I went with the parser since I'm building a UI for users to enable/disable plugins I n eed all the metadata. > > >basically what I need is get the author, version,etc. I found a > >pkg_resources.EggMetadata but i'm not sure how to instantiate it > >because I don't know what a "importer" means in this context. > > You don't need to instantiate those metadata classes; that's done for > you automatically by functions like 'find_distributions()'. The > entrypoint.dist attribute will give you a distribution with the right > type of metadata already attached. Likewise, distribution objects > obtained from a WorkingSet or Environment will be properly initialized as well. > great thanks for the pointer, I have worked an attr based class which turned out to be a bit more complicated than what I though in the first place but it does the job nice (and not that cleanly) here is a copy if someone is interested. import pkg_resources import StringIO import rfc822 fields= ['name' ,'version' ,'platform' ,'summary' ,'description' ,'keywords' ,'home_page' ,'author' ,'author_email' ,'license'] class pkg_info_parsed(object): def __init__(self,dist,missingMsg=None): metadata=StringIO.StringIO(dist.get_metadata('PKG-INFO')) messages=rfc822.Message(metadata) print messages.items() for field in fields: if field in ['home_page','author_email']: prop=field.replace('_','-') else: prop=field value=messages.getheader(prop) if missingMsg: if not value or value == 'UNKNOWN': value=missingMsg setattr(self,field,value) one=pkg_resources.iter_entry_points('deluge.plugin').next() metadata=pkg_info_parsed(one.dist,'Not avaliable') print metadata.name print metadata.version print metadata.platform print metadata.summary print metadata.description print metadata.keywords print metadata.home_page print metadata.author print metadata.author_email print metadata.license >