From ianb at colorstudy.com Fri Jun 3 20:51:20 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 03 Jun 2005 13:51:20 -0500 Subject: [Distutils] EasyInstall: bootstrapping Message-ID: <42A0A6A8.4010207@colorstudy.com> I added a subcommand to Paste "paster install X", which is basically just easy_install with some hooks into Paste's configuration. But anyway, now I'm wondering how to get the whole thing going. You can't run easy_install without setuptools installed. I'm inclined to install easy_install with easy_install, but obviously that won't work ;) Of course, I can download the egg and put that into the path; that's what Paste's setup.py does currently (which also gives me package_data on older Pythons as well). But where should I put the egg? For setup.py it doesn't matter that much, but it matters more when I want to keep it around for future commands. Should I download it, put it on the path, then use it to install itself? I dunno, I'm just feeling a little mixed up. I'm also thinking I could start installing things globally now that there's essentially versioned imports... except that I probably have to hardcode the versions when deploying, so I don't accidentally upgrade apps without meaning to. Or I can stick with app-packages to be extra safe. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at imagescape.com Fri Jun 3 22:56:36 2005 From: ianb at imagescape.com (Ian Bicking) Date: Fri, 03 Jun 2005 15:56:36 -0500 Subject: [Distutils] EasyInstall: installing from svn Message-ID: <42A0C404.9060906@imagescape.com> Now I'm feeling confused about how installation from svn should work. Or more generally, how deployment and development work. Usually in development, I like to install libraries directly out of svn (no distutil step). Then I can fiddle with them if necessary, commit changes (or create diffs if I don't have commit access), etc. This is all much easier to do when there's no distutil step. When deploying, that's not really what I want to do, even though I might very well be deploying from a repository. What easy_install does now with svn repositories is good in that case. The problem is that the two are conflicting. If I require a package, I'm not going to get the svn checkout, even if the checkout was on the path initially. Since the egg package names don't map to Python packages, you can't even really tell that a package is already available without egg installation. The only way I can think of right now to safely do this would be: try: import wsgiutils except ImportError: require('WSGI-Utils') Ick. So how can I set up a working environment that is close, but not too close, to the deployment environment? Maybe there's a way that, up front, I can specify which packages are made available through alternate methods? E.g., fulfill_requirement('WSGI-Utils', 'checkouts/WSGIUtils') (where adding the second path to sys.path will fulfill the requirement, irrespective of any eggs). Ian From bob at redivi.com Fri Jun 3 23:07:11 2005 From: bob at redivi.com (Bob Ippolito) Date: Fri, 3 Jun 2005 14:07:11 -0700 Subject: [Distutils] EasyInstall: installing from svn In-Reply-To: <42A0C404.9060906@imagescape.com> References: <42A0C404.9060906@imagescape.com> Message-ID: On Jun 3, 2005, at 1:56 PM, Ian Bicking wrote: > Now I'm feeling confused about how installation from svn should work. > Or more generally, how deployment and development work. > > Usually in development, I like to install libraries directly out of > svn > (no distutil step). Then I can fiddle with them if necessary, commit > changes (or create diffs if I don't have commit access), etc. This is > all much easier to do when there's no distutil step. In development, with svn libraries, I check out the source tree and create a .pth file . This .pth file does one of two things: 1. If the source requires distutils to build stuff (i.e. PyObjC), then I will point the .pth at the distutils build directory, so I can simply "python setup.py build" to update. 2. If the source does not (i.e. Twisted), I point the .pth at the right place in the source tree From an EasyInstall perspective, 2 is hard to figure out because there is no general way (without running distutils, anyway) to figure out where the right place is. You could do it heuristically, but you'd probably be wrong :) -bob From ianb at colorstudy.com Fri Jun 3 23:07:39 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 03 Jun 2005 16:07:39 -0500 Subject: [Distutils] EasyInstall: non-distutil packages Message-ID: <42A0C69B.2010001@colorstudy.com> I'm starting to think that some sort of patch system is the best way to get non-distutil packages installed. One of the reasons is that I just got it in my head that I'd like to install Javascript libraries using this stuff, and they certainly aren't going to have setup.py's. I don't actually know how that's going to work... but I think there's something useful there. But then, I'm not sure if patches are even that important. Instead maybe it makes sense to create a package that doesn't include the data, but fetches it on installation. You could just distribute a zip file with one member (setup.py) that used urllib or svn to fetch the installation. But, well, the more appropriate way to do this would be to use easy_install's functions for downloading and unpacking. But... does that mean that the setup.py has the location hardcoded of the non-distutil package it is installing? That's a little annoying. I dunno... I haven't thought this through, but this is what I'm thinking at the moment. (I happen to be deploying some software today, and using easy_install a little, and thinking about how I can use it more) -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Fri Jun 3 23:12:53 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 03 Jun 2005 17:12:53 -0400 Subject: [Distutils] EasyInstall: installing from svn In-Reply-To: <42A0C404.9060906@imagescape.com> Message-ID: <5.1.1.6.0.20050603170402.02209f10@mail.telecommunity.com> At 03:56 PM 6/3/2005 -0500, Ian Bicking wrote: >Now I'm feeling confused about how installation from svn should work. >Or more generally, how deployment and development work. > >Usually in development, I like to install libraries directly out of svn >(no distutil step). Then I can fiddle with them if necessary, commit >changes (or create diffs if I don't have commit access), etc. This is >all much easier to do when there's no distutil step. > >When deploying, that's not really what I want to do, even though I might >very well be deploying from a repository. What easy_install does now >with svn repositories is good in that case. > >The problem is that the two are conflicting. If I require a package, >I'm not going to get the svn checkout, even if the checkout was on the >path initially. Since the egg package names don't map to Python >packages, you can't even really tell that a package is already available >without egg installation. The only way I can think of right now to >safely do this would be: > >try: > import wsgiutils >except ImportError: > require('WSGI-Utils') > >Ick. So how can I set up a working environment that is close, but not >too close, to the deployment environment? Maybe there's a way that, up >front, I can specify which packages are made available through alternate >methods? E.g., fulfill_requirement('WSGI-Utils', 'checkouts/WSGIUtils') >(where adding the second path to sys.path will fulfill the requirement, >irrespective of any eggs). Ah, you've encountered the part of the egg docs that I haven't written yet. Basically, the solution is to have a PackageName.egg-info directory in the dependent package. pkg_resources.require() will see the directory and use it for the metadata. It will then know that the directory *contianing* the .egg-info directory is the directory that needs to be (and already is) on sys.path. Here's how you do it: go to the library you want to use for development, and run setup.py bdist_egg. This will create (as a side effect) a PackageName.egg-info subdirectory in the directory that you're going to be putting on sys.path. pkg_resources automatically detects this "development egg" and knows what version is present. This can work even for several packages installed in the same sys.path directory, you just have to have a PackageName.egg-info for each one, and you'll need to move or symlink them from their individual package locations. Basically, you're not the first person to have this problem, or rather, you are the first person to have the problem, you're just not the first person to think of it. :) At PyCon, I was describing the egg system design to Fred Drake, and he pointed out the problem of require()-ing something you're developing in this fashion, so later I came up with the PackageName.egg-info trick as a fix. (Originally, PackageName.egg-info was called EGG-INFO.in or something like that, and it lived in a different place; later I realized that the staging area could be reused to provide metadata for in-development packages, and renamed it to use the distro name so that more than one could live in the same parent directory on sys.path.) From pje at telecommunity.com Fri Jun 3 21:53:49 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 03 Jun 2005 15:53:49 -0400 Subject: [Distutils] EasyInstall: bootstrapping In-Reply-To: <42A0A6A8.4010207@colorstudy.com> Message-ID: <5.1.1.6.0.20050603153243.02201bd8@mail.telecommunity.com> At 01:51 PM 6/3/2005 -0500, Ian Bicking wrote: >I added a subcommand to Paste "paster install X", which is basically >just easy_install with some hooks into Paste's configuration. > >But anyway, now I'm wondering how to get the whole thing going. You >can't run easy_install without setuptools installed. I'm inclined to >install easy_install with easy_install, but obviously that won't work ;) > Of course, I can download the egg and put that into the path; that's >what Paste's setup.py does currently (which also gives me package_data >on older Pythons as well). But where should I put the egg? For >setup.py it doesn't matter that much, but it matters more when I want to >keep it around for future commands. Should I download it, put it on the >path, then use it to install itself? > >I dunno, I'm just feeling a little mixed up. I'm also thinking I could >start installing things globally now that there's essentially versioned >imports... except that I probably have to hardcode the versions when >deploying, so I don't accidentally upgrade apps without meaning to. Or >I can stick with app-packages to be extra safe. Hm. I would suggest that the safest tack to take is try importing pkg_resources and do a 'require()' for the setuptools version that you need. If the import fails or the version fails, bail out of setup.py, asking for the user to install the correct setuptools. I think that trying to invoke easy_install recursively or guess where the egg should be installed is a bad idea. (E.g., guessing about the egg location will cause easy_install's sandboxing to shut you down anyway) However, it would certainly be nice to have a way for setuptools-using packages to bootstrap the setuptools install -- and that includes setuptools itself. (My head is starting to spin now.) Hm. Unfortunately, it seems that the only other sane way to do this is to distribute Paste as an egg, and require people to use easy_install to install it, because then any dependency back to setuptools can be resolved properly once easy_install's automatic dependency installation is in place. That doesn't help you much *now*, I realize. The big problem is that 'pkg_resources' really needs to be a singleton; there can't be multiple versions of it floating around at the same time, due to it containing global registries. Otherwise I'd tell you to just stick pkg_resources and easy_install in a package of your own and not worry about it. Anyway, this is definitely a case for putting at least pkg_resources into the standard library as soon as it stabilizes sufficiently. From pje at telecommunity.com Sat Jun 4 19:22:43 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 04 Jun 2005 13:22:43 -0400 Subject: [Distutils] Python Eggs: Major documentation update Message-ID: <5.1.1.6.0.20050604131904.0220cab8@mail.telecommunity.com> I've just completed a major update to the Python Eggs documentation at: http://peak.telecommunity.com/DevCenter/PythonEggs There is an all-new API reference that gives detailed coverage of the most important classes and functions provided by the "pkg_resources" runtime module, as well as an expanded "Developing with Eggs" section that explains more about namespace packages, developing from source checkouts, and how to update modules to find their data files using the resource API instead of by __file__ manipulation that won't work from inside zip files. Please let me know if you have any questions or find any problems. Thanks! From ianb at colorstudy.com Sat Jun 4 23:05:09 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sat, 04 Jun 2005 16:05:09 -0500 Subject: [Distutils] EasyInstall: bootstrapping In-Reply-To: <5.1.1.6.0.20050603153243.02201bd8@mail.telecommunity.com> References: <5.1.1.6.0.20050603153243.02201bd8@mail.telecommunity.com> Message-ID: <42A21785.3010106@colorstudy.com> Phillip J. Eby wrote: > Hm. I would suggest that the safest tack to take is try importing > pkg_resources and do a 'require()' for the setuptools version that you > need. If the import fails or the version fails, bail out of setup.py, > asking for the user to install the correct setuptools. I think that > trying to invoke easy_install recursively or guess where the egg should > be installed is a bad idea. (E.g., guessing about the egg location will > cause easy_install's sandboxing to shut you down anyway) > > However, it would certainly be nice to have a way for setuptools-using > packages to bootstrap the setuptools install -- and that includes > setuptools itself. (My head is starting to spin now.) > > Hm. Unfortunately, it seems that the only other sane way to do this is > to distribute Paste as an egg, and require people to use easy_install to > install it, because then any dependency back to setuptools can be > resolved properly once easy_install's automatic dependency installation > is in place. I wouldn't mind so much requiring easy_install to be installed -- if that's the only firm prerequesite, that's okay. But a Paste checkout gives you docs and examples and unit tests, which aren't part of an egg, so Paste as installed by easy_install isn't complete. One option would be if there was an option to install something the same way you recommend using a development checkout -- don't create an egg, just make sure the PackageName.egg-info directory is in place. Incidentally, should there be a egg_setup command or something short of creating an egg you won't use to create that special directory?) Also, if I was just going to get rid of paster install (which is fine, no big investment there), it does have the plugin-enabling code (where plugins can register themselves with Paste). I'm not sure where that would go; you could put the code into plugins themselves -- very reasonable -- except how do they find the appropriate Paste installation? I hate PYTHONPATH, and that's where this would lead. > That doesn't help you much *now*, I realize. The big problem is that > 'pkg_resources' really needs to be a singleton; there can't be multiple > versions of it floating around at the same time, due to it containing > global registries. Otherwise I'd tell you to just stick pkg_resources > and easy_install in a package of your own and not worry about it. Well, I'm wondering if it's okay to just download the setuptools egg and put it on sys.path manually (obviously not using require() or anything in setuptools). That's pretty easy to do. The only problem is when they have a version of setuptools that is older than required -- if I try to import it (not using require()) and it's too old, then I have to delete it from sys.modules, which starts getting kind of long. Especially since I can't write a library to do this ;) I have thought in the past about writing a setup.py-builder, which might make this more reasonable. > Anyway, this is definitely a case for putting at least pkg_resources > into the standard library as soon as it stabilizes sufficiently. That might help. But only maybe. Having distutils in the stdlib isn't making things any less complicated right now. I almost think it makes sense to just have a hard line for packages that use this stuff: if setuptools isn't installed (or is too old), then go install it and start again. That way, assuming it is popular, it can become kind of standard regardless of your Python version. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Sun Jun 5 00:08:51 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 04 Jun 2005 18:08:51 -0400 Subject: [Distutils] EasyInstall: bootstrapping In-Reply-To: <42A21785.3010106@colorstudy.com> References: <5.1.1.6.0.20050603153243.02201bd8@mail.telecommunity.com> <5.1.1.6.0.20050603153243.02201bd8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050604174402.02193cb8@mail.telecommunity.com> At 04:05 PM 6/4/2005 -0500, Ian Bicking wrote: >Phillip J. Eby wrote: >>Hm. I would suggest that the safest tack to take is try importing >>pkg_resources and do a 'require()' for the setuptools version that you >>need. If the import fails or the version fails, bail out of setup.py, >>asking for the user to install the correct setuptools. I think that >>trying to invoke easy_install recursively or guess where the egg should >>be installed is a bad idea. (E.g., guessing about the egg location will >>cause easy_install's sandboxing to shut you down anyway) >>However, it would certainly be nice to have a way for setuptools-using >>packages to bootstrap the setuptools install -- and that includes >>setuptools itself. (My head is starting to spin now.) >>Hm. Unfortunately, it seems that the only other sane way to do this is >>to distribute Paste as an egg, and require people to use easy_install to >>install it, because then any dependency back to setuptools can be >>resolved properly once easy_install's automatic dependency installation >>is in place. > >I wouldn't mind so much requiring easy_install to be installed -- if >that's the only firm prerequesite, that's okay. But a Paste checkout >gives you docs and examples and unit tests, which aren't part of an egg, >so Paste as installed by easy_install isn't complete. Well, there's always -b, as of 0.3a3, which keeps the unpacked source distribution after installation. Does that help? >One option would be if there was an option to install something the same >way you recommend using a development checkout -- don't create an egg, >just make sure the PackageName.egg-info directory is in place. Ugh. I don't want to do that for stuff that isn't either a development checkout or part of the standard library. Anywhere in between and you start to mess up version management. Note that when an .egg-info directory is in a directory on sys.path, it means the egg in question is *already* on sys.path, which means you can't require() a version that conflicts with it. So, .egg-info is really a trick for development. EasyInstall already takes care of everything you need for installing as an egg directory, so installing .egg-info doesn't really provide any benefit that I see. Now, if somebody wants to manually set a .pth file (or PYTHONPATH et al) to point to a Paste checkout and its included .egg-info, that's fine: just add the .egg-info directory to subversion after you've generated it, and you're all set. This is in fact recommended for development scenarios. You just don't want to try and do this by actually *installing* the package and .egg-info to site-packages, because then you can't get it back out again. > Incidentally, should there be a egg_setup command or something short of > creating an egg you won't use to create that special directory?) I guess I could factor it out into an 'egg_info' command, and call it from 'bdist_egg'. It might take some work to prevent duplication, though, and a big problem is that it wants to build a 'native_libs.txt' in the info directory, which it has to get the information for from the install_lib command. But I could maybe not generate that file in the 'egg_info' command, but then there's duplication of process in bdist_egg to find the egg-info directory. No, it doesn't make any sense to do all that extra work. The idea that the egg is something "you won't use" is an illusion in the long run, and meanwhile the build process helps validates the correctness of your ``setup.py`` and much of the rest of your package. The disk space and CPU to build the egg should be minimal, and you only need to do it again if you change your package's metadata (e.g. its version). The rest of the time you can use it as-is from subversion or CVS. So just use bdist_egg. >Also, if I was just going to get rid of paster install (which is fine, no >big investment there), it does have the plugin-enabling code (where >plugins can register themselves with Paste). I'm not sure where that >would go; you could put the code into plugins themselves -- very >reasonable -- except how do they find the appropriate Paste >installation? I hate PYTHONPATH, and that's where this would lead. I don't understand what you're trying to do here. Why do plugins need to find Paste? Shouldn't Paste find the plugins? My general approach to application plugin installation is that applications have a plugin directory where the plugins' eggs get dropped (either via EasyInstall, or by the app itself). The application then just uses ``find_distributions(directory)`` to iterate over the distributions and check for plugin metadata, and do any registering of services provided by the plugins. (Or, you can use the ``AvailableDistributions`` class to automatically filter, index, and sort the eggs found.) For larger application frameworks comparable to Eclipse in the Java world, those apps would likely keep a registry of already-installed plugins and their registered services, and thus would only scan the plugin directory to locate newly- installed plugins. Some applications might also have an explicit installation process of their own. Anyway, as you can see, there are many architectures you can use to manage application plugins using Python Eggs. I should probably expand on the above a bit and add it to the docs, but I'd really like the rest of EasyInstall to be done, so that it can use EasyInstall as an example of downloading packages and resolving dependencies automatically. (Although in truth most application plugin systems will probably want to just download eggs directly rather than building them, and have some application-centric directory of download URLs.) >I almost think it makes sense to just have a hard line for packages that >use this stuff: if setuptools isn't installed (or is too old), then go >install it and start again. That way, assuming it is popular, it can >become kind of standard regardless of your Python version. That's my inclination as well. I suspect that web application deployment is a killer use case for eggs, too. Certainly it was the use case (besides Chandler) that pushed me to take action on designing and implementing Python Eggs. From p.f.moore at gmail.com Sun Jun 5 17:16:17 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 5 Jun 2005 16:16:17 +0100 Subject: [Distutils] EasyInstall: scripts In-Reply-To: <429A6961.9070809@colorstudy.com> References: <5.1.1.6.0.20050529183121.02444b88@mail.telecommunity.com> <429A6961.9070809@colorstudy.com> Message-ID: <79990c6b050605081641b66b4@mail.gmail.com> On 5/30/05, Ian Bicking wrote: > Phillip J. Eby wrote: > > It might help if there were some kind of metadata for scripts, like to > > indicate whether something is a command-line utility, a graphical > > application, etc. Then distutils could tweak the file extension and/or > > build a custom launcher for it (like Fredrik Lundh's exemaker for > > Windows) as appropriate for the platform. That information could then > > be added to EGG-INFO and/or EGG-SCRIPTS, and used by EasyInstall to do > > script installation and uninstallation. > > That's what I'd be inclined towards. Like, this is what one script I > have looks like: Scripts are a *pain*, basically because so much of the peripheral issues are OS-specific. On Unix, the general preference seems to be a script with no extension, and a #! line, written in Python, for most cases (Unix is nice and uniform like that :-)) On Windows, the file extension determines executability. Opinion seems divided (probably not evenly) between people who prefer a .py extension (along with appropriate PATHEXT/file association magic, which the default install does *not* do) and people who prefer a .bat (or even .exe) wrapper. On Windows, GUI and console applications need to be run with different Python executables, adding another level of complexity. I'd like to see the basic "wrap up a script" functionality tidied up - leave start menu changes, shortcuts, aliases, PATH changes, etc etc, as OS-specific enhancements. Experienced users can do this for themselves, and packagers can write platform-specific "installer" scripts where appropriate. (Personally, I dislike most installers' choices for start menu items anyway :-)) The python -m option solves 90% of the script issue in my view. Simply write your driver script as a Python module, and install it as such. Then it can be invoked via python -m (or pythonw -m if it's a GUI application). Most of the remaining 10% would be covered by extending -m to packages. There is a Cookbook recipe which demonstrates a module which can do this (so we have "python -m execpackage my.package.script arg1 arg2...") but it would be nice if this were built in. This is mainly a namespace issue, though. For eggs, we're 90% of the way to executable eggs. All that's needed is a way of adding a particular egg to sys.path, then using the -m machinery (with a standard module name, such as main, or maybe requiring the user to specify a module, effectively an egg "subcommand", or possibly reading egg metadata). This could be implemented as an egg.py module, so you'd have python -m egg path/to/my/sample.egg arg... (invokes the main module, or one from egg metadata). python -m egg -m subcmd path/to/my/sample.egg arg... (invokes the subcmd module - maybe -m for the subcmd option name isn't an ideal choice...) Of course, on Unix, this can be handled by temporary environment variable setting, like PYTHONPATH=path/to/my/sample.egg python -m main arg... but the point here is to make it cross-platform. As I say, the command lines can look a bit clumsy, but wrapping that up in an alias, or desktop shortcut, or whatever, is something I explicitly defer to platform-specific installers. You'll notice that I've completely sidestepped the idea that distutils build "appropriate" wrappers or launchers, or tweak extensions. Past experience has shown that it's incredibly hard to get any consensus on this sort of thing. But it *is* possible (IMHO) to get 90% of the way there, just by getting people to use -m compatible modules. Of course, that isn't 2.3 compatible. But I'd rather move forward than keep trying to work around the same issues - -m was added to address something that people had tried, and failed, to solve in pure Python 2.3, so why reopen that again? Paul. From p.f.moore at gmail.com Sun Jun 5 17:40:45 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 5 Jun 2005 16:40:45 +0100 Subject: [Distutils] EasyInstall: List of installed packages Message-ID: <79990c6b05060508407f220969@mail.gmail.com> Is there a way of asking EasyInstall "what packages are installed"? Without that, I can't see myself moving away from bdist_wininst. I know that a registry of installed packages starts to move away from the "easy" bit of "EasyInstall", but to be honest, EasyInstall is no easier *for me* than double clicking on a bdist_wininst executable. Paul. From pje at telecommunity.com Sun Jun 5 20:12:57 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 05 Jun 2005 14:12:57 -0400 Subject: [Distutils] EasyInstall: List of installed packages In-Reply-To: <79990c6b05060508407f220969@mail.gmail.com> Message-ID: <5.1.1.6.0.20050605140329.02192468@mail.telecommunity.com> At 04:40 PM 6/5/2005 +0100, Paul Moore wrote: >Is there a way of asking EasyInstall "what packages are installed"? Not the command-line script. However "ls site-packages/*.egg" or "dir site-packages\*.egg" depending on your platform will give you the answer. I though about having a --list-packages option, but I tend to dislike having options that completely change a script's behavior and complicate the logic of determining whether you supplied valid options. Still, I suppose I could add it if it just made it list all installed packages after completing its regular operations. That list could get pretty long, though. Maybe there should be a separate command altogether; certainly it'd be easy to write one. Here's some sample code for a package lister: import pkg_resources distros = pkg_resources.AvailableDistributions() # scan sys.path for key in distros: for dist in distros[key]: print dist.name, dist.version If you give AvailableDistributions a list of directories, it'll search those instead of sys.path. Or, you can call its 'scan()' method to scan additional directories after you've created it. >Without that, I can't see myself moving away from bdist_wininst. > >I know that a registry of installed packages starts to move away from >the "easy" bit of "EasyInstall", but to be honest, Ah, I see the confusion. Yes, there's a "registry" of packages installed by EasyInstall. They're eggs, so they're in directories or files named for the distribution, version, python version, and platform. So the installation *itself* is the registry, and therefore can't be corrupted or out of date. > EasyInstall is no >easier *for me* than double clicking on a bdist_wininst executable. Perhaps. I hope this weekend however to complete PyPI integration for EasyInstall. It's still not a GUI, but it'll be convenient. From pje at telecommunity.com Sun Jun 5 20:28:46 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 05 Jun 2005 14:28:46 -0400 Subject: [Distutils] EasyInstall: scripts In-Reply-To: <79990c6b050605081641b66b4@mail.gmail.com> References: <429A6961.9070809@colorstudy.com> <5.1.1.6.0.20050529183121.02444b88@mail.telecommunity.com> <429A6961.9070809@colorstudy.com> Message-ID: <5.1.1.6.0.20050605141313.021be488@mail.telecommunity.com> At 04:16 PM 6/5/2005 +0100, Paul Moore wrote: >The python -m option solves 90% of the script issue in my view. Simply >write your driver script as a Python module, and install it as such. >Then it can be invoked via python -m (or pythonw -m if it's a GUI >application). > >Most of the remaining 10% would be covered by extending -m to >packages. There is a Cookbook recipe which demonstrates a module which >can do this (so we have "python -m execpackage my.package.script arg1 >arg2...") but it would be nice if this were built in. This is mainly a >namespace issue, though. > >For eggs, we're 90% of the way to executable eggs. All that's needed >is a way of adding a particular egg to sys.path, then using the -m >machinery (with a standard module name, such as main, or maybe >requiring the user to specify a module, effectively an egg >"subcommand", or possibly reading egg metadata). This could be >implemented as an egg.py module, so you'd have > > python -m egg path/to/my/sample.egg arg... > >(invokes the main module, or one from egg metadata). > > python -m egg -m subcmd path/to/my/sample.egg arg... > >(invokes the subcmd module - maybe -m for the subcmd option name isn't >an ideal choice...) > >Of course, on Unix, this can be handled by temporary environment >variable setting, like I like these ideas, but for the moment, I think we need to make existing packages' scripts work, or there's not going to be enough initial adoption. My current plan is to simply tack an extra line between the #! line and the rest of the script, assuming it has either a .py extension or a #! line containing "python". The line would read something like: import pkg_resources; pkg_resources.require("OwningPackage==exactversion"); del pkg_resources This would work as long as there were no 'from __future__' import lines in the script. (Which we could check for, and move the line after, but I'm not sure I want to get that complicated, certainly not in the first version of the feature.) The script would then be installed according to normal distutils practices in all other respects. Of course, to support this, bdist_egg will need to add the scripts to an 'EGG-INFO/scripts' subdirectory. That way, you get them even when installing from an egg, and when you switch active library versions using EasyInstall, it can remove the old scripts and install the new ones. >But it *is* possible (IMHO) to get 90% of the way >there, just by getting people to use -m compatible modules. Of course, >that isn't 2.3 compatible. But I'd rather move forward than keep >trying to work around the same issues - -m was added to address >something that people had tried, and failed, to solve in pure Python >2.3, so why reopen that again? I'd definitely like to encourage people to do that, and build up assistive infrastructure for it. But, EasyInstall has to have a strategy for dealing with today's scripts, or it's not very useful today. From ianb at colorstudy.com Sun Jun 5 21:19:52 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 05 Jun 2005 14:19:52 -0500 Subject: [Distutils] EasyInstall: scripts In-Reply-To: <79990c6b050605081641b66b4@mail.gmail.com> References: <5.1.1.6.0.20050529183121.02444b88@mail.telecommunity.com> <429A6961.9070809@colorstudy.com> <79990c6b050605081641b66b4@mail.gmail.com> Message-ID: <42A35058.6010800@colorstudy.com> Paul Moore wrote: > On 5/30/05, Ian Bicking wrote: > >>Phillip J. Eby wrote: >> >>>It might help if there were some kind of metadata for scripts, like to >>>indicate whether something is a command-line utility, a graphical >>>application, etc. Then distutils could tweak the file extension and/or >>>build a custom launcher for it (like Fredrik Lundh's exemaker for >>>Windows) as appropriate for the platform. That information could then >>>be added to EGG-INFO and/or EGG-SCRIPTS, and used by EasyInstall to do >>>script installation and uninstallation. >> >>That's what I'd be inclined towards. Like, this is what one script I >>have looks like: > > > Scripts are a *pain*, basically because so much of the peripheral > issues are OS-specific. > > On Unix, the general preference seems to be a script with no > extension, and a #! line, written in Python, for most cases (Unix is > nice and uniform like that :-)) > > On Windows, the file extension determines executability. Opinion seems > divided (probably not evenly) between people who prefer a .py > extension (along with appropriate PATHEXT/file association magic, > which the default install does *not* do) and people who prefer a .bat > (or even .exe) wrapper. On Windows, GUI and console applications need > to be run with different Python executables, adding another level of > complexity. I agree that Windows scripts are messy and poorly understood. I for one don't have any opinion or knowledge about how to make a well-behaved Windows script... > I'd like to see the basic "wrap up a script" functionality tidied up - > leave start menu changes, shortcuts, aliases, PATH changes, etc etc, > as OS-specific enhancements. Experienced users can do this for > themselves, and packagers can write platform-specific "installer" > scripts where appropriate. (Personally, I dislike most installers' > choices for start menu items anyway :-)) > > The python -m option solves 90% of the script issue in my view. Simply > write your driver script as a Python module, and install it as such. > Then it can be invoked via python -m (or pythonw -m if it's a GUI > application). ... however, I know how to make a good script on Unix and Mac machines, and I'm hardly going to give that up for Windows. I'd have a hard time replacing: paster serve with: python -m execpackage paste.app_setup serve Or even: /path/to/svn/checkout/scripts/paster serve With: PYTHONPATH=/path/to/svn/checkout python -m execpackage paste.app_setup serve And I don't even know how to do that on Windows. So, right now the status quo is perfectly fine on Unixy machines, and we can't make that worse. > Most of the remaining 10% would be covered by extending -m to > packages. There is a Cookbook recipe which demonstrates a module which > can do this (so we have "python -m execpackage my.package.script arg1 > arg2...") but it would be nice if this were built in. This is mainly a > namespace issue, though. > > For eggs, we're 90% of the way to executable eggs. All that's needed > is a way of adding a particular egg to sys.path, then using the -m > machinery (with a standard module name, such as main, or maybe > requiring the user to specify a module, effectively an egg > "subcommand", or possibly reading egg metadata). This could be > implemented as an egg.py module, so you'd have > > python -m egg path/to/my/sample.egg arg... > > (invokes the main module, or one from egg metadata). > > python -m egg -m subcmd path/to/my/sample.egg arg... > > (invokes the subcmd module - maybe -m for the subcmd option name isn't > an ideal choice...) > > Of course, on Unix, this can be handled by temporary environment > variable setting, like > > PYTHONPATH=path/to/my/sample.egg python -m main arg... > > but the point here is to make it cross-platform. > > As I say, the command lines can look a bit clumsy, but wrapping that > up in an alias, or desktop shortcut, or whatever, is something I > explicitly defer to platform-specific installers. I would be fine with that too, but I think installers need to take up the cause before this will work. And anyway, easy_install is an installer, so this is within its scope. I think I'd like these semantics: setup(... script_packages=[{'paster': 'paste.app_setup:main'}]) Then a script is generated by that name (with some extension on Windows, I don't care what), and it does: import paste.app_setup sys.exit(main(sys.argv) or 0) Well... somewhere in there it would be nice to make the distinction between console and GUI applications, to make that script use pythonw on Windows. One issue, though, is how to run the script differently. For instance, how to use a different installation/version of Python, or how to best setup or change sys.path. For instance, I always try to detect if the script is being run out of a checkout, and fix up the path when doing so (not only does this avoid PYTHONPATH setup, it means that you won't accidentally run a script with a perhaps older installed version of a package if you forget to fix PYTHONPATH). > You'll notice that I've completely sidestepped the idea that distutils > build "appropriate" wrappers or launchers, or tweak extensions. Past > experience has shown that it's incredibly hard to get any consensus on > this sort of thing. But it *is* possible (IMHO) to get 90% of the way > there, just by getting people to use -m compatible modules. Of course, > that isn't 2.3 compatible. But I'd rather move forward than keep > trying to work around the same issues - -m was added to address > something that people had tried, and failed, to solve in pure Python > 2.3, so why reopen that again? I don't see why this should be so difficult to come to consensus on. So far consensus hasn't been very useful, because consensus would just turn into ad hoc scripts and installation processes. Recipes can't build consensus; but setuptools is actual code, not a recipe. And how strongly can people feel about .bat vs. .py vs. .exe? These aren't interesting enough problems to deserve fanaticism ;) -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From p.f.moore at gmail.com Sun Jun 5 22:55:46 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 5 Jun 2005 21:55:46 +0100 Subject: [Distutils] EasyInstall: List of installed packages In-Reply-To: <5.1.1.6.0.20050605140329.02192468@mail.telecommunity.com> References: <79990c6b05060508407f220969@mail.gmail.com> <5.1.1.6.0.20050605140329.02192468@mail.telecommunity.com> Message-ID: <79990c6b050605135577366316@mail.gmail.com> On 6/5/05, Phillip J. Eby wrote: > At 04:40 PM 6/5/2005 +0100, Paul Moore wrote: > >Is there a way of asking EasyInstall "what packages are installed"? > > Not the command-line script. However "ls site-packages/*.egg" or "dir > site-packages\*.egg" depending on your platform will give you the answer. [...] > Ah, I see the confusion. Yes, there's a "registry" of packages installed > by EasyInstall. They're eggs, so they're in directories or files named for > the distribution, version, python version, and platform. So the > installation *itself* is the registry, and therefore can't be corrupted or > out of date. I see. On the other hand, there's this .pth file containing the egg name, so you can't just add & remove egg files, you do need to go through the EasyInstall interface or things get mixed up. So there is some (relatively minor) level at which things are more "managed" than just file copies. And of course, egg files with non-standard names are "not in the registry" to some extent. So there's still a sense in which a registry is needed... Paul. From pje at telecommunity.com Sun Jun 5 23:35:39 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 05 Jun 2005 17:35:39 -0400 Subject: [Distutils] EasyInstall: List of installed packages In-Reply-To: <79990c6b050605135577366316@mail.gmail.com> References: <5.1.1.6.0.20050605140329.02192468@mail.telecommunity.com> <79990c6b05060508407f220969@mail.gmail.com> <5.1.1.6.0.20050605140329.02192468@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050605173006.022019b8@mail.telecommunity.com> At 09:55 PM 6/5/2005 +0100, Paul Moore wrote: >On 6/5/05, Phillip J. Eby wrote: > > At 04:40 PM 6/5/2005 +0100, Paul Moore wrote: > > >Is there a way of asking EasyInstall "what packages are installed"? > > > > Not the command-line script. However "ls site-packages/*.egg" or "dir > > site-packages\*.egg" depending on your platform will give you the answer. >[...] > > Ah, I see the confusion. Yes, there's a "registry" of packages installed > > by EasyInstall. They're eggs, so they're in directories or files named for > > the distribution, version, python version, and platform. So the > > installation *itself* is the registry, and therefore can't be corrupted or > > out of date. > >I see. On the other hand, there's this .pth file containing the egg >name, so you can't just add & remove egg files, you do need to go >through the EasyInstall interface or things get mixed up. Not really; invalid entries in .pth do nothing but maybe slow down some imports. EasyInstall isn't confused by them at all, and will remove them the next time you do an install operation involving a distribution with the same name. > So there is >some (relatively minor) level at which things are more "managed" than >just file copies. Extremely minor. :) Note that the .pth is not a registry; the egg files and directories themselves are the registry, so it's a very robust approach. >And of course, egg files with non-standard names are >"not in the registry" to some extent. Well, you can always look at EGG-INFO/PKG-INFO. The egg runtime system only does this under very limited circumstances, for performance reasons. So you could create a tool that would verify eggs' name and version from PKG-INFO. From p.f.moore at gmail.com Mon Jun 6 00:06:48 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 5 Jun 2005 23:06:48 +0100 Subject: [Distutils] EasyInstall: scripts In-Reply-To: <42A35058.6010800@colorstudy.com> References: <5.1.1.6.0.20050529183121.02444b88@mail.telecommunity.com> <429A6961.9070809@colorstudy.com> <79990c6b050605081641b66b4@mail.gmail.com> <42A35058.6010800@colorstudy.com> Message-ID: <79990c6b05060515067f57faaf@mail.gmail.com> On 6/5/05, Ian Bicking wrote: > I agree that Windows scripts are messy and poorly understood. I for one > don't have any opinion or knowledge about how to make a well-behaved > Windows script... I do :-) Or at least, I think I do... :-) > ... however, I know how to make a good script on Unix and Mac machines, > and I'm hardly going to give that up for Windows. But if you make your code work via -m, then at least you haven't left us poor Windows users without any driver script at all! > I'd have a hard time replacing: > > paster serve > > with: > > python -m execpackage paste.app_setup serve > > Or even: > > /path/to/svn/checkout/scripts/paster serve > > With: > > PYTHONPATH=/path/to/svn/checkout python -m execpackage > paste.app_setup serve > > And I don't even know how to do that on Windows. > > So, right now the status quo is perfectly fine on Unixy machines, and we > can't make that worse. I didn't intend for one moment to suggest that we should. But I do feel, quite strongly, that it's platform-specific. People keep trying to make wrapper scripts "just work" cross-platform, and that's what I see as the mistake. OTOH, python -m execpackage paste.app_setup serve *can* be cross-platform. So writing a Unix wrapper round that, and a different Windows-specific wrapper, etc, is not an unreasonable approach. (Nor is writing a more capable Unix-specific implementation). Why do you need your "paster" script to be more complex than #!/bin/sh exec python -m execpackage paste.app_setup $* (excuse my pidgin Unix - assume I knew enough to write this properly :-))? What *I* am trying to move away from is having non-trivial cross-platform code in .py files installed as scripts. An example is with Twisted, twistd.py is installed as a script, but it has some non-trivial platform detection logic in it. It wouldn't take much to move that logic to the twisted.scripts.twistd module (or somewhere similar) so the twistd daemon could be run directly from a python -m invocation. Keep a trivial wrapper by all means, but don't make the wrapper the only way of getting the functionality. > I don't see why this should be so difficult to come to consensus on. So > far consensus hasn't been very useful, because consensus would just turn > into ad hoc scripts and installation processes. Recipes can't build > consensus; but setuptools is actual code, not a recipe. And how > strongly can people feel about .bat vs. .py vs. .exe? These aren't > interesting enough problems to deserve fanaticism ;) Welcome to Windows :-) Seriously, some issues: 1. .exe is binary, so not customisable. Also, you need two versions of any wrapper exe, one a console app and one a GUI one. But this is not a bad option. Fredrik Lundh has implemented something along these likes (exemaker). It's not a popular option though - maybe because the binaries must be built on Windows, maybe because of virus concerns, I don't know. 2. The default Python installer doesn't do all of the necessary magicto make .py files "executable". That's a deliberate policy decision, and not likely to change. It's my preference, though (I make the necessary changes manually). 3. .bat files are mildly, but annoyingly, non-cross-platform. As Windows 9x and command.com die, this is becoming less of an issue, but it's still there. Also, if you hit Ctrl-C to interrupt a .bat wrapper, the OS generates a horribly unfriendly "Abort batch job (Yes/No/Cancel)?" prompt. Oh, and there is no way to use .bat files to create a GUI wrapper - you *always* get a console window. As I say, my preference is (2), and I can argue that it's the only good one. But it ain't going to happen, so we're left with irritating workarounds. As long as I can get all I need by just ignoring the Python scripts directory, that's fine - but while some scripts include nontrivial logic, I can't do that. Paul. From pje at telecommunity.com Mon Jun 6 00:22:48 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 05 Jun 2005 18:22:48 -0400 Subject: [Distutils] EasyInstall 0.4a1: now with PyPI autodownload Message-ID: <5.1.1.6.0.20050605180352.02208810@mail.telecommunity.com> I've just released a new version of EasyInstall, that supports automatic package location via PyPI; installing a package can now be as simple as: easy_install "SQLObject>=0.6" There are many new options and features related to this; see the EasyInstall home page for more information: http://peak.telecommunity.com/DevCenter/EasyInstall For the benefit of the PyPI maintainers, here is a summary of EasyInstall's fairly minimal assumptions about PyPI's current behavior, assuming that "base-url" is the root URL of the package index: * Going to "base-url/SomePackage" produces an HTML page that either has a title containing "Index of Packages" and links to zero or more pages for specific versions, or else it is a single-version package page. * Single-version package pages may have a home page and download URL link, each of which occurs after 'Home Page' and 'Download URL' respectively, if present. * Going to "base-url/" (note trailing '/') produces an HTML page containing links to all active versions of registered packages * Links to package pages always have URLs of the form "base-url/SomePackage/itsVersion" - i.e., exactly two path parts following the base URL, with no query strings, parameters, fragments, etc. EasyInstall should continue to work with PyPI if these assumptions continue to hold. However, I'd also like to suggest that PyPI deprecate the use of spaces and other non-alphanumeric characters (other than '-') in package names, and move to a case-insensitive matching mechanism for package names. (Currently, if a user types a package name in the wrong case, EasyInstall downloads the full package list in order to do its own case-insensitive search.) By the way, EasyInstall does not rely solely on the download URL of a PyPI entry, nor does it assume that the download URL is in fact the URL where the package's source distribution is found. Instead, EasyInstall inspects the URLs for whether the extension suggests an egg or source distribution. If not, it retrieves the listed URL, and if it contains HTML, it scans the HTML for links to egg or source distributions (again identified by extension). It does this for both the home page and the download URL, in case there is a usable download link on the package's home page. This approach was chosen to maximise the odds of successful downloading, given the current contents of PyPI. From postmaster at defecondo.com Mon Jun 6 15:13:10 2005 From: postmaster at defecondo.com (postmaster@defecondo.com) Date: Mon, 06 Jun 2005 15:13:10 +0200 Subject: [Distutils] Allarme Virus Message-ID: <20050606131338.2B2FE1E4003@bag.python.org> Salve. Il server non è riuscito a recapitare il messaggio perchè è infetto da virus. Leggi le righe successive per il dettaglio del problema. The message has been infected virus that you send to vigilanza at defecondo.com delivery failure: the message has been infected virus. --- Below this line is a report of the anti-virus engine. -------------------------------------- Scan started: Mon Jun 6 15:13:07 2005 35866391/your_document.zip: Worm.SomeFool.P FOUND 35866391/your_document.zip: Removed -- summary -- Known viruses: 35438 Engine version: 0.85 Scanned directories: 1 Scanned files: 2 Infected files: 1 Data scanned: 0.03 MB Time: 3.062 sec (0 m 3 s) --- Below this line is a copy of the message. Received: from unknown (HELO defecondo.com) ([62.10.78.206]) (envelope-sender ) by [192.168.188.97] with ESMTP for ; Mon, 06 Jun 2005 15:13:06 +0200 From: distutils-sig at python.org To: vigilanza at defecondo.com Subject: Stolen document Date: Mon, 6 Jun 2005 15:20:32 +0200 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPart_000_0016----=_NextPart_000_0016" X-Priority: 3 X-MSMail-Priority: Normal From pje at telecommunity.com Mon Jun 6 16:44:19 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 06 Jun 2005 10:44:19 -0400 Subject: [Distutils] A plan for scripts (in EasyInstall) Message-ID: <5.1.1.6.0.20050606091347.01dd1f00@mail.telecommunity.com> Here's my semi-final plan for doing scripts with EasyInstall; please let me know if you see any issues with it: * Make bdist_egg stick the scripts under an EGG-INFO/scripts/ subdirectory of the egg, by abusing the install_scripts command. (Note: this does *not* mean they will be copied to 'PackageName.egg-info', only that they will be included in the built egg file. Currently, subdirectories of .egg-info are not included in the built egg, so this is completely separate.) * Add 'metadata_isdir' and 'metadata_listdir' APIs to pkg_resources, to allow inspecting the contents of EGG-INFO/scripts * Add these options to the easy_install command: --exclude-scripts, -x Don't install scripts --scripts-to=DIR, -t DIR Install scripts to DIR The --scripts-to option would default to being the *same as the installation directory*, unless you're installing to site-packages, in which case they would go to the Python default location for installing scripts. Why the installation directory? Because if you run the scripts from there, the egg(s) will be in a directory on sys.path, meaning that 'require()' will then work. In essence, I'm assuming that the normal use case for specifying an install-dir is to create an "application directory" filled with the eggs needed to run a specific application. For example, on a Unix-like system you might be installing to your ~/bin. The downside to this assumption is that since the scripts are in the same place, they might become importable when that's not intended. So if you're installing to a personal ~/lib/python, you will probably want to use -x or -t to override. Anyway, if you are installing scripts, easy_install will just do what the distutils does now to install them in their specified locations. This basically means giving them executable permissions and munging the #! line if applicable. On a multi-version install (-m or -i), it seems like we should also add a line like: from pkg_resources import require; require("thispackage==version"); del require But, as I've pointed out before, it's a tricky modification, as it would need to be inserted just before the first *executable* line of code in the script, which is often not the first line. Docstrings and __future__ statements both have to be skipped, or else the script could be broken by the modification. Further, even a successful modification is going to change the script's line numbering, which could have tech support implications. So, I'm somewhat reluctant to do this without a way to turn it off (other than by skipping scripts). There also needs to be a way to verify that the script is in fact a Python script! (Presumably by checking for a .py/.pyw extension or a #! line containing "python".) Also, adding such a 'require()' line might be more restrictive than necessary; the script might include its own require() already! So, here's an alternative possibility. Let's suppose that the script *is* a Python script. What if we ran it from inside the egg? We could write out the script as a stub loader, looking something like this: #!python <-- copied from original script, if present import pkg_resources pkg_resources.run_main("scriptname", "EggName") The 'run_main' function would do several things: * require() the appropriate package(s) * Clear everything but __name__ from the __main__ namespace * Load the script file into memory, and "poke" it into linecache.cache, so that tracebacks from the script execution will still show its source code, with correct line numbers * exec the script in __main__, using something like: maindict['__file__'] = pseudo_filename code = compile(script_source, pseudo_filename, "exec") exec code in maindict, maindict Hm. execfile() could also be used if the script file actually exists, in which case we could also skip the seeding of linecache. Probably we can add a run_script() method to the IMetadataProvider interface, so that different egg formats can handle this appropriately. Now that we've come this far, it becomes clear that these "scripts" are nothing more than bootstraps -- which means that in a future version I can imagine allowing more user-friendly installation options, like .exe files on Windows, "applications" on the Mac, and extension-stripping everywhere else. However, it may be that the choice of script installation policy is largely a matter of vehement personal preference, so there should probably be a way to configure that. It could also include a way to define custom installation policies in Python modules, and a way to select a particular policy at runtime, e.g.: easy_install --script-policy=mymodule.foo_policy ... At this point, however, easy_install options will have gotten complex enough to warrant configuration files for standard settings. We could probably hijack the existing distutils configuration scheme for that, though, treating 'easy_install' as if it were a distutils command. Whew. I think that about covers it. Thoughts, anyone? From ianb at colorstudy.com Mon Jun 6 18:00:24 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 06 Jun 2005 11:00:24 -0500 Subject: [Distutils] A plan for scripts (in EasyInstall) In-Reply-To: <5.1.1.6.0.20050606091347.01dd1f00@mail.telecommunity.com> References: <5.1.1.6.0.20050606091347.01dd1f00@mail.telecommunity.com> Message-ID: <42A47318.50407@colorstudy.com> Phillip J. Eby wrote: > Here's my semi-final plan for doing scripts with EasyInstall; please let me > know if you see any issues with it: > > * Make bdist_egg stick the scripts under an EGG-INFO/scripts/ subdirectory > of the egg, by abusing the install_scripts command. (Note: this does *not* > mean they will be copied to 'PackageName.egg-info', only that they will be > included in the built egg file. Currently, subdirectories of .egg-info are > not included in the built egg, so this is completely separate.) > > * Add 'metadata_isdir' and 'metadata_listdir' APIs to pkg_resources, to > allow inspecting the contents of EGG-INFO/scripts > > * Add these options to the easy_install command: > > --exclude-scripts, -x Don't install scripts > --scripts-to=DIR, -t DIR Install scripts to DIR Should these options match distutils? Well, --install-scripts -- I don't think it has an --exclude-scripts. I don't really see a need for the short form, but it doesn't really matter. > The --scripts-to option would default to being the *same as the > installation directory*, unless you're installing to site-packages, in > which case they would go to the Python default location for installing scripts. > > Why the installation directory? Because if you run the scripts from there, > the egg(s) will be in a directory on sys.path, meaning that 'require()' > will then work. In essence, I'm assuming that the normal use case for > specifying an install-dir is to create an "application directory" filled > with the eggs needed to run a specific application. For example, on a > Unix-like system you might be installing to your ~/bin. > > The downside to this assumption is that since the scripts are in the same > place, they might become importable when that's not intended. So if you're > installing to a personal ~/lib/python, you will probably want to use -x or > -t to override. How would they be importable? Assuming you don't put __init__.py in scripts/, it shouldn't be a problem, right? As for putting them in the installation direction, it's not a great place but at least they won't be lost. > Anyway, if you are installing scripts, easy_install will just do what the > distutils does now to install them in their specified locations. This > basically means giving them executable permissions and munging the #! line > if applicable. On a multi-version install (-m or -i), it seems like we > should also add a line like: > > from pkg_resources import require; require("thispackage==version"); del > require > > But, as I've pointed out before, it's a tricky modification, as it would > need to be inserted just before the first *executable* line of code in the > script, which is often not the first line. Docstrings and __future__ > statements both have to be skipped, or else the script could be broken by > the modification. Further, even a successful modification is going to > change the script's line numbering, which could have tech support > implications. So, I'm somewhat reluctant to do this without a way to turn > it off (other than by skipping scripts). There also needs to be a way to > verify that the script is in fact a Python script! (Presumably by checking > for a .py/.pyw extension or a #! line containing "python".) > > Also, adding such a 'require()' line might be more restrictive than > necessary; the script might include its own require() already! Well, they certainly don't have require() lines yet, so you can set precedent now. Personally I don't worry much about changing line numbers slightly -- in general it is easy and beneficial to make scripts very small, and I'd be worried about the generated code (which might be very useful to know for debugging) being obscured by tricky processes. All the fiddling with a fake __main__ module and whatnot seems unnecessarily complicated, all to enable a crufty process. In particular, I find __main__ annoying because of the double-import problem (the same module existing as __main__ and as its true name) -- it works okay for small utilities, but if I want to give the script any polish at all I'd want to get rid of __main__. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Mon Jun 6 18:59:47 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 06 Jun 2005 12:59:47 -0400 Subject: [Distutils] A plan for scripts (in EasyInstall) In-Reply-To: <42A47318.50407@colorstudy.com> References: <5.1.1.6.0.20050606091347.01dd1f00@mail.telecommunity.com> <5.1.1.6.0.20050606091347.01dd1f00@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050606123945.02454ae8@mail.telecommunity.com> At 11:00 AM 6/6/2005 -0500, Ian Bicking wrote: >Phillip J. Eby wrote: >>Here's my semi-final plan for doing scripts with EasyInstall; please let >>me know if you see any issues with it: >>* Make bdist_egg stick the scripts under an EGG-INFO/scripts/ >>subdirectory of the egg, by abusing the install_scripts command. (Note: >>this does *not* mean they will be copied to 'PackageName.egg-info', only >>that they will be included in the built egg file. Currently, >>subdirectories of .egg-info are not included in the built egg, so this is >>completely separate.) >>* Add 'metadata_isdir' and 'metadata_listdir' APIs to pkg_resources, to >>allow inspecting the contents of EGG-INFO/scripts >>* Add these options to the easy_install command: >> --exclude-scripts, -x Don't install scripts >> --scripts-to=DIR, -t DIR Install scripts to DIR > >Should these options match distutils? Well, --install-scripts -- I don't >think it has an --exclude-scripts. I don't really see a need for the >short form, but it doesn't really matter. I could allow it as a synonym, I suppose, but since optparse lets you abbreviate longopts, I try to put as much distinction into the prefixes as possible. With scripts-to, --scr is sufficient to uniquely identify it. >>The downside to this assumption is that since the scripts are in the same >>place, they might become importable when that's not intended. So if >>you're installing to a personal ~/lib/python, you will probably want to >>use -x or -t to override. > >How would they be importable? Assuming you don't put __init__.py in >scripts/, it shouldn't be a problem, right? Um, they'd be importable if the --install-dir is on sys.path, and they have .py extensions. >All the fiddling with a fake __main__ module and whatnot seems >unnecessarily complicated, It's not fake -- no more so than 'python -m' is. The only thing faked anywhere in my final proposal is the script filename and linecache, and even those are only faked for zipped files, so that you can get a traceback that includes source. >all to enable a crufty process. Well, it is called *Easy*Install, after all, not "Might install future packages that are up to our newly-invented standards." :) > In particular, I find __main__ annoying because of the double-import > problem (the same module existing as __main__ and as its true name) -- it > works okay for small utilities, but if I want to give the script any > polish at all I'd want to get rid of __main__. Well-written packages do this today; their scripts will usually either just be a stub that imports main() from somewhere, or their "if __name__=='__main__'" block imports main from itself. Such packages are not penalized by EasyInstall, and not-so-well-written packages won't be broken any more than they already are. I think we should *also* have some kind of "new-style scripts" option, where you list metadata in your .egg-info directory or setup.py to describe what modules can be run as "main" modules, and whether they're console or GUI applications. But, for EasyInstall the first priority is to support existing distutils scripts, for backward compatibility. We can add the fancy new stuff afterwards, maybe in setuptools 0.5, which at the current rate of development should only be a couple of weeks from now. ;) (The things on my list for the 0.4.x releases are basic script support and automatic installation of a newly-installed package's dependencies. In other words, a basically feature-complete package for end users. Extensibility and developer-oriented features (like logging, and the ability to replace the standard URL opener with an app-specific one) will be in the 0.5.x series.) From pje at telecommunity.com Wed Jun 8 17:56:35 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 08 Jun 2005 11:56:35 -0400 Subject: [Distutils] EasyInstall --> distutils command + setuptools enhancements? Message-ID: <5.1.1.6.0.20050608105243.01fd52d0@mail.telecommunity.com> As I move towards implementing features for EasyInsall like having distutils-style configuration files, and refactoring the 'main()' function for reusability and extension by other packages, I began to notice something: I was basically reinventing a distutils Command object. Sure, if I subclass Command, I'm stuck with a crummy 'log', and an ancient ancestor of 'optparse' (i.e. 'fancy_getopt'), but I get config file parsing for free, I get a known structure for arranging things, and I can trivially get access to the work done by other distutils commands, like finding out where the 'install' command would've installed libraries and scripts. Not only that, but if I wanted to implement the commands other people have been asking for, like to list installed packages, uninstall packages, search PyPI, etc., I could just add them to the same harness. What's more, people could add third-party commands using the --command-packages option under 2.4 (and I could perhaps backport support into setuptools so that it would work with 2.3 as well, at least for EasyInstall). My main concern with this approach is lengthening the command line, because now you'd have to specify a command, resulting in things like this: python -m easy_install get SQLObject The current name seems to clash with the idea of having different commands, too. It almost seems like it should be more like: python -m package get SQLObject python -m package list python -m package delete mechanize python -m package check FooBar[someopt,otheropt] python -m package download somepkg python -m package extract otherpkg In addition to providing a bit more flexibility, this would let us break up the somewhat monolithic structure of the current installer code, so that you could use individual pieces as shown, like extracting a package to a directory without building or installing it. Not only would that make these commands available to the user, but it would also make them available to packages that use setuptools in their setup.py, such that a package could e.g. download and extract the source of some C library that it depends on, if it's not installed on the user's system. The 'download' command could delegate to 'download_url', 'download_file', and 'download_req' subcommands, corresponding to the current features. 'extract' would delegate to 'download', and so on. There'd probably be a 'setup' command, too, to run a setup script under sandboxing, etc. So what are the downsides to this, apart from needing to turn lots of little methods into entire Command subclasses? Well, I imagine it makes it harder for somebody to build-in the commands to non-distutils programs, unless there's an easy way to create and run the commands, like if I added a class method like 'invoke(**args)' to a base class, so that you could say stuff like 'package.download_url.invoke(url="whatever")' in order to call 'em from inside the program. I've already added something similar to the 'bdist_egg' comamnd called 'call_command(name,**args)', so that you can more easily invoke other commands as subroutines. Features like those could well be useful for the distutils as well, so perhaps 'setuptools.Command' should include them. Right now, setuptools exports the same Command class as the distutils, but I could easily change that, so that Commands can be easily created and invoked from non-distutils code. The other downside that needs mitigation is the logging crud. Distutils is spattered with write() calls to sys.stderr and sys.stdout, and it inconsistently uses its own 'log.' calls (e.g., it has messages beginning with the word "Warning" that nonetheless use 'log.info()' as the level!). It has stuff that depends on a distutils-specific verbosity level, and so on. All of this can and should be fixed in Python 2.5, but EasyInstall needs it now, so setuptools will probably do some monkeypatching here and there to get 'distutils.log' rerouted to 'logging.getLogger("distutils")', and perhaps hack things in such a way that the distutils verbose/quiet stuff is only used to change the log level, and and restores the level when commands are finished. I'm not entirely sure how feasible this is. Some of the sys.stderr writes are also not monkeypatchable without replacing giant routines, and there are close to 70 print statements spread across 16 modules. Even just going through to verify which ones are debug prints is a PITA. However, if someone were trying to integrate these commands into some type of GUI application, it might cause occasional problems unless stdout and stderr were replaced or redirected somehow. Still, the bulk of the logging issues can probably be safely handled in a couple of monkeypatches to Command, CCompiler, and text_file.TextFile; most of the rest occur under pretty obscure circumstances. So, I think an 80% solution is quite doable. I don't really want to tangle with the fancy_getopt vs. optparse distinction, though. Or any of the other places where a utility module in the distutils has been left behind by a newfangled version in the standard library. :( I mean, on one level it's always tempting to try to write "Distutils 2", but at this point there's so much accumulated knowledge in distutils that I don't think it's reasonable to expect a replacement to ever be doable in a single rewrite. Refactoring the existing stuff is the only way to go. Anyway, now I'm digressing. Thoughts, anyone? From ianb at colorstudy.com Wed Jun 8 19:05:36 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 08 Jun 2005 12:05:36 -0500 Subject: [Distutils] EasyInstall --> distutils command + setuptools enhancements? In-Reply-To: <5.1.1.6.0.20050608105243.01fd52d0@mail.telecommunity.com> References: <5.1.1.6.0.20050608105243.01fd52d0@mail.telecommunity.com> Message-ID: <42A72560.7000702@colorstudy.com> Phillip J. Eby wrote: > As I move towards implementing features for EasyInsall like having > distutils-style configuration files, and refactoring the 'main()' function > for reusability and extension by other packages, I began to notice > something: I was basically reinventing a distutils Command object. Are you thinking solely about a two-level command? It's a little confusing, because it seems like EasyInstall and distutils are kind of separate, and I'm not clear what relation you are proposing. If it's just that there be two levels to the command, just like in distutils or cvs, among other commands, then yes, I think that's a good idea, and I can imagine quite a few commands over time. As for the specific implementation, I don't know that distutil's Command is worth using if it's outdated. > So what are the downsides to this, apart from needing to turn lots of > little methods into entire Command subclasses? Well, I imagine it makes it > harder for somebody to build-in the commands to non-distutils programs, > unless there's an easy way to create and run the commands, like if I added > a class method like 'invoke(**args)' to a base class, so that you could say > stuff like 'package.download_url.invoke(url="whatever")' in order to call > 'em from inside the program. I've already added something similar to the > 'bdist_egg' comamnd called 'call_command(name,**args)', so that you can > more easily invoke other commands as subroutines. Would it make more sense to just keep the command-line interface down to a thin wrapper around module-level functions? -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Wed Jun 8 19:21:30 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 08 Jun 2005 13:21:30 -0400 Subject: [Distutils] EasyInstall --> distutils command + setuptools enhancements? In-Reply-To: <42A72560.7000702@colorstudy.com> References: <5.1.1.6.0.20050608105243.01fd52d0@mail.telecommunity.com> <5.1.1.6.0.20050608105243.01fd52d0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050608131932.01fd5348@mail.telecommunity.com> At 12:05 PM 6/8/2005 -0500, Ian Bicking wrote: >Phillip J. Eby wrote: >>As I move towards implementing features for EasyInsall like having >>distutils-style configuration files, and refactoring the 'main()' >>function for reusability and extension by other packages, I began to >>notice something: I was basically reinventing a distutils Command object. > >Are you thinking solely about a two-level command? It's a little >confusing, because it seems like EasyInstall and distutils are kind of >separate, and I'm not clear what relation you are proposing. > >If it's just that there be two levels to the command, just like in >distutils or cvs, among other commands, then yes, I think that's a good >idea, and I can imagine quite a few commands over time. > >As for the specific implementation, I don't know that distutil's Command >is worth using if it's outdated. Yes, but it *integrates* with existing functionality, which is useful. It would also make it easier for setup scripts to use EasyInstall functionality as part of their build process. >>So what are the downsides to this, apart from needing to turn lots of >>little methods into entire Command subclasses? Well, I imagine it makes >>it harder for somebody to build-in the commands to non-distutils >>programs, unless there's an easy way to create and run the commands, like >>if I added a class method like 'invoke(**args)' to a base class, so that >>you could say stuff like 'package.download_url.invoke(url="whatever")' in >>order to call 'em from inside the program. I've already added something >>similar to the 'bdist_egg' comamnd called 'call_command(name,**args)', so >>that you can more easily invoke other commands as subroutines. > >Would it make more sense to just keep the command-line interface down to a >thin wrapper around module-level functions? Well, the argument metadata still has to exist somewhere. And most of the functions would end up needing to have access to various data items. The structure definitely mimics the structure of the distutils commands, and it seems silly to go off and create another framework, especially since I already have to import the distutils anyway. From ianb at colorstudy.com Wed Jun 8 19:23:19 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 08 Jun 2005 12:23:19 -0500 Subject: [Distutils] EasyInstall --> distutils command + setuptools enhancements? In-Reply-To: <5.1.1.6.0.20050608131932.01fd5348@mail.telecommunity.com> References: <5.1.1.6.0.20050608105243.01fd52d0@mail.telecommunity.com> <5.1.1.6.0.20050608105243.01fd52d0@mail.telecommunity.com> <5.1.1.6.0.20050608131932.01fd5348@mail.telecommunity.com> Message-ID: <42A72987.9040502@colorstudy.com> Phillip J. Eby wrote: >> Are you thinking solely about a two-level command? It's a little >> confusing, because it seems like EasyInstall and distutils are kind of >> separate, and I'm not clear what relation you are proposing. >> >> If it's just that there be two levels to the command, just like in >> distutils or cvs, among other commands, then yes, I think that's a >> good idea, and I can imagine quite a few commands over time. >> >> As for the specific implementation, I don't know that distutil's >> Command is worth using if it's outdated. > > > Yes, but it *integrates* with existing functionality, which is useful. > It would also make it easier for setup scripts to use EasyInstall > functionality as part of their build process. I'm still not clear. Do you mean you intend people to do: python setup.py install_depends ? I'm just confused. If it's a matter of using EasyInstall functionality, it seems like the Python-level interface is more useful. Or do you mean, so that "python setup.py install" can pass its arguments on to EasyInstall commands (e.g., the --install-lib option will be passed onto installed dependencies?) -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Wed Jun 8 19:39:39 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 08 Jun 2005 13:39:39 -0400 Subject: [Distutils] EasyInstall --> distutils command + setuptools enhancements? In-Reply-To: <42A72987.9040502@colorstudy.com> References: <5.1.1.6.0.20050608131932.01fd5348@mail.telecommunity.com> <5.1.1.6.0.20050608105243.01fd52d0@mail.telecommunity.com> <5.1.1.6.0.20050608105243.01fd52d0@mail.telecommunity.com> <5.1.1.6.0.20050608131932.01fd5348@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050608133342.02e15c80@mail.telecommunity.com> At 12:23 PM 6/8/2005 -0500, Ian Bicking wrote: >Phillip J. Eby wrote: >>>Are you thinking solely about a two-level command? It's a little >>>confusing, because it seems like EasyInstall and distutils are kind of >>>separate, and I'm not clear what relation you are proposing. >>> >>>If it's just that there be two levels to the command, just like in >>>distutils or cvs, among other commands, then yes, I think that's a good >>>idea, and I can imagine quite a few commands over time. >>> >>>As for the specific implementation, I don't know that distutil's Command >>>is worth using if it's outdated. >> >>Yes, but it *integrates* with existing functionality, which is useful. >>It would also make it easier for setup scripts to use EasyInstall >>functionality as part of their build process. > >I'm still not clear. Do you mean you intend people to do: > > python setup.py install_depends That wasn't what I had in mind, although now that you mention it, it sounds kind of handy. Not so much that people would type it directly, as that folks who don't want their users to have to install EasyInstall, could use it as part of their build process. Hm. That's actually a fantastic idea. Thanks! :) >? I'm just confused. If it's a matter of using EasyInstall >functionality, it seems like the Python-level interface is more >useful. Or do you mean, so that "python setup.py install" can pass its >arguments on to EasyInstall commands (e.g., the --install-lib option will >be passed onto installed dependencies?) Something like that. I actually more meant that things like the current logging verbosity would be shared, configuration files shared, etc. However, this is actually a fantastic thing for the new idea (allowing packages using setuptools to install dependencies), because it means that certain kinds of arguments passed to the parent script will be usable by the child scripts (e.g. compiler options, install locations, etc.). Sweet! Interestingly, this is what I originally set out to make setuptools do, i.e. know about dependencies and be able to download/install them, but the work kind of stalled as I ran into complex issues of selecting the right platform and distribution to install. EasyInstall addressed these issues and solved them starting from a point of view based on a totally different use case, but seems to have found its way full circle now. :) From ianb at colorstudy.com Wed Jun 8 20:20:34 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 08 Jun 2005 13:20:34 -0500 Subject: [Distutils] EasyInstall --> distutils command + setuptools enhancements? In-Reply-To: <5.1.1.6.0.20050608133342.02e15c80@mail.telecommunity.com> References: <5.1.1.6.0.20050608131932.01fd5348@mail.telecommunity.com> <5.1.1.6.0.20050608105243.01fd52d0@mail.telecommunity.com> <5.1.1.6.0.20050608105243.01fd52d0@mail.telecommunity.com> <5.1.1.6.0.20050608131932.01fd5348@mail.telecommunity.com> <5.1.1.6.0.20050608133342.02e15c80@mail.telecommunity.com> Message-ID: <42A736F2.9060304@colorstudy.com> Phillip J. Eby wrote: >> I'm still not clear. Do you mean you intend people to do: >> >> python setup.py install_depends > > > That wasn't what I had in mind, although now that you mention it, it > sounds kind of handy. Not so much that people would type it directly, > as that folks who don't want their users to have to install EasyInstall, > could use it as part of their build process. > > Hm. That's actually a fantastic idea. Thanks! :) Um, okay ;) It seems a little strange to me, but I suppose it does solve the bootstrapping problem. Well, not "solve", but for instance I can just keep (or update) the code in Paste's setup.py to fetch setuptools and add it to sys.path, though that code does feel a bit crufty, since it's never going to be a very robust installation process. Where will the normal script go then? Like, when you want to install a package manually, and you don't have its setup.py file yet? Will the easy_install script just be a kind of metadata-less version of distutils' setup()? -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Wed Jun 8 21:06:26 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 08 Jun 2005 15:06:26 -0400 Subject: [Distutils] EasyInstall --> distutils command + setuptools enhancements? In-Reply-To: <42A736F2.9060304@colorstudy.com> References: <5.1.1.6.0.20050608133342.02e15c80@mail.telecommunity.com> <5.1.1.6.0.20050608131932.01fd5348@mail.telecommunity.com> <5.1.1.6.0.20050608105243.01fd52d0@mail.telecommunity.com> <5.1.1.6.0.20050608105243.01fd52d0@mail.telecommunity.com> <5.1.1.6.0.20050608131932.01fd5348@mail.telecommunity.com> <5.1.1.6.0.20050608133342.02e15c80@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050608150420.01fd5720@mail.telecommunity.com> At 01:20 PM 6/8/2005 -0500, Ian Bicking wrote: >Phillip J. Eby wrote: >>>I'm still not clear. Do you mean you intend people to do: >>> >>> python setup.py install_depends >> >>That wasn't what I had in mind, although now that you mention it, it >>sounds kind of handy. Not so much that people would type it directly, as >>that folks who don't want their users to have to install EasyInstall, >>could use it as part of their build process. >>Hm. That's actually a fantastic idea. Thanks! :) > >Um, okay ;) It seems a little strange to me, but I suppose it does solve >the bootstrapping problem. Well, not "solve", but for instance I can just >keep (or update) the code in Paste's setup.py to fetch setuptools and add >it to sys.path, though that code does feel a bit crufty, since it's never >going to be a very robust installation process. > >Where will the normal script go then? Like, when you want to install a >package manually, and you don't have its setup.py file yet? Will the >easy_install script just be a kind of metadata-less version of distutils' >setup()? Well, it'd be replaced with a new 'package' module, that would contain a start script to run the various setuptools commands that represent the current 'easy_install' functionality. e.g.,: python -m package get SQLObject From pje at telecommunity.com Wed Jun 8 22:25:42 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 08 Jun 2005 16:25:42 -0400 Subject: [Distutils] EasyInstall --> distutils command + setuptools enhancements? In-Reply-To: <42A736F2.9060304@colorstudy.com> References: <5.1.1.6.0.20050608133342.02e15c80@mail.telecommunity.com> <5.1.1.6.0.20050608131932.01fd5348@mail.telecommunity.com> <5.1.1.6.0.20050608105243.01fd52d0@mail.telecommunity.com> <5.1.1.6.0.20050608105243.01fd52d0@mail.telecommunity.com> <5.1.1.6.0.20050608131932.01fd5348@mail.telecommunity.com> <5.1.1.6.0.20050608133342.02e15c80@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050608161458.01fdb540@mail.telecommunity.com> At 01:20 PM 6/8/2005 -0500, Ian Bicking wrote: >Phillip J. Eby wrote: >>>I'm still not clear. Do you mean you intend people to do: >>> >>> python setup.py install_depends >> >>That wasn't what I had in mind, although now that you mention it, it >>sounds kind of handy. Not so much that people would type it directly, as >>that folks who don't want their users to have to install EasyInstall, >>could use it as part of their build process. >>Hm. That's actually a fantastic idea. Thanks! :) > >Um, okay ;) It seems a little strange to me, but I suppose it does solve >the bootstrapping problem. Well, not "solve", but for instance I can just >keep (or update) the code in Paste's setup.py to fetch setuptools and add >it to sys.path, though that code does feel a bit crufty, since it's never >going to be a very robust installation process. We could solve that a little better by creating an 'ez_boot' module or some such, for people to use in their setup scripts. You'd just do "import ez_boot" in your setup script before trying to import setuptools, and it would do the job of stuff like: REQUIRED_VERSION="setuptools>=0.5a" try: import pkg_resources except ImportError: download_latest_setuptools() add_setuptools_to_sys_path() set_self_installation_flag() else: try: pkg_resources.require(REQUIRED_VERSION) except pkg_resources.VersionConflict: ask_user_to_upgrade_and_restart() Where the functions of course have to be defined with something sane. :) Oh, and the self-installation flag would basically tell setuptools it's bootstrapping, and it should install itself as if it were a dependency of the package when you install it. That is, when you run 'setup.py install' for the containing package, it'll also install the setuptools egg, thereby avoiding the need for any future packages to do the download. From ianb at colorstudy.com Wed Jun 8 22:24:55 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 08 Jun 2005 15:24:55 -0500 Subject: [Distutils] EasyInstall --> distutils command + setuptools enhancements? In-Reply-To: <5.1.1.6.0.20050608161458.01fdb540@mail.telecommunity.com> References: <5.1.1.6.0.20050608133342.02e15c80@mail.telecommunity.com> <5.1.1.6.0.20050608131932.01fd5348@mail.telecommunity.com> <5.1.1.6.0.20050608105243.01fd52d0@mail.telecommunity.com> <5.1.1.6.0.20050608105243.01fd52d0@mail.telecommunity.com> <5.1.1.6.0.20050608131932.01fd5348@mail.telecommunity.com> <5.1.1.6.0.20050608133342.02e15c80@mail.telecommunity.com> <5.1.1.6.0.20050608161458.01fdb540@mail.telecommunity.com> Message-ID: <42A75417.6050801@colorstudy.com> Phillip J. Eby wrote: > We could solve that a little better by creating an 'ez_boot' module or > some such, for people to use in their setup scripts. You'd just do > "import ez_boot" in your setup script before trying to import > setuptools, and it would do the job of stuff like: So ez_boot.py would get distributed alongside setup.py? That seems like a good idea. It would also be a sneak way to get people to install pkg_resources et al ;) -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ronaldoussoren at mac.com Thu Jun 9 21:36:55 2005 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 9 Jun 2005 12:36:55 -0700 Subject: [Distutils] EasyInstall and build-time dependencies Message-ID: Are there any plans to teach EasyInstall about build time dependencies? And example of this would be the dependency of PIL on libjpeg and libtiff. Ronald From pje at telecommunity.com Thu Jun 9 22:14:42 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 09 Jun 2005 16:14:42 -0400 Subject: [Distutils] EasyInstall and build-time dependencies In-Reply-To: Message-ID: <5.1.1.6.0.20050609155831.01fa8808@mail.telecommunity.com> At 12:36 PM 6/9/2005 -0700, Ronald Oussoren wrote: >Are there any plans to teach EasyInstall about build time >dependencies? And example of this would be the dependency of PIL on >libjpeg and libtiff. Yes and no. I plan to refactor EasyInstall so that packages that want to, can use its find-and-download capabilities to install their own dependencies of that sort. However, if PyPI ever includes metadata that EasyInstall could consult about a package's non-Python dependencies, it could be used to inform the user. There's also the possibility that we could enhance Eggs to be able to deal with headers and libraries, and then create distutils wrappers for building libraries or finding them on the system. Then, PIL could depend on "py-libjpeg" and "py-libtiff" wrappers that could be used to find the headers and libraries for compiling the PIL extensions. But, I mainly expect build-time dependencies to have to be handled by the to-be-built package, not by EasyInstall directly. The package with the dependencies would use setuptools' new EasyInstall-based commands to locate and install the dependencies before building, placing the dependencies' eggs in the building package's dist directory, where the parent installer will notice and install them along with the building package. So, in terms of your example, PIL would have to use setuptools in its setup.py, and request build-time availability of py-libtiff and py-libjpeg. Setuptools would download them each to emporary build/ subdirectorie, and finish by putting py_libtiff-blahblah.egg and py_libtiff-blahblah.egg in dist/. PIL would then build, having found out what subdirectories to use as include and library dirs for its build process. You'd then have a PIL .egg in dist/ as well. The EasyInstall that downloaded PIL in the first place would then spot the eggs and install them in *its* dist/, or site-packages or --install-dir or whatever the final resting place should be. So how do these hypothetical py_libtiff/py_libjpeg packages work? Well, presumably they're going to be rather platform and even packaging-system specific. For Linux and friends, it'll probably look in the "usual places" for these things, or maybe use rpmfind or other packaging system indexes to see if it can install what you need that way. Or maybe it'll build from source. Heck if I know. :) That's really designing a whole 'nother level of installation system, that I'm not trying to tackle just yet. Personally, I'd just as soon see package authors and secondary distributors just build platform-appropriate .egg files and let people download those, rather than trying to solve this. Relatively few Python packages contain a significant number of extensions, and it seems to me that most of them that do, don't have any external library dependencies. The exceptions are things like PIL, PyGame, DBAPI modules, and other "interface with some other system" modules. However, typically these are offered as win32.exe's, or .rpms, or whatever else. So packaging as .eggs certainly makes sense there too. From anthony at interlink.com.au Fri Jun 10 09:23:29 2005 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri, 10 Jun 2005 17:23:29 +1000 Subject: [Distutils] Moving distutils minimum required version to 2.3 Message-ID: <200506101723.30704.anthony@interlink.com.au> I've checked distutils into the python sandbox for a major chunk of refactoring, including killing distutils.log and distutils.fancy_getopt. I plan to upgrade the minimum required version of distutils to 2.3 - I will be using the logging and optparse modules from the stdlib. This means that the version of distutils included in 2.4 will be the last version guaranteed to work on 2.1 and 2.2 -- but I seriously question whether anyone is ever going to be making standalone distutils distributions again. If you know of any problems with this, please let me know. Thanks, Anthony -- Anthony Baxter It's never too late to have a happy childhood. From pje at telecommunity.com Fri Jun 10 15:27:04 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 10 Jun 2005 09:27:04 -0400 Subject: [Distutils] Moving distutils minimum required version to 2.3 In-Reply-To: <200506101723.30704.anthony@interlink.com.au> Message-ID: <5.1.1.6.0.20050610092635.01fa4a58@mail.telecommunity.com> At 05:23 PM 6/10/2005 +1000, Anthony Baxter wrote: >I will be using >the logging and optparse modules from the stdlib. Don't forget textwrap. ;) From ronaldoussoren at mac.com Fri Jun 10 17:12:13 2005 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Fri, 10 Jun 2005 08:12:13 -0700 Subject: [Distutils] EasyInstall and build-time dependencies In-Reply-To: <5.1.1.6.0.20050609155831.01fa8808@mail.telecommunity.com> References: <5.1.1.6.0.20050609155831.01fa8808@mail.telecommunity.com> Message-ID: <447098C8-4AF0-4264-9C9A-C27B110EFE39@mac.com> On 9-jun-2005, at 13:14, Phillip J. Eby wrote: > At 12:36 PM 6/9/2005 -0700, Ronald Oussoren wrote: > >> Are there any plans to teach EasyInstall about build time >> dependencies? And example of this would be the dependency of PIL on >> libjpeg and libtiff. >> > > Yes and no. I plan to refactor EasyInstall so that packages that > want to, can use its find-and-download capabilities to install > their own dependencies of that sort. However, if PyPI ever includes > metadata that EasyInstall could consult about a package's non- > Python dependencies, it could be used to inform the user. That's good enough for me. For the major reason for adding explicit dependencies to normal C libraries would be to avoid strange failures when installing a package through EasyInstall. It would be nice if EasyInstall could give a nice message about missing C libraries, that obviously requires the cooperation of the packages. > > There's also the possibility that we could enhance Eggs to be able > to deal with headers and libraries, and then create distutils > wrappers for building libraries or finding them on the system. > Then, PIL could depend on "py-libjpeg" and "py-libtiff" wrappers > that could be used to find the headers and libraries for compiling > the PIL extensions. My major gripe about PIL's build procedure is that it *optionally* depends on some graphics libraries. This means that I end up with a (for me at least) mostly useles version of PIL if I forget to install libjpeg first. It's amazing how little code and infrastructure is needed to build CPAN-like functionality for Python. Keep up the good work. Ronald From pje at telecommunity.com Sun Jun 12 18:08:48 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 12 Jun 2005 12:08:48 -0400 Subject: [Distutils] EasyInstall --> distutils command + setuptools enhancements? In-Reply-To: <42A75417.6050801@colorstudy.com> References: <5.1.1.6.0.20050608161458.01fdb540@mail.telecommunity.com> <5.1.1.6.0.20050608133342.02e15c80@mail.telecommunity.com> <5.1.1.6.0.20050608131932.01fd5348@mail.telecommunity.com> <5.1.1.6.0.20050608105243.01fd52d0@mail.telecommunity.com> <5.1.1.6.0.20050608105243.01fd52d0@mail.telecommunity.com> <5.1.1.6.0.20050608131932.01fd5348@mail.telecommunity.com> <5.1.1.6.0.20050608133342.02e15c80@mail.telecommunity.com> <5.1.1.6.0.20050608161458.01fdb540@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050610214422.01f6dcc8@mail.telecommunity.com> At 03:24 PM 6/8/2005 -0500, Ian Bicking wrote: >Phillip J. Eby wrote: >>We could solve that a little better by creating an 'ez_boot' module or >>some such, for people to use in their setup scripts. You'd just do >>"import ez_boot" in your setup script before trying to import setuptools, >>and it would do the job of stuff like: > >So ez_boot.py would get distributed alongside setup.py? That seems like a >good idea. It would also be a sneak way to get people to install >pkg_resources et al ;) Not only that, but it would help with packaging "legacy" (non-distutils) packages, because you could just stick a setup.py and a copy of ez_boot.py together to make a source distribution that downloads the "legacy" package before it builds itself. By the way, I ended up refactoring easy_install to be a single distutils command, and moving all the download-related parts into the PackageIndex class, which is now in the 'setuptools.package_index' module. Archive extraction support went to 'setuptools.archive_util', and sandboxed running of setup files went to 'setuptools.sandbox.run_setup'. So all these bits are now reusable outside EasyInstall. I still need to get logging hooked up, although one concern I have is that the distutils uses a single global log object, so run_setup is going to need to save and restore its state, but I'm going to have to figure out how to do that in a "forward compatible" way with Anthony's planned upgrade to distutils.log. (Because there is no public API to *get* the log level from distutils.log, so saving it isn't possible without depending on an implementation detail.) I haven't finalized the name for the bootstrap module yet, but I think it's going to be 'ez_setup'. I also need to finalize a policy for where it will download setuptools from, unless of course it works something like: from ez_setup import use_setuptools use_setuptools("0.4a3", "http://peak.telecommunity.com/dist/") Where you give it a specific version and a base URL, in case you want to ensure accessibility for your users. But even there, it'll probably be nice to have defaults. Anyway, that would probably be implemented with something like: def use_setuptools(version="latest", download_base="someplace on pypi") try: from pkg_resources import require except ImportError: # download, add to sys.path, etc. else: require("setuptools>="+version") From pje at telecommunity.com Mon Jun 13 00:15:57 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 12 Jun 2005 18:15:57 -0400 Subject: [Distutils] FYI: EasyInstall 0.4a2 with scripts and 'ez_setup' bootstrapper Message-ID: <5.1.1.6.0.20050612153100.01f6d848@mail.telecommunity.com> Just a heads-up for any early adopters... I've released EasyInstall 0.4a2, with support for installing scripts, and a new 'ez_setup.py' script/module that you can use to install setuptools. Plus, you can include it in a package that needs setuptools, and it will automatically download setuptools for you, for use during your installation procedure. (It won't actually *install* setuptools on the user's system in that mode, however; I'll tackle that in a later release. Also, see the script's docstrings for instructions on integrating it with your setup script.) If you've been embedding easy_install facilities in another program or setup script, be aware that the embedding interface changed dramatically in this release due to the distutils integration, but it should be comparatively stable from here on out. Also, the installation procedure for easy_install has changed; be sure to consult the updated documentation at: http://peak.telecommunity.com/DevCenter/EasyInstall Please help me test this new version; it is a *major* upgrade and refactoring compared to the last release. I tried a variety of scenarios, but I don't yet have any automated regression tests for many of its features. From rtomayko at gmail.com Mon Jun 13 03:26:11 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Sun, 12 Jun 2005 21:26:11 -0400 Subject: [Distutils] FYI: EasyInstall 0.4a2 with scripts and 'ez_setup' bootstrapper In-Reply-To: <5.1.1.6.0.20050612153100.01f6d848@mail.telecommunity.com> References: <5.1.1.6.0.20050612153100.01f6d848@mail.telecommunity.com> Message-ID: <28E9B657-E05B-4B24-AFD3-88FB9410AD86@gmail.com> On Jun 12, 2005, at 6:15 PM, Phillip J. Eby wrote: > Please help me test this new version; it is a *major* upgrade and > refactoring compared to the last release. I'm getting the following on two separate packages that contains scripts when building eggs w/ 0.4a2. Python-2.4/MacOS-10.4 $ python setup.py --command-packages=setuptools.command bdist_egg copying and adjusting bin/pudge -> build/scripts-2.4 changing mode of build/scripts-2.4/pudge from 664 to 775 creating build/bdist.darwin-8.1.0-Power_Macintosh/egg/EGG-INFO creating build/bdist.darwin-8.1.0-Power_Macintosh/egg/EGG-INFO/scripts copying build/scripts-2.4/pudge -> build/bdist.darwin-8.1.0- Power_Macintosh/egg/EGG-INFO/scripts changing mode of build/bdist.darwin-8.1.0-Power_Macintosh/egg/EGG- INFO/scripts/pudge to 775 Traceback (most recent call last): File "setup.py", line 70, in ? setup(**config) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/distutils/core.py", line 149, in setup dist.run_commands() File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/distutils/dist.py", line 946, in run_commands self.run_command(cmd) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "build\bdist.win32\egg\setuptools\command\bdist_egg.py", line 160, in run AttributeError: 'Distribution' object has no attribute 'egg_name' This release is a significant milestone and I'm willing to help in any way needed. Thanks, Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From rtomayko at gmail.com Mon Jun 13 04:04:35 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Sun, 12 Jun 2005 22:04:35 -0400 Subject: [Distutils] FYI: EasyInstall 0.4a2 with scripts and 'ez_setup' bootstrapper In-Reply-To: <28E9B657-E05B-4B24-AFD3-88FB9410AD86@gmail.com> References: <5.1.1.6.0.20050612153100.01f6d848@mail.telecommunity.com> <28E9B657-E05B-4B24-AFD3-88FB9410AD86@gmail.com> Message-ID: <9E314C67-0D1E-402F-B300-14490B165543@gmail.com> On Jun 12, 2005, at 9:26 PM, Ryan Tomayko wrote: > On Jun 12, 2005, at 6:15 PM, Phillip J. Eby wrote: > >> Please help me test this new version; it is a *major* upgrade and >> refactoring compared to the last release. >> > > I'm getting the following on two separate packages that contains > scripts when building eggs w/ 0.4a2. > > Python-2.4/MacOS-10.4 My bad. I had old pkg_resources.pyc and easy_install.pyc files lying around. Using FIGNORE in bash isn't worth it sometimes... Ryan From rtomayko at gmail.com Mon Jun 13 06:16:39 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Mon, 13 Jun 2005 00:16:39 -0400 Subject: [Distutils] setuptools-0.4a2: Eggs, scripts, and __file__ Message-ID: From "[Distutils] A plan for scripts (in EasyInstall)" On Mon Jun 6 16:44:19 CEST 2005, Phillip J. Eby wrote: > The 'run_main' function would do several things: > > * Clear everything but __name__ from the __main__ namespace > * exec the script in __main__, using something like: > > maindict['__file__'] = pseudo_filename code = > compile(script_source, pseudo_filename, "exec") exec code in > maindict, maindict It seems that bullet one is happening but not bullet two? I have scripts that are attempting to use __file__ but failing with: NameError: name '__file__' is not defined The script looks like it would work properly if it was given a pseudo filename but this has me thinking about what the best way to detect development environments in scripts will look like in an eggified environment. My wrapper scripts generally look something like the following to detect whether the script is being run from a development location or a deployed location:: import sys from os.path import dirname, abspath, join, exists devel_dir = dirname(dirname(abspath(__file__))) if exists(join(devel_dir, 'support')): sys.path.insert(1, join(devel_dir, 'support')) sys.path.insert(1, devel_dir) from package.module import main main(sys.argv) Assuming a directory layout of: [devel-dir]/bin (scripts in here) [devel-dir]/support (packages/modules) [devel-dir]/[package-name] At first I thought I should switch from using path operations on __file__ to using `pkg_resources.resource_isdir` and `resource_filename` but that doesn't make any sense - if the script is running from within a deployed egg, I'm not using it from a development environment and the resource_*** functions don't make sense in __main__ context anyway. So my current thinking is that the existing idiom should remain and that a pseudo filename shouldn't pose any problems in the scenarios I'm dealing with: * Deployed egg: don't tamper with sys.path * Deployed site-packages: don't tamper with sys.path * Development environment: insert development paths At any rate, I'm guessing that __file__ appears in a good percentage of scripts in the wild. I've switched mine to use sys.argv[0] in place of __file__ but I'll see if I can get a patch together for getting a psuedo __file__ into the exec dict (but don't wait up - I'm not familiar with the code and figuring out how to get the psuedo filename will take me infinite orders of magnitude longer than you ;) Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From pje at telecommunity.com Mon Jun 13 07:15:38 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 13 Jun 2005 01:15:38 -0400 Subject: [Distutils] setuptools-0.4a2: Eggs, scripts, and __file__ In-Reply-To: Message-ID: <5.1.1.6.0.20050613005536.01de8158@mail.telecommunity.com> At 12:16 AM 6/13/2005 -0400, Ryan Tomayko wrote: > From "[Distutils] A plan for scripts (in EasyInstall)" > > >On Mon Jun 6 16:44:19 CEST 2005, Phillip J. Eby wrote: > > The 'run_main' function would do several things: > > > > > * Clear everything but __name__ from the __main__ namespace > > > * exec the script in __main__, using something like: > > > > maindict['__file__'] = pseudo_filename code = > > compile(script_source, pseudo_filename, "exec") exec code in > > maindict, maindict > >It seems that bullet one is happening but not bullet two? I have >scripts that are attempting to use __file__ but failing with: > > NameError: name '__file__' is not defined I'll fix this in the next release. >The script looks like it would work properly if it was given a pseudo >filename but this has me thinking about what the best way to detect >development environments in scripts will look like in an eggified >environment. That's the wrong question to ask, IMO. Think about how to make the script work exactly the same in all environments, instead. :) >My wrapper scripts generally look something like the >following to detect whether the script is being run from a >development location or a deployed location:: > > import sys > from os.path import dirname, abspath, join, exists > devel_dir = dirname(dirname(abspath(__file__))) > if exists(join(devel_dir, 'support')): > sys.path.insert(1, join(devel_dir, 'support')) > sys.path.insert(1, devel_dir) > from package.module import main > main(sys.argv) > >Assuming a directory layout of: > > [devel-dir]/bin (scripts in here) > [devel-dir]/support (packages/modules) > [devel-dir]/[package-name] Have you read this: http://peak.telecommunity.com/DevCenter/PythonEggs#developing-with-eggs The complexity you're incurring here is unnecessary; between require() and .pth files you should never need to mess with sys.path manually. >At first I thought I should switch from using path operations on >__file__ to using `pkg_resources.resource_isdir` and >`resource_filename` but that doesn't make any sense - if the script >is running from within a deployed egg, I'm not using it from a >development environment Not true; see the link above. When you do development from your distutils package root, your development code *is* in an egg. However, you still shouldn't be checking __file__ or fiddling with sys.path, and there is no need anyway. Here's one idea of what your tree could look like: devel_dir/ some_dependency-1.2.egg otherpackage-3.9-whatever.egg package_you_are_working_on/ __init__.py some_module.py somescript.py MyPackage.egg-info/ <- this makes devel_dir a "source egg" for MyPackage setup.py build/ dist/ MyPackage-1.1.egg <- built egg; not actually used for development You'll notice that everything is just thrown into the same directory; if you use this setup for development, everything will "just work". So, you create a setup.py for your package, and you run setup.py bdist_egg; this will dump an egg in dist/, and create MyPackage.egg-info, marking devel_dir as a "development egg". Install any other packages you need to the current directory using 'easy_install -xd. package_you_need' (the -x excludes their scripts). Now you're ready to party. Make your scripts use 'require()' to ask for 'MyPackage'; when you run them (whether you are in the devel_dir or not), they will find MyPackage.egg-info, find the dependencies declared, and add all the needed .eggs to sys.path automatically. This is just *one* layout that works. The dependency eggs don't have to be in devel_dir; they can be anywhere that will already be on sys.path, like site-packages or a directory named in a .pth file in site-packages. Does this explain it better? One side benefit of egg-based installation is that you can dump as many libraries in site-packages as you want and not worry about version conflicts, so it's definitely how I plan to do most development. The directory where a script is located, however, takes precedence over site-packages, which means that even if you have the package you're developing already installed in site-packages, your development egg will take precedence if the script you're running is in that directory. > and the resource_*** functions don't make >sense in __main__ context anyway. So my current thinking is that the >existing idiom should remain and that a pseudo filename shouldn't >pose any problems in the scenarios I'm dealing with: > > * Deployed egg: don't tamper with sys.path > * Deployed site-packages: don't tamper with sys.path > * Development environment: insert development paths You should be able to make that last one work the same way; i.e., without tampering with sys.path. If you can't, please explain your situation further, because I want pkg_resources to be able to prevent all future sys.path munging by anything but EasyInstall itself, and by extensible applications that have to manage plugin directories. From rtomayko at gmail.com Mon Jun 13 09:12:35 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Mon, 13 Jun 2005 03:12:35 -0400 Subject: [Distutils] setuptools-0.4a2: Eggs, scripts, and __file__ In-Reply-To: <5.1.1.6.0.20050613005536.01de8158@mail.telecommunity.com> References: <5.1.1.6.0.20050613005536.01de8158@mail.telecommunity.com> Message-ID: <119B7D68-B609-4161-8756-8489CEA06671@gmail.com> On Jun 13, 2005, at 1:15 AM, Phillip J. Eby wrote: >> The script looks like it would work properly if it was given a pseudo >> filename but this has me thinking about what the best way to detect >> development environments in scripts will look like in an eggified >> environment. >> > > That's the wrong question to ask, IMO. Think about how to make the > script work exactly the same in all environments, instead. :) I'd love to except I can't assume setuptools and eggs-based dependencies in all environments at the moment. In particular, Linux distributions like Fedora probably won't be moving to egg based packaging for some time. If I'm lucky I might see python RPM maintainers phase in package.egg-info directories on top of the normal site-packages layout over the next few months. What this adds up to--if I'm not missing something--is that I can't assume require() is going to work. I need to be able to fallback into assuming that all dependencies will be laid out for me by some other package management system (in this case RPM). I don't think the setuptools dependency will be hard to deal with but egg versions of other dependencies is probably going to be a problem for a little while. I can assume that require() will be there but I'd have to try/expect/pass on DependencyNotFound exceptions or something. What I'd prefer is to keep require() out of the code completely and use .egg-info/depends.txt instead. If I'm running out of an egg, I want setuptools to manage requiring everything before my script is even called. This should give me all of the benefits of eggs when I'm using them and fallback to the old-style manual dependency management otherwise. Does that make sense? > Have you read this: > > http://peak.telecommunity.com/DevCenter/PythonEggs#developing-with- > eggs > > The complexity you're incurring here is unnecessary; between require > () and .pth files you should never need to mess with sys.path > manually. I've absolutely read it and agree completely with the concept. I can't assume that require() will work in all scenarios, however. But your conclusion is still valid I think. If I move to egg dependencies in development and assume that either setuptools or some other package management utility will setup sys.path correctly, I should be able to get rid of manual sys.path hackery. >> At first I thought I should switch from using path operations on >> __file__ to using `pkg_resources.resource_isdir` and >> `resource_filename` but that doesn't make any sense - if the script >> is running from within a deployed egg, I'm not using it from a >> development environment > > Not true; see the link above. I should have been more clear. I was speaking to when my script is being run from EGG-INFO/scripts/somescript as opposed to [devel- package]/scripts/somescript or /usr/bin/somescript (deployed via RPM). Where the script file *is* provides the information needed to determine whether/how to setup sys.path. The resource_*** functions provide no information about where the script actually lives and so the entire exercise of moving that code to use those functions was in vain. The point is moot at any rate as I don't think I'll be needing sys.path munging anymore. > When you do development from your distutils package root, your > development code *is* in an egg. However, you still shouldn't be > checking __file__ or fiddling with sys.path, and there is no need > anyway. Here's one idea of what your tree could look like: > > > > So, you create a setup.py for your package, and you run setup.py > bdist_egg; this will dump an egg in dist/, and create MyPackage.egg- > info, marking devel_dir as a "development egg". > > Install any other packages you need to the current directory using > 'easy_install -xd. package_you_need' (the -x excludes their > scripts). Now you're ready to party. Make your scripts use > 'require()' to ask for 'MyPackage'; when you run them (whether you > are in the devel_dir or not), they will find MyPackage.egg-info, > find the dependencies declared, and add all the needed .eggs to > sys.path automatically. So this is where I need to figure something out because I'd like to either not use require() in those scripts or will need to try/except/ pass around DependencyNotFound exceptions in cases where eggs won't be available for dependencies. Or maybe... When I require('MyPackage'), does setuptools look at MyPackage.egg- info/depends.txt and require everything else for me? I'm assuming it does and don't see why it wouldn't. If that's the case, I might be able to make my scripts as simple as:: from pkg_resources import require, find_distributions if list(find_distributions('MyPackage')): require('MyPackage') import MyPackage MyPackage.main() If find_distributions() yields any results then we can assume that we're running as an egg, if not we assume that we're running old- school and that some other package manager has laid everything out nicely already. The downside to this approach is that I would have to be sure to NOT distribute MyPackage.egg-info with RPMs and other packages, which kind of rules out any phased approach to bringing egg based packaging to Fedora's stock RPMs. It might be better to just patch some flag into my script during the RPM build that would tell it whether to use require or not:: use_require = 1 if use_require: require('MyPackage') import MyPackage MyPackage.main() The RPM spec would have to patch that use_require line to be zero but that's a single call to sed. If that's all the finagling I have to do in the spec file it would be a good day. I don't know - none of these seem to be perfect solutions, but none of them would have taken me as much time to implement as writing this email either. Still, it seems worth pointing out that keeping the number of code level require() calls to a minimum and having some way of switching those few calls off and on based on environment is something packages that need to be included in a non-egg-based distribution will need to think about. > Does this explain it better? One side benefit of egg-based > installation is that you can dump as many libraries in site- > packages as you want and not worry about version conflicts, so it's > definitely how I plan to do most development. The directory where > a script is located, however, takes precedence over site-packages, > which means that even if you have the package you're developing > already installed in site-packages, your development egg will take > precedence if the script you're running is in that directory. Right. I think that makes a lot of sense and will definitely be moving to eggs in development as you've described. >> and the resource_*** functions don't make >> sense in __main__ context anyway. So my current thinking is that the >> existing idiom should remain and that a pseudo filename shouldn't >> pose any problems in the scenarios I'm dealing with: >> >> * Deployed egg: don't tamper with sys.path >> * Deployed site-packages: don't tamper with sys.path >> * Development environment: insert development paths >> > > You should be able to make that last one work the same way; i.e., > without tampering with sys.path. If you can't, please explain your > situation further, because I want pkg_resources to be able to > prevent all future sys.path munging by anything but EasyInstall > itself, and by extensible applications that have to manage plugin > directories. No, I think that covers sys.path munging. I'm still a little shaky on how I should know whether to rely on require() or not but we'll see what happens. Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From pje at telecommunity.com Mon Jun 13 18:21:55 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 13 Jun 2005 12:21:55 -0400 Subject: [Distutils] setuptools-0.4a2: Eggs, scripts, and __file__ In-Reply-To: <119B7D68-B609-4161-8756-8489CEA06671@gmail.com> References: <5.1.1.6.0.20050613005536.01de8158@mail.telecommunity.com> <5.1.1.6.0.20050613005536.01de8158@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050613113419.01deb5d8@mail.telecommunity.com> At 03:12 AM 6/13/2005 -0400, Ryan Tomayko wrote: >On Jun 13, 2005, at 1:15 AM, Phillip J. Eby wrote: >>>The script looks like it would work properly if it was given a pseudo >>>filename but this has me thinking about what the best way to detect >>>development environments in scripts will look like in an eggified >>>environment. >> >>That's the wrong question to ask, IMO. Think about how to make the >>script work exactly the same in all environments, instead. :) > >I'd love to except I can't assume setuptools and eggs-based >dependencies in all environments at the moment. In particular, Linux >distributions like Fedora probably won't be moving to egg based >packaging for some time. If I'm lucky I might see python RPM >maintainers phase in package.egg-info directories on top of the >normal site-packages layout over the next few months. What this adds >up to--if I'm not missing something--is that I can't assume require() >is going to work. I need to be able to fallback into assuming that >all dependencies will be laid out for me by some other package >management system (in this case RPM). > >I don't think the setuptools dependency will be hard to deal with but >egg versions of other dependencies is probably going to be a problem >for a little while. Are you distributing an application, or a library? If you're distributing a library, you don't need require() in library code. If it's an application, you can handle your own dependencies by force-installing the eggs in the application script directory using EasyInstall, and then just require() your main package. > I can assume that require() will be there but I'd >have to try/expect/pass on DependencyNotFound exceptions or >something. What I'd prefer is to keep require() out of the code >completely and use .egg-info/depends.txt instead. That's ideal for library code; startup scripts should just require() their target package, and that's almost more for development than anything else, since scripts installed by EasyInstall do the necessary require() work in pkg_resources.run_main(). > If I'm running out >of an egg, I want setuptools to manage requiring everything before my >script is even called. Yep, it'll do that. >This should give me all of the benefits of >eggs when I'm using them and fallback to the old-style manual >dependency management otherwise. Does that make sense? Um, yeah, except I don't think you really need to fall back, just because people have other stuff installed. The worst that's going to happen is that you're going to force reinstallation of dependencies they already have, just to get them into eggs. (Or make them create .egg-info directories to tell the system the stuff is already installed.) Hm. What if you created .egg directories and symlinked the dependencies into them during your installation process? Or if there were some way to create the .egg-info directories automatically from packaging system databases, or from inspecting module contents? Setuptools has some code that can look for the setting of constants or presence of symbols in modules, without importing them. Perhaps I could extend this somehow so that a transitional package like yours could include additional info in the setup script, that checks for these dependencies and tags them somehow? Or maybe this could be done by metadata -- you put a legacy.py file in your egg-info, and when processing your egg's dependencies, if pkg_resources can't find a package you need, it would call a function in legacy.py that would check for the dependency using setuptools' inspection facilities, and return a path and a guess at a version number. How does that sound? >If I move to egg dependencies >in development and assume that either setuptools or some other >package management utility will setup sys.path correctly, I should be >able to get rid of manual sys.path hackery. I assume most packaging systems install to site-packages, so if you're doing applications, it's basically going to boil down to eggs in the script directory plus whatever's in site-packages. >When I require('MyPackage'), does setuptools look at MyPackage.egg- >info/depends.txt and require everything else for me? I'm assuming it >does and don't see why it wouldn't. Yes, it does. > If that's the case, I might be >able to make my scripts as simple as:: > > from pkg_resources import require, find_distributions > if list(find_distributions('MyPackage')): Don't do this. find_distributions() yields distributions found in a directory or zipfile; it doesn't take a package name, it takes a sys.path entry. I'm not sure exactly what you're trying to do here. If you just want to know if your script is running from a development location (and therefore needs to call require() to set up dependencies), couldn't you just check for 'MyPackage.egg-info' in the sys.path[0] (script) directory? e.g.: import sys, os if os.path.isdir(os.path.join(sys.path[0],"MyPackage.egg-info")): from pkg_resources import require require("MyPackage") # ensures dependencies get processed If this is what you want, perhaps we can create a standard recipe in pkg_resources, like maybe 'script_package("MyPackage")', that only does the require if you're a development egg and not being run from run_main(). >The downside to this approach is that I would have to be sure to NOT >distribute MyPackage.egg-info with RPMs and other packages, which >kind of rules out any phased approach to bringing egg based packaging >to Fedora's stock RPMs. I don't know if .egg-info is a good idea for RPMs. .egg-info is primarily intended for *development*, not deployment, because you can't easily override a package installed with .egg-info in site-packages. In fact, the only way you can normally override it is to install an egg alongside the script. My current idea for how RPMs and other packagers should install eggs is just to dump them in site-packages as egg files or directories, and let people use require() or else use EasyInstall to set the active package. Hey, wait a second... if you can put install/uninstall scripts in packages, couldn't installing or uninstalling an RPM ask EasyInstall to fix up the easyinstall.pth file? This would let packagers distribute as eggs, but without breaking users' expectations that the package would be available to "just import". If somebody explicitly wants to support multiversion for a package, they can run 'EasyInstall -m PackageName' to reset it to multi-version after installing a new version. EasyInstall doesn't have everything that's needed to do this yet (no "uninstall" mode), but perhaps it's a good option to add, and then packagers could standardize on this approach. >I don't know - none of these seem to be perfect solutions, but none >of them would have taken me as much time to implement as writing this >email either. Still, it seems worth pointing out that keeping the >number of code level require() calls to a minimum and having some way >of switching those few calls off and on based on environment is >something packages that need to be included in a non-egg-based >distribution will need to think about. Yep. From rtomayko at gmail.com Tue Jun 14 00:00:56 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Mon, 13 Jun 2005 18:00:56 -0400 Subject: [Distutils] setuptools-0.4a2: Eggs, scripts, and __file__ In-Reply-To: <5.1.1.6.0.20050613113419.01deb5d8@mail.telecommunity.com> References: <5.1.1.6.0.20050613005536.01de8158@mail.telecommunity.com> <5.1.1.6.0.20050613005536.01de8158@mail.telecommunity.com> <5.1.1.6.0.20050613113419.01deb5d8@mail.telecommunity.com> Message-ID: <596A0467-191B-4598-8178-4C1CA7A9A741@gmail.com> On Jun 13, 2005, at 12:21 PM, Phillip J. Eby wrote: > Are you distributing an application, or a library? If you're > distributing a library, you don't need require() in library code. > If it's an application, you can handle your own dependencies by > force-installing the eggs in the application script directory using > EasyInstall, and then just require() your main package. Libraries, applications, and libraries with helper scripts. >> This should give me all of the benefits of >> eggs when I'm using them and fallback to the old-style manual >> dependency management otherwise. Does that make sense? > > Um, yeah, except I don't think you really need to fall back, just > because people have other stuff installed. The worst that's going > to happen is that you're going to force reinstallation of > dependencies they already have, just to get them into eggs. (Or > make them create .egg-info directories to tell the system the stuff > is already installed.) That's the problem. I'm trying to figure out a general plan of attack that Linux/BSD package maintainers can adopt for python packages that want to use eggs / setuptools. Here's some numbers on how many python packages are included in a couple different distributions: On a OS X darwinports box: $ port list | grep -e '^py-' | wc -l 237 On a Fedora 3 box (Core + Extras): $ yum list all | grep -e '^py' -e 'python' | wc -l 78 I don't have any debian or gentoo boxes handy but I imagine they'd weigh in somewhere around the darwinports number. None of these packages are currently provided as eggs or with .egg- info directories when they are installed to site-packages and they have complex dependency relationships that are managed by the distribution's utility (port, yum, apt-get, emerge, etc.) This creates a problem for these packages because it means that they can not assume dependencies will always be egg managed. If they start adding require() calls to their scripts, they will break under these environments. require() is an all or nothing proposition for distributions and that means there will need to be a planned "upgrade" period or something for all packages. As a more specific example, I contribute to two packages that are distributed with Fedora Core: python-urlgrabber and yum. yum depends on python-urlgrabber and python-elementtree. Now, if I wanted to move yum to be egg based and use require(), I would also need to ensure that all yum's dependencies are egg based. When yum (and its dependencies) are installed from RPM, they must all be in egg format (or at least provide .egg-info dirs). If not, the yum script will fail. So a single package using require() can cause a snowball effect where many other packages would need to be upgraded to egg format as well. In time, this may be a good thing because it could accelerate adoption of eggs but for the time being it makes it really hard to use require(). > Hm. What if you created .egg directories and symlinked the > dependencies into them during your installation process? Or if > there were some way to create the .egg-info directories > automatically from packaging system databases, or from inspecting > module contents? Setuptools has some code that can look for the > setting of constants or presence of symbols in modules, without > importing them. Perhaps I could extend this somehow so that a > transitional package like yours could include additional info in > the setup script, that checks for these dependencies and tags them > somehow? Yes, yes. Along those lines. > Or maybe this could be done by metadata -- you put a legacy.py file > in your egg-info, and when processing your egg's dependencies, if > pkg_resources can't find a package you need, it would call a > function in legacy.py that would check for the dependency using > setuptools' inspection facilities, and return a path and a guess at > a version number. > > How does that sound? That would solve my problem perfectly. >> If that's the case, I might be >> able to make my scripts as simple as:: >> >> from pkg_resources import require, find_distributions >> if list(find_distributions('MyPackage')): >> > > Don't do this. find_distributions() yields distributions found in > a directory or zipfile; it doesn't take a package name, it takes a > sys.path entry. Ahhh.. > I'm not sure exactly what you're trying to do here. If you just > want to know if your script is running from a development location > (and therefore needs to call require() to set up dependencies), > couldn't you just check for 'MyPackage.egg-info' in the sys.path[0] > (script) directory? > > e.g.: > > import sys, os > if os.path.isdir(os.path.join(sys.path[0],"MyPackage.egg-info")): > from pkg_resources import require > require("MyPackage") # ensures dependencies get processed > > If this is what you want, perhaps we can create a standard recipe > in pkg_resources, like maybe 'script_package("MyPackage")', that > only does the require if you're a development egg and not being run > from run_main(). > >> The downside to this approach is that I would have to be sure to NOT >> distribute MyPackage.egg-info with RPMs and other packages, which >> kind of rules out any phased approach to bringing egg based packaging >> to Fedora's stock RPMs. >> > > I don't know if .egg-info is a good idea for RPMs. .egg-info is > primarily intended for *development*, not deployment, because you > can't easily override a package installed with .egg-info in site- > packages. In fact, the only way you can normally override it is to > install an egg alongside the script. > > My current idea for how RPMs and other packagers should install > eggs is just to dump them in site-packages as egg files or > directories, and let people use require() or else use EasyInstall > to set the active package. Right. But that's going to require significant lobbying and effort to get all core python packages included in a distribution migrated over to eggs. This is the root of my dilemma. The only realistic approach seems to be supporting dual egg and non-egg deployment for a little while. The legacy.py proposal is one method and I threw out a couple earlier. But yea.. I think this is the root of the problem. > Hey, wait a second... if you can put install/uninstall scripts in > packages, couldn't installing or uninstalling an RPM ask > EasyInstall to fix up the easyinstall.pth file? This would let > packagers distribute as eggs, but without breaking users' > expectations that the package would be available to "just import". > If somebody explicitly wants to support multiversion for a package, > they can run 'EasyInstall -m PackageName' to reset it to multi- > version after installing a new version. I think that would be great but it assumes modification to a whole lot of existing packages. Ideally, I'd like to be able to start using eggs and require() in my packages without having to convince everyone else to do so just yet. Not that I won't be trying to convince people, I just don't want to rely on other packages being eggified in Fedora Core, darwinports, etc for upcoming releases. > EasyInstall doesn't have everything that's needed to do this yet > (no "uninstall" mode), but perhaps it's a good option to add, and > then packagers could standardize on this approach. I'd be happy to advocate to / work with packagers once we get a basic set of best practices together. It seems like there are a lot of options here - we just need to iron out the details. Thanks, Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From rtomayko at gmail.com Tue Jun 14 01:33:34 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Mon, 13 Jun 2005 19:33:34 -0400 Subject: [Distutils] PATCH: setuptools-0.4a2 resource_filename prob Message-ID: Ran into a minor issue with resource_filename: Traceback (most recent call last): File "pkg_resources.py", line 388, in resource_filename File "pkg_resources.py", line 728, in get_resource_filename File "pkg_resources.py", line 740, in _extract_resource File "pkg_resources.py", line 569, in resource_isdir NameError: global name 'resource_name' is not defined One liner: Index: pkg_resources.py =================================================================== RCS file: /cvsroot/python/python/nondist/sandbox/setuptools/ pkg_resources.py,v retrieving revision 1.29 diff -u -r1.29 pkg_resources.py --- pkg_resources.py 12 Jun 2005 21:47:33 -0000 1.29 +++ pkg_resources.py 13 Jun 2005 23:26:44 -0000 @@ -566,7 +566,7 @@ return yield_lines(self.get_metadata(name)) def resource_isdir(self,name): - return self._isdir(self._fn(self.module_path, resource_name)) + return self._isdir(self._fn(self.module_path, name)) def metadata_isdir(self,name): return self.egg_info and self._isdir(self._fn (self.egg_info,name)) Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From pje at telecommunity.com Tue Jun 14 01:51:22 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 13 Jun 2005 19:51:22 -0400 Subject: [Distutils] setuptools-0.4a2: Eggs, scripts, and __file__ In-Reply-To: <596A0467-191B-4598-8178-4C1CA7A9A741@gmail.com> References: <5.1.1.6.0.20050613113419.01deb5d8@mail.telecommunity.com> <5.1.1.6.0.20050613005536.01de8158@mail.telecommunity.com> <5.1.1.6.0.20050613005536.01de8158@mail.telecommunity.com> <5.1.1.6.0.20050613113419.01deb5d8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050613183935.03194178@mail.telecommunity.com> At 06:00 PM 6/13/2005 -0400, Ryan Tomayko wrote: >On a OS X darwinports box: > > $ port list | grep -e '^py-' | wc -l > 237 > >On a Fedora 3 box (Core + Extras): > > $ yum list all | grep -e '^py' -e 'python' | wc -l > 78 Impressive. :) >I don't have any debian or gentoo boxes handy but I imagine they'd >weigh in somewhere around the darwinports number. > >None of these packages are currently provided as eggs or with .egg- info >directories when they are installed to site-packages and they >have complex dependency relationships that are managed by the >distribution's utility (port, yum, apt-get, emerge, etc.) This >creates a problem for these packages because it means that they can >not assume dependencies will always be egg managed. If they start >adding require() calls to their scripts, they will break under these >environments. require() is an all or nothing proposition for >distributions and that means there will need to be a planned >"upgrade" period or something for all packages. > >As a more specific example, I contribute to two packages that are >distributed with Fedora Core: python-urlgrabber and yum. yum depends >on python-urlgrabber and python-elementtree. Now, if I wanted to move >yum to be egg based and use require(), I would also need to ensure >that all yum's dependencies are egg based. When yum (and its >dependencies) are installed from RPM, they must all be in egg format >(or at least provide .egg-info dirs). If not, the yum script will fail. > >So a single package using require() can cause a snowball effect where >many other packages would need to be upgraded to egg format as well. >In time, this may be a good thing because it could accelerate >adoption of eggs but for the time being it makes it really hard to >use require(). I'm not seeing how this is any different than if you just started requiring a newer version of a package. I mean, if 'yum' needed a newer version of elementtree, it would force an upgrade. So why can't you just rely on a later "port number"? ISTM that most packaging systems have something like '-1' or 'p1' or 'nb1' (NetBSD) tagged on a revision to identify changes in the packaging or platform-specific patches applied. Couldn't you use that to make your 'yum' RPM depend on egg-packaged versions of its dependencies? I understand you're saying it's a big problem, but the truth is that relatively few existing Python packages have a lot of dependencies; the dependency tree of the 237 darwinports is probably extremely flat. The problems today of depending on anything are such that few people do; this makes it relatively simple for the maintainer of a single port to just go ahead and upgrade the dependencies, too (organizational issues notwithstanding). But I am obviously no expert in these matters, so I defer to you here. I'm just saying that distribution packages that depend on more than one or two other packages are rare in Python today, and the things that do get depended on, tend to be frequently used, so when you do port a dependency, it significantly reduces the number of dependencies that *need* to be ported. Thus, I think that although the problem appears huge in potential, I think that the actual interconnectedness of the packages is probably quite small. >>Or maybe this could be done by metadata -- you put a legacy.py file >>in your egg-info, and when processing your egg's dependencies, if >>pkg_resources can't find a package you need, it would call a >>function in legacy.py that would check for the dependency using >>setuptools' inspection facilities, and return a path and a guess at >>a version number. >> >>How does that sound? > >That would solve my problem perfectly. I'll give this some thought for the 0.5/0.6 releases, then. Interestingly enough, this technique could possibly give someone the opportunity to do things like look for dynamic link libraries or headers, check operating system versions, etc. >>I'm not sure exactly what you're trying to do here. If you just >>want to know if your script is running from a development location >>(and therefore needs to call require() to set up dependencies), >>couldn't you just check for 'MyPackage.egg-info' in the sys.path[0] >>(script) directory? >> >>e.g.: >> >> import sys, os >> if os.path.isdir(os.path.join(sys.path[0],"MyPackage.egg-info")): >> from pkg_resources import require >> require("MyPackage") # ensures dependencies get processed >> >>If this is what you want, perhaps we can create a standard recipe >>in pkg_resources, like maybe 'script_package("MyPackage")', that >>only does the require if you're a development egg and not being run >>from run_main(). You didn't answer this, by the way. >I'd be happy to advocate to / work with packagers once we get a basic >set of best practices together. It seems like there are a lot of >options here - we just need to iron out the details. Yeah; I think that basically the best approach for packaging systems will be to run EasyInstall during install and uninstall to modify the easyinstall.pth file. I also think that if in Python 2.5 we can change the bdist_* commands to create packages this way, then that should help, too. From pje at telecommunity.com Tue Jun 14 03:42:00 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 13 Jun 2005 21:42:00 -0400 Subject: [Distutils] PATCH: setuptools-0.4a2 resource_filename prob In-Reply-To: Message-ID: <5.1.1.6.0.20050613214023.01de99d8@mail.telecommunity.com> At 07:33 PM 6/13/2005 -0400, Ryan Tomayko wrote: >Ran into a minor issue with resource_filename: > > Traceback (most recent call last): > File "pkg_resources.py", line 388, in resource_filename > File "pkg_resources.py", line 728, in get_resource_filename > File "pkg_resources.py", line 740, in _extract_resource > File "pkg_resources.py", line 569, in resource_isdir > NameError: global name 'resource_name' is not defined Fixed in 0.4a3, along with the __file__ problem. I haven't done anything yet on implementing 'legacy.py', as there are still some details to think about. From rtomayko at gmail.com Tue Jun 14 13:00:39 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Tue, 14 Jun 2005 07:00:39 -0400 Subject: [Distutils] PATCH: setuptools-0.4a2 resource_filename prob In-Reply-To: <5.1.1.6.0.20050613214023.01de99d8@mail.telecommunity.com> References: <5.1.1.6.0.20050613214023.01de99d8@mail.telecommunity.com> Message-ID: On Jun 13, 2005, at 9:42 PM, Phillip J. Eby wrote: > At 07:33 PM 6/13/2005 -0400, Ryan Tomayko wrote: > >> Ran into a minor issue with resource_filename: >> >> Traceback (most recent call last): >> File "pkg_resources.py", line 388, in resource_filename >> File "pkg_resources.py", line 728, in get_resource_filename >> File "pkg_resources.py", line 740, in _extract_resource >> File "pkg_resources.py", line 569, in resource_isdir >> NameError: global name 'resource_name' is not defined > > Fixed in 0.4a3, along with the __file__ problem. Great. > I haven't done anything yet on implementing 'legacy.py', as there > are still some details to think about. No prob. I'm not in a big rush or anything. I just wanted to raise the issue sooner than later. Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From rtomayko at gmail.com Tue Jun 14 13:04:35 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Tue, 14 Jun 2005 07:04:35 -0400 Subject: [Distutils] setuptools-0.4a2: Eggs, scripts, and __file__ In-Reply-To: <5.1.1.6.0.20050613183935.03194178@mail.telecommunity.com> References: <5.1.1.6.0.20050613113419.01deb5d8@mail.telecommunity.com> <5.1.1.6.0.20050613005536.01de8158@mail.telecommunity.com> <5.1.1.6.0.20050613005536.01de8158@mail.telecommunity.com> <5.1.1.6.0.20050613113419.01deb5d8@mail.telecommunity.com> <5.1.1.6.0.20050613183935.03194178@mail.telecommunity.com> Message-ID: On Jun 13, 2005, at 7:51 PM, Phillip J. Eby wrote: > At 06:00 PM 6/13/2005 -0400, Ryan Tomayko wrote: >> So a single package using require() can cause a snowball effect where >> many other packages would need to be upgraded to egg format as well. >> In time, this may be a good thing because it could accelerate >> adoption of eggs but for the time being it makes it really hard to >> use require(). > > I'm not seeing how this is any different than if you just started > requiring a newer version of a package. I mean, if 'yum' needed a > newer version of elementtree, it would force an upgrade. So why > can't you just rely on a later "port number"? ISTM that most > packaging systems have something like '-1' or 'p1' or > 'nb1' (NetBSD) tagged on a revision to identify changes in the > packaging or platform-specific patches applied. Couldn't you use > that to make your 'yum' RPM depend on egg-packaged versions of its > dependencies? Sure. It's just that down-level cascading upgrades like this are generally discouraged if they can be avoided. The legacy.py approach or a mechanism like it would be preferable until such time as a general policy for packaging with eggs can be devised by the distribution. It's not a show stopper, just a mild concern. > I understand you're saying it's a big problem, but the truth is > that relatively few existing Python packages have a lot of > dependencies; the dependency tree of the 237 darwinports is > probably extremely flat. The problems today of depending on > anything are such that few people do; this makes it relatively > simple for the maintainer of a single port to just go ahead and > upgrade the dependencies, too (organizational issues notwithstanding). Perhaps your right. The nice thing is that in the case of libraries (elementtree for example) no upstream changes are needed to the code and the package can be eggified by the packager. This makes the whole thing a bit less of an issue. > But I am obviously no expert in these matters, so I defer to you > here. I'm just saying that distribution packages that depend on > more than one or two other packages are rare in Python today, and > the things that do get depended on, tend to be frequently used, so > when you do port a dependency, it significantly reduces the number > of dependencies that *need* to be ported. Thus, I think that > although the problem appears huge in potential, I think that the > actual interconnectedness of the packages is probably quite small. I guess we'll find out in the coming months as maintainers become more aware of the advantage of egg based development / deployment. Like I was saying earlier, this may be to our advantage as packages moving to eggs will nudge others in that direction as well. >>> Or maybe this could be done by metadata -- you put a legacy.py file >>> in your egg-info, and when processing your egg's dependencies, if >>> pkg_resources can't find a package you need, it would call a >>> function in legacy.py that would check for the dependency using >>> setuptools' inspection facilities, and return a path and a guess at >>> a version number. >>> >>> How does that sound? >> >> That would solve my problem perfectly. > > I'll give this some thought for the 0.5/0.6 releases, then. > > Interestingly enough, this technique could possibly give someone > the opportunity to do things like look for dynamic link libraries > or headers, check operating system versions, etc. Yea. I was thinking the same thing. Having a "package preload" area could be useful in a variety of ways. >>> I'm not sure exactly what you're trying to do here. If you just >>> want to know if your script is running from a development location >>> (and therefore needs to call require() to set up dependencies), >>> couldn't you just check for 'MyPackage.egg-info' in the sys.path[0] >>> (script) directory? >>> >>> e.g.: >>> >>> import sys, os >>> if os.path.isdir(os.path.join(sys.path[0],"MyPackage.egg- >>> info")): >>> from pkg_resources import require >>> require("MyPackage") # ensures dependencies get processed >>> >>> If this is what you want, perhaps we can create a standard recipe >>> in pkg_resources, like maybe 'script_package("MyPackage")', that >>> only does the require if you're a development egg and not being run >>> from run_main(). >>> > > You didn't answer this, by the way. I'm hoping this won't be necessary with legacy.py or some variation thereof. I wasn't trying to detect a development environment so much as I was trying to detect whether I was egg managed or some-external- package-manager managed. The idea was to limit require() calls to when my package was an egg or had egg meta data, otherwise assume some external package manager (or manual setup) is responsible for setting up sys.path. >> I'd be happy to advocate to / work with packagers once we get a basic >> set of best practices together. It seems like there are a lot of >> options here - we just need to iron out the details. > > Yeah; I think that basically the best approach for packaging > systems will be to run EasyInstall during install and uninstall to > modify the easyinstall.pth file. I also think that if in Python > 2.5 we can change the bdist_* commands to create packages this way, > then that should help, too. I think so too. Wide changes to packages are not unexpected when the python version changes so if this did make it into 2.5, I might even be able to convince fedora-devel packagers to move python packages to eggs or at least provide .egg-info directories and calls to EasyInstall for Fedora Core 5. I'm not very close to the BSD ports or other packaging communities but I wouldn't have a problem advocating the same to them if things went well with fedora. Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From rtomayko at gmail.com Tue Jun 14 14:17:55 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Tue, 14 Jun 2005 08:17:55 -0400 Subject: [Distutils] PATCH: setuptools-0.4a3 resource loading from egg problem Message-ID: <0F9C799F-9D0D-4E31-847A-CF4304540CBD@gmail.com> I've been getting the following when trying to use resource_filename when my package is in a zipped egg (since somewhere around 0.3a2 or so): Traceback (most recent call last): File "/usr/local/bin/somescript.py", line 4, in ? pkg_resources.run_main('somepackage==1.0', 'somescript.py') File "setuptools/pkg_resources.py", line 110, in run_main require(dist_spec)[0].metadata.run_script(script_name, __main__.__dict__) File "setuptools/pkg_resources.py", line 599, in run_script exec script_code in namespace, namespace File "/usr/local/bin/somescript.py", line 7, in ? File "egg/somepackage/somemodule.py", line 3, in ? File "setuptools/pkg_resources.py", line 388, in resource_filename return get_provider(package_name).get_resource_filename( File "setuptools/pkg_resources.py", line 728, in get_resource_filename return self._extract_resource(manager, resource_name) File "setuptools/pkg_resources.py", line 745, in _extract_resource zip_stat = self.zipinfo[os.path.join(*self.prefix+parts)] KeyError: 'somepackage/someresource.txt' Here's a test egg that illustrates the problem: Install that and then run the included "somescript.py". You should see the traceback above. The problem seems to be with loading resources relative to a module in a package. The egg contains a package "somepackage" and a module within that package "somemodule". somemodule calls ``resource_filename (__name__, 'someresource.txt')``. The ZipProvider seems to handle this case badly. Attached is a patch against 0.4a3 that fixes this particular case but I'm not confident it won't cause problems elsewhere. I'll comment each hunk here with an explanation of what I think may be happening. @@ -573,7 +573,7 @@ def resource_listdir(self,resource_name): - return self._listdir(self._fn(self.egg_info,resource_name)) + return self._listdir(self._fn(self.module_path, resource_name)) def metadata_listdir(self,name): if self.egg_info: Here we see NullProvider.resource_listdir using [path]/EGG_INFO instead of [path]. This seems like a simple typo to me as there are resource_XXX and metadata_XXX methods, the first based on the module path and the second set based on the EGG_INFO dir. @@ -703,7 +703,7 @@ def __init__(self, module): DefaultProvider.__init__(self,module) self.zipinfo = zipimport._zip_directory_cache [self.loader.archive] - self.zip_pre = self.loader.archive+os.sep + self.zip_pre = self.module_path + os.sep def _short_name(self, path): if path.startswith(self.zip_pre): This is where I was a bit shaky: loader_archive = /path/to/the-egg.egg module_path = /path/to/the-egg.egg/package Shouldn't resource loading be relative to the package directory and not the root of the egg? @@ -738,7 +738,7 @@ def _extract_resource(self, manager, resource_name): if self.resource_isdir(resource_name): - return self._extract_dir(resource_name) + return self._extract_directory(manager, resource_name) parts = resource_name.split('/') zip_path = os.path.join(self.module_path, *parts) I believe this was a simple typo / refactoring-miss. This isn't breaking the test case but is causing problems in another package that is using resource_filename to get a directory. Thanks, Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ -------------- next part -------------- A non-text attachment was scrubbed... Name: zip-resource-loading.patch Type: application/octet-stream Size: 1107 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20050614/23661a54/zip-resource-loading.obj -------------- next part -------------- From pje at telecommunity.com Tue Jun 14 14:47:50 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 14 Jun 2005 08:47:50 -0400 Subject: [Distutils] PATCH: setuptools-0.4a3 resource loading from egg problem In-Reply-To: <0F9C799F-9D0D-4E31-847A-CF4304540CBD@gmail.com> Message-ID: <5.1.1.6.0.20050614083214.01e0cb00@mail.telecommunity.com> At 08:17 AM 6/14/2005 -0400, Ryan Tomayko wrote: >Here's a test egg that illustrates the problem: > > > >Install that and then run the included "somescript.py". You should >see the traceback above. I don't; it works fine on my (Windows) PC, assuming all it does is spit out a filename. >Attached is a patch against 0.4a3 that fixes this particular case but >I'm not confident it won't cause problems elsewhere. I'll comment >each hunk here with an explanation of what I think may be happening. > >@@ -573,7 +573,7 @@ > def resource_listdir(self,resource_name): >- return self._listdir(self._fn(self.egg_info,resource_name)) >+ return self._listdir(self._fn(self.module_path, resource_name)) > def metadata_listdir(self,name): > if self.egg_info: > >Here we see NullProvider.resource_listdir using [path]/EGG_INFO >instead of [path]. This seems like a simple typo to me as there are >resource_XXX and metadata_XXX methods, the first based on the module >path and the second set based on the EGG_INFO dir. Yeah, that's a typo. :( >@@ -703,7 +703,7 @@ > def __init__(self, module): > DefaultProvider.__init__(self,module) > self.zipinfo = zipimport._zip_directory_cache [self.loader.archive] >- self.zip_pre = self.loader.archive+os.sep >+ self.zip_pre = self.module_path + os.sep > def _short_name(self, path): > if path.startswith(self.zip_pre): > >This is where I was a bit shaky: > > loader_archive = /path/to/the-egg.egg > module_path = /path/to/the-egg.egg/package > >Shouldn't resource loading be relative to the package directory and >not the root of the egg? zip_pre is used only to remove the front end of path strings in order to get them out of the zipimport module's zip directory cache. It has to be loader.archive+os.sep, since that's how zipimport constructs its cache. So this should not be changed. >@@ -738,7 +738,7 @@ > def _extract_resource(self, manager, resource_name): > if self.resource_isdir(resource_name): >- return self._extract_dir(resource_name) >+ return self._extract_directory(manager, resource_name) > parts = resource_name.split('/') > zip_path = os.path.join(self.module_path, *parts) > >I believe this was a simple typo / refactoring-miss. Yeah, I really need to get creative about how to unit test some of this stuff so I can get it back up to full test coverage. :( However, the refactorings I've just completed will make testing easier. For example, I can now subclass PackageIndex and replace its open_url() method with a mock/stub for testing purposes. Similarly, I can subclass easy_install and NullProvider to stub out methods for testing as well. In the meantime, though, I'm definitely paying for my lack of upfront testing in several areas, as the entire system has grown from a few hundred to a few thousand lines of code. From rtomayko at gmail.com Tue Jun 14 15:43:41 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Tue, 14 Jun 2005 09:43:41 -0400 Subject: [Distutils] PATCH: setuptools-0.4a3 resource loading from egg problem In-Reply-To: <5.1.1.6.0.20050614083214.01e0cb00@mail.telecommunity.com> References: <5.1.1.6.0.20050614083214.01e0cb00@mail.telecommunity.com> Message-ID: <30AC3EEB-EBBF-4D95-AB38-03C449FDD8B6@gmail.com> On Jun 14, 2005, at 8:47 AM, Phillip J. Eby wrote: > At 08:17 AM 6/14/2005 -0400, Ryan Tomayko wrote: >> Here's a test egg that illustrates the problem: >> >> >> >> Install that and then run the included "somescript.py". You should >> see the traceback above. >> > > I don't; it works fine on my (Windows) PC, assuming all it does is > spit out a filename. Oops. Sorry, that test is bad. The problem only seems to manifest itself when you're trying to use resource_filename on a directory, not on a filename. I put another test up there. If you could check that out I'd appreciate it: The package structure: somepackage/__init__.py somepackage/somemodule.py somepackage/wtf/someresource.txt somemodule.py is calling resource_filename as follows:: resource_filename(__name__, 'wtf') This is failing on my end with a KeyError. The full patch fixes it but if I revert the zip_pre change I get the KeyError. >> @@ -703,7 +703,7 @@ >> def __init__(self, module): >> DefaultProvider.__init__(self,module) >> self.zipinfo = zipimport._zip_directory_cache >> [self.loader.archive] >> - self.zip_pre = self.loader.archive+os.sep >> + self.zip_pre = self.module_path + os.sep >> def _short_name(self, path): >> if path.startswith(self.zip_pre): >> >> This is where I was a bit shaky: >> >> loader_archive = /path/to/the-egg.egg >> module_path = /path/to/the-egg.egg/package >> >> Shouldn't resource loading be relative to the package directory and >> not the root of the egg? >> > > zip_pre is used only to remove the front end of path strings in > order to get them out of the zipimport module's zip directory > cache. It has to be loader.archive+os.sep, since that's how > zipimport constructs its cache. So this should not be changed. It also seems to be used to figure out key names in the zip index and this seems to be the problem. > In the meantime, though, I'm definitely paying for my lack of > upfront testing in several areas, as the entire system has grown > from a few hundred to a few thousand lines of code. That's been my experience with unit testing: pay me now or pay me later, but you'll pay. Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From pje at telecommunity.com Tue Jun 14 17:22:01 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 14 Jun 2005 11:22:01 -0400 Subject: [Distutils] PATCH: setuptools-0.4a3 resource loading from egg problem In-Reply-To: <30AC3EEB-EBBF-4D95-AB38-03C449FDD8B6@gmail.com> References: <5.1.1.6.0.20050614083214.01e0cb00@mail.telecommunity.com> <5.1.1.6.0.20050614083214.01e0cb00@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050614111327.01dfa830@mail.telecommunity.com> At 09:43 AM 6/14/2005 -0400, Ryan Tomayko wrote: >It also seems to be used to figure out key names in the zip index and >this seems to be the problem. Actually, the problem is more subtle than that, but basically the same. The right solution is not to change zip_pre, but rather to have the directory operations trim a different prefix. Here's a trial change: def _isdir(self,path): return self._dir_name(path) in self._index() def _listdir(self,path): return list(self._index().get(self._dir_name(path), ())) def _dir_name(self,path): if path.startswith(self.module_path+os.sep): path = path[len(self.module_path+os.sep):] path = path.replace(os.sep,'/') if path.endswith('/'): path = path[:-1] return path Replace these functions in ZipProvider, and then make the start of extract_resource look like this: def _extract_resource(self, manager, resource_name): if self.resource_isdir(resource_name): return self._extract_directory(manager, resource_name) And you should get working code (works for me, IOW). These fixes will go out in 0.4a4; thanks for all your help in testing. From rtomayko at gmail.com Tue Jun 14 17:43:22 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Tue, 14 Jun 2005 11:43:22 -0400 Subject: [Distutils] PATCH: setuptools-0.4a3 resource loading from egg problem In-Reply-To: <5.1.1.6.0.20050614111327.01dfa830@mail.telecommunity.com> References: <5.1.1.6.0.20050614083214.01e0cb00@mail.telecommunity.com> <5.1.1.6.0.20050614083214.01e0cb00@mail.telecommunity.com> <5.1.1.6.0.20050614111327.01dfa830@mail.telecommunity.com> Message-ID: On Jun 14, 2005, at 11:22 AM, Phillip J. Eby wrote: > At 09:43 AM 6/14/2005 -0400, Ryan Tomayko wrote: > >> It also seems to be used to figure out key names in the zip index and >> this seems to be the problem. > > Actually, the problem is more subtle than that, but basically the > same. The right solution is not to change zip_pre, but rather to > have the directory operations trim a different prefix. Here's a > trial change: > > Works for me. > And you should get working code (works for me, IOW). These fixes > will go out in 0.4a4; thanks for all your help in testing. No prob. Everything seems to be working well with the few packages I'm testing this out on. Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From pje at telecommunity.com Tue Jun 14 18:01:08 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 14 Jun 2005 12:01:08 -0400 Subject: [Distutils] 0.4a4 released (was Re: PATCH: setuptools-0.4a3 resource loading from egg problem) In-Reply-To: References: <5.1.1.6.0.20050614111327.01dfa830@mail.telecommunity.com> <5.1.1.6.0.20050614083214.01e0cb00@mail.telecommunity.com> <5.1.1.6.0.20050614083214.01e0cb00@mail.telecommunity.com> <5.1.1.6.0.20050614111327.01dfa830@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050614115908.01e0a968@mail.telecommunity.com> At 11:43 AM 6/14/2005 -0400, Ryan Tomayko wrote: >On Jun 14, 2005, at 11:22 AM, Phillip J. Eby wrote: >>At 09:43 AM 6/14/2005 -0400, Ryan Tomayko wrote: >> >>>It also seems to be used to figure out key names in the zip index and >>>this seems to be the problem. >> >>Actually, the problem is more subtle than that, but basically the >>same. The right solution is not to change zip_pre, but rather to >>have the directory operations trim a different prefix. Here's a >>trial change: >> >> > >Works for me. > >>And you should get working code (works for me, IOW). These fixes >>will go out in 0.4a4; thanks for all your help in testing. > >No prob. Everything seems to be working well with the few packages >I'm testing this out on. I just pushed out 0.4a4; it also adds support for the standard -distutils q/-v/-n/-O options, cleans up the progress messages a bit, and supports downloading packages uploaded to PyPI. From ianb at colorstudy.com Wed Jun 15 20:17:29 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 15 Jun 2005 13:17:29 -0500 Subject: [Distutils] EasyInstall: suggested usage patterns Message-ID: <42B070B9.2010209@colorstudy.com> Hi -- I'm getting ready to go out of town, so I haven't been able to track everything that's gone on in 0.4 (or 0.5 now?), and I might not be able to follow up on this conversation very thoroughly, but anyway... I feel like we need several suggested ways to use easy_install and eggs, given different cases. This is probably a documentation task; I'm more interested in how I *should* use this, rather than how I *can*. I guess that's the same as best practices. In turn I'd like to echo these best practices in any documentation or examples that use easy_install/eggs. Anyway, here's some cases: Development * I am developing an application. I don't have a distutil script for the application. * I am simultaneously developing distutils-based libraries. I want to edit these during my development process. * I also have some distutils libraries that I'm not developing, but I need to install. * I also have some libraries laying around that I already installed (before easy_install existed). And some that are from OS packages. * I have other Python stuff on my computer that I don't want to mess up because of my ongoing development. So, given this, what commands do I run? Where do I put files? What if I don't have permission to put files in global locations (site-packages)? How do I clean up after myself later? Later, this becomes... Deployment * I have developed a web application. Maybe it also doesn't have a distutils script...? It could, though currently I don't develop one for my web applications. Also, I sometimes make hot fixes, especially when the application is deployed but not yet live. * How and where should libraries be installed? How should application dependencies be expressed? * Some libraries are internal, and so aren't available from a public location. Maybe on the web with HTTP auth, though I'm more inclined to simply keep them in a branch in the private repository. Or fetch over scp. * Should I change my require()s to use a specific version of the libraries, so that I don't accidentally upgrade (/break) the application when a later application is installed? How do I manage that process? * Later a library might be broken and I need to fix it. Is there anything I should do on installation so I can later track who uses what library? Also so I can collect unused versions of libraries. Non-code * I'd like to distribute some data that doesn't have much (or maybe any) code. This might be a Javascript library, or a skin for the application (a bunch of templates and images), or something like that. Can I facilitate that with easy_install? Enabling plugins * I have library A, and library B. Library B optionally provides a "plugin" to library A, but both are usable in isolation. Library B needs to inject stuff into library A -- i.e., at runtime some code in library B enhances library A. How do I make this work? How do I make library A aware of library B? Other People's Code * Someone wrote some code I'd like to use. But it's poorly packaged -- maybe no setup.py, or maybe a bad one. For instance, I've decided that zpt.sf.net's setup.py is just broken -- you can't use extra_path, no package, and provide an __init__.py all at once. I'd like to write my own setup.py, but use that package. And it's on SF, so I'd like to use easy_install to download the package. Those are some of the things I'd like to do now -- easy_install doesn't have to magically make all of them work wonderfully; if I have to do things by hand, keep separate records, write custom code, or whatever, that's fine; I just want to know what I should be doing right now for each of these cases. Also, I'm interested in conventions we can define so that we all start doing the same thing. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Wed Jun 15 22:45:50 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 15 Jun 2005 16:45:50 -0400 Subject: [Distutils] EasyInstall: suggested usage patterns In-Reply-To: <42B070B9.2010209@colorstudy.com> Message-ID: <5.1.1.6.0.20050615150342.01e00b28@mail.telecommunity.com> At 01:17 PM 6/15/2005 -0500, Ian Bicking wrote: >Hi -- I'm getting ready to go out of town, so I haven't been able to >track everything that's gone on in 0.4 (or 0.5 now?), and I might not be >able to follow up on this conversation very thoroughly, but anyway... > >I feel like we need several suggested ways to use easy_install and eggs, >given different cases. This is probably a documentation task; I'm more >interested in how I *should* use this, rather than how I *can*. I guess >that's the same as best practices. In turn I'd like to echo these best >practices in any documentation or examples that use easy_install/eggs. > >Anyway, here's some cases: > >Development > >* I am developing an application. I don't have a distutil script for >the application. So your application's startup code needs to either require() all the libraries you need, or require an egg representing the application, which then lists all its dependencies. To create an egg representing the application, just create a MyApp.egg-info directory containing a PKG-INFO with a non-empty version number. Then you can create a depends.txt with your dependencies. >* I am simultaneously developing distutils-based libraries. I want to >edit these during my development process. Check out their source, run bdist_egg on them, and add their source directories to PYTHONPATH or a .pth file. >* I also have some distutils libraries that I'm not developing, but I >need to install. Run 'easy_install whatever' to get them. >* I also have some libraries laying around that I already installed >(before easy_install existed). And some that are from OS packages. If you have conflicting packages, you'll have to uninstall them. If they're not conflicting, you can create .egg-info directories alongside the packages with a PKG-INFO indicating their version numbers, in order for you to be able to safely include them in your depends.txt >* I have other Python stuff on my computer that I don't want to mess up >because of my ongoing development. Okay, then you are going to have to use PYTHONPATH or have some kind of hook that your application uses to add locations to sys.path. The built eggs, however, can at least be dumped in the same directory as your app's startup script. Also if you're very careful, you can probably do subversion or CVS checkout setups such that all your development libraries can live in the same directory, with a bunch of .egg-info directories in there as well. This can be a bit tricky to set up, but if you're doing simultaneous development of a bunch of libraries, it might be worth the trouble. >So, given this, what commands do I run? Where do I put files? My (evolving) thinking on this is the notion of "project directories", where a project directory contains: * A setup.py (optional but recommended) * A setup.cfg (optional, but useful for configuring easy_install) * .egg-info directory and package directory checkouts for all libraries I'm developing as part of the project * All scripts being developed or used by the project * All eggs installed solely for the project's use > What if >I don't have permission to put files in global locations >(site-packages)? Use '-d.' and run EasyInstall in the project directory. > How do I clean up after myself later? Wipe out the project directory. In one of the next phases of EasyInstall development, I plan to add a 'package' script with various subcommands like 'list', 'upgrade', etc. I was already thinking it would be useful to have a 'package create' command that would create a basic project directory for you. Now that I've thought about your scenario here a bit, I think I'd like to also have a 'package develop' that would let you check out packages and their .egg-infos (given some kind of configuration of their CVS/SVN locations) and add them to your depends.txt if they weren't already mentioned. I can also envision a variety of tools springing up in package.py for manipulating the project environment. >Later, this becomes... > >Deployment > >* I have developed a web application. Maybe it also doesn't have a >distutils script...? Note that with setuptools, a setup script doesn't have to say much besides metadata like name, version, author, etc. You can use the 'find_packages()' function to automatically include all packages (perhaps excepting those that have been somehow marked as being checked out by 'package.py develop'), so it's mostly a matter of listing scripts and such. Note, by the way, that information like author, license, and a number of other things could easily be defaulted by a configuration file, too, if set in e.g. the ~/.pydistutils.cfg file on a per-user basis. > It could, though currently I don't develop one for >my web applications. Also, I sometimes make hot fixes, especially when >the application is deployed but not yet live. Surely this could be done by deploying a project directory? >* How and where should libraries be installed? How should application >dependencies be expressed? Same as before: any directory on sys.path, and via depends.txt >* Some libraries are internal, and so aren't available from a public >location. Maybe on the web with HTTP auth, though I'm more inclined to >simply keep them in a branch in the private repository. Or fetch over scp. Sure; EasyInstall also supports "find_links" pages that list links to source archives or eggs, so you can use this technique to access them more easily by putting the download pages in your configuration file(s). >* Should I change my require()s to use a specific version of the >libraries, so that I don't accidentally upgrade (/break) the application >when a later application is installed? How do I manage that process? That won't prevent breakage, if you end up with a conflict between the versions required by multiple components. Sadly, I have no silver bullet for you regarding management. >* Later a library might be broken and I need to fix it. Is there >anything I should do on installation so I can later track who uses what >library? Also so I can collect unused versions of libraries. Commands to do these things should probably be added to the 'package.py' script, once it exists. The information is certainly there. >Non-code > >* I'd like to distribute some data that doesn't have much (or maybe any) >code. This might be a Javascript library, or a skin for the application >(a bunch of templates and images), or something like that. Can I >facilitate that with easy_install? You need at least a zero-length __init__.py that the data lives alongside, but yes. (i.e., using an otherwise-empty package as a data carrier is fine.) >Enabling plugins > >* I have library A, and library B. Library B optionally provides a >"plugin" to library A, but both are usable in isolation. Library B >needs to inject stuff into library A -- i.e., at runtime some code in >library B enhances library A. How do I make this work? How do I make >library A aware of library B? I plan to add an 'add_activation_listener()' API to pkg_resources that will let library A get callbacks when new eggs are activated (and will also get callbacks for the eggs that are already activated when the listener is added). This will let library A ask for files in library B's metadata (EGG-INFO/.egg-info) directory, and do any desired automatic registration. Library A will have to define a metadata standard for what filename it looks for and what the contents should be. I expect Paste, Twisted, PEAK, and Zope will probably all want to define such standards. Of course, performance for this is O(nm) where n is the number of listeners and m is the number of eggs, but n will usually be small. However, zipped distributions can check for the existence of files with a simple dictionary lookup, so performance there will be good. At some point we'll need to start pushing people to make their eggs "compression-ready, for enhanced performance", so EasyInstall can be '--zip-ok' by default instead of only doing it for packages distributed as .egg files. >Other People's Code > >* Someone wrote some code I'd like to use. But it's poorly packaged -- >maybe no setup.py, or maybe a bad one. For instance, I've decided that >zpt.sf.net's setup.py is just broken -- you can't use extra_path, no >package, and provide an __init__.py all at once. I'd like to write my >own setup.py, but use that package. And it's on SF, so I'd like to use >easy_install to download the package. Make your setup.py use 'ez_setup' to bootstrap setuptools. Then create a setuptools.package_index.PackageIndex instance, then call its 'download()' method to retrieve the desired package, and use setuptools.archive_util.extract_archive() to unpack it. Then, write a flag file so that if setup.py is rerun you know you already did all that! Use appropriate directory information in the setup() call so that stuff is built from the extracted package, and of course supply correct metadata in your parent setup script. Voila. >Those are some of the things I'd like to do now -- easy_install doesn't >have to magically make all of them work wonderfully; if I have to do >things by hand, keep separate records, write custom code, or whatever, >that's fine; I just want to know what I should be doing right now for >each of these cases. Also, I'm interested in conventions we can define >so that we all start doing the same thing. Luckily, it's pretty magically wonderful for most of what you describe, and for the rest there mostly already exist plans to take care of 'em. Mostly, the support is still a bit weak on the development side of things, but I think we can fix those right up with some "wizards" in package.py to perform common operations like creating a package, setting your default author name and other data, doing CVS/SVN checkouts, etc. (And perhaps the folks doing Python IDE's will start putting menu items in their systems to run these commands, too.) Also, documentation is really lacking right now, so if folks can contribute How-To's, tutorials, and similar material to complement my overview and reference docs, it'll help other people start using all this. From ianb at colorstudy.com Thu Jun 16 07:30:59 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 16 Jun 2005 00:30:59 -0500 Subject: [Distutils] EasyInstall: development environment Message-ID: <42B10E93.7030806@colorstudy.com> Working from Phillip's last email, here's my proposed list of steps for creating a development environment; some of these could be automated, but lets presume the automation does the same thing this manual process does... Application name: MyApp # We don't care where this directory is, except that we put all our # in-development checkouts here... cd ~/devel-directory mkdir MyApp # or an svn repository /MyApp/trunk, and check out to MyApp/ cd MyApp mkdir MyApp.egg-info echo "Name: MyApp Version: 0.0" > MyApp.egg-info/PKG-INFO echo "SQLObject>=0.6.1 pysqlite2 Cheetah" > MyApp.egg-info/depends.txt # But I'm going to be developing SQLObject, but not pysqlite and # Cheetah... svn co http://svn.colorstudy.com/trunk/SQLObject ../SQLObject cd ../SQLObject # Well, this isn't right, because SQLObject doesn't import # setuptools... I guess I have to run package/easy_install somehow? python setup.py bdist_egg cd .. export PYTHONPATH=`pwd` cd MyApp mkdir app-packages # What do I run here to get the dependencies that aren't already # fulfilled? easy_install -d app-packages (?) # But then I try to install pysqlite, but there's some weird conflict, # and I wish I was just using the OS package... so what do I do to # satisfy the dependency? Preferably without writing to # site-packages/, where pysqlite is installed? # and where are my scripts at this point? What should MyApp/setup.py # look like? What should MyApp/ look like? Should it have a package # underneath it? -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Thu Jun 16 07:41:11 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 16 Jun 2005 00:41:11 -0500 Subject: [Distutils] EasyInstall: suggested usage patterns In-Reply-To: <5.1.1.6.0.20050615150342.01e00b28@mail.telecommunity.com> References: <5.1.1.6.0.20050615150342.01e00b28@mail.telecommunity.com> Message-ID: <42B110F7.3090800@colorstudy.com> Phillip J. Eby wrote: >> Deployment >> >> * I have developed a web application. Maybe it also doesn't have a >> distutils script...? > > > Note that with setuptools, a setup script doesn't have to say much > besides metadata like name, version, author, etc. You can use the > 'find_packages()' function to automatically include all packages > (perhaps excepting those that have been somehow marked as being checked > out by 'package.py develop'), so it's mostly a matter of listing scripts > and such. > > Note, by the way, that information like author, license, and a number of > other things could easily be defaulted by a configuration file, too, if > set in e.g. the ~/.pydistutils.cfg file on a per-user basis. You mean when generating setup.py? I wouldn't want the setup.py to simply omit these details and then pick them up from a configuration file. I might forget to explicitly put those values in if that's the case, then someone else will get different values based on their environment. >> It could, though currently I don't develop one for >> my web applications. Also, I sometimes make hot fixes, especially when >> the application is deployed but not yet live. > > > Surely this could be done by deploying a project directory? You mean, deploy the application as a checkout with the egg-info directory? That's what I would be inclined to do. But it should be explicit if this is the recommended way to deploy if you are expecting to do hot fixes. >> * Some libraries are internal, and so aren't available from a public >> location. Maybe on the web with HTTP auth, though I'm more inclined to >> simply keep them in a branch in the private repository. Or fetch over >> scp. > > > Sure; EasyInstall also supports "find_links" pages that list links to > source archives or eggs, so you can use this technique to access them > more easily by putting the download pages in your configuration file(s). This might be useful to list in a ~/.pydistutils.cfg -- I could put the URL to some central page (for my company) that in turn pointed to places in a private repository. Or even just to internally preferred/tested releases of other people's software (assuming that directory took precedence over PyPI when installing without an explicit version). >> * Should I change my require()s to use a specific version of the >> libraries, so that I don't accidentally upgrade (/break) the application >> when a later application is installed? How do I manage that process? > > > That won't prevent breakage, if you end up with a conflict between the > versions required by multiple components. Sadly, I have no silver > bullet for you regarding management. Well, presumably I've tested my application in development, and I installed specific versions of packages during development. If I could record the actual versions I installed (some of which had explicitly installed versions, and some which were just the most recent release) then I could duplicate that tested environment later when the most recent versions had changed. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Thu Jun 16 17:52:49 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 16 Jun 2005 11:52:49 -0400 Subject: [Distutils] EasyInstall: suggested usage patterns In-Reply-To: <42B110F7.3090800@colorstudy.com> References: <5.1.1.6.0.20050615150342.01e00b28@mail.telecommunity.com> <5.1.1.6.0.20050615150342.01e00b28@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050616114457.0306fb10@mail.telecommunity.com> At 12:41 AM 6/16/2005 -0500, Ian Bicking wrote: >Phillip J. Eby wrote: >>>Deployment >>> >>>* I have developed a web application. Maybe it also doesn't have a >>>distutils script...? >> >>Note that with setuptools, a setup script doesn't have to say much >>besides metadata like name, version, author, etc. You can use the >>'find_packages()' function to automatically include all packages (perhaps >>excepting those that have been somehow marked as being checked out by >>'package.py develop'), so it's mostly a matter of listing scripts and such. >>Note, by the way, that information like author, license, and a number of >>other things could easily be defaulted by a configuration file, too, if >>set in e.g. the ~/.pydistutils.cfg file on a per-user basis. > >You mean when generating setup.py? I wouldn't want the setup.py to simply >omit these details and then pick them up from a configuration file. I >might forget to explicitly put those values in if that's the case, then >someone else will get different values based on their environment. Don't forget about setup.cfg, which lives alongside setup.py. I'm assuming that you would put in defaults in your ~/.pydistutils.cfg, and then they would be used to generate entries in setup.cfg when you create the project. Why not in setup.py? Because programs can edit setup.cfg more easily than they can setup.py. It would be nice if you could literally use a stock setup.py with no human-edited content, at least for "pure" packages with no special needs. >>> It could, though currently I don't develop one for >>>my web applications. Also, I sometimes make hot fixes, especially when >>>the application is deployed but not yet live. >> >>Surely this could be done by deploying a project directory? > >You mean, deploy the application as a checkout with the egg-info >directory? That's what I would be inclined to do. But it should be >explicit if this is the recommended way to deploy if you are expecting to >do hot fixes. I just look at it as still being a development environment in that case. >>>* Some libraries are internal, and so aren't available from a public >>>location. Maybe on the web with HTTP auth, though I'm more inclined to >>>simply keep them in a branch in the private repository. Or fetch over scp. >> >>Sure; EasyInstall also supports "find_links" pages that list links to >>source archives or eggs, so you can use this technique to access them >>more easily by putting the download pages in your configuration file(s). > >This might be useful to list in a ~/.pydistutils.cfg -- I could put the >URL to some central page (for my company) that in turn pointed to places >in a private repository. Or even just to internally preferred/tested >releases of other people's software (assuming that directory took >precedence over PyPI when installing without an explicit version). EasyInstall only looks on PyPI if it can't find a package that satisfies your request via the --find-links. That's true *even if* you specify a version. >>>* Should I change my require()s to use a specific version of the >>>libraries, so that I don't accidentally upgrade (/break) the application >>>when a later application is installed? How do I manage that process? >> >>That won't prevent breakage, if you end up with a conflict between the >>versions required by multiple components. Sadly, I have no silver bullet >>for you regarding management. > >Well, presumably I've tested my application in development, and I >installed specific versions of packages during development. If I could >record the actual versions I installed (some of which had explicitly >installed versions, and some which were just the most recent release) then >I could duplicate that tested environment later when the most recent >versions had changed. Sure; you can just code the specific required versions into your depends.txt. (Which I'm thinking of renaming to 'requires.txt', btw.) From pje at telecommunity.com Thu Jun 16 18:45:54 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 16 Jun 2005 12:45:54 -0400 Subject: [Distutils] EasyInstall: development environment In-Reply-To: <42B10E93.7030806@colorstudy.com> Message-ID: <5.1.1.6.0.20050616115256.02040008@mail.telecommunity.com> At 12:30 AM 6/16/2005 -0500, Ian Bicking wrote: >Working from Phillip's last email, here's my proposed list of steps for >creating a development environment; some of these could be automated, >but lets presume the automation does the same thing this manual process >does... > >Application name: MyApp > ># We don't care where this directory is, except that we put all our ># in-development checkouts here... >cd ~/devel-directory > >mkdir MyApp ># or an svn repository /MyApp/trunk, and check out to MyApp/ >cd MyApp >mkdir MyApp.egg-info > >echo "Name: MyApp >Version: 0.0" > MyApp.egg-info/PKG-INFO > >echo "SQLObject>=0.6.1 >pysqlite2 >Cheetah" > MyApp.egg-info/depends.txt > ># But I'm going to be developing SQLObject, but not pysqlite and ># Cheetah... > >svn co http://svn.colorstudy.com/trunk/SQLObject ../SQLObject >cd ../SQLObject ># Well, this isn't right, because SQLObject doesn't import ># setuptools... I guess I have to run package/easy_install somehow? >python setup.py bdist_egg > >cd .. >export PYTHONPATH=`pwd` > >cd MyApp >mkdir app-packages ># What do I run here to get the dependencies that aren't already ># fulfilled? >easy_install -d app-packages (?) > ># But then I try to install pysqlite, but there's some weird conflict, ># and I wish I was just using the OS package... so what do I do to ># satisfy the dependency? Preferably without writing to ># site-packages/, where pysqlite is installed? > ># and where are my scripts at this point? What should MyApp/setup.py ># look like? What should MyApp/ look like? Should it have a package ># underneath it? Okay, I lost you somewhere here. Here's my vision of your "project" directory: MyApp/ MyApp.egg-info/ # project egg-info SQLObject.egg/ # checkout of SQLObject's "project" directory pysqlite2-2.1-py2.4.egg/ # eggs installed with easy_install -d. Cheetah-1.9-py2.4.egg/ some_sqlobject_script # scripts installed by easy_install cheetah_compile_script pysqlite_shell_script.py myapp_script1.py # scripts, packages, and modules from my project myapp_script2.py myapp_package1/ myapp_somemodule.py setup.py # setup stuff setup.cfg So, a "project" directory always contains a .egg-info for the project, along with its setup.py and setup.cfg. Thus, if you want to develop multiple libraries in a single development project, you can just check out those libraries' project directories, giving them names with .egg extensions. (Note: right now the nested project's SQLObject.egg-info has to be renamed to EGG-INFO for metadata to work right, but I plan to fix this so that a simple checkout will suffice in future.) There are still a few wrinkles in this layout, mostly relating to scripts. If scripts all use require, and you use subproject scripts (like an SQLObject script in this scenario) by running the subproject's build_scripts+install_scripts targeting the main project's directory, then it all should work. However, if the subproject isn't designed for use in eggs, it won't work because the egg won't be on sys.path. So, there will really need to be a way to get easy_install to do a scripts-only install from a subproject directory. (And it's going to have to use different stub scripts, because the subproject's scripts won't be under its .egg-info directory.) The parent project's scripts don't have to require() the main project, since it's already on sys.path when they run. But, if they don't, then the script's requirements won't be activated. So that's a bit of a problem, although I suppose we could just say you have to use require('MyApp') or something similar in your scripts. The other problem with this approach is that it assumes an overall project directory layout that doesn't necessarily match what people use today. Zope and PEAK, for example, put their packages in a 'src' subdirectory under the "project directory", and that totally won't work with this approach unless there's a way to somehow configure the paths differently. It makes me think that maybe there needs to be a ".egg-link" file format that just contains a path to the a project's source root. Then, the layout might look like this instead: MyApp/ MyApp.egg-info/ # project egg-info SQLObject.project/ # checkout of SQLObject's "project" directory SQLObject.egg-link # pointer to SQLObject's package root pysqlite2-2.1-py2.4.egg/ # eggs installed with easy_install -d. Cheetah-1.9-py2.4.egg/ # ... etc. This is more complex to set up, but it should work for any existing distutils projects, and on systems that don't have symlinks. There are still enough wrinkles to make my head hurt, though. For example, the parent project still has to have a single-directory layout, unless we complicate things even more... MyApp/ bin/ # all scripts, eggs, links get installed here SQLObject/ # subprojects can get checked out here, though # and an .egg-link is put in bin setup.py # setup stuff has to live in the root setup.cfg src/ # but you can put stuff under src or lib if you want # or in this directory if you prefer Hm. I suppose that actually isn't any more complicated. However, whenever you add or remove scripts from the MyApp project, you'll have to run something to update your stub(s) in bin/. I'm also not sure 'bin' is the best name for the directory; it might already be used by some projects. We could call it 'run' or 'runtime' or something like that, though. So with this layout, you just 'easy_install -dbin' your non-development dependencies, and it'll even create the bin directory for you. You checkout subprojects, and then run some kind of setup command, e.g.: cd SQLObject setup.py dev_link -d../bin To create the needed .egg-link script stubs in bin/. In practice, you'd probably do some sort of easy_install or package.py command from the main directory, so that it can work with projects that don't use setuptools. Hm. Okay, that actually seems pretty reasonable, since it can now handle project-specific directory layouts with ease. The only fixed parts are that you need to select a 'bin' location that doesn't clash with the location of the source of your project's scripts. The downside is that '.egg-link' is purely a hypothetical concept at this point, and possibly introduces some new wrinkles into the egg installation process. However, experimenters could pretty much get the same effect right now by hand-creating a .egg symlink to the subproject source root, and making an EGG-INFO symlink in the source root pointing to Subproject.egg-info. That would work today, although a lot of the script pieces aren't fully ready for that configuration. Your thoughts? From pje at telecommunity.com Fri Jun 17 18:01:08 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 17 Jun 2005 12:01:08 -0400 Subject: [Distutils] EasyInstall: Issue: installing versioned modules by default, specifying version at runtime, lazy load highest by default. In-Reply-To: References: <5.1.1.6.0.20050616114457.0306fb10@mail.telecommunity.com> <5.1.1.6.0.20050615150342.01e00b28@mail.telecommunity.com> <42B110F7.3090800@colorstudy.com> <5.1.1.6.0.20050616114457.0306fb10@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050617113951.01e30468@home.unrestrained.com> At 01:35 PM 6/17/2005 +0100, Clayton Brown wrote: >Sometime ago, well at least before this modern marvel of eggs/easy >install etc, I wrote / adapted a bootstrap loader to enable module >versioning / specification at time of import, its default behaviour >was to load the highest version at the level of specification eg. > >_foo_version__ == "1.2.3" will load 1.2.3.99 etc where one exists >but no specification will import 99.99.99.99 etc. > >ref: http://claytonbrown.com/cbwiki/ModuleVersioning / >http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/285215 > >could the easy_install / import mechanism accomodate such installation >/ import hooks, (perhaps by just importing a module to invoke this >behaviour) so that by default eggs are all installed and versioned, >and if no version is specified by using require the highest version is >loaded, and when require() is used unless exact version is specified >it will import higher versions at the depth of specification should >they become available. Though by the looks of pkg_resources.require() >this may just be for system binary dependancies, not modules. > >my idea is that it would be nice to use -m on every installed package >to get versioned modules in site-packages, but not force require() >within scripts,just default to importing highest released version if >require() doesn't specifify a differnet version. When EasyInstall installs a script, it actually installs a stub that effectively require()s the coresponding egg, then runs the original script. So, you don't actually have to use require() at all, if you use EasyInstall to install some existing project's distribution. The exact version that supplied the scripts will be require()d for you automatically. require() is really only intended to bootstrap activation of the script's distribution; it isn't intended to be used throughout an application anyway. One very important reason why not, is that require() embedded in modules is not discoverable. There is no way to automatically find out what versions of things a package requires, and thereby automatically install those versions, if it is buried in code. This is why eggs have a depends.txt file where all the dependencies are centrally listed, allowing automated tools to deal with them. These dependencies are based on project names, not on individual modules or packages, because project names are discoverable (e.g. via PyPI) and package names are potentially far more numerous. So, although your approach has lots of merit, it doesn't accomplish the larger goal for Eggs, which is to support upgradeable applications and application servers, and to make it easy to add/remove Python packages from an application or Python installation. Versioned importing is just the tip of the iceberg for eggs, something that almost falls out as a side effect, rather than being a direct goal. Really, eggs more have the goal of *eliminating the need* for versioned importing, rather than implementing it! :) From rtomayko at gmail.com Mon Jun 20 05:32:35 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Sun, 19 Jun 2005 23:32:35 -0400 Subject: [Distutils] EasyInstall: Issue: wrapper script weirdness Message-ID: <4FC8B052-491C-4E9B-AF28-4EEF2596E443@gmail.com> I was able to package py.test (and the rest of py.lib) into an egg without much issue. I ran into a small problem with the py.test script, however, and the issue could effect other scripts as well. There is a main script, "py.test", that is very short:: from _Findlay import py py.test.cmdline.main() py.lib is packaged such that "_findpy.py" is installed as a script so it sits next to py.test under PKG-INFO/scripts and a wrapper is generated at [script-dir]/_findpy.py as well. _findpy just does some sys.path munging on import and then brings in the py module. For clarity, we have a set of files that look like this (on my machine): /usr/local/bin/py.test (setuptools wrapper) /usr/local/bin/_findpy.py (setuptools wrapper) py.test.egg/PKG-INFO/scripts/py.test (actual script) py.test.egg/PKG-INFO/scripts/_findpy.py (actual module) When I try to run the script using the setuptools generated wrapper I get:: ImportError: cannot import name py This is happening on the "from _findpy import py" line in PKG-INFO/ scripts/py.test. The text of ImportError looked more like a circular import problem than a "module not found" problem so I checked it out and the PKG-INFO/scripts/_findpy module is indeed being loaded and executed and it is finding the "py" package okay. What's happening is that '/usr/local/bin' comes before 'PKG-INFO/ scripts' on sys.path so that execution moves something like this: 1. `/usr/local/bin/py.test` 2. execfile `PKG-INFO/scripts/py.test` 3. import _findpy (found at `/usr/local/bin/_findpy.py`) 4. execfile `PKG-INFO/scripts/_findpy.py` Step three is bad. It would probably be best to make py.lib's scripts a little less complex (installing _findpy.py as a script doesn't strike me as an elegant approach). On the other hand, the way setuptools lays out sys.path when wrapper scripts are invoked results in unexpected behavior. If a script is importing a module it believes to be in the same directory, it is definitely not wanting to import the wrapper version. It may be best to remove the wrapper script directory from sys.path entirely or at least push it back behind the PKG-INFO/ scripts directory. I can't think of a single scenario where a wrapped script would want to import another wrapper. I haven't tried anything yet but will fire off a patch if I get in there to have a look. Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From rtomayko at gmail.com Mon Jun 20 06:03:28 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Mon, 20 Jun 2005 00:03:28 -0400 Subject: [Distutils] cracked eggs Message-ID: I've noticed that many packages that use __file__ and path operations instead of the pkg_resources / PEP 302 APIs to load package data usually work fine if you move the .egg file out of the way and then extract it to a directory with the same name. I have a little shell script ("crack_egg") that does this for me but I was wondering if this might be something that package authors could specify for their packages somehow, or maybe easy_install could take a command line arg or something? I guess it gets pretty messy. I remember reading somewhere that .egg files speed up import times over .egg directories because the zip index is quicker than the file system or something. But is it really that significant? Considering that a decent number of potential eggs are broken due to the resource API I wonder if it might be nice to have the option to prefer .egg directories for all packages installed by easy_install (e.g. by defaulting some option in ~/pydistutils.cfg). here's that script: $ cat $(which crack_egg) #!/bin/bash # usage: crack_egg egg-file [egg-file] ... for arg in $@ ; do mv $arg{,.zip} mkdir $arg unzip -d $arg $arg.zip done Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From pje at telecommunity.com Mon Jun 20 17:13:53 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 20 Jun 2005 11:13:53 -0400 Subject: [Distutils] cracked eggs In-Reply-To: Message-ID: <5.1.1.6.0.20050620110843.01e11a40@mail.telecommunity.com> At 12:03 AM 6/20/2005 -0400, Ryan Tomayko wrote: >I've noticed that many packages that use __file__ and path operations >instead of the pkg_resources / PEP 302 APIs to load package data >usually work fine if you move the .egg file out of the way and then >extract it to a directory with the same name. I have a little shell >script ("crack_egg") that does this for me but I was wondering if >this might be something that package authors could specify for their >packages somehow, or maybe easy_install could take a command line arg >or something? It does! The --zip-ok/-z flag. By default, EasyInstall unpacks eggs that it builds from source. However, it assumes that packages being *distributed* as eggs are "egg-ready". >I remember reading somewhere that .egg files speed up import times >over .egg directories because the zip index is quicker than the file >system or something. But is it really that significant? The issue is that it gets worse as you add more and more directories to sys.path. The zip index is quicker than the filesystem because it's loaded at most once, and kept in memory thereafter. So the disk access for a zipfile is O(1), but for a directory it's O(N). > Considering >that a decent number of potential eggs are broken due to the resource >API I wonder if it might be nice to have the option to prefer .egg >directories for all packages installed by easy_install (e.g. by >defaulting some option in ~/pydistutils.cfg). It's already the default for eggs built from source; you have to explicitly request installation as a zip file. Eventually, the option will default the other way, when most everything out there works as eggs anyway. From pje at telecommunity.com Mon Jun 20 17:20:40 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 20 Jun 2005 11:20:40 -0400 Subject: [Distutils] EasyInstall: Issue: wrapper script weirdness In-Reply-To: <4FC8B052-491C-4E9B-AF28-4EEF2596E443@gmail.com> Message-ID: <5.1.1.6.0.20050620111406.02f8eed0@mail.telecommunity.com> At 11:32 PM 6/19/2005 -0400, Ryan Tomayko wrote: >It would probably be best to make py.lib's scripts a little less >complex (installing _findpy.py as a script doesn't strike me as an >elegant approach). I think there's probably a trivial fix - add _findpy to the 'py_modules' list in setup.py, and remove it from 'scripts'. >On the other hand, the way setuptools lays out >sys.path when wrapper scripts are invoked results in unexpected >behavior. If a script is importing a module it believes to be in the >same directory, it is definitely not wanting to import the wrapper >version. It may be best to remove the wrapper script directory from >sys.path entirely or at least push it back behind the PKG-INFO/ scripts >directory. I can't think of a single scenario where a wrapped >script would want to import another wrapper. Well, it does occur to me that the part of EasyInstall that's broken here is the assumption that a wrapper will be run as __main__. What probably ought to happen is that if you do import something as a wrapper script, is that it ought to either: 1. work (by running the script in whatever module it is) 2. barf (and tell you what package is broken) 3. punt (by trying to replace itself with a module of the same name later on the path) Unfortunately I can think of scenarios where each of these is the "right thing" to do. #1 should work for py.test, but #3 makes more sense if the importee is not from the same package as the importer. My general inclination for now is to fix py.test and see if the issue comes up with anything else. It's easier to get a line on the right fix for something when you have more than one example to generalize from. :) From rtomayko at gmail.com Mon Jun 20 17:31:46 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Mon, 20 Jun 2005 11:31:46 -0400 Subject: [Distutils] cracked eggs In-Reply-To: <5.1.1.6.0.20050620110843.01e11a40@mail.telecommunity.com> References: <5.1.1.6.0.20050620110843.01e11a40@mail.telecommunity.com> Message-ID: <11BAAD7E-8B41-419A-9F40-0F3FFB4E083D@gmail.com> Gah. Color me a dolt. That all makes a ton of sense. Sorry... Ryan On Jun 20, 2005, at 11:13 AM, Phillip J. Eby wrote: > At 12:03 AM 6/20/2005 -0400, Ryan Tomayko wrote: > >> I've noticed that many packages that use __file__ and path operations >> instead of the pkg_resources / PEP 302 APIs to load package data >> usually work fine if you move the .egg file out of the way and then >> extract it to a directory with the same name. I have a little shell >> script ("crack_egg") that does this for me but I was wondering if >> this might be something that package authors could specify for their >> packages somehow, or maybe easy_install could take a command line arg >> or something? >> > > It does! The --zip-ok/-z flag. By default, EasyInstall unpacks > eggs that it builds from source. However, it assumes that packages > being *distributed* as eggs are "egg-ready". > > > >> I remember reading somewhere that .egg files speed up import times >> over .egg directories because the zip index is quicker than the file >> system or something. But is it really that significant? >> > > The issue is that it gets worse as you add more and more > directories to sys.path. The zip index is quicker than the > filesystem because it's loaded at most once, and kept in memory > thereafter. So the disk access for a zipfile is O(1), but for a > directory it's O(N). > > > >> Considering >> that a decent number of potential eggs are broken due to the resource >> API I wonder if it might be nice to have the option to prefer .egg >> directories for all packages installed by easy_install (e.g. by >> defaulting some option in ~/pydistutils.cfg). >> > > It's already the default for eggs built from source; you have to > explicitly request installation as a zip file. Eventually, the > option will default the other way, when most everything out there > works as eggs anyway. > > Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From rtomayko at gmail.com Mon Jun 20 18:35:33 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Mon, 20 Jun 2005 12:35:33 -0400 Subject: [Distutils] EasyInstall: Issue: wrapper script weirdness In-Reply-To: <5.1.1.6.0.20050620111406.02f8eed0@mail.telecommunity.com> References: <5.1.1.6.0.20050620111406.02f8eed0@mail.telecommunity.com> Message-ID: On Jun 20, 2005, at 11:20 AM, Phillip J. Eby wrote: > At 11:32 PM 6/19/2005 -0400, Ryan Tomayko wrote: >> It would probably be best to make py.lib's scripts a little less >> complex (installing _findpy.py as a script doesn't strike me as an >> elegant approach). > > I think there's probably a trivial fix - add _findpy to the > 'py_modules' list in setup.py, and remove it from 'scripts'. That does it. I'm not sure there's any down side to this approach so I'll propose it to the maintainers. >> On the other hand, the way setuptools lays out >> sys.path when wrapper scripts are invoked results in unexpected >> behavior. If a script is importing a module it believes to be in the >> same directory, it is definitely not wanting to import the wrapper >> version. It may be best to remove the wrapper script directory from >> sys.path entirely or at least push it back behind the PKG-INFO/ >> scripts directory. I can't think of a single scenario where a wrapped >> script would want to import another wrapper. > > Well, it does occur to me that the part of EasyInstall that's > broken here is the assumption that a wrapper will be run as > __main__. What probably ought to happen is that if you do import > something as a wrapper script, is that it ought to either: > > 1. work (by running the script in whatever module it is) > 2. barf (and tell you what package is broken) > 3. punt (by trying to replace itself with a module of the same name > later on the path) > > Unfortunately I can think of scenarios where each of these is the > "right thing" to do. #1 should work for py.test, but #3 makes more > sense if the importee is not from the same package as the importer. > > My general inclination for now is to fix py.test and see if the > issue comes up with anything else. It's easier to get a line on > the right fix for something when you have more than one example to > generalize from. :) Agreed. Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From mal at egenix.com Tue Jun 21 10:01:26 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 21 Jun 2005 10:01:26 +0200 Subject: [Distutils] Moving distutils minimum required version to 2.3 In-Reply-To: <200506101723.30704.anthony@interlink.com.au> References: <200506101723.30704.anthony@interlink.com.au> Message-ID: <42B7C956.4040801@egenix.com> Anthony Baxter wrote: > I've checked distutils into the python sandbox for a major > chunk of refactoring, including killing distutils.log and > distutils.fancy_getopt. I plan to upgrade the minimum > required version of distutils to 2.3 - I will be using > the logging and optparse modules from the stdlib. This > means that the version of distutils included in 2.4 will > be the last version guaranteed to work on 2.1 and 2.2 -- > but I seriously question whether anyone is ever going to > be making standalone distutils distributions again. That's not really the point: the reason we keep the compatibility to Python 2.1 is that package builders and users building the packages themselves will want to create builds for various Python versions, so that need a distutils version that works with mutiple recent Python versions. If you now jump to Python 2.3 as requirement, package authors will have to adapt their setup.py and associated distutils files to work with yet another variation of backend system. This is bound to break the simple install instructions of "python setup.py install" one way or another. It will also not be possible to use new developments in distutils for packages targetted for Python versions prior to Python 2.3 and - because distutils is normally extended via subclassing - make it harder to have setup.pys which work across multiple Python versions. I'd suggest to have distutils use the new tools from Python 2.3 if available, but still continue to ship its own versions for Python versions prior to 2.3. If you really want to bump the minimum required version for the complete package, then the next obvious choice would be Python 2.2 - in general, I think keeping backwards compatibility to the last 3 released Python versions is a good approach and gives enough time to the users, packagers and authors to adapt their software to new requirements. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 21 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From cmtaylor at ti.com Tue Jun 21 17:12:01 2005 From: cmtaylor at ti.com (Taylor, Martin) Date: Tue, 21 Jun 2005 10:12:01 -0500 Subject: [Distutils] Absolute destination path in data_files Message-ID: In the document "Distributing Python Modules" by Greg Ward, section "2.6 Installing Additional Files" has the following example: setup(... data_files=[('bitmaps', ['bm/b1.gif', 'bm/b2.gif']), ('config', ['cfg/data.cfg']), ('/etc/init.d', ['init-script'])] ) which I understand to be showing, in the last line, that a path proceeded by a "/" is an absolute destination path for the file names following it. When I try this in the following example: setup(name="FOO", description="FOO - A Generic Package", version=__version__, package_dir = {"FOO" : "Core", "FOO.EngA" : "EngA", "FOO.EngB" : "EngB"}, packages = ["FOO", "FOO.EngA", "FOO.EngB"], data_files=[('/FOO/FOO_BAR', ["UtilA\CMDTestClient.py", "UtilA\StopTestServer.py", "UtilA\TestClient.py", "UtilA\TestServer.py"] )], author="C. Martin Taylor", author_email="cmtaylor at ti.com " ) and then build it with the command: python setup.py bdist --format=wininst I get the following error messages: C:\FOO>python setup.py bdist --format=wininst running bdist running bdist_wininst running build running build_py installing to build\bdist.win32\wininst running install_lib creating build\bdist.win32\wininst creating build\bdist.win32\wininst\PURELIB creating build\bdist.win32\wininst\PURELIB\FOO creating build\bdist.win32\wininst\PURELIB\FOO\EngA copying build\lib\FOO\EngA\EngA.py -> build\bdist.win32\wininst\PURELIB\FOO\EngA copying build\lib\FOO\EngA\__init__.py -> build\bdist.win32\wininst\PURELIB\FOO\EngA creating build\bdist.win32\wininst\PURELIB\FOO\EngB copying build\lib\FOO\EngB\EngB.py -> build\bdist.win32\wininst\PURELIB\FOO\EngB copying build\lib\FOO\EngB\EngBConstants.py -> build\bdist.win32\wininst\PURELIB\FOO\EngB copying build\lib\FOO\EngB\__init__.py -> build\bdist.win32\wininst\PURELIB\FOO\EngB copying build\lib\FOO\FOO.py -> build\bdist.win32\wininst\PURELIB\FOO copying build\lib\FOO\FOOLog.py -> build\bdist.win32\wininst\PURELIB\FOO copying build\lib\FOO\FOO_CommonCode.py -> build\bdist.win32\wininst\PURELIB\FOO copying build\lib\FOO\FOO_UDF.py -> build\bdist.win32\wininst\PURELIB\FOO copying build\lib\FOO\FOO_Version.py -> build\bdist.win32\wininst\PURELIB\FOO copying build\lib\FOO\__init__.py -> build\bdist.win32\wininst\PURELIB\FOO running install_data creating build\bdist.win32\wininst\DATA Traceback (most recent call last): File "setup.py", line 49, in ? author_email="cmtaylor at ti.com " File "C:\Python24\lib\distutils\core.py", line 149, in setup dist.run_commands() File "C:\Python24\lib\distutils\dist.py", line 946, in run_commands self.run_command(cmd) File "C:\Python24\lib\distutils\dist.py", line 966, in run_command cmd_obj.run() File "C:\Python24\lib\distutils\command\bdist.py", line 146, in run self.run_command(cmd_name) File "C:\Python24\lib\distutils\cmd.py", line 333, in run_command self.distribution.run_command(command) File "C:\Python24\lib\distutils\dist.py", line 966, in run_command cmd_obj.run() File "C:\Python24\lib\distutils\command\bdist_wininst.py", line 152, in run install.run() File "C:\Python24\lib\distutils\command\install.py", line 505, in run self.run_command(cmd_name) File "C:\Python24\lib\distutils\cmd.py", line 333, in run_command self.distribution.run_command(command) File "C:\Python24\lib\distutils\dist.py", line 966, in run_command cmd_obj.run() File "C:\Python24\lib\distutils\command\install_data.py", line 62, in run dir = convert_path(f[0]) File "C:\Python24\lib\distutils\util.py", line 89, in convert_path raise ValueError, "path '%s' cannot be absolute" % pathname ValueError: path '/FOO/FOO_BAR' cannot be absolute Does anyone have an idea what is wrong here? Am I misinterpreting the document, or is there a bug in ActivePython 2.4? Thank you, C. Martin Taylor Sr. Test Automation Specialist Texas Instruments, Inc. Educational and Productivity Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20050621/4d670820/attachment.htm From pje at telecommunity.com Tue Jun 21 20:10:33 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 21 Jun 2005 14:10:33 -0400 Subject: [Distutils] Absolute destination path in data_files In-Reply-To: Message-ID: <5.1.1.6.0.20050621140552.01fb6780@mail.telecommunity.com> At 10:12 AM 6/21/2005 -0500, Taylor, Martin wrote: >ValueError: path '/FOO/FOO_BAR' cannot be absolute > >Does anyone have an idea what is wrong here? Am I misinterpreting the >document, or is there a bug in ActivePython 2.4? I think the documentation is wrong, and that you can't use absolute paths. If you want to install data with a package, the best way is to use the package_data argument instead of data_files. This will install the data inside the package's own directory. See the documentation for more details. By the way, judging from your setup.py, are you sure you don't want those 'UtilA' scripts to be installed as scripts instead? (using scripts=[...]) From pje at telecommunity.com Tue Jun 21 22:16:41 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 21 Jun 2005 16:16:41 -0400 Subject: [Distutils] Absolute destination path in data_files In-Reply-To: Message-ID: <5.1.1.6.0.20050621161400.01fa2138@mail.telecommunity.com> At 01:39 PM 6/21/2005 -0500, Taylor, Martin wrote: >You're probably right about using "scripts=", but I can't figure out its >syntax, either (the manual is VERY LIGHT on useful examples). If I >listed some scripts, I'd want to put them in a more package-specific >place than C:\Python24\Scripts. But I can't figure out if that's >possible. You can create and distribute a setup.cfg file with your package, containing something like: [install] install_scripts = /FOO/Scripts/Whatever And users can override it with --install-scripts or by editing setup.cfg This will have no effect on the bdist_wininst installer (or other wrappers like EasyInstall, bdist_rpm, etc.), only on packages installed using setup.py From anthony at interlink.com.au Wed Jun 22 15:52:44 2005 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed, 22 Jun 2005 23:52:44 +1000 Subject: [Distutils] Moving distutils minimum required version to 2.3 In-Reply-To: <42B7C956.4040801@egenix.com> References: <200506101723.30704.anthony@interlink.com.au> <42B7C956.4040801@egenix.com> Message-ID: <200506222352.47525.anthony@interlink.com.au> On Tuesday 21 June 2005 18:01, M.-A. Lemburg wrote: > That's not really the point: the reason we keep the compatibility > to Python 2.1 is that package builders and users building > the packages themselves will want to create builds for > various Python versions, so that need a distutils version > that works with mutiple recent Python versions. I'm not planning on _breaking_ compatibility at the language level, but to depend on stuff that's in the std lib in newer versions. Existing versions will still work fine, if you have the backported extensions - and in any case, someone could ship a version that bundled up textwrap, optparse and logging, and whatever else was needed with the newer code (since that's the main thing I'd be looking to change). The rationale for this change is this: Distutils has problems. Part of the reason nobody wants to touch it is that it's really quite complicated code - and part of the reason for _that_ is that it re-implements a huge amount of stuff that's now part of the std library. At some point, we either need to - Clean up the code so people can work on it without having their brain leak out the ears, or - Start an entirely new replacement for distutils (distutils2?) The former is a whole lot less work. "It's fine, and should remain as it is for ever and ever" is not a an outcome I'd like to see. > If you now jump to Python 2.3 as requirement, package > authors will have to adapt their setup.py and associated > distutils files to work with yet another variation of > backend system. This is bound to break the simple install > instructions of "python setup.py install" one way or > another. I don't see how this be the result. > It will also not be possible to use new developments in > distutils for packages targetted for Python versions prior > to Python 2.3 and - because distutils is normally extended > via subclassing - make it harder to have setup.pys which > work across multiple Python versions. Again, I'm _not_ planning to break code. I'm planning to make the code easier to work on -- so that hopefully, going forward, we can get people working on distutils to improve it, rather than everyone writing enormously complicated setup.py scripts that are just working around basic flaws in the current code. As an example: distutils' current option parsing code (fancy_getopt) behaves quite differently to both optparse and getopt. I don't see how this could be considered a good thing. > I'd suggest to have distutils use the new tools from Python > 2.3 if available, but still continue to ship its own versions > for Python versions prior to 2.3. And suddenly the distutils code is even _more_ complex than it is now! Making it use _either_ distutils.log or logging does not make the code cleaner, or easier to work with. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From pje at telecommunity.com Wed Jun 22 17:16:43 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 22 Jun 2005 11:16:43 -0400 Subject: [Distutils] Moving distutils minimum required version to 2.3 In-Reply-To: <200506222352.47525.anthony@interlink.com.au> References: <42B7C956.4040801@egenix.com> <200506101723.30704.anthony@interlink.com.au> <42B7C956.4040801@egenix.com> Message-ID: <5.1.1.6.0.20050622111314.01f93f28@mail.telecommunity.com> At 11:52 PM 6/22/2005 +1000, Anthony Baxter wrote: >As an example: distutils' current option parsing code (fancy_getopt) >behaves quite differently to both optparse and getopt. I don't see >how this could be considered a good thing. Well, if it changes how command classes have to express their metadata, it might be a bad thing to replace it. > > I'd suggest to have distutils use the new tools from Python > > 2.3 if available, but still continue to ship its own versions > > for Python versions prior to 2.3. > >And suddenly the distutils code is even _more_ complex than it >is now! Making it use _either_ distutils.log or logging does >not make the code cleaner, or easier to work with. I thought the plan was simply to make distutils.log use logging "under the hood". If that's the case, then none of the logging client code should change. So if somebody wants to use the old version of distutils.log, that's not a backward compatibility problem. What I worry about is that there are lots of packages out there that do crazy hacks to the distutils in order to accomplish things, and these hacks often rely on obscure properties of the current implementation. So, making it easier to work on (i.e. change) the distutils code base might actually be counterproductive. :) From rtomayko at gmail.com Thu Jun 23 21:25:40 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Thu, 23 Jun 2005 15:25:40 -0400 Subject: [Distutils] EasyInstall: scripts aren't executable? Message-ID: <95E69641-62F9-40C9-825F-FD7CD3B91E00@gmail.com> So EasyInstall wrapper scripts don't get any executable bits set when installed. Was this an intentional omission or just oversight? Note that you won't see the problem if the script already exists with +x when it is installed. diff -u -r1.22 easy_install.py --- easy_install.py 15 Jun 2005 02:23:47 -0000 1.22 +++ easy_install.py 23 Jun 2005 17:32:20 -0000 @@ -238,7 +238,8 @@ f = open(target,"w") f.write(script_text) f.close() - + from os import chmod + chmod(target, 0755) Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From pje at telecommunity.com Thu Jun 23 22:43:06 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 23 Jun 2005 16:43:06 -0400 Subject: [Distutils] EasyInstall: scripts aren't executable? In-Reply-To: <95E69641-62F9-40C9-825F-FD7CD3B91E00@gmail.com> Message-ID: <5.1.1.6.0.20050623164239.02ef8008@mail.telecommunity.com> At 03:25 PM 6/23/2005 -0400, Ryan Tomayko wrote: >So EasyInstall wrapper scripts don't get any executable bits set when >installed. Was this an intentional omission or just oversight? It's a bug; I'll fix it in 0.5a3. From pje at telecommunity.com Sat Jun 25 22:14:20 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 25 Jun 2005 16:14:20 -0400 Subject: [Distutils] Terminology for distributions, eggs, setuptools, etc. Message-ID: <5.1.1.6.0.20050616220458.01e144f0@mail.telecommunity.com> I'm trying to define some meaningful terms for talking about eggs and what they contain, as I prepare for some refactoring to make the egg runtime's class and attribute names be more consistent. Currently, the terminology driving those names has been kind of vague and handwavy; I'd like to get a little more precise. So, the following is an attempt at working out a coherent vocabulary, conceptual framework, and architectural overview of "Pluggable Distributions" (i.e., eggs). My plan is to refactor pkg_resources, setuptools.packge_index, and possibly some other parts of setuptools to match the more formal concept hierarchy laid out here. I'd appreciate any feedback, but it needs to be fairly soon as there should ideally be only one "great renaming", marking the transition of setuptools from 0.5 to 0.6, with a stable API for extenders going forward. Pluggable Distributions of Python Software ========================================== A "Distribution" is a collection of files that represent a "Release" of a "Project" as of a particular point in time, denoted by a "Version". Releases may have zero or more "Requirements", which indicate what releases of another project the release requires in order to function. A Requirement names the other project, expresses some criteria as to what releases of that project are acceptable, and lists any "Extras" that the requiring release may need from that project. (An Extra is an optional feature of a Release, that can only be used if its additional Requirements are satisfied.) Notice, by the way, that this definition of Distribution is broad enough to include directories containing Python packages or modules, not just "built distributions" created by the distutils. For example, the directory containing the Python standard library is a "distribution" by this definition, and so are the directories you edit your project's code in! In other words, every copy of a project's code is a "distribution", even if you don't take any special steps to make it one. Not all distributions are "importable", however. An "importable" distribution is one whose file or directory name can be referenced on sys.path, to allow importing modules or packages from that distribution. So, simple directories of code (and zipfiles with the correct internal layout) are "importable distributions", but most of the distributions built by the distutils (source archives and binary installers) are not. A "Project" is a library, framework, script, application, or collection of data or other files relevant to Python. "Projects" must have unique names, in order to tell them apart. Currently, PyPI is useful as a way of registering project names for uniqueness, because the 'name' argument to distutils 'setup()' command is used to identify the project on PyPI, as well as to generate Distributions' file names. A "Pluggable Distribution" or "Pluggable", is an importable distribution that satisfies these important additional properties: 1. Its project name and distribution format can be unambiguously determined from file or directory names, without actually examining any file contents. (Most distutils distribution formats cannot guarantee this, because they do not place any restrictions on project name strings, and thus allow ambiguity as to what part of their filenames is the project name, and what part is the version.) 2. A pluggable distribution contains metadata identifying its release's version, requirements, extras, and any additional requirements needed to implement those extras. It may also contain other metadata specific to an application or framework, to support integrating the pluggable's project with that application or framework. Distributions that satisfy these two properties are thus "pluggable", because they can be automatically discovered and "activated" (by adding them to sys.path), then used for importing Python modules or accessing other resource files and directories that are part of the distributed project. The Working Set --------------- The collection of distributions that are currently activated is called a Working Set. Note that a Working Set can contain any importable distribution, not just pluggable ones. For example, the Python standard library is an importable distribution that will usually be part of the Working Set, even though it is not pluggable. Similarly, when you are doing development work on a project, the files you are editing are also a Distribution. (And, with a little attention to the directory names used, and including some additional metadata, such a "development distribution" can be made pluggable as well.) When Python runs a program, that program must have all its requirements met by importable distributions in the working set. Initially, a Python program's Working Set consists only of the importable distributions (whether pluggable or not) listed in sys.path, such as the directory containing the program's __main__ script, and the directories containing the standard library and site-packages. If these are the only distributions that the program requires, then of course that program can run. However, if some of the requirements are not satisfied by the working set, this can lead to errors that may be hard to diagnose. So, if a Python program were made part of a Project, and the project explicitly defines its Requirements, which are then expressed as part of a Pluggable Distribution, then a runtime facility could automatically attempt to locate suitable pluggables and add them to the working set, or at least give a more specific error message if a requirement can't be satisfied. The Environment --------------- A set of directories that may be searched for pluggable distributions is called an Environment. By default, the Environment consists of all existing directories on sys.path, plus any distribution sources registered with the runtime. Given an Environment, and a Requirement to be satisfied, our proposed runtime facility would search the environment for pluggable distributions that satisfy the requirement (and the requirements of those distributions, recursively), such that it returns a list of distributions to be added to the working set, or raises a DependencyNotFound error. Note that a Working Set should not contain multiple distributions for the same project, so the runtime system must not propose to add a pluggable distribution to a Working Set if that set already contains a pluggable for the same project. If a project's requirements can't be met without adding a conflicting pluggable to the working set, a VersionConflict error is raised. (Unlike a working set, an Environment may contain more than one pluggable for a given project, because these are simply distributions that are *available* to be activated.) Python Eggs ----------- "Python Eggs" are distributions in specific formats that implement the concept of a "Pluggable Distribution". An egg may be a zipfile or directory whose name ends with '.egg', that contains Python modules or packages, plus an 'EGG-INFO' subdirectory containing metadata. An egg may also be a directory containing one or more 'ProjectName.egg-info' subdirectories with metadata. The latter form is primarily intended to add discoverability to distributions that -- for whatever reason -- cannot be restructured to the primary egg format. For example, by placing appropriate .egg-info directories in site-packages, one could document what distributions are already installed in that directory. While this would not make those releases capable of being individually activated, it does allow the runtime system to be aware that any requirements for those projects are already met, and to know that it should not attempt to add any other releases of those projects to the working set. The last form of egg is a '.egg-link' file. These exist to support symbolic linking on platforms that do not natively support symbolic links (e.g. Windows). These consist simply of a single line indicating the location of a directory that contains either an EGG-INFO or ProjectName.egg-info subdirectory. This format will be used by project management utilities to add an in-development distribution to the development Environment. Initialization, Development, and Deployment ------------------------------------------- Pluggable distributions can be manually made part of the working set by modifying sys.path. This can be done via PYTHONPATH, .pth files, or direct code manipulation. However, it is generally more useful to put distributions in the working set by automatically locating them in an appropriate Environment. The default Environment is the directories already on sys.path, so simply placing pluggable distributions in those directories suffices to make them available for adding to the working set. But *something* must add them to the working set, even if it is just to designate the project the current program is part of, so that its dependencies can be automatically resolved and added to the working set. This means that either a program's start scripts must invoke the runtime facility and make this initial request, or there must be some automatic means by which this is accomplished. The EasyInstall program accomplishes this by creating wrapper scripts when a distribution is installed. The wrapper scripts know what project the "real" script is part of, and so can ensure that the right working set is active when the scripts run. The scripts' author does not need to invoke the runtime facility directly, nor do they even need to be aware that it exists. For development, however, one does not generally want to have to "install" scripts that one is actively editing. So, future versions of the runtime facility will have an option to automatically create wrapper scripts that invoke the in-development versions of the scripts, rather than versions installed in eggs. This will allow developers to continue to write scripts without embedding any project or version information in them. Essentially, for development purposes, there will be a tool to "install" an in-development distribution to a given Environment, using a symlink or .egg-link file to include the distribution, and generating wrapper scripts to invoke any "main program" scripts in the project. Thus, a user's development Environment can include one or more projects whose source code he or she is editing, as well as any number of built distributions. He or she can then also build source or binary distributions of their project for deployment, whenever it is necessary or convenient to do so. From pje at telecommunity.com Sun Jun 26 00:04:54 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 25 Jun 2005 18:04:54 -0400 Subject: [Distutils] Installing dependencies Message-ID: <5.1.1.6.0.20050625171339.01fa5d98@mail.telecommunity.com> I don't quite know yet how EasyInstall should handle dependencies. I do know that I want it to use the metadata from a built egg to determine those dependencies, and that it should also include dependencies for any "extras" (see my last terminology post) that were requested on the EasyInstall command line. I also know that these dependencies can't really be handled by pretending they were included on the original command line, because that implies e.g. copying the package to the installation directory, and possibly munging .pth files. For example, if you already have one of the dependencies installed in --multi-version mode, but are installing a new package in single-version, you don't necessarily want to switch the existing package to single-version, do you? Then again, maybe you do. After all, if you're installing in single-version, you probably want stuff to "just work", and resetting the current version of the dependency would be the right thing in that case. EasyInstall should probably just be louder about the fact that it's changing the currently-active version. Maybe a post-installation report about what versions of relevant packages are active (w/"before and after" version information) would address this issue. Okay, what about the reverse scenario? You're installing --multi-version, but one of the dependencies is already single-version? That really needs to be left alone. And what if there's a conflicting requirement with an existing single-version install? Changing the active version might break something else, so we'd need to spit out a warning, explaining how to fix it (by changing the active version of the dependencies). Finally, EasyInstall currently always ensures there is a copy of the desired distribution in the installation directory. So, if you are installing somewhere other than site-packages, but your dependencies can be met using distributions in site-packages, should we still copy the distributions? I think it's always a safe default to do the copying, because disk space is cheap, and it makes the installation more independent of the Python installation and environment used to create it. However, I can also see that sometimes you might want to avoid this, but I'm not really sure what the option should be called. --no-copy, perhaps? (meaning, don't copy distributions found on sys.path to the --install-dir.) On a related note, it seems to me that setuptools can and maybe should change its 'install' command to actually use EasyInstall, such that running 'setup.py install' for a package using setuptools, does the same thing as it would have if you'd used EasyInstall to install it. This might force some people to use 'require()' who otherwise would not have, but I'm not sure how much of a problem that is in practice. Really, if Python 2.5 ends up supporting .pth files in any sys.path directory, the problem will go away because EasyInstall could avoid forcing --multi-version when a custom installation directory is used. Anyway, aside from the easy package management and other features of using easy_install as the 'install' command, it would also enable the traditional 'setup.py install' to include the distributed package's dependencies as well. This means that we could finally make it easy for people to distribute packages with external dependencies, because the only thing they'll have to include is the 'ez_setup.py' bootstrap script alongside their setup.py. The principal problem I see with this is that some distutils commands internally access the 'install' command in order to get information or to perform a pseudo-installation for creating binary distributions. So, there might be some interesting technical hurdles in order to make it such that running 'install' from the command line does one thing, but 'install' called internally from a 'bdist' command does something else. I'll have to investigate that a bit. It would also be nice if you could specify your dependencies as part of the 'setup()' script, but it looks like some people are trying to use the 'requires' keyword for PEP 314's non-semantic metadata. Personally I think PEP 314 is going in the wrong direction at the moment, because it doesn't specify any semantics for the metadata. Having more fields for people to fill out with data we can't use isn't very helpful. And, at the same time, the PEP 314 spec is too strict about how versions are defined, and doesn't support "extra" (optional) requirements. I should probably make a concrete proposal for revising PEP 314 to: 1. Define semantics for "Requires" strings (PyPI project names) 2. Suggest removing "Provides", in favor of distributing only a single "provided" thing per distribution at the present time. 3. Suggest renaming "Obsoletes" to "Conflicts" 4. Define a package version syntax based on setuptools' rules 5. Allow Download-URL to be a link to an HTML page containing direct links to downloadable distributions, so that PyPI and Sourceforge download pages are acceptable (note that PEP 314 doesn't currently let you publish links to binary distributions, only source). What I'd like to do with setuptools is allow you to use keywords like the following, in order to define your distribution's required dependencies and any optional extra features that incur additional dependencies: setup( ... requires = ["SomePackage>=1.2", "OtherPackage"], extras = dict( reST = ["docutils>=0.3.5", "restEdit>=0.3"], XML = ["xmlplus"], ... ) ... ) But that won't work too well if the 'requires' conflicts with what PEP 314 is after, and the current Python CVS trunk implements. From pje at telecommunity.com Mon Jun 27 05:36:51 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 26 Jun 2005 23:36:51 -0400 Subject: [Distutils] PyPI upload doesn't support eggs? Message-ID: <5.1.1.6.0.20050626233332.020f7408@mail.telecommunity.com> I can't seem to convince the "upload" command to send egg distributions to PyPI. I'm using the CVS head version of "upload", and modified bdist_egg to add the necessary information to distribution.dist_files, but attempting to upload gives this response (using --show-response): Submitting dist\setuptools-0.5a4-py2.3.egg to http://www.python.org/pypi Upload failed (500): There's been a problem with your request --------------------------------------------------------------------------- Erro r... There's been a problem with your request exceptions.ValueError: invalid distribution file ------------------------------- -------------------------------------------- Any suggestions? From pje at telecommunity.com Mon Jun 27 06:05:02 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 27 Jun 2005 00:05:02 -0400 Subject: [Distutils] PyPI upload doesn't support eggs? In-Reply-To: <5.1.1.6.0.20050626233332.020f7408@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050626235632.020fbe60@mail.telecommunity.com> I just tried doing the upload via the web interface, and got the same error. It appears that PyPI doesn't support uploading eggs, which is surprising because I could've sworn that the sprinters at PyCon said they were putting eggs on the list of allowed filetypes. But I just found the appropriate PyPI source code, and '.egg' isn't on the list of allowed file extensions, so I'll go ahead and file an RFE for this. At 11:36 PM 6/26/2005 -0400, Phillip J. Eby wrote: >I can't seem to convince the "upload" command to send egg distributions to >PyPI. I'm using the CVS head version of "upload", and modified bdist_egg >to add the necessary information to distribution.dist_files, but attempting >to upload gives this response (using --show-response): > >Submitting dist\setuptools-0.5a4-py2.3.egg to http://www.python.org/pypi >Upload failed (500): There's been a problem with your request >--------------------------------------------------------------------------- >Erro >r... > >There's been a problem with your request > >exceptions.ValueError: invalid distribution file >------------------------------- >-------------------------------------------- > >Any suggestions? From rtomayko at gmail.com Mon Jun 27 10:30:05 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Mon, 27 Jun 2005 04:30:05 -0400 Subject: [Distutils] .egg media type? (was: PyPI upload doesn't support eggs?) In-Reply-To: <5.1.1.6.0.20050626235632.020fbe60@mail.telecommunity.com> References: <5.1.1.6.0.20050626235632.020fbe60@mail.telecommunity.com> Message-ID: <273A8ED8-A991-44EE-802B-7EEDCBD95A9E@gmail.com> On Jun 27, 2005, at 12:05 AM, Phillip J. Eby wrote: > I just tried doing the upload via the web interface, and got the same > error. It appears that PyPI doesn't support uploading eggs, which is > surprising because I could've sworn that the sprinters at PyCon > said they > were putting eggs on the list of allowed filetypes. But I just > found the > appropriate PyPI source code, and '.egg' isn't on the list of > allowed file > extensions, so I'll go ahead and file an RFE for this. On an almost related note, what should the MIME content/media type be for serving egg files via HTTP? It may be advantageous to use "application/zip" [1]. A derivative like "application/egg+zip" would allow the media type to be associated with easy_install or other handlers while preserving the base media type information (although I'm not completely clear on whether derivatives require registration). [1] http://www.iana.org/assignments/media-types/application/zip Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From fdrake at gmail.com Mon Jun 27 15:55:04 2005 From: fdrake at gmail.com (Fred Drake) Date: Mon, 27 Jun 2005 09:55:04 -0400 Subject: [Distutils] [Catalog-sig] .egg media type? (was: PyPI upload doesn't support eggs?) In-Reply-To: <273A8ED8-A991-44EE-802B-7EEDCBD95A9E@gmail.com> References: <5.1.1.6.0.20050626235632.020fbe60@mail.telecommunity.com> <273A8ED8-A991-44EE-802B-7EEDCBD95A9E@gmail.com> Message-ID: <9cee7ab80506270655231e466f@mail.gmail.com> On 6/27/05, Ryan Tomayko wrote: > On an almost related note, what should the MIME content/media type be > for serving egg files via HTTP? It may be advantageous to use > "application/zip" [1]. Perhaps. > A derivative like "application/egg+zip" would allow the media type to > be associated with easy_install or other handlers while preserving > the base media type information (although I'm not completely clear on > whether derivatives require registration). "application/egg+zip" would require registration. I've not seen the "+foo" annotation style used for anything other than "+xml" yet, but this use makes sense to me. -Fred -- Fred L. Drake, Jr. Zope Corporation From bob at redivi.com Mon Jun 27 16:32:48 2005 From: bob at redivi.com (Bob Ippolito) Date: Mon, 27 Jun 2005 10:32:48 -0400 Subject: [Distutils] [Catalog-sig] .egg media type? (was: PyPI upload doesn't support eggs?) In-Reply-To: <9cee7ab80506270655231e466f@mail.gmail.com> References: <5.1.1.6.0.20050626235632.020fbe60@mail.telecommunity.com> <273A8ED8-A991-44EE-802B-7EEDCBD95A9E@gmail.com> <9cee7ab80506270655231e466f@mail.gmail.com> Message-ID: <51DD7AFE-007A-4E1D-9BAE-55B7C5B547D2@redivi.com> On Jun 27, 2005, at 9:55 AM, Fred Drake wrote: > On 6/27/05, Ryan Tomayko wrote: > >> On an almost related note, what should the MIME content/media type be >> for serving egg files via HTTP? It may be advantageous to use >> "application/zip" [1]. >> > > Perhaps. -1. Eggs should be opaque to software that doesn't necessarily know what an egg is. For a point of reference, Java's jar format is 'application/java-archive'. -bob From rtomayko at gmail.com Mon Jun 27 16:33:53 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Mon, 27 Jun 2005 10:33:53 -0400 Subject: [Distutils] [Catalog-sig] .egg media type? (was: PyPI upload doesn't support eggs?) In-Reply-To: <9cee7ab80506270655231e466f@mail.gmail.com> References: <5.1.1.6.0.20050626235632.020fbe60@mail.telecommunity.com> <273A8ED8-A991-44EE-802B-7EEDCBD95A9E@gmail.com> <9cee7ab80506270655231e466f@mail.gmail.com> Message-ID: <0BEC27EF-7197-4BD2-B2C6-F99C9D41E150@gmail.com> On Jun 27, 2005, at 9:55 AM, Fred Drake wrote: >> A derivative like "application/egg+zip" would allow the media type to >> be associated with easy_install or other handlers while preserving >> the base media type information (although I'm not completely clear on >> whether derivatives require registration). >> > > "application/egg+zip" would require registration. I've not seen the > "+foo" annotation style used for anything other than "+xml" yet, but > this use makes sense to me. Same here. I'm still trying to dig it up but I think I remember seeing the "+foo" annotation explicitly specified in the normative application/xml media type specification. But I'm not seeing it so I could be wrong. Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From rtomayko at gmail.com Mon Jun 27 16:51:52 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Mon, 27 Jun 2005 10:51:52 -0400 Subject: [Distutils] [Catalog-sig] .egg media type? (was: PyPI upload doesn't support eggs?) In-Reply-To: <51DD7AFE-007A-4E1D-9BAE-55B7C5B547D2@redivi.com> References: <5.1.1.6.0.20050626235632.020fbe60@mail.telecommunity.com> <273A8ED8-A991-44EE-802B-7EEDCBD95A9E@gmail.com> <9cee7ab80506270655231e466f@mail.gmail.com> <51DD7AFE-007A-4E1D-9BAE-55B7C5B547D2@redivi.com> Message-ID: <82E0D872-53C5-4971-B4F9-F13386D24DEC@gmail.com> On Jun 27, 2005, at 10:32 AM, Bob Ippolito wrote: > On Jun 27, 2005, at 9:55 AM, Fred Drake wrote: >> On 6/27/05, Ryan Tomayko wrote: >>> On an almost related note, what should the MIME content/media >>> type be >>> for serving egg files via HTTP? It may be advantageous to use >>> "application/zip" [1]. >> >> Perhaps. > -1. Eggs should be opaque to software that doesn't necessarily > know what an egg is. Yea, "application/zip" would be bad as it would clash with zip handling applications, making it very unlikely that an egg handling application could be provided as the default handler. But what about "application/egg+zip"? If the "+foo" annotation is doable, it would allow egg handling applications to be registered but also allow fallback to zip handling applications when there is no egg handling application available. I don't see the downside. > For a point of reference, Java's jar format is 'application/java- > archive'. I apologize for being cynical but, given Java's history of disregarding/butchering web and internet architecture, I'd be surprised if that was put in place by anyone having experience with internet media types. Media types are like view source/HTML, it's easy to miss the huge body of spec text dedicated to their behavior. I really don't know the history of "application/java-archive" though, so take that as an uninformed, biased, and probably a little unfair jab. Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From rtomayko at gmail.com Mon Jun 27 19:17:18 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Mon, 27 Jun 2005 13:17:18 -0400 Subject: [Distutils] [Catalog-sig] .egg media type? (was: PyPI upload doesn't support eggs?) In-Reply-To: <0BEC27EF-7197-4BD2-B2C6-F99C9D41E150@gmail.com> References: <5.1.1.6.0.20050626235632.020fbe60@mail.telecommunity.com> <273A8ED8-A991-44EE-802B-7EEDCBD95A9E@gmail.com> <9cee7ab80506270655231e466f@mail.gmail.com> <0BEC27EF-7197-4BD2-B2C6-F99C9D41E150@gmail.com> Message-ID: <6320650D-DD11-4A2E-8042-E471A54A15DA@gmail.com> On Jun 27, 2005, at 10:33 AM, Ryan Tomayko wrote: > On Jun 27, 2005, at 9:55 AM, Fred Drake wrote: >> "application/egg+zip" would require registration. I've not seen the >> "+foo" annotation style used for anything other than "+xml" yet, but >> this use makes sense to me. > > Same here. I'm still trying to dig it up but I think I remember > seeing the "+foo" annotation explicitly specified in the normative > application/xml media type specification. But I'm not seeing it so > I could be wrong. The "+xml" suffix is the only suffixing scheme at IANA and is normatively defined in the RFC 3023 XML Media Types [1]. The section linked to below contains justification for inventing the suffixing scheme, which is almost as interesting as it is amusing. :) [1] Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From fdrake at gmail.com Mon Jun 27 20:04:17 2005 From: fdrake at gmail.com (Fred Drake) Date: Mon, 27 Jun 2005 14:04:17 -0400 Subject: [Distutils] [Catalog-sig] .egg media type? (was: PyPI upload doesn't support eggs?) In-Reply-To: <6320650D-DD11-4A2E-8042-E471A54A15DA@gmail.com> References: <5.1.1.6.0.20050626235632.020fbe60@mail.telecommunity.com> <273A8ED8-A991-44EE-802B-7EEDCBD95A9E@gmail.com> <9cee7ab80506270655231e466f@mail.gmail.com> <0BEC27EF-7197-4BD2-B2C6-F99C9D41E150@gmail.com> <6320650D-DD11-4A2E-8042-E471A54A15DA@gmail.com> Message-ID: <9cee7ab805062711042967c1f3@mail.gmail.com> On 6/27/05, Ryan Tomayko wrote: > The "+xml" suffix is the only suffixing scheme at IANA and is > normatively defined in the RFC 3023 XML Media Types [1]. The section > linked to below contains justification for inventing the suffixing > scheme, which is almost as interesting as it is amusing. :) The "XML is the first generic data format that has seemed to justify special treatment" (A.14) in certainly humorous, especially given the context of this discussion. I'm firmly in the "application/egg+zip" camp at this point. -Fred -- Fred L. Drake, Jr. Zope Corporation From martin at v.loewis.de Mon Jun 27 23:05:12 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 27 Jun 2005 23:05:12 +0200 Subject: [Distutils] [Catalog-sig] PyPI upload doesn't support eggs? In-Reply-To: <5.1.1.6.0.20050626235632.020fbe60@mail.telecommunity.com> References: <5.1.1.6.0.20050626235632.020fbe60@mail.telecommunity.com> Message-ID: <42C06A08.1010504@v.loewis.de> Phillip J. Eby wrote: > I just tried doing the upload via the web interface, and got the same > error. It appears that PyPI doesn't support uploading eggs, which is > surprising because I could've sworn that the sprinters at PyCon said they > were putting eggs on the list of allowed filetypes. I'm not sure whom specifically you are referring to - I personally don't recall saying such a thing. Regards, Martin From pje at telecommunity.com Mon Jun 27 23:16:20 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 27 Jun 2005 17:16:20 -0400 Subject: [Distutils] [Catalog-sig] PyPI upload doesn't support eggs? In-Reply-To: <42C06A08.1010504@v.loewis.de> References: <5.1.1.6.0.20050626235632.020fbe60@mail.telecommunity.com> <5.1.1.6.0.20050626235632.020fbe60@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050627171310.036590e0@mail.telecommunity.com> At 11:05 PM 6/27/2005 +0200, Martin v. L?wis wrote: >Phillip J. Eby wrote: > > I just tried doing the upload via the web interface, and got the same > > error. It appears that PyPI doesn't support uploading eggs, which is > > surprising because I could've sworn that the sprinters at PyCon said they > > were putting eggs on the list of allowed filetypes. > >I'm not sure whom specifically you are referring to - I personally >don't recall saying such a thing. I think it was Richard Jones who said he was including .egg on the list. I gave him some additional info about the format (e.g. that eggs include an EGG-INFO/PKG-INFO file), but he said he didn't think it was worth bothering with checking beyond the filename for .egg. (I mention this only to highlight the point that we did talk for a minute or so on the subject, not to get into the subject of the .egg format's internal layout.) Anyway, I got the impression from what was said that he was actually literally implementing the filetype check as we spoke about it, which is why I was surprised to find it's not actually in there. From mathieu.gontier at fft.be Thu Jun 30 10:49:29 2005 From: mathieu.gontier at fft.be (mathieu gontier) Date: Thu, 30 Jun 2005 10:49:29 +0200 Subject: [Distutils] Can distutils manage static libraries and fortran compilers ? Message-ID: <42C3B219.4010402@fft.be> Hello I work on an finite element framework and Python bindings have been developped. Actually, we use Boost.Build as build system but we would like to change it. We have two kinds of problems. First, the framework is used for generate pure C++ applications with static libraries ; second, the framework have some dependencies to fortran projects. Then, in order to know if Distutils can be use in our case, I would like to know if : - Can distutil compile static C/C++ libraries ? - Can distutil compile C/C++ applications ? - Can distutil manage fortran compiler and compile fortran libraries and applications ? Thank for your answers, -------------- next part -------------- A non-text attachment was scrubbed... Name: mathieu.gontier.vcf Type: text/x-vcard Size: 321 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20050630/adecbea8/mathieu.gontier.vcf From berthold at despammed.com Thu Jun 30 12:32:06 2005 From: berthold at despammed.com (=?iso-8859-15?q?Berthold_H=F6llmann?=) Date: Thu, 30 Jun 2005 12:32:06 +0200 Subject: [Distutils] Can distutils manage static libraries and fortran compilers ? References: <42C3B219.4010402@fft.be> Message-ID: mathieu gontier writes: > Hello > > I work on an finite element framework and Python bindings have been > developped. > Actually, we use Boost.Build as build system but we would like to > change it. > > We have two kinds of problems. First, the framework is used for > generate pure C++ applications with static libraries ; second, the > framework have some dependencies to fortran projects. > Then, in order to know if Distutils can be use in our case, I would > like to know if : > - Can distutil compile static C/C++ libraries ? Yes > - Can distutil compile C/C++ applications ? Yes, with some extra code in the setup file. > - Can distutil manage fortran compiler and compile fortran libraries > and applications ? Yes, in collaboration with scipy_distutils -- __ Address: G / \ L Germanischer Lloyd phone: +49-40-36149-7374 -+----+- Vorsetzen 35 P.O.Box 111606 fax : +49-40-36149-7320 \__/ D-20459 Hamburg D-20416 Hamburg From barry at python.org Thu Jun 30 15:16:03 2005 From: barry at python.org (Barry Warsaw) Date: Thu, 30 Jun 2005 09:16:03 -0400 Subject: [Distutils] --build-base and install Message-ID: <1120137363.9638.54.camel@geddy.wooz.org> I'm trying to build and install pysqlite-1.1.6 using --build-base, because I want the files created during the build to be placed in a specific directory outside the source directory. --build-base works just fine with the 'build' command, but not at all with the 'install' command. How is 'install' supposed to find the files that 'build' placed elsewhere? Is the lack of --build-base support a problem with pysqlite's setup.py script or a more fundamental problem with distutils? Or am I just being dense? FTR: Python 2.4.1 Thanks, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/distutils-sig/attachments/20050630/f3829d6a/attachment.pgp From pbagora at yahoo.com Thu Jun 30 15:59:04 2005 From: pbagora at yahoo.com (Pranav Bagora) Date: Thu, 30 Jun 2005 06:59:04 -0700 (PDT) Subject: [Distutils] Question about checking version Message-ID: <20050630135904.61146.qmail@web32415.mail.mud.yahoo.com> Hello, i am a newbie using distutils... i m trying to make a installer for windows which will compile few python files. What i want to know is that if i am creating a newer version of the installer how do i check for the previous versions installed and also install newer versions in a separate directory. Thanks Pranav Bagora Graduate Student SUNY Binghamton. ================ Add:- 261 Main Street Apt No. 11 Binghamton NY. ================ Phone:- 607 349 8788 email:- pbagora at binghamton.edu ____________________________________________________ Yahoo! Sports Rekindle the Rivalries. Sign up for Fantasy Football http://football.fantasysports.yahoo.com From p.f.moore at gmail.com Thu Jun 30 16:01:17 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 30 Jun 2005 15:01:17 +0100 Subject: [Distutils] --build-base and install In-Reply-To: <1120137363.9638.54.camel@geddy.wooz.org> References: <1120137363.9638.54.camel@geddy.wooz.org> Message-ID: <79990c6b0506300701107e2adc@mail.gmail.com> On 6/30/05, Barry Warsaw wrote: > I'm trying to build and install pysqlite-1.1.6 using --build-base, > because I want the files created during the build to be placed in a > specific directory outside the source directory. > > --build-base works just fine with the 'build' command, but not at all > with the 'install' command. How is 'install' supposed to find the files > that 'build' placed elsewhere? Is the lack of --build-base support a > problem with pysqlite's setup.py script or a more fundamental problem > with distutils? Or am I just being dense? > > FTR: Python 2.4.1 Have you tried python setup.py build --build-base=whatever install ? I chain the commands like this to switch compilers on Windows (build --compiler=mingw32 bdist_wininst) and it works fine for that. Paul. From barry at python.org Thu Jun 30 16:29:20 2005 From: barry at python.org (Barry Warsaw) Date: Thu, 30 Jun 2005 10:29:20 -0400 Subject: [Distutils] --build-base and install In-Reply-To: <79990c6b0506300701107e2adc@mail.gmail.com> References: <1120137363.9638.54.camel@geddy.wooz.org> <79990c6b0506300701107e2adc@mail.gmail.com> Message-ID: <1120141760.9636.63.camel@geddy.wooz.org> On Thu, 2005-06-30 at 10:01, Paul Moore wrote: > Have you tried > > python setup.py build --build-base=whatever install > > ? > > I chain the commands like this to switch compilers on Windows (build > --compiler=mingw32 bdist_wininst) and it works fine for that. Thanks Paul, that does work perfectly. I didn't realize you could chain commands like that! -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/distutils-sig/attachments/20050630/3c8bc2c8/attachment.pgp From theller at python.net Thu Jun 30 16:39:11 2005 From: theller at python.net (Thomas Heller) Date: Thu, 30 Jun 2005 16:39:11 +0200 Subject: [Distutils] --build-base and install References: <1120137363.9638.54.camel@geddy.wooz.org> <79990c6b0506300701107e2adc@mail.gmail.com> <1120141760.9636.63.camel@geddy.wooz.org> Message-ID: <64vvdi40.fsf@python.net> Barry Warsaw writes: > On Thu, 2005-06-30 at 10:01, Paul Moore wrote: > >> Have you tried >> >> python setup.py build --build-base=whatever install >> >> ? >> >> I chain the commands like this to switch compilers on Windows (build >> --compiler=mingw32 bdist_wininst) and it works fine for that. > > Thanks Paul, that does work perfectly. I didn't realize you could chain > commands like that! Another way is to write a setup.cfg file - that way you could not forget to specify the flags all the time. Thomas From mal at egenix.com Thu Jun 30 16:49:23 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 30 Jun 2005 16:49:23 +0200 Subject: [Distutils] --build-base and install In-Reply-To: <1120141760.9636.63.camel@geddy.wooz.org> References: <1120137363.9638.54.camel@geddy.wooz.org> <79990c6b0506300701107e2adc@mail.gmail.com> <1120141760.9636.63.camel@geddy.wooz.org> Message-ID: <42C40673.3080804@egenix.com> Barry Warsaw wrote: > On Thu, 2005-06-30 at 10:01, Paul Moore wrote: > > >>Have you tried >> >> python setup.py build --build-base=whatever install >> >>? >> >>I chain the commands like this to switch compilers on Windows (build >>--compiler=mingw32 bdist_wininst) and it works fine for that. > > > Thanks Paul, that does work perfectly. I didn't realize you could chain > commands like that! Aside: you can also put these options into the setup.cfg file, so that you don't have to retype them every time. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 30 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From pje at telecommunity.com Thu Jun 30 17:13:00 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 30 Jun 2005 11:13:00 -0400 Subject: [Distutils] --build-base and install In-Reply-To: <42C40673.3080804@egenix.com> References: <1120141760.9636.63.camel@geddy.wooz.org> <1120137363.9638.54.camel@geddy.wooz.org> <79990c6b0506300701107e2adc@mail.gmail.com> <1120141760.9636.63.camel@geddy.wooz.org> Message-ID: <5.1.1.6.0.20050630111113.02539cd8@mail.telecommunity.com> At 04:49 PM 6/30/2005 +0200, M.-A. Lemburg wrote: >Barry Warsaw wrote: > > On Thu, 2005-06-30 at 10:01, Paul Moore wrote: > > > > > >>Have you tried > >> > >> python setup.py build --build-base=whatever install > >> > >>? > >> > >>I chain the commands like this to switch compilers on Windows (build > >>--compiler=mingw32 bdist_wininst) and it works fine for that. > > > > > > Thanks Paul, that does work perfectly. I didn't realize you could chain > > commands like that! > >Aside: you can also put these options into the setup.cfg file, >so that you don't have to retype them every time. Not only that, if you want to always select a particular compiler, you can put it in your $HOME/pydistutils.cfg (Windows) or ~/.pydistutils.cfg (Unixes), or in a distutils.cfg file in the package directory for distutils (any platform). I do this for any new Python install on Windows, and never have to use the --compiler or -c option. From ianb at colorstudy.com Thu Jun 30 19:51:56 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 30 Jun 2005 12:51:56 -0500 Subject: [Distutils] [Catalog-sig] .egg media type? In-Reply-To: <9cee7ab805062711042967c1f3@mail.gmail.com> References: <5.1.1.6.0.20050626235632.020fbe60@mail.telecommunity.com> <273A8ED8-A991-44EE-802B-7EEDCBD95A9E@gmail.com> <9cee7ab80506270655231e466f@mail.gmail.com> <0BEC27EF-7197-4BD2-B2C6-F99C9D41E150@gmail.com> <6320650D-DD11-4A2E-8042-E471A54A15DA@gmail.com> <9cee7ab805062711042967c1f3@mail.gmail.com> Message-ID: <42C4313C.5040102@colorstudy.com> Fred Drake wrote: > On 6/27/05, Ryan Tomayko wrote: > >>The "+xml" suffix is the only suffixing scheme at IANA and is >>normatively defined in the RFC 3023 XML Media Types [1]. The section >>linked to below contains justification for inventing the suffixing >>scheme, which is almost as interesting as it is amusing. :) > > > The "XML is the first generic data format that has seemed to justify > special treatment" (A.14) in certainly humorous, especially given the > context of this discussion. > > I'm firmly in the "application/egg+zip" camp at this point. But shouldn't that be application/x-egg+zip ? -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From fdrake at gmail.com Thu Jun 30 21:51:25 2005 From: fdrake at gmail.com (Fred Drake) Date: Thu, 30 Jun 2005 15:51:25 -0400 Subject: [Distutils] [Catalog-sig] .egg media type? In-Reply-To: <42C4313C.5040102@colorstudy.com> References: <5.1.1.6.0.20050626235632.020fbe60@mail.telecommunity.com> <273A8ED8-A991-44EE-802B-7EEDCBD95A9E@gmail.com> <9cee7ab80506270655231e466f@mail.gmail.com> <0BEC27EF-7197-4BD2-B2C6-F99C9D41E150@gmail.com> <6320650D-DD11-4A2E-8042-E471A54A15DA@gmail.com> <9cee7ab805062711042967c1f3@mail.gmail.com> <42C4313C.5040102@colorstudy.com> Message-ID: <9cee7ab80506301251296daa6b@mail.gmail.com> On 6/30/05, Ian Bicking wrote: > But shouldn't that be application/x-egg+zip ? Not if it's registered. -Fred -- Fred L. Drake, Jr. Zope Corporation