From ianb at colorstudy.com Mon Jul 4 01:01:09 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 03 Jul 2005 18:01:09 -0500 Subject: [Distutils] easy_install: setup.py-less packages Message-ID: <42C86E35.1070701@colorstudy.com> Hi; trying to get back into this again... So, there's a package at http://svn.saddi.com/flup/trunk/ that I want to install, but it has no setup.py file. I think that file should look something like: from setuptools import setup setup(name="flup", packages=['flup', 'flup.middleware', 'flup.resolver', 'flup.server'] version='0.0-devel-r???') And I'd probably add other things, but I think that's all that's required. So... how should I make this happen? Should I hardcode 'http://svn.saddi.com/flup/trunk' into the setup.py file? Can I make the location overrideable with an option? How do I download the checkout? What was in a function before is now in setuptools.package_index.PackageIndex, and involves all sorts of data that doesn't seem to imply (because it's not a package without the setup.py file, and there's no index to get it from). Should I calculate r??? on my own, and if so where? Right before I call setup()? I need the repository location to do so, so if the repository location was overrideable then I'd need to get the real location. Once the files are checkout out, how do I put it in place? Is there a way to set the "base path" for the packages, so I could say "install these packages, found in /tmp/wherever-it-was-downloaded-to"? -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Mon Jul 4 01:01:19 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 03 Jul 2005 18:01:19 -0500 Subject: [Distutils] easy_install: package index Message-ID: <42C86E3F.1000100@colorstudy.com> I notice there is an option for the index URL. But it doesn't look like a multi-option, i.e., you can't give several URLs, to be checked in turn. That would be useful. It'd also be nice if you could configure this; for instance, there's several libraries I'm using that haven't had proper releases yet, so they aren't in PyPI. If I could tell someone to just add a particular line to ~/.setup.cfg or somesuch, then that would make it easier. Maybe that's already possible? But that'd be no good if that disabled the use of PyPI. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Mon Jul 4 03:07:34 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 03 Jul 2005 20:07:34 -0500 Subject: [Distutils] eggs: Python version independence Message-ID: <42C88BD6.4000507@colorstudy.com> Most (90%+) of the libraries I work with don't depend on a Python version. But unfortunately when using easy_install/eggs they always have an explicit version, and multiple installations are necessary otherwise. Generally speaking, all pure-Python packages are version independent. At least on the source level. I suppose .pyc files aren't always (though they haven't changed for a long time, have they?) -- would it be reasonable to have version-independent packages? -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From bob at redivi.com Mon Jul 4 03:32:02 2005 From: bob at redivi.com (Bob Ippolito) Date: Sun, 3 Jul 2005 15:32:02 -1000 Subject: [Distutils] eggs: Python version independence In-Reply-To: <42C88BD6.4000507@colorstudy.com> References: <42C88BD6.4000507@colorstudy.com> Message-ID: <5B4A94BE-75E9-4564-96E2-2BB27941A582@redivi.com> On Jul 3, 2005, at 3:07 PM, Ian Bicking wrote: > Most (90%+) of the libraries I work with don't depend on a Python > version. But unfortunately when using easy_install/eggs they always > have an explicit version, and multiple installations are necessary > otherwise. You shouldn't have a site-packages that multiple Python interpreters are stomping on.. so I don't see how you can avoid this anyway. > Generally speaking, all pure-Python packages are version independent. > At least on the source level. I suppose .pyc files aren't always > (though they haven't changed for a long time, have they?) -- would > it be > reasonable to have version-independent packages? Python 2.4 can use Python 2.3 bytecode, but not vice versa. -bob From pje at telecommunity.com Mon Jul 4 04:19:08 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 03 Jul 2005 22:19:08 -0400 Subject: [Distutils] easy_install: setup.py-less packages In-Reply-To: <42C86E35.1070701@colorstudy.com> Message-ID: <5.1.1.6.0.20050703221421.01ac2b78@mail.telecommunity.com> At 06:01 PM 7/3/2005 -0500, Ian Bicking wrote: >Hi; trying to get back into this again... > >So, there's a package at http://svn.saddi.com/flup/trunk/ that I want to >install, but it has no setup.py file. I think that file should look >something like: > >from setuptools import setup >setup(name="flup", > packages=['flup', 'flup.middleware', 'flup.resolver', 'flup.server'] > version='0.0-devel-r???') > > >And I'd probably add other things, but I think that's all that's >required. So... how should I make this happen? Should I hardcode >'http://svn.saddi.com/flup/trunk' into the setup.py file? Can I make >the location overrideable with an option? How do I download the >checkout? Why don't you just use a subversion "external" to include the package in your own as a subdirectory? Then use the package_dir setting in setup to refer to the subdirectory. You then publish your wrapping package as a subversion URL. This won't work for auto-discovery from PyPI unless your URL ends with Flup.egg or something like that, but you can always give it to EasyInstall as a URL, or build eggs and publish those. >What was in a function before is now in >setuptools.package_index.PackageIndex, and involves all sorts of data >that doesn't seem to imply (because it's not a package without the >setup.py file, and there's no index to get it from). Well, it still doesn't stop you from using it to download. Be aware also that the API for the non-downloading parts of PackageIndex might change significantly between now and 0.6, due to the redesign docs I posted here a week or two ago. >Should I calculate r??? on my own, and if so where? Right before I call >setup()? I need the repository location to do so, so if the repository >location was overrideable then I'd need to get the real location. Note that bdist_egg has an option to pull this for you, although currently it uses the setup.py directory so that doesn't actually help in this case. >Once the files are checkout out, how do I put it in place? Is there a >way to set the "base path" for the packages, so I could say "install >these packages, found in /tmp/wherever-it-was-downloaded-to"? I don't understand. From pje at telecommunity.com Mon Jul 4 04:24:58 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 03 Jul 2005 22:24:58 -0400 Subject: [Distutils] easy_install: package index In-Reply-To: <42C86E3F.1000100@colorstudy.com> Message-ID: <5.1.1.6.0.20050703221914.03922b48@mail.telecommunity.com> At 06:01 PM 7/3/2005 -0500, Ian Bicking wrote: >I notice there is an option for the index URL. But it doesn't look like >a multi-option, i.e., you can't give several URLs, to be checked in >turn. -f /--find-links does this; it takes whitespace separated URLs, but they have to be of pages that have direct links to downloadable packages. I'm not sure how I could make something that would be sane in the presence of conflicting information between multiple PyPI-like indexes. > That would be useful. It'd also be nice if you could configure >this; for instance, there's several libraries I'm using that haven't had >proper releases yet, so they aren't in PyPI. If I could tell someone to >just add a particular line to ~/.setup.cfg or somesuch, then that would >make it easier. Maybe that's already possible? But that'd be no good >if that disabled the use of PyPI. It's ~/.pydistutils.cfg, and you want to put this in it: [easy_install] find_links = url_of_download_page1 url_of_download_page2 ... This will only disable PyPI for requests that can be met using links found on the listed pages. And if you use the --upgrade flag, it'll scan PyPI anyway, and take the latest and best version (given the request it's trying to fulfill) from whichever place it was found. Note, by the way, that these links don't have to be maintained by the authors; you can make one page for e.g. Paste, that has direct links to all the project distributions somebody needs, and just let them put that in their config or use it with -f. The only real downside to putting things in find_links is that the web pages have to be read each time you run easy_install. Maybe somebody could contribute some caching code and we could cache the pages locally up to a certain age. From rtomayko at gmail.com Mon Jul 4 05:43:31 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Sun, 3 Jul 2005 23:43:31 -0400 Subject: [Distutils] buildutils Message-ID: <470C4369-0B81-4337-A304-FDC84F165418@gmail.com> I'm getting ready to announce/release a new project I've been working on over the past week or so called "buildutils". It's a set of extensions to distutils that are oriented toward the build process. For example, there are commands for generating MD5/SHA-1 checksum files for distributables, running py.test based unit tests, pushing doc/distributables to another site via ssh/scp, etc. My goal is to remove make from my python development process. The package consists almost entirely of additional distutils commands; there's very little additional apparatus. It uses setuptools, taking advantage of its dependency resolution features to install command dependencies when commands are executed, which lets us put all commands in the box but not require dependencies until commands are actually used. Everything is still a little rough but there's more information here: http://buildutils.lesscode.org/ I was wondering if there would be any interest in directing mailing list discussion to the distutils-sig mailing list. Build commands may not be part of the SIG's charter but I figure most of the discussion will be relevant. The expertise of the members of this list would be a huge help. Perhaps I can try to point command development related discussion here while pushing support and usage related discussion to a different list? I apologize if I'm being dense. I'm not familiar with SIG process but would like to involve as much of the community as possible. Thanks, Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From ianb at colorstudy.com Mon Jul 4 07:47:12 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 04 Jul 2005 00:47:12 -0500 Subject: [Distutils] easy_install: setup.py-less packages In-Reply-To: <5.1.1.6.0.20050703221421.01ac2b78@mail.telecommunity.com> References: <5.1.1.6.0.20050703221421.01ac2b78@mail.telecommunity.com> Message-ID: <42C8CD60.8060909@colorstudy.com> Phillip J. Eby wrote: > At 06:01 PM 7/3/2005 -0500, Ian Bicking wrote: > >> Hi; trying to get back into this again... >> >> So, there's a package at http://svn.saddi.com/flup/trunk/ that I want to >> install, but it has no setup.py file. I think that file should look >> something like: >> >> from setuptools import setup >> setup(name="flup", >> packages=['flup', 'flup.middleware', 'flup.resolver', >> 'flup.server'] >> version='0.0-devel-r???') >> >> >> And I'd probably add other things, but I think that's all that's >> required. So... how should I make this happen? Should I hardcode >> 'http://svn.saddi.com/flup/trunk' into the setup.py file? Can I make >> the location overrideable with an option? How do I download the >> checkout? > > > Why don't you just use a subversion "external" to include the package in > your own as a subdirectory? Then use the package_dir setting in setup > to refer to the subdirectory. I'd never looked at those before; that seems to be just right for this. In other cases I might just download the file(s) on my own (e.g., archives online, etc.). > You then publish your wrapping package as a subversion URL. This won't > work for auto-discovery from PyPI unless your URL ends with Flup.egg or > something like that, but you can always give it to EasyInstall as a URL, > or build eggs and publish those. Yes. Hmm... I suppose I could create something that redirected from a detectable URL to the actual URL. That's kind of hacky though. Only being able to list actual eggs decreases the utility of a custom index. Could it also look at title attributes or something? It would be neat if you could download a working development copy with easy_install too. I'd like to encourage, or at least assist, people who want to interact with the code itself; and if you use a checkout you can at least use things like svn diff, even if you can't necessarily commit. >> Should I calculate r??? on my own, and if so where? Right before I call >> setup()? I need the repository location to do so, so if the repository >> location was overrideable then I'd need to get the real location. > > > Note that bdist_egg has an option to pull this for you, although > currently it uses the setup.py directory so that doesn't actually help > in this case. Cool... though yes, it wouldn't work quite right here. >> Once the files are checkout out, how do I put it in place? Is there a >> way to set the "base path" for the packages, so I could say "install >> these packages, found in /tmp/wherever-it-was-downloaded-to"? > > > I don't understand. Nevermind that, I think externals will work for me. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From rtomayko at gmail.com Mon Jul 4 08:46:51 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Mon, 4 Jul 2005 02:46:51 -0400 Subject: [Distutils] easy_install: setup.py-less packages In-Reply-To: <42C8CD60.8060909@colorstudy.com> References: <5.1.1.6.0.20050703221421.01ac2b78@mail.telecommunity.com> <42C8CD60.8060909@colorstudy.com> Message-ID: On Jul 4, 2005, at 1:47 AM, Ian Bicking wrote: > Phillip J. Eby wrote: >> You then publish your wrapping package as a subversion URL. This >> won't >> work for auto-discovery from PyPI unless your URL ends with >> Flup.egg or >> something like that, but you can always give it to EasyInstall as >> a URL, >> or build eggs and publish those. > > Yes. Hmm... I suppose I could create something that redirected from a > detectable URL to the actual URL. That's kind of hacky though. Only > being able to list actual eggs decreases the utility of a custom > index. > Could it also look at title attributes or something? > > It would be neat if you could download a working development copy with > easy_install too. I'd like to encourage, or at least assist, > people who > want to interact with the code itself; and if you use a checkout > you can > at least use things like svn diff, even if you can't necessarily > commit. That's interesting. If you could point to subversion repositories from an index page, you could throw a page somewhere with a big list of project subversion URLs that use some kind of convention on naming... foo-devel bar-devel such that $ easy_install.py foo-devel ... would grab and install the latest foo-devel from subversion. This would make a ton of sense in my development layout right now. Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From rtomayko at gmail.com Mon Jul 4 09:51:54 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Mon, 4 Jul 2005 03:51:54 -0400 Subject: [Distutils] easy_install: setup.py-less packages In-Reply-To: References: <5.1.1.6.0.20050703221421.01ac2b78@mail.telecommunity.com> <42C8CD60.8060909@colorstudy.com> Message-ID: <4C63DF5C-DCCC-4B46-8828-EDB935ADC529@gmail.com> >> Phillip J. Eby wrote: >>> You then publish your wrapping package as a subversion URL. This >>> won't >>> work for auto-discovery from PyPI unless your URL ends with >>> Flup.egg or >>> something like that, but you can always give it to EasyInstall as >>> a URL, >>> or build eggs and publish those. >> > Ian Bicking wrote: >> Yes. Hmm... I suppose I could create something that redirected >> from a >> detectable URL to the actual URL. That's kind of hacky though. Only >> being able to list actual eggs decreases the utility of a custom >> index. >> Could it also look at title attributes or something? >> >> It would be neat if you could download a working development copy >> with >> easy_install too. I'd like to encourage, or at least assist, >> people who >> want to interact with the code itself; and if you use a checkout >> you can >> at least use things like svn diff, even if you can't necessarily >> commit. >> Ryan Tomayko wrote: > That's interesting. If you could point to subversion repositories > from an index page, you could throw a page somewhere with a big > list of project subversion URLs that use some kind of convention on > naming... > > foo-devel > bar-devel That's going to be ambiguous though, right. How do you know those URLs are subversion repositories without having to go out and HEAD each resource or something. We can use the rel attribute: foo-devel From the HTML 4.01 specification, rel attribute [1]: > This attribute describes the relationship from the current > document to the anchor specified by the href attribute. The > value of this attribute is a space-separated list of link types. From the HTML 4.01 specification, link-types [2]: > Authors may wish to define additional link types not described > in this specification. If they do so, they should use a profile > to cite the conventions used to define the link types. Please see > the profile attribute of the HEAD element for more details. [1]: http://www.w3.org/TR/REC-html40/struct/links.html#adef-rel [2]: http://www.w3.org/TR/REC-html40/types.html#type-links Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From claytonbrowns at gmail.com Mon Jul 4 11:12:19 2005 From: claytonbrowns at gmail.com (Clayton Brown) Date: Mon, 4 Jul 2005 10:12:19 +0100 Subject: [Distutils] eggs: Python version independence In-Reply-To: <5B4A94BE-75E9-4564-96E2-2BB27941A582@redivi.com> References: <42C88BD6.4000507@colorstudy.com> <5B4A94BE-75E9-4564-96E2-2BB27941A582@redivi.com> Message-ID: Sort of on this topic, handy hint...to maintain a couple of python versions i did this with a bash sript (/usr/local/bin/pypan.sh) called 'pypan' funnily given all the naming issues of late. This just keeps my python instance in sync with each other for when i need to migrate to 2.4 ---8<----pypan.sh-------8<-----8<----------8<-----8<----------8<-----8<----------8<-----8<----------8<----- echo "==================================================" echo "PyPAN - Python Site Packages Installer running...." echo "==================================================" echo "python23 - installing using: /usr/local/Python2.3.4/bin/easy_install.py" python23 /usr/local/Python2.3.4/bin/easy_install.py $1 echo "\n\n" echo "python24 - installing using: /usr/local/bin/easy_install.py" python24 /usr/local/bin/easy_install.py $1 ---8<----pypan.sh-------8<-----8<----------8<-----8<----------8<-----8<----------8<-----8<----------8<----- Ill probably extend this to also use wget and keep a repository of all source packages, but recall seing this functionality in easy_install itself - and since it has better dig options etc best use it i think, long term i would like to use the (-m) option by default to always maintain mutliple versions of a package, but in my first attempt this require() method described failed to import the package that i had installed multiple versions of. Ideally python itself could be extended with the functionality easy_install provides, particularly to resolve latest version when not specified, and to alllow version specification at run time. From pje at telecommunity.com Mon Jul 4 18:23:55 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 04 Jul 2005 12:23:55 -0400 Subject: [Distutils] easy_install: setup.py-less packages In-Reply-To: <4C63DF5C-DCCC-4B46-8828-EDB935ADC529@gmail.com> References: <5.1.1.6.0.20050703221421.01ac2b78@mail.telecommunity.com> <42C8CD60.8060909@colorstudy.com> Message-ID: <5.1.1.6.0.20050704115529.0266e3c0@mail.telecommunity.com> At 03:51 AM 7/4/2005 -0400, Ryan Tomayko wrote: >>>Phillip J. Eby wrote: >>>>You then publish your wrapping package as a subversion URL. This >>>>won't >>>>work for auto-discovery from PyPI unless your URL ends with >>>>Flup.egg or >>>>something like that, but you can always give it to EasyInstall as >>>>a URL, >>>>or build eggs and publish those. > >>Ian Bicking wrote: >>>Yes. Hmm... I suppose I could create something that redirected >>>from a >>>detectable URL to the actual URL. That's kind of hacky though. Only >>>being able to list actual eggs decreases the utility of a custom >>>index. >>> Could it also look at title attributes or something? >>> >>>It would be neat if you could download a working development copy >>>with >>>easy_install too. I'd like to encourage, or at least assist, >>>people who >>>want to interact with the code itself; and if you use a checkout >>>you can >>>at least use things like svn diff, even if you can't necessarily >>>commit. > >Ryan Tomayko wrote: >>That's interesting. If you could point to subversion repositories >>from an index page, you could throw a page somewhere with a big >>list of project subversion URLs that use some kind of convention on >>naming... >> >> foo-devel >> bar-devel > >That's going to be ambiguous though, right. How do you know those >URLs are subversion repositories without having to go out and HEAD >each resource or something. > >We can use the rel attribute: > > > foo-devel > Um, yeah. You guys probably haven't noticed this, but EasyInstall detects links using a regular expression that looks for "href="; it doesn't even try to parse HTML, and I don't really want it to start trying, either. It'd be better if we could put the information in the URL itself, e.g. "http://example.com/svn/foo/trunk#egg=foo_devel". IOW, I could have EasyInstall extract information from a specially-formatted "fragment" portion of a URL, which should be safe because it doesn't get transmitted to the server. EasyInstall already knows when it fetches a subversion http: URL, and automatically does a checkout in that case. Of course, the only current use case for such fragment identifiers is to mark a non-egg URL as a discoverable subversion checkout. For this purpose, EasyInstall really needs to know the effective version number, unless it just treats subversion checkouts as being the highest possible version number for purposes of selecting a download target. Either that, or the development checkout of a project needs to be treated as a project with a different name. For example, the development project for "foo" could be "foo_devel", and therefore installing "foo_devel" gives you something different. Or, there could be a command-line option to EasyInstall (--develop, perhaps) that requests subversion checkouts take precedence over other kinds of links (so that they don't have to have a version number in the link). In this case, we could make the fragment syntax be something like "#svn=foo" to indicate that the link is a subversion checkout for the "foo" project. Okay, so here's the idea. I add #svn=projectname detection support, but treat such links as the lowest possible version number (an empty version), unless the --develop flag is supplied or the fragment includes a version number. In this way, subversion links are always discoverable but not used unless they are either the only link and you have no version installed, or the person creating the link has identified the version. But with the --develop flag, what happens is that instead of installing the package and its dependencies, you just check out the named package(s) to a directory (or directories) named for the project under(s) your designated --build directory. It then runs a "setup.py develop" command on each of the checked-out projects to create links and script wrappers in your --install directory, so that you can use them. The "develop" command (which doesn't exist yet, but which I'd originally planned to write this weekend) installs an .egg-link file in the target directory that points to the development source root, and runs 'build_ext -i' to build any C extensions in-place, so they'll run from the source directory. Oh, and it also updates the project .egg-info so that it's usable as an egg, and puts stub scripts in the --script-dir that run the original source scripts after the appropriate 'require()' operation(s). This then allows you to develop in one directory, while having that code "deployed" to another. (In fact, you can have your development checkout deployed to multiple target locations at the same time, if desired.) There would also be an --uninstall flag for the develop command to let you remove the wrappers and .egg-link, so that you could then install a non-development version to the target. Anyway, a --develop option for EasyInstall could be added in order to run "setup.py develop" on checked-out development versions of packages you're working on, using EasyInstall's --install-dir and --script-dir as the base for the "develop" commands. I'm also planning, by the way, to update setuptools' "test" command to support running from a built egg; right now it still does a traditional distutils "install" instead. From pje at telecommunity.com Mon Jul 4 18:45:41 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 04 Jul 2005 12:45:41 -0400 Subject: [Distutils] eggs: Python version independence In-Reply-To: References: <5B4A94BE-75E9-4564-96E2-2BB27941A582@redivi.com> <42C88BD6.4000507@colorstudy.com> <5B4A94BE-75E9-4564-96E2-2BB27941A582@redivi.com> Message-ID: <5.1.1.6.0.20050704124211.03737960@mail.telecommunity.com> At 10:12 AM 7/4/2005 +0100, Clayton Brown wrote: >Ill probably extend this to also use wget and keep a repository of all >source packages, but recall seing this functionality in easy_install >itself - and since it has better dig options etc best use it i think, Right now the closest thing is to use the -b option, and the downloaded file will be left in the build directory. But that won't help your script find and reuse the download, I'm afraid. >long term i would like to use the (-m) option by default to always >maintain mutliple versions of a package, but in my first attempt this >require() method described failed to import the package that i had >installed multiple versions of. What happened instead? >Ideally python itself could be extended with the functionality >easy_install provides, particularly to resolve latest version when not >specified, and to alllow version specification at run time. It does allow version specification at runtime; in fact there are already hooks that can be used to make require() automatically run easy_install, and these hooks are in fact already used to install packages' declared dependencies. However, these hooks have to be explicitly activated; simply building this capability into Python's default behavior would be an extreme security risk. From rtomayko at gmail.com Mon Jul 4 19:47:04 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Mon, 4 Jul 2005 13:47:04 -0400 Subject: [Distutils] eggs: Python version independence In-Reply-To: <5.1.1.6.0.20050704124211.03737960@mail.telecommunity.com> References: <5B4A94BE-75E9-4564-96E2-2BB27941A582@redivi.com> <42C88BD6.4000507@colorstudy.com> <5B4A94BE-75E9-4564-96E2-2BB27941A582@redivi.com> <5.1.1.6.0.20050704124211.03737960@mail.telecommunity.com> Message-ID: <7BA24CA4-C1A7-44E5-931F-9E504A05A311@gmail.com> On Jul 4, 2005, at 12:45 PM, Phillip J. Eby wrote: > It does allow version specification at runtime; in fact there are > already > hooks that can be used to make require() automatically run > easy_install, > and these hooks are in fact already used to install packages' declared > dependencies. > > However, these hooks have to be explicitly activated; simply > building this > capability into Python's default behavior would be an extreme > security risk. What's the best way to turn that on? I haven't seen it referenced in the doc anywhere. I have a case where that would be useful. Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From pje at telecommunity.com Mon Jul 4 20:47:44 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 04 Jul 2005 14:47:44 -0400 Subject: [Distutils] eggs: Python version independence In-Reply-To: <7BA24CA4-C1A7-44E5-931F-9E504A05A311@gmail.com> References: <5.1.1.6.0.20050704124211.03737960@mail.telecommunity.com> <5B4A94BE-75E9-4564-96E2-2BB27941A582@redivi.com> <42C88BD6.4000507@colorstudy.com> <5B4A94BE-75E9-4564-96E2-2BB27941A582@redivi.com> <5.1.1.6.0.20050704124211.03737960@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050704144340.02671e10@mail.telecommunity.com> At 01:47 PM 7/4/2005 -0400, Ryan Tomayko wrote: >On Jul 4, 2005, at 12:45 PM, Phillip J. Eby wrote: >>It does allow version specification at runtime; in fact there are >>already >>hooks that can be used to make require() automatically run >>easy_install, >>and these hooks are in fact already used to install packages' declared >>dependencies. >> >>However, these hooks have to be explicitly activated; simply >>building this >>capability into Python's default behavior would be an extreme >>security risk. > >What's the best way to turn that on? I haven't seen it referenced in >the doc anywhere. I have a case where that would be useful. Create an instance of AvailableDistributions or a subclass thereof, either overriding the 'obtain()' method, or passing in an 'installer' argument to resolve(). EasyInstall does this in its 'process_distribution()' method, for example. This API is going to change in 0.6 though, when I refactor to match the new spec. In the new spec, 'resolve()' will be an operation on a WorkingSet, which will be allowed to search various sources (e.g. collections of distributions) for possible resolutions. This will make more sense, IMO, than asking a collection to resolve a requirement in terms of a working set. Indeed, the awkwardness of that very mechanism was one of the things that made me realize the architecture needed refactoring. From dparson at csc.com Wed Jul 6 00:00:36 2005 From: dparson at csc.com (dparson@csc.com) Date: Tue, 5 Jul 2005 18:00:36 -0400 Subject: [Distutils] RETURNED MAIL: DATA FORMAT ERROR Message-ID: <20050705220042.9202E1E4012@bag.python.org> The original message was received at Tue, 5 Jul 2005 18:00:36 -0400 from csc.com [208.137.142.50] ----- The following addresses had permanent fatal errors ----- distutils-sig at python.org ----- Transcript of session follows ----- ... while talking to python.org.: >>> DATA <<< 400-aturner; -RMS-E-CRE, ACP file create failed <<< 400-aturner; -SYSTEM-F-EXDISKQUOTA, disk quota exceeded <<< 400 -------------- next part -------------- A non-text attachment was scrubbed... Name: document.zip Type: application/octet-stream Size: 29076 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20050705/4b1a46d1/document-0001.obj From pje at telecommunity.com Wed Jul 6 06:19:52 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 06 Jul 2005 00:19:52 -0400 Subject: [Distutils] setuptools 0.5a5: "develop" and "test" commands Message-ID: <5.1.1.6.0.20050705235229.02684aa0@mail.telecommunity.com> I've just released setuptools 0.5a5, which updates the old "test" command to do in-place build-and-test instead of the previous install-and-test behavior. It also automatically require()s the package's dependencies, if any, and updates the .egg-info metadata. I also added a new "develop" command, which is like an install that doesn't actually install the package. It runs "build_ext --inplace" (so that any C extensions get built in the source directory) and then installs a .egg-link pointing to the source tree, optionally updating easy-install.pth and installing any dependencies for the package. (It only takes a subset of the easy_install command options, however, so it's currently better if you install your dependencies before using the "develop" command, especially if you want to "develop" any of those as well.) The "develop" command basically adds your source checkout to the eggs that will be searched at runtime to resolve a 'require()' request, and if you're installing to site-packages, then it will be added to sys.path as well (unless you use --multi-version or -m). This lets you edit your source checkout and do development without having to actually install the package. Wrapper scripts are installed to the --script-dir; the wrappers do a require() and then execfile() your in-development script source files, so you don't have to re-run "develop" just because you've changed script contents. The only time you need to re-run the "develop" command is when your metadata (e.g. version or requirements) changes, or if you add a new script, or need to rebuild changed C extensions. When you're done with development, "setup.py develop --uninstall" will remove the .egg-link and any easy-install.pth entry for your development tree, but you will need to update or replace any installed scripts by hand. All of these commands respect the standard distutils .cfg files, so you can e.g. use setup.cfg to set options for developing and testing. There were a couple of other important changes in this release; easy_install the *module* is now found under setuptools.command; easy_install the *script* no longer contains the actual command code. Also, a new "egg_info" command was split out of "bdist_egg", so that it could be used by "test" and "develop" as well as "bdist_egg". As a result of this, "bdist_egg" no longer supports the various "tag" options that it did before; you must now do something like this:: setup.py egg_info --tag-svn-revision bdist_egg if you want to set tagging options. You can also use this with the "develop" and "test" commands, e.g.:: setup.py egg_info --tag-svn-revision develop in order to change the version info that will be used for the "virtual egg". At this point, I think setuptools now has sufficient features to support development of packages with "cheap" external dependencies. That is, a package using setuptools to create its setup script can add external dependencies with almost no effort on the part of the package developer, and no effort at all for the package installer. And, with further effort by persons familiar with the appropriate packaging systems, the dependency information included in setup.py could be used to generate native packages with dependency information (assuming there is some uniform mapping from PyPI project names to platform package names). In other words, I think we've now achieved an interim state of packaging nirvana. There are many additional stages of enlightenment possible, but I think we are now in a position to begin the journey. Of course, having some docs would help quite a lot, as right now there's no formal documentation for stuff like version/requirement syntax, how to specify requirements in the setup() call, etc. I'd like to start upgrading my existing packages to use the new features before I try and write any, but if any brave souls happen to get to it first, well, the more the merrier where docs are concerned. The one area where docs might be a problem for a while is the actual pkg_resources infrastructure, since I intend to do a major refactoring of it in the 0.6 releases. In addition to restructuring based on the cleaner concepts of distributions, environments, working sets, etc., it will also be test-driven, such that nearly all the pkg_resources code will have unit tests for it, and as much of the setuptools command code as possible will too. I'll also be trying to clean up the older attempts at a dependency management system that lurk deeper in setuptools; these never quite worked out before. Finally, there are a few older features that are now of questionable utility given the ease of adding external dependencies. But, I'm not sure if I can remove any/all of these things yet, because I don't know if anybody (besides Fred Drake and myself) has based anything interesting on those features. So, those changes might have to wait for the 0.7 release series, depending. From rtomayko at gmail.com Wed Jul 6 08:05:06 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Wed, 6 Jul 2005 02:05:06 -0400 Subject: [Distutils] setuptools 0.5a5: "develop" and "test" commands In-Reply-To: <5.1.1.6.0.20050705235229.02684aa0@mail.telecommunity.com> References: <5.1.1.6.0.20050705235229.02684aa0@mail.telecommunity.com> Message-ID: <441E438A-82F0-42EE-9294-E8F7F3371858@gmail.com> On Jul 6, 2005, at 12:19 AM, Phillip J. Eby wrote: > There were a couple of other important changes in this release; > easy_install the *module* is now found under setuptools.command; This is breaking setuptools upgrades using ez_setup.py. If you run the latest ez_setup.py on a system that has a previous version of setuptools installed, you get an ImportError: Traceback (most recent call last): File "ez_setup.py", line 160, in ? main(sys.argv[1:]) File "ez_setup.py", line 149, in main from setuptools.command.easy_install import main ImportError: No module named easy_install I was able to get around it with the following patch: $ diff ez_setup.py{.orig,} 149c149,152 < from setuptools.command.easy_install import main --- > try: > from setuptools.command.easy_install import main > except ImportError: > from easy_install import main Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From pje at telecommunity.com Wed Jul 6 08:29:56 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 06 Jul 2005 02:29:56 -0400 Subject: [Distutils] setuptools 0.5a5: "develop" and "test" commands In-Reply-To: <441E438A-82F0-42EE-9294-E8F7F3371858@gmail.com> References: <5.1.1.6.0.20050705235229.02684aa0@mail.telecommunity.com> <5.1.1.6.0.20050705235229.02684aa0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050706022735.02674a38@mail.telecommunity.com> At 02:05 AM 7/6/2005 -0400, Ryan Tomayko wrote: >On Jul 6, 2005, at 12:19 AM, Phillip J. Eby wrote: >>There were a couple of other important changes in this release; >>easy_install the *module* is now found under setuptools.command; > >This is breaking setuptools upgrades using ez_setup.py. If you run >the latest ez_setup.py on a system that has a previous version of >setuptools installed, you get an ImportError: > > Traceback (most recent call last): > File "ez_setup.py", line 160, in ? > main(sys.argv[1:]) > File "ez_setup.py", line 149, in main > from setuptools.command.easy_install import main > ImportError: No module named easy_install > >I was able to get around it with the following patch: > >$ diff ez_setup.py{.orig,} >149c149,152 >< from setuptools.command.easy_install import main >--- > > try: > > from setuptools.command.easy_install import main > > except ImportError: > > from easy_install import main Okay, I updated the download for that. Luckily, ez_setup isn't versioned or included in the main setuptools package, so it didn't necessitate a new release version. Testing EasyInstall and friends is currently a pain because there's no clean way to unit test most distutils commands. I'm really going to have to extend the OS sandbox code I wrote so it can do a virtual filesystem for testing purposes. From dangoor at gmail.com Wed Jul 6 16:30:17 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Wed, 6 Jul 2005 10:30:17 -0400 Subject: [Distutils] eggs and py2app/py2exe Message-ID: <3f085ecd05070607304c358d27@mail.gmail.com> Let's say you've got some libraries installed as eggs. If you run py2app or py2exe on the application, it seems like you'd want the egg (or at least it's contents) to come along with the application. Or, at least, I'd prefer it for the eggs to come along with the app, rather than being downloaded via require. (For curiousity's sake, I just tried it and found that the contents of the eggs were left behind when I ran py2app. Not at all surprising, given that I don't think py2app or py2exe know about eggs yet...) Anyone have opinions on how a standalone app should look with eggs? Kevin From rtomayko at gmail.com Wed Jul 6 16:52:36 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Wed, 6 Jul 2005 10:52:36 -0400 Subject: [Distutils] setuptools 0.5a5: "develop" and "test" commands In-Reply-To: <5.1.1.6.0.20050706022735.02674a38@mail.telecommunity.com> References: <5.1.1.6.0.20050705235229.02684aa0@mail.telecommunity.com> <5.1.1.6.0.20050705235229.02684aa0@mail.telecommunity.com> <5.1.1.6.0.20050706022735.02674a38@mail.telecommunity.com> Message-ID: <73C00BE9-8214-4976-800E-1C87E81398C1@gmail.com> On Jul 6, 2005, at 2:29 AM, Phillip J. Eby wrote: > At 02:05 AM 7/6/2005 -0400, Ryan Tomayko wrote: > Testing EasyInstall and friends is currently a pain because there's > no clean way to unit test most distutils commands. I'm really > going to have to extend the OS sandbox code I wrote so it can do a > virtual filesystem for testing purposes. I'm having the same problem right now. Testing commands--even commands like distutils that have a sane and programmatical interface--has always been a big pain for me. The problem seems similar to that of testing UI code. I'll be interested to take a peek at what you throw together here... Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From pje at telecommunity.com Wed Jul 6 17:58:20 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 06 Jul 2005 11:58:20 -0400 Subject: [Distutils] eggs and py2app/py2exe In-Reply-To: <3f085ecd05070607304c358d27@mail.gmail.com> Message-ID: <5.1.1.6.0.20050706115029.026798d8@mail.telecommunity.com> At 10:30 AM 7/6/2005 -0400, Kevin Dangoor wrote: >Let's say you've got some libraries installed as eggs. If you run >py2app or py2exe on the application, it seems like you'd want the egg >(or at least it's contents) to come along with the application. Or, at >least, I'd prefer it for the eggs to come along with the app, rather >than being downloaded via require. > >(For curiousity's sake, I just tried it and found that the contents of >the eggs were left behind when I ran py2app. Not at all surprising, >given that I don't think py2app or py2exe know about eggs yet...) > >Anyone have opinions on how a standalone app should look with eggs? The egg runtime is designed to allow "baskets" - zip files containing multiple eggs - in order to support py2exe and similar scenarios. Let's say you have two dependencies: foo 1.2 and bar 3.4. Your py2exe-generated zipfile contents should look like this: some_main_module.py some_stdlibmodule.py ...etc. foo-1.2-py2.4-win32.egg/ EGG-INFO/ PKG-INFO some_foo_module.py ...etc. bar-3.4-py2.4-win32.egg/ EGG-INFO/ PKG-INFO some_bar_module.py ...etc. In other words, you can zip up a bunch of directory eggs into a single zipfile, and the runtime will treat them as separate eggs. Of course, 'pkg_resources.py' needs to be among the modules placed directly in the zipfile, or else it won't be findable. Another alternative that's possible is to just include all the modules and packages directly, copying EGG-INFO dirs to ProjectName.egg-info dirs. This makes the packages discoverable with all their metadata, but doesn't provide any flexibility of activation. However, for a py2exe/py2app scenario, you probably don't need that flexibility for anything you're hardwiring into the executable anyway. The metadata is more useful on the build side, in that tools like py2exe and py2app could use setuptools' dependency info to assemble the application from the designated eggs, rather than trying to sniff out all the dependencies by introspection. From dangoor at gmail.com Wed Jul 6 20:06:26 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Wed, 6 Jul 2005 14:06:26 -0400 Subject: [Distutils] eggs and py2app/py2exe In-Reply-To: <5.1.1.6.0.20050706115029.026798d8@mail.telecommunity.com> References: <3f085ecd05070607304c358d27@mail.gmail.com> <5.1.1.6.0.20050706115029.026798d8@mail.telecommunity.com> Message-ID: <3f085ecd050706110657de370b@mail.gmail.com> On 7/6/05, Phillip J. Eby wrote: > The egg runtime is designed to allow "baskets" - zip files containing > multiple eggs - in order to support py2exe and similar scenarios. Aha. That is handy. Given how easy eggs are to work with, I'm certain that I'll be able to get them working with the existing py2app/py2exe, even if the process is a bit more manual. It seems like I just need to be sure that pkg_resources makes it in to the app, and that the eggs all end up in one of the spots that are naturally added to sys.path by the py2app/py2exe runtimes. Thanks for the tips! Kevin From dangoor at gmail.com Wed Jul 6 21:29:47 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Wed, 6 Jul 2005 15:29:47 -0400 Subject: [Distutils] Something that doesn't run from an egg Message-ID: <3f085ecd05070612295f2a2da8@mail.gmail.com> This is not wholly surprising, but py.test doesn't appear to run from an egg. (I say it's not surprising, given that py is doing its own import trickery...) The way it fails is interesting. I can import py from a python prompt, but not from the py.test script. /usr/local/bin/python Python 2.4.1 (#2, Mar 31 2005, 00:05:10) [GCC 3.3 20030304 (Apple Computer, Inc. build 1666)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import py.test >>> py.test py lib is at /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/py-0.8.0_alpha1-py2.4.egg/py/__init__.pyc Traceback (most recent call last): File "/usr/local/bin/py.test", line 4, in ? pkg_resources.run_main('py==0.8.0-alpha1', 'py.test') File "build\bdist.win32\egg\pkg_resources.py", line 110, in run_main File "build\bdist.win32\egg\pkg_resources.py", line 599, in run_script File "/Library/Frameworks/Python.framework/Versions/2.4/bin/py.test", line 3, in ? import pkg_resources ImportError: cannot import name py Another of the pylib scripts, py.countloc, works fine. That one may not depend on the rest of the package, though. I'd thought I'd toss this out there in case anyone cares, but I'm not sure if anyone will care about this particular case. Kevin From pje at telecommunity.com Wed Jul 6 21:32:53 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 06 Jul 2005 15:32:53 -0400 Subject: [Distutils] Something that doesn't run from an egg In-Reply-To: <3f085ecd05070612295f2a2da8@mail.gmail.com> Message-ID: <5.1.1.6.0.20050706153213.02678088@mail.telecommunity.com> Can you give me the download URL for the py source distro so I can try to reproduce this? At 03:29 PM 7/6/2005 -0400, Kevin Dangoor wrote: >This is not wholly surprising, but py.test doesn't appear to run from >an egg. (I say it's not surprising, given that py is doing its own >import trickery...) The way it fails is interesting. I can import py >from a python prompt, but not from the py.test script. > >/usr/local/bin/python >Python 2.4.1 (#2, Mar 31 2005, 00:05:10) >[GCC 3.3 20030304 (Apple Computer, Inc. build 1666)] on darwin >Type "help", "copyright", "credits" or "license" for more information. > >>> import py.test > >>> > >py.test >py lib is at >/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/py-0.8.0_alpha1-py2.4.egg/py/__init__.pyc >Traceback (most recent call last): > File "/usr/local/bin/py.test", line 4, in ? > pkg_resources.run_main('py==0.8.0-alpha1', 'py.test') > File "build\bdist.win32\egg\pkg_resources.py", line 110, in run_main > File "build\bdist.win32\egg\pkg_resources.py", line 599, in run_script > File "/Library/Frameworks/Python.framework/Versions/2.4/bin/py.test", >line 3, in ? > import pkg_resources >ImportError: cannot import name py > >Another of the pylib scripts, py.countloc, works fine. That one may >not depend on the rest of the package, though. > >I'd thought I'd toss this out there in case anyone cares, but I'm not >sure if anyone will care about this particular case. > >Kevin >_______________________________________________ >Distutils-SIG maillist - Distutils-SIG at python.org >http://mail.python.org/mailman/listinfo/distutils-sig From dangoor at gmail.com Wed Jul 6 22:01:32 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Wed, 6 Jul 2005 16:01:32 -0400 Subject: [Distutils] Something that doesn't run from an egg In-Reply-To: <5.1.1.6.0.20050706153213.02678088@mail.telecommunity.com> References: <3f085ecd05070612295f2a2da8@mail.gmail.com> <5.1.1.6.0.20050706153213.02678088@mail.telecommunity.com> Message-ID: <3f085ecd0507061301383d80cf@mail.gmail.com> On 7/6/05, Phillip J. Eby wrote: > Can you give me the download URL for the py source distro so I can try to > reproduce this? Right now, it's only available from subversion: svn co http://codespeak.net/svn/py/dist py-dist Here's the Getting Started page for it: http://codespeak.net/py/current/doc/getting-started.html Kevin From rtomayko at gmail.com Wed Jul 6 22:13:35 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Wed, 6 Jul 2005 16:13:35 -0400 Subject: [Distutils] Something that doesn't run from an egg In-Reply-To: <3f085ecd05070612295f2a2da8@mail.gmail.com> References: <3f085ecd05070612295f2a2da8@mail.gmail.com> Message-ID: <3DC9F38E-B6C1-469E-8FDE-372F192D8EB5@gmail.com> On Jul 6, 2005, at 3:29 PM, Kevin Dangoor wrote: > This is not wholly surprising, but py.test doesn't appear to run from > an egg. (I say it's not surprising, given that py is doing its own > import trickery...) The way it fails is interesting. I can import py > from a python prompt, but not from the py.test script. See also: Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From dangoor at gmail.com Wed Jul 6 22:16:23 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Wed, 6 Jul 2005 16:16:23 -0400 Subject: [Distutils] Something that doesn't run from an egg In-Reply-To: <3DC9F38E-B6C1-469E-8FDE-372F192D8EB5@gmail.com> References: <3f085ecd05070612295f2a2da8@mail.gmail.com> <3DC9F38E-B6C1-469E-8FDE-372F192D8EB5@gmail.com> Message-ID: <3f085ecd050706131647fb8026@mail.gmail.com> On 7/6/05, Ryan Tomayko wrote: > See also: > > That's funny. I was looking around a little in the distutils archive earlier today, and I happened to miss that one. Nice to know that I'm not the only one who'd like to see an egg of py.test. Kevin From pje at telecommunity.com Wed Jul 6 22:46:44 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 06 Jul 2005 16:46:44 -0400 Subject: [Distutils] Something that doesn't run from an egg In-Reply-To: <3f085ecd0507061301383d80cf@mail.gmail.com> References: <5.1.1.6.0.20050706153213.02678088@mail.telecommunity.com> <3f085ecd05070612295f2a2da8@mail.gmail.com> <5.1.1.6.0.20050706153213.02678088@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050706164122.02671900@mail.telecommunity.com> At 04:01 PM 7/6/2005 -0400, Kevin Dangoor wrote: >On 7/6/05, Phillip J. Eby wrote: > > Can you give me the download URL for the py source distro so I can try to > > reproduce this? > >Right now, it's only available from subversion: > >svn co http://codespeak.net/svn/py/dist py-dist Note that you can just "easy_install http://codespeak.net/svn/py/dist" to install it directly from subversion, without needing a manual checkout. (You *do* need a subversion client on your PATH, however.) Here's a patch to setuptools that appears to fix the problem (by not running imported scripts as __main__), and it's probably a good idea to add it to setuptools in any case. Meanwhile, however, somebody should probably tell the py folks that _findpy is a horrible, *horrible* hack and should be shot. :) Index: pkg_resources.py =================================================================== RCS file: /cvsroot/python/python/nondist/sandbox/setuptools/pkg_resources.py,v retrieving revision 1.36 diff -u -r1.36 pkg_resources.py --- pkg_resources.py 6 Jul 2005 03:46:16 -0000 1.36 +++ pkg_resources.py 6 Jul 2005 20:42:14 -0000 @@ -104,11 +104,11 @@ def run_main(dist_spec, script_name): """Locate distribution `dist_spec` and run its `script_name` script""" - import __main__ - __main__.__dict__.clear() - __main__.__dict__.update({'__name__':'__main__'}) - require(dist_spec)[0].metadata.run_script(script_name, __main__.__dict__) - + ns = sys._getframe(1).f_globals + name = ns['__name__'] + ns.clear() + ns['__name__'] = name + require(dist_spec)[0].metadata.run_script(script_name, ns) From dangoor at gmail.com Thu Jul 7 12:53:39 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Thu, 7 Jul 2005 06:53:39 -0400 Subject: [Distutils] Something that doesn't run from an egg In-Reply-To: <5.1.1.6.0.20050706164122.02671900@mail.telecommunity.com> References: <3f085ecd05070612295f2a2da8@mail.gmail.com> <5.1.1.6.0.20050706153213.02678088@mail.telecommunity.com> <3f085ecd0507061301383d80cf@mail.gmail.com> <5.1.1.6.0.20050706164122.02671900@mail.telecommunity.com> Message-ID: <3f085ecd05070703531d8e7792@mail.gmail.com> On 7/6/05, Phillip J. Eby wrote: > At 04:01 PM 7/6/2005 -0400, Kevin Dangoor wrote: > >On 7/6/05, Phillip J. Eby wrote: > > > Can you give me the download URL for the py source distro so I can try to > > > reproduce this? > > > >Right now, it's only available from subversion: > > > >svn co http://codespeak.net/svn/py/dist py-dist > > Note that you can just "easy_install http://codespeak.net/svn/py/dist" to > install it directly from subversion, without needing a manual > checkout. (You *do* need a subversion client on your PATH, however.) Neat. easy_install seems to have quite a few tricks up its sleeve. > Here's a patch to setuptools that appears to fix the problem (by not > running imported scripts as __main__), and it's probably a good idea to add > it to setuptools in any case. Meanwhile, however, somebody should probably > tell the py folks that _findpy is a horrible, *horrible* hack and should be > shot. :) Cool. I'll give this a whirl. Kevin From mwh at python.net Thu Jul 7 14:30:18 2005 From: mwh at python.net (Michael Hudson) Date: Thu, 07 Jul 2005 13:30:18 +0100 Subject: [Distutils] Something that doesn't run from an egg In-Reply-To: <5.1.1.6.0.20050706164122.02671900@mail.telecommunity.com> (Phillip J. Eby's message of "Wed, 06 Jul 2005 16:46:44 -0400") References: <5.1.1.6.0.20050706153213.02678088@mail.telecommunity.com> <3f085ecd05070612295f2a2da8@mail.gmail.com> <5.1.1.6.0.20050706153213.02678088@mail.telecommunity.com> <5.1.1.6.0.20050706164122.02671900@mail.telecommunity.com> Message-ID: <2mll4ivlwl.fsf@starship.python.net> "Phillip J. Eby" writes: > Here's a patch to setuptools that appears to fix the problem (by not > running imported scripts as __main__), and it's probably a good idea to add > it to setuptools in any case. Meanwhile, however, somebody should probably > tell the py folks that _findpy is a horrible, *horrible* hack and should be > shot. :) I think they know :) Cheers, mwh -- But since I'm not trying to impress anybody in The Software Big Top, I'd rather walk the wire using a big pole, a safety harness, a net, and with the wire not more than 3 feet off the ground. -- Grant Griffin, comp.lang.python From dangoor at gmail.com Thu Jul 7 14:58:48 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Thu, 7 Jul 2005 08:58:48 -0400 Subject: [Distutils] Something that doesn't run from an egg In-Reply-To: <5.1.1.6.0.20050706164122.02671900@mail.telecommunity.com> References: <3f085ecd05070612295f2a2da8@mail.gmail.com> <5.1.1.6.0.20050706153213.02678088@mail.telecommunity.com> <3f085ecd0507061301383d80cf@mail.gmail.com> <5.1.1.6.0.20050706164122.02671900@mail.telecommunity.com> Message-ID: <3f085ecd0507070558630aaf65@mail.gmail.com> On 7/6/05, Phillip J. Eby wrote: > Here's a patch to setuptools that appears to fix the problem (by not > running imported scripts as __main__), and it's probably a good idea to add > it to setuptools in any case. Meanwhile, however, somebody should probably > tell the py folks that _findpy is a horrible, *horrible* hack and should be > shot. :) I just gave the patch a try. I just wanted to confirm that it's behaving the same way for me that it did for you. Running the py.test script no longer has trouble finding py, because of your patch. However, it still doesn't run (due to other things that py.test is doing). Specifically: py.error.ENOENT: [No such file or directory]: /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/py-0.8.0_alpha1-py2.4.egg/py/test/defaultconftest.py Is that what you saw, or did py.test actually run for you? Kevin From pje at telecommunity.com Thu Jul 7 16:01:21 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 07 Jul 2005 10:01:21 -0400 Subject: [Distutils] Something that doesn't run from an egg In-Reply-To: <3f085ecd0507070558630aaf65@mail.gmail.com> References: <5.1.1.6.0.20050706164122.02671900@mail.telecommunity.com> <3f085ecd05070612295f2a2da8@mail.gmail.com> <5.1.1.6.0.20050706153213.02678088@mail.telecommunity.com> <3f085ecd0507061301383d80cf@mail.gmail.com> <5.1.1.6.0.20050706164122.02671900@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050707100023.028db920@mail.telecommunity.com> At 08:58 AM 7/7/2005 -0400, Kevin Dangoor wrote: >On 7/6/05, Phillip J. Eby wrote: > > Here's a patch to setuptools that appears to fix the problem (by not > > running imported scripts as __main__), and it's probably a good idea to add > > it to setuptools in any case. Meanwhile, however, somebody should probably > > tell the py folks that _findpy is a horrible, *horrible* hack and should be > > shot. :) > >I just gave the patch a try. I just wanted to confirm that it's >behaving the same way for me that it did for you. Running the py.test >script no longer has trouble finding py, because of your patch. >However, it still doesn't run (due to other things that py.test is >doing). Specifically: > >py.error.ENOENT: [No such file or directory]: >/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/py-0.8.0_alpha1-py2.4.egg/py/test/defaultconftest.py > >Is that what you saw, or did py.test actually run for you? Yes, it actually ran, and tried to test setuptools. :) So I gave it the --help option and it showed help. That's all I did with it. I was using Windows Python 2.3. From rtomayko at gmail.com Thu Jul 7 16:11:46 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Thu, 7 Jul 2005 10:11:46 -0400 Subject: [Distutils] Something that doesn't run from an egg In-Reply-To: <3f085ecd0507070558630aaf65@mail.gmail.com> References: <3f085ecd05070612295f2a2da8@mail.gmail.com> <5.1.1.6.0.20050706153213.02678088@mail.telecommunity.com> <3f085ecd0507061301383d80cf@mail.gmail.com> <5.1.1.6.0.20050706164122.02671900@mail.telecommunity.com> <3f085ecd0507070558630aaf65@mail.gmail.com> Message-ID: <50B550BD-BD72-46E4-9C64-4384FEF2E224@gmail.com> On Jul 7, 2005, at 8:58 AM, Kevin Dangoor wrote: > On 7/6/05, Phillip J. Eby wrote: > >> Here's a patch to setuptools that appears to fix the problem (by not >> running imported scripts as __main__), and it's probably a good >> idea to add >> it to setuptools in any case. Meanwhile, however, somebody should >> probably >> tell the py folks that _findpy is a horrible, *horrible* hack and >> should be >> shot. :) >> > > I just gave the patch a try. I just wanted to confirm that it's > behaving the same way for me that it did for you. Running the py.test > script no longer has trouble finding py, because of your patch. > However, it still doesn't run (due to other things that py.test is > doing). Specifically: > > py.error.ENOENT: [No such file or directory]: > /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ > site-packages/py-0.8.0_alpha1-py2.4.egg/py/test/defaultconftest.py I think this is due to the py library using __file__ and path operations instead of pkg_resources to load package data. I wasn't able to run it within an egg. If you crack the egg with something like the following, it seems to work fine:: cd /Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages mv py-0.8.0_alpha1-py2.4.egg py-0.8.0_alpha1-py2.4.egg.zip mkdir py-0.8.0_alpha1-py2.4.egg cd py-0.8.0_alpha1-py2.4.egg unzip ../py-0.8.0_alpha1-py2.4.egg.zip Now you should be able to run everything and as a kinda-egg (e.g. metadata and scripts are available). Of course, there's not much value in using setuptools at all at this point. :) I think the best path for getting the py lib to work with easy_install would be to just fix up their distutils setup and build a normal tar.gz source distributions. We can worry about porting the code to use pkg_resources at a later time. They've chosen to employ some interesting techniques for distribution, however, so I'm not sure how bad the problem is. Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From dangoor at gmail.com Thu Jul 7 16:25:20 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Thu, 7 Jul 2005 10:25:20 -0400 Subject: [Distutils] py2app and eggs Message-ID: <3f085ecd05070707252ade4c2c@mail.gmail.com> I've done some quick experimentation with py2app and packages installed as eggs (stopping short of monkeying with py2app itself). As it currently stands, py2app's dependency finder misses the dependencies of modules within eggs. I was easily able to get my setup script to copy the eggs I needed into the app's site-packages and generate a .pth file to use the eggs, but found that the app wouldn't run because of standard library modules that hadn't been included. In some respects, eggsploding the egg's packages into a directory and letting py2app pick the modules up from there is beneficial, because that would probably include just a subset of the egg in the final app. A smaller final app is a good thing. py2app also includes just the .pyc files this way, whereas the eggs I have include the source. I haven't tried py2exe yet, but I will probably end up with a similar approach. Kevin From dangoor at gmail.com Thu Jul 7 16:39:49 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Thu, 7 Jul 2005 10:39:49 -0400 Subject: [Distutils] Something that doesn't run from an egg In-Reply-To: <50B550BD-BD72-46E4-9C64-4384FEF2E224@gmail.com> References: <3f085ecd05070612295f2a2da8@mail.gmail.com> <5.1.1.6.0.20050706153213.02678088@mail.telecommunity.com> <3f085ecd0507061301383d80cf@mail.gmail.com> <5.1.1.6.0.20050706164122.02671900@mail.telecommunity.com> <3f085ecd0507070558630aaf65@mail.gmail.com> <50B550BD-BD72-46E4-9C64-4384FEF2E224@gmail.com> Message-ID: <3f085ecd0507070739115c384@mail.gmail.com> On 7/7/05, Ryan Tomayko wrote: > cd /Library/Frameworks/Python.framework/Versions/2.4/lib/ > python2.4/site-packages > mv py-0.8.0_alpha1-py2.4.egg py-0.8.0_alpha1-py2.4.egg.zip > mkdir py-0.8.0_alpha1-py2.4.egg > cd py-0.8.0_alpha1-py2.4.egg > unzip ../py-0.8.0_alpha1-py2.4.egg.zip > > Now you should be able to run everything and as a kinda-egg (e.g. > metadata and scripts are available). Of course, there's not much > value in using setuptools at all at this point. :) Yep. Of course, I already had a running py lib, I was just turning some parts used by my project into eggs. > I think the best path for getting the py lib to work with > easy_install would be to just fix up their distutils setup and build > a normal tar.gz source distributions. We can worry about porting the > code to use pkg_resources at a later time. They've chosen to employ > some interesting techniques for distribution, however, so I'm not > sure how bad the problem is. That's true... At least py lib is pure python, so you can easy_install it without having to have a functional C compiler. I'll give this a whirl on another day... Kevin From rtomayko at gmail.com Thu Jul 7 18:02:27 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Thu, 7 Jul 2005 12:02:27 -0400 Subject: [Distutils] ANN: buildutils-0.1.0 - Distutils extensions for developing Python libraries and applications. Message-ID: <1BED7FC7-1F45-4D67-87F6-AE98C93B2DCD@gmail.com> I'm happy to announce that an initial 0.1.0 version of Python Buildutils is available. The `buildutils` package contains extensions to Python's standard distribution utilities (`distutils`) that are often useful during the development of Python projects. Buildutils was created to scratch an itch: removing ``make`` from the Python development process and partially to gain a better understanding of how `distutils` works. The following extension commands are included: announce send a release announcement to mailing lists like python-announce-list at python.org checksum generate MD5 and SHA-1 checksum files for distributables. etags generate an TAGS file over all packages and module (for use in Emacs). flakes find lint using the pyflakes utility. info dumps information about the project. publish push distributables and documentation up to a project site using ssh/scp/sftp. pudge build Python documentation from restructured text documents and Python doc strings. pytest run py.test unit tests. stats dump statistics on the number of lines, files, modules, packages, etc. use bring in a working version of a dependency (uses setuptools egg stuff). For more information, including User's Guide, Command Reference, and installation options, visit the Buildutils project site: There's also a bit more information on the release at the lesscode.org blog: Ryan Tomayko From pje at telecommunity.com Thu Jul 7 19:12:54 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 07 Jul 2005 13:12:54 -0400 Subject: [Distutils] ANN: buildutils-0.1.0 - Distutils extensions for developing Python libraries and applications. In-Reply-To: <1BED7FC7-1F45-4D67-87F6-AE98C93B2DCD@gmail.com> Message-ID: <5.1.1.6.0.20050707125928.02673310@mail.telecommunity.com> At 12:02 PM 7/7/2005 -0400, Ryan Tomayko wrote: >I'm happy to announce that an initial 0.1.0 version of Python Buildutils >is available. Looks nice; I could easily see wanting to steal some of those commands for setuptools. :) Interestingly, there's some overlap between the commands as they exist now. For example, your 'use' command could use the setuptools "develop" command to install and uninstall development eggs safely on all platforms (note that Windows doesn't do symlinks) and your code is making a false assumption that .egg-info will always be in the package checkout's root. It might be better to have your 'use' command just be a wrapper for running the 'develop' command in each of the named projects, so that those details will work right. (i.e., AFAICT 'use' is short for cd-ing to each project and doing "pbu develop".) Also, there's some limited overlap between the new "upload" in setuptools and your "publish"; the difference being that "upload" only does source and egg uploads to PyPI. (I just implemented a workaround for the absence of egg support in PyPI.) From pje at telecommunity.com Thu Jul 7 19:27:45 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 07 Jul 2005 13:27:45 -0400 Subject: [Distutils] buildutils' use of setuptools (was Re: ANN: buildutils-0.1.0 - Distutils extensions for developing Python libraries and applications. In-Reply-To: <1BED7FC7-1F45-4D67-87F6-AE98C93B2DCD@gmail.com> Message-ID: <5.1.1.6.0.20050707131713.02673600@mail.telecommunity.com> At 12:02 PM 7/7/2005 -0400, Ryan Tomayko wrote: >I'm happy to announce that an initial 0.1.0 version of Python Buildutils >is available. By the way, I notice your extension doc suggests subclassing distutils.cmd.Command rather than setuptools.Command, and your setup.py doesn't use setuptools.setup(). You might want to revise those, because using setuptools.setup() provides important additional features that distutils.core.setup() cannot: * running "setup.py install" installs the package as an egg+.pth, for easy uninstalls and upgrades * running "setup.py sdist" with no MANIFEST.in file will automatically include all files under revision control (CVS or svn) in the project tree, not just those found by the distutils default search algorithm that searches only for source files, not docs, headers, or data. * running "setup.py install" when setuptools is not installed will bootstrap an actual installation of setuptools, whereas your current arrangement will not actually install it, just put it on sys.path during the execution of setup.py Also, 'depends.txt' is now unneeded; if you use 'from setuptools import setup' then you can pass 'install_requires' and 'extras_require' keywords to setup() and get the same effect. (If you do this, you should remove the depends.txt file from your .egg-info directory, so as not to list them twice; the keyword arguments get written to a different file.) From rtomayko at gmail.com Thu Jul 7 19:34:55 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Thu, 7 Jul 2005 13:34:55 -0400 Subject: [Distutils] ANN: buildutils-0.1.0 - Distutils extensions for developing Python libraries and applications. In-Reply-To: <5.1.1.6.0.20050707125928.02673310@mail.telecommunity.com> References: <5.1.1.6.0.20050707125928.02673310@mail.telecommunity.com> Message-ID: <23C54FE5-5F84-4417-970D-45677C5E8887@gmail.com> On Jul 7, 2005, at 1:12 PM, Phillip J. Eby wrote: > At 12:02 PM 7/7/2005 -0400, Ryan Tomayko wrote: > >> I'm happy to announce that an initial 0.1.0 version of Python >> Buildutils >> is available. > > Looks nice; I could easily see wanting to steal some of those > commands for setuptools. :) > > Interestingly, there's some overlap between the commands as they > exist now. Yep. I've been meaning to talk to you about that, in fact. Buildutils depends on and utilizes setuptools quite a bit. So there's really no reason not to move buildutils commands and functionality down into setuptools if they seem more relevant there. Ditto with distutils really. We may want to just start a conversation to establish whether splitting commands up as we our now is a even the best way to do it. Do we need distutils, setuptools, and buildutils? I think there's a good and obvious separation but we could probably pound out some general guidelines for where different types of functionality should go. > For example, your 'use' command could use the setuptools "develop" > command to install and uninstall development eggs safely on all > platforms (note that Windows doesn't do symlinks) and your code is > making a false assumption that .egg-info will always be in the > package checkout's root. It might be better to have your 'use' > command just be a wrapper for running the 'develop' command in each > of the named projects, so that those details will work right. > (i.e., AFAICT 'use' is short for cd-ing to each project and doing > "pbu develop".) Absolutely. I need to revisit use now that you've added support for develop in setuptools. I didn't even document it because I figured it would be retooled or removed completely. I added it back when we first had our discussion about development environment layout. > Also, there's some limited overlap between the new "upload" in > setuptools and your "publish"; the difference being that "upload" > only does source and egg uploads to PyPI. (I just implemented a > workaround for the absence of egg support in PyPI.) publish is a bit weird right now and I imagine it will be refactored considerably in the coming month. I wonder if if might be better to use subcommands here? e.g. publish_pypi, publish_scp, etc. and then a top level publish command that determines which subcommands to run based on setup.cfg or other metadata. Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From rtomayko at gmail.com Thu Jul 7 19:42:00 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Thu, 7 Jul 2005 13:42:00 -0400 Subject: [Distutils] buildutils' use of setuptools (was Re: ANN: buildutils-0.1.0 - Distutils extensions for developing Python libraries and applications. In-Reply-To: <5.1.1.6.0.20050707131713.02673600@mail.telecommunity.com> References: <5.1.1.6.0.20050707131713.02673600@mail.telecommunity.com> Message-ID: On Jul 7, 2005, at 1:27 PM, Phillip J. Eby wrote: > At 12:02 PM 7/7/2005 -0400, Ryan Tomayko wrote: >> I'm happy to announce that an initial 0.1.0 version of Python >> Buildutils >> is available. > > By the way, I notice your extension doc suggests subclassing > distutils.cmd.Command rather than setuptools.Command, and your > setup.py doesn't use setuptools.setup(). You might want to revise > those, because using setuptools.setup() provides important > additional features that distutils.core.setup() cannot: > > * running "setup.py install" installs the package as an egg+.pth, > for easy uninstalls and upgrades > > * running "setup.py sdist" with no MANIFEST.in file will > automatically include all files under revision control (CVS or svn) > in the project tree, not just those found by the distutils default > search algorithm that searches only for source files, not docs, > headers, or data. > > * running "setup.py install" when setuptools is not installed will > bootstrap an actual installation of setuptools, whereas your > current arrangement will not actually install it, just put it on > sys.path during the execution of setup.py > > Also, 'depends.txt' is now unneeded; if you use 'from setuptools > import setup' then you can pass 'install_requires' and > 'extras_require' keywords to setup() and get the same effect. (If > you do this, you should remove the depends.txt file from your .egg- > info directory, so as not to list them twice; the keyword arguments > get written to a different file.) These are all things I have on my list. I have a hard time keeping pace with your progress on setuptools. :) But yea, my plan is to try to get this straightened out for a quick release tonight. Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From jon.drukman at gamespot.com Fri Jul 8 08:58:31 2005 From: jon.drukman at gamespot.com (Jon Drukman) Date: Thu, 7 Jul 2005 23:58:31 -0700 Subject: [Distutils] Out of Office AutoReply: [CNET NETWORKS AV: VIRUS REMOVED] Delivery reports about your e-mail Message-ID: <676DC304356D354C9690D973CD29AE5C26DB91@CNET6.cnet.cnwk> I am out of the office until Monday July 11. Please contact Noel Morrison or Scott Bedard if your matter is urgent. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20050707/3d5b4d2d/attachment.htm From rtomayko at gmail.com Fri Jul 8 17:16:25 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Fri, 8 Jul 2005 11:16:25 -0400 Subject: [Distutils] ANN: buildutils-0.1.0 - Distutils extensions for developing Python libraries and applications. In-Reply-To: <5.1.1.6.0.20050707125928.02673310@mail.telecommunity.com> References: <5.1.1.6.0.20050707125928.02673310@mail.telecommunity.com> Message-ID: <9DF33FBF-9081-4A1C-85A9-F2745F996268@gmail.com> On Jul 7, 2005, at 1:12 PM, Phillip J. Eby wrote: > Interestingly, there's some overlap between the commands as they > exist now. For example, your 'use' command could use the > setuptools "develop" command to install and uninstall development > eggs safely on all platforms (note that Windows doesn't do > symlinks) and your code is making a false assumption that .egg-info > will always be in the package checkout's root. It might be better > to have your 'use' command just be a wrapper for running the > 'develop' command in each of the named projects, so that those > details will work right. (i.e., AFAICT 'use' is short for cd-ing > to each project and doing "pbu develop".) [warning: epiphany occurs half way through this message, read entirely before responding] Ah yes. I just sat down to take a look at the use command and I remember now the problem I was trying to solve. The develop command seems to rely on modifying the easy_install.pth in the site-packages directory and works much like Bob Ippolito's description of developing with pth files [1]. While this isn't a huge problem for me right now, there are times where I'd prefer to either not modify sys.path for all other python applications or will not have permissions to modify site-packages. Ideally, I'd like to only modify the directory of the project I'm working on and possibly the directory of the project I'm using. Now that I've said all that, it occurs to me that this might all be possible now. Ah yes, it is. Forgive me, I'm not trying hard enough. Let's say I have three projects: devel/kid devel/buildutils devel/pudge If I want to use the development version of kid from buildutils, I would do: cd devel/kid python setup.py develop --install-dir=../buildutils --script- dir=../buildutils -m And now as long as I start python within the devel/buildutils directory, everything seems to work just as I would have hoped. Very nice, Phillip. I may still leave the use command in tact and have it perform these commands on multiple projects the way it does today. For example, if I wanted to use the development versions of kid and pudge from buildutils: pbu use --projects=kid,pudge And to no longer use those projects: pbu use --stop --projects=kid,pudge Rock on. Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ [1] http://bob.pythonmac.org/archives/2005/02/06/using-pth-files-for- python-development/ From pje at telecommunity.com Fri Jul 8 19:00:19 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 08 Jul 2005 13:00:19 -0400 Subject: [Distutils] ANN: buildutils-0.1.0 - Distutils extensions for developing Python libraries and applications. In-Reply-To: <9DF33FBF-9081-4A1C-85A9-F2745F996268@gmail.com> References: <5.1.1.6.0.20050707125928.02673310@mail.telecommunity.com> <5.1.1.6.0.20050707125928.02673310@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050708124511.03758c68@mail.telecommunity.com> At 11:16 AM 7/8/2005 -0400, Ryan Tomayko wrote: >If I want to use the development version of kid from buildutils, I >would do: > > cd devel/kid > python setup.py develop --install-dir=../buildutils --script- > dir=../buildutils -m FYI, --script-dir defaults to --install-dir if you explicitly set --install-dir, so the above could read: python setup.py develop --install-dir=../buildutils -m or for the ultimate in cryptic brevity: python setup.py develop -md../buildutils :) >I may still leave the use command in tact and have it perform these >commands on multiple projects the way it does today. For example, if >I wanted to use the development versions of kid and pudge from >buildutils: > > pbu use --projects=kid,pudge > >And to no longer use those projects: > > pbu use --stop --projects=kid,pudge I'm not sure if you're saying you're going to keep having 'use' do what it does now to implement this, or whether you mean you're going to have it run "setup.py develop" with the right options in its targets. I'd recommend the latter, since your current implementation doesn't work for projects that put their source code in a subdirectory of the project directory (e.g. a "lib" or "src" subdirectory). If you make it a wrapper for "develop", then any setuptools-using project should be able to use it, without being forced to follow your personal choice of project directory layout. :) If you do this, you should also use the current package's: os.path.abspath(self.get_finalized_command('egg_info').egg_base) as the --install-dir for the develop commands, so that projects with "src" or "lib" dirs can be the target of the use command, as well as their source. (egg_base is the directory where the project's .egg-info will be located; getting it from the 'egg_info' command allows advanced users to override it in setup.cfg if it can't be determined from the distribution package_dir setting(s).) By the way, the whole idea of using develop to link one package to another is quite interesting; I hadn't thought of making one package the target of another like this, but it seems useful and ought to work. Even more interesting is that if you run "develop" on the package that has had other projects develop-linked to it, then the whole tree of dependencies becomes available in the target. However, you should be careful to avoid creating a cycle of develop-links, because it will cause an infinite loop at runtime. pkg_resources does not have any ability to detect such link cycles at the moment, although I could probably add something without too much difficulty. From dangoor at gmail.com Fri Jul 8 21:14:05 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Fri, 8 Jul 2005 15:14:05 -0400 Subject: [Distutils] ANN: buildutils-0.1.0 - Distutils extensions for developing Python libraries and applications. In-Reply-To: <5.1.1.6.0.20050708124511.03758c68@mail.telecommunity.com> References: <5.1.1.6.0.20050707125928.02673310@mail.telecommunity.com> <9DF33FBF-9081-4A1C-85A9-F2745F996268@gmail.com> <5.1.1.6.0.20050708124511.03758c68@mail.telecommunity.com> Message-ID: <3f085ecd05070812143e535ce2@mail.gmail.com> This is a very handy conversation... it coincides with work I'm doing quite nicely. On 7/8/05, Phillip J. Eby wrote: > FYI, --script-dir defaults to --install-dir if you explicitly set > --install-dir, so the above could read: > > python setup.py develop --install-dir=../buildutils -m > > or for the ultimate in cryptic brevity: > > python setup.py develop -md../buildutils > > :) > This is a good trick. Unfortunately, it's not working for me. I have an egg installed for a package (A) that is required by a package (B) I'm working on. I wanted to debug something in A, so I upped the version number in A's setup script and ran the command you give up above. In the top level directory of project B, there is now an A.egg-link. Very cool. From that directory (and with that directory in sys.path), I was unable to find the newer version of A using pkg_resources.require. Removing the older egg from the easy-install.pth file fixed this problem... I was able to find the new version that was linked in just fine. This is with setuptools 0.5a8. As an aside, the thing that I'm trying to debug in A is a "Not a directory" error that I think is a __file__ usage. Ahh, the joys of being among the first to try to exploit this stuff. But I must say that, despite this, eggs are working great! Kevin From rtomayko at gmail.com Fri Jul 8 21:37:19 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Fri, 8 Jul 2005 15:37:19 -0400 Subject: [Distutils] ANN: buildutils-0.1.0 - Distutils extensions for developing Python libraries and applications. In-Reply-To: <3f085ecd05070812143e535ce2@mail.gmail.com> References: <5.1.1.6.0.20050707125928.02673310@mail.telecommunity.com> <9DF33FBF-9081-4A1C-85A9-F2745F996268@gmail.com> <5.1.1.6.0.20050708124511.03758c68@mail.telecommunity.com> <3f085ecd05070812143e535ce2@mail.gmail.com> Message-ID: <87F66F13-14E5-47B5-AA65-AE00EC9CFB9D@gmail.com> On Jul 8, 2005, at 3:14 PM, Kevin Dangoor wrote: > As an aside, the thing that I'm trying to debug in A is a "Not a > directory" error that I think is a __file__ usage. Ahh, the joys of > being among the first to try to exploit this stuff. But I must say > that, despite this, eggs are working great! Yea. I'd like to chime in on that too and just say that after working with setuptools for the past couple of weeks I think it's going to work out nicely. So there's some market validation for you :) Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From pje at telecommunity.com Fri Jul 8 21:39:01 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 08 Jul 2005 15:39:01 -0400 Subject: [Distutils] ANN: buildutils-0.1.0 - Distutils extensions for developing Python libraries and applications. In-Reply-To: <3f085ecd05070812143e535ce2@mail.gmail.com> References: <5.1.1.6.0.20050708124511.03758c68@mail.telecommunity.com> <5.1.1.6.0.20050707125928.02673310@mail.telecommunity.com> <9DF33FBF-9081-4A1C-85A9-F2745F996268@gmail.com> <5.1.1.6.0.20050708124511.03758c68@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050708153022.02674b68@mail.telecommunity.com> At 03:14 PM 7/8/2005 -0400, Kevin Dangoor wrote: >This is a very handy conversation... it coincides with work I'm doing >quite nicely. > >On 7/8/05, Phillip J. Eby wrote: > > FYI, --script-dir defaults to --install-dir if you explicitly set > > --install-dir, so the above could read: > > > > python setup.py develop --install-dir=../buildutils -m > > > > or for the ultimate in cryptic brevity: > > > > python setup.py develop -md../buildutils > > > > :) > > > >This is a good trick. Unfortunately, it's not working for me. >... >Removing the older egg from the easy-install.pth file fixed this >problem... I was able to find the new version that was linked in just >fine. Sounds like it is working for you, then. :) Seriously, pkg_resources is very conservative right now about versioning; if you install something without the -m flag, then you are telling it that that is the version you want to use, period. Part of the problem is that pkg_resources adds new versions to the *end* of sys.path, which means that if you have an older version already on sys.path when you ask for the newer one, there's no way that pkg_resources can safely change the path to allow the new version to take effect. The moral is that you probably want to use -m even for site-packages installations, unless there's a good reason to do otherwise. The reason -m doesn't get used by default for site-packages is because I want people whose use case is just "install this thing and make it work like I'm used to" to have a good experience with EasyInstall. :) Conversely, I figure smart people like you will know (or learn) that they want --multi-version/-m for all their site-packages installs. :) (Eventually, we can change this default, once the majority of packages declare their dependencies.) >As an aside, the thing that I'm trying to debug in A is a "Not a >directory" error that I think is a __file__ usage. You might try grepping for use of __file__ and __path__ in .py files; it's rare to touch either of those for any reason but data file access. From dangoor at gmail.com Fri Jul 8 22:21:37 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Fri, 8 Jul 2005 16:21:37 -0400 Subject: [Distutils] ANN: buildutils-0.1.0 - Distutils extensions for developing Python libraries and applications. In-Reply-To: <5.1.1.6.0.20050708153022.02674b68@mail.telecommunity.com> References: <5.1.1.6.0.20050707125928.02673310@mail.telecommunity.com> <9DF33FBF-9081-4A1C-85A9-F2745F996268@gmail.com> <5.1.1.6.0.20050708124511.03758c68@mail.telecommunity.com> <3f085ecd05070812143e535ce2@mail.gmail.com> <5.1.1.6.0.20050708153022.02674b68@mail.telecommunity.com> Message-ID: <3f085ecd05070813213883b86b@mail.gmail.com> On 7/8/05, Phillip J. Eby wrote: > >Removing the older egg from the easy-install.pth file fixed this > >problem... I was able to find the new version that was linked in just > >fine. > > Sounds like it is working for you, then. :) > > Seriously, pkg_resources is very conservative right now about versioning; > if you install something without the -m flag, then you are telling it that > that is the version you want to use, period. Part of the problem is that > pkg_resources adds new versions to the *end* of sys.path, which means that > if you have an older version already on sys.path when you ask for the newer > one, there's no way that pkg_resources can safely change the path to allow > the new version to take effect. Ahh, that makes sense to me now. I discovered that I even had another copy of that package lying around on my path (left over from when I expanded the egg for py2app), which certainly couldn't have been helping things... Kevin From trust_no_1 at spymac.com Sun Jul 10 23:53:48 2005 From: trust_no_1 at spymac.com (Jean-Christophe G.) Date: Sun, 10 Jul 2005 21:53:48 +0000 (UTC) Subject: [Distutils] =?utf-8?q?bdist=5Fnsi?= Message-ID: Just a short announce concerning the bdist_nsi mod: http://bdist-nsi.sourceforge.net/ It's an alternative way to build win32 package installer using Nsis. This adds : Silent install, Modern UI, internationnalization, uninstall Have a look at the screenshots! It's still in early stage development but works quite well on all package I tested without any modification as long as your setup.py is not esotheric From hoffman at ebi.ac.uk Mon Jul 11 16:39:45 2005 From: hoffman at ebi.ac.uk (Michael Hoffman) Date: Mon, 11 Jul 2005 15:39:45 +0100 Subject: [Distutils] Setuptools and non-system installs Message-ID: I was trying to install setuptools in my home directory on a system I don't have root access to using the command "python setup.py install --home=~". Of course, since it installs as an egg, I can't use it unless I *specifically* add it to my PYTHONPATH. This would be a pain should it be necessary to add for every egg I install. I've added a workaround to my ~/lib/python/sitecustomize.py (~/lib/python is in PYTHONPATH) which uses the existing machinery in site.py to process .pth files in all of the entries of sys.path, not just the default sitedirs. I've noticed, however, that Guido has rejected a patch to have the default site.py search .pth files in other directories, mainly because all the extra path searching at each startup might slow down startup on some systems. The best solution I can think of is to think of is to have some directory that is explicitly searched for .pth files (or even .egg/.zip files) and has them added to the path. This would still require some changes to sitecustomize.py (and hopefully eventually site.py), so it keeps eggs from working OOTB on this type of system. What do you think is the best way to have eggs loaded from non-system installs? -- Michael Hoffman European Bioinformatics Institute From pje at telecommunity.com Mon Jul 11 17:32:27 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 11 Jul 2005 11:32:27 -0400 Subject: [Distutils] Setuptools and non-system installs In-Reply-To: Message-ID: <5.1.1.6.0.20050711111639.026a90b0@mail.telecommunity.com> At 03:39 PM 7/11/2005 +0100, Michael Hoffman wrote: >I was trying to install setuptools in my home directory on a system I >don't have root access to using the command "python setup.py install >--home=~". Of course, since it installs as an egg, I can't use it >unless I *specifically* add it to my PYTHONPATH. This would be a pain >should it be necessary to add for every egg I install. It isn't necessary. All you need to do is ensure that the setuptools egg is on PYTHONPATH; everything else can then be handled by EasyInstall-generated script wrappers or by manual use of require(). So, basically, if you have this non-root scenario, you have to put the setuptools egg directly on your PYTHONPATH, and everything should be fine after that. >I've added a workaround to my ~/lib/python/sitecustomize.py >(~/lib/python is in PYTHONPATH) which uses the existing machinery in >site.py to process .pth files in all of the entries of sys.path, not >just the default sitedirs. Unfortunately EasyInstall can't know that you're doing this, so it's not going to generate a .pth file in your ~/lib/python directory. Basically, you need to learn to live with --multi-version mode, which has benefits as well as drawbacks. When EasyInstall installs in multi-version mode, you can't just fire up the interpreter and import the package, you have to either call require() to ask for the package by name and version, *or* you can run any script that was installed with the package and it'll work just fine because EasyInstall creates a wrapper around the script that does the require() for you. If you're using packages that depend on each other, or creating your own packages, you can create a wrapper package that's just a setup.py and whatever programs you need to run. In the setup.py, you declare your dependencies on the other packages, and then you run "setup.py develop" to generate wrappers for your new scripts and set up your project as an egg. The steps are detailed in the new setuptools Developers' Guide at: http://peak.telecommunity.com/DevCenter/setuptools In particular, pay close attention to the sections on "Basic Use", "Declaring Dependencies", "Development Mode", and on the "develop" command. >I've noticed, however, that Guido has rejected a patch > to have the default site.py search >.pth files in other directories, mainly because all the extra path >searching at each startup might slow down startup on some systems. > >The best solution I can think of is to think of is to have some >directory that is explicitly searched for .pth files (or even >.egg/.zip files) and has them added to the path. This would still >require some changes to sitecustomize.py (and hopefully eventually >site.py), so it keeps eggs from working OOTB on this type of system. > >What do you think is the best way to have eggs loaded from non-system >installs? I don't think that the patch makes much difference; there's nothing stopping web hosters or others from adding a line like this to a .pth file in site-packages now: import os; addsitedir(os.path.expanduser('~/lib/python2.3')) # or 2.4 Which will do the same as the patch proposes to do for every line in each .pth file. If you can get your service provider to put this in a .pth file in site-packages, you're pretty much set, except for the fact that EasyInstall will still think you need to do a multi-version install. However, you won't need PYTHONPATH and you won't need to manage it except by manipulating .pth files in your ~/lib/python2.3 or whatever directory. And, I could probably add an option to EasyInstall (and setuptools install/develop commands) to force .pth usage even if you're installing somewhere other than site-packages. From p.f.moore at gmail.com Mon Jul 11 17:58:04 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 11 Jul 2005 16:58:04 +0100 Subject: [Distutils] setuptools for people behind a firewall Message-ID: <79990c6b05071108585afc3bcd@mail.gmail.com> I've just had a look at the new documentation for setuptools. I've not read it all in detail yet, but one thing struck me regarding the "automatically download dependencies" feature. It isn't going to work for people (like me) stuck behind a firewall that Python doesn't support (Windows NTLM based firewall). Obviously, setuptools is never going to be able to resolve a situation like this, nor would I expect it to. But can I suggest two possible changes to make it easier for people with limited internet access? 1. A "manual download" mode, where setuptools lists the files which it wants you to obtain, and then leaves it to you how you get them. I'm not sure how plausible this would be, given the necessarily iterative process involved in resolving dependencies, but even a little help would be useful (a report of unresolved dependencies when run with a --no-download flag would be the most basic help). 2. A way of specifying an external command to use to download files over HTTP. This would (for example) allow me to use curl, which does support HTLM proxies, rather than relying on Python's built-in HTTP support, which doesn't. Regards, Paul. From rtomayko at gmail.com Mon Jul 11 18:06:57 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Mon, 11 Jul 2005 12:06:57 -0400 Subject: [Distutils] setuptools for people behind a firewall In-Reply-To: <79990c6b05071108585afc3bcd@mail.gmail.com> References: <79990c6b05071108585afc3bcd@mail.gmail.com> Message-ID: <6AA7FD94-7D9A-43E2-9372-5938A504FEB2@gmail.com> One thing you could do is keep a bunch of eggs, .tar.gz's, exe's, whatever in a directory on a web server with directory indexes turned on and then add that page to the find_links options in you ~/.pydistutils.cfg file. Here's mine:: [easy_install] find_links=http://lesscode.org/eggs/ http://peak.telecommunity.com/dist/ Then, just evolve a process for placing things on the behind-the- firewall server and you should be good. Ryan On Jul 11, 2005, at 11:58 AM, Paul Moore wrote: > I've just had a look at the new documentation for setuptools. I've not > read it all in detail yet, but one thing struck me regarding the > "automatically download dependencies" feature. > > It isn't going to work for people (like me) stuck behind a firewall > that Python doesn't support (Windows NTLM based firewall). Obviously, > setuptools is never going to be able to resolve a situation like this, > nor would I expect it to. But can I suggest two possible changes to > make it easier for people with limited internet access? > > 1. A "manual download" mode, where setuptools lists the files which it > wants you to obtain, and then leaves it to you how you get them. I'm > not sure how plausible this would be, given the necessarily iterative > process involved in resolving dependencies, but even a little help > would be useful (a report of unresolved dependencies when run with a > --no-download flag would be the most basic help). > > 2. A way of specifying an external command to use to download files > over HTTP. This would (for example) allow me to use curl, which does > support HTLM proxies, rather than relying on Python's built-in HTTP > support, which doesn't. > > Regards, > Paul. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From ianb at colorstudy.com Mon Jul 11 18:17:24 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 11 Jul 2005 11:17:24 -0500 Subject: [Distutils] setuptools for people behind a firewall In-Reply-To: <79990c6b05071108585afc3bcd@mail.gmail.com> References: <79990c6b05071108585afc3bcd@mail.gmail.com> Message-ID: <42D29B94.1000508@colorstudy.com> Paul Moore wrote: > 2. A way of specifying an external command to use to download files > over HTTP. This would (for example) allow me to use curl, which does > support HTLM proxies, rather than relying on Python's built-in HTTP > support, which doesn't. Someone mentioned curl to me in a ticket, but didn't mention specifically why it was "better". I assume now that NTLM was the issue. They submitted this patch that used pycurl: http://pythonpaste.org/trac/attachment/ticket/10/build-pkg.diff I imagine that would be an easy change to setuptools as well. Though it would be nice if setuptools could detect the case where urllib failed due to an NTLM firewall, so it could suggest (in the error message) that the user install pycurl. I don't know how easy pycurl is to install -- I wonder in practice if command-line interaction with curl would be easier to get people to install (pycurl doesn't appear to have up-to-date windows installers). -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From p.f.moore at gmail.com Mon Jul 11 21:57:35 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 11 Jul 2005 20:57:35 +0100 Subject: [Distutils] setuptools for people behind a firewall References: <79990c6b05071108585afc3bcd@mail.gmail.com> <42D29B94.1000508@colorstudy.com> Message-ID: Ian Bicking writes: > I don't know how easy pycurl is to install -- I wonder in practice > if command-line interaction with curl would be easier to get people > to install (pycurl doesn't appear to have up-to-date windows > installers). I was certainly just thinking of being able to supply a command line which, given a URL, would produce the file on stdout (in the case of curl, "curl %s" is the basic command, possibly with proxy-type options added). I don't see the need for using pycurl - and doing so would exclude use of other utilities like wget. Paul. -- The trouble with being punctual is that nobody's there to appreciate it. -- Franklin P. Jones From p.f.moore at gmail.com Mon Jul 11 21:54:08 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 11 Jul 2005 20:54:08 +0100 Subject: [Distutils] setuptools for people behind a firewall References: <79990c6b05071108585afc3bcd@mail.gmail.com> <6AA7FD94-7D9A-43E2-9372-5938A504FEB2@gmail.com> Message-ID: Ryan Tomayko writes: > One thing you could do is keep a bunch of eggs, .tar.gz's, exe's, > whatever in a directory on a web server with directory indexes turned > on and then add that page to the find_links options in you > ~/.pydistutils.cfg file. That's pretty neat. It's obvious in retrospect, but I never thought of it... > Then, just evolve a process for placing things on the behind-the- > firewall server and you should be good. Ah, there's the difficulty, of course :-) That's where a utility to report the dependencies would help... (It may be pretty trivial, but I don't see immediately how to do this - it seems to me that the current setuptools documentation is more for *creators* of eggs, than for *users* of them...) Paul. -- The trouble with being punctual is that nobody's there to appreciate it. -- Franklin P. Jones From dangoor at gmail.com Mon Jul 11 22:19:41 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Mon, 11 Jul 2005 16:19:41 -0400 Subject: [Distutils] pkg_resources: getting to resources in other packages Message-ID: <3f085ecd05071113194399e4ed@mail.gmail.com> I've found a scenario that isn't working, and I figured I'd toss this out there to see if a change is needed in setuptools or in my scenario... Picture something like this: toppackage __init__.py somemod.py lowerpackage __init__.py someresource.txt In somemod.py: import pkg_resources s = pkg_resource.resource_string("toppackage.lowerpackage", "someresource.txt") This will fail with a KeyError in get_provider(moduleName). If there is an import toppackage.lowerpackage before the resource_string call, then it works fine. In this particular instance, I'm going to go ahead and implement logic to ensure that the package has been imported. But, I was figuring that it might be nice if pkg_resource handled this... What say you all? Kevin From p.f.moore at gmail.com Mon Jul 11 23:32:54 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 11 Jul 2005 22:32:54 +0100 Subject: [Distutils] easy_install - some thoughts Message-ID: I just built a couple of eggs, for interest, experimentation, and generally "having a go" with setuptools. A couple of things struck me about the end-user experience with easy_install: 1. It would be much better if easy_install was usable via python -m, either as well as or instead of the existing easy_install.py script. On Windows, C:\Python24\Scripts is not normally on %PATH%, so easy_install isn't directly available after installing setuptools, without an additional step. 2. It would be useful if the -f option worked with a file: URL (or better still, a simple directory name), to handle local repositories. 3. I still dislike the fact that I can't get a listing of installed eggs. And uninstalling by deletion isn't good, IMHO. I am *strongly* averse to using raw file operations (delete) on files in Python\lib\site-packages. Elsewhere, fine, but I view the Python standard library as a "managed" area, and management tools should exist for it. That's the key benefit to me of bdist_wininst, and I don't see eggs providing it. 4. --dry-run doesn't work as expected: >\Apps\Python24\Scripts\easy_install.py --dry-run dist/alarm-1.0-py2.4-win32.egg Processing alarm-1.0-py2.4-win32.egg Copying alarm-1.0-py2.4-win32.egg to C:\Apps\Python24\Lib\site-packages Traceback (most recent call last): File "C:\Apps\Python24\Scripts\easy_install.py", line 18, in ? main(sys.argv[1:]) File "C:\Apps\Python24\Lib\site-packages\setuptools\command\easy_install.py", line 987, in main setup(script_args = ['-q','easy_install', '-v']+argv, **kw) File "C:\Apps\Python24\Lib\site-packages\setuptools\__init__.py", line 51, in setup return distutils.core.setup(**attrs) File "C:\Apps\Python24\lib\distutils\core.py", line 149, in setup dist.run_commands() File "C:\Apps\Python24\Lib\site-packages\setuptools\dist.py", line 419, in run _commands self.run_command(cmd) File "C:\Apps\Python24\lib\distutils\dist.py", line 966, in run_command cmd_obj.run() File "C:\Apps\Python24\Lib\site-packages\setuptools\command\easy_install.py", line 222, in run self.easy_install(spec) File "C:\Apps\Python24\Lib\site-packages\setuptools\command\easy_install.py", line 260, in easy_install return self.install_item(None, spec, tmpdir, True) File "C:\Apps\Python24\Lib\site-packages\setuptools\command\easy_install.py", line 294, in install_item dists = self.install_eggs(download, tmpdir) File "C:\Apps\Python24\Lib\site-packages\setuptools\command\easy_install.py", line 414, in install_eggs return [self.install_egg(dist_filename, tmpdir)] File "C:\Apps\Python24\Lib\site-packages\setuptools\command\easy_install.py", line 491, in install_egg return self.egg_distribution(destination) File "C:\Apps\Python24\Lib\site-packages\setuptools\command\easy_install.py", line 456, in egg_distribution metadata = EggMetadata(zipimport.zipimporter(egg_path)) zipimport.ZipImportError: not a Zip file Don't get me wrong - eggs are definitely going in the right direction. But there's still a way to go (for me, at least) yet. Some more unfocused and rambling thoughts: A nice to have feature would be the ability for easy_install to pick apart a bdist_wininst installer and build an egg from it. This would enormously help Windows users who don't have a compiler, but do want to use eggs - in my view, it's fairly "all or nothing" choice. You *don't* want to have part of your installation managed via bdist_wininst installers (more accurately, by the OS native package management), and another part by EasyInstall. On a related note, had you thought of having EasyInstall put eggs in a specific subdirectory? Something like site-packages/eggs? As you're using .pth files, this wouldn't be any harder to manage, would it? I can see a use case for a "combination" egg. One which packages together a group of packages. For example, if I want a standard set of packages installed on all machines, so that users can assume they are present when writing one-off scripts. If I could package *existing* eggs up into a combination site-std.egg (this is a pure aggregation operation, and should not require the ability to build extensions) then I could install the one egg across the site (and version that independently of the individual packages). There's a similar facility available in TCL's "starkits" (see http://www.equi4.com/starkit.html) - which are well worth a look for ideas, as they are fairly similar in concept to eggs. Paul. -- Once the game is over, the King and the pawn go back in the same box. -- Italian Proverb From pje at telecommunity.com Tue Jul 12 05:04:03 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 11 Jul 2005 23:04:03 -0400 Subject: [Distutils] setuptools for people behind a firewall In-Reply-To: <79990c6b05071108585afc3bcd@mail.gmail.com> Message-ID: <5.1.1.6.0.20050711225244.038cc890@mail.telecommunity.com> At 04:58 PM 7/11/2005 +0100, Paul Moore wrote: >I've just had a look at the new documentation for setuptools. I've not >read it all in detail yet, but one thing struck me regarding the >"automatically download dependencies" feature. > >It isn't going to work for people (like me) stuck behind a firewall >that Python doesn't support (Windows NTLM based firewall). Obviously, >setuptools is never going to be able to resolve a situation like this, >nor would I expect it to. Have you tried APS? (i.e., http://ntlmaps.sf.net/ ) Its pages seem to suggest it can authenticate to NTLM proxy servers like the one you're dealing with, and it sounds like a general-purpose solution to the proxying problem. The only issue is that you'd need to configure your system such that urllib considers the APS address to be the proxy to use, but then *all* Python apps (or any app that reads the same proxy configuration) will be able to get out past the firewall. >But can I suggest two possible changes to >make it easier for people with limited internet access? > >1. A "manual download" mode, where setuptools lists the files which it >wants you to obtain, and then leaves it to you how you get them. I'm >not sure how plausible this would be, given the necessarily iterative >process involved in resolving dependencies, but even a little help >would be useful (a report of unresolved dependencies when run with a >--no-download flag would be the most basic help). EasyInstall can't *find* the files without HTTP, since that's what it uses to talk to PyPI. So "manual download" mode would mean you'd effectively need to find all the files yourself! I'm not sure how much help EasyInstall could actually be in such a process. >2. A way of specifying an external command to use to download files >over HTTP. This would (for example) allow me to use curl, which does >support HTLM proxies, rather than relying on Python's built-in HTTP >support, which doesn't. This is potentially possible, but EasyInstall's PyPI searches and SourceForge download support need to be able to detect the MIME type of an HTTP response in order to decide whether it has an HTML page or not. If APS doesn't work for you, this might be an option, it's just a tricky one to implement in ez_setup.py, which wants to be small and simple and do no command-line parsing of its own, because it's also a module that gets imported by setup scripts. OTOH, perhaps an environment variable would be the way to go there, if we have to. From pje at telecommunity.com Tue Jul 12 05:09:27 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 11 Jul 2005 23:09:27 -0400 Subject: [Distutils] setuptools for people behind a firewall In-Reply-To: References: <79990c6b05071108585afc3bcd@mail.gmail.com> <6AA7FD94-7D9A-43E2-9372-5938A504FEB2@gmail.com> Message-ID: <5.1.1.6.0.20050711230451.01ac5b28@mail.telecommunity.com> At 08:54 PM 7/11/2005 +0100, Paul Moore wrote: >Ah, there's the difficulty, of course :-) That's where a utility to >report the dependencies would help... (It may be pretty trivial, but I >don't see immediately how to do this - it seems to me that the current >setuptools documentation is more for *creators* of eggs, than for >*users* of them...) Did you also look at these docs: http://peak.telecommunity.com/DevCenter/PythonEggs http://peak.telecommunity.com/DevCenter/EasyInstall Of course, the main reason for little documentation devoted to "users" of eggs is that really eggs should be a mostly transparent thing. As for dependency analysis, the current API to find eggs in a directory is pkg_resources.find_distributions(dirname_or_filename), which iterates over the egg(s), yielding Distribution objects. Distribution objects have a 'depends()' method which returns a list of Requirement objects describing what the distribution needs. With that API, you should easily be able to create a simple script to dump an egg's dependencies, although I think the API may change slightly in a future release. (e.g. depends() will probably have a different name before 1.0 rolls around.) From pje at telecommunity.com Tue Jul 12 05:09:54 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 11 Jul 2005 23:09:54 -0400 Subject: [Distutils] pkg_resources: getting to resources in other packages In-Reply-To: <3f085ecd05071113194399e4ed@mail.gmail.com> Message-ID: <5.1.1.6.0.20050711230933.027221d8@mail.telecommunity.com> At 04:19 PM 7/11/2005 -0400, Kevin Dangoor wrote: >In this particular instance, I'm going to go ahead and implement logic >to ensure that the package has been imported. But, I was figuring that >it might be nice if pkg_resource handled this... I'll put this on my list for the next release. From pje at telecommunity.com Tue Jul 12 05:34:53 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 11 Jul 2005 23:34:53 -0400 Subject: [Distutils] easy_install - some thoughts In-Reply-To: Message-ID: <5.1.1.6.0.20050711231010.038c2228@mail.telecommunity.com> At 10:32 PM 7/11/2005 +0100, Paul Moore wrote: >1. It would be much better if easy_install was usable via python -m, > either as well as or instead of the existing easy_install.py > script. I actually goofed in removing this; I'll put it back in the next release. >2. It would be useful if the -f option worked with a file: URL (or > better still, a simple directory name), to handle local > repositories. A good idea, which I've been thinking about. I'll add it in the next release, such that if an argument to -f is not a URL, it gets checked for being an existing directory. If it's a directory, I'll scan the contents as if they were links on a webpage. If it's not an existing directory, I'll print a warning (since errors should not pass silently) and proceed. >3. I still dislike the fact that I can't get a listing of installed > eggs. I suggest writing a simple script using pkg_resources.find_distributions(), and perhaps adding some documentation and contributing it to setuptools. :) >And uninstalling by deletion isn't good, IMHO. I am > *strongly* averse to using raw file operations (delete) on files > in Python\lib\site-packages. Elsewhere, fine, but I view the > Python standard library as a "managed" area, and management tools > should exist for it. That's the key benefit to me of bdist_wininst, > and I don't see eggs providing it. So write a pkg_delete script and contribute it, too. I've got no objection to these other tools, they just haven't been my top priority and aren't likely to be for a while. I'd like to focus on the core engine/API, in such a way that other people can easily write these sorts of management tools around it. >4. --dry-run doesn't work as expected: > > >\Apps\Python24\Scripts\easy_install.py --dry-run > dist/alarm-1.0-py2.4-win32.egg > >Processing alarm-1.0-py2.4-win32.egg >Copying alarm-1.0-py2.4-win32.egg to C:\Apps\Python24\Lib\site-packages >zipimport.ZipImportError: not a Zip file Sadly, this is about what *I* expect, because most distutils commands can only get so far into a --dry-run before some step depends on the nonexistent output of the preceding step. I've been making the assumption that it's more useful to have a dry run that tells you most of what it does before dying, than to have no dry-run capability at all. It's possible that I could make later steps say, "I'm in dry run mode and I don't know what the heck to do here", but I'm really not sure how I'd start. If you or someone else has patches, I'll be happy to look at incorporating them. Again, my focus for the next few releases is going to be on hardening pkg_resources to make it more robust and to have a more usable API. >A nice to have feature would be the ability for easy_install to pick >apart a bdist_wininst installer and build an egg from it. Then I guess it's a good thing I implemented it already. :) Try running EasyInstall on a win32.exe file, and you'll find that it actually does a pretty good job of turning it into an egg. If you were to run "easy_install pysqlite", for example, (assuming you could get past your firewall) you'd notice that it finds and downloads the bdist_wininst installer for PySQLite, converts it to an egg, and then installs it. This feature is only obscurely referenced in the docs and changelog, so I don't really blame you for missing it. :) >On a related note, had you thought of having EasyInstall put eggs in >a specific subdirectory? Something like site-packages/eggs? As you're >using .pth files, this wouldn't be any harder to manage, would it? Yes, it would, actually, especially for non-system installs, as Michael Hoffman was asking about here recently. The main reason for putting eggs in sys.path directories, however, is to support automatic discovery and adding them to sys.path at *runtime*, dynamically. I also wanted to take advantage of certain properties of sys.path, like the fact that Python's start script has its directory added to sys.path. This then makes eggs in the script directory easily discoverable. In fact, if you set an --install-dir with EasyInstall, and set the --always-copy flag, then your installation directory is an "application directory". Scripts and eggs are both copied there, including all eggs needed to run the application. Now you can just ship that directory and it can be used anywhere you have Python and setuptools installed. Anyway, I think putting eggs in directories on sys.path is a natural extension of the sys.path concept, that seems to me to avoid the mental overhead of having a separate search path that works differently. >I can see a use case for a "combination" egg. One which packages >together a group of packages. For example, if I want a standard set >of packages installed on all machines, so that users can assume they >are present when writing one-off scripts. If I could package >*existing* eggs up into a combination site-std.egg This is called a "basket" (as in putting all your eggs in one basket!), and it's actually already supported, as long as you put your hypothetical site-std.egg directly on sys.path. (It ought to be able to be just placed in a *directory* on sys.path, but when looking over the code just now I spotted a bug that prevents it from working unless you actually put the .egg file directly on sys.path. I'll fix this bug in the next release, so that dropping the egg in a sys.path directory is all you should need to do.) Anyway, the format of it is that you zip up the *unpacked* form of the eggs, directory names and all, so that if you have foo-1.5 and bar-2.7 then the interior of the combined egg needs to look like this: foo-1.5-py2.3-win32.egg/ # foo egg contents bar-2.7-py2.3-win32.egg/ # bar egg contents So, the combination process is trivial enough that you can do it by hand, or with a simple script. Just unzip the old eggs, if they weren't installed as directories to begin with, then zip the resulting directories. Anyway, if you end up with something nice and want to contribute it, that's great too. As with the other tools you've suggested, it's something I might get around to eventually, but there are some things I'm dying to get working first in the core engine, and I also have a lot of stuff I'd like to actually *use* the existing tools for right now. :) From pje at telecommunity.com Tue Jul 12 07:48:00 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 12 Jul 2005 01:48:00 -0400 Subject: [Distutils] setuptools 0.5a10 released with your bugs/features Message-ID: <5.1.1.6.0.20050712013849.0267e278@mail.telecommunity.com> * auto-import when requested resource's module isn't imported (Kevin Dangoor) * fix detection of "baskets" in sys.path directories (Paul Moore) * allow directories as arguments to --find-links (Paul Moore) * fix python -m easy_install not working (Paul Moore) Enjoy. (P.S. this also fixes some breakage w/buildutils 0.1.1 trying to import stuff from the 'easy_install' module. But note that buildutils doesn't seem to be smart enough about installing dependencies yet anyway, because trying to install pudge at runtime complains about it not being found in PyPI. I'm guessing that if Ryan's not going to put these modules up on PyPI, he needs to add some --find-links to the easy_install subcommand that buildutils runs to install a command's dependencies.) (P.P.S. That reminds me... Ryan, don't bother putting http://peak.telecommunity.com/dist/ in your --find-links list; now that I've got automatic PyPI uploading figured out, I plan to put future "stable" releases on PyPI, and snapshot releases (of projects other than setuptools itself) will go to http://peak.telecommunity.com/snapshots/ in the future.) From rtomayko at gmail.com Tue Jul 12 10:09:31 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Tue, 12 Jul 2005 04:09:31 -0400 Subject: [Distutils] setuptools 0.5a10 released with your bugs/features In-Reply-To: <5.1.1.6.0.20050712013849.0267e278@mail.telecommunity.com> References: <5.1.1.6.0.20050712013849.0267e278@mail.telecommunity.com> Message-ID: <8137DC99-4BE7-4C8F-A432-DE526501D3EA@gmail.com> On Jul 12, 2005, at 1:48 AM, Phillip J. Eby wrote: > (P.S. this also fixes some breakage w/buildutils 0.1.1 trying to > import > stuff from the 'easy_install' module. But note that buildutils > doesn't > seem to be smart enough about installing dependencies yet anyway, > because > trying to install pudge at runtime complains about it not being > found in > PyPI. I'm still working on an initial release of pudge. It shouldn't be too long now. > I'm guessing that if Ryan's not going to put these modules up on > PyPI, he needs to add some --find-links to the easy_install > subcommand that > buildutils runs to install a command's dependencies.) > > (P.P.S. That reminds me... Ryan, don't bother putting > http://peak.telecommunity.com/dist/ in your --find-links list; now > that > I've got automatic PyPI uploading figured out, I plan to put future > "stable" releases on PyPI, and snapshot releases (of projects other > than > setuptools itself) will go to http://peak.telecommunity.com/ > snapshots/ in > the future.) Will do. Thanks. Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From p.f.moore at gmail.com Tue Jul 12 10:50:17 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 12 Jul 2005 09:50:17 +0100 Subject: [Distutils] setuptools for people behind a firewall In-Reply-To: <5.1.1.6.0.20050711225244.038cc890@mail.telecommunity.com> References: <79990c6b05071108585afc3bcd@mail.gmail.com> <5.1.1.6.0.20050711225244.038cc890@mail.telecommunity.com> Message-ID: <79990c6b05071201507cccbc99@mail.gmail.com> On 7/12/05, Phillip J. Eby wrote: > At 04:58 PM 7/11/2005 +0100, Paul Moore wrote: > >I've just had a look at the new documentation for setuptools. I've not > >read it all in detail yet, but one thing struck me regarding the > >"automatically download dependencies" feature. > > > >It isn't going to work for people (like me) stuck behind a firewall > >that Python doesn't support (Windows NTLM based firewall). Obviously, > >setuptools is never going to be able to resolve a situation like this, > >nor would I expect it to. > > Have you tried APS? (i.e., http://ntlmaps.sf.net/ ) Its pages seem to > suggest it can authenticate to NTLM proxy servers like the one you're > dealing with, and it sounds like a general-purpose solution to the proxying > problem. The only issue is that you'd need to configure your system such > that urllib considers the APS address to be the proxy to use, but then > *all* Python apps (or any app that reads the same proxy configuration) will > be able to get out past the firewall. Yes, I have used APS and it is a reasonably good workaround. However, there is a definite disadvantage, in that it isn't set up to be launched as a service on Windows, which means that I can't have it "always running" (actually, having a permanently running proxy probably isn't that good an idea - I'm not enough of a security expert to be sure I haven't left a hole by doing so). So it tends to be left around, to run "when needed", which in practice means that when I do need it (pretty infrequently) I have to remember where it is, how to start it, etc etc. All of the above is fixable, but I don't have the time to do so, and the project seems pretty static, so I don't expect this sort of usability improvement to come from the project. But yes, I'll keep it in mind as an option. > This is potentially possible, but EasyInstall's PyPI searches and > SourceForge download support need to be able to detect the MIME type > of an HTTP response in order to decide whether it has an HTML page or not. curl -i includes the HTTP headers. But I understand that it's additional work for a very limited requirement, and there's a more general workaround available, so that's OK. Thanks for the explanation. Paul. From p.f.moore at gmail.com Tue Jul 12 11:23:47 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 12 Jul 2005 10:23:47 +0100 Subject: [Distutils] setuptools for people behind a firewall In-Reply-To: <5.1.1.6.0.20050711230451.01ac5b28@mail.telecommunity.com> References: <79990c6b05071108585afc3bcd@mail.gmail.com> <6AA7FD94-7D9A-43E2-9372-5938A504FEB2@gmail.com> <5.1.1.6.0.20050711230451.01ac5b28@mail.telecommunity.com> Message-ID: <79990c6b050712022316ce38d7@mail.gmail.com> On 7/12/05, Phillip J. Eby wrote: > At 08:54 PM 7/11/2005 +0100, Paul Moore wrote: > >Ah, there's the difficulty, of course :-) That's where a utility to > >report the dependencies would help... (It may be pretty trivial, but I > >don't see immediately how to do this - it seems to me that the current > >setuptools documentation is more for *creators* of eggs, than for > >*users* of them...) > > Did you also look at these docs: > > http://peak.telecommunity.com/DevCenter/PythonEggs > http://peak.telecommunity.com/DevCenter/EasyInstall I'd forgotten about these, but found them last night. I've not had a chance to read them through yet, but will do so. > Of course, the main reason for little documentation devoted to "users" of > eggs is that really eggs should be a mostly transparent thing. Hmm, yes. It's that "site-packages should be managed by tools" attitude of mine which muddies the waters a bit. Maybe it's a Windows thing... > As for dependency analysis, the current API to find eggs in a directory is > pkg_resources.find_distributions(dirname_or_filename), which iterates over > the egg(s), yielding Distribution objects. Distribution objects have a > 'depends()' method which returns a list of Requirement objects describing > what the distribution needs. With that API, you should easily be able to > create a simple script to dump an egg's dependencies, although I think the > API may change slightly in a future release. (e.g. depends() will probably > have a different name before 1.0 rolls around.) Cool. I feel the need for such a script (and possibly some others) so I'll have a go at an egg_utils script/module which I'll contribute back when I'm happy with it. Paul. From rtomayko at gmail.com Tue Jul 12 11:49:30 2005 From: rtomayko at gmail.com (Ryan Tomayko) Date: Tue, 12 Jul 2005 05:49:30 -0400 Subject: [Distutils] setuptools for people behind a firewall In-Reply-To: <79990c6b050712022316ce38d7@mail.gmail.com> References: <79990c6b05071108585afc3bcd@mail.gmail.com> <6AA7FD94-7D9A-43E2-9372-5938A504FEB2@gmail.com> <5.1.1.6.0.20050711230451.01ac5b28@mail.telecommunity.com> <79990c6b050712022316ce38d7@mail.gmail.com> Message-ID: On Jul 12, 2005, at 5:23 AM, Paul Moore wrote: > Hmm, yes. It's that "site-packages should be managed by tools" > attitude of mine which muddies the waters a bit. Maybe it's a Windows > thing... It's not. In many cases, it will be impossible for users without root/ wheel level permissions to modify the site-packages directory at all on *nix systems that have proper package management, which is just about everything now. Additionally, most Redhat/Fedora admins will tell you that having anything other than rpm touching the RPM managed site-packages directory is bad form (there's actually good reasons for this), even when you do have permissions (I'm not sure how prevalent this mindset is outside the Redhat/Fedora community). I think that many people running systems built on package management will want to setup an alternative install_dir (/usr/local/lib/ pythonX.X/site-packages or perhaps /opt/pythonX.X/site-packages). This is why I think the only-load-pth-from-site-packages issue is important. Then again, wrapper scripts make it less of an issue. teeder-totter-teeder-totter I'm still waiting to see how people maintaining python packages for Linux/BSD distributions feel about eggs. So far I don't think they've had to think about it a whole lot. Ryan Tomayko rtomayko at gmail.com http://naeblis.cx/rtomayko/ From p.f.moore at gmail.com Tue Jul 12 12:11:23 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 12 Jul 2005 11:11:23 +0100 Subject: [Distutils] easy_install - some thoughts In-Reply-To: <5.1.1.6.0.20050711231010.038c2228@mail.telecommunity.com> References: <5.1.1.6.0.20050711231010.038c2228@mail.telecommunity.com> Message-ID: <79990c6b050712031170cc2803@mail.gmail.com> On 7/12/05, Phillip J. Eby wrote: > At 10:32 PM 7/11/2005 +0100, Paul Moore wrote: > >2. It would be useful if the -f option worked with a file: URL (or > > better still, a simple directory name), to handle local > > repositories. > > A good idea, which I've been thinking about. I'll add it in the next > release, such that if an argument to -f is not a URL, it gets checked for > being an existing directory. If it's a directory, I'll scan the contents > as if they were links on a webpage. If it's not an existing directory, > I'll print a warning (since errors should not pass silently) and proceed. Agreed. Thanks. > >3. I still dislike the fact that I can't get a listing of installed > > eggs. > > I suggest writing a simple script using pkg_resources.find_distributions(), > and perhaps adding some documentation and contributing it to setuptools. :) > > > >And uninstalling by deletion isn't good, IMHO. > > So write a pkg_delete script and contribute it, too. See my earlier message. I will do so. > I've got no objection to these other tools, they just haven't been my top > priority and aren't likely to be for a while. I'd like to focus on the core > engine/API, in such a way that other people can easily write these > sorts of management tools around it. Great. I'll see what I can do to contribute in that area. > >4. --dry-run doesn't work as expected: [...] > Sadly, this is about what *I* expect, [...] > I've been making the assumption that it's more useful to have a dry run > that tells you most of what it does before dying, than to have no dry-run > capability at all. My reason for using --dry-run was because while I'm happy to do the build steps in a work area, I don't want to do the install until I see what will happen (just in case I've got something wrong!) So for me, dry-run capability is only really crucial for the install step. I'd actually done all of the previous steps (the build bits) for real in any case, so the dry run did nothing before failing... But I see what you mean, it's standard distutils machinery so you haven't got much freedom to change it, and ultimately when enough things aren't done, the remaining steps are meaningless anyway. > >A nice to have feature would be the ability for easy_install to pick > >apart a bdist_wininst installer and build an egg from it. > > Then I guess it's a good thing I implemented it already. :) Cool! Now give back the time machine... > Try running EasyInstall on a win32.exe file, and you'll find that it > actually does a pretty good job of turning it into an egg. If you were to run > "easy_install pysqlite", for example, (assuming you could get past your > firewall) you'd notice that it finds and downloads the bdist_wininst > installer for PySQLite, converts it to an egg, and then installs it. This > feature is only obscurely referenced in the docs and changelog, so I don't > really blame you for missing it. :) Quick question, is there a "convert but don't install yet" option? So I get an egg on my PC, but not in site-packages. > >On a related note, had you thought of having EasyInstall put eggs in > >a specific subdirectory? Something like site-packages/eggs? As you're > >using .pth files, this wouldn't be any harder to manage, would it? > > Yes, it would, actually, especially for non-system installs, as Michael > Hoffman was asking about here recently. The main reason for putting eggs > in sys.path directories, however, is to support automatic discovery and > adding them to sys.path at *runtime*, dynamically. I also wanted to take > advantage of certain properties of sys.path, like the fact that Python's > start script has its directory added to sys.path. This then makes eggs in > the script directory easily discoverable. > > In fact, if you set an --install-dir with EasyInstall, and set the > --always-copy flag, then your installation directory is an "application > directory". Scripts and eggs are both copied there, including all eggs > needed to run the application. Now you can just ship that directory and it > can be used anywhere you have Python and setuptools installed. > > Anyway, I think putting eggs in directories on sys.path is a natural > extension of the sys.path concept, that seems to me to avoid the mental > overhead of having a separate search path that works differently. > > > >I can see a use case for a "combination" egg. One which packages > >together a group of packages. For example, if I want a standard set > >of packages installed on all machines, so that users can assume they > >are present when writing one-off scripts. If I could package > >*existing* eggs up into a combination site-std.egg > > This is called a "basket" (as in putting all your eggs in one basket!), and > it's actually already supported, I told you - give back that time machine! :-) Seriously, I'm getting more and more impressed... > Anyway, if you end up with something nice and want to contribute it, that's > great too. As with the other tools you've suggested, it's something I > might get around to eventually, but there are some things I'm dying to get > working first in the core engine, and I also have a lot of stuff I'd like > to actually *use* the existing tools for right now. :) Will do. I'm much closer to having a feel for where I'd be adding value and not duplicating effort, so I'll certainly see what I can do. Thanks for answering my dumb questions. Paul. From micktwomey at gmail.com Tue Jul 12 13:01:45 2005 From: micktwomey at gmail.com (Michael Twomey) Date: Tue, 12 Jul 2005 12:01:45 +0100 Subject: [Distutils] Problem with setuptools >=0.5a9 and develop command Message-ID: <50a522ca0507120401144f1f4b@mail.gmail.com> Hi, I've been busily converting some of my smaller servers to eggs, and I've hit a problem with the develop command. From setuptools 0.5a9 onwards the develop command barfs with the following traceback: $ python2.4 setup.py develop --help Traceback (most recent call last): File "setup.py", line 33, in ? install_requires = [ File "/usr/lib/python2.4/site-packages/setuptools-0.5a10-py2.4.egg/setuptools/__init__.py", line 51, in setup File "/usr/lib/python2.4/distutils/core.py", line 135, in setup ok = dist.parse_command_line() File "/usr/lib/python2.4/site-packages/setuptools-0.5a10-py2.4.egg/setuptools/dist.py", line 107, in parse_command_line File "/usr/lib/python2.4/distutils/dist.py", line 432, in parse_command_line args = self._parse_command_opts(parser, args) File "/usr/lib/python2.4/site-packages/setuptools-0.5a10-py2.4.egg/setuptools/dist.py", line 385, in _parse_command_opts File "/usr/lib/python2.4/distutils/dist.py", line 530, in _parse_command_opts parser.set_negative_aliases(negative_opt) File "/usr/lib/python2.4/distutils/fancy_getopt.py", line 143, in set_negative_aliases self._check_alias_dict(negative_alias, "negative alias") File "/usr/lib/python2.4/distutils/fancy_getopt.py", line 125, in _check_alias_dict raise DistutilsGetoptError, \ distutils.errors.DistutilsGetoptError: invalid negative alias 'always-unzip': option 'always-unzip' not defined Looking at commands/develop.py I notice that it's the only command which sets 'command_consumes_arguments = False', and it also doesn't override the negative_opt dict. I tried setting 'negative_opt = {}' and it started working again. So this patch "fixes" the issue though I don't know the internals of distutils/setuptools to tell you exactly why: --- /tmp/setuptools-0.5a10/setuptools/command/develop.py 2005-07-06 19:15:52.000000000 +0100 +++ setuptools/command/develop.py 2005-07-12 11:50:04.000000000 +0100 @@ -21,6 +21,8 @@ boolean_options = [ 'multi-version', 'exclude-scripts', 'always-copy', 'uninstall' ] + + negative_opt = {} command_consumes_arguments = False # override base Is anyone else seeing this problem? I've reproduced it on a debian linux box and my apple laptop. cheers, Michael From ianb at colorstudy.com Tue Jul 12 17:50:12 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 12 Jul 2005 10:50:12 -0500 Subject: [Distutils] setuptools for people behind a firewall In-Reply-To: References: <79990c6b05071108585afc3bcd@mail.gmail.com> <6AA7FD94-7D9A-43E2-9372-5938A504FEB2@gmail.com> <5.1.1.6.0.20050711230451.01ac5b28@mail.telecommunity.com> <79990c6b050712022316ce38d7@mail.gmail.com> Message-ID: <42D3E6B4.3040504@colorstudy.com> Ryan Tomayko wrote: > I'm still waiting to see how people maintaining python packages for > Linux/BSD distributions feel about eggs. So far I don't think they've > had to think about it a whole lot. Hopefully this will make package generation much much easier -- the only issue I can see is the mismatch in the names of packages and dependencies (since generally package names are parameterized with "py-" or "python-" or somesuch). Actually putting the one file in its place should be easy, of course. Well... the other issue is that eggs support the installation of multiple versions simultaneously; for most packaging systems to support this the package name has to grow a version number too, which is rather awkward. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Tue Jul 12 18:01:42 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 12 Jul 2005 12:01:42 -0400 Subject: [Distutils] easy_install - some thoughts In-Reply-To: <79990c6b050712031170cc2803@mail.gmail.com> References: <5.1.1.6.0.20050711231010.038c2228@mail.telecommunity.com> <5.1.1.6.0.20050711231010.038c2228@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050712115839.0268a400@mail.telecommunity.com> At 11:11 AM 7/12/2005 +0100, Paul Moore wrote: >Quick question, is there a "convert but don't install yet" option? So >I get an egg on my PC, but not in site-packages. Yes, it's called --install-dir, and you set it to the directory where you'd like it to dump the egg. :) If you don't want script wrappers in that directory too, use --exclude-scripts as well. So, "easy_install -xd somedir pysqlite" would dump the pysqlite egg in "somedir". Or, if the .exe is on your machine already then "easy_install -xd somedir somepackage.win32.exe" will do the trick. (Note that EasyInstall only manipulates .pth files when installing to site-packages, so this should do exactly what you want.) From pje at telecommunity.com Tue Jul 12 18:10:28 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 12 Jul 2005 12:10:28 -0400 Subject: [Distutils] Problem with setuptools >=0.5a9 and develop command In-Reply-To: <50a522ca0507120401144f1f4b@mail.gmail.com> Message-ID: <5.1.1.6.0.20050712120324.0267c758@mail.telecommunity.com> At 12:01 PM 7/12/2005 +0100, Michael Twomey wrote: >Looking at commands/develop.py I notice that it's the only command >which sets 'command_consumes_arguments = False', and it also doesn't >override the negative_opt dict. I tried setting 'negative_opt = {}' >and it started working again. Yeah, it looks like I broke "develop" when I added the '--always-unzip' option to the "easy_install" command (which "develop" subclasses). That's the sucky thing about not having automated tests. Thanks for the bug report, and your fix is indeed the correct one. It never occurred to me that adding an option to the base class would affect the subclass that way. I'm pushing out 0.5a11 now to fix this. Sorry about that. From micktwomey at gmail.com Tue Jul 12 18:18:57 2005 From: micktwomey at gmail.com (Michael Twomey) Date: Tue, 12 Jul 2005 17:18:57 +0100 Subject: [Distutils] Problem with setuptools >=0.5a9 and develop command In-Reply-To: <5.1.1.6.0.20050712120324.0267c758@mail.telecommunity.com> References: <50a522ca0507120401144f1f4b@mail.gmail.com> <5.1.1.6.0.20050712120324.0267c758@mail.telecommunity.com> Message-ID: <50a522ca050712091842e9df48@mail.gmail.com> Great, thanks. Michael On 7/12/05, Phillip J. Eby wrote: > At 12:01 PM 7/12/2005 +0100, Michael Twomey wrote: > > >Looking at commands/develop.py I notice that it's the only command > >which sets 'command_consumes_arguments = False', and it also doesn't > >override the negative_opt dict. I tried setting 'negative_opt = {}' > >and it started working again. > > Yeah, it looks like I broke "develop" when I added the '--always-unzip' > option to the "easy_install" command (which "develop" subclasses). That's > the sucky thing about not having automated tests. Thanks for the bug > report, and your fix is indeed the correct one. It never occurred to me > that adding an option to the base class would affect the subclass that way. > > I'm pushing out 0.5a11 now to fix this. Sorry about that. > > From p.f.moore at gmail.com Tue Jul 12 20:37:30 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 12 Jul 2005 19:37:30 +0100 Subject: [Distutils] easy_install - some thoughts In-Reply-To: <5.1.1.6.0.20050712115839.0268a400@mail.telecommunity.com> References: <5.1.1.6.0.20050711231010.038c2228@mail.telecommunity.com> <79990c6b050712031170cc2803@mail.gmail.com> <5.1.1.6.0.20050712115839.0268a400@mail.telecommunity.com> Message-ID: <79990c6b0507121137e089153@mail.gmail.com> On 7/12/05, Phillip J. Eby wrote: > At 11:11 AM 7/12/2005 +0100, Paul Moore wrote: > >Quick question, is there a "convert but don't install yet" option? So > >I get an egg on my PC, but not in site-packages. > > Yes, it's called --install-dir, and you set it to the directory where you'd > like it to dump the egg. :) If you don't want script wrappers in that > directory too, use --exclude-scripts as well. So, "easy_install -xd > somedir pysqlite" would dump the pysqlite egg in "somedir". Neat, thanks. By the way, I don't know if the sqlite web page has changed, but your standard example of sqlite doesn't work for me: >\Apps\Python24\Scripts\easy_install.py -xd test sqlite Searching for sqlite Reading http://www.python.org/pypi/sqlite/ Reading http://pysqlite.org/ No local packages or download links found for sqlite error: Could not find distribution for Requirement('sqlite', [], ()) I tried a few others - cx_Oracle, ctypes, kinterbasdb, python-dateutil, and they didn't work either. PIL *nearly* did - it found the PIL 1.1.5 Windows installer for Python 2.4, but it then failed with error: c:\docume~1\gustav\locals~1\temp\easy_install-k1vu-u\PIL-1.1.5.win32-py2.4.exe is not a valid distutils Windows .exe which is probably wrong, as the URL was correct and the file is a Python installer, but I can't diagnose much further as the temp file was deleted - could it be binary/text file issues? Actually, if I download the file manually and do "easy_install.py -xd test PIL-1.1.5.win32-py2.4.exe" I still get the same error, so it's deeper than that... I can't think of much more I can do to diagnose this for you, but let me know if there's anything you'd like me to try. This is with setuptools-0.5a9, btw, I haven't had time to download the new version yet (there's no Windows installer, so I'll have to use the egg, which shouldn't be hard but I'd rather not compound the issues by using a mis-installed package :-)) Paul.. From dangoor at gmail.com Tue Jul 12 20:45:23 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Tue, 12 Jul 2005 14:45:23 -0400 Subject: [Distutils] easy_install - some thoughts In-Reply-To: <79990c6b0507121137e089153@mail.gmail.com> References: <5.1.1.6.0.20050711231010.038c2228@mail.telecommunity.com> <79990c6b050712031170cc2803@mail.gmail.com> <5.1.1.6.0.20050712115839.0268a400@mail.telecommunity.com> <79990c6b0507121137e089153@mail.gmail.com> Message-ID: <3f085ecd0507121145423cef4@mail.gmail.com> On 7/12/05, Paul Moore wrote: > By the way, I don't know if the sqlite web page has changed, but your > standard example of sqlite doesn't work for me: It may be pysqlite. > This is with setuptools-0.5a9, btw, I haven't had time to download the > new version yet (there's no Windows installer, so I'll have to use the > egg, which shouldn't be hard but I'd rather not compound the issues by > using a mis-installed package :-)) easy_install -U setuptools is the easiest way to upgrade. And, once you have the new version, you will be able to do python -m easy_install -U setuptools to keep up to date. Kevin From p.f.moore at gmail.com Tue Jul 12 21:41:14 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 12 Jul 2005 20:41:14 +0100 Subject: [Distutils] easy_install - some thoughts In-Reply-To: <3f085ecd0507121145423cef4@mail.gmail.com> References: <5.1.1.6.0.20050711231010.038c2228@mail.telecommunity.com> <79990c6b050712031170cc2803@mail.gmail.com> <5.1.1.6.0.20050712115839.0268a400@mail.telecommunity.com> <79990c6b0507121137e089153@mail.gmail.com> <3f085ecd0507121145423cef4@mail.gmail.com> Message-ID: <79990c6b050712124167224b6f@mail.gmail.com> On 7/12/05, Kevin Dangoor wrote: > easy_install -U setuptools > > is the easiest way to upgrade. And, once you have the new version, you > will be able to do python -m easy_install -U setuptools to keep up to > date. That conflicts with a bdist_wininst installation. Shouldn't be too hard to sort out, but there's a little bit of a chicken and egg issue here :-) Actually, ez_setup.py works fine on Windows - so that may well be a better option for Windows users with network access. However, if setuptools is installed as an egg, python -m easy_install doesn't work (and putting the egg on PYTHONPATH, or even renaming it as .zip didn't help - this may be a bug in Python's -m support...) Paul. From njriley at uiuc.edu Tue Jul 12 22:06:26 2005 From: njriley at uiuc.edu (Nicholas Riley) Date: Tue, 12 Jul 2005 15:06:26 -0500 Subject: [Distutils] ez_setup doesn't finish partial install Message-ID: <20050712200626.GA30024@uiuc.edu> I was trying to install setuptools, and I ended up running into a permissions error: % MACOSX_DEPLOYMENT_TARGET=10.4 python ez_setup.py Downloading http://www.python.org/packages/source/s/setuptools/setuptools-0.5a11-py2.3.egg.zip Processing setuptools-0.5a11-py2.3.egg Copying setuptools-0.5a11-py2.3.egg to /System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages Adding setuptools 0.5a11 to easy-install.pth file Installing easy_install.py script to /System/Library/Frameworks/Python.framework/Versions/2.3/bin error: /System/Library/Frameworks/Python.framework/Versions/2.3/bin/easy_install.py: Permission denied So I fixed permissions and tried again: % MACOSX_DEPLOYMENT_TARGET=10.4 python ez_setup.py Setuptools successfully installed or upgraded. But after the above, I still don't have an easy_install.py script. I had to delete the easy-install and setuptools .egg and .pth files from /Library/Python/2.3/site-packages to get it to install easy_install.py. -- Nicholas Riley | From bob at redivi.com Tue Jul 12 22:44:54 2005 From: bob at redivi.com (Bob Ippolito) Date: Tue, 12 Jul 2005 10:44:54 -1000 Subject: [Distutils] ez_setup doesn't finish partial install In-Reply-To: <20050712200626.GA30024@uiuc.edu> References: <20050712200626.GA30024@uiuc.edu> Message-ID: <6C473B28-F5F7-4475-A3D2-9C0EAD216764@redivi.com> On Jul 12, 2005, at 10:06 AM, Nicholas Riley wrote: > I was trying to install setuptools, and I ended up running into a > permissions error: > > % MACOSX_DEPLOYMENT_TARGET=10.4 python ez_setup.py > Downloading http://www.python.org/packages/source/s/setuptools/ > setuptools-0.5a11-py2.3.egg.zip > Processing setuptools-0.5a11-py2.3.egg > Copying setuptools-0.5a11-py2.3.egg to /System/Library/Frameworks/ > Python.framework/Versions/2.3/lib/python2.3/site-packages > Adding setuptools 0.5a11 to easy-install.pth file > Installing easy_install.py script to /System/Library/Frameworks/ > Python.framework/Versions/2.3/bin > error: /System/Library/Frameworks/Python.framework/Versions/2.3/bin/ > easy_install.py: Permission denied The default location for binaries is in-framework, which is almost always the last place you want to have it. Especially for the System Python. You want to override this default like bdist_mpkg does. I don't know the ez_setup syntax for it, but the distutils install command option is --install-scripts=/usr/local/bin -bob From p.f.moore at gmail.com Tue Jul 12 23:11:40 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 12 Jul 2005 22:11:40 +0100 Subject: [Distutils] easy_install - some thoughts In-Reply-To: <3f085ecd0507121145423cef4@mail.gmail.com> References: <5.1.1.6.0.20050711231010.038c2228@mail.telecommunity.com> <79990c6b050712031170cc2803@mail.gmail.com> <5.1.1.6.0.20050712115839.0268a400@mail.telecommunity.com> <79990c6b0507121137e089153@mail.gmail.com> <3f085ecd0507121145423cef4@mail.gmail.com> Message-ID: <79990c6b05071214113e8cb082@mail.gmail.com> On 7/12/05, Kevin Dangoor wrote: > On 7/12/05, Paul Moore wrote: > > By the way, I don't know if the sqlite web page has changed, but your > > standard example of sqlite doesn't work for me: > > It may be pysqlite. It's not, it's Python 2.4 :-( It looks like setuptools.command.easy_install is doing some ugly hacks to read data out of the raw bdist_wininst file, in extract_wininst_cfg, and the file format has changed in Python 2.4. For 2.4, it looks like the "tag" has just changed from 0x1234567A to 0x1234567B... So just checking both values should work. I've attached a patch. Still doesn't work, as I get an error about the (not-)zip-safe file: >python -m easy_install -xd eggs Python\ctypes-0.9.6.win32-py2.4.exe Processing ctypes-0.9.6.win32-py2.4.exe ctypes.__init__: module references __file__ ctypes.com.register: module references __file__ error: c:\docume~1\gustav\locals~1\temp\easy_install-unhoeu\ctypes-0.9.6-py2.4-win32.egg.tmp\ EGG-INFO\not-zip-safe: No such file or directory It looks like the issue is in bdist_egg.py, write_safety_flag, where an ensure_directory() call is needed. There's a patch for this below, as well. But even then, things are going odd, as if I try to build an egg for ctypes (which isn't zip-safe) all of the files and subdirectories in the egg directory are missing the first two characters... Hmm, it happens with zip-safe eggs, as well, so I don't know - maybe it's related to the bdist_wininst unpacking code. Sorry, I can't debug this any further. BTW, it would be useful to have some type of debug flag which left the temporary directory around for debugging... Paul. --- setuptools\command\easy_install.py.orig 2005-07-10 22:26:46.000000000 +0100 +++ setuptools\command\easy_install.py 2005-07-12 21:30:04.718750000 +0100 @@ -877,7 +877,7 @@ import struct, StringIO, ConfigParser tag, cfglen, bmlen = struct.unpack("0x1234567A: + if tag<>0x1234567A and tag<>0x1234567B: return None # not a valid tag f.seek(prepended-(12+cfglen+bmlen)) --- \Apps\Python24\Lib\site-packages\setuptools\command\bdist_egg.py.orig 2005-07-10 23:32:28.000000000 +0100 +++ \Apps\Python24\Lib\site-packages\setuptools\command\bdist_egg.py 2005-07-12 21:55:35.781250000 +0100 @@ -8,7 +8,7 @@ from distutils.dir_util import remove_tree, mkpath from distutils.sysconfig import get_python_version, get_python_lib from distutils import log -from pkg_resources import get_platform, Distribution +from pkg_resources import get_platform, Distribution, ensure_directory from types import CodeType def write_stub(resource, pyfile): @@ -315,7 +315,9 @@ def write_safety_flag(egg_dir, safe): # Write a zip safety flag file flag = safe and 'zip-safe' or 'not-zip-safe' - open(os.path.join(egg_dir,'EGG-INFO',flag),'w').close() + filename = os.path.join(egg_dir,'EGG-INFO',flag) + ensure_directory(filename) + open(filename, 'w').close() From pje at telecommunity.com Wed Jul 13 00:13:17 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 12 Jul 2005 18:13:17 -0400 Subject: [Distutils] easy_install - some thoughts In-Reply-To: <79990c6b0507121137e089153@mail.gmail.com> References: <5.1.1.6.0.20050712115839.0268a400@mail.telecommunity.com> <5.1.1.6.0.20050711231010.038c2228@mail.telecommunity.com> <79990c6b050712031170cc2803@mail.gmail.com> <5.1.1.6.0.20050712115839.0268a400@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050712181220.026aa250@mail.telecommunity.com> At 07:37 PM 7/12/2005 +0100, Paul Moore wrote: >This is with setuptools-0.5a9, btw, I haven't had time to download the >new version yet (there's no Windows installer, so I'll have to use the >egg, which shouldn't be hard but I'd rather not compound the issues by >using a mis-installed package :-)) After a9 I got rid of the Windows installer, as I forgot that it was installing an "unmanaged" version of setuptools. From pje at telecommunity.com Wed Jul 13 00:15:01 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 12 Jul 2005 18:15:01 -0400 Subject: [Distutils] easy_install - some thoughts In-Reply-To: <79990c6b050712124167224b6f@mail.gmail.com> References: <3f085ecd0507121145423cef4@mail.gmail.com> <5.1.1.6.0.20050711231010.038c2228@mail.telecommunity.com> <79990c6b050712031170cc2803@mail.gmail.com> <5.1.1.6.0.20050712115839.0268a400@mail.telecommunity.com> <79990c6b0507121137e089153@mail.gmail.com> <3f085ecd0507121145423cef4@mail.gmail.com> Message-ID: <5.1.1.6.0.20050712181338.0268f388@mail.telecommunity.com> At 08:41 PM 7/12/2005 +0100, Paul Moore wrote: >However, if setuptools is installed as an egg, python -m easy_install >doesn't work (and putting the egg on PYTHONPATH, or even renaming it >as .zip didn't help - this may be a bug in Python's -m support...) Darnit. I forgot about this issue. You can't run a script-in-a-zip with 'python -m', which means that I need to make setuptools itself be "not zip safe". Which sucks, since I'd like it to be a poster child for zip safety, but oh well. From pje at telecommunity.com Wed Jul 13 00:42:17 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 12 Jul 2005 18:42:17 -0400 Subject: [Distutils] ez_setup doesn't finish partial install In-Reply-To: <20050712200626.GA30024@uiuc.edu> Message-ID: <5.1.1.6.0.20050712181512.0298a4f8@mail.telecommunity.com> At 03:06 PM 7/12/2005 -0500, Nicholas Riley wrote: >I was trying to install setuptools, and I ended up running into a >permissions error: > >% MACOSX_DEPLOYMENT_TARGET=10.4 python ez_setup.py >Downloading >http://www.python.org/packages/source/s/setuptools/setuptools-0.5a11-py2.3.egg.zip >Processing setuptools-0.5a11-py2.3.egg >Copying setuptools-0.5a11-py2.3.egg to >/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages >Adding setuptools 0.5a11 to easy-install.pth file >Installing easy_install.py script to >/System/Library/Frameworks/Python.framework/Versions/2.3/bin >error: >/System/Library/Frameworks/Python.framework/Versions/2.3/bin/easy_install.py: >Permission denied > >So I fixed permissions and tried again: > >% MACOSX_DEPLOYMENT_TARGET=10.4 python ez_setup.py >Setuptools successfully installed or upgraded. > >But after the above, I still don't have an easy_install.py script. I >had to delete the easy-install and setuptools .egg and .pth files from >/Library/Python/2.3/site-packages to get it to install >easy_install.py. FYI, you can run "ez_setup -U setuptools" to force an upgrade/reinstall. I've added a message about this after the "installed or upgraded" message, that will appear in the next release. From pje at telecommunity.com Wed Jul 13 00:28:09 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 12 Jul 2005 18:28:09 -0400 Subject: [Distutils] ez_setup doesn't finish partial install In-Reply-To: <6C473B28-F5F7-4475-A3D2-9C0EAD216764@redivi.com> References: <20050712200626.GA30024@uiuc.edu> <20050712200626.GA30024@uiuc.edu> Message-ID: <5.1.1.6.0.20050712182624.03764878@mail.telecommunity.com> At 10:44 AM 7/12/2005 -1000, Bob Ippolito wrote: >You want to override this default like bdist_mpkg does. I don't know >the ez_setup syntax for it, but the distutils install command option >is --install-scripts=/usr/local/bin It's also the option for easy_install, and easy_install will take its default from the same place the distutils does; so you can do something like this: [install] install_scripts = /usr/local/bin in ~/.pydistutils.cfg and it will work for all distutils installs as well as EasyInstall. From pje at telecommunity.com Wed Jul 13 00:40:29 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 12 Jul 2005 18:40:29 -0400 Subject: [Distutils] easy_install - some thoughts In-Reply-To: <79990c6b05071214113e8cb082@mail.gmail.com> References: <3f085ecd0507121145423cef4@mail.gmail.com> <5.1.1.6.0.20050711231010.038c2228@mail.telecommunity.com> <79990c6b050712031170cc2803@mail.gmail.com> <5.1.1.6.0.20050712115839.0268a400@mail.telecommunity.com> <79990c6b0507121137e089153@mail.gmail.com> <3f085ecd0507121145423cef4@mail.gmail.com> Message-ID: <5.1.1.6.0.20050712182814.0299c538@mail.telecommunity.com> At 10:11 PM 7/12/2005 +0100, Paul Moore wrote: >On 7/12/05, Kevin Dangoor wrote: > > On 7/12/05, Paul Moore wrote: > > > By the way, I don't know if the sqlite web page has changed, but your > > > standard example of sqlite doesn't work for me: > > > > It may be pysqlite. > >It's not, it's Python 2.4 :-( Aha. :) >It looks like setuptools.command.easy_install is doing some ugly hacks >to read data out of the raw bdist_wininst file, in Well, no uglier than the hacks that put them in there. :) >extract_wininst_cfg, and the file format has changed in Python 2.4. > >For 2.4, it looks like the "tag" has just changed from 0x1234567A to >0x1234567B... So just checking both values should work. I've attached >a patch. That's not the end of the difference, I'm afraid. See below. >Still doesn't work, as I get an error about the (not-)zip-safe file: > > >python -m easy_install -xd eggs Python\ctypes-0.9.6.win32-py2.4.exe >Processing ctypes-0.9.6.win32-py2.4.exe >ctypes.__init__: module references __file__ >ctypes.com.register: module references __file__ >error: >c:\docume~1\gustav\locals~1\temp\easy_install-unhoeu\ctypes-0.9.6-py2.4-win32.egg.tmp\ >EGG-INFO\not-zip-safe: No such file or directory > >It looks like the issue is in bdist_egg.py, write_safety_flag, where >an ensure_directory() call is needed. There's a patch for this below, >as well. Actually, the problem is that install_exe() isn't creating the EGG-INFO directory in the first place, but your fix is okay as a workaround. (The difference is that if converting from exe->egg ever needs any other metadata files, the problem could recur; but by creating EGG-INFO up front in install_exe(), creating other metadata files should be fine.) >But even then, things are going odd, as if I try to build an egg for >ctypes (which isn't zip-safe) all of the files and subdirectories in >the egg directory are missing the first two characters... Hmm, it >happens with zip-safe eggs, as well, so I don't know - maybe it's >related to the bdist_wininst unpacking code. Sorry, I can't debug this >any further. And I can't reproduce it. However, I have noticed that the format change for Python 2.4 includes a change to what data is in the 'cfgdata' string, and so I changed this line: cfg.readfp(StringIO.StringIO(f.read(cfglen))) to read: cfg.readfp(StringIO.StringIO(f.read(cfglen).split(chr(0),1)[0])) In order to ignore anything after the first NUL character in the cfgdata. I don't know if that was causing your two-character problem or not, as I can't reproduce it with Python 2.4 and the ctypes .exe for Python 2.4. >BTW, it would be useful to have some type of debug flag which left the >temporary directory around for debugging... See the --build-dir/-b option, which is what I used to do my own investigation of the above. The time machine strikes again! :) Unfortunately, the time machine won't let me fix bugs because it would create an infinite time loop; if I only know about the bug because you report it, then going back to fix the bug would mean you wouldn't report it, and so on. After the first thousand levels of recursion, the time machine throws a RuntimeError("Paradox stablization loop limit exceeded") and kicks me back to the present. :) From pje at telecommunity.com Wed Jul 13 02:45:38 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 12 Jul 2005 20:45:38 -0400 Subject: [Distutils] setuptools 0.5a12 Message-ID: <5.1.1.6.0.20050712202139.0378be38@mail.telecommunity.com> Hopefully, this is the last of the current spew of bugfix releases, although in this case I also threw in some revisions to pkg_resources' handling of resource extraction. You can now control the cache explicitly using a PYTHON_EGGS_CACHE variable, and on Windows the default if you don't specify it is now a "Python-Eggs" directory under the current user's "application data" directory. Also, extraction takes place to a temporary filename that is then renamed to the final name, in order to prevent race conditions with other threads or processes. So, assuming this change works well, I will start making the zip safety checks a little less stringent, so that mere use of C extensions or data files won't disqualify an egg from being installed in zipped form. However, that will be in the 0.6 series of releases, which will be focused mainly on getting both pkg_resources and the entire easy_install family of code under a solid test regime. I'm currently releasing as many new bugs as I am new features, so that's got to stop right away. My current plan is to see if I can jury-rig doctest to do tests on the output of various easy_install commands, running them in various temporary environments. The result of this process may be that EasyInstall ends up with additional options to control testing behavior. (For example, I can't test easy-install.pth manipulation by installing to a temporary directory!) As part of the move to full test automation, I'll also be refactoring pkg_resources to match the design I posted here a while back, and implementing core engine features like Ryan's "legacy.py" option, my "add_activation_listener()", etc. The PythonEggs page could really use some updating, too, to catch up even to the current API, let alone the future one. It also has some bits that should be removed since they document manual ways to do things that setuptools does automatically. (e.g. creating a 'depends.txt' file when setuptools now just uses the 'install_requires' and 'extras_require' keywords. From dangoor at gmail.com Wed Jul 13 04:51:44 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Tue, 12 Jul 2005 22:51:44 -0400 Subject: [Distutils] py2app and eggs part deux Message-ID: <3f085ecd050712195157300596@mail.gmail.com> I really want to use eggs, but I *have* to use py2app. Basically, the issues that I see are: 1) modulegraph needs to know about eggs so that it can track down the dependencies of things in the eggs 2) the eggs should either be put into the site-packages.zip file in the bundle in "basket" form, or should be tossed directly into the site-packages directory. For the first point, I'm curious if modulegraph already knows about zip packages. I didn't see anything in the code that immediately jumped out at me saying "hey! this package is zipped up". What I had seen was that the contents of eggs were just being ignored. If module graph does have some knowledge about zip libraries, that would probably translate well. As for #2, the ideal would probably be to put everything in basket form in site-packages.zip. I figure that by opening up the eggs and only pulling out the referenced parts, the overall app can be smaller. Any package resources need to get in there, though. For expedience, though, I'm quite find with a little bit larger app that has the .egg files stored directly in site-packages with a .pth file. Any commentary on the difficulty involved here? (Bob?) Thanks, Kevin From bob at redivi.com Wed Jul 13 05:04:37 2005 From: bob at redivi.com (Bob Ippolito) Date: Tue, 12 Jul 2005 17:04:37 -1000 Subject: [Distutils] py2app and eggs part deux In-Reply-To: <3f085ecd050712195157300596@mail.gmail.com> References: <3f085ecd050712195157300596@mail.gmail.com> Message-ID: <4B264DB0-5273-41B6-9693-BE9591B9A5A0@redivi.com> On Jul 12, 2005, at 4:51 PM, Kevin Dangoor wrote: > I really want to use eggs, but I *have* to use py2app. Basically, the > issues that I see are: > > 1) modulegraph needs to know about eggs so that it can track down the > dependencies of things in the eggs > > 2) the eggs should either be put into the site-packages.zip file in > the bundle in "basket" form, or should be tossed directly into the > site-packages directory. > > For the first point, I'm curious if modulegraph already knows about > zip packages. I didn't see anything in the code that immediately > jumped out at me saying "hey! this package is zipped up". What I had > seen was that the contents of eggs were just being ignored. If module > graph does have some knowledge about zip libraries, that would > probably translate well. > > As for #2, the ideal would probably be to put everything in basket > form in site-packages.zip. I figure that by opening up the eggs and > only pulling out the referenced parts, the overall app can be smaller. > Any package resources need to get in there, though. For expedience, > though, I'm quite find with a little bit larger app that has the .egg > files stored directly in site-packages with a .pth file. > > Any commentary on the difficulty involved here? (Bob?) The most expedient option would be to specify the eggs and a pth as resources (or data_files) manually. Second to that, you could make sure all the eggs are unzipped and on sys.path when the setup.py is running (this doesn't solve the resources thing, but modulegraph doesn't do that anyway). As far as fixing the implementation, I don't know.. I don't really have time to work on open source Python stuff right now. Patches accepted, of course. -bob From dangoor at gmail.com Wed Jul 13 05:18:46 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Tue, 12 Jul 2005 23:18:46 -0400 Subject: [Distutils] py2app and eggs part deux In-Reply-To: <4B264DB0-5273-41B6-9693-BE9591B9A5A0@redivi.com> References: <3f085ecd050712195157300596@mail.gmail.com> <4B264DB0-5273-41B6-9693-BE9591B9A5A0@redivi.com> Message-ID: <3f085ecd0507122018488e5a70@mail.gmail.com> On 7/12/05, Bob Ippolito wrote: > The most expedient option would be to specify the eggs and a pth as > resources (or data_files) manually. > > Second to that, you could make sure all the eggs are unzipped and on > sys.path when the setup.py is running (this doesn't solve the > resources thing, but modulegraph doesn't do that anyway). > > As far as fixing the implementation, I don't know.. I don't really > have time to work on open source Python stuff right now. Patches > accepted, of course. That's why I was asking about complexity... I'm considering digging in myself, but I haven't touched py2app's code before. I was all in favor of the most expedient option that you mention above. However, the problem that I ran into was that standard library modules that were needed by the eggs weren't getting picked up (since the eggs were just "data files"). If there is an easy way to run modulegraph on the eggs, then maybe a short-term solution is: - add an option to specify required eggs (using the setuptools syntax, ie "FooBar >= 27.2") - grab these eggs and drop them into the generated site-packages dir with a pth file - scan them for any modules they need and add those to the overall dependency graph (this is where I'm fuzziest on the difficulty level) So, really... the main problem to solve is finding out what is required by the modules in a given egg. Kevin From bob at redivi.com Wed Jul 13 05:29:50 2005 From: bob at redivi.com (Bob Ippolito) Date: Tue, 12 Jul 2005 17:29:50 -1000 Subject: [Distutils] py2app and eggs part deux In-Reply-To: <3f085ecd0507122018488e5a70@mail.gmail.com> References: <3f085ecd050712195157300596@mail.gmail.com> <4B264DB0-5273-41B6-9693-BE9591B9A5A0@redivi.com> <3f085ecd0507122018488e5a70@mail.gmail.com> Message-ID: On Jul 12, 2005, at 5:18 PM, Kevin Dangoor wrote: > On 7/12/05, Bob Ippolito wrote: > >> The most expedient option would be to specify the eggs and a pth as >> resources (or data_files) manually. >> >> Second to that, you could make sure all the eggs are unzipped and on >> sys.path when the setup.py is running (this doesn't solve the >> resources thing, but modulegraph doesn't do that anyway). >> >> As far as fixing the implementation, I don't know.. I don't really >> have time to work on open source Python stuff right now. Patches >> accepted, of course. >> > > That's why I was asking about complexity... I'm considering digging in > myself, but I haven't touched py2app's code before. I really don't know, I haven't looked at the code recently, especially the modulegraph stuff. > I was all in favor of the most expedient option that you mention > above. However, the problem that I ran into was that standard library > modules that were needed by the eggs weren't getting picked up (since > the eggs were just "data files"). If there is an easy way to run > modulegraph on the eggs, then maybe a short-term solution is: Well, just do the second suggestion then. "Install" the eggs somewhere, unzipped, and make sure it's on sys.path during setup.py. Maybe you could crank the ez_setup wheels (or whatever module is appropriate, I haven't looked at eggs yet) at the top of your setup.py and automate this. -bob From pje at telecommunity.com Wed Jul 13 06:28:57 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 13 Jul 2005 00:28:57 -0400 Subject: [Distutils] py2app and eggs part deux In-Reply-To: References: <3f085ecd0507122018488e5a70@mail.gmail.com> <3f085ecd050712195157300596@mail.gmail.com> <4B264DB0-5273-41B6-9693-BE9591B9A5A0@redivi.com> <3f085ecd0507122018488e5a70@mail.gmail.com> Message-ID: <5.1.1.6.0.20050713001237.03721558@mail.telecommunity.com> At 05:29 PM 7/12/2005 -1000, Bob Ippolito wrote: >Well, just do the second suggestion then. "Install" the eggs >somewhere, unzipped, and make sure it's on sys.path during setup.py. >Maybe you could crank the ez_setup wheels (or whatever module is >appropriate, I haven't looked at eggs yet) at the top of your >setup.py and automate this. Well, if it just has to be on sys.path during setup.py, then it should be working now, shouldn't it? I mean, if he installed them to Python's site-packages. In the alternative, he could create a wrapper command that calls require("project==version") to find all the eggs and put them on sys.path. So, I guess my suggestion to Kevin is to try installing to a directory on sys.path, using the new --always-unzip option to force the packages to be unzipped, and then try hacking py2app to do a require() before looking for dependencies (consulting the setuptools "test" command for an example). I suspect this means that the packages will get merged into one another with no distinction at runtime, but whatever works, works. One interesting thing that eggs do, however, in the problem of module-finding is that they potentially reduce the question to one of just identifying what portions of the standard library each egg uses, which in some ways is a significantly easier problem because import tricks (e.g. dynamic imports, exec's, etc.) involving the standard library occur far less often. It might be worth adding something modulegraph-ish to setuptools to support this, although it's not at the top of my list right now. From bob at redivi.com Wed Jul 13 06:42:12 2005 From: bob at redivi.com (Bob Ippolito) Date: Tue, 12 Jul 2005 18:42:12 -1000 Subject: [Distutils] py2app and eggs part deux In-Reply-To: <5.1.1.6.0.20050713001237.03721558@mail.telecommunity.com> References: <3f085ecd0507122018488e5a70@mail.gmail.com> <3f085ecd050712195157300596@mail.gmail.com> <4B264DB0-5273-41B6-9693-BE9591B9A5A0@redivi.com> <3f085ecd0507122018488e5a70@mail.gmail.com> <5.1.1.6.0.20050713001237.03721558@mail.telecommunity.com> Message-ID: On Jul 12, 2005, at 6:28 PM, Phillip J. Eby wrote: > At 05:29 PM 7/12/2005 -1000, Bob Ippolito wrote: > >> Well, just do the second suggestion then. "Install" the eggs >> somewhere, unzipped, and make sure it's on sys.path during setup.py. >> Maybe you could crank the ez_setup wheels (or whatever module is >> appropriate, I haven't looked at eggs yet) at the top of your >> setup.py and automate this. >> > > Well, if it just has to be on sys.path during setup.py, then it > should be working now, shouldn't it? I mean, if he installed them > to Python's site-packages. In the alternative, he could create a > wrapper command that calls require("project==version") to find all > the eggs and put them on sys.path. Like modulefinder, modulegraph doesn't know anything about fancy PEP 302 stuff yet. > So, I guess my suggestion to Kevin is to try installing to a > directory on sys.path, using the new --always-unzip option to force > the packages to be unzipped, and then try hacking py2app to do a > require() before looking for dependencies (consulting the > setuptools "test" command for an example). I'm not sure you'd really want to hack py2app anyway, just the setup.py before setup() is called. > I suspect this means that the packages will get merged into one > another with no distinction at runtime, but whatever works, works. Yeah, I'm not sure why you would possibly care about this at runtime for an application that's "frozen", anyway. > One interesting thing that eggs do, however, in the problem of > module-finding is that they potentially reduce the question to one > of just identifying what portions of the standard library each egg > uses, which in some ways is a significantly easier problem because > import tricks (e.g. dynamic imports, exec's, etc.) involving the > standard library occur far less often. It might be worth adding > something modulegraph-ish to setuptools to support this, although > it's not at the top of my list right now. Modulegraph should already know about all of the tricks in the stdlib (at least in 2.3, not sure if I updated it for 2.4) including the modules that are imported by the core behind your back (warnings, etc.).. It doesn't contain anything py2app specific, it might as well be used as-is. http://svn.red-bean.com/bob/py2app/trunk/src/ (modulegraph depends on altgraph, the other stuff you can ignore) -bob From pje at telecommunity.com Wed Jul 13 06:42:51 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 13 Jul 2005 00:42:51 -0400 Subject: [Distutils] setuptools for people behind a firewall In-Reply-To: References: <79990c6b050712022316ce38d7@mail.gmail.com> <79990c6b05071108585afc3bcd@mail.gmail.com> <6AA7FD94-7D9A-43E2-9372-5938A504FEB2@gmail.com> <5.1.1.6.0.20050711230451.01ac5b28@mail.telecommunity.com> <79990c6b050712022316ce38d7@mail.gmail.com> Message-ID: <5.1.1.6.0.20050713003052.02690788@mail.telecommunity.com> At 05:49 AM 7/12/2005 -0400, Ryan Tomayko wrote: >I think that many people running systems built on package management >will want to setup an alternative install_dir (/usr/local/lib/ >pythonX.X/site-packages or perhaps /opt/pythonX.X/site-packages). >This is why I think the only-load-pth-from-site-packages issue is >important. Then again, wrapper scripts make it less of an issue. >teeder-totter-teeder-totter Keep in mind that you only have to create one .pth file per Python installation in order to enable this setup now. e.g., create an 'altinstall.pth' with this in it: import os,site; site.addsitedir(os.expanduser('~/lib/python2.X/site-packages') import site; site.addsitedir('/opt/python2.X/site-packages') Those lines will not only put the other directories on sys.path, but it will also load all the .pth files in those directories as well. Of course, EasyInstall won't create an easy-install.pth in the /opt directory or ~/lib directories, at least not until/unless it grows a --site-dirs option for you to use to specify alternate directories. I could see an OS distro being set up like this, with 'altinstall.pth' in site-packages, and a 'distutils.cfg' file in the OS-installed distutils, containing something like: [easy_install] site_dirs = /opt/python2.X/site-packages, ~/lib/python2.X/site-packages And this would basically make it so that you could install Python packages either site-wide or in your home directory, and still use easy-install.pth files. Of course, there are problems with this setup for distros that have privileged programs running in Python; those programs will need to run with python -S (to prevent bringing in site-packages and friends) so that those programs can control their own environment more closely. There are always trade-offs. From pje at telecommunity.com Wed Jul 13 06:59:27 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 13 Jul 2005 00:59:27 -0400 Subject: [Distutils] py2app and eggs part deux In-Reply-To: References: <5.1.1.6.0.20050713001237.03721558@mail.telecommunity.com> <3f085ecd0507122018488e5a70@mail.gmail.com> <3f085ecd050712195157300596@mail.gmail.com> <4B264DB0-5273-41B6-9693-BE9591B9A5A0@redivi.com> <3f085ecd0507122018488e5a70@mail.gmail.com> <5.1.1.6.0.20050713001237.03721558@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050713004844.0269f3f8@mail.telecommunity.com> At 06:42 PM 7/12/2005 -1000, Bob Ippolito wrote: >On Jul 12, 2005, at 6:28 PM, Phillip J. Eby wrote: > >>At 05:29 PM 7/12/2005 -1000, Bob Ippolito wrote: >> >>>Well, just do the second suggestion then. "Install" the eggs >>>somewhere, unzipped, and make sure it's on sys.path during setup.py. >>>Maybe you could crank the ez_setup wheels (or whatever module is >>>appropriate, I haven't looked at eggs yet) at the top of your >>>setup.py and automate this. >> >>Well, if it just has to be on sys.path during setup.py, then it >>should be working now, shouldn't it? I mean, if he installed them >>to Python's site-packages. In the alternative, he could create a >>wrapper command that calls require("project==version") to find all >>the eggs and put them on sys.path. > >Like modulefinder, modulegraph doesn't know anything about fancy PEP >302 stuff yet. I meant, if he installed with --always-unzip to the system site-packages directory, such that EasyInstall put the needed paths in easy-install.pth. >>So, I guess my suggestion to Kevin is to try installing to a >>directory on sys.path, using the new --always-unzip option to force >>the packages to be unzipped, and then try hacking py2app to do a >>require() before looking for dependencies (consulting the >>setuptools "test" command for an example). > >I'm not sure you'd really want to hack py2app anyway, just the >setup.py before setup() is called. Er, no, because then he'd have to change the setup script between building the package and making an app out of it. It needs to be something that runs as a command, *not* something in the setup script. >>I suspect this means that the packages will get merged into one >>another with no distinction at runtime, but whatever works, works. > >Yeah, I'm not sure why you would possibly care about this at runtime >for an application that's "frozen", anyway. If the application is composed of plugins, you'd want the plugins' metadata to be available, so there'd be additional work involved to get that put back together. >Modulegraph should already know about all of the tricks in the stdlib >(at least in 2.3, not sure if I updated it for 2.4) including the >modules that are imported by the core behind your back (warnings, >etc.).. It doesn't contain anything py2app specific, it might as >well be used as-is. > >http://svn.red-bean.com/bob/py2app/trunk/src/ >(modulegraph depends on altgraph, the other stuff you can ignore) Interesting. For py2exe this stuff makes some sense to me, but for py2app I wonder if you really even care about finding stdlib dependencies anyway, if your target Mac OS version has the stdlib already installed. It seems that in that case you could just use something like: setup.py easy_install --always-unzip --always-copy --install-dir=appdir . to dump all the unzipped eggs and scripts needed by the current project into 'appdir', and then do whatever Mac OS magic is needed to turn the directory into an application. Oh, and you'd also need to throw in a copy of pkg_resources, I suppose, since it's not part of Python... yet. :) This approach isn't really an option for py2exe, since there's no version of Windows that bundles Python, but for the many Unix-y operating systems (including Mac OS) that provide Python as a standard feature, it's not a bad choice. From bob at redivi.com Wed Jul 13 07:28:31 2005 From: bob at redivi.com (Bob Ippolito) Date: Tue, 12 Jul 2005 19:28:31 -1000 Subject: [Distutils] py2app and eggs part deux In-Reply-To: <5.1.1.6.0.20050713004844.0269f3f8@mail.telecommunity.com> References: <5.1.1.6.0.20050713001237.03721558@mail.telecommunity.com> <3f085ecd0507122018488e5a70@mail.gmail.com> <3f085ecd050712195157300596@mail.gmail.com> <4B264DB0-5273-41B6-9693-BE9591B9A5A0@redivi.com> <3f085ecd0507122018488e5a70@mail.gmail.com> <5.1.1.6.0.20050713001237.03721558@mail.telecommunity.com> <5.1.1.6.0.20050713004844.0269f3f8@mail.telecommunity.com> Message-ID: <182DF1F7-2A09-473A-BEBA-DC9941285B22@redivi.com> On Jul 12, 2005, at 6:59 PM, Phillip J. Eby wrote: > At 06:42 PM 7/12/2005 -1000, Bob Ippolito wrote: > >> On Jul 12, 2005, at 6:28 PM, Phillip J. Eby wrote: >> >> >>> At 05:29 PM 7/12/2005 -1000, Bob Ippolito wrote: >>> >>> >>>> Well, just do the second suggestion then. "Install" the eggs >>>> somewhere, unzipped, and make sure it's on sys.path during >>>> setup.py. >>>> Maybe you could crank the ez_setup wheels (or whatever module is >>>> appropriate, I haven't looked at eggs yet) at the top of your >>>> setup.py and automate this. >>>> >>> >>> Well, if it just has to be on sys.path during setup.py, then it >>> should be working now, shouldn't it? I mean, if he installed them >>> to Python's site-packages. In the alternative, he could create a >>> wrapper command that calls require("project==version") to find all >>> the eggs and put them on sys.path. >>> >> >> Like modulefinder, modulegraph doesn't know anything about fancy PEP >> 302 stuff yet. >> > > I meant, if he installed with --always-unzip to the system site- > packages directory, such that EasyInstall put the needed paths in > easy-install.pth. > > > >>> So, I guess my suggestion to Kevin is to try installing to a >>> directory on sys.path, using the new --always-unzip option to force >>> the packages to be unzipped, and then try hacking py2app to do a >>> require() before looking for dependencies (consulting the >>> setuptools "test" command for an example). >>> >> >> I'm not sure you'd really want to hack py2app anyway, just the >> setup.py before setup() is called. >> > > Er, no, because then he'd have to change the setup script between > building the package and making an app out of it. It needs to be > something that runs as a command, *not* something in the setup > script. Right, well, there's easy enough ways around that... subclass away :) >>> I suspect this means that the packages will get merged into one >>> another with no distinction at runtime, but whatever works, works. >>> >> >> Yeah, I'm not sure why you would possibly care about this at runtime >> for an application that's "frozen", anyway. >> > > If the application is composed of plugins, you'd want the plugins' > metadata to be available, so there'd be additional work involved to > get that put back together. This is only really a concern for people doing egg-centric development (hah). I don't think this is going to be a problem just yet :) > >> Modulegraph should already know about all of the tricks in the stdlib >> (at least in 2.3, not sure if I updated it for 2.4) including the >> modules that are imported by the core behind your back (warnings, >> etc.).. It doesn't contain anything py2app specific, it might as >> well be used as-is. >> >> http://svn.red-bean.com/bob/py2app/trunk/src/ >> (modulegraph depends on altgraph, the other stuff you can ignore) >> > > Interesting. For py2exe this stuff makes some sense to me, but for > py2app I wonder if you really even care about finding stdlib > dependencies anyway, if your target Mac OS version has the stdlib > already installed. It seems that in that case you could just use > something like: > > setup.py easy_install --always-unzip --always-copy --install- > dir=appdir . > > to dump all the unzipped eggs and scripts needed by the current > project into 'appdir', and then do whatever Mac OS magic is needed > to turn the directory into an application. Oh, and you'd also need > to throw in a copy of pkg_resources, I suppose, since it's not part > of Python... yet. :) > > This approach isn't really an option for py2exe, since there's no > version of Windows that bundles Python, but for the many Unix-y > operating systems (including Mac OS) that provide Python as a > standard feature, it's not a bad choice. You really don't want to depend on the OS installed Python. The biggest reason is because Python 2.3 is going to disappear in 10.5, so your application breaks. -bob From ianb at colorstudy.com Wed Jul 13 08:39:12 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 13 Jul 2005 01:39:12 -0500 Subject: [Distutils] setuptools: MANIFEST generation Message-ID: <42D4B710.3090000@colorstudy.com> I noticed in the documentation it said that setuptools can find files based on version control when creating source distributions. This almost never applies to my packages (at least the well made ones ;) because I have html files generated from txt sources, and I like to include the html. Is there any way to tell setuptools to add certain files, but still automatically detect everything else? It seemed to do an okay job when I created a MANIFEST.in with everything but "recursive-include . *.html" commented out, but there might be something I'm missing about it. Additionally, it would be nice to be able to exclude files, like files I use for internal package administration, or large svn externals which should be separate packages. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From mal at egenix.com Wed Jul 13 09:13:15 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 13 Jul 2005 09:13:15 +0200 Subject: [Distutils] easy_install - some thoughts In-Reply-To: <79990c6b05071214113e8cb082@mail.gmail.com> References: <5.1.1.6.0.20050711231010.038c2228@mail.telecommunity.com> <79990c6b050712031170cc2803@mail.gmail.com> <5.1.1.6.0.20050712115839.0268a400@mail.telecommunity.com> <79990c6b0507121137e089153@mail.gmail.com> <3f085ecd0507121145423cef4@mail.gmail.com> <79990c6b05071214113e8cb082@mail.gmail.com> Message-ID: <42D4BF0B.2080106@egenix.com> Paul Moore wrote: > On 7/12/05, Kevin Dangoor wrote: > >>On 7/12/05, Paul Moore wrote: >> >>>By the way, I don't know if the sqlite web page has changed, but your >>>standard example of sqlite doesn't work for me: >> >>It may be pysqlite. > > > It's not, it's Python 2.4 :-( > > It looks like setuptools.command.easy_install is doing some ugly hacks > to read data out of the raw bdist_wininst file, in > extract_wininst_cfg, and the file format has changed in Python 2.4. > > For 2.4, it looks like the "tag" has just changed from 0x1234567A to > 0x1234567B... So just checking both values should work. I've attached > a patch. > > Still doesn't work, as I get an error about the (not-)zip-safe file: > > >>python -m easy_install -xd eggs Python\ctypes-0.9.6.win32-py2.4.exe > > Processing ctypes-0.9.6.win32-py2.4.exe > ctypes.__init__: module references __file__ > ctypes.com.register: module references __file__ > error: c:\docume~1\gustav\locals~1\temp\easy_install-unhoeu\ctypes-0.9.6-py2.4-win32.egg.tmp\ > EGG-INFO\not-zip-safe: No such file or directory > > It looks like the issue is in bdist_egg.py, write_safety_flag, where > an ensure_directory() call is needed. There's a patch for this below, > as well. Talking about "safety": shouldn't this be addressed in a standard way, ie. signed packages ? At the very least, I'd expect the downloader to compare an MD5 checksum stored in PyPI with the one from the downloaded file. Of course, using GPG and checking the signature based on the public key of the author would be even better. After all... we don't want "easy" to spell "eval". -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 13 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From ianb at colorstudy.com Wed Jul 13 09:27:39 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 13 Jul 2005 02:27:39 -0500 Subject: [Distutils] setuptools: extras_require Message-ID: <42D4C26B.80500@colorstudy.com> I can't see any way to get extras_require to be installed, unless some other package indicates an extra feature in the package that needs to be supported. For example, if you want HTTP support in Paste, you can install WSGIUtils. But this isn't required. How can I get WSGIUtils installed? Or, more specifically, there's a bunch of examples distributed in the Paste source distribution; these examples require lots of extra software, even if Paste doesn't. How can I easily get all the examples working? -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From p.f.moore at gmail.com Wed Jul 13 10:50:08 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 13 Jul 2005 09:50:08 +0100 Subject: [Distutils] setuptools 0.5a12 In-Reply-To: <5.1.1.6.0.20050712202139.0378be38@mail.telecommunity.com> References: <5.1.1.6.0.20050712202139.0378be38@mail.telecommunity.com> Message-ID: <79990c6b0507130150313ed14@mail.gmail.com> On 7/13/05, Phillip J. Eby wrote: > Hopefully, this is the last of the current spew of bugfix releases, Could you retain a pointer to the source package on the setuptools page, for firewall-challenged people? I found it, by reading the source to ez_setup, but it's a bit of a roundabout route... (I know, I should set up APS, but time is short :-)) Thanks, Paul. From p.f.moore at gmail.com Wed Jul 13 11:02:10 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 13 Jul 2005 10:02:10 +0100 Subject: [Distutils] easy_install - some thoughts In-Reply-To: <5.1.1.6.0.20050712182814.0299c538@mail.telecommunity.com> References: <5.1.1.6.0.20050711231010.038c2228@mail.telecommunity.com> <79990c6b050712031170cc2803@mail.gmail.com> <5.1.1.6.0.20050712115839.0268a400@mail.telecommunity.com> <79990c6b0507121137e089153@mail.gmail.com> <3f085ecd0507121145423cef4@mail.gmail.com> <79990c6b05071214113e8cb082@mail.gmail.com> <5.1.1.6.0.20050712182814.0299c538@mail.telecommunity.com> Message-ID: <79990c6b05071302023c3f4860@mail.gmail.com> On 7/12/05, Phillip J. Eby wrote: > >But even then, things are going odd, as if I try to build an egg for > >ctypes (which isn't zip-safe) all of the files and subdirectories in > >the egg directory are missing the first two characters... Hmm, it > >happens with zip-safe eggs, as well, so I don't know - maybe it's > >related to the bdist_wininst unpacking code. Sorry, I can't debug this > >any further. > > And I can't reproduce it. However, I have noticed that the format change > for Python 2.4 includes a change to what data is in the 'cfgdata' string, > and so I changed this line: > > cfg.readfp(StringIO.StringIO(f.read(cfglen))) > > to read: > > cfg.readfp(StringIO.StringIO(f.read(cfglen).split(chr(0),1)[0])) > > In order to ignore anything after the first NUL character in the > cfgdata. I don't know if that was causing your two-character problem or > not, as I can't reproduce it with Python 2.4 and the ctypes .exe for Python > 2.4. Looks like that fixed the problem. Thanks. > After the first thousand levels of recursion, the time machine throws a > RuntimeError("Paradox stablization loop limit exceeded") and kicks > me back to the present. :) So *that's* why Guido resists implementing tail-call optimisation! :-) Paul From p.f.moore at gmail.com Wed Jul 13 12:20:42 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 13 Jul 2005 11:20:42 +0100 Subject: [Distutils] Adding egg support (Resource API) to existing packages Message-ID: <79990c6b05071303201318ac71@mail.gmail.com> I have been looking at how I can use eggs to manage the packages I currently use in my Python installation. Mostly, everything has gone extremely smoothly, and I can build eggs either from bdist_wininst executables, or from source packages, without any problems (and without needing to modify the package sources!). With one package (Python Dateutils) I noticed that the resulting egg was "not zip safe", and I thought I'd take a look at why. It turns out to be a relatively simple issue, as the package contains a data file which isn't accessed via the resource manager protocol. Patching the source isn't hard, but it left me with a dilemma. I'd like to submit the change back to the package author, but I'm not sure how receptive he'd be to converting to setuptools (ie, adding a dependency on setuptools, changing setup.py, etc). I'd like to be able to offer a minimal change, which just switched to using the resource management protocol - people who wanted to build eggs could change setup.py, or just use the --command-packages option in Python 2.4. The problem is, how to depend on *just* the pkg_resources module. I can bundle it with the application, which is simplest, but produces a maintenance burden. I can just say that users must install setuptools, but this seems ugly, as there is no obvious reason why a date utility package should depend on a setup utility package. Having pkg_resources in the standard library is clearly the best long-term solution, but in the shorter term, shouldn't pkg_resources be unbundled from setuptools? In many ways, isn't that the *point* of setuptools/eggs, to ease unbundling of packages - and so, shouldn't setuptools "practice what it preaches", and unbundle itself...? I realise that this could cause chicken and egg problems, where setuptools needs pkg_resources to ensure that it can support not having pkg_resources bundled, but even so... Any comments/suggestions? Paul. From dangoor at gmail.com Wed Jul 13 13:20:36 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Wed, 13 Jul 2005 07:20:36 -0400 Subject: [Distutils] py2app and eggs part deux In-Reply-To: References: <3f085ecd050712195157300596@mail.gmail.com> <4B264DB0-5273-41B6-9693-BE9591B9A5A0@redivi.com> <3f085ecd0507122018488e5a70@mail.gmail.com> Message-ID: <3f085ecd050713042018ae2e36@mail.gmail.com> On 7/12/05, Bob Ippolito wrote: > > On Jul 12, 2005, at 5:18 PM, Kevin Dangoor wrote: > > That's why I was asking about complexity... I'm considering digging in > > myself, but I haven't touched py2app's code before. > > I really don't know, I haven't looked at the code recently, > especially the modulegraph stuff. OK. > Well, just do the second suggestion then. "Install" the eggs > somewhere, unzipped, and make sure it's on sys.path during setup.py. > Maybe you could crank the ez_setup wheels (or whatever module is > appropriate, I haven't looked at eggs yet) at the top of your > setup.py and automate this. I hadn't realized that you haven't looked at eggs yet. I remembered reading on Phillip's blog that you had been involved in the early work and thought you might have continued following it. I'll take a quick look to see if I can conveniently add the egg's modules to the module graph. Failing that, I'll just work it the way that you and Phillip are suggesting. Kevin From p.f.moore at gmail.com Wed Jul 13 14:07:49 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 13 Jul 2005 13:07:49 +0100 Subject: [Distutils] A quick question on pkg_resources.find_distributions Message-ID: <79990c6b05071305072a09e9b9@mail.gmail.com> I have only one egg installed in my site-packages, namely setuptools. However, with the following script: import sys import pkg_resources def list_packages(): dists = [] for p in sys.path: for dist in pkg_resources.find_distributions(p): dists.append((p, dist)) for p, dist in dists: print p, dist.name, dist.version if __name__ == '__main__': list_packages() I get the output: C:\Apps\Python24\lib\site-packages setuptools 0.5a12 C:\Apps\Python24\Lib\site-packages\setuptools-0.5a12-py2.4.egg setuptools 0.5a12 How come I see setuptools twice? It clearly doesn't matter for purposes of importing it, but it makes a listing of installed packages look messy. A complementary question - how can I detect when two Distribution instances refer to the "same" egg. Is comparing the path attribute sufficient? Thanks, Paul. From p.f.moore at gmail.com Wed Jul 13 14:13:46 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 13 Jul 2005 13:13:46 +0100 Subject: [Distutils] A quick question on pkg_resources.find_distributions In-Reply-To: <79990c6b05071305072a09e9b9@mail.gmail.com> References: <79990c6b05071305072a09e9b9@mail.gmail.com> Message-ID: <79990c6b0507130513398385d6@mail.gmail.com> On 7/13/05, Paul Moore wrote: > C:\Apps\Python24\lib\site-packages setuptools 0.5a12 > C:\Apps\Python24\Lib\site-packages\setuptools-0.5a12-py2.4.egg setuptools 0.5a12 [...] > A complementary question - how can I detect when two Distribution > instances refer to the "same" egg. Is comparing the path attribute > sufficient? It's not, but using os.path.normcase on the path attribute seems OK (note that the case of the "lib" component in the 2 paths above differs - this is Windows, where the file system is case-insensitive). Paul. From p.f.moore at gmail.com Wed Jul 13 16:00:33 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 13 Jul 2005 15:00:33 +0100 Subject: [Distutils] setuptools for people behind a firewall In-Reply-To: <79990c6b050712022316ce38d7@mail.gmail.com> References: <79990c6b05071108585afc3bcd@mail.gmail.com> <6AA7FD94-7D9A-43E2-9372-5938A504FEB2@gmail.com> <5.1.1.6.0.20050711230451.01ac5b28@mail.telecommunity.com> <79990c6b050712022316ce38d7@mail.gmail.com> Message-ID: <79990c6b050713070079db9ad0@mail.gmail.com> On 7/12/05, Paul Moore wrote: > Hmm, yes. It's that "site-packages should be managed by tools" > attitude of mine which muddies the waters a bit. Maybe it's a Windows > thing... > > > As for dependency analysis, the current API to find eggs in a directory is > > pkg_resources.find_distributions(dirname_or_filename), which iterates over > > the egg(s), yielding Distribution objects. Distribution objects have a > > 'depends()' method which returns a list of Requirement objects describing > > what the distribution needs. With that API, you should easily be able to > > create a simple script to dump an egg's dependencies, although I think the > > API may change slightly in a future release. (e.g. depends() will probably > > have a different name before 1.0 rolls around.) > > Cool. I feel the need for such a script (and possibly some others) so > I'll have a go at an egg_utils script/module which I'll contribute > back when I'm happy with it. I'm playing around with writing some utility/management scripts at the moment. Listing installed distributions is easy, and I see how dependency analysis can be handled with the above APIs. I was looking at an uninstall utility, and hit a possible problem: Given a Distribution object (maybe derived from a user's command line, maybe from selection off a GUI) I can "uninstall" the Distribution by simply removing the egg (file or directory). However, according to the documentation, before I delete files for "the currently installed version of a package", I need to run easy_install -m to ensure that Python doesn't continue to search for it. So, three questions: 1. How can I tell from a Distribution instance if it is "the currently installed version"? 2. How can I do the equivalent of easy_install -m in code? 3. Can eggs be in site-packages, but not locatable via find_distribution? I can't see anything in the Distribution API documentation, and I'm a little hazy on what happens in the face of multiple Distributions of the same package, all in site-packages at once but only one "installed". And I don't have a suitable set of eggs or a "clean" environment to try things out. Thanks, Paul. From dangoor at gmail.com Wed Jul 13 17:15:40 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Wed, 13 Jul 2005 11:15:40 -0400 Subject: [Distutils] py2app and eggs part deux In-Reply-To: <3f085ecd050713042018ae2e36@mail.gmail.com> References: <3f085ecd050712195157300596@mail.gmail.com> <4B264DB0-5273-41B6-9693-BE9591B9A5A0@redivi.com> <3f085ecd0507122018488e5a70@mail.gmail.com> <3f085ecd050713042018ae2e36@mail.gmail.com> Message-ID: <3f085ecd05071308154d3ea216@mail.gmail.com> Either something has changed (for the better!) or I'm losing my mind... When I first started playing with the eggs (setuptools 0.5a5, I think), py2app pretty much ignored what was in there. This morning, I removed the egg expansion stuff from my setup script and voila! With setuptools 0.5a12, py2app spots the modules in the eggs and handles them just like any other! So, I just need to figure out what to do about included resources and I'm all set. Kevin From pje at telecommunity.com Wed Jul 13 17:47:52 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 13 Jul 2005 11:47:52 -0400 Subject: [Distutils] setuptools: MANIFEST generation In-Reply-To: <42D4B710.3090000@colorstudy.com> Message-ID: <5.1.1.6.0.20050713114506.0387eea0@mail.telecommunity.com> At 01:39 AM 7/13/2005 -0500, Ian Bicking wrote: >I noticed in the documentation it said that setuptools can find files >based on version control when creating source distributions. > >This almost never applies to my packages (at least the well made ones ;) >because I have html files generated from txt sources, and I like to >include the html. Is there any way to tell setuptools to add certain >files, but still automatically detect everything else? It seemed to do >an okay job when I created a MANIFEST.in with everything but >"recursive-include . *.html" commented out, but there might be something >I'm missing about it. > >Additionally, it would be nice to be able to exclude files, like files I >use for internal package administration, or large svn externals which >should be separate packages. Hm. Good points. I'll put it on my list to see if there's a way to integrate this, although it may be that it's already integrated. That is, if the distutils sdist command already applies its "get default files" before processing MANIFEST.in, the net result should be the same, with the exception that you'll have to use -f all the time to get the MANIFEST regenerated when anything changes. (Currently, setuptools' sdist regenerates MANIFEST every time if you have no MANIFEST.in, thereby saving you the trouble.) Hm. Maybe there should be a MANIFEST.extra that would do the same thing. Anyway, something to think about for *after* I get the current stuff on a testing regime. >-- >Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org >_______________________________________________ >Distutils-SIG maillist - Distutils-SIG at python.org >http://mail.python.org/mailman/listinfo/distutils-sig From pje at telecommunity.com Wed Jul 13 17:52:12 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 13 Jul 2005 11:52:12 -0400 Subject: [Distutils] setuptools: extras_require In-Reply-To: <42D4C26B.80500@colorstudy.com> Message-ID: <5.1.1.6.0.20050713114758.03883b80@mail.telecommunity.com> At 02:27 AM 7/13/2005 -0500, Ian Bicking wrote: >I can't see any way to get extras_require to be installed, unless some >other package indicates an extra feature in the package that needs to be >supported. > >For example, if you want HTTP support in Paste, you can install >WSGIUtils. But this isn't required. How can I get WSGIUtils installed? Assuming the "extra" in Paste is "HTTP", you would use: easy_install Paste[HTTP] Even if Paste itself is already installed, this will ensure that anything in extras_require['HTTP'] gets installed too. > Or, more specifically, there's a bunch of examples distributed in the >Paste source distribution; these examples require lots of extra >software, even if Paste doesn't. How can I easily get all the examples >working? Define an "examples" extra, then use: easy_install Paste[examples] Note, by the way, that if the examples need the HTTP extra, then you can specify that like this: setup( ... extras_require = dict( HTTP = ['WSGIUtils'], examples = ['Paste[HTTP]', ...other stuff here...], ) ) That is, it's okay to have cyclical dependencies; the dependency resolution mechanism is smart enough to figure out what to do. From pje at telecommunity.com Wed Jul 13 17:54:45 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 13 Jul 2005 11:54:45 -0400 Subject: [Distutils] setuptools 0.5a12 In-Reply-To: <79990c6b0507130150313ed14@mail.gmail.com> References: <5.1.1.6.0.20050712202139.0378be38@mail.telecommunity.com> <5.1.1.6.0.20050712202139.0378be38@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050713115236.0389cee8@mail.telecommunity.com> At 09:50 AM 7/13/2005 +0100, Paul Moore wrote: >On 7/13/05, Phillip J. Eby wrote: > > Hopefully, this is the last of the current spew of bugfix releases, > >Could you retain a pointer to the source package on the setuptools >page, for firewall-challenged people? I found it, by reading the >source to ez_setup, but it's a bit of a roundabout route... The PyPI home of setuptools is: http://python.org/pypi/setuptools And you'll always find the most current release download links there, for everything except ez_setup.py (because PyPI doesn't accept .py files). From pje at telecommunity.com Wed Jul 13 18:09:30 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 13 Jul 2005 12:09:30 -0400 Subject: [Distutils] Adding egg support (Resource API) to existing packages In-Reply-To: <79990c6b05071303201318ac71@mail.gmail.com> Message-ID: <5.1.1.6.0.20050713115507.0387a9c0@mail.telecommunity.com> At 11:20 AM 7/13/2005 +0100, Paul Moore wrote: >Patching the source isn't hard, but it left me with a dilemma. I'd like to >submit the change back to the package author, but I'm not sure how receptive >he'd be to converting to setuptools (ie, adding a dependency on setuptools, >changing setup.py, etc). I'd like to be able to offer a minimal change, which >just switched to using the resource management protocol - people who wanted to >build eggs could change setup.py, or just use the --command-packages option in >Python 2.4. > >The problem is, how to depend on *just* the pkg_resources module. I can bundle >it with the application, which is simplest, but produces a maintenance burden. Worse - it will result in conflicts when the package is installed in an unmanaged way, because each package bundling pkg_resources will overwrite ones bundled by others. Of course, as long as the user *only* uses easy_install to install packages, they'll be fine, but if they install even *one* package that bundles pkg_resources, they can break their installation. >I can just say that users must install setuptools, but this seems ugly, as >there is no obvious reason why a date utility package should depend on a setup >utility package. I think you're missing something. It makes more sense to bundle ez_setup.py than pkg_resources. See this link: http://peak.telecommunity.com/DevCenter/setuptools#using-setuptools-without-bundling-it This will take care of ensuring that a recent setuptools is installed on the user's machine. It's a relatively minor change, although it does of course still introduce a dependency in the sense that the package author needs to be comfortable with letting setuptools handle the actual installation. >Having pkg_resources in the standard library is clearly the best long-term >solution, but in the shorter term, shouldn't pkg_resources be unbundled from >setuptools? In many ways, isn't that the *point* of setuptools/eggs, to ease >unbundling of packages - and so, shouldn't setuptools "practice what it >preaches", and unbundle itself...? If pkg_resources could be in the standard library, there would be no need to tie it to setuptools. But pkg_resources *isn't* in the standard library, so the best practical way to get it installed is for it to piggyback on setuptools, which has a bootstrap facility that'll get it installed. Practicality beats purity here, I think. >I realise that this could cause chicken and >egg problems, where setuptools needs pkg_resources to ensure that it can >support not having pkg_resources bundled, but even so... You're also missing the other half of the problem, where pkg_resources needs setuptools so it can be installed in a way that's safe from being overwritten by random packages reinstalling it! This is also why other packages can't bundle it, without effectively bundling setuptools too. So, the simplest solution is to Just Drink the Kool-Aid and use setuptools already. It's not like there aren't numerous other benefits to using it for your packaging, anyway. From pje at telecommunity.com Wed Jul 13 18:15:21 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 13 Jul 2005 12:15:21 -0400 Subject: [Distutils] A quick question on pkg_resources.find_distributions In-Reply-To: <79990c6b05071305072a09e9b9@mail.gmail.com> Message-ID: <5.1.1.6.0.20050713120950.03889e68@mail.telecommunity.com> At 01:07 PM 7/13/2005 +0100, Paul Moore wrote: >I have only one egg installed in my site-packages, namely setuptools. >However, with the following script: > >import sys >import pkg_resources > >def list_packages(): > dists = [] > for p in sys.path: > for dist in pkg_resources.find_distributions(p): > dists.append((p, dist)) > > for p, dist in dists: > print p, dist.name, dist.version > >if __name__ == '__main__': > list_packages() > >I get the output: > >C:\Apps\Python24\lib\site-packages setuptools 0.5a12 >C:\Apps\Python24\Lib\site-packages\setuptools-0.5a12-py2.4.egg setuptools >0.5a12 > >How come I see setuptools twice? You need to look at the .path attributes. What's happening is that setuptools is on sys.path directly, but it's also in a *directory* on sys.path. So, you're calling find_distributions() on site-packages, but you're also calling it directly on the egg, so you get it twice. In the future, I may separate these kinds of returns, such that there's a way to ask find_distributions() to give you only exact matches, because this will be needed for implementing the WorkingSet concept in the pkg_resources refactoring. >A complementary question - how can I detect when two Distribution >instances refer to the "same" egg. Is comparing the path attribute >sufficient? You need os.path.normcase on Windows, and os.path.normpath/os.path.realpath/etc. to be really sure. Sane path comparison is unfortunately somewhat of a black art in Python. From pje at telecommunity.com Wed Jul 13 18:21:30 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 13 Jul 2005 12:21:30 -0400 Subject: [Distutils] setuptools for people behind a firewall In-Reply-To: <79990c6b050713070079db9ad0@mail.gmail.com> References: <79990c6b050712022316ce38d7@mail.gmail.com> <79990c6b05071108585afc3bcd@mail.gmail.com> <6AA7FD94-7D9A-43E2-9372-5938A504FEB2@gmail.com> <5.1.1.6.0.20050711230451.01ac5b28@mail.telecommunity.com> <79990c6b050712022316ce38d7@mail.gmail.com> Message-ID: <5.1.1.6.0.20050713121544.03885720@mail.telecommunity.com> At 03:00 PM 7/13/2005 +0100, Paul Moore wrote: >On 7/12/05, Paul Moore wrote: >Given a Distribution object (maybe derived from a user's command line, >maybe from selection off a GUI) I can "uninstall" the Distribution by >simply removing the egg (file or directory). Make sure you *only* do this to a directory if it has a '.egg' extension; otherwise you could delete a package installed using "develop"! > However, according to the >documentation, before I delete files for "the currently installed >version of a package", I need to run easy_install -m to >ensure that Python doesn't continue to search for it. So, three >questions: > >1. How can I tell from a Distribution instance if it is "the currently >installed version"? If its .path attribute matches an entry in sys.path. But you'd probably be better off manipulating easy-install.pth directly, via PthDistributions. >2. How can I do the equivalent of easy_install -m in code? There's a PthDistributions class in setuptools.command.easy_install; look at it and the code that uses it. Unfortunately, this code is targeted for refactoring when pkg_resources gets refactored, but hopefully its API won't change much. >3. Can eggs be in site-packages, but not locatable via find_distribution? Um, only if you don't look for them. I'm not sure I understand the question. >I can't see anything in the Distribution API documentation, and I'm a >little hazy on what happens in the face of multiple Distributions of >the same package, all in site-packages at once but only one >"installed". You mean only one "activated" (they're all "installed"). What happens is that when you find_distributions('site-packages') they will all be listed. However, when you find_distributions() on the path entry that makes a particular one current, that one will show up again. From pje at telecommunity.com Wed Jul 13 18:22:50 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 13 Jul 2005 12:22:50 -0400 Subject: [Distutils] py2app and eggs part deux In-Reply-To: <3f085ecd05071308154d3ea216@mail.gmail.com> References: <3f085ecd050713042018ae2e36@mail.gmail.com> <3f085ecd050712195157300596@mail.gmail.com> <4B264DB0-5273-41B6-9693-BE9591B9A5A0@redivi.com> <3f085ecd0507122018488e5a70@mail.gmail.com> <3f085ecd050713042018ae2e36@mail.gmail.com> Message-ID: <5.1.1.6.0.20050713122205.03914e28@mail.telecommunity.com> At 11:15 AM 7/13/2005 -0400, Kevin Dangoor wrote: >Either something has changed (for the better!) or I'm losing my mind... > >When I first started playing with the eggs (setuptools 0.5a5, I >think), py2app pretty much ignored what was in there. This morning, I >removed the egg expansion stuff from my setup script and voila! With >setuptools 0.5a12, py2app spots the modules in the eggs and handles >them just like any other! You might clean up any build directories involved, just in case there's residue from when you had the expansion code, because you might be getting a false positive success right now. Just a thought. From pje at telecommunity.com Wed Jul 13 18:49:55 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 13 Jul 2005 12:49:55 -0400 Subject: [Distutils] easy_install - some thoughts In-Reply-To: <42D4BF0B.2080106@egenix.com> References: <79990c6b05071214113e8cb082@mail.gmail.com> <5.1.1.6.0.20050711231010.038c2228@mail.telecommunity.com> <79990c6b050712031170cc2803@mail.gmail.com> <5.1.1.6.0.20050712115839.0268a400@mail.telecommunity.com> <79990c6b0507121137e089153@mail.gmail.com> <3f085ecd0507121145423cef4@mail.gmail.com> <79990c6b05071214113e8cb082@mail.gmail.com> Message-ID: <5.1.1.6.0.20050713113547.026851e8@mail.telecommunity.com> At 09:13 AM 7/13/2005 +0200, M.-A. Lemburg wrote: > > It looks like the issue is in bdist_egg.py, write_safety_flag, where > > an ensure_directory() call is needed. There's a patch for this below, > > as well. > >Talking about "safety": shouldn't this be addressed in a standard >way, ie. signed packages ? "Zip safety" refers to whether the package can be safely installed as a zip file; i.e., whether the package is likely to work once it has been installed that way. It's not about "safety" in some security sense. >At the very least, I'd expect the downloader to compare an MD5 >checksum stored in PyPI with the one from the downloaded file. >Of course, using GPG and checking the signature based on the >public key of the author would be even better. At the moment, PyPI only stores MD5's and signatures for packages uploaded to PyPI itself, which is an extremely small minority of packages, so I haven't implemented this yet. However, easy_install runs fine on local files, so you can download and verify files before running easy_install on them. If somebody wants to contribute patches for MD5 and signing, that would certainly be nice. From dangoor at gmail.com Wed Jul 13 19:02:20 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Wed, 13 Jul 2005 13:02:20 -0400 Subject: [Distutils] py2app and eggs part deux In-Reply-To: <5.1.1.6.0.20050713122205.03914e28@mail.telecommunity.com> References: <3f085ecd050712195157300596@mail.gmail.com> <4B264DB0-5273-41B6-9693-BE9591B9A5A0@redivi.com> <3f085ecd0507122018488e5a70@mail.gmail.com> <3f085ecd050713042018ae2e36@mail.gmail.com> <3f085ecd05071308154d3ea216@mail.gmail.com> <5.1.1.6.0.20050713122205.03914e28@mail.telecommunity.com> Message-ID: <3f085ecd05071310026272efa2@mail.gmail.com> On 7/13/05, Phillip J. Eby wrote: > At 11:15 AM 7/13/2005 -0400, Kevin Dangoor wrote: > >Either something has changed (for the better!) or I'm losing my mind... > > > >When I first started playing with the eggs (setuptools 0.5a5, I > >think), py2app pretty much ignored what was in there. This morning, I > >removed the egg expansion stuff from my setup script and voila! With > >setuptools 0.5a12, py2app spots the modules in the eggs and handles > >them just like any other! > > You might clean up any build directories involved, just in case there's > residue from when you had the expansion code, because you might be getting > a false positive success right now. Just a thought. Before sending that message this morning, I had poked around in my build directory and didn't see the packages there... apparently, they were there though. I deleted the whole build directory and I'm now getting the missing std lib modules problem that I was before. Oh well. From ianb at colorstudy.com Wed Jul 13 20:02:39 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 13 Jul 2005 13:02:39 -0500 Subject: [Distutils] setuptools: extras_require In-Reply-To: <5.1.1.6.0.20050713114758.03883b80@mail.telecommunity.com> References: <5.1.1.6.0.20050713114758.03883b80@mail.telecommunity.com> Message-ID: <42D5573F.3040108@colorstudy.com> Phillip J. Eby wrote: >> Or, more specifically, there's a bunch of examples distributed in the >> Paste source distribution; these examples require lots of extra >> software, even if Paste doesn't. How can I easily get all the examples >> working? > > > Define an "examples" extra, then use: > > easy_install Paste[examples] > > Note, by the way, that if the examples need the HTTP extra, then you can > specify that like this: > > setup( > ... > extras_require = dict( > HTTP = ['WSGIUtils'], > examples = ['Paste[HTTP]', ...other stuff here...], > ) > ) > > That is, it's okay to have cyclical dependencies; the dependency > resolution mechanism is smart enough to figure out what to do. Cool... now I'm thinking about how I should distribute the examples. They are already part of the source distribution (i.e., in the MANIFEST but not the packages). They aren't a package at all, really, more like data (even the .py files). I'm not sure what I should be shooting for. This is okay: easy_install.py Paste[HTTP,examples] But this doesn't actually install the examples themselves, just the prerequisites for them. I can't figure out how example installation would work, since examples really aren't installed globally in any way, and shouldn't be eggified. The examples could be separated into a separate package, but I don't think that would help. If easy_install.py had a develop option that would do it. Like: easy_install.py Paste[HTTP,examples] easy_install.py --develop Paste Where --develop downloads the package (not the egg), and does "setup.py develop" or something. Maybe, kind of -- at least, you'd end up with the entire Paste source package, but the prereqs would be installed normally. And I guess if you didn't run the first command, all the prereqs would be installed in development? I don't know. The same issues perhaps apply to documentation (though personally documentation matters much less to me, since it can be published directly to the web). -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Thu Jul 14 05:05:04 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 13 Jul 2005 22:05:04 -0500 Subject: [Distutils] setuptools: extras_require In-Reply-To: <5.1.1.6.0.20050713114758.03883b80@mail.telecommunity.com> References: <5.1.1.6.0.20050713114758.03883b80@mail.telecommunity.com> Message-ID: <42D5D660.3040902@colorstudy.com> Phillip J. Eby wrote: > At 02:27 AM 7/13/2005 -0500, Ian Bicking wrote: > >> I can't see any way to get extras_require to be installed, unless some >> other package indicates an extra feature in the package that needs to be >> supported. >> >> For example, if you want HTTP support in Paste, you can install >> WSGIUtils. But this isn't required. How can I get WSGIUtils installed? > > > Assuming the "extra" in Paste is "HTTP", you would use: > > easy_install Paste[HTTP] > > Even if Paste itself is already installed, this will ensure that > anything in extras_require['HTTP'] gets installed too. How can I do this if I have an unpacked version of Paste on disk? I can install that with "easy_install.py .", but I can't do "easy_install.py .[HTTP]". Is there a way to install the dependencies with setup.py? "python setup.py depends" doesn't do much of anything. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Thu Jul 14 05:11:29 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 13 Jul 2005 23:11:29 -0400 Subject: [Distutils] setuptools: extras_require In-Reply-To: <42D5D660.3040902@colorstudy.com> References: <5.1.1.6.0.20050713114758.03883b80@mail.telecommunity.com> <5.1.1.6.0.20050713114758.03883b80@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050713230704.02684848@mail.telecommunity.com> At 10:05 PM 7/13/2005 -0500, Ian Bicking wrote: >Phillip J. Eby wrote: >>At 02:27 AM 7/13/2005 -0500, Ian Bicking wrote: >> >>>I can't see any way to get extras_require to be installed, unless some >>>other package indicates an extra feature in the package that needs to be >>>supported. >>> >>>For example, if you want HTTP support in Paste, you can install >>>WSGIUtils. But this isn't required. How can I get WSGIUtils installed? >> >>Assuming the "extra" in Paste is "HTTP", you would use: >> easy_install Paste[HTTP] >>Even if Paste itself is already installed, this will ensure that anything >>in extras_require['HTTP'] gets installed too. > >How can I do this if I have an unpacked version of Paste on disk? I can >install that with "easy_install.py .", but I can't do "easy_install.py .[HTTP]" easy_install.py . Paste[HTTP] Or: python setup.py develop easy_install.py Paste[HTTP] In either case, the Paste requirement will be met by the installed (or "developed") version. Of course, it seems like it would be best to add an '--extras' argument to the 'install' and 'develop' commands, as then you could just use --extras=HTTP as an argument to either. I'll look at that for a future release. >Is there a way to install the dependencies with setup.py? "python setup.py >depends" doesn't do much of anything. That's because it's an undocumented command that was never finished. It was based on a much older idea of how to accomplish dependency installation, that didn't involve eggs. It'll go away or be replaced by something useful eventually. :) From pje at telecommunity.com Thu Jul 14 05:21:34 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 13 Jul 2005 23:21:34 -0400 Subject: [Distutils] setuptools: extras_require In-Reply-To: <42D5573F.3040108@colorstudy.com> References: <5.1.1.6.0.20050713114758.03883b80@mail.telecommunity.com> <5.1.1.6.0.20050713114758.03883b80@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050713231259.029b5fc0@mail.telecommunity.com> At 01:02 PM 7/13/2005 -0500, Ian Bicking wrote: >If easy_install.py had a develop option that would do it. Like: > > easy_install.py Paste[HTTP,examples] > easy_install.py --develop Paste > >Where --develop downloads the package (not the egg), and does "setup.py >develop" or something. Maybe, kind of -- at least, you'd end up with the >entire Paste source package, but the prereqs would be installed normally. The problem is no way to make links to checkouts, remember? I guess we could finally implement that anchor idea, and then you could include the link in your PyPI long_description, where easy_install could find it. > And I guess if you didn't run the first command, all the prereqs would > be installed in development? Yeah, the develop command installs dependencies now. Personally, I don't think a --develop option should be recursive; I think it should apply only to the packages you explicitly list. > I don't know. The same issues perhaps apply to documentation (though > personally documentation matters much less to me, since it can be > published directly to the web). There might also be some use to having tests appear only in development checkouts, and not in the actual eggs. Setuptools has an undocumented "feature" system that lets you use --with-X and --without-X options to control whether a set of packages or extensions are included in the build, although there are some limitations to its functioning when you don't clean up the "build" directory between option changes. (Which is one reason it's undocumented.) From ianb at colorstudy.com Thu Jul 14 05:59:23 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 13 Jul 2005 22:59:23 -0500 Subject: [Distutils] setuptools: extras_require In-Reply-To: <5.1.1.6.0.20050713231259.029b5fc0@mail.telecommunity.com> References: <5.1.1.6.0.20050713114758.03883b80@mail.telecommunity.com> <5.1.1.6.0.20050713114758.03883b80@mail.telecommunity.com> <5.1.1.6.0.20050713231259.029b5fc0@mail.telecommunity.com> Message-ID: <42D5E31B.3040703@colorstudy.com> Phillip J. Eby wrote: > At 01:02 PM 7/13/2005 -0500, Ian Bicking wrote: > >> If easy_install.py had a develop option that would do it. Like: >> >> easy_install.py Paste[HTTP,examples] >> easy_install.py --develop Paste >> >> Where --develop downloads the package (not the egg), and does >> "setup.py develop" or something. Maybe, kind of -- at least, you'd >> end up with the entire Paste source package, but the prereqs would be >> installed normally. > > > The problem is no way to make links to checkouts, remember? I guess we > could finally implement that anchor idea, and then you could include the > link in your PyPI long_description, where easy_install could find it. Yes, I was thinking the two went together. I had forgotten about the anchor idea; I'll give that another look and try to produce a patch. What would that link actually look like? http://svn.pythonpaste.org/Paste/trunk#Paste-devel-trunk ? I'm not sure what the version number should be. >> And I guess if you didn't run the first command, all the prereqs >> would be installed in development? > > > Yeah, the develop command installs dependencies now. > > Personally, I don't think a --develop option should be recursive; I > think it should apply only to the packages you explicitly list. That seems sensible to me as well. >> I don't know. The same issues perhaps apply to documentation >> (though personally documentation matters much less to me, since it can >> be published directly to the web). > > > There might also be some use to having tests appear only in development > checkouts, and not in the actual eggs. Setuptools has an undocumented > "feature" system that lets you use --with-X and --without-X options to > control whether a set of packages or extensions are included in the > build, although there are some limitations to its functioning when you > don't clean up the "build" directory between option changes. (Which is > one reason it's undocumented.) What would X be? --with-foo.tests? I go back and forth with including tests in the package that's being tested, or putting it in a sibling directory. Generally I'd see tests as a development files like examples or documentation. Hmm... I can almost see these as being like scripts, being installed in a separate configurable location. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Thu Jul 14 06:28:20 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 14 Jul 2005 00:28:20 -0400 Subject: [Distutils] setuptools: extras_require In-Reply-To: <42D5E31B.3040703@colorstudy.com> References: <5.1.1.6.0.20050713231259.029b5fc0@mail.telecommunity.com> <5.1.1.6.0.20050713114758.03883b80@mail.telecommunity.com> <5.1.1.6.0.20050713114758.03883b80@mail.telecommunity.com> <5.1.1.6.0.20050713231259.029b5fc0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050714002327.0268a1c0@mail.telecommunity.com> At 10:59 PM 7/13/2005 -0500, Ian Bicking wrote: >What would that link actually look like? >http://svn.pythonpaste.org/Paste/trunk#Paste-devel-trunk ? I'm not sure >what the version number should be. I was assuming it would be like #egg-dev=Paste, just identifying the project, not a version. That doesn't address the possibility of branches, but offhand I'd say that's a YAGNI. >>There might also be some use to having tests appear only in development >>checkouts, and not in the actual eggs. Setuptools has an undocumented >>"feature" system that lets you use --with-X and --without-X options to >>control whether a set of packages or extensions are included in the >>build, although there are some limitations to its functioning when you >>don't clean up the "build" directory between option changes. (Which is >>one reason it's undocumented.) > >What would X be? --with-foo.tests? Well, if that was the name of the Feature object, I suppose. :) In practice, you couldn't have a '.'. See: http://svn.eby-sarna.com/PyProtocols/setup.py?view=markup and look at the 'speedups' feature, that implements a --with-speedups/--without-speedups pair. From p.f.moore at gmail.com Thu Jul 14 11:26:57 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 14 Jul 2005 10:26:57 +0100 Subject: [Distutils] Adding egg support (Resource API) to existing packages In-Reply-To: <5.1.1.6.0.20050713115507.0387a9c0@mail.telecommunity.com> References: <79990c6b05071303201318ac71@mail.gmail.com> <5.1.1.6.0.20050713115507.0387a9c0@mail.telecommunity.com> Message-ID: <79990c6b050714022654f1f809@mail.gmail.com> On 7/13/05, Phillip J. Eby wrote: > At 11:20 AM 7/13/2005 +0100, Paul Moore wrote: > >I can just say that users must install setuptools, but this seems ugly, as > >there is no obvious reason why a date utility package should depend on a setup > >utility package. > > I think you're missing something. It makes more sense to bundle > ez_setup.py than pkg_resources. See this link: I had read that link. I think I failed to express my problem well enough. I am not the author of the python-dateutils package, so I am acting as an interested user, who is considering submitting a patch to the author. My patch will help *me* build an egg from the package, but unless (until) the author becomes interested in easy_setup, setuptools and eggs for his own reasons, I don't expect him to be building eggs himself. What I want to do is to send a patch to the author, saying "Hey - I like your package, but for my needs the following patch would help. Others may also benefit, so would it be possible to include it in the standard distribution?" Now, at this level, setuptools/pkg_utils isn't relevant. I'd have the same issues if I was advocating a dependency on *any* other package - why should the author impose an extra installation burden on all of his users just because I'd like an additional feature? That's precisely the issue that eggs are designed to address, but we aren't yet in a position where we can assume everyone is using eggs, and the problem is solved. So I have to minimise my dependency requirements, and carefully judge the costs and benefits - particularly for other users who may not get any benefit at all! In addition, bundling ez_setup imposes a burden on the package *author*, who will potentially need to make changes to his own build process as a result - even if it's only to make sure that he has an internet connection when he builds, so that ez_setup can do its stuff. And the same burden applies to anyone using the package who still wants to use "python setup.py install". Don't get me wrong - ez_setup is a great solution to the transitional issue of getting from where we are now to a position where everyone can be assumed to have (some version of) setuptools installed. But that's good for setup/distribution issues, not resource lookup. What I'm saying, is that I see a need for *another* transitional process - enabling users and/or package authors to make their code "resource safe", *without* having to buy into the egg/setuptools transition. I see the resource API as a major benefit in isolation - it's a general tool which can be used with zipimport, or other import hooks, and is not egg-specific. There's very little downside (IMHO) to using the resource API. On the other hand, I see eggs as a less clear-cut benefit - many people will still prefer installation processes tied into the OS package management facilities (even something as basic as bdist_wininst). Alongside this, I see setuptools as basically an egg-enabling change - and as such tied to the "less clear-cut" side of the equation. I'd like to see pkg_resources separated from this, as I see far less reason why anyone might object to it. [This is why I feel that pkg_resource is pretty much a no-brainer for inclusion into the stdlib, whereas setuptools may be more controversial]. > You're also missing the other half of the problem, where pkg_resources > needs setuptools so it can be installed in a way that's safe from being > overwritten by random packages reinstalling it! This is also why other > packages can't bundle it, without effectively bundling setuptools too. So, > the simplest solution is to Just Drink the Kool-Aid and use setuptools > already. It's not like there aren't numerous other benefits to using it > for your packaging, anyway. Hmm, I've read this a number of times and it still seems to me to be saying that all of setuptools' package management goodies break horribly if an unmanaged copy of a package is installed - making it a very "all or nothing" solution. While I accept that something like require("setuptools=0.5a7,=0.5a11,=0.5a12") is probably not going to work well if there's an unmanaged copy of setuptools on sys.path, can't you make some attempt in simpler cases (maybe introspect for __version__, and if not present assume the unmanaged version is "wrong", and put a good version at the *front* of sys.path - or something). Some form of graceful degradation should be available, surely? What if I install a new package, which I know depends on a number of major packages I already have installed - and the new one is an egg, but the old ones have been there for ages and are used by a number of existing scripts. Surely I don't have to switch everything to managed eggs in one "big bang" just to get the one new package installed? I can't see people doing that in a production environment. If I misread your paragraph above, my apologies. Cheers, Paul. From p.f.moore at gmail.com Thu Jul 14 12:26:45 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 14 Jul 2005 11:26:45 +0100 Subject: [Distutils] setuptools for people behind a firewall In-Reply-To: <5.1.1.6.0.20050713121544.03885720@mail.telecommunity.com> References: <79990c6b05071108585afc3bcd@mail.gmail.com> <6AA7FD94-7D9A-43E2-9372-5938A504FEB2@gmail.com> <5.1.1.6.0.20050711230451.01ac5b28@mail.telecommunity.com> <79990c6b050712022316ce38d7@mail.gmail.com> <79990c6b050713070079db9ad0@mail.gmail.com> <5.1.1.6.0.20050713121544.03885720@mail.telecommunity.com> Message-ID: <79990c6b050714032652d46773@mail.gmail.com> On 7/13/05, Phillip J. Eby wrote: > At 03:00 PM 7/13/2005 +0100, Paul Moore wrote: > >On 7/12/05, Paul Moore wrote: > >Given a Distribution object (maybe derived from a user's command line, > >maybe from selection off a GUI) I can "uninstall" the Distribution by > >simply removing the egg (file or directory). > > Make sure you *only* do this to a directory if it has a '.egg' extension; > otherwise you could delete a package installed using "develop"! Hmm, I'm only looking on sys.path - I hadn't imagined that development software would be added to the *default* sys.path... But I take your point. Actually, as I'm only looking at these utilities from the POV of managing site-packages, maybe I should just strip out any entries from sys.path which aren't under that directory (although I don't know if I can find that directory on Unix - on Windows, it's under sys.(exec)prefix, but I don't know directory structures under Unix so well). I'll have to check the source of easy_install to see how it decides where to copy files *to* (in the absence of user overrides...) [... omitted some stuff about PthDistributions which I need to review ...] > >3. Can eggs be in site-packages, but not locatable via find_distribution? > > Um, only if you don't look for them. I'm not sure I understand the question. Sorry - I'm doing find_distribution() on every entry in sys.path. What I was getting at, is whether that process could miss any eggs which easy_install may have put into site-packages. Again, the key point I forgot to clarify is that I want to keep track of "things that easy_install could have added to site-packages" - as I'm trying to add tools to do the operations easy_install doesn't supply (list, uninstall are the key ones) so that there's no requirement for the user to manually work inside site-packages. > >I can't see anything in the Distribution API documentation, and I'm a > >little hazy on what happens in the face of multiple Distributions of > >the same package, all in site-packages at once but only one > >"installed". > > You mean only one "activated" (they're all "installed"). What happens is > that when you find_distributions('site-packages') they will all be > listed. However, when you find_distributions() on the path entry that > makes a particular one current, that one will show up again. Ah, OK. That helps me understand the reason for the doubled entries mentioned above as well. Thanks, Paul. From pje at telecommunity.com Thu Jul 14 17:04:55 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 14 Jul 2005 11:04:55 -0400 Subject: [Distutils] Adding egg support (Resource API) to existing packages In-Reply-To: <79990c6b050714022654f1f809@mail.gmail.com> References: <5.1.1.6.0.20050713115507.0387a9c0@mail.telecommunity.com> <79990c6b05071303201318ac71@mail.gmail.com> <5.1.1.6.0.20050713115507.0387a9c0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050714101747.02680170@mail.telecommunity.com> At 10:26 AM 7/14/2005 +0100, Paul Moore wrote: >What I'm saying, is that I see a need for *another* transitional >process - enabling users and/or package authors to make their code >"resource safe", *without* having to buy into the egg/setuptools >transition. It still seems to me that you're missing something. If I don't care about eggs, why would I care about resource safety? Its only benefit is to allow my code to be put in an egg! :) >I see the resource API as a major benefit in isolation - >it's a general tool which can be used with zipimport, or other import >hooks, and is not egg-specific. In principle that's true. In practice, the only other tool I know of where it'd actually be *relevant* today would be py2exe -- and you don't get all of the features there that you get with eggs. For example, resource_filename() works with eggs and unmanaged packages, but not with py2exe. Indeed, the majority of pkg_resources' code deals with things that are only of relevance to eggs; simple resource safety is hardly any code at all; something like: if hasattr(module,'__loader__'): if hasattr(module.__loader__,'get_data'): # code to call get_data w/appropriate value else: # error else: # use __file__ This doesn't give you resource_filename() and it doesn't let you put C extensions in a zipfile, but it's adequate to implement basic zip-safety. > There's very little downside (IMHO) to >using the resource API. You have to bundle pkg_resources, and then deal with potential version conflicts to it. The bundling isn't a big deal, but the version conflict issue is very big because it means you could break someone's whole system by installing an old version of pkg_resources that can't deal with some new feature used by other packages they've installed. > > You're also missing the other half of the problem, where pkg_resources > > needs setuptools so it can be installed in a way that's safe from being > > overwritten by random packages reinstalling it! This is also why other > > packages can't bundle it, without effectively bundling setuptools too. So, > > the simplest solution is to Just Drink the Kool-Aid and use setuptools > > already. It's not like there aren't numerous other benefits to using it > > for your packaging, anyway. > >Hmm, I've read this a number of times and it still seems to me to be >saying that all of setuptools' package management goodies break >horribly if an unmanaged copy of a package is installed No, it's saying that if an unmanaged copy of *pkg_resources* is installed, you could break somebody's whole installation, because somebody who just bundles it could be distributing a version that doesn't work with stuff somebody already has installed. This is one of the many reasons that "bundling sucks", but it's downright dangerous for something like pkg_resources. Let me repeat: bundling pkg_resources without setuptools is *dangerous*, because you could screw up somebody's system just by not keeping your pkg_resources version up-to-date. You could say, "well, I'll have my setup check the version", but now you're modifying setup.py a bunch, and by the time you make it safe, you could've spent less time just slapping ez_setup.py in there and been done with it. > - making it a >very "all or nothing" solution. While I accept that something like >require("setuptools=0.5a7,=0.5a11,=0.5a12") is probably not going to >work well if there's an unmanaged copy of setuptools on sys.path, >can't you make some attempt in simpler cases (maybe introspect for >__version__, and if not present assume the unmanaged version is >"wrong", and put a good version at the *front* of sys.path - or >something). Some form of graceful degradation should be available, >surely? So come up with a proposal for *how*, that doesn't involve duplicating setuptools' code in pkg_resources or vice versa. Unless you're suggesting I put ez_setup inside of pkg_resources as some kind of trojan? :) More seriously, I suppose I could write some kind of bootstrap code in pkg_resources that scans sys.path for a setuptools egg and then reloads pkg_resources from it if it's not running in an egg. That would make it safe to bundle pkg_resources as long as it was a version with that feature. It would be hideous and stupid, though, because it'd be doubling the import time for people who do have setuptools. I want installing setuptools to be a benefit, not a penalty. My perspective on this is that I don't really care if packages use pkg_resources now; if nobody does it until pkg_resources is in the stdlib, that's okay by me. Eggs work fine in unpacked form, they just slow down top-level imports a little. > What if I install a new package, which I know depends on a >number of major packages I already have installed - and the new one is >an egg, but the old ones have been there for ages and are used by a >number of existing scripts. Surely I don't have to switch everything >to managed eggs in one "big bang" just to get the one new package >installed? This isn't a problem for most eggs; it's strictly a problem with overwriting the egg runtime. From pje at telecommunity.com Thu Jul 14 17:14:36 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 14 Jul 2005 11:14:36 -0400 Subject: [Distutils] setuptools for people behind a firewall In-Reply-To: <79990c6b050714032652d46773@mail.gmail.com> References: <5.1.1.6.0.20050713121544.03885720@mail.telecommunity.com> <79990c6b05071108585afc3bcd@mail.gmail.com> <6AA7FD94-7D9A-43E2-9372-5938A504FEB2@gmail.com> <5.1.1.6.0.20050711230451.01ac5b28@mail.telecommunity.com> <79990c6b050712022316ce38d7@mail.gmail.com> <79990c6b050713070079db9ad0@mail.gmail.com> <5.1.1.6.0.20050713121544.03885720@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050714110519.01ac89f0@mail.telecommunity.com> At 11:26 AM 7/14/2005 +0100, Paul Moore wrote: >On 7/13/05, Phillip J. Eby wrote: > > At 03:00 PM 7/13/2005 +0100, Paul Moore wrote: > > >On 7/12/05, Paul Moore wrote: > > >Given a Distribution object (maybe derived from a user's command line, > > >maybe from selection off a GUI) I can "uninstall" the Distribution by > > >simply removing the egg (file or directory). > > > > Make sure you *only* do this to a directory if it has a '.egg' extension; > > otherwise you could delete a package installed using "develop"! > >Hmm, I'm only looking on sys.path - I hadn't imagined that development >software would be added to the *default* sys.path... By default, if you use "develop" on a package, it becomes part of the default sys.path; this makes sense on machines that are "development" machines. If you are developing on a machine that also has production software, you need to use a different staging area (--install-dir) for the develop command. This is easily set on a per-user or sitewide basis; in a shop making use of these tools, you'd probably do this (as root): setup.py setopt -g -c develop -o install_dir -s /somewhere/staging To set the sitewide staging area. Hmm, actually, it would probably make more sense to have user-specific staging areas, by making the default something like ~/staging. I should probably think about adding expanduser() support to a lot of setuptools' options. >But I take your point. Actually, as I'm only looking at these >utilities from the POV of managing site-packages, maybe I should just >strip out any entries from sys.path which aren't under that directory >(although I don't know if I can find that directory on Unix - on >Windows, it's under sys.(exec)prefix, but I don't know directory >structures under Unix so well). I'll have to check the source of >easy_install to see how it decides where to copy files *to* (in the >absence of user overrides...) There's a get_python_lib() function in distutils that I use. However, I've recently discovered that the situation on Mac OS X is more complex; users have a ~/Library/python2.X/site-packages directory as well. >Sorry - I'm doing find_distribution() on every entry in sys.path. What >I was getting at, is whether that process could miss any eggs which >easy_install may have put into site-packages. No; find_distributions(site_packages) would list everything installed in that directory, whether active or not, plus distributions for anything that was installed as a development link in site-packages. That is, even if *all* you call find_distributions() on is site-packages, it's still going to include distributions whose .path is elsewhere, and that's fine because you want to know about those anyway; just list them with a "(Development)" status or something in your output. From p.f.moore at gmail.com Thu Jul 14 17:44:04 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 14 Jul 2005 16:44:04 +0100 Subject: [Distutils] Adding egg support (Resource API) to existing packages In-Reply-To: <5.1.1.6.0.20050714101747.02680170@mail.telecommunity.com> References: <79990c6b05071303201318ac71@mail.gmail.com> <5.1.1.6.0.20050713115507.0387a9c0@mail.telecommunity.com> <79990c6b050714022654f1f809@mail.gmail.com> <5.1.1.6.0.20050714101747.02680170@mail.telecommunity.com> Message-ID: <79990c6b05071408443e900c6a@mail.gmail.com> On 7/14/05, Phillip J. Eby wrote: > At 10:26 AM 7/14/2005 +0100, Paul Moore wrote: > >What I'm saying, is that I see a need for *another* transitional > >process - enabling users and/or package authors to make their code > >"resource safe", *without* having to buy into the egg/setuptools > >transition. > > It still seems to me that you're missing something. If I don't care about > eggs, why would I care about resource safety? Its only benefit is to allow > my code to be put in an egg! :) You don't care about eggs (yet). Your users (one of whom is me) do, and would like to package your code in an egg via python setup.py --compiler-package=setuptools.command bdist_egg. This works fine, with no source changes, but the resulting egg isn't zip-safe because of resource usage issues. Your user (me) can maintain a small local patch to switch to the resource API, but it would be nice to fold that patch into the distribution, and so avoid the need for a local patch. But the patch that is small for your user with setuptools installed, is large *for you* because you don't (yet) buy into the benefits of eggs (and hence the patch adds a dependency you don't want). So you reject the patch. Please be aware that the point about eggs that excites *me* is the single-file aspect. I'm pretty neutral over unpacked eggs at the moment, so zip-safety is a big bonus for me. I think we're going to have to agree to differ here. Thanks for sticking with it this far. Paul. From p.f.moore at gmail.com Thu Jul 14 17:50:48 2005 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 14 Jul 2005 16:50:48 +0100 Subject: [Distutils] setuptools for people behind a firewall In-Reply-To: <5.1.1.6.0.20050714110519.01ac89f0@mail.telecommunity.com> References: <79990c6b05071108585afc3bcd@mail.gmail.com> <6AA7FD94-7D9A-43E2-9372-5938A504FEB2@gmail.com> <5.1.1.6.0.20050711230451.01ac5b28@mail.telecommunity.com> <79990c6b050712022316ce38d7@mail.gmail.com> <79990c6b050713070079db9ad0@mail.gmail.com> <5.1.1.6.0.20050713121544.03885720@mail.telecommunity.com> <79990c6b050714032652d46773@mail.gmail.com> <5.1.1.6.0.20050714110519.01ac89f0@mail.telecommunity.com> Message-ID: <79990c6b050714085039afe0b2@mail.gmail.com> On 7/14/05, Phillip J. Eby wrote: > At 11:26 AM 7/14/2005 +0100, Paul Moore wrote: > > > >Hmm, I'm only looking on sys.path - I hadn't imagined that development > >software would be added to the *default* sys.path... > > By default, if you use "develop" on a package, it becomes part of the > default sys.path; this makes sense on machines that are "development" > machines. Ah! I don't work that way, so this hadn't occurred to me. No wonder I didn't follow the discussion on "develop" :-) I'll have to think about this. > This is easily set on a per-user or sitewide basis; in a > shop making use of these tools, you'd probably do this (as root): > > setup.py setopt -g -c develop -o install_dir -s /somewhere/staging Ooh, these setup commands that do global stuff unrelated to the package whose setup.py is being used confuse me! More to think about... > There's a get_python_lib() function in distutils that I use. However, I've > recently discovered that the situation on Mac OS X is more complex; users > have a ~/Library/python2.X/site-packages directory as well. > > > >Sorry - I'm doing find_distribution() on every entry in sys.path. What > >I was getting at, is whether that process could miss any eggs which > >easy_install may have put into site-packages. > > No; find_distributions(site_packages) would list everything installed in > that directory, whether active or not, plus distributions for anything that > was installed as a development link in site-packages. That is, even if > *all* you call find_distributions() on is site-packages, it's still going > to include distributions whose .path is elsewhere, and that's fine because > you want to know about those anyway; just list them with a "(Development)" > status or something in your output. Sounds reasonable. There are lots of complexities here I hadn't considered, because I'm working purely from the POV of a *user* of eggs, not a *producer* of them. Maybe that's why bdist_wininst installers, simplistic though they are, suit a lot of my needs. (But that's leading onto another philosophical ramble, which I'll spare you... :-)) Paul. From dangoor at gmail.com Thu Jul 14 17:56:34 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Thu, 14 Jul 2005 11:56:34 -0400 Subject: [Distutils] Adding egg support (Resource API) to existing packages In-Reply-To: <79990c6b05071408443e900c6a@mail.gmail.com> References: <79990c6b05071303201318ac71@mail.gmail.com> <5.1.1.6.0.20050713115507.0387a9c0@mail.telecommunity.com> <79990c6b050714022654f1f809@mail.gmail.com> <5.1.1.6.0.20050714101747.02680170@mail.telecommunity.com> <79990c6b05071408443e900c6a@mail.gmail.com> Message-ID: <3f085ecd050714085664b24262@mail.gmail.com> On 7/14/05, Paul Moore wrote: > You don't care about eggs (yet). Your users (one of whom is me) do, > and would like to package your code in an egg via python setup.py > --compiler-package=setuptools.command bdist_egg. This works fine, with > no source changes, but the resulting egg isn't zip-safe because of > resource usage issues. While I agree with Phillip on this one, I do know what you're saying. I've wondered that about some of the packages I'm putting into eggs. I'm pretty certain this situation will arise where a project maintainer won't go for a setuptools dependency. Here's one way to deal with it: Let's say the project's code is in the package: pythonfoobar What if you toss pkg_resources into pythonfoobar. Then, in all of the modules you touch, you just do: from pythonfoobar import pkg_resources Then, once the package maintainer comes to his or her senses and realizes that the world is A Better Place With Eggs (tm), there are two simple ways to introduce the setuptools dependency: 1) project wide search and replace 2) in pythonfoobar/__init__ just import pkg_resources (this is assuming you're not using the fancy setuptools namespaces... not many projects need that, though.) Ideally, people will go along with setuptools. But, at least this workaround won't break people's systems. Kevin From pje at telecommunity.com Thu Jul 14 17:55:42 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 14 Jul 2005 11:55:42 -0400 Subject: [Distutils] Adding egg support (Resource API) to existing packages In-Reply-To: <79990c6b05071408443e900c6a@mail.gmail.com> References: <5.1.1.6.0.20050714101747.02680170@mail.telecommunity.com> <79990c6b05071303201318ac71@mail.gmail.com> <5.1.1.6.0.20050713115507.0387a9c0@mail.telecommunity.com> <79990c6b050714022654f1f809@mail.gmail.com> <5.1.1.6.0.20050714101747.02680170@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050714115240.029a76a0@mail.telecommunity.com> At 04:44 PM 7/14/2005 +0100, Paul Moore wrote: >On 7/14/05, Phillip J. Eby wrote: > > At 10:26 AM 7/14/2005 +0100, Paul Moore wrote: > > >What I'm saying, is that I see a need for *another* transitional > > >process - enabling users and/or package authors to make their code > > >"resource safe", *without* having to buy into the egg/setuptools > > >transition. > > > > It still seems to me that you're missing something. If I don't care about > > eggs, why would I care about resource safety? Its only benefit is to allow > > my code to be put in an egg! :) > >You don't care about eggs (yet). Your users (one of whom is me) do, >and would like to package your code in an egg via python setup.py >--compiler-package=setuptools.command bdist_egg. This works fine, with >no source changes, but the resulting egg isn't zip-safe because of >resource usage issues. > >Your user (me) can maintain a small local patch to switch to the >resource API, but it would be nice to fold that patch into the >distribution, and so avoid the need for a local patch. But the patch >that is small for your user with setuptools installed, is large *for >you* because you don't (yet) buy into the benefits of eggs (and hence >the patch adds a dependency you don't want). So you reject the patch. So give them a patch that's egg-safe *without* using pkg_resources! It'll be slightly more complicated to implement, but won't introduce *any* dependencies. Then use "easy_install -z" to force it into a zipped egg. Voila, you're done. From pje at telecommunity.com Thu Jul 14 18:14:58 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 14 Jul 2005 12:14:58 -0400 Subject: [Distutils] Adding egg support (Resource API) to existing packages In-Reply-To: <3f085ecd050714085664b24262@mail.gmail.com> References: <79990c6b05071408443e900c6a@mail.gmail.com> <79990c6b05071303201318ac71@mail.gmail.com> <5.1.1.6.0.20050713115507.0387a9c0@mail.telecommunity.com> <79990c6b050714022654f1f809@mail.gmail.com> <5.1.1.6.0.20050714101747.02680170@mail.telecommunity.com> <79990c6b05071408443e900c6a@mail.gmail.com> Message-ID: <5.1.1.6.0.20050714115733.036f6928@mail.telecommunity.com> At 11:56 AM 7/14/2005 -0400, Kevin Dangoor wrote: >On 7/14/05, Paul Moore wrote: > > You don't care about eggs (yet). Your users (one of whom is me) do, > > and would like to package your code in an egg via python setup.py > > --compiler-package=setuptools.command bdist_egg. This works fine, with > > no source changes, but the resulting egg isn't zip-safe because of > > resource usage issues. > >While I agree with Phillip on this one, I do know what you're saying. >I've wondered that about some of the packages I'm putting into eggs. > >I'm pretty certain this situation will arise where a project >maintainer won't go for a setuptools dependency. Here's one way to >deal with it: > >Let's say the project's code is in the package: > >pythonfoobar > >What if you toss pkg_resources into pythonfoobar. Then, in all of the >modules you touch, you just do: > >from pythonfoobar import pkg_resources *shudder*. Please don't. pkg_resources needs to be a singleton, partly because sys.path is a singleton, and partly because there needs to be a single place to register adapters for PEP 302 loaders and importers, and in the future, egg listeners. If you guys *really* want to do this, what we need to do is create a super-simple "mini-Me" form of pkg_resources, that provides exactly two functions: resource_string and resource_stream. The code for these will be very short and relatively simple to do, and will not impact any other part of the egg runtime system; it will just allow portable resource access. We can call it 'pkg_resources_lite', or something similar, and I can change the safety analysis to ignore the fact it uses __file__ (i.e. if the module's named pkg_resources_lite). >2) in pythonfoobar/__init__ just import pkg_resources (this is >assuming you're not using the fancy setuptools namespaces... not many >projects need that, though.) Namespaces...? Do you mean namespace packages? >Ideally, people will go along with setuptools. But, at least this >workaround won't break people's systems. Um, yeah. I think, after all this discussion, that the simple way to fix this is going to be for somebody to produce and maintain a pkg_resources_lite that people can bundle inside their distributions like this. It'll even be smaller than ez_setup, although it won't come in the convenient svn:externals format of ez_setup. Hm. You know, if pkg_resources_lite is written so that it tries to import from pkg_resources, then I can make it "step aside and let the professional handle it" if the user has the full pkg_resources package. This would let it take advantage of any support for registered import hooks, etc. that the full package was doing. Something like this: import sys, os def _get_path_and_loader(package_name, resource_name): try: module = sys.modules[moduleName] except KeyError: __import__(moduleName) module = sys.modules[moduleName] loader = getattr(module, '__loader__', None) parts = resource_name.split('/') path = os.path.join(os.path.dirname(module.__file__),*parts) return path, loader def resource_string(package_name, resource_name): path,loader = _get_path_and_loader(package_name, resource_name) if loader is None: f = open(path,'rb') try: return f.read() finally: f.close() else: return loader.get_data(path) def resource_stream(package_name, resource_name): path,loader = _get_path_and_loader(package_name, resource_name) if loader is None: return open(path,'rb') else: from cStringIO import StringIO return StringIO(loader.get_data(path)) try: from pkg_resources import resource_string, resource_stream except ImportError: pass Only with some docstrings and comments added. :) Anyway, this thing would work with eggs or py2exe or just plain zipfiles. It won't necessarily work with other PEP 302 loader objects, but neither does pkg_resources unless you register suitable adapters for the functionality needed. From dangoor at gmail.com Thu Jul 14 20:40:52 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Thu, 14 Jul 2005 14:40:52 -0400 Subject: [Distutils] Adding egg support (Resource API) to existing packages In-Reply-To: <5.1.1.6.0.20050714115733.036f6928@mail.telecommunity.com> References: <79990c6b05071303201318ac71@mail.gmail.com> <5.1.1.6.0.20050713115507.0387a9c0@mail.telecommunity.com> <79990c6b050714022654f1f809@mail.gmail.com> <5.1.1.6.0.20050714101747.02680170@mail.telecommunity.com> <79990c6b05071408443e900c6a@mail.gmail.com> <3f085ecd050714085664b24262@mail.gmail.com> <5.1.1.6.0.20050714115733.036f6928@mail.telecommunity.com> Message-ID: <3f085ecd05071411405f3c1a78@mail.gmail.com> On 7/14/05, Phillip J. Eby wrote: > *shudder*. Please don't. pkg_resources needs to be a singleton, partly > because sys.path is a singleton, and partly because there needs to be a > single place to register adapters for PEP 302 loaders and importers, and in > the future, egg listeners. Ahh, OK. I hadn't looked closely at all that was going on in there. > If you guys *really* want to do this, what we need to do is create a > super-simple "mini-Me" form of pkg_resources, that provides exactly two > functions: resource_string and resource_stream. I don't really want to do this. I consider it entirely likely that I'll produce a couple of patches for other projects to make them zip-safe. I don't know yet if those projects will take the patches and the setuptools dependencies... we'll see when that time comes. Being a believer in YAGNI, I wouldn't have brought this up until someone actually told me "nope, I won't do that." The pkg_resources_lite idea seems reasonable, should this discussion leave the hypothetical. Kevin From rkern at ucsd.edu Fri Jul 15 01:12:49 2005 From: rkern at ucsd.edu (Robert Kern) Date: Thu, 14 Jul 2005 16:12:49 -0700 Subject: [Distutils] Adding egg support (Resource API) to existing packages In-Reply-To: <5.1.1.6.0.20050714115240.029a76a0@mail.telecommunity.com> References: <5.1.1.6.0.20050714101747.02680170@mail.telecommunity.com> <79990c6b05071303201318ac71@mail.gmail.com> <5.1.1.6.0.20050713115507.0387a9c0@mail.telecommunity.com> <79990c6b050714022654f1f809@mail.gmail.com> <5.1.1.6.0.20050714101747.02680170@mail.telecommunity.com> <79990c6b05071408443e900c6a@mail.gmail.com> <5.1.1.6.0.20050714115240.029a76a0@mail.telecommunity.com> Message-ID: Phillip J. Eby wrote: > So give them a patch that's egg-safe *without* using pkg_resources! It'll > be slightly more complicated to implement, but won't introduce *any* > dependencies. But first see if they'll drink the Kool Aid. Only if they really hate the idea of depending on setuptools should you bother with this strategy. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From phil_schwartz at users.sourceforge.net Sat Jul 16 19:38:30 2005 From: phil_schwartz at users.sourceforge.net (Phil Schwartz) Date: Sat, 16 Jul 2005 10:38:30 -0700 (PDT) Subject: [Distutils] Building rpms for different interpreters? Message-ID: I scanned the distutils mailing list archive to no avail, so hopefully this question hasn't been asked and answered before. Basically, my ReleaseForge application uses distutils to generate zip, tar and rpm packages. Everything has worked perfectly up until now because I now want to generate a python2.4 specific rpm in addition to a python2.3 rpm. My development system is FC3 and has rpms of Python2.3 (default system wide) and Python2.4 installed: $ rpm -q python python2.4 python-2.3.4-13.1 python2.4-2.4-1pydotorg When I issue the following command: $ python setup.py bdist_rpm the generated rpm files have references to /usr/lib/python2.3/site-packages. This is ofcourse expected since python2.3 is the default. However, what is unexepected is that when I invoke python2.4 directly to generate the rpm: $ python2.4 setup.py bdist_rpm the generated rpm still has references to /usr/lib/python2.3/site-packages rather than /usr/lib/python2.4/site-packages. $ python2.4 Python 2.4 (#1, Nov 30 2004, 11:25:14) [GCC 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)] on linux2 >> import sys >>> sys.path ['', '/usr/lib/python24.zip', '/usr/lib/python2.4', '/usr/lib/python2.4/plat-linux2', '/usr/lib/python2.4/lib-tk', '/usr/lib/python2.4/lib-dynload', '/usr/lib/python2.4/site-packages'] What I'd like to see is that when I invoke "python2.4 setup.py bdist_rpm" are for the files in the generated rpm to reference /usr/lib/python2.4/site-packages rather than the respective 2.3 directory. FWIW, the 2.3 and 2.4 generated rpms will be essentially the same (w/ the only difference being the location of site-packages) What painfully obvious point am I missing? Thanks for any help, Phil Schwartz http://releaseforge.sourceforge.net From mal at egenix.com Sat Jul 16 21:08:16 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Sat, 16 Jul 2005 21:08:16 +0200 Subject: [Distutils] Building rpms for different interpreters? In-Reply-To: References: Message-ID: <42D95B20.9080305@egenix.com> Phil Schwartz wrote: > I scanned the distutils mailing list archive to no avail, so hopefully > this question hasn't been asked and answered before. > > Basically, my ReleaseForge application uses distutils to generate zip, tar > and rpm packages. Everything has worked perfectly up until now because I > now want to generate a python2.4 specific rpm in addition to a > python2.3 rpm. > > My development system is FC3 and has rpms of Python2.3 (default system > wide) and Python2.4 installed: > > $ rpm -q python python2.4 > python-2.3.4-13.1 > python2.4-2.4-1pydotorg > > When I issue the following command: > > $ python setup.py bdist_rpm > > the generated rpm files have references to > /usr/lib/python2.3/site-packages. This is ofcourse expected since > python2.3 is the default. > > However, what is unexepected is that when I invoke python2.4 directly to > generate the rpm: > > $ python2.4 setup.py bdist_rpm > > the generated rpm still has references to /usr/lib/python2.3/site-packages > rather than /usr/lib/python2.4/site-packages. > > $ python2.4 > > Python 2.4 (#1, Nov 30 2004, 11:25:14) > [GCC 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)] on linux2 > >>>import sys >>> >>>>sys.path > > ['', '/usr/lib/python24.zip', '/usr/lib/python2.4', > '/usr/lib/python2.4/plat-linux2', '/usr/lib/python2.4/lib-tk', > '/usr/lib/python2.4/lib-dynload', '/usr/lib/python2.4/site-packages'] > > What I'd like to see is that when I invoke "python2.4 setup.py bdist_rpm" > are for the files in the generated rpm to reference > /usr/lib/python2.4/site-packages rather than the respective 2.3 directory. > > FWIW, the 2.3 and 2.4 generated rpms will be essentially the same (w/ the > only difference being the location of site-packages) > > What painfully obvious point am I missing? In addition to building with Python 2.4, you also have to point bdist_rpm to the Python interpreter to use when running the rpm build command: $(PYTHON) setup.py \ bdist_rpm \ --python $(PYTHON) \ --release $(PACKAGERELEASE) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 16 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From rkern at ucsd.edu Sat Jul 16 21:34:36 2005 From: rkern at ucsd.edu (Robert Kern) Date: Sat, 16 Jul 2005 12:34:36 -0700 Subject: [Distutils] Req: easy_install.py --site-dirs Message-ID: I tried doing a bit of EasyInstall evangelism, and apparently, the Kool-Aid isn't quite sweet enough yet. ;-) I got feedback from someone trying the very first example with SQLObject in the documentation. He is using Debian Linux and thus does not install anything that's not from a .deb into /usr/lib. User-installed Python packages need to go into /usr/local/lib/python2.x/site-packages. To encourage this, Debian's site.py is patched to add that to the list of .pth-enabled directories. I imagine that similarly conscientious/anal distributions do likewise. My friend is a bit more conservative than that, even. He manages /usr/local using GNU Stow so he can make installations as non-root. This is very important to him. He's even willing to use tricky .pth hacks[1] to permit this. Fortunately, all of his technical concerns can be addressed by adding --site-dirs. [1] As documented here: http://article.gmane.org/gmane.comp.python.distutils.devel/1895 -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From hawk.it at tiscali.it Sat Jul 16 22:19:21 2005 From: hawk.it at tiscali.it (Vincenzo Di Massa) Date: Sat, 16 Jul 2005 22:19:21 +0200 Subject: [Distutils] Req: easy_install.py --site-dirs In-Reply-To: References: Message-ID: <200507162219.21380.hawk.it@tiscali.it> Hello, I'm new to this list so I will introduce myself... I'm an italian Ph.D. student working on http://udu.wiki.ubuntu.com/PythonModulePackaging http://udu.wiki.ubuntu.com/PackagesFromPyPi as google's SummerOfCode project. Basically ubuntu (a debian derivative distro) wants: 1. a tool to create packages of python modules "automatically" from PyPI (like EasyInstall does with eggs) 2. a setup where on python upgrade all compatible python modules are migrated to the new installation. I'm quite new to distutils so I'd like comments / critics / suggestions... I hope ubuntu will adopt my work for breezy: I will actually code what we all decide is the best solution (they pay me for this work). Regards Vincenzo Alle 21:34, sabato 16 luglio 2005, Robert Kern ha scritto: > I tried doing a bit of EasyInstall evangelism, and apparently, the > Kool-Aid isn't quite sweet enough yet. ;-) > > I got feedback from someone trying the very first example with SQLObject > in the documentation. He is using Debian Linux and thus does not install > anything that's not from a .deb into /usr/lib. User-installed Python > packages need to go into /usr/local/lib/python2.x/site-packages. To > encourage this, Debian's site.py is patched to add that to the list of > .pth-enabled directories. I imagine that similarly conscientious/anal > distributions do likewise. > > My friend is a bit more conservative than that, even. He manages > /usr/local using GNU Stow so he can make installations as non-root. This > is very important to him. He's even willing to use tricky .pth hacks[1] > to permit this. > > Fortunately, all of his technical concerns can be addressed by adding > --site-dirs. > > [1] As documented here: > http://article.gmane.org/gmane.comp.python.distutils.devel/1895 From pje at telecommunity.com Sat Jul 16 23:58:48 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 16 Jul 2005 17:58:48 -0400 Subject: [Distutils] Req: easy_install.py --site-dirs In-Reply-To: Message-ID: <5.1.1.6.0.20050716175658.029d9f80@mail.telecommunity.com> Okay, I've checked in a preliminary version of --site-dirs support in the Python CVS HEAD of nondist/sandbox/setuptools; why don't you give that a try? It may be a few hours before it's available for anonymous checkout, however, as Sourceforge only updates their anoncvs snapshots every six hours or so, IIRC. If you have Python developer privileges, you can get it sooner than that. At 12:34 PM 7/16/2005 -0700, Robert Kern wrote: >I tried doing a bit of EasyInstall evangelism, and apparently, the >Kool-Aid isn't quite sweet enough yet. ;-) > >I got feedback from someone trying the very first example with SQLObject >in the documentation. He is using Debian Linux and thus does not install >anything that's not from a .deb into /usr/lib. User-installed Python >packages need to go into /usr/local/lib/python2.x/site-packages. To >encourage this, Debian's site.py is patched to add that to the list of >.pth-enabled directories. I imagine that similarly conscientious/anal >distributions do likewise. > >My friend is a bit more conservative than that, even. He manages >/usr/local using GNU Stow so he can make installations as non-root. This >is very important to him. He's even willing to use tricky .pth hacks[1] >to permit this. > >Fortunately, all of his technical concerns can be addressed by adding >--site-dirs. > >[1] As documented here: >http://article.gmane.org/gmane.comp.python.distutils.devel/1895 > >-- >Robert Kern >rkern at ucsd.edu > >"In the fields of hell where the grass grows high > Are the graves of dreams allowed to die." > -- Richard Harter > >_______________________________________________ >Distutils-SIG maillist - Distutils-SIG at python.org >http://mail.python.org/mailman/listinfo/distutils-sig From ianb at colorstudy.com Sun Jul 17 01:17:25 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sat, 16 Jul 2005 18:17:25 -0500 Subject: [Distutils] easy_install: fragment support Message-ID: <42D99585.1020102@colorstudy.com> I've attached a patch to detect fragments. There was a couple ideas bandied about; this seems the simplest. You use a link do: url#egg=package You can include a version, otherwise it's unversioned. The only package I've tested it on is "flup", listed on http://pythonpaste.org/package_index.html -- in that case there's no competing or versioned listings in PyPI; if there was the entry would simply be ignored and would be useless. PJE aso mentioned calling the packages things like flup_devel, so that it's a completely different package from flup. The problem, then, is that requirements can't be satisfied by the development package. It seems better to have some magic version string for development, that is neither more or less than other versions (or at least it depends on context -- it's the highest version number when using --develop [should that be implemented], and the lowest otherwise). But that's only useful given a --develop option to easy_install.py. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: package_index_fragment.diff Url: http://mail.python.org/pipermail/distutils-sig/attachments/20050716/85969276/package_index_fragment.diff From pje at telecommunity.com Sun Jul 17 04:50:45 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 16 Jul 2005 22:50:45 -0400 Subject: [Distutils] easy_install: fragment support In-Reply-To: <42D99585.1020102@colorstudy.com> Message-ID: <5.1.1.6.0.20050716224813.03717f68@mail.telecommunity.com> At 06:17 PM 7/16/2005 -0500, Ian Bicking wrote: >I've attached a patch to detect fragments. Would you mind doing it as a unidiff? I'm having trouble understanding/applying it without context. Thanks. >It seems better to have some magic version string for development, that is >neither more or less than other versions (or at least it depends on >context -- it's the highest version number when using --develop [should >that be implemented], and the lowest otherwise). But that's only useful >given a --develop option to easy_install.py. I was actually thinking that such packages should have their Distribution object be of a new 'type', which would have lower precedence than other distribution types unless development mode was in effect (in which case other distribution types would be ignored). From ianb at colorstudy.com Sun Jul 17 05:03:32 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sat, 16 Jul 2005 22:03:32 -0500 Subject: [Distutils] easy_install: fragment support In-Reply-To: <5.1.1.6.0.20050716224813.03717f68@mail.telecommunity.com> References: <5.1.1.6.0.20050716224813.03717f68@mail.telecommunity.com> Message-ID: <42D9CA84.1070501@colorstudy.com> Phillip J. Eby wrote: > At 06:17 PM 7/16/2005 -0500, Ian Bicking wrote: > >> I've attached a patch to detect fragments. > > > Would you mind doing it as a unidiff? I'm having trouble > understanding/applying it without context. Thanks. Sure, attached again... >> It seems better to have some magic version string for development, >> that is neither more or less than other versions (or at least it >> depends on context -- it's the highest version number when using >> --develop [should that be implemented], and the lowest otherwise). >> But that's only useful given a --develop option to easy_install.py. > > > I was actually thinking that such packages should have their > Distribution object be of a new 'type', which would have lower > precedence than other distribution types unless development mode was in > effect (in which case other distribution types would be ignored). You mean like a Distribution subclass? Would that class just be used anytime there wasn't any version information, or would there be a way of indicating that it was a development version in the filename/fragment? -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: package_index_fragment.diff Url: http://mail.python.org/pipermail/distutils-sig/attachments/20050716/24956b5c/package_index_fragment-0001.diff From pje at telecommunity.com Sun Jul 17 05:37:45 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 16 Jul 2005 23:37:45 -0400 Subject: [Distutils] easy_install: fragment support In-Reply-To: <42D9CA84.1070501@colorstudy.com> References: <5.1.1.6.0.20050716224813.03717f68@mail.telecommunity.com> <5.1.1.6.0.20050716224813.03717f68@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050716233331.02695fb0@mail.telecommunity.com> At 10:03 PM 7/16/2005 -0500, Ian Bicking wrote: >Phillip J. Eby wrote: >>At 06:17 PM 7/16/2005 -0500, Ian Bicking wrote: >> >>>I've attached a patch to detect fragments. >> >>Would you mind doing it as a unidiff? I'm having trouble >>understanding/applying it without context. Thanks. > >Sure, attached again... Thanks; I hope you don't mind if I don't include the "make emacs happy" bit that I don't understand. :) >>>It seems better to have some magic version string for development, that >>>is neither more or less than other versions (or at least it depends on >>>context -- it's the highest version number when using --develop [should >>>that be implemented], and the lowest otherwise). >>>But that's only useful given a --develop option to easy_install.py. >> >>I was actually thinking that such packages should have their Distribution >>object be of a new 'type', which would have lower precedence than other >>distribution types unless development mode was in effect (in which case >>other distribution types would be ignored). > >You mean like a Distribution subclass? No, I mean a distribution type (the 'distro_type' attribute of Distribution). There are currently type codes for eggs, binary distributions (e.g. win32.exe), and source distributions. The type code is a tie-breaker in sorting, when two distributions have the same version; if more than one suitable package of the highest version is available, EasyInstall chooses to use eggs or binaries in preference to source distributions. Similarly, I was thinking we would select a type code that would cause CVS/Subversion checkouts to have an even lower precedence unless --develop was supplied. I'm also thinking that --develop should take an argument: a parent directory in which the package(s) should be checked out. From rkern at ucsd.edu Sun Jul 17 06:01:41 2005 From: rkern at ucsd.edu (Robert Kern) Date: Sat, 16 Jul 2005 21:01:41 -0700 Subject: [Distutils] Req: easy_install.py --site-dirs In-Reply-To: <5.1.1.6.0.20050716175658.029d9f80@mail.telecommunity.com> References: <5.1.1.6.0.20050716175658.029d9f80@mail.telecommunity.com> Message-ID: Phillip J. Eby wrote: > Okay, I've checked in a preliminary version of --site-dirs support in the > Python CVS HEAD of nondist/sandbox/setuptools; why don't you give that a > try? It may be a few hours before it's available for anonymous checkout, > however, as Sourceforge only updates their anoncvs snapshots every six > hours or so, IIRC. If you have Python developer privileges, you can get it > sooner than that. Ah, now that's service! It works fine for me. I would suggest changing the altinstall.pth example, though. Python doesn't seem to like addsitedir('/Users/kern/lib/python2.4') Although it will be just happy with import site; site.addsitedir('/Users/kern/lib/python2.4') Thank you very much! -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From bob at redivi.com Sun Jul 17 06:03:06 2005 From: bob at redivi.com (Bob Ippolito) Date: Sat, 16 Jul 2005 18:03:06 -1000 Subject: [Distutils] easy_install: fragment support In-Reply-To: <5.1.1.6.0.20050716233331.02695fb0@mail.telecommunity.com> References: <5.1.1.6.0.20050716224813.03717f68@mail.telecommunity.com> <5.1.1.6.0.20050716224813.03717f68@mail.telecommunity.com> <5.1.1.6.0.20050716233331.02695fb0@mail.telecommunity.com> Message-ID: On Jul 16, 2005, at 5:37 PM, Phillip J. Eby wrote: > At 10:03 PM 7/16/2005 -0500, Ian Bicking wrote: > >> Phillip J. Eby wrote: >> >>> At 06:17 PM 7/16/2005 -0500, Ian Bicking wrote: >>> >>> >>>> I've attached a patch to detect fragments. >>>> >>> >>> Would you mind doing it as a unidiff? I'm having trouble >>> understanding/applying it without context. Thanks. >>> >> >> Sure, attached again... >> > > Thanks; I hope you don't mind if I don't include the "make emacs > happy" bit > that I don't understand. :) The "make emacs happy" comment is to fix syntax highlighting. It thinks there is an unclosed single quote string (I guess it doesn't understand r"""strings""" properly). -bob From ianb at colorstudy.com Sun Jul 17 06:13:29 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sat, 16 Jul 2005 23:13:29 -0500 Subject: [Distutils] easy_install: fragment support In-Reply-To: <5.1.1.6.0.20050716233331.02695fb0@mail.telecommunity.com> References: <5.1.1.6.0.20050716224813.03717f68@mail.telecommunity.com> <5.1.1.6.0.20050716224813.03717f68@mail.telecommunity.com> <5.1.1.6.0.20050716233331.02695fb0@mail.telecommunity.com> Message-ID: <42D9DAE9.3080308@colorstudy.com> Phillip J. Eby wrote: > At 10:03 PM 7/16/2005 -0500, Ian Bicking wrote: > >> Phillip J. Eby wrote: >> >>> At 06:17 PM 7/16/2005 -0500, Ian Bicking wrote: >>> >>>> I've attached a patch to detect fragments. >>> >>> >>> Would you mind doing it as a unidiff? I'm having trouble >>> understanding/applying it without context. Thanks. >> >> >> Sure, attached again... > > > Thanks; I hope you don't mind if I don't include the "make emacs happy" > bit that I don't understand. :) Like Bob said, it's to fix Emacs syntax highlighting, which doesn't understand triple-quoted strings. package_index.py looks like one giant string in Emacs without it. > I'm also thinking that --develop should take an argument: a parent > directory in which the package(s) should be checked out. That sounds like a good idea; I hadn't thought through exactly how that would work. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Sun Jul 17 06:17:38 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 17 Jul 2005 00:17:38 -0400 Subject: [Distutils] Req: easy_install.py --site-dirs In-Reply-To: References: <5.1.1.6.0.20050716175658.029d9f80@mail.telecommunity.com> <5.1.1.6.0.20050716175658.029d9f80@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050717001604.02696630@mail.telecommunity.com> At 09:01 PM 7/16/2005 -0700, Robert Kern wrote: >Phillip J. Eby wrote: > > Okay, I've checked in a preliminary version of --site-dirs support in the > > Python CVS HEAD of nondist/sandbox/setuptools; why don't you give that a > > try? It may be a few hours before it's available for anonymous checkout, > > however, as Sourceforge only updates their anoncvs snapshots every six > > hours or so, IIRC. If you have Python developer privileges, you can > get it > > sooner than that. > >Ah, now that's service! It works fine for me. I would suggest changing >the altinstall.pth example, though. Python doesn't seem to like > > addsitedir('/Users/kern/lib/python2.4') > >Although it will be just happy with > > import site; site.addsitedir('/Users/kern/lib/python2.4') Um, the example I put in the EasyInstall doc for --site-dirs was: import os; addsitedir(os.path.expanduser('~/lib/python2.3')) So I'm not sure what you mean here. From rkern at ucsd.edu Sun Jul 17 06:22:55 2005 From: rkern at ucsd.edu (Robert Kern) Date: Sat, 16 Jul 2005 21:22:55 -0700 Subject: [Distutils] Req: easy_install.py --site-dirs In-Reply-To: <5.1.1.6.0.20050717001604.02696630@mail.telecommunity.com> References: <5.1.1.6.0.20050716175658.029d9f80@mail.telecommunity.com> <5.1.1.6.0.20050716175658.029d9f80@mail.telecommunity.com> <5.1.1.6.0.20050717001604.02696630@mail.telecommunity.com> Message-ID: Phillip J. Eby wrote: > At 09:01 PM 7/16/2005 -0700, Robert Kern wrote: >>Ah, now that's service! It works fine for me. I would suggest changing >>the altinstall.pth example, though. Python doesn't seem to like >> >> addsitedir('/Users/kern/lib/python2.4') >> >>Although it will be just happy with >> >> import site; site.addsitedir('/Users/kern/lib/python2.4') > > Um, the example I put in the EasyInstall doc for --site-dirs was: > > import os; addsitedir(os.path.expanduser('~/lib/python2.3')) > > So I'm not sure what you mean here. Eventually someone is going to want to do addsitedir('/usr/local/stow/easy_install/lib/python2.4/site-packages') which won't work. The explicit "import site" encourages people not to tread that path. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From pje at telecommunity.com Sun Jul 17 06:35:43 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 17 Jul 2005 00:35:43 -0400 Subject: [Distutils] Req: easy_install.py --site-dirs In-Reply-To: References: <5.1.1.6.0.20050717001604.02696630@mail.telecommunity.com> <5.1.1.6.0.20050716175658.029d9f80@mail.telecommunity.com> <5.1.1.6.0.20050716175658.029d9f80@mail.telecommunity.com> <5.1.1.6.0.20050717001604.02696630@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050717003438.02981378@mail.telecommunity.com> At 09:22 PM 7/16/2005 -0700, Robert Kern wrote: >Phillip J. Eby wrote: > > At 09:01 PM 7/16/2005 -0700, Robert Kern wrote: > > >>Ah, now that's service! It works fine for me. I would suggest changing > >>the altinstall.pth example, though. Python doesn't seem to like > >> > >> addsitedir('/Users/kern/lib/python2.4') > >> > >>Although it will be just happy with > >> > >> import site; site.addsitedir('/Users/kern/lib/python2.4') > > > > Um, the example I put in the EasyInstall doc for --site-dirs was: > > > > import os; addsitedir(os.path.expanduser('~/lib/python2.3')) > > > > So I'm not sure what you mean here. > >Eventually someone is going to want to do > > addsitedir('/usr/local/stow/easy_install/lib/python2.4/site-packages') But I didn't give that (or anything like it) as an example in the documentation, so I still don't understand what it is you're asking me to do. From rkern at ucsd.edu Sun Jul 17 06:45:22 2005 From: rkern at ucsd.edu (Robert Kern) Date: Sat, 16 Jul 2005 21:45:22 -0700 Subject: [Distutils] Req: easy_install.py --site-dirs In-Reply-To: <5.1.1.6.0.20050717003438.02981378@mail.telecommunity.com> References: <5.1.1.6.0.20050717001604.02696630@mail.telecommunity.com> <5.1.1.6.0.20050716175658.029d9f80@mail.telecommunity.com> <5.1.1.6.0.20050716175658.029d9f80@mail.telecommunity.com> <5.1.1.6.0.20050717001604.02696630@mail.telecommunity.com> <5.1.1.6.0.20050717003438.02981378@mail.telecommunity.com> Message-ID: Phillip J. Eby wrote: > At 09:22 PM 7/16/2005 -0700, Robert Kern wrote: > >>Phillip J. Eby wrote: >> >>>At 09:01 PM 7/16/2005 -0700, Robert Kern wrote: >> >>>>Ah, now that's service! It works fine for me. I would suggest changing >>>>the altinstall.pth example, though. Python doesn't seem to like >>>> >>>> addsitedir('/Users/kern/lib/python2.4') >>>> >>>>Although it will be just happy with >>>> >>>> import site; site.addsitedir('/Users/kern/lib/python2.4') >>> >>>Um, the example I put in the EasyInstall doc for --site-dirs was: >>> >>> import os; addsitedir(os.path.expanduser('~/lib/python2.3')) >>> >>>So I'm not sure what you mean here. >> >>Eventually someone is going to want to do >> >> addsitedir('/usr/local/stow/easy_install/lib/python2.4/site-packages') > > But I didn't give that (or anything like it) as an example in the > documentation, so I still don't understand what it is you're asking me to do. When I went to modify the example to suit my needs, I thought, "Okay, I don't really need to use os.path.expanduser(), so I'll just remove the 'import os'." That didn't work. If instead the example had an explicit "import site; site.addsitedir(...)", I would never have tried the plain "addsitedir(...)" which doesn't work. An explicit statement that the bare "addsitedir(...)" doesn't work would be fine, too. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From bob at redivi.com Sun Jul 17 07:30:13 2005 From: bob at redivi.com (Bob Ippolito) Date: Sat, 16 Jul 2005 19:30:13 -1000 Subject: [Distutils] Req: easy_install.py --site-dirs In-Reply-To: <5.1.1.6.0.20050717001604.02696630@mail.telecommunity.com> References: <5.1.1.6.0.20050716175658.029d9f80@mail.telecommunity.com> <5.1.1.6.0.20050716175658.029d9f80@mail.telecommunity.com> <5.1.1.6.0.20050717001604.02696630@mail.telecommunity.com> Message-ID: <6CE43AEA-0ED5-45D7-AF8C-DC4F920532CB@redivi.com> On Jul 16, 2005, at 6:17 PM, Phillip J. Eby wrote: > At 09:01 PM 7/16/2005 -0700, Robert Kern wrote: > >> Phillip J. Eby wrote: >> >>> Okay, I've checked in a preliminary version of --site-dirs >>> support in the >>> Python CVS HEAD of nondist/sandbox/setuptools; why don't you give >>> that a >>> try? It may be a few hours before it's available for anonymous >>> checkout, >>> however, as Sourceforge only updates their anoncvs snapshots >>> every six >>> hours or so, IIRC. If you have Python developer privileges, you can >>> >> get it >> >>> sooner than that. >>> >> >> Ah, now that's service! It works fine for me. I would suggest >> changing >> the altinstall.pth example, though. Python doesn't seem to like >> >> addsitedir('/Users/kern/lib/python2.4') >> >> Although it will be just happy with >> >> import site; site.addsitedir('/Users/kern/lib/python2.4') >> > > Um, the example I put in the EasyInstall doc for --site-dirs was: > > import os; addsitedir(os.path.expanduser('~/lib/python2.3')) > > So I'm not sure what you mean here. addsitedir is not a built-in, that's what he means. -bob From bob at redivi.com Sun Jul 17 07:32:16 2005 From: bob at redivi.com (Bob Ippolito) Date: Sat, 16 Jul 2005 19:32:16 -1000 Subject: [Distutils] Req: easy_install.py --site-dirs In-Reply-To: References: <5.1.1.6.0.20050717001604.02696630@mail.telecommunity.com> <5.1.1.6.0.20050716175658.029d9f80@mail.telecommunity.com> <5.1.1.6.0.20050716175658.029d9f80@mail.telecommunity.com> <5.1.1.6.0.20050717001604.02696630@mail.telecommunity.com> <5.1.1.6.0.20050717003438.02981378@mail.telecommunity.com> Message-ID: <338FEC10-7700-4199-A83E-93833399FDB8@redivi.com> On Jul 16, 2005, at 6:45 PM, Robert Kern wrote: > Phillip J. Eby wrote: > >> At 09:22 PM 7/16/2005 -0700, Robert Kern wrote: >> >> >>> Phillip J. Eby wrote: >>> >>> >>>> At 09:01 PM 7/16/2005 -0700, Robert Kern wrote: >>>> >>> >>> >>>>> Ah, now that's service! It works fine for me. I would suggest >>>>> changing >>>>> the altinstall.pth example, though. Python doesn't seem to like >>>>> >>>>> addsitedir('/Users/kern/lib/python2.4') >>>>> >>>>> Although it will be just happy with >>>>> >>>>> import site; site.addsitedir('/Users/kern/lib/python2.4') >>>>> >>>> >>>> Um, the example I put in the EasyInstall doc for --site-dirs was: >>>> >>>> import os; addsitedir(os.path.expanduser('~/lib/ >>>> python2.3')) >>>> >>>> So I'm not sure what you mean here. >>>> >>> >>> Eventually someone is going to want to do >>> >>> addsitedir('/usr/local/stow/easy_install/lib/python2.4/site- >>> packages') >>> >> >> But I didn't give that (or anything like it) as an example in the >> documentation, so I still don't understand what it is you're >> asking me to do. >> > > When I went to modify the example to suit my needs, I thought, > "Okay, I > don't really need to use os.path.expanduser(), so I'll just remove the > 'import os'." That didn't work. If instead the example had an explicit > "import site; site.addsitedir(...)", I would never have tried the > plain > "addsitedir(...)" which doesn't work. > > An explicit statement that the bare "addsitedir(...)" doesn't work > would > be fine, too. pth files eval lines that start with the word "import". If there's no import there, it will add it to sys.path. The fact that addsitedir works bare is uh.. coincidence, because it happens to be eval'ed in the scope of the site module. I wouldn't write it like that. -bob From Jack.Jansen at cwi.nl Sun Jul 17 23:16:55 2005 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Sun, 17 Jul 2005 23:16:55 +0200 Subject: [Distutils] Wanted: ideas for using distutils with preprocessors/dependencies Message-ID: <0B3941A7-89BB-4892-AD69-08C7240BF3A1@cwi.nl> I'm currently looking at integrating bgen with distutils. Bgen is a little-known part of the Python core distribution: it is similar to swig, in that it generates C extension modules. In some respects it is more powerful than swig, the main one being that it reads standard C .h files in stead of adorned .i files. (Incidentally, this is also the reason it's part of core python, it is used to generate the MacOS API modules, so these are (almost) automatically updated when Apple adds new functionality. At least, that was true under MacOS9 and will be again when I get my act together:-). But bgen has a lot of disadvantages when compared to swig, the main one being that it is a rather fearsome tool to try and master. Integration with distutils is one of the things I want to do to lower that barrier. But now I'm at a loss as to how to proceed. I had a look at how swig is integrated into distutils, and I don't really like it, it smells like a hack. And, according to the comments in the source and the manual, the author agrees with me:-) Swig support is basically done in the build_ext command, by filtering out all ".i" files in the source file list very early in the process, running swig on them, and replacing them by the .c or .cpp equivalents. I can see various ways of adding bgen support, but I'm not sure which one is the best one, and/or whether there are other options. So I'd be interested in hearing what other people think, and how other packages have added a preprocessor to distutils. There's a fair amount of Python code needed to drive bgen, at least for interfaces to complex APIs (bridging C types to Python, handling callbacks, how to parse the specific .h files for this API, etc). Currently that code is in two .py files but it will be put in a class, probably modeled somewhat after Extension (but having C/C++ source files as output in stead of dynamic extension modules). What I don't know is how I'd connect this to the Extension object that will create the extension module. Ideally I'd like the bgen process to be optional. In other words, the distribution packager has three options: (a) include the bgen C output in the distribution and don't run bgen unless the end users specifically asks for it; (b) include the bgen C output but only run bgen if the normal timestamp dependencies require it; or (c) always run bgen. But it doesn't seem the Extension object currently has any support for such make-like chaining, and I'm not sure how to add it. One way would be to allow non-strings in the sources argument, and do something smart there. A similar mod could be used for libraries and extra_objects to allow chaining there too. Another way would be to add a "dependencies" argument, where those dependencies are objects that get run early, and can add their results to sources, libraries and extra_objects. I think this latter solution is probably better, as such a dependency object could modify multiple arguments of Extension in one fell swoop. As a somewhat contrived example, an "OptionalJPEGSupport" dependency could check whether the relevant libraries and include files are available to enable JPEG support in an imaging package, and then add the right source files, libraries, defines, library paths and include paths to the relevant Extension arguments. But all of this is made quite a bit more difficult (I think) by the fact the Extension doesn't really do anything, it's only a container and all the logic is in build_ext. Maybe I should follow the paradigm set by "build_clib", and add a "build_bgen" command with build_ext picking up the results? And maybe there are better solutions that I haven't thought of yet? -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From lars at ibp.de Mon Jul 18 01:08:23 2005 From: lars at ibp.de (Lars Immisch) Date: Mon, 18 Jul 2005 01:08:23 +0200 Subject: [Distutils] Wanted: ideas for using distutils with preprocessors/dependencies In-Reply-To: <0B3941A7-89BB-4892-AD69-08C7240BF3A1@cwi.nl> References: <0B3941A7-89BB-4892-AD69-08C7240BF3A1@cwi.nl> Message-ID: <42DAE4E7.70103@ibp.de> Hi, > I'm currently looking at integrating bgen with distutils. > > Bgen is a little-known part of the Python core distribution: it is > similar to swig, in that it generates C extension modules. In some > respects it is more powerful than swig, the main one being that it > reads standard C .h files in stead of adorned .i files. > (Incidentally, this is also the reason it's part of core python, it > is used to generate the MacOS API modules, so these are (almost) > automatically updated when Apple adds new functionality. At least, > that was true under MacOS9 and will be again when I get my act > together:-). But bgen has a lot of disadvantages when compared to > swig, the main one being that it is a rather fearsome tool to try and > master. Integration with distutils is one of the things I want to do > to lower that barrier. > > But now I'm at a loss as to how to proceed. I had a look at how swig > is integrated into distutils, and I don't really like it, it smells > like a hack. And, according to the comments in the source and the > manual, the author agrees with me:-) Swig support is basically done > in the build_ext command, by filtering out all ".i" files in the > source file list very early in the process, running swig on them, and > replacing them by the .c or .cpp equivalents. I've read distutils SWIG support, and I would (politely) say it is an afterthought. The main problem is that distutils is fixed on a two-stage compilation process: compile and link. Additional processing steps are just not supported. > I can see various ways of adding bgen support, but I'm not sure which > one is the best one, and/or whether there are other options. So I'd > be interested in hearing what other people think, and how other > packages have added a preprocessor to distutils. > > There's a fair amount of Python code needed to drive bgen, at least > for interfaces to complex APIs (bridging C types to Python, handling > callbacks, how to parse the specific .h files for this API, etc). > Currently that code is in two .py files but it will be put in a > class, probably modeled somewhat after Extension (but having C/C++ > source files as output in stead of dynamic extension modules). > > What I don't know is how I'd connect this to the Extension object > that will create the extension module. Ideally I'd like the bgen > process to be optional. In other words, the distribution packager has > three options: (a) include the bgen C output in the distribution and > don't run bgen unless the end users specifically asks for it; (b) > include the bgen C output but only run bgen if the normal timestamp > dependencies require it; or (c) always run bgen. > > But it doesn't seem the Extension object currently has any support > for such make-like chaining, and I'm not sure how to add it. One way > would be to allow non-strings in the sources argument, and do > something smart there. A similar mod could be used for libraries and > extra_objects to allow chaining there too. Another way would be to > add a "dependencies" argument, where those dependencies are objects > that get run early, and can add their results to sources, libraries > and extra_objects. I think this latter solution is probably better, > as such a dependency object could modify multiple arguments of > Extension in one fell swoop. As a somewhat contrived example, an > "OptionalJPEGSupport" dependency could check whether the relevant > libraries and include files are available to enable JPEG support in > an imaging package, and then add the right source files, libraries, > defines, library paths and include paths to the relevant Extension > arguments. > > But all of this is made quite a bit more difficult (I think) by the > fact the Extension doesn't really do anything, it's only a container > and all the logic is in build_ext. Maybe I should follow the paradigm > set by "build_clib", and add a "build_bgen" command with build_ext > picking up the results? And maybe there are better solutions that I > haven't thought of yet? I'd like full blown support for more compilation stages in distutils. I tried to come up with something nonobtrusive and failed. Also, the current stated goal of distutils maintenance seems to be: don't break existing installers. That flatly rules out any sort of rewrite. - Lars From pje at telecommunity.com Mon Jul 18 04:26:56 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 17 Jul 2005 22:26:56 -0400 Subject: [Distutils] Heads up: major setuptools update in CVS Message-ID: <5.1.1.6.0.20050717220626.026aecf8@mail.telecommunity.com> FYI, I've just checked in 0.6a0, which is a pretty major refactoring of certain aspects of the pkg_resources implementation and API. The original purpose was to make the API conform to the design spec I posted a few weeks ago, but in the process I ended up finding lots of latent bugs that I was able to stomp. For example, much of the setuptools/EasyInstall internals were vulnerable to being confused by case-insensitive filesytems, and to being fooled by symlinks into thinking that a pair of paths were different even though they referred to the same file. There are also a lot of new API features, mainly to do with the new WorkingSet class, which embeds the logic needed to manage a "working set" of active (importable) distributions. This allows you to do things like: # List all distributions that can be imported without another # require() call # for dist in pkg_resources.working_set: print dist You can also use the 'subscribe' method of a working set to add a callback that will be invoked for every active distribution (those already present in the working set, plus any that are added later). This is so that frameworks and extensible applications can register to scan plugins for new and useful metadata. Anyway, because of the breadth of the changes, I'd like assistance with testing on different platforms, Python versions, etc. before I turn this into an official 0.6a1 release. Note that if you have any scripts that use any pkg_resources APIs for Distribution, Requirement, or AvailableDistributions objects, they may need to be updated; see the changelog in setuptools.txt (in the CVS version of setuptools) for details of the changes. P.S. I've also implemented a stricter version of Ian Bicking's #egg syntax, and included docs on how to use it in setuptools.txt under the heading, "Making your package available for EasyInstall". I have not, however, implemented a --develop option yet. From mal at egenix.com Mon Jul 18 09:47:25 2005 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 18 Jul 2005 09:47:25 +0200 Subject: [Distutils] Wanted: ideas for using distutils with preprocessors/dependencies In-Reply-To: <0B3941A7-89BB-4892-AD69-08C7240BF3A1@cwi.nl> References: <0B3941A7-89BB-4892-AD69-08C7240BF3A1@cwi.nl> Message-ID: <42DB5E8D.6080500@egenix.com> Hi Jack, Jack Jansen wrote: > I'm currently looking at integrating bgen with distutils. > ... > > But now I'm at a loss as to how to proceed. I had a look at how swig > is integrated into distutils, and I don't really like it, it smells > like a hack. And, according to the comments in the source and the > manual, the author agrees with me:-) Swig support is basically done > in the build_ext command, by filtering out all ".i" files in the > source file list very early in the process, running swig on them, and > replacing them by the .c or .cpp equivalents. > > I can see various ways of adding bgen support, but I'm not sure which > one is the best one, and/or whether there are other options. So I'd > be interested in hearing what other people think, and how other > packages have added a preprocessor to distutils. Due to the natur of distutils, it is easily possible to add a few more stages to the build process here and there. If you just want to do some extra processing before building an extension, the simplest way to hook into the process is by extending build_ext. If you want your command to get automatically checked and processed, you have to subclass the build class itself and add the command as sub-command. For examples on how this can be done, have a look at mxSetup.py which you can find in egenix-mx-base. It has support for auto configuration, building Unix libraries and various other things we needed in distutils. Works great and distutils made it easy to add the new features to our setup.pys. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 18 2005) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From rkern at ucsd.edu Mon Jul 18 10:04:27 2005 From: rkern at ucsd.edu (Robert Kern) Date: Mon, 18 Jul 2005 01:04:27 -0700 Subject: [Distutils] Wanted: ideas for using distutils with preprocessors/dependencies In-Reply-To: <0B3941A7-89BB-4892-AD69-08C7240BF3A1@cwi.nl> References: <0B3941A7-89BB-4892-AD69-08C7240BF3A1@cwi.nl> Message-ID: Jack Jansen wrote: > I'm currently looking at integrating bgen with distutils. Take a look at scipy_distutils. One thing that you can do is pass in a function as the sources argument of Extension. For example, to build the C source from a Pyrex source: def generate_c_from_pyrex(extension, build_dir): name = extension.name.split('.')[-1] source = extension.depends[0] target = os.path.join(build_dir, name+'.c') if newer_group(extension.depends, target): options = Main.CompilationOptions( defaults=Main.default_options, output_file=target) pyrex_result = Main.compile(source, options=options) if pyrex_result.num_errors != 0: raise RuntimeError("%d errors in Pyrex compile" % pyrex_result.num_errors) return target scipy_distutils adds a build_src command that precedes build_ext that executes these functions. It's a bit of a hack, and sometimes is a bit finicky, but it works. http://www.scipy.net/cgi-bin/viewcvsx.cgi/scipy_core/scipy_distutils/ -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From dangoor at gmail.com Tue Jul 19 16:10:01 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Tue, 19 Jul 2005 10:10:01 -0400 Subject: [Distutils] setuptools and Mac OS X minor updates Message-ID: <3f085ecd0507190710133bbdfb@mail.gmail.com> This morning, I updated to Mac OS 10.4.2 and then discovered that one of my eggs could no longer be found. Rebuilding the egg gave me the answer: Darwin had bumped from 8.1 to 8.2. My guess is that an egg built under Mac OS 10.4.2 will work just fine on a 10.4.1 Mac. It would be a drag to have a separate egg for each minor revision of the OS. Should this be done differently? I seem to remember an option for setting the platform, but I wouldn't want to do something that would break how easy_install does things. Kevin From pje at telecommunity.com Tue Jul 19 16:18:08 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 19 Jul 2005 10:18:08 -0400 Subject: [Distutils] setuptools and Mac OS X minor updates In-Reply-To: <3f085ecd0507190710133bbdfb@mail.gmail.com> Message-ID: <5.1.1.6.0.20050719101443.028c4dc0@mail.telecommunity.com> At 10:10 AM 7/19/2005 -0400, Kevin Dangoor wrote: >This morning, I updated to Mac OS 10.4.2 and then discovered that one >of my eggs could no longer be found. Rebuilding the egg gave me the >answer: Darwin had bumped from 8.1 to 8.2. > >My guess is that an egg built under Mac OS 10.4.2 will work just fine >on a 10.4.1 Mac. It would be a drag to have a separate egg for each >minor revision of the OS. > >Should this be done differently? I seem to remember an option for >setting the platform, but I wouldn't want to do something that would >break how easy_install does things. The get_platform() and compatible_platforms() functions in pkg_resources are the place to look. If you have a good understanding of a particular platform, please propose a standard for coding and interpreting its platform tag. Win32 is my main development platform, and "win32" is all Python ever generates as a platform, so I have no knowledge of what the issues may be on various other systems, especially Mac (which I have zero experience with). But those two functions are wide open for patches to deal with the vagaries of other operating systems, as you'll see from the XXX comments there. Please propose algorithms or submit patches for operating systems you're familiar with. Thanks! From dangoor at gmail.com Tue Jul 19 16:43:22 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Tue, 19 Jul 2005 10:43:22 -0400 Subject: [Distutils] setuptools and Mac OS X minor updates In-Reply-To: <5.1.1.6.0.20050719101443.028c4dc0@mail.telecommunity.com> References: <3f085ecd0507190710133bbdfb@mail.gmail.com> <5.1.1.6.0.20050719101443.028c4dc0@mail.telecommunity.com> Message-ID: <3f085ecd050719074332d24efd@mail.gmail.com> On 7/19/05, Phillip J. Eby wrote: > The get_platform() and compatible_platforms() functions in pkg_resources > are the place to look. If you have a good understanding of a particular > platform, please propose a standard for coding and interpreting its > platform tag. Win32 is my main development platform, and "win32" is all > Python ever generates as a platform, so I have no knowledge of what the > issues may be on various other systems, especially Mac (which I have zero > experience with). OK. I'll move my question over to the python-mac list to see how it *should* work. I'm not familiar enough with the cross-version issues on Mac OS to definitively say what the answer should be... Kevin From njriley at uiuc.edu Wed Jul 20 00:06:06 2005 From: njriley at uiuc.edu (Nicholas Riley) Date: Tue, 19 Jul 2005 17:06:06 -0500 Subject: [Distutils] setuptools.pth being installed with wrong permissions Message-ID: <20050719220606.GA2117@uiuc.edu> Hi, I was installing setuptools on another machine so I could use the egg-based plugin support in the upcoming Trac 0.9 (projects.edgewall.com/trac). It's looking really neat - you'll be able to upload an egg via the Web and have it installed into the currently active Trac, or just drop an egg into a directory. Previously I had no version of setuptools installed on this machine, so I downloaded ez_setup.py and ran it. It didn't work the first time, because setuptools.pth got installed with the wrong permissions (my umask is 077). The egg itself had the correct permissions, however. -- Nicholas Riley | From pje at telecommunity.com Wed Jul 20 03:51:45 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 19 Jul 2005 21:51:45 -0400 Subject: [Distutils] setuptools.pth being installed with wrong permissions In-Reply-To: <20050719220606.GA2117@uiuc.edu> Message-ID: <5.1.1.6.0.20050719214359.028565d8@mail.telecommunity.com> At 05:06 PM 7/19/2005 -0500, Nicholas Riley wrote: >Hi, > >I was installing setuptools on another machine so I could use the >egg-based plugin support in the upcoming Trac 0.9 >(projects.edgewall.com/trac). It's looking really neat - you'll be >able to upload an egg via the Web and have it installed into the >currently active Trac, or just drop an egg into a directory. Cool. Note that if you use EasyInstall, you could also just have people type in the name or URL of a package, and then Trac could find and install it. Also, as of the CVS HEAD of setuptools, there are now facilities you can easily use to perform an action upon availability of an egg. So, you could look for a 'trac-config.py' metadata file (that the package distributor puts in their package.egg-info directory before building) and run it so the egg can register its Trac modules, if any. >Previously I had no version of setuptools installed on this machine, >so I downloaded ez_setup.py and ran it. It didn't work the first >time, because setuptools.pth got installed with the wrong permissions >(my umask is 077). The egg itself had the correct permissions, >however. Interesting. Did easy-install.pth get the right permissions, or no? I'd have expected them to end up with the same permissions. Directories probably get correct permissions, but I'm not doing any explicit permissions setting anywhere. By the way, I'd suggest testing against the current CVS HEAD as soon as practical, because there are some significant API changes in there. Also, that's where any permissions bug fix would go, at least until 0.6a1 is released. From njriley at uiuc.edu Wed Jul 20 04:10:54 2005 From: njriley at uiuc.edu (Nicholas Riley) Date: Tue, 19 Jul 2005 21:10:54 -0500 Subject: [Distutils] setuptools.pth being installed with wrong permissions In-Reply-To: <5.1.1.6.0.20050719214359.028565d8@mail.telecommunity.com> References: <20050719220606.GA2117@uiuc.edu> <5.1.1.6.0.20050719214359.028565d8@mail.telecommunity.com> Message-ID: <20050720021054.GA14309@uiuc.edu> On Tue, Jul 19, 2005 at 09:51:45PM -0400, Phillip J. Eby wrote: > Cool. Note that if you use EasyInstall, you could also just have people > type in the name or URL of a package, and then Trac could find and install > it. I'll suggest this as another option. > >Previously I had no version of setuptools installed on this machine, > >so I downloaded ez_setup.py and ran it. It didn't work the first > >time, because setuptools.pth got installed with the wrong permissions > >(my umask is 077). The egg itself had the correct permissions, > >however. > > Interesting. Did easy-install.pth get the right permissions, or no? I'd > have expected them to end up with the same permissions. Directories > probably get correct permissions, but I'm not doing any explicit > permissions setting anywhere. No, it didn't get the right permissions either, at least when I tried it this time. This appears to have been a red herring; I actually did two installs, the second as root so easy_install.py could go into /usr/local/bin, and the second install overwrote everything from the first install -except- the .pth files. The permissions were wrong on the contents of the egg in the first install, too. > By the way, I'd suggest testing against the current CVS HEAD as soon as > practical, because there are some significant API changes in there. Also, > that's where any permissions bug fix would go, at least until 0.6a1 is > released. I just confirmed I see the same thing with a simple python setup.py install on the CVS HEAD setuptools. This may just be a generic distutils problem that I've never seen before. -- Nicholas Riley | From xemacs-beta at xemacs.org Wed Jul 20 13:29:11 2005 From: xemacs-beta at xemacs.org (xemacs-beta@xemacs.org) Date: Wed, 20 Jul 2005 13:29:11 +0200 Subject: [Distutils] Returned mail: see transcript for details Message-ID: <20050720113008.32A171E4002@bag.python.org> This message was undeliverable due to the following reason(s): Your message could not be delivered because the destination computer was unreachable within the allowed queue period. The amount of time a message is queued before it is returned depends on local configura- tion parameters. Most likely there is a network problem that prevented delivery, but it is also possible that the computer is turned off, or does not have a mail system running right now. Your message was not delivered within 8 days: Host 105.68.58.128 is not responding. The following recipients did not receive this message: Please reply to postmaster at python.org if you feel this message to be in error. -------------- next part -------------- A non-text attachment was scrubbed... Name: document.zip Type: application/octet-stream Size: 29466 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20050720/5990b419/document-0001.obj From ianb at colorstudy.com Wed Jul 20 21:41:19 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 20 Jul 2005 14:41:19 -0500 Subject: [Distutils] easy_install: application description files Message-ID: <42DEA8DF.1020908@colorstudy.com> Are there currently any thoughts about application descriptions? For reasons actually unrelated to easy_install, I'm going to be adding files describing applications at work. These files contain all sorts of company-specific metadata. I'd like to include descriptions of dependencies, for use by various tools (or maybe just by easy_install, should it do everything I want). Right now all the metadata goes in setup(), which is perhaps okay, but requires distutils/setuptools to implement all the features, because the file itself isn't (realistically) parseable without running it. But in this context an ini file would probably be better. Phillip, I know you like the ini files, so you must have something in mind for this. I'm not seeing anything particular in the documentation, though. Or do I put everything in setup() and use some of the files produced by that? I see .egg-info/requires.txt contains the requirement information (BTW, it's missing a trailing newline). Should I just use that directory as my application-metadata storage area? Would I add that directory to my repository? If so, is it recommended I ignore top_level.txt and requires.txt as derivative? Completely unrelated, easy_install installs executable .pyc files. Is that intended? They actually don't seem to be .pyc files either, just copies of the .py files. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Wed Jul 20 22:16:01 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 20 Jul 2005 16:16:01 -0400 Subject: [Distutils] easy_install: application description files In-Reply-To: <42DEA8DF.1020908@colorstudy.com> Message-ID: <5.1.1.6.0.20050720160514.026ce0e0@mail.telecommunity.com> At 02:41 PM 7/20/2005 -0500, Ian Bicking wrote: >For reasons actually unrelated to easy_install, I'm going to be adding >files describing applications at work. These files contain all sorts of >company-specific metadata. I'd like to include descriptions of >dependencies, for use by various tools (or maybe just by easy_install, >should it do everything I want). > >Right now all the metadata goes in setup(), which is perhaps okay, but >requires distutils/setuptools to implement all the features, because the >file itself isn't (realistically) parseable without running it. But in >this context an ini file would probably be better. Phillip, I know you >like the ini files, so you must have something in mind for this. I'm >not seeing anything particular in the documentation, though. > >Or do I put everything in setup() and use some of the files produced by >that? I see .egg-info/requires.txt contains the requirement information >(BTW, it's missing a trailing newline). Should I just use that >directory as my application-metadata storage area? Yes. > Would I add that directory to my repository? Yes. >If so, is it recommended I ignore top_level.txt and requires.txt as >derivative? That's up to you. They're overwritten whenever setuptools does something that needs them. (That goes for the PKG-INFO file as well.) >Completely unrelated, easy_install installs executable .pyc files. Is >that intended? They actually don't seem to be .pyc files either, just >copies of the .py files. Ah, I see it now; if you name a script with a .py extension in an egg that's unpacked to a directory, it ends up compiling the script during the byte-compilation pass, and then the script installation pass thinks they're two scripts. I'll fix this in CVS sometime tonight. From BounceReplies at fmglobal.com Fri Jul 22 15:12:25 2005 From: BounceReplies at fmglobal.com (BounceReplies@fmglobal.com) Date: Fri, 22 Jul 2005 14:12:25 +0100 (BST) Subject: [Distutils] Mail Delivery (failure ricardo.vieira@fmglobal.com) (FMGMSWRET-02EX) Message-ID: <20050722131410.0F75B1E4007@bag.python.org> Your message has not been delivered to its intended recipient at FM Global. The e-mail's content and/or its attached files may not comply with FM Global's Electronic Communications Policy. If this is a legitimate business e-mail, please forward this e-mail to ReturnedMail at fmglobal.com. You will receive an acknowledgement once it has been received. We will then review the message content and determine if it should be released to the intended recipient. You may also contact the FM Global Corporate Help Desk in Johnston, R.I., USA at +1 (1)401 275 3000 x 1299. From hawk78_it at yahoo.it Sat Jul 23 10:48:53 2005 From: hawk78_it at yahoo.it (Vincenzo Di Massa) Date: Sat, 23 Jul 2005 10:48:53 +0200 Subject: [Distutils] Easy Install source download Message-ID: <200507231048.53319.hawk78_it@yahoo.it> Hi, I'm developing an easy_install based packager. It will create distribution specific packages containing eggs. (deb, rpm...) I would like to generate source packages from which binary egg-packages are built. I spent one day looking at setuptools code, but I did not find how to force easy_install to download a source distribution even if an egg is available. I think package_index has code I can call to download just the tar.gz or zip source... but how? thanks in advance Vincenzo From pje at telecommunity.com Sat Jul 23 17:37:15 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 23 Jul 2005 11:37:15 -0400 Subject: [Distutils] Easy Install source download In-Reply-To: <200507231048.53319.hawk78_it@yahoo.it> Message-ID: <5.1.1.6.0.20050723113251.026e5958@mail.telecommunity.com> At 10:48 AM 7/23/2005 +0200, Vincenzo Di Massa wrote: >Hi, >I'm developing an easy_install based packager. >It will create distribution specific packages containing eggs. (deb, rpm...) > >I would like to generate source packages from which binary egg-packages are >built. > >I spent one day looking at setuptools code, but I did not find how to force >easy_install to download a source distribution even if an egg is available. > >I think package_index has code I can call to download just the tar.gz or zip >source... but how? Just give easy_install the URL of the source archive. You can use a standalone PackageIndex to do your PyPI search ahead of time; just use the find_packages() method, then do something similar to what the obtain() function does to iterate over the available packages. A package with a precedence of SOURCE_DIST or CHECKOUT_DIST is presumably what you want, and its 'location' attribute will be what you should give to easy_install. (Note that what I'm saying is correct only for the CVS HEAD (0.6a0) of setuptools, not the released 0.5 versions, which don't have a 'location' attribute or CHECKOUT_DIST precedence.) From pje at telecommunity.com Sun Jul 24 04:42:57 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 23 Jul 2005 22:42:57 -0400 Subject: [Distutils] --develop => --editable + --build-directory Message-ID: <5.1.1.6.0.20050723191944.026e3480@mail.telecommunity.com> I tried to implement a --develop option for EasyInstall, but ran into a few snags that forced me to rethink the option a bit. Here's what I've settled on, that can actually be made to work. I'm adding an --editable/-e option, which checks out or extracts a package to a project-named subdirectory of the --build-directory. Unlike the original concept of a "--develop" option, this does *not* run the "develop" command on the checked out or extracted package. It simply fetches and extracts it for you. The problem I had was that it's very difficult to integrate the target package's setup script with the parent easy_install command in order to maintain state properly between the two. In particular, there's no way to tell if a setup script uses setuptools or not, and no way to ensure that the setup script isn't going to just modify stuff on its own, as some packages' setup scripts do. Using the sandbox facility doesn't really help, since the "develop" command *needs* to be able to write to the install directory. So, in the end, the simplest thing was just to make it "--editable" and then you can go to the directory and run "setup.py develop" yourself, which will also take care of that package's dependencies. (Or of course you can run "easy_install ." to install it normally, again with its dependencies.) In order to use --editable, you must also specify a --build-directory. Checked-out or extracted packages are copied into that directory. If you specify --build-directory but not --editable, then if there is already a package with the desired name in the build directory, you'll just get a warning. But if you specify --editable and a package of that name exists, you'll get an error message and an abort. To use --editable, you must also specify only project names and versions on the command line. If you want to download from a URL or filename, you must specify it as a --find-links/-f value instead. (Note that this means you need to tag checkout URLs with a "#egg=" suffix.) Finally, if you want to force EasyInstall to check out a subversion URL instead of a binary distribution, you'll need to ensure that you specify a version that matches the checkout link. I recommend using #egg=projectname-dev to tag links to development checkouts, so that people can do:: easy_install -eb ~/projects projectname==dev to specifically check out the development version in editable form to their ~/projects/projectname directory. The --editable flag causes easy_install to only look for source distributions or checkouts, but a source distribution will take precedence over a checkout if both match the specified version. All in all, there some caveats to using --editable, but that's mainly because there aren't many #egg links out there yet, nor are conventions for using them established as yet. Anyway, this is all now in the CVS HEAD of setuptools; see EasyInstall.txt for more docs and examples. From pje at telecommunity.com Sun Jul 24 18:49:03 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 24 Jul 2005 12:49:03 -0400 Subject: [Distutils] [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config In-Reply-To: <42E359AB.5010002@colorstudy.com> Message-ID: <5.1.1.6.0.20050724114256.0272b7e0@mail.telecommunity.com> [cc:ed to distutils-sig because much of the below is about a new egg feature; follow-ups about the web stuff should stay on web-sig] At 04:04 AM 7/24/2005 -0500, Ian Bicking wrote: >So maybe here's a deployment spec we can start with. It looks like: > > [feature1] > someapplication.somemodule.some_function > > [feature2] > someapplication.somemodule.some_function2 > >You can't get dumber than that! There should also be a "no-feature" >section; maybe one without a section identifier, or some special section >identifier. > >It goes in the .egg-info directory. This way elsewhere you can say: > > application = SomeApplication[feature1] I like this a lot, although for a different purpose than the format Chris and I were talking about. I see this fitting into that format as maybe: [feature1 from SomeApplication] # configuration here >And it's quite unambiguous. Note that there is *no* "configuration" in >the egg-info file, because you can't put any configuration related to a >deployment in an .egg-info directory, because it's not specific to any >deployment. Obviously we still need a way to get configuration in >there, but lets say that's a different matter. Easily fixed via what I've been thinking of as the "deployment descriptor"; I would call your proposal here the "import map". Basically, an import map describes a mapping from some sort of feature name to qualified names in the code. I have an extension that I would make, though. Instead of using sections for features, I would use name/value pairs inside of sections named for the kind of import map. E.g.: [wsgi.app_factories] feature1 = somemodule:somefunction feature2 = another.module:SomeClass ... [mime.parsers] application/atom+xml = something:atom_parser ... In other words, feature maps could be a generic mechanism offered by setuptools, with a 'Distribution.load_entry_point(kind,name)' API to retrieve the desired object. That way, we don't end up reinventing this idea for dozens of frameworks or pluggable applications that just need a way to find a few simple entry points into the code. In addition to specifying the entry point, each entry in the import map could optionally list the "extras" that are required if that entry point is used. It could also issue a 'require()' for the corresponding feature if it has any additional requirements listed in the extras_require dictionary. So, I'm thinking that this would be implemented with an entry_points.txt file in .egg-info, but supplied in setup.py like this: setup( ... entry_points = { "wsgi.app_factories": dict( feature1 = "somemodule:somefunction", feature2 = "another.module:SomeClass [extra1,extra2]", ), "mime.parsers": { "application/atom+xml": "something:atom_parser [feedparser]" } }, extras_require = dict( feedparser = [...], extra1 = [...], extra2 = [...], ) ) Anyway, this would make the most common use case for eggs-as-plugins very easy: an application or framework would simply define entry points, and plugin projects would declare the ones they offer in their setup script. I think this is a fantastic idea and I'm about to leap into implementing it. :) >This puts complex middleware construction into the function that is >referenced. This function might be, in turn, an import from a >framework. Or it might be some complex setup specific to the >application. Whatever. > >The API would look like: > > wsgiapp = wsgiref.get_egg_application('SomeApplication[feature1]') > >Which ultimately resolves to: > > wsgiapp = some_function() > >get_egg_application could also take a pkg_resources.Distribution object. Yeah, I'm thinking that this could be implemented as something like: import pkg_resources def get_wsgi_app(project_name, app_name, *args, **kw): dist = pkg_resources.require(project_name)[0] return dist.load_entry_point('wsgi.app_factories', app_name)(*args,**kw) with all the heavy lifting happening in the pkg_resources.Distribution class, along with maybe a new EntryPoint class (to handle parsing entry point specifiers and to do the loading of them. >Open issues? Yep, there's a bunch. This requires the rest of the >configuration to be done quite lazily. Not sure I follow you; the deployment descriptor could contain all the configuration; see the Web-SIG post I made just previous to this one. > But I can fit this into source >control; it is about *all* I can fit into source control (I can't have >any filenames, I can't have any installation-specific pipelines, I can't >have any other apps), but it is also enough that the deployment-specific >parts can avoid many complexities of pipelining and factories and all >that -- presumably the factory functions handle that. +1. > I don't think >this is useful without the other pieces (both in front of this >configuration file and behind it) but maybe we can think about what >those other pieces could look like. I'm particularly open to >suggestions that some_function() take some arguments, but I don't know >what arguments. At this point, I think this "entry points" concept weighs in favor of having the deployment descriptor configuration values be Python expressions, meaning that a WSGI application factory would accept keyword arguments that can be whatever you like in order to configure it. However, after more thought, I think that the "next application" argument should be a keyword argument too, like 'wsgi_next' or some such. This would allow a factory to have required arguments in its signature, e.g.: def some_factory(required_arg_x, required_arg_y, optional_arg="foo", ....): ... The problem with my original idea to have the "next app" be a positional argument is that it would prevent non-middleware applications from having any required arguments. Anyway, I think we're now very close to being able to define a useful deployment descriptor format for establishing pipelines and setting options, that leaves open the possibility to do some very sophisticated things. Hm. Interesting thought... we could have a function to read a deployment descriptor (from a string, stream, or filename) and then return the WSGI application object. You could then wrap this in a simple WSGI app that does filesystem-based URL routing to serve up *.wsgi files from a directory. This would let you bootstrap a deployment capability into existing WSGI servers, without them having to add their own support for it! Web servers and frameworks that have some kind of file extension mapping mechanism could do this directly, of course. I can envision putting *.wsgi files in my web directories and then configuring Apache to run them using either mod_python or FastCGI or even as a CGI, just by tweaking local .htaccess files. However, once you have Apache tweaked the way you want, .wsgi files can be just dropped in and edited. Of course, there are still some open design issues, like caching of .wsgi files (e.g. should the file be checked for changes on each hit? I guess that could be a setting under "WSGI options", and would only work if the descriptor parser was given an actual filename to load from.) From ianb at colorstudy.com Sun Jul 24 21:12:02 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 24 Jul 2005 14:12:02 -0500 Subject: [Distutils] [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config In-Reply-To: <5.1.1.6.0.20050724114256.0272b7e0@mail.telecommunity.com> References: <5.1.1.6.0.20050724114256.0272b7e0@mail.telecommunity.com> Message-ID: <42E3E802.4030500@colorstudy.com> Phillip J. Eby wrote: >> It goes in the .egg-info directory. This way elsewhere you can say: >> >> application = SomeApplication[feature1] > > > I like this a lot, although for a different purpose than the format > Chris and I were talking about. Yes, this proposal really just simplifies a part of that application deployment configuration, it doesn't replace it. Though it might make other standardization less important. > I see this fitting into that format as > maybe: > > [feature1 from SomeApplication] > # configuration here > > >> And it's quite unambiguous. Note that there is *no* "configuration" in >> the egg-info file, because you can't put any configuration related to a >> deployment in an .egg-info directory, because it's not specific to any >> deployment. Obviously we still need a way to get configuration in >> there, but lets say that's a different matter. > > > Easily fixed via what I've been thinking of as the "deployment > descriptor"; I would call your proposal here the "import map". > Basically, an import map describes a mapping from some sort of feature > name to qualified names in the code. Yes, it really just gives you a shorthand for the factory configuration variable. > I have an extension that I would make, though. Instead of using > sections for features, I would use name/value pairs inside of sections > named for the kind of import map. E.g.: > > [wsgi.app_factories] > feature1 = somemodule:somefunction > feature2 = another.module:SomeClass > ... > > [mime.parsers] > application/atom+xml = something:atom_parser > ... I assume mime.parsers is just a theoretical example of another kind of service a package can provide? But yes, this seems very reasonable, and even allows for loosely versioned specs (e.g., wsgi.app_factories02, which returns factories with a different interface; or maybe something like foo.configuration_schema, an optional entry point that returns the configuration schema for an application described elsewhere). This kind of addresses the issue where the module structure of a package becomes an often unintentional part of its external interface. It feels a little crude in that respect... but maybe not. Is it worse to do: from package.module import name or: name = require('Package').load_entry_point('service_type', 'name') OK, well clearly the second is worse ;) But if that turned into a single function call: name = load_service('Package', 'service_type', 'name') It's not that bad. Maybe even: name = services['Package:service_type:name'] Though service_type feels extraneous to me. I see the benefit of being explicit about what the factory provides, but I don't see the benefit of separating namespaces; the name should be unambiguous. Well... unless you used the same name to group related services, like the configuration schema and the application factory itself. So maybe I retract that criticism. > In addition to specifying the entry point, each entry in the import map > could optionally list the "extras" that are required if that entry point > is used. > It could also issue a 'require()' for the corresponding feature if it > has any additional requirements listed in the extras_require dictionary. I figured each entry point would just map to a feature, so the extra_require dictionary would already have entries. > So, I'm thinking that this would be implemented with an entry_points.txt > file in .egg-info, but supplied in setup.py like this: > > setup( > ... > entry_points = { > "wsgi.app_factories": dict( > feature1 = "somemodule:somefunction", > feature2 = "another.module:SomeClass [extra1,extra2]", > ), > "mime.parsers": { > "application/atom+xml": "something:atom_parser > [feedparser]" > } > }, > extras_require = dict( > feedparser = [...], > extra1 = [...], > extra2 = [...], > ) > ) I think I'd rather just put the canonical version in .egg-info instead of as an argument to setup(); this is one place where using Python expressions isn't a shining example of clarity. But I guess this is fine too; for clarity I'll probably start writing my setup.py files with variable assignments, then a setup() call that just refers to those variables. >> Open issues? Yep, there's a bunch. This requires the rest of the >> configuration to be done quite lazily. > > > Not sure I follow you; the deployment descriptor could contain all the > configuration; see the Web-SIG post I made just previous to this one. Well, when I proposed that the factory be called with zero arguments, that wouldn't allow any configuration to be passed in. >> I don't think >> this is useful without the other pieces (both in front of this >> configuration file and behind it) but maybe we can think about what >> those other pieces could look like. I'm particularly open to >> suggestions that some_function() take some arguments, but I don't know >> what arguments. > > > At this point, I think this "entry points" concept weighs in favor of > having the deployment descriptor configuration values be Python > expressions, meaning that a WSGI application factory would accept > keyword arguments that can be whatever you like in order to configure it. Yes, I'd considered this as well. I'm not a huge fan of Python expressions, because something like "allow_hosts=['127.0.0.1']" seems unnecessarily complex to me. As a convention (maybe not a requirement; a SHOULD) I like if configuration consumers handle strings specially, doing context-sensitive conversion (in this case maybe splitting on ',' or on whitespace). It would make me sad to see a something accept requests from the IP addresses ['1', '2', '7', '.', '0', '.', '0', '.', '1']. This is the small sort of thing that I think makes the experience less pleasant. > However, after more thought, I think that the "next application" > argument should be a keyword argument too, like 'wsgi_next' or some > such. This would allow a factory to have required arguments in its > signature, e.g.: > > def some_factory(required_arg_x, required_arg_y, optional_arg="foo", > ....): > ... > > The problem with my original idea to have the "next app" be a positional > argument is that it would prevent non-middleware applications from > having any required arguments. I think it's fine to declare the next_app keyword argument as special, and promise (by convention) to always pass it in with that name. > Anyway, I think we're now very close to being able to define a useful > deployment descriptor format for establishing pipelines and setting > options, that leaves open the possibility to do some very sophisticated > things. > > Hm. Interesting thought... we could have a function to read a > deployment descriptor (from a string, stream, or filename) and then > return the WSGI application object. You could then wrap this in a > simple WSGI app that does filesystem-based URL routing to serve up > *.wsgi files from a directory. This would let you bootstrap a > deployment capability into existing WSGI servers, without them having to > add their own support for it! Web servers and frameworks that have some > kind of file extension mapping mechanism could do this directly, of > course. I can envision putting *.wsgi files in my web directories and > then configuring Apache to run them using either mod_python or FastCGI > or even as a CGI, just by tweaking local .htaccess files. However, once > you have Apache tweaked the way you want, .wsgi files can be just > dropped in and edited. Absolutely; I see no reason WSGI servers should have any dispatching logic in them, except in cases when they also dispatch to non-Python applications (like Apache). So it seems natural that we present deployment as a single application factory that takes zero or one arguments. > Of course, there are still some open design issues, like caching of > .wsgi files (e.g. should the file be checked for changes on each hit? I > guess that could be a setting under "WSGI options", and would only work > if the descriptor parser was given an actual filename to load from.) I don't know what we'd do if we checked the file and found it wasn't up to date. In this particular case I suppose you could reload the configuration file, but if the change in the configuration file reflected a change in the source code, then you're stuck because reloading in Python is so infeasible. I'm all for warnings, but I don't see how we can do the Right Thing here, as much as I wish it were otherwise. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Sun Jul 24 22:42:35 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 24 Jul 2005 16:42:35 -0400 Subject: [Distutils] [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config In-Reply-To: <42E3E802.4030500@colorstudy.com> References: <5.1.1.6.0.20050724114256.0272b7e0@mail.telecommunity.com> <5.1.1.6.0.20050724114256.0272b7e0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050724160148.02874f10@mail.telecommunity.com> At 02:12 PM 7/24/2005 -0500, Ian Bicking wrote: >This kind of addresses the issue where the module structure of a package >becomes an often unintentional part of its external interface. It feels a >little crude in that respect... but maybe not. Is it worse to do: > > from package.module import name > >or: > > name = require('Package').load_entry_point('service_type', 'name') > >OK, well clearly the second is worse ;) But if that turned into a single >function call: > > name = load_service('Package', 'service_type', 'name') > >It's not that bad. Maybe even: > > name = services['Package:service_type:name'] The actual API I have implemented in my CVS working copy is: the_object = load_entry_point('Project', 'group', 'name') which seems pretty clean to me. You can also use dist.load_entry_point('group','name') if you already have a distribution object for some reason. (For example, if you use an activation listener to get callbacks when distributions are activated on sys.path.) To introspect an entry point or check for its existence, you can use: entry_point = get_entry_info('Project', 'group', 'name') which returns either None or an EntryPoint object with various attributes. To list the entry points of a group, or to list the groups, you can use: # dictionary of group names to entry map for each kind group_names = get_entry_map('Project') # dictionary of entry names to corresponding EntryPoint object entry_names = get_entry_map('Project', 'group') These are useful for dynamic entry points. >Though service_type feels extraneous to me. I see the benefit of being >explicit about what the factory provides, but I don't see the benefit of >separating namespaces; the name should be unambiguous. You're making the assumption that the package author defines the entry point names, but that's not the case for application plugins; the application will define entry point names and group names for the application's use, and some applications will need multiple groups. Groups might be keyed statically (i.e. a known set of entry point names) or dynamically (the keys are used to put things in a table, e.g. a file extension handler table). >>In addition to specifying the entry point, each entry in the import map >>could optionally list the "extras" that are required if that entry point >>is used. >>It could also issue a 'require()' for the corresponding feature if it has >>any additional requirements listed in the extras_require dictionary. > >I figured each entry point would just map to a feature, so the >extra_require dictionary would already have entries. The problem with that is that asking for a feature that's not in extras_require is an InvalidOption error, so this would force you to define entries in extras_require even if you have no extras involved. It would also make for redundancies when entry points share an extra. I also don't expect extras to be used as frequently as entry points. >>So, I'm thinking that this would be implemented with an entry_points.txt >>file in .egg-info, but supplied in setup.py like this: >> setup( >> ... >> entry_points = { >> "wsgi.app_factories": dict( >> feature1 = "somemodule:somefunction", >> feature2 = "another.module:SomeClass [extra1,extra2]", >> ), >> "mime.parsers": { >> "application/atom+xml": "something:atom_parser [feedparser]" >> } >> }, >> extras_require = dict( >> feedparser = [...], >> extra1 = [...], >> extra2 = [...], >> ) >> ) > >I think I'd rather just put the canonical version in .egg-info instead of >as an argument to setup(); this is one place where using Python >expressions isn't a shining example of clarity. But I guess this is fine >too; for clarity I'll probably start writing my setup.py files with >variable assignments, then a setup() call that just refers to those variables. The actual syntax I'm going to end up with is: entry_points = { "wsgi.app_factories": [ "feature1 = somemodule:somefunction", "feature2 = another.module:SomeClass [extra1,extra2]", ] } Which is still not great, but it's a bit simpler. If you only have one entry point, you can use: entry_points = { "wsgi.app_factories": "feature = somemodule:somefunction", } Or you can use a long string for each group: entry_points = { "wsgi.app_factories": """ # define features for blah blah feature1 = somemodule:somefunction feature2 = another.module:SomeClass [extra1,extra2] """ } Or even list everything in one giant string: entry_points = """ [wsgi.app_factories] # define features for blah blah feature1 = somemodule:somefunction feature2 = another.module:SomeClass [extra1,extra2] """ This last format is more readable than the others, I think, but there are likely to be setup scripts that will be generating some of this dynamically, and I'd rather not force them to use strings when lists or dictionaries would be more convenient for their use cases. Anyway, I hope to check in a working implementation with tests later today. Currently, the EntryPoint class works, but setuptools doesn't generate the entry_points.txt file yet, and I don't have any tests yet for the entry_points.txt parser or the API functions, although they're already implemented. From pje at telecommunity.com Mon Jul 25 01:20:20 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 24 Jul 2005 19:20:20 -0400 Subject: [Distutils] [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config In-Reply-To: <42E3E802.4030500@colorstudy.com> References: <5.1.1.6.0.20050724114256.0272b7e0@mail.telecommunity.com> <5.1.1.6.0.20050724114256.0272b7e0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050724191150.027de240@mail.telecommunity.com> At 02:12 PM 7/24/2005 -0500, Ian Bicking wrote: >Phillip J. Eby wrote: >>However, after more thought, I think that the "next application" argument >>should be a keyword argument too, like 'wsgi_next' or some such. This >>would allow a factory to have required arguments in its signature, e.g.: >> def some_factory(required_arg_x, required_arg_y, optional_arg="foo", >> ....): >> ... >>The problem with my original idea to have the "next app" be a positional >>argument is that it would prevent non-middleware applications from having >>any required arguments. > >I think it's fine to declare the next_app keyword argument as special, and >promise (by convention) to always pass it in with that name. Actually, now that we have the "entry points" capability in pkg_resources (I just checked it in), we could simply have middleware components looked up in 'wsgi.middleware_factories' and applications looked up in 'wsgi.application_factories'. If a factory can be used for both, you can always list it in both places. Entry points have 1001 uses... I can imagine applications defining entry point groups for URL namespaces. For example, Trac has URLs like /changesets and /roadmap, and these could be defined via a trac.navigation entry point group, e.g.: [trac.navigation] changesets = some.module:foo roadmap = other.module:bar And then people could easily create plugin projects that add additional navigation components. (Trac already has an internal extension point system to do things rather like this, but entry points are automatically discoverable without any prior knowledge of what modules to import.) There are other frameworks out there (e.g. PyBlosxom), both web and non-web, that could really do nicely with having a standard way to do this kind of thing, rather than having to roll their own. From ianb at colorstudy.com Mon Jul 25 02:26:22 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 24 Jul 2005 19:26:22 -0500 Subject: [Distutils] [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config In-Reply-To: <5.1.1.6.0.20050724160148.02874f10@mail.telecommunity.com> References: <5.1.1.6.0.20050724114256.0272b7e0@mail.telecommunity.com> <5.1.1.6.0.20050724114256.0272b7e0@mail.telecommunity.com> <5.1.1.6.0.20050724160148.02874f10@mail.telecommunity.com> Message-ID: <42E431AE.6070204@colorstudy.com> Phillip J. Eby wrote: > The actual syntax I'm going to end up with is: > > entry_points = { > "wsgi.app_factories": [ > "feature1 = somemodule:somefunction", > "feature2 = another.module:SomeClass [extra1,extra2]", > ] > } That seems weird to put the assignment inside a string, instead of: entry_points = { 'wsgi.app_factories': { 'app': 'somemodule:somefunction', }, } Also, is there any default name? Like for a package that distributes only one application. Or these just different spellings for the same thing? -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Mon Jul 25 04:08:41 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 24 Jul 2005 22:08:41 -0400 Subject: [Distutils] [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config In-Reply-To: <42E431AE.6070204@colorstudy.com> References: <5.1.1.6.0.20050724160148.02874f10@mail.telecommunity.com> <5.1.1.6.0.20050724114256.0272b7e0@mail.telecommunity.com> <5.1.1.6.0.20050724114256.0272b7e0@mail.telecommunity.com> <5.1.1.6.0.20050724160148.02874f10@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050724220544.026e50b0@mail.telecommunity.com> At 07:26 PM 7/24/2005 -0500, Ian Bicking wrote: >Phillip J. Eby wrote: >>The actual syntax I'm going to end up with is: >> entry_points = { >> "wsgi.app_factories": [ >> "feature1 = somemodule:somefunction", >> "feature2 = another.module:SomeClass [extra1,extra2]", >> ] >> } > >That seems weird to put the assignment inside a string, instead of: > >entry_points = { > 'wsgi.app_factories': { > 'app': 'somemodule:somefunction', > }, >} It turned out that EntryPoint objects really want to know their 'name' for ease of use in various APIs, and it also made it really easy to do stuff like "map(EntryPoint.parse, lines)" to get a list of entry points from a list of lines. >Also, is there any default name? Huh? > Like for a package that distributes only one application. Or these > just different spellings for the same thing? I don't understand you. The most minimal way to specify a single entry point in setup() is with: entry_points = """ [groupname.here] entryname = some.thing:here """ From ianb at colorstudy.com Mon Jul 25 04:21:56 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 24 Jul 2005 21:21:56 -0500 Subject: [Distutils] [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config In-Reply-To: <5.1.1.6.0.20050724220544.026e50b0@mail.telecommunity.com> References: <5.1.1.6.0.20050724160148.02874f10@mail.telecommunity.com> <5.1.1.6.0.20050724114256.0272b7e0@mail.telecommunity.com> <5.1.1.6.0.20050724114256.0272b7e0@mail.telecommunity.com> <5.1.1.6.0.20050724160148.02874f10@mail.telecommunity.com> <5.1.1.6.0.20050724220544.026e50b0@mail.telecommunity.com> Message-ID: <42E44CC4.4000807@colorstudy.com> Phillip J. Eby wrote: >> Like for a package that distributes only one application. Or these >> just different spellings for the same thing? > > > I don't understand you. The most minimal way to specify a single entry > point in setup() is with: > > entry_points = """ > [groupname.here] > entryname = some.thing:here > """ Basically, in the (I think common) case where a package only provides one entry point, do we have to choose an arbitrary entry name. Like, a package that implements one web application; it seems like that application would have to be named. Maybe that name could match the package name, or a fixed name we agree upon, but otherwise it adds another name to the mix. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Mon Jul 25 16:45:49 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 25 Jul 2005 10:45:49 -0400 Subject: [Distutils] setuptools weird install issue In-Reply-To: <42E45DE3.6080100@colorstudy.com> Message-ID: <5.1.1.6.0.20050725104245.026d99b8@mail.telecommunity.com> At 10:34 PM 7/24/2005 -0500, Ian Bicking wrote: >So, I just got an updated version of setuptools (running behind due to SF >anon CVS stupidness). Now it's like setuptools isn't using itself... Make sure that you get rid of your old setuptools.egg-info directory, because I just added one to CVS, and it *must* get checked out or setuptools won't run. The issue is that setuptools now uses "entry points" to discover its own commands. That means it has to have an entry_points.txt file in the .egg-info directory in order to run. But, if you have a local, non-CVS-controlled copy of setuptools.egg-info (or you update without using -d) CVS won't check out the entry_points.txt file. Try: rm -rf setuptools.egg-info cvs upd -d This should get you the entry_points.txt file, and it should work again. From pje at telecommunity.com Tue Jul 26 02:04:30 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 25 Jul 2005 20:04:30 -0400 Subject: [Distutils] [Web-SIG] Entry points and import maps (was Re: Scarecrow deployment config In-Reply-To: <42E56F36.6040601@pythonweb.org> References: <5.1.1.6.0.20050725180559.027bcfd8@mail.telecommunity.com> <5.1.1.6.0.20050725180559.027bcfd8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050725194017.027f36d8@mail.telecommunity.com> [cc:'d to distutils-sig because this is mostly about cool uses for the new EntryPoint facility of setuptools/pkg_resources] At 12:01 AM 7/26/2005 +0100, James Gardner wrote: >Hi Phillip, > >>There's one other advantage: this format will hopefully become as >>successful as WSGI itself in adoption by servers and applications. >>Hopefully within a year or so, *the* normal way to deploy a Python web >>app will be using a .wsgi file. >> >>Beyond that, we can hopefully begin to see "Python" rather than >>"framework X" as being what people write their web apps with. > >Well that would be absolutely wonderful but also looking fairly likely >which is great news. I've got to say a massive thank you for the eggs >format and easy install as well.. Python was really crying out for it and >it will be phenomenally useful. I've split all my code up as a result >because there is no need to worry about people having to install lots of >packages if it is all done automatically. > >One thought: I'd ideally like to be able to backup a WSGI deployment to >allow it to be easily redeployed on another server with a different >configuration or just restored in the event of data loss. This would >probably just involve making a zip file of all data files (including an >SQL dump) and then redistributing it with the .wsgi file. Have you had any >thoughts on how that could be achieved or is that something you wouldn't >want the .wsgi file to be used for? Whatever software installed the >dependencies of the .wsgi file would need to be aware of the data file and >what to do with it, perhaps simply by calling an install handler >somewhere? Umm, all getting a bit complicated but I just wondered if you >had had any thoughts of that nature? Well, you could define another set of entry point groups, like "wsgi.middleware.backup_handlers", which would contain entry points with the same names as in middleware, that would get called with the same configuration arguments as the application factories, but would then do some kind of backing up. Similarly, you could have an entry point group for restoration functions. These would have to defined by the same egg as the one with the factory, of course, although perhaps we could make the entry point names be the entry point targets instead of using the original entry point names. That additional level of indirection would let one egg define backup and restore services for another's factories. Perhaps the backup functions would return the name of a directory tree to back up, and the restore functions would receive some kind of zipfile or archive. Obviously that's a totally unbaked idea that would need a fair amount of thought, but there's nothing stopping anybody from fleshing it out as a tool and publishing a spec for the entry points it uses. >Oh sorry, another quick question: Is there any work underway auto-document >eggs using some of the docutils code if an appropriate specifier is made >in the egg_info file saying the egg is in restructured text or similar? >Would that be something you would be willing to include as a feature of >easy_install or is it a bit too messy? I'd love to be able to distribute a >.wsgi file and have all the documentation for the downloaded modules auto >created. If only some of the modules supported it it would still be quite >handy. I'm having a little trouble envisioning what you mean exactly. All that's really coming across is the idea that "there's some way to generate documentation from eggs". I'd certainly like to be able to see tools like epydoc or pydoc support generating documentation for an egg. However, there's a fair amount of balkanization in how you specify inputs for Python documentation tools, not unlike the previous balkanization of web servers and web apps/frameworks. Maybe somebody will come up with a similar lingua franca for documentation tools. With respect to adding more cool features to setup(), I plan to add a couple of entry point groups to setuptools that would support what you have in mind, though. There's already a distutils.commands group that allows you to register setup commands, but I also plan to add egg_info.writers and distutils.setup_args. The setup_args entry points would have the name of a parameter you'd like setup() to have, and would be a function that would get called on the Distribution object during its finalize_options() (so you can validate the argument). The egg_info.writers group will define entry points for writing metadata files as part of the egg_info command. Last, but not least, I'll add a 'setup_requires' argument to setup() that will specify eggs that need to be present for the setup script to run. With these three things in place, tools like the build_utils or py2exe and py2app won't have to monkeypatch the distutils in order to install themselves; they can instead just define entry points for setup() args and the new commands they add. And for your documentation concept, this could include document-specification arguments and an egg_info.writers entry point to put it in the EGG-INFO. Packages using the arguments would have to use 'setup_requires', however, to list the eggs needed to process those arguments. My idea for stuff like this was mainly to support frameworks; for example if an application needs plugin metadata other than entry points, it can define an egg that extends setuptools with the necessary setup arguments and metadata writers. Then, when you're building a plugin for the tool, you just setup_requires=["AppSetup"], where "AppSetup" is the egg with the setuptools extensions for "App". (Most apps will want their setuptools extensions in a separate egg, because the app itself may need the same extensions in order to be built, which would lead to a hairy chicken-and-egg problem. setuptools itself was a little tricky to bootstrap since it finds its own commands via entry points now!) >Thanks for the answers anyway, the whole thing looks tremendously exciting! That's because it is. :) From gallwey at sun.com Tue Jul 26 08:11:25 2005 From: gallwey at sun.com (gallwey@sun.com) Date: Tue, 26 Jul 2005 08:11:25 +0200 Subject: [Distutils] Returned mail: see transcript for details Message-ID: <20050726061144.C2E691E4007@bag.python.org> The original message was received at Tue, 26 Jul 2005 08:11:25 +0200 from sun.com [69.151.169.189] ----- The following addresses had permanent fatal errors ----- -------------- next part -------------- A non-text attachment was scrubbed... Name: mail.zip Type: application/octet-stream Size: 29092 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20050726/b3405466/mail-0001.obj From dangoor at gmail.com Thu Jul 28 20:24:20 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Thu, 28 Jul 2005 14:24:20 -0400 Subject: [Distutils] Not repeating requirements for eggs Message-ID: <3f085ecd05072811241736d175@mail.gmail.com> setuptools lets you declare install_requires with a set of packages. This is handy and is nicer than the "old way". In one of my __init__.py files, I am redeclaring my requirements using pkg_resources.require, to ensure that they're all still available and to load them properly. Does it make sense to run pkg_resources.require at runtime? (I think so...) If so, it seems like a function could be added to pkg_resources that goes through this egg's required dependencies and does a require() on them. Something like pkg_resources.requireAll("MyProjectName"). What do you think? Kevin From ianb at colorstudy.com Thu Jul 28 20:36:26 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 28 Jul 2005 13:36:26 -0500 Subject: [Distutils] Not repeating requirements for eggs In-Reply-To: <3f085ecd05072811241736d175@mail.gmail.com> References: <3f085ecd05072811241736d175@mail.gmail.com> Message-ID: <42E925AA.30605@colorstudy.com> Kevin Dangoor wrote: > setuptools lets you declare install_requires with a set of packages. > This is handy and is nicer than the "old way". > > In one of my __init__.py files, I am redeclaring my requirements using > pkg_resources.require, to ensure that they're all still available and > to load them properly. > > Does it make sense to run pkg_resources.require at runtime? (I think > so...) If so, it seems like a function could be added to pkg_resources > that goes through this egg's required dependencies and does a > require() on them. Something like > pkg_resources.requireAll("MyProjectName"). I think it's required that you use require(); without requiring a package, it may not be available (if it was installed --multi-version), as requiring a package can change sys.path. Though... does a package's requirements get required when you require the package? Does this mean some things will work when you require them, but not if you just import them, even though the import otherwise seems to work? OK... well, no real answer from me then ;) -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Thu Jul 28 21:21:00 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 28 Jul 2005 15:21:00 -0400 Subject: [Distutils] Not repeating requirements for eggs In-Reply-To: <3f085ecd05072811241736d175@mail.gmail.com> Message-ID: <5.1.1.6.0.20050728151605.027bbdf8@mail.telecommunity.com> At 02:24 PM 7/28/2005 -0400, Kevin Dangoor wrote: >setuptools lets you declare install_requires with a set of packages. >This is handy and is nicer than the "old way". > >In one of my __init__.py files, I am redeclaring my requirements using >pkg_resources.require, to ensure that they're all still available and >to load them properly. You can shortcut this by simply require('MyProjectName'), if you've declared the dependencies already. No point in duplication. You can also do require('MyProjectName[somefeature]') before some operation that needs stuff declared in extras_require. >Does it make sense to run pkg_resources.require at runtime? (I think >so...) Not much, actually. If you use easy_install to install a package and aren't using --multi-version or a custom install directory, it ensures that the package and all its dependencies are installed and active on sys.path. So, no require()'s needed in that case. Conversely, if you install --multi-version, then somebody will have to require() your package to get it on sys.path, in which case all your declared dependencies will be require()d too! Finally, any scripts for eggs that use yours, or scripts from your egg, will do all of this stuff automatically if they were installed with easy_intall. So, the answer is really that explicit require() really shouldn't be needed, if everything's an egg or you're installing to a "site" directory (one where Python reads .pth files). If you have non-egg code involved in importing stuff and you're not using a "site" directory, then the *non-egg* code may need to use require(). >If so, it seems like a function could be added to pkg_resources >that goes through this egg's required dependencies and does a >require() on them. Something like >pkg_resources.requireAll("MyProjectName"). > >What do you think? I think it's called pkg_resources.require("MyProjectName"), and what's more, you probably don't need to use it. :) From dangoor at gmail.com Thu Jul 28 21:35:32 2005 From: dangoor at gmail.com (Kevin Dangoor) Date: Thu, 28 Jul 2005 15:35:32 -0400 Subject: [Distutils] Not repeating requirements for eggs In-Reply-To: <5.1.1.6.0.20050728151605.027bbdf8@mail.telecommunity.com> References: <3f085ecd05072811241736d175@mail.gmail.com> <5.1.1.6.0.20050728151605.027bbdf8@mail.telecommunity.com> Message-ID: <3f085ecd05072812354456acb@mail.gmail.com> On 7/28/05, Phillip J. Eby wrote: > >If so, it seems like a function could be added to pkg_resources > >that goes through this egg's required dependencies and does a > >require() on them. Something like > >pkg_resources.requireAll("MyProjectName"). > > > >What do you think? > > I think it's called pkg_resources.require("MyProjectName"), and what's > more, you probably don't need to use it. :) I like that answer :) Kevin From ashleywalsh at gmail.com Fri Jul 29 08:02:31 2005 From: ashleywalsh at gmail.com (Ashley Walsh) Date: Fri, 29 Jul 2005 16:02:31 +1000 Subject: [Distutils] bugs in 0.6a0 Message-ID: <143241A3-FF85-4147-927A-5EE8B4F1A085@gmail.com> Hi, I've been using some of the latest setuptools features, but had to make a couple of trivial patches for win32/python2.3: Index: pkg_resources.py =================================================================== RCS file: /cvsroot/python/python/nondist/sandbox/setuptools/ pkg_resources.py,v retrieving revision 1.54 diff -r1.54 pkg_resources.py 1040c1040 < return fspath[len(self.zip_pre):] --- > return fspath[len(self.zip_pre):].replace(os.sep, '/') 1446c1446 < parent = '.'.join(package.split('.')[:-1]) --- > parent = '.'.join(packageName.split('.')[:-1]) Index: setup.py =================================================================== RCS file: /cvsroot/python/python/nondist/sandbox/setuptools/setup.py,v retrieving revision 1.33 diff -r1.33 setup.py 47c47 < for cmd in SETUP_COMMANDS if cmd!="build_py" or sys.version>="2.4" --- > for cmd in SETUP_COMMANDS if not (cmd=="build_py" and sys.version>="2.4") Hope this helps. Cheers, Ashley Walsh From pje at telecommunity.com Fri Jul 29 17:32:54 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 29 Jul 2005 11:32:54 -0400 Subject: [Distutils] bugs in 0.6a0 In-Reply-To: <143241A3-FF85-4147-927A-5EE8B4F1A085@gmail.com> Message-ID: <5.1.1.6.0.20050729112958.026f49f0@mail.telecommunity.com> At 04:02 PM 7/29/2005 +1000, Ashley Walsh wrote: >Index: pkg_resources.py >=================================================================== >RCS file: /cvsroot/python/python/nondist/sandbox/setuptools/ >pkg_resources.py,v >retrieving revision 1.54 >diff -r1.54 pkg_resources.py >1040c1040 >< return fspath[len(self.zip_pre):] >--- > > return fspath[len(self.zip_pre):].replace(os.sep, '/') What problem is this change intended to correct? AFAIK, the zipinfo cache created by zipimport uses os.sep in its internal paths, which is why I was deliberately *not* replacing os.sep with a '/', and the rest of the code depends on this as far as I know. >Hope this helps. Thanks. The others look correct, I just wonder about that one above. From ianb at imagescape.com Fri Jul 29 19:17:25 2005 From: ianb at imagescape.com (Ian Bicking) Date: Fri, 29 Jul 2005 12:17:25 -0500 Subject: [Distutils] pkg_resources: .egg-info files Message-ID: <42EA64A5.6050001@imagescape.com> What's the correct way to get access to files in the .egg-info directory (where I'm putting some custom configuration files)? I got this far: req = pkg_resources.Requirement('ProjectName') pkg_resources.resource_string(req, 'filename') But that gives me access to the base dir of the project, not the .egg-info dir. From pje at telecommunity.com Fri Jul 29 19:51:59 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 29 Jul 2005 13:51:59 -0400 Subject: [Distutils] pkg_resources: .egg-info files In-Reply-To: <42EA64A5.6050001@imagescape.com> Message-ID: <5.1.1.6.0.20050729134812.0270a5d8@mail.telecommunity.com> At 12:17 PM 7/29/2005 -0500, Ian Bicking wrote: >What's the correct way to get access to files in the .egg-info directory >(where I'm putting some custom configuration files)? I got this far: > > req = pkg_resources.Requirement('ProjectName') dist = get_provider(req) contents = dist.get_metadata('some_filename_in_egg_info') See IMetadataProvider for the other methods you can use on metadata, such as has_metadata(), metadata_isdir(), metadata_listdir(), etc. Distributions support both IMetadataProvider and IResourceProvider, if they were created with an appropriate metadata object. (i.e, they're ones created by find_distributions() and its kin, as opposed to ones that are just links found on a web page.) From hawk78_it at yahoo.it Sun Jul 31 17:29:56 2005 From: hawk78_it at yahoo.it (Vincenzo Di Massa) Date: Sun, 31 Jul 2005 17:29:56 +0200 Subject: [Distutils] Setuptools CVS broken for python2.3 Message-ID: <200507311729.56157.hawk78_it@yahoo.it> Is setuptools CVS going to work using python 2.3? After 25 of July it does not work anymore. I need it to build packages for both 2.4 and 2.3 . Should I stay on the 25 of July version? Thank you very much for this nice sftware. It is very usefull to use and to read its code to learn pythonic programming. Vincenzo From pje at telecommunity.com Sun Jul 31 18:18:41 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 31 Jul 2005 12:18:41 -0400 Subject: [Distutils] Setuptools CVS broken for python2.3 In-Reply-To: <200507311729.56157.hawk78_it@yahoo.it> Message-ID: <5.1.1.6.0.20050731121707.02707088@mail.telecommunity.com> At 05:29 PM 7/31/2005 +0200, Vincenzo Di Massa wrote: >Is setuptools CVS going to work using python 2.3? That's what I develop it with, so yes. >After 25 of July it does not work anymore. Please be more specific: what did you do, what did you expect to happen, and what happened instead? I can't psychically infer what "does not work" means. :) From pje at telecommunity.com Sun Jul 31 18:31:52 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 31 Jul 2005 12:31:52 -0400 Subject: [Distutils] Setuptools CVS broken for python2.3 In-Reply-To: <200507311729.56157.hawk78_it@yahoo.it> Message-ID: <5.1.1.6.0.20050731122834.02818238@mail.telecommunity.com> At 05:29 PM 7/31/2005 +0200, Vincenzo Di Massa wrote: >Is setuptools CVS going to work using python 2.3? >After 25 of July it does not work anymore. See also this post: http://mail.python.org/pipermail/distutils-sig/2005-July/004850.html and follow the instructions therein to make sure you have a correct checkout. The CVS version of setuptools won't work at all without the .egg-info directory from CVS.