From safesticks at yahoo.com Sat Apr 1 04:18:51 2006 From: safesticks at yahoo.com (none nonoe) Date: Fri, 31 Mar 2006 18:18:51 -0800 (PST) Subject: [Distutils] copy_tree? Message-ID: <20060401021851.21887.qmail@web35908.mail.mud.yahoo.com> After running this (break.py):: import os, shutil, distutils.dir_util os.mkdir("./a") os.popen("touch ./a/testfile") distutils.dir_util.copy_tree("./a", "./b") shutil.rmtree("./b") distutils.dir_util.copy_tree("./a", "./b") I get this error:: Traceback (most recent call last): File "break.py", line 8, in ? distutils.dir_util.copy_tree("./a", "./b") File "C:\Python24\lib\distutils\dir_util.py", line 175, in copy_tree preserve_times, update, dry_run=dry_run) File "C:\Python24\lib\distutils\file_util.py", line 165, in copy_file _copy_file_contents(src, dst) File "C:\Python24\lib\distutils\file_util.py", line 47, in _copy_file_contents fdst = open(dst, 'wb') IOError: [Errno 2] No such file or directory: './b\\testfile' Am I doing something wrong? S --------------------------------- Talk is cheap. Use Yahoo! Messenger to make PC-to-Phone calls. Great rates starting at 1¢/min. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20060331/d3a6a147/attachment.htm From cwmoad at gmail.com Mon Apr 3 17:24:52 2006 From: cwmoad at gmail.com (Charlie Moad) Date: Mon, 3 Apr 2006 11:24:52 -0400 Subject: [Distutils] Upgrading dependencies Message-ID: <6382066a0604030824g6795b324vffd2d4d33e0bd271@mail.gmail.com> Is it currently possible to upgrade dependencies as well when upgrading a packages? If not, this would be a really nice feature to add to easy_install. Maybe a call like: easy_install --upgrade --upgrade-deps Package Obviously it makes sense to leave this off by default. Thanks, Charlie From jjl at pobox.com Mon Apr 3 19:21:44 2006 From: jjl at pobox.com (John J Lee) Date: Mon, 3 Apr 2006 18:21:44 +0100 (GMT Standard Time) Subject: [Distutils] Upgrading dependencies In-Reply-To: <6382066a0604030824g6795b324vffd2d4d33e0bd271@mail.gmail.com> References: <6382066a0604030824g6795b324vffd2d4d33e0bd271@mail.gmail.com> Message-ID: On Mon, 3 Apr 2006, Charlie Moad wrote: > Is it currently possible to upgrade dependencies as well when > upgrading a packages? If not, this would be a really nice feature to > add to easy_install. Maybe a call like: [...] I believe this is something Phillip hopes to do in 0.7, by adding support for a "nest" command :-) Searching in the gmane archives for "nest command" might tell you what you want... John From schwehr at gmail.com Wed Apr 5 01:30:46 2006 From: schwehr at gmail.com (Kurt Schwehr) Date: Tue, 4 Apr 2006 19:30:46 -0400 Subject: [Distutils] setuptools in fink Message-ID: <37de72610604041630t70d5c955x8f95fdfb6b1de122@mail.gmail.com> Hi All, For Mac OSX 10.4 folks... Now that the sf.net cvs server is back alive, I have added setuptools and sqlalchemy to the 10.4-transitional/unstable tree. Any reports of success, failure, or suggestions would be greatly appreciated! Once this has been out there for a few days, I will try package a couple other things that need setuptools. BTW, I am holding off on pysqlite2 in fink since I think that will break existing trac installs. Any suggestions on handling that? fink selfupdate fink install setuptools-py24 fink install sqlalchemy-py24 Thanks, -kurt -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20060404/34bf59b0/attachment.htm From schwehr at gmail.com Thu Apr 6 15:15:39 2006 From: schwehr at gmail.com (Kurt Schwehr) Date: Thu, 6 Apr 2006 09:15:39 -0400 Subject: [Distutils] man page generation and other documentation Message-ID: <37de72610604060615h5cc81a8chf982426a4a4dfcbc@mail.gmail.com> I posted this on comp.lang.python and only got one email back. Maybe someone here could point me to a package python package that does man pages right. I know I am a dinosaur for still liking man pages. And a follow on... I just grabbed that 3.0 alpha of epydoc and after using it for two weeks, I REALLY like it, but it has some quirks. I haven't heard back from the author, but what is the current state of documentation best practice for python. I got frustrated by lack of features from pydoc and doxygen doesn't really understand python that well. Thanks much! -kurt my nntp post... I am rearranging the layout of one of my python projects so that it more closely conforms to how most python projects seem to work. I now have a structure like this: seismic-py - setup.py - seismic - bulk of the code - scripts - programs that go in bin I am using OptionParser, help2man, groff and man2html to provide man pages. Everything was sitting in the top level directory, so it was clear where to put these, but where do I put the foo.help2man files that contain extra text for the man pages? If I put it in scripts, that is pretty easy to cope with, but I was thinking about a docs directory, but then the build process might be more difficult. Or should I be putting in another string in each executable that contains this extra man page info? Then it would show up in epydoc as well. Maybe something like __help2man__ = ''' [AUTHOR] Kurt Schwehr [SEE ALSO] segysql.py [DESCRIPTION] .PP The --coord-unit option is designed to allow use of databases that exclude the CoordUnit field. This field is probably the same for all traces in the majority of SEGY data files, so most segy-py drivers will want to exclude coordunit from the short list (see segysql.py). The values are taken from page 14 of the SEG-Y Rev 1 specification -1 = Follow field 89-90 of the trace header 1 = Length (meters or feet) [NOT supported] 2 = Seconds of arc 3 = Degrees, minutes, seconds (DMS) [NOT SUPPORTED] ''' I am still in the middle of shuffling the tree about, but it is available here... https://cowfish.unh.edu/projects/seismic-py/ Any thoughts would be greatly appreciated! I am still trying to understand the best practices for python packaging. seismic-py - setup.py - seismic - __init.py__ - bulk of the code *.py - scripts - programs that go in bin/the users executable path (no .py extension) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20060406/09d310dd/attachment.html From jjl at pobox.com Sat Apr 8 19:09:37 2006 From: jjl at pobox.com (John J Lee) Date: Sat, 8 Apr 2006 17:09:37 +0000 (UTC) Subject: [Distutils] setuptools and Python 2.5? Message-ID: I seem to remember mention on python-dev of parts of setuptools going into Python 2.5. Did that / will that happen? If yes, any tips on updating packages & testing against 2.5 setuptools? John From pje at telecommunity.com Sat Apr 8 23:28:59 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 08 Apr 2006 17:28:59 -0400 Subject: [Distutils] setuptools and Python 2.5? In-Reply-To: Message-ID: <5.1.1.6.0.20060408172337.03736808@mail.telecommunity.com> At 05:09 PM 4/8/2006 +0000, John J Lee wrote: >I seem to remember mention on python-dev of parts of setuptools going into >Python 2.5. Did that / will that happen? What's happened so far is that 2.5 distutils has an 'install_egg_info' command to ensure that non-setuptools packages get their package metadata installed. >If yes, any tips on updating packages & testing against 2.5 setuptools? Make sure your setup script does not produce any warnings about your .egg-info directory or your namespace packages. The version that should go into Python 2.5 will probably not offer backward compatibility for the issues that today's setuptools warn about, as that would force me to retain that compatibility for years from now. So, check for warnings about undeclared namespace packages and misnamed egg-info directories, and update them accordingly. From mfogels at gmail.com Mon Apr 10 20:17:13 2006 From: mfogels at gmail.com (Mars) Date: Mon, 10 Apr 2006 14:17:13 -0400 Subject: [Distutils] Deployment with setuptools: a basket-of-eggs approach Message-ID: <3676e0c0604101117x20dfe770ndefe5ab1248b46f8@mail.gmail.com> Hello all, I was reading through my backlog of Daily Python URL's and saw that the topics of deployment and configuration managment seem to be getting some attention lately. We have been having issues similar to this at my company, and I was hoping for some feedback on a solution I devised for these problems. In our situation: - we have only a handful of developers, maintaining a very large legacy code base. - different applications use different versions of the same library on the same machine - developers still want to push bug fixes out without visiting every application - developers want to write new (non-compatible) versions of their libraries - developers want other developers to use their new libraries instead of the old ones The plan I have borrows a bit from the Java and Python development communities, and relies heavily on setuptools, policy, and some developer dicipline (we are professionals, after all). *Outline* - Developers outline Families of modules for common enterprise-wide tasks - Developers build Packages (python .egg files) from Families - Developers build Applications which are composed of Packages *Packages* - Packages are deployed to a SharedDirectory - Package versions have three number parts, Major.Minor.Bugfix - New package features, or *any* change in the package functionality, changes _at least_ the Minor version number. - Bugfixes should not change any functionality, and thus they only update the Bugfix version number - Developers write ChangeLogs for Packages before deployment *SharedDirectory* - Servers mount the packages SharedDirectory *ChangeLogs* - ChangeLogs outline any new features or Bugfixes in a Package - ChangeLogs are sent to all other developers after deployment *Applications* - Applications find their dependancies in the SharedDirectory - Applications have startup scripts that 'require' the developer-specified packages (for example, setuptools entry_points and automatic script generation do this) - Applications 'require' only the first two numbers of the package version, Major.Minor Some observations for deployers: - All applications should automatically upgrade if there is a bugfix version. - Changes in functionality are isolated from legacy applications. - Because applications are self-contained in setuptools .egg files, we can specify the shared directory as a global installation source when using easy_install. - Applications can be installed locally in a fashion similar to Java applications, which are often a collection of .jar files and data thrown into a common directory. We can do the same with "easy_install -f /opt/eggs -zmad /opt/myapp". - The application script itself could be shared from the central drive, if we write a custom script that does not use the shebang line. This has the side-effect of automatically upgrading *everyone's* version, which can be a blessing or a curse when you take versioning of run-time data into account. Some observations for developers: - The version numbering policy is flexible, but one *must* end up with a deployment policy where business-critical applications do not have even the most minor of functionality changes foisted upon them (for one solution see Java-style installations, outlined above). - Developers know what is happening to a package at a grain higher than that presented by Subversion, thanks to a published ChangeLog. - Developers are free to chose any version of the available packages, because they know that they are all available at install-time. They can upgrade and downgrade versions at will. - Developers are still responsible for upgrading legacy applications with more functional library versions, but that is what unit tests are for (you /do/ write unit tests, don't you?) - It helps to use a setuptools alias to easy_install when building packages for deployment. You can then type "python setup.py deploy" in the source directory as a quick and easy shortcut for building the required .egg. There are a few concerns that cross into release handling, and code library maintenance and care: - Regular library reviews and a comfortable package end-of-life schedule should be used to help prune the "supported packages" tree. - A relaxed package release schedule, with more features per-release, should help slow things down to a comfortable balance between time spent on upgrades, bugfixes, and features. - If the version numbers are climbing too fast, ask whether there should be more features per-release, or if the package should be put into 'alpha' or 'beta' status until it stabilizes. - Ask if anybody else is using the package you are maintaining. If not, ask yourself if it has be pre-maturely extract from an application (remember 'You Aint Gonna Need It'). - Ask if a quickly-rising package version indicates a family of modules and functionality that should be re-arranged. This could happen if someone is developing a new application and dumping too much application-specific functionality into an existing family. Can the family be split along functional lines? Should the functionality be kept in the application? (YAGNI again). Well, that's about it. You almost need to resurrect the role of 'Project Librarian' to keep track of it all! ;) I would love to hear people's feedback on this idea, as I am sure that I am not the first to tread this path. Maris From p.f.moore at gmail.com Tue Apr 11 00:02:15 2006 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 10 Apr 2006 23:02:15 +0100 Subject: [Distutils] Using easy_install without net access Message-ID: <79990c6b0604101502u206690aenb3b370770df149e4@mail.gmail.com> I'm trying to install a setuptools-based package (RuleDispatch) but there are no suitable binary eggs on the net (Python 2.5 on Windows). So I'm effectively building all of my own eggs and installing as if I have no net access [1]. According to http://peak.telecommunity.com/DevCenter/EasyInstall, """If you can't use APS, or don't have internet access at all, you will need to first download the appropriate .egg file from the setuptools PyPI page using a computer with internet access. Place the egg in the same directory as ez_setup.py on the target computer before running it.""" I tried this, but I can't get it to work - ez_setup.py still tries to go to CheeseShop. Do I need some command line flags not mentioned on the EasyInstall page? I've tried the most obvious ones, but to no avail. Actually, I got somewhere, by building a setuptools egg, adding it to PYTHONPATH, then running ez_setup.py. That "installed" setuptools, but didn't move it anywhere - it's still in my temporary "Data" directory. >From there, easy_install -f . RuleDispatch seems to be getting somewhere (I now need to build PyProtocols, but that's OK). But still nothing's getting copied into site-packages. I'm clearly doing something wrong, but I have no idea what. At the risk of going on about it, this is far messier than the traditional "python setup.py bdist_wininst" and run the installer... (BTW, what I ultimately want to do is to try out RuleDispatch on a machine which doesn't have net access. The rest is just annoying cruft. For now, I got this by building a wininst installer for setuptools, installing it, building wininst installers for pyprotocols and ruledispatch, uninstalling setuptools, and then using the wininst installers - as neither package uses setuptools at runtime...) Thanks, Paul. [1] Actually, one of the machines I'm doing this on is not (easily) connected to the net in any case. From amk at amk.ca Tue Apr 11 00:00:11 2006 From: amk at amk.ca (A.M. Kuchling) Date: Mon, 10 Apr 2006 18:00:11 -0400 Subject: [Distutils] PEP 243: in need of updating? Message-ID: <20060410220011.GA25936@localhost.localdomain> PEP 243 describes how to upload packages to a package repository. According to Richard, PyPI implements a modified version of PEP 243. (For a start, I'll bet it doesn't return a X-Swalow-Status header.) Could someone please update PEP 243 to match the PyPI reality? Or explain what the differences are? --amk From pje at telecommunity.com Tue Apr 11 00:27:43 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 10 Apr 2006 18:27:43 -0400 Subject: [Distutils] Using easy_install without net access In-Reply-To: <79990c6b0604101502u206690aenb3b370770df149e4@mail.gmail.co m> Message-ID: <5.1.1.6.0.20060410182350.05d16208@mail.telecommunity.com> At 11:02 PM 4/10/2006 +0100, Paul Moore wrote: >I'm trying to install a setuptools-based package (RuleDispatch) but >there are no suitable binary eggs on the net (Python 2.5 on Windows). >So I'm effectively building all of my own eggs and installing as if I >have no net access [1]. According to >http://peak.telecommunity.com/DevCenter/EasyInstall, > >"""If you can't use APS, or don't have internet access at all, you >will need to first download the appropriate .egg file from the >setuptools PyPI page using a computer with internet access. Place the >egg in the same directory as ez_setup.py on the target computer before >running it.""" > >I tried this, but I can't get it to work - ez_setup.py still tries to >go to CheeseShop. Do I need some command line flags not mentioned on >the EasyInstall page? I've tried the most obvious ones, but to no >avail. Hm. That seems odd. It should just install itself. Have you tried running "ez_setup.py --allow-hosts=None -f. setuptools"? >Actually, I got somewhere, by building a setuptools egg, adding it to >PYTHONPATH, then running ez_setup.py. That "installed" setuptools, but >didn't move it anywhere - it's still in my temporary "Data" directory. > >From there, easy_install -f . RuleDispatch seems to be getting >somewhere (I now need to build PyProtocols, but that's OK). But still >nothing's getting copied into site-packages. > >I'm clearly doing something wrong, but I have no idea what. Me either. I'm also not sure how to go about reproducing the problem. >At the risk of going on about it, this is far messier than the >traditional "python setup.py bdist_wininst" and run the installer... That actually works now with setuptools-based packages, by the way. >(BTW, what I ultimately want to do is to try out RuleDispatch on a >machine which doesn't have net access. The rest is just annoying >cruft. For now, I got this by building a wininst installer for >setuptools, installing it, building wininst installers for pyprotocols >and ruledispatch, uninstalling setuptools, and then using the wininst >installers - as neither package uses setuptools at runtime...) It's quite safe to do that; setuptools now fully supports bdist_wininst. From p.f.moore at gmail.com Tue Apr 11 00:32:35 2006 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 10 Apr 2006 23:32:35 +0100 Subject: [Distutils] Using easy_install without net access In-Reply-To: <5.1.1.6.0.20060410182350.05d16208@mail.telecommunity.com> References: <5.1.1.6.0.20060410182350.05d16208@mail.telecommunity.com> Message-ID: <79990c6b0604101532n7222142bk32d1ad5a87e5b005@mail.gmail.com> On 4/10/06, Phillip J. Eby wrote: > Hm. That seems odd. It should just install itself. Have you tried > running "ez_setup.py --allow-hosts=None -f. setuptools"? I tried -f . but missed --allow-hosts=None. No time to try it right now, but I'll give it a go later. (Just for completeness - see below). > >At the risk of going on about it, this is far messier than the > >traditional "python setup.py bdist_wininst" and run the installer... > > That actually works now with setuptools-based packages, by the way. [...] > It's quite safe to do that; setuptools now fully supports bdist_wininst. That's brilliant news - I thought I saw some egg-info directories appear, but I hadn't understood the significance. Thanks, Paul. From pje at telecommunity.com Tue Apr 11 00:42:10 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 10 Apr 2006 18:42:10 -0400 Subject: [Distutils] Using easy_install without net access In-Reply-To: <79990c6b0604101532n7222142bk32d1ad5a87e5b005@mail.gmail.co m> References: <5.1.1.6.0.20060410182350.05d16208@mail.telecommunity.com> <5.1.1.6.0.20060410182350.05d16208@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060410184100.0369a0f0@mail.telecommunity.com> At 11:32 PM 4/10/2006 +0100, Paul Moore wrote: >On 4/10/06, Phillip J. Eby wrote: > > Hm. That seems odd. It should just install itself. Have you tried > > running "ez_setup.py --allow-hosts=None -f. setuptools"? > >I tried -f . but missed --allow-hosts=None. No time to try it right >now, but I'll give it a go later. (Just for completeness - see below). > > > >At the risk of going on about it, this is far messier than the > > >traditional "python setup.py bdist_wininst" and run the installer... > > > > That actually works now with setuptools-based packages, by the way. >[...] > > It's quite safe to do that; setuptools now fully supports bdist_wininst. > >That's brilliant news - I thought I saw some egg-info directories >appear, but I hadn't understood the significance. Actually, I guess I should qualify that a bit -- it's only *fully* supported if the package has no dependencies, in that bdist_wininst has no provision for installing dependencies. From mfogels at gmail.com Tue Apr 11 15:38:48 2006 From: mfogels at gmail.com (Mars) Date: Tue, 11 Apr 2006 09:38:48 -0400 Subject: [Distutils] Deployment with setuptools: a basket-of-eggs approach In-Reply-To: <87irpgcxoc.fsf@localhost.localdomain> References: <3676e0c0604101117x20dfe770ndefe5ab1248b46f8@mail.gmail.com> <87irpgcxoc.fsf@localhost.localdomain> Message-ID: <3676e0c0604110638u697271d3p4c8223482afed8e2@mail.gmail.com> On 11 Apr 2006 10:40:35 +0200, Iwan Vosloo XXXXX wrote: > > Hi Maris, > > what do you mean by 'deploying' an egg to the SharedDirectory? Will > you install it there, or merely put it there so that installs > elsewhere can fetch the egg from the repository in SharedDirectory? > > > I also wondered what platforms are involved in your environment? > > -i > Hi Iwan. According to the documentation for the pkg_resources API, eggs need to be placed on sys.path somehow for the 'require' function's automatic resource discovery to work. The easiest way to do this is to add the SharedDirectory to the PYTHONPATH. With respect to deployment, I believe that simply copying the .egg to the SharedDirectory should be enough. This is similar to the instructions in the easy_install documentation under 'Installing on Un-networked Machines'. Thus, developers would execute the following: % easy_install -f /opt/eggs -zmaxd /opt/eggs mypackage This will have the effect of building and copying all eggs to the SharedDirectory, as well as copying all third-party eggs and dependancies. As an aside, this raises some interesting questions: should third-party eggs be treated as run-time data, and live with the application, or should they be treated the same as all of our internal eggs? What is this the most efficient way to upgrade versions? Also, the latter approach leaves some important questions about open-ended versioning. For example, what if I have a chain of dependancies, like so: MyPackage-1.0 -> eggA-1.0 -> eggB-1.0 What happens if I upgrade to eggB-2.0, which has an API that is incompatible with eggB-1.0? If I originally stated the dependancy as 'require(eggB)', I now have to go back and force eggA-1.0 to use the earlier version of eggB somehow. If it is a third-party package, I am in trouble, as I may not be able to force a specific version. I would then have to test all combinations of dependancies across all applications and versions... So, I just re-discovered a form of DLL Hell.... I could impliment a policy of freezing the dependancies to a specific Major and Minor version when an application or package states it's requirements. So a developer _must_ state the requirement as 'require(eggB==1.0)'. The onus then falls on the developer to expand the range of supported versions of dependancies as they are published (which is what a good unit test suite will do). Anyway, as far as platforms are concerned, we have a mix of Windows and Linux i386 systems, no 64bit systems. Directory sharing is implemented through CIFS (Samba). I should also mention that we are not working with modules that contain C extensions. I hope this answers your questions, Maris P.S. I CC'd this to distutils-sig, as I feel there is some information here that other people may find valuable. From iv at lantic.net Tue Apr 11 16:31:44 2006 From: iv at lantic.net (Iwan Vosloo) Date: 11 Apr 2006 16:31:44 +0200 Subject: [Distutils] Deployment with setuptools: a basket-of-eggs approach In-Reply-To: <3676e0c0604110638u697271d3p4c8223482afed8e2@mail.gmail.com> References: <3676e0c0604101117x20dfe770ndefe5ab1248b46f8@mail.gmail.com> <87irpgcxoc.fsf@localhost.localdomain> <3676e0c0604110638u697271d3p4c8223482afed8e2@mail.gmail.com> Message-ID: <871ww4chf3.fsf@localhost.localdomain> Hi Maris, Ok, I see... You can thus assume in your environment that the network will always be there. I was wondering whether you've ever looked at something like Debian's apt. (Mentioned here just to learn from it, not to advocate its use.) Apt is a wonderful tool for keeping repositories and installing packages. It does not solve all problems - and has the drawback that it only allows one version of something on a system (but you can trick it by having different package names...). The hell you're talking about is something that Debian (and, I suppose other distros) has a lot of experience in managing. And, for Debian, apt is the tool. (I don't know the others.) Of course there are also a number of conventions and policies that play with to make it work. I find it odd that you call upon unit testing. Is the issue not actually integration testing? I think that the only way to deal with the possible complexities of many packages and dependencies is to impose restictions on when and how things are released. For example, all the packages in Debian release X are tested to work together well (this is integration testing). So, in Debian, you don't only have packages, you also have a set-of-versioned packages (also versioned) which is the release of the entire distro. Any new version of a package, or new package that should work with that distro would need to be tested with all the rest of the packages in that release of the distro. I suppose disallowing more than one version of a package on a machine (like they have done) is one way of simplifying things. And the standard workaround for special packages that need more versions, is to include part of the version in the name. For example "gcc-3.4" (version 3.4.2) can be installed alongside "gcc-4.0" (version 4.0.3). With your scheme, your respository of eggs is also like a single, shared installation of eggs. And it may be argued that there is a difference between putting something in a shared repository (which means "it is now officially released") versus installing a package on a machine where it is used. When you install it, you care about other localised things too that are not versioned, such as local config or user data. And things like apt include ways and means for you to upgrade user data and deal with config sensibly. It may be that the simplification of making "install" and "release" one thing is useful in an environment, I guess. But in some environments the simplification may introduce other hellish issues. (Sorry, I'm just thinking aloud, because I am also faced with such problems and, by habit, always build things around apt... So its interesting to see other thoughts.) (And, I've no clue as to what platforms apt is available on...) -i From mfogels at gmail.com Tue Apr 11 19:02:24 2006 From: mfogels at gmail.com (Mars) Date: Tue, 11 Apr 2006 13:02:24 -0400 Subject: [Distutils] Deployment with setuptools: a basket-of-eggs approach In-Reply-To: <871ww4chf3.fsf@localhost.localdomain> References: <3676e0c0604101117x20dfe770ndefe5ab1248b46f8@mail.gmail.com> <87irpgcxoc.fsf@localhost.localdomain> <3676e0c0604110638u697271d3p4c8223482afed8e2@mail.gmail.com> <871ww4chf3.fsf@localhost.localdomain> Message-ID: <3676e0c0604111002i38ac40bcldd8d243aac2bdfdd@mail.gmail.com> On 11 Apr 2006 16:31:44 +0200, Iwan Vosloo XXXXX wrote: > > Hi Maris, > > Ok, I see... > > You can thus assume in your environment that the network will always > be there. > > I was wondering whether you've ever looked at something like Debian's > apt. (Mentioned here just to learn from it, not to advocate its use.) > Apt is a wonderful tool for keeping repositories and installing > packages. It does not solve all problems - and has the drawback that > it only allows one version of something on a system (but you can trick > it by having different package names...). > I am not completely familiar with the way that Debian handles package releases, but I do have experience with managing apt, ebuilds, rpms, and source installs. I do not believe that these systems are an option since some of our end-users are on Windows, and we do not have an in-house admistrator to handle application upgrades. Since end-users will be handling upgrades, and because of the widely varying technical skills of our users we must also have to try and have a trivial upgrade process (which easy_install can do, with a bit of help for runtime data and future-proofing). Since our servers are all Fedora Linux, we could try installing Cygwin on the Windows machines, and add facilities to automatically build two RPMS, one for Fedora, one for Cygwin, and install those. But it would be very hard to justify that extra complexity in a shop our size. One very nice thing about using setuptools and easy_install is that it keeps the application lifecycle within the Python world. > The hell you're talking about is something that Debian (and, I suppose > other distros) has a lot of experience in managing. And, for Debian, > apt is the tool. (I don't know the others.) Of course there are also > a number of conventions and policies that play with to make it work. > > I find it odd that you call upon unit testing. Is the issue not > actually integration testing? > You are right, it would be integration testing. With a technical staff as small as ours (3 people) I was thinking that coding unit tests for the boundaries between your package and its dependancies would help ease the work load of upgrading. I was hoping to ease adoption of the new system by combining integration testing with a framework that people are already familiar with. > I think that the only way to deal with the possible complexities of > many packages and dependencies is to impose restictions on when and > how things are released. For example, all the packages in Debian > release X are tested to work together well (this is integration > testing). So, in Debian, you don't only have packages, you also have > a set-of-versioned packages (also versioned) which is the release of > the entire distro. Any new version of a package, or new package that > should work with that distro would need to be tested with all the rest > of the packages in that release of the distro. > That level of testing would require the roles of either Project Librarian or Build Master, neither of which we could justify at a company our size. Especially given the rate at which new projects are written. > I suppose disallowing more than one version of a package on a machine > (like they have done) is one way of simplifying things. And the > standard workaround for special packages that need more versions, is > to include part of the version in the name. For example "gcc-3.4" > (version 3.4.2) can be installed alongside "gcc-4.0" (version 4.0.3). > Alas, that is not an option with our legacy applications, which mix library versions onto a single system, with the additional requirement to be able to apply bug fixes to the systems easily (without a sysadmin). > With your scheme, your respository of eggs is also like a single, > shared installation of eggs. And it may be argued that there is a > difference between putting something in a shared repository (which > means "it is now officially released") versus installing a package on > a machine where it is used. When you install it, you care about other > localised things too that are not versioned, such as local config or > user data. And things like apt include ways and means for you to > upgrade user data and deal with config sensibly. > > It may be that the simplification of making "install" and "release" > one thing is useful in an environment, I guess. But in some > environments the simplification may introduce other hellish issues. > Those other hellish issues are influenced by the vague, situation-specific part I am attempting to grasp the full shape of. It seems to hinge on a handful of critical requirements, such as "two applications, one system, different versions of libraries", "fast/easy deployment to all systems", "fast/easy upgrades to all installed applications", or "mixed operating systems". One side effect of the 'require() only a single version' policy: why wouldn't I just bundle everything anyway? Since I want only a single version of the library, because that is all that I can guarantee to be stable, then why not enforce the restriction physically? Then we would have the situation you speak of, making the distinction between 'installing on the local machine', and 'deploying to the shared directory'. I would have to install most applications as a multi-install in order to meet the requirement of "two applications, one system, different versions of libraries". So we get: # deploy % easy_install -f /opt/eggs -zmaxd /opt/eggs mypackage # install % easy_install -f /opt/eggs -zmad /some/where mypackage We now have the Java-style install I outlined in my first email (this is the part where I wish that easy_install had an option to install *everything* in the target directory, instead of trying to use /usr/bin or whatever for scripts, etc. We can get around this with a custom setup.cfg and the "$base" distutils variable) This satisfies the additional requirments of "multiple operating systems" and "easy/fast installation" That leaves us with the requirement for "easy/fast upgrades". Under the bundled setup we still have to visit every server in order to push out a bugfix version of a library. What is more, we have to visit every individual application that was installed using the multi-install option. One way around this would be to specify a subset of a version number to which easy_install could be applied. Something like this: % easy_install --upgrade mypackage==X.X.Y We could achieve this by using pkg_resources to get the application's dependancy tree, as well as all installed versions of all dependancies. Then it is a simple matter of parsing the version string for each package and spliting out the 'X.X' part. Then, for each package in the tree, we re-run easy install with this information: os.system('easy_install -f /opt/eggs --upgrade mypackage==X.X') easy_install should then grab and install any bugfix versions as required. As far as pushing or pulling application bugfixes, this could be handled by a cron script or scheduled task. One per application, which means more work to install any given app, because setuptools can't handle this at the moment. But it would work! > > (Sorry, I'm just thinking aloud, because I am also faced with such > problems and, by habit, always build things around apt... So its > interesting to see other thoughts.) > No problem there, thinking aloud is often why discussing ideas with people helps so much! Maris From matt at bogosian.net Wed Apr 12 09:32:59 2006 From: matt at bogosian.net (Matthew Bogosian) Date: Wed, 12 Apr 2006 00:32:59 -0700 Subject: [Distutils] py2app + setuptools-0.6a9 = *poof* Message-ID: <2AD77FA9-E936-44D5-B3FC-E9EEA62D7FF8@bogosian.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello all, My apologies in advance if this has already been covered, but I am very confused by the interaction between py2app and setuptools. I have installed Universal-MacPython-2.4.3 from (I have an Intel-based Mac) and built py2app 0.2.0 from source: > $ export PATH="/usr/local/bin:${PATH}" > $ svco http://svn.red-bean.com/bob/py2app/releases/py2app-0.2.0 > ... > $ cd py2app-0.2.0 > $ python setup.py bdist_mpkg --open > ... Now when I download ClientForm from and try to build it using py2app, I get the following: > $ tar xpvf ClientForm-0.2.2.tar.gz > $ cd ClientForm-0.2.2 > $ bdist_mpkg > > ---------------------------------------------------------------------- > ----- > This script requires setuptools version 0.6a9 to run (even to display > help). I will attempt to download it for you (from > http://cheeseshop.python.org/packages/2.4/s/setuptools/), but > you may need to enable firewall access for this script first. > I will start the download in 15 seconds. > ---------------------------------------------------------------------- > ----- > Downloading http://cheeseshop.python.org/packages/2.4/s/setuptools/ > setuptools-0.6a9-py2.4.egg > Traceback (most recent call last): > File "/usr/local/bin/bdist_mpkg", line 3, in ? > main() > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ > python2.4/site-packages/py2app/bdist_mpkg/scripts/ > script_bdist_mpkg.py", line 24, in main > execfile(sys.argv[0], g, g) > File "setup.py", line 88, in ? > ez_setup.use_setuptools() > File "/Users/matt/Desktop/macpython/orig/ClientForm-0.2.2/ > ez_setup/__init__.py", line 84, in use_setuptools > import setuptools; setuptools.bootstrap_install_from = egg > File "/Users/matt/Desktop/macpython/orig/ClientForm-0.2.2/ > setuptools-0.6a9-py2.4.egg/setuptools/__init__.py", line 3, in ? > File "/Users/matt/Desktop/macpython/orig/ClientForm-0.2.2/ > setuptools-0.6a9-py2.4.egg/setuptools/dist.py", line 29, in ? > File "/Users/matt/Desktop/macpython/orig/ClientForm-0.2.2/ > setuptools-0.6a9-py2.4.egg/setuptools/dist.py", line 24, in > _get_unpatched > AssertionError: distutils has already been patched by py2app.install.Distribution at 0x589390> I'm now at a total loss. Is there any advice anyone can give me on now to get this to work (besides "don't use (py2app|ClientForm)")? Any help is greatly appreciated. --Matt -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (Darwin) iD8DBQFEPK0snLpDzL5I7l8RAqy1AJ0bp2Yt7Nj3O1dSZm0+upcszZEH6ACeMvG8 XAWwdG9j3iv5MgQdKFAEIT4= =Gj4O -----END PGP SIGNATURE----- From pje at telecommunity.com Wed Apr 12 18:27:20 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 12 Apr 2006 12:27:20 -0400 Subject: [Distutils] py2app + setuptools-0.6a9 = *poof* In-Reply-To: <2AD77FA9-E936-44D5-B3FC-E9EEA62D7FF8@bogosian.net> Message-ID: <5.1.1.6.0.20060412122343.01fa8008@mail.telecommunity.com> At 12:32 AM 4/12/2006 -0700, Matthew Bogosian wrote: > > AssertionError: distutils has already been patched by > py2app.install.Distribution at 0x589390> > >I'm now at a total loss. Is there any advice anyone can give me on >now to get this to work (besides "don't use (py2app|ClientForm)")? >Any help is greatly appreciated. Whenever setuptools is used with a tool (such as py2exe or py2app), setuptools must be imported first. In your specific situation, you should first install the latest version of setuptools, then replace your existing setup command with: python -c "import setuptools; execfile('setup.py')" stuff where "stuff" is whatever you would have put after "python setup.py" on the command line. This is the easiest way to force setuptools to be imported first, before anything else has a chance to attach itself to the distutils. This still doesn't guarantee success, since it's possible for other programs to monkeypatch the distutils in ways that break setuptools. But it should at least move you forward another step. From ianb at colorstudy.com Wed Apr 12 19:04:37 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 12 Apr 2006 12:04:37 -0500 Subject: [Distutils] Deployment with setuptools: a basket-of-eggs approach In-Reply-To: <3676e0c0604101117x20dfe770ndefe5ab1248b46f8@mail.gmail.com> References: <3676e0c0604101117x20dfe770ndefe5ab1248b46f8@mail.gmail.com> Message-ID: <443D3325.2070409@colorstudy.com> Hi there; a bunch of ideas here, more than I can digest all at once I suppose. Mars wrote: > Hello all, > > I was reading through my backlog of Daily Python URL's and saw that > the topics of deployment and configuration managment seem to be > getting some attention lately. We have been having issues similar to > this at my company, and I was hoping for some feedback on a solution I > devised for these problems. > > In our situation: > - we have only a handful of developers, maintaining a very large > legacy code base. > - different applications use different versions of the same library > on the same machine > - developers still want to push bug fixes out without visiting every > application > - developers want to write new (non-compatible) versions of their libraries > - developers want other developers to use their new libraries instead > of the old ones > > The plan I have borrows a bit from the Java and Python development > communities, and relies heavily on setuptools, policy, and some > developer dicipline (we are professionals, after all). > > *Outline* > - Developers outline Families of modules for common enterprise-wide tasks > - Developers build Packages (python .egg files) from Families > - Developers build Applications which are composed of Packages > > *Packages* > - Packages are deployed to a SharedDirectory > - Package versions have three number parts, Major.Minor.Bugfix > - New package features, or *any* change in the package functionality, > changes _at least_ the Minor version number. Given some other things you mention here, I think metadata updates will be common, which I assume would be grouped in with bugfixes in the version. > - Bugfixes should not change any functionality, and thus they only > update the Bugfix version number > - Developers write ChangeLogs for Packages before deployment > > *SharedDirectory* > - Servers mount the packages SharedDirectory > > *ChangeLogs* > - ChangeLogs outline any new features or Bugfixes in a Package > - ChangeLogs are sent to all other developers after deployment > > *Applications* > - Applications find their dependancies in the SharedDirectory > - Applications have startup scripts that 'require' the > developer-specified packages (for example, setuptools entry_points and > automatic script generation do this) > - Applications 'require' only the first two numbers of the package > version, Major.Minor Yes, requiring a fixed version doesn't work in my experience. Requiring a range should, i.e., >=1.5,<1.6. After testing you may change that requirement, and so you'll have to release. E.g., you add a new feature, you test it works with an old version of another library, and you change that requirement to >=1.5,<1.7 and release that other library as a bugfix release. This can work for applications as well as libraries. I was talking to someone about .NET and how it handles versioning, and he mentioned that you can configure overrides. So if you have a package A that works for B>=1.5,<1.6, and you confirm that at least in a specific context B==1.7.3 works, then you can add something to that *context* (not to either package) that overrides the requirement with B>=1.5,<1.6,>=1.7.3. This would decrease the number of metadata-updating releases you would have to do. I do everything as a "library", in that they are all setuptools packages. This works well for me. However, there's some as-yet-undefined entity above that. An installation, or an environment. For me that thing is a "website", though even that isn't strictly true -- sometimes a website has more than one environment (e.g., if there's a conflict I don't care to resolve), or more than one website has the same environment (if I want to run more than one website in the same process). So no concrete term really works. But the idea of an environment seems like an important abstraction to work in. I think it is similar to what you are thinking of as an application. A feature I'd really like is the ability to do an svn commit in concert that also updated some metadata. So, lets say I have package A and B, and A requires B, and I'm updating A but have to make a change in B to go with it. So I might say: fancysvn commit -m "changes" --update-requirements=A A/ B/ Then it would commit the changes in B, get the new version (which in development would be based on the svn revision), update A's entry_points.txt (in this model setup.py wouldn't hold that data), and then commit A. I end up doing this a lot, but without this command I seldom update the requirements like I should. > Some observations for deployers: > - All applications should automatically upgrade if there is a bugfix version. For this you need a central database of application installations. Otherwise it seems like a simple script to do the update. > - Changes in functionality are isolated from legacy applications. > - Because applications are self-contained in setuptools .egg files, > we can specify the shared directory as a global installation source > when using easy_install. I just use an internal web directory for all the packages, though local access would be faster. I've come to find actual .tar.gz or .egg creation a bit tedious, and have been considering using svn checkouts for everything. Then the central index would be a series of links to svn repository locations. That would require a link for every tag, e.g., svn://repos/Package/tags/0.3#egg=Package-0.3 Potentially such a page could be automatically generated from an svn hook, since the links are obvious if you stick to a conventional svn layout. > - Applications can be installed locally in a fashion similar to Java > applications, which are often a collection of .jar files and data > thrown into a common directory. We can do the same with "easy_install > -f /opt/eggs -zmad /opt/myapp". > - The application script itself could be shared from the central > drive, if we write a custom script that does not use the shebang line. > This has the side-effect of automatically upgrading *everyone's* > version, which can be a blessing or a curse when you take versioning > of run-time data into account. Yeah, that scares me a bit. I'd rather keep track automatically, then upgrade individually. Also easier to roll back just one piece if necessary. If you stick with the idea of an isolated environment -- instead of a big pool -- it also means you don't have to lean to heavily on the versioning features of setuptools. The advantage there being that the system is more translatable to other environments and languages. > Some observations for developers: > - The version numbering policy is flexible, but one *must* end up > with a deployment policy where business-critical applications do not > have even the most minor of functionality changes foisted upon them > (for one solution see Java-style installations, outlined above). I think the versioning will be pretty important. For instance, the business needs of versioning have to be kept separate -- you'll have fairly formal requirements for how versions are assigned. > - Developers know what is happening to a package at a grain higher > than that presented by Subversion, thanks to a published ChangeLog. Do you have any particular thoughts on how the ChangeLog is generated/presented? I'm not very good at keeping logs at different levels of granularity (right now I typically only keep per-commit logs of work). > - Developers are free to chose any version of the available packages, > because they know that they are all available at install-time. They > can upgrade and downgrade versions at will. Though it improves over time, as some packages reach maturity, in my experience work typically involves poking at more than one package. So during development I want everything to be editable. This strongly favors working against the trunk (or at least a branch), since editing an older version of a package means you have to be quite careful about actually deploying. Some of the mini-branch techniques people talk about might help here. Mostly people talk about this in the context of distributed systems, but I believe the Divmod guys are doing mini branches in subversion. > - Developers are still responsible for upgrading legacy applications > with more functional library versions, but that is what unit tests are > for (you /do/ write unit tests, don't you?) Buildbot would probably be a great boon here as well. While you don't want to accidentally "upgrade" a working deployed app, having tests run on all combinations of the libraries and apps will give you some idea of how much breakage has occurred, and when. > - It helps to use a setuptools alias to easy_install when building > packages for deployment. You can then type "python setup.py deploy" > in the source directory as a quick and easy shortcut for building the > required .egg. I've certainly found minor conveniences like these to be strong incentives to actually doing the right thing each time. > There are a few concerns that cross into release handling, and code > library maintenance and care: > - Regular library reviews and a comfortable package end-of-life > schedule should be used to help prune the "supported packages" tree. > - A relaxed package release schedule, with more features per-release, > should help slow things down to a comfortable balance between time > spent on upgrades, bugfixes, and features. > - If the version numbers are climbing too fast, ask whether there > should be more features per-release, or if the package should be put > into 'alpha' or 'beta' status until it stabilizes. What is the problem you are solving here? Version numbers are just numbers, after all. I guess one advantage of dated versions, or integer-only versions, is that you avoid some emotional resistance to increasing versions since the versions are relatively meaningless. > - Ask if anybody else is using the package you are maintaining. If > not, ask yourself if it has be pre-maturely extract from an > application (remember 'You Aint Gonna Need It'). > - Ask if a quickly-rising package version indicates a family of > modules and functionality that should be re-arranged. This could > happen if someone is developing a new application and dumping too much > application-specific functionality into an existing family. Can the > family be split along functional lines? Should the functionality be > kept in the application? (YAGNI again). That's definitely been an issue I've seen. There's definitely some internal packages which need splitting, for instance, since large portions have been highly stable, and updates elsewhere in the package shouldn't effect the release schedule of that stable core. Also, if I want to know what a change will effect, I'll know that better if the granularity of that change is clearer. > Well, that's about it. You almost need to resurrect the role of > 'Project Librarian' to keep track of it all! ;) I think more tool and reporting support is definitely called for. In many cases instead of relying too heavily on setuptools loading the right package, I think it would be better to think in terms of an evironment builder with static analysis of the result. E.g., you have a deployment that includes apps X, Y, and Z, then you start adding version requirements there, see what that brings in, report on any conflicts, and have the ability to do that speculatively. That is, where setuptools primarily checks this things at runtime now, based on a specific installation, this would be a little more look-before-you-leap. All the functions are in pkg_resources already, but there's no frontend to really look at this stuff. > I would love to hear people's feedback on this idea, as I am sure that > I am not the first to tread this path. I'm definitely interested in this stuff too; but at least in the context of Python there's not a whole lot of experience in these things. Or at best, people have set up their own ad hoc systems. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Wed Apr 12 19:29:33 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 12 Apr 2006 13:29:33 -0400 Subject: [Distutils] Deployment with setuptools: a basket-of-eggs approach In-Reply-To: <443D3325.2070409@colorstudy.com> References: <3676e0c0604101117x20dfe770ndefe5ab1248b46f8@mail.gmail.com> <3676e0c0604101117x20dfe770ndefe5ab1248b46f8@mail.gmail.com> Message-ID: <5.1.1.6.0.20060412132811.01f903e0@mail.telecommunity.com> At 12:04 PM 4/12/2006 -0500, Ian Bicking wrote: >Yes, requiring a fixed version doesn't work in my experience. Requiring >a range should, i.e., >=1.5,<1.6. You probably want "<1.6a", to avoid pre-releases. It especially helps if you use a version number scheme that requires certain digits to be bumped when there's an API change. For example, the above would always be a good idea if the minor release number indicates API changes. From bob at redivi.com Wed Apr 12 19:50:32 2006 From: bob at redivi.com (Bob Ippolito) Date: Wed, 12 Apr 2006 10:50:32 -0700 Subject: [Distutils] py2app + setuptools-0.6a9 = *poof* In-Reply-To: <2AD77FA9-E936-44D5-B3FC-E9EEA62D7FF8@bogosian.net> References: <2AD77FA9-E936-44D5-B3FC-E9EEA62D7FF8@bogosian.net> Message-ID: On Apr 12, 2006, at 12:32 AM, Matthew Bogosian wrote: > My apologies in advance if this has already been covered, but I am > very confused by the interaction between py2app and setuptools. They're more or less completely incompatible at this point. py2app 0.2.x has no idea what setuptools is, the next major version will be dependent on setuptools (and probably also Python 2.4). -bob From matt at bogosian.net Wed Apr 12 20:58:29 2006 From: matt at bogosian.net (Matthew Bogosian) Date: Wed, 12 Apr 2006 11:58:29 -0700 Subject: [Distutils] py2app + setuptools-0.6a9 = *poof* In-Reply-To: References: <2AD77FA9-E936-44D5-B3FC-E9EEA62D7FF8@bogosian.net> Message-ID: <80154579-3E3F-42E5-8236-3BAD18E22995@bogosian.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Apr 12, 2006, at 10:50 , Bob Ippolito wrote: > On Apr 12, 2006, at 12:32 AM, Matthew Bogosian wrote: > >> My apologies in advance if this has already been covered, but I am >> very confused by the interaction between py2app and setuptools. > > They're more or less completely incompatible at this point. py2app > 0.2.x has no idea what setuptools is, the next major version will > be dependent on setuptools (and probably also Python 2.4). Not to sound irreverent or impatient, but do you have any idea when the next major version will be available? (I understand that putting food on the table takes precedence.) --Matt -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (Darwin) iD8DBQFEPU3VnLpDzL5I7l8RAvAGAJ9M0msWe0x6IQE3zb1Jp0hY3m/izgCfShpX xIsXhuH/uCn8FaV0hbikzzs= =o6ja -----END PGP SIGNATURE----- From bob at redivi.com Wed Apr 12 21:24:13 2006 From: bob at redivi.com (Bob Ippolito) Date: Wed, 12 Apr 2006 12:24:13 -0700 Subject: [Distutils] py2app + setuptools-0.6a9 = *poof* In-Reply-To: <80154579-3E3F-42E5-8236-3BAD18E22995@bogosian.net> References: <2AD77FA9-E936-44D5-B3FC-E9EEA62D7FF8@bogosian.net> <80154579-3E3F-42E5-8236-3BAD18E22995@bogosian.net> Message-ID: <845F1BC2-99DB-4BCF-BA70-438E6A61CD89@redivi.com> On Apr 12, 2006, at 11:58 AM, Matthew Bogosian wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Apr 12, 2006, at 10:50 , Bob Ippolito wrote: > >> On Apr 12, 2006, at 12:32 AM, Matthew Bogosian wrote: >> >>> My apologies in advance if this has already been covered, but I am >>> very confused by the interaction between py2app and setuptools. >> >> They're more or less completely incompatible at this point. >> py2app 0.2.x has no idea what setuptools is, the next major >> version will be dependent on setuptools (and probably also Python >> 2.4). > > Not to sound irreverent or impatient, but do you have any idea when > the next major version will be available? (I understand that > putting food on the table takes precedence.) It'll be available when I get around to it.. which is some completely indeterminate amount of time. I'm not currently working on anything related to Mac OS X specifically at this time, so I'm not in a hurry. For the most part, eggs are a better solution than bdist_mpkg anyway (except for peripheral stuff like docs, applications, and examples). When I do get around to working on py2app's next major version, my focus will be to support the building of applications. bdist_mpkg is mostly obsolete in a setuptools world. -bob From iv at lantic.net Thu Apr 13 11:44:15 2006 From: iv at lantic.net (Iwan Vosloo) Date: 13 Apr 2006 11:44:15 +0200 Subject: [Distutils] Deployment with setuptools: a basket-of-eggs approach In-Reply-To: <3676e0c0604111002i38ac40bcldd8d243aac2bdfdd@mail.gmail.com> References: <3676e0c0604101117x20dfe770ndefe5ab1248b46f8@mail.gmail.com> <87irpgcxoc.fsf@localhost.localdomain> <3676e0c0604110638u697271d3p4c8223482afed8e2@mail.gmail.com> <871ww4chf3.fsf@localhost.localdomain> <3676e0c0604111002i38ac40bcldd8d243aac2bdfdd@mail.gmail.com> Message-ID: <87slohbyj4.fsf@localhost.localdomain> Hi Maris, I'm not suggesting using apt - just thinking about the difference in models. I suppose the pure python equivalent of apt and its repositories would be to have your own private PyPi (the respository), and using easy_install to install things from there onto individual machines. This does not necessitate more work: installing and updating packages on individual machines _from_ this central repository can also be automated. For example by running a script in cron every night or so. This gives you a set of _released_ packages (those in the repository), and different machines where instances of packages are _installed_. I suppose there is a correlation between what Philip calls an environment in this sense, and a machine in the debian/apt world. An environment is just a more abstract way of looking at it allowing more flexibility. Also, I tend to package everything as an egg - applications also. Eggs are just a way to package software to me, not libraries? -i From iv at lantic.net Thu Apr 13 11:47:10 2006 From: iv at lantic.net (Iwan Vosloo) Date: 13 Apr 2006 11:47:10 +0200 Subject: [Distutils] Deployment with setuptools: a basket-of-eggs approach In-Reply-To: <87slohbyj4.fsf@localhost.localdomain> References: <3676e0c0604101117x20dfe770ndefe5ab1248b46f8@mail.gmail.com> <87irpgcxoc.fsf@localhost.localdomain> <3676e0c0604110638u697271d3p4c8223482afed8e2@mail.gmail.com> <871ww4chf3.fsf@localhost.localdomain> <3676e0c0604111002i38ac40bcldd8d243aac2bdfdd@mail.gmail.com> <87slohbyj4.fsf@localhost.localdomain> Message-ID: <87odz5bye9.fsf@localhost.localdomain> Iwan Vosloo writes: > I suppose there is a correlation between what Philip calls an > environment in this sense, and a machine in the debian/apt world. An > environment is just a more abstract way of looking at it allowing more > flexibility. Sorrry, that was meant to be Ian, not Philip... -i From mfogels at gmail.com Thu Apr 13 17:04:02 2006 From: mfogels at gmail.com (Mars) Date: Thu, 13 Apr 2006 11:04:02 -0400 Subject: [Distutils] Deployment with setuptools Message-ID: <3676e0c0604130804q302886d9xe5947317015d1ca4@mail.gmail.com> This has been a very good discussion. I have been doing a bit more thinking regarding why I intended to use a packaging tool in the first place. I am relying on the package versioning scheme to ensure that critical libraries line up across families of applications. Libraries for internal company-wide file formats are a good example of this. I also think that it is easier for a deployers and developers to have an object or entity that must be equivalent across all applications for the application to be valid. It is easier than referencing a Subversion tag. Given that, what would my ideal application lifecycle look like? I would want applications to be registered to the machine, but I want libraries to be local to the execution environment. Sort of like using the 'multi-version' option, but the environment name is still in a central database. Then I could issue the following command to update every applicaton on the system: % easy_install --list-registered | xargs -i easy_install -U - I get: - multiple independant environments - a central database of registered environments - a database of packages on a per-environment basis by looking at the physical package names in each environment - the ability to copy elements across environments easily - I can not share elements across environments - I can use elements from the system-wide shared repository. It would be nice to install packages as either zip, editable, or development mode on a per-environment basis. Heck, why not have the ability to install the entire environment in development mode, with one command? I prefer the distutils installation scheme for --home or --prefix or --root lined up across operating systems. But this is contrary to distutils' general case installation scenario. (perhaps a new keyword, '--environment' or '--deploy' would make sense? Or we could use setup.cfg) As far as general versioning and release policies, the method outlined py the py.lib project looks to be light-weight and flexible (micro-releases, API backwards compatability guaranteed through testing, etc.): http://codespeak.net/py/current/doc/why_py.html#upcoming-release-policy-api-guarantees This works well with the automatic upgrading of micro-releases made possible by the '1.0>=,<2.0a' version specification. The micro-branching techniques that Ian mentioned go nicely with the previously mentioned release policy and testing guideline. http://divmod.org/trac/wiki/BranchBasedDevelopment The overall process looks like a good method to guarantee that the trunk never gets broken (which can be a *very* annoying problem, for the reasons the Divmod guys outline). It is also possible that micro-branching would solve the ChangeLog issues that Ian spoke of. Since each micro-branch merge is for a specific feature it would be possible to create a nice formatted commit message, and use something like svn-to-changelog on the trunk to get the whole ChangeLog automatically (even if the output format from svn2cl is ugly...): http://ch.tudelft.nl/~arthur/svn2cl/ It would be nice to write my code in an entirely version-agnostic manner by relying on the environment entry points to handle the package specification. Then my import statements are just normal Python code. The idea of an environment-specific package dependancy override is very nice. Almost a necessity, in fact. We have to handle the override at resource aquisition time (before site.py is generated?) It would be nice to hook setuptools into scons or aap or even make for the time being, so that I could handle a more customized environment setup. Things like configuration templates, log file stubs, auto-generated documentation, an environment-specific directory structure, etc. could all be there. (or maybe I could just check the entire environment template out of Subversion, and let easy_install handle filling the template with Python code.) To support the enviroment templates we would need to somehow extract some of the package data to a standard location. I considered using eager_resources for this, but it is not quite eager enough for my requirments. I could write an extension, 'very_eager_resources', to do this instead. I believe that everything here is possible using setuptools, easy_install, and pkg_resources in it's current form, with a few custom extensions. There is definitely a focus on using setuptools as an environment building and maintenance tool, or at least as something to glue more specialized tools together. The challenge is to create a process that is light-weight enough to be accepted for day-to-day use, instead of re-creating something like Java's EAR and WAR specifications. Maris From ianb at colorstudy.com Thu Apr 13 21:40:46 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 13 Apr 2006 14:40:46 -0500 Subject: [Distutils] Deployment with setuptools In-Reply-To: <3676e0c0604130804q302886d9xe5947317015d1ca4@mail.gmail.com> References: <3676e0c0604130804q302886d9xe5947317015d1ca4@mail.gmail.com> Message-ID: <443EA93E.6010405@colorstudy.com> Mars wrote: > I prefer the distutils installation scheme for --home or --prefix or > --root lined up across operating systems. But this is contrary to > distutils' general case installation scenario. (perhaps a new > keyword, '--environment' or '--deploy' would make sense? Or we could > use setup.cfg) I don't have time to respond to all of this now, but have you looked at working-env.py? http://blog.ianbicking.org/working-env.html You activate it by putting the working environment on your $PYTHONPATH, and then easy_install should automatically install in that environment and there is a custom distutils.cfg local to that environment. It's like virtual-python.py, except a bit lighter. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From mfogels at gmail.com Thu Apr 13 21:55:24 2006 From: mfogels at gmail.com (Mars) Date: Thu, 13 Apr 2006 15:55:24 -0400 Subject: [Distutils] Deployment with setuptools In-Reply-To: <443EA93E.6010405@colorstudy.com> References: <3676e0c0604130804q302886d9xe5947317015d1ca4@mail.gmail.com> <443EA93E.6010405@colorstudy.com> Message-ID: <3676e0c0604131255t404dee0fn25e86f97b2d4caa@mail.gmail.com> On 4/13/06, Ian Bicking wrote: > Mars wrote: > > I prefer the distutils installation scheme for --home or --prefix or > > --root lined up across operating systems. But this is contrary to > > distutils' general case installation scenario. (perhaps a new > > keyword, '--environment' or '--deploy' would make sense? Or we could > > use setup.cfg) > > I don't have time to respond to all of this now, but have you looked at > working-env.py? http://blog.ianbicking.org/working-env.html > > You activate it by putting the working environment on your $PYTHONPATH, > and then easy_install should automatically install in that environment > and there is a custom distutils.cfg local to that environment. It's > like virtual-python.py, except a bit lighter. > > > > -- > Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org > I have heard working-env.py mentioned elsewhere, but have not checked it out myself. I will look into it. Thanks! Maris From schwehr at gmail.com Fri Apr 14 11:54:19 2006 From: schwehr at gmail.com (Kurt Schwehr) Date: Fri, 14 Apr 2006 05:54:19 -0400 Subject: [Distutils] externally managed packages Message-ID: <37de72610604140254n34cec64ew318094a676145ba6@mail.gmail.com> Hi All, The externally managed option seems to be working great for fink/macosx, but I have a question. Are these file necessary or should I be removing them from the install? Thanks, -kurt /sw/lib/python2.4/site-packages/SQLAlchemy-0.1.6-py2.4.egg-info /sw/lib/python2.4/site-packages/SQLAlchemy-0.1.6-py2.4.egg-info/PKG-INFO /sw/lib/python2.4/site-packages/SQLAlchemy-0.1.6-py2.4.egg-info/SOURCES.txt /sw/lib/python2.4/site-packages/SQLAlchemy-0.1.6-py2.4.egg-info /top_level.txt -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20060414/23381f9a/attachment.html From pje at telecommunity.com Fri Apr 14 17:36:07 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 14 Apr 2006 11:36:07 -0400 Subject: [Distutils] externally managed packages In-Reply-To: <37de72610604140254n34cec64ew318094a676145ba6@mail.gmail.co m> Message-ID: <5.1.1.6.0.20060414113405.01e32120@mail.telecommunity.com> At 05:54 AM 4/14/2006 -0400, Kurt Schwehr wrote: >Hi All, > >The externally managed option seems to be working great for fink/macosx, >but I have a question. Are these file necessary or should I be removing >them from the install? Yes, they're necessary. No, don't remove them. They are there so that packages with dynamic (run-time) dependencies can discover what version of the package is installed. Not only that, but many packages have files of their own in .egg-info, that are used by themselves or by other systems they plug into. In Python 2.5, even distutils-installed packages will have some of this stuff installed, again so that other packages can discover what is installed on sys.path. From pje at telecommunity.com Fri Apr 14 20:25:44 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 14 Apr 2006 14:25:44 -0400 Subject: [Distutils] setuptools 0.6 beta branch created - trunk is 0.7 alpha Message-ID: <5.1.1.6.0.20060414141329.04167760@mail.telecommunity.com> I've just created a SVN branch for setuptools 0.6 beta and beyond, which can now be accessed with 'setuptools=dev06'. 'setuptools==dev' now refers to the 0.7 development trunk. I had to do this now because there's a fair amount of surgery I need to do to make setuptools merge with the Python 2.5 standard library, including module moves, renames, etc. So, to not lose all the hard-won stability of 0.6 as it is, I've split things up. Naturally, I hope that 0.7 will be just as stable as 0.6; it's more of a "have something always ready to release" issue. There's going to be some fairly intense reshufflings of the source code, though, including a switch from reStructuredText to PythonDoc LaTeX for the documentation, due to the need for integration with the Python documentation system. The C source code for the launcher will get moved, etc. There is also going to be some code migrated from pkg_resources to the stdlib's "pkgutil" module - specifically, the importer-specific infrastructure for filesystem and zipfile packages, as it's already been duplicated two other places in the stdlib. So the plan is to consolidate the various versions of this into pkgutil. setuptools 0.7 will include a replacement version of pkgutil for use by version of Python <2.5, but the setuptools bundled with Python 2.5 will just use the stdlib pkgutil. I don't plan to make any API changes to pkg_resources, however. It's only some of the implementation that's moving. It's also likely that Python 2.5 and up will use 'python -m easy_install' as the default way to invoke easy_install, but the older versions will retain their separate easy_install executable. I'll keep you posted here if there are any changes to the above. Please keep in mind that if you are currently relying on 'setuptools==dev' being relatively stable, you should consider whether you want to start using 'setuptools==dev06' instead, at least until an actual 0.7a1 release is available. In the meantime, I'm assuming that all is going well with 0.6a11, and that it can reasonably be bumped to 0.6b1 sometime next month. Please let me know if there are any issues outstanding with the 0.6 versions. Thanks. From vijayendra.bapte at gmail.com Tue Apr 18 06:03:52 2006 From: vijayendra.bapte at gmail.com (vijayendra bapte) Date: Tue, 18 Apr 2006 09:33:52 +0530 Subject: [Distutils] Howto include .dll files and .so files Message-ID: Hi All, I have created one package. It contains few binary files such as .so and .dll files. I want to include those files in my package distribution. I tried adding those files using data files but it was not working. Can any one please guide me how to include those files in my distribution. Regards, Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20060418/8cab4059/attachment.html From pje at telecommunity.com Tue Apr 18 06:38:18 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 18 Apr 2006 00:38:18 -0400 Subject: [Distutils] setuptools now in Python 2.5 trunk Message-ID: <5.1.1.6.0.20060418001001.01faac38@mail.telecommunity.com> Just a heads-up: setuptools 0.7a1.dev-r45519 is now in the Python 2.5 stdlib trunk. The docs haven't been converted yet, nor have the various PEPs and NEWS or whatsnew docs been updated. But it's in, and the tests run as part of the Python 2.5 regression test suite. I had to tweak a few things in setuptools 0.7 in order to make everything work cleanly across 2.3, 2.4, and 2.5, but it's done now. And, as a side benefit, setuptools 0.7 now installs its own fixes for the pydoc and doctest problems with modules in zipfiles! (And Python 2.5 doesn't have those problems to begin with; it also supports showing tracebacks and warnings and other things using source code pulled from .egg or .zip files.) These fixes are automatically substituted for the corresponding 2.3 and 2.4 stdlib modules whenever you install setuptools 0.7 with an older Python version, unless you are installing setuptools as a system package. (That's because system packages installed to site-packages can't override stdlib modules, but .eggs installed anywhere (even site-packages) *can*.) So, if you get a chance to try the curren SVN head of Python, or the 2.5 alpha 2 release, let me know if you encounter any issues with setuptools. Also, if you're using the setuptools 0.7 trunk with any other Python version, please let me know about any issues there as well. Thanks. (By the way, in setuptools 0.7 I've reinstated support for running with "python -m easy_install", because that was easier than trying to get the Python build process to install a script or executable.) From ianb at colorstudy.com Wed Apr 19 19:47:23 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 19 Apr 2006 12:47:23 -0500 Subject: [Distutils] Further extending of setup.py In-Reply-To: <4415B40D.4010601@colorstudy.com> References: <4415B40D.4010601@colorstudy.com> Message-ID: <444677AB.6090205@colorstudy.com> Phillip -- any thoughts on this? If this went in, this would allow me to start splitting up and simplifying some commands, as outlined in: http://blog.ianbicking.org/paster-and-scripts.html Ian Bicking wrote: > I'd like to get rid of the "paster" script, moving some of what it does > into setup.py or elsewhere. > > To a degree that is possible, but I think requires some additions to > setuptools: > > * An entry point group that is not globally looked up, like > distutils.commands. This would be a set of commands the package > provides only for itself. I expect it to look like: > > [distutils.commands.local] > sql_record = sqlobject.manage.distextension:sql_record > > Exactly like what we have currently, just not looked up globally. > > * Additionally, or probably even better, something that enumerates > locations for commands. E.g.: > > extra_commands=['SQLObject'] > > And then it would look in the SQLObject egg for > distutils.commands.local, and apply everything found there to this > package. Right now, for instance, buildutils adds a "pytest" command to > every project, even though it only applies to some projects (for some > commands this is ok, like "info", so two different entry points makes > sense). A project could list itself to provide its own custom commands; > I think that won't be too circular. Typically commands you provide for > yourself or someone else are different -- e.g., the SQLObject commands > don't apply to SQLObject itself. > > * Everything that can be done on a deployed egg will probably go in > app/egg-specific command-line scripts, and maybe I'll make a little > framework to make these easier to write, but that's entirely an > implementation detail. But I'm also thinking that extra_commands could > be used as a hint to that framework, and would kind of facilitate > coordination of in-development commands with after-deployment commands. From pje at telecommunity.com Wed Apr 19 21:42:30 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 19 Apr 2006 15:42:30 -0400 Subject: [Distutils] Further extending of setup.py In-Reply-To: <444677AB.6090205@colorstudy.com> References: <4415B40D.4010601@colorstudy.com> <4415B40D.4010601@colorstudy.com> Message-ID: <5.1.1.6.0.20060419152353.01e3b3c0@mail.telecommunity.com> At 12:47 PM 4/19/2006 -0500, Ian Bicking wrote: >Phillip -- any thoughts on this? If this went in, this would allow me >to start splitting up and simplifying some commands, as outlined in: >http://blog.ianbicking.org/paster-and-scripts.html > >Ian Bicking wrote: > > I'd like to get rid of the "paster" script, moving some of what it does > > into setup.py or elsewhere. > > > > To a degree that is possible, but I think requires some additions to > > setuptools: > > > > * An entry point group that is not globally looked up, like > > distutils.commands. This would be a set of commands the package > > provides only for itself. I expect it to look like: > > > > [distutils.commands.local] > > sql_record = sqlobject.manage.distextension:sql_record > > > > Exactly like what we have currently, just not looked up globally. > > > > * Additionally, or probably even better, something that enumerates > > locations for commands. E.g.: > > > > extra_commands=['SQLObject'] You can pass a 'cmdclass' argument to setup() to accomplish these things. Normally, it's expected to be a dictionary, but as long as you give it an object that implements 'in', 'get()', 'keys()' and setitem/getitem operations, you should be good to go. (The distutils and setuptools both cache their own lookups by assigning items back into the cmdclass object.) If you come up with a nice "command map" object that lets you do something like: setup( cmdclass = CommandMap( # configuration stuff here ), ... ) then it would make a nice addition to setuptools for people that need it. > >A project could list itself to provide its own custom commands; > > I think that won't be too circular. That's only 'cause you haven't tried it yet. Setuptools has to do this, sort of, and trust me, it's a bitch. :) > > Typically commands you provide for > > yourself or someone else are different -- e.g., the SQLObject commands > > don't apply to SQLObject itself. That doesn't matter; the problem is that until your egg_info is generated, you don't have a pkg_resources.Distribution in existence to provide the entry points. > > * Everything that can be done on a deployed egg will probably go in > > app/egg-specific command-line scripts, and maybe I'll make a little > > framework to make these easier to write, but that's entirely an > > implementation detail. I actually do want "nest" to provide a nice mini-framework for writing command-line apps as functions -- sort of a stripped-down cross between peak.running.commands and peak.running.options. One key feature will be that the commands have to accept an argument that's a logger for them to use, and the verbosity of that logger will be managed by the caller/framework, not the function. (Meaning that verbosity options will be handled by the framework when running via the command line.) Although perhaps I should just write this hypothetical framework as a separate package... but I want nest to be a part of the core setuptools distribution. I guess if you're going to depend on the command line framework, you might as well depend on setuptools, since you'll need to have setuptools in order to be able to depend on the hypothetical command line package. :) (Confusing, isn't it?) From ianb at colorstudy.com Wed Apr 19 22:14:06 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 19 Apr 2006 15:14:06 -0500 Subject: [Distutils] Further extending of setup.py In-Reply-To: <5.1.1.6.0.20060419152353.01e3b3c0@mail.telecommunity.com> References: <4415B40D.4010601@colorstudy.com> <4415B40D.4010601@colorstudy.com> <5.1.1.6.0.20060419152353.01e3b3c0@mail.telecommunity.com> Message-ID: <44469A0E.5020704@colorstudy.com> Phillip J. Eby wrote: > At 12:47 PM 4/19/2006 -0500, Ian Bicking wrote: > >> Phillip -- any thoughts on this? If this went in, this would allow me >> to start splitting up and simplifying some commands, as outlined in: >> http://blog.ianbicking.org/paster-and-scripts.html >> >> Ian Bicking wrote: >> > I'd like to get rid of the "paster" script, moving some of what it does >> > into setup.py or elsewhere. >> > >> > To a degree that is possible, but I think requires some additions to >> > setuptools: >> > >> > * An entry point group that is not globally looked up, like >> > distutils.commands. This would be a set of commands the package >> > provides only for itself. I expect it to look like: >> > >> > [distutils.commands.local] >> > sql_record = sqlobject.manage.distextension:sql_record >> > >> > Exactly like what we have currently, just not looked up globally. >> > >> > * Additionally, or probably even better, something that enumerates >> > locations for commands. E.g.: >> > >> > extra_commands=['SQLObject'] > > > You can pass a 'cmdclass' argument to setup() to accomplish these > things. Normally, it's expected to be a dictionary, but as long as you > give it an object that implements 'in', 'get()', 'keys()' and > setitem/getitem operations, you should be good to go. (The distutils > and setuptools both cache their own lookups by assigning items back into > the cmdclass object.) It looks like, if I do this, it will also have to implement the current default behavior of finding distutils.commands providers. Which isn't a big deal as long as that doesn't change, I guess. Though cmdclass won't get a reference to the setuptools.dist.Distribution object, so some things might not be possible. > If you come up with a nice "command map" object that lets you do > something like: > > setup( > cmdclass = CommandMap( > # configuration stuff here > ), > ... > ) > > then it would make a nice addition to setuptools for people that need it. OK, I'll look into that. I'm guessing it'll be something like CommandMap([eggs]), and maybe some other options, though I can't think of any at the moment. Maybe also reading .egg-info/command_map.txt if not given any list of eggs -- I'd like to keep options for putting metadata into .egg-info/*.txt files instead of setup.py. >> >A project could list itself to provide its own custom commands; >> > I think that won't be too circular. > > > That's only 'cause you haven't tried it yet. Setuptools has to do this, > sort of, and trust me, it's a bitch. :) Yeah... it probably isn't important anyway, I can't think of any cases where I'd really want to use this. >> > * Everything that can be done on a deployed egg will probably go in >> > app/egg-specific command-line scripts, and maybe I'll make a little >> > framework to make these easier to write, but that's entirely an >> > implementation detail. > > > I actually do want "nest" to provide a nice mini-framework for writing > command-line apps as functions -- sort of a stripped-down cross between > peak.running.commands and peak.running.options. One key feature will be > that the commands have to accept an argument that's a logger for them to > use, and the verbosity of that logger will be managed by the > caller/framework, not the function. (Meaning that verbosity options > will be handled by the framework when running via the command line.) In my code I do a lot of specific testing of verbosity, to try to avoid redudancy. For instance: def ensure_dir(dir): if verbosity >= 2: print 'Trying to make directory %s' % dir if os.path.exists(dir): if verbosity >= 2: print ' (directory already exists)' elif verbosity >= 1: print 'Directory %s already exists' % dir It's a little annoying to write this, but log-style output is fairly crappy; a better compromise might be: def ensure_dir(dir): log.debug('Trying to make directory %s' % dir) if os.path.exists(dir): log.debug(' (directory already exists)') log.message((log.INFO, log.DEBUG), 'Directory %s already exists' % dir) I think the features for a simple two-level command framework would be fairly simple: * Register commands somehow (entry points, I suppose). Commands have a name and aliases. * Commands can be called to run them, with all arguments after the command name. * Commands have a help() method or something like that, that returns a string help message. Probably two forms -- a long-form help, and short-form. Maybe only the short form? * The framework provides "cmd -h", "cmd help", "cmd -h subcommand", and "cmd help subcommand". The last two might just call "cmd subcommand -h" internally. * I suppose the framework needs to be given some global help message. * It should be possible to hide commands or aliases from help. E.g., commands that represent an internal interface, or aliases that are only provided for backward compatibility. * Maybe a standard way to activate specific eggs, e.g., "cmd --require='Foo==1.0' ...". Maybe also a way of controlling the PYTHONPATH, since setting environmental variables is not equally easy in all environments. Everything else should be possible to build from there with other sub-frameworks. Self-referencing entry point loading would be nice. Then, for instance, you could do: entry_points=""" [whatever.this.group.is] sql-setup = sqlobject.commands:setup_command """ If setup_command is called with the package's Distribution object, then it can read config files or whatever from the egg, and you don't have to create little bits of glue all over the place. Hmm... putting information into the entry point name is problematic here, because SQLObject would give a set of commands which would be a bit tedious to enumerate, and it would be better if you could introduce the whole set of commands with one entry point. Also, it doesn't resolve the problem of aliases. Though it does give you an opportunity to resolve naming conflicts by renaming a command. > Although perhaps I should just write this hypothetical framework as a > separate package... but I want nest to be a part of the core setuptools > distribution. I guess if you're going to depend on the command line > framework, you might as well depend on setuptools, since you'll need to > have setuptools in order to be able to depend on the hypothetical > command line package. :) (Confusing, isn't it?) Would setuptools stop using the distutils command framework? -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Wed Apr 19 22:52:13 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 19 Apr 2006 16:52:13 -0400 Subject: [Distutils] Further extending of setup.py In-Reply-To: <44469A0E.5020704@colorstudy.com> References: <5.1.1.6.0.20060419152353.01e3b3c0@mail.telecommunity.com> <4415B40D.4010601@colorstudy.com> <4415B40D.4010601@colorstudy.com> <5.1.1.6.0.20060419152353.01e3b3c0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060419164334.040a13a8@mail.telecommunity.com> At 03:14 PM 4/19/2006 -0500, Ian Bicking wrote: >It looks like, if I do this, it will also have to implement the current >default behavior of finding distutils.commands providers. No it won't. You should just look up things that you want to provide; setuptools and distutils will do their own thing if your __contains__ says False or your get() returns the default when they ask for a command class. They will then call your __setitem__ to cache whatever they find. >>If you come up with a nice "command map" object that lets you do >>something like: >> setup( >> cmdclass = CommandMap( >> # configuration stuff here >> ), >> ... >> ) >>then it would make a nice addition to setuptools for people that need it. > >OK, I'll look into that. I'm guessing it'll be something like >CommandMap([eggs]), and maybe some other options, though I can't think of >any at the moment. Maybe also reading .egg-info/command_map.txt if not >given any list of eggs -- I'd like to keep options for putting metadata >into .egg-info/*.txt files instead of setup.py. Yecch. When I first started doing egg stuff for setuptools, I thought it'd be nice to just edit files in .egg-info, and then I quickly got sick of it. IMO, setup.py should be a one-stop-shop. It's easier to explain to setup authors, too. I can understand not wanting to go to the trouble of implementing an egg_info "writer" plugin, but trying to show people how to actually use these things, I think your users are a lot better off if all they have to do is pass in a setup keyword or two. >I think the features for a simple two-level command framework would be >fairly simple: > >* Register commands somehow (entry points, I suppose). Commands have a >name and aliases. >* Commands can be called to run them, with all arguments after the command >name. >* Commands have a help() method or something like that, that returns a >string help message. Probably two forms -- a long-form help, and >short-form. Maybe only the short form? >* The framework provides "cmd -h", "cmd help", "cmd -h subcommand", and >"cmd help subcommand". The last two might just call "cmd subcommand -h" >internally. >* I suppose the framework needs to be given some global help message. >* It should be possible to hide commands or aliases from help. E.g., >commands that represent an internal interface, or aliases that are only >provided for backward compatibility. >* Maybe a standard way to activate specific eggs, e.g., "cmd >--require='Foo==1.0' ...". Maybe also a way of controlling the >PYTHONPATH, since setting environmental variables is not equally easy in >all environments. > >Everything else should be possible to build from there with other >sub-frameworks. You left out configuration. >Self-referencing entry point loading would be nice. Then, for instance, >you could do: > > entry_points=""" > [whatever.this.group.is] > sql-setup = sqlobject.commands:setup_command > """ > >If setup_command is called with the package's Distribution object, then it >can read config files or whatever from the egg, and you don't have to >create little bits of glue all over the place. > >Hmm... putting information into the entry point name is problematic here, >because SQLObject would give a set of commands which would be a bit >tedious to enumerate, and it would be better if you could introduce the >whole set of commands with one entry point. Also, it doesn't resolve the >problem of aliases. Though it does give you an opportunity to resolve >naming conflicts by renaming a command. Sorry, I don't really understand what you're trying to do here. Are you talking about adding something to "nest", or...? >Would setuptools stop using the distutils command framework? No - it's necessary for compatibility, and to handle the intricate interdependencies where some commands depend on values computed by other commands. Nest has much simpler needs that would be better served by less implicit coupling. From ianb at colorstudy.com Wed Apr 19 23:12:31 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 19 Apr 2006 16:12:31 -0500 Subject: [Distutils] Further extending of setup.py In-Reply-To: <5.1.1.6.0.20060419164334.040a13a8@mail.telecommunity.com> References: <5.1.1.6.0.20060419152353.01e3b3c0@mail.telecommunity.com> <4415B40D.4010601@colorstudy.com> <4415B40D.4010601@colorstudy.com> <5.1.1.6.0.20060419152353.01e3b3c0@mail.telecommunity.com> <5.1.1.6.0.20060419164334.040a13a8@mail.telecommunity.com> Message-ID: <4446A7BF.9080104@colorstudy.com> Phillip J. Eby wrote: > At 03:14 PM 4/19/2006 -0500, Ian Bicking wrote: > >> It looks like, if I do this, it will also have to implement the >> current default behavior of finding distutils.commands providers. > > > No it won't. You should just look up things that you want to provide; > setuptools and distutils will do their own thing if your __contains__ > says False or your get() returns the default when they ask for a command > class. They will then call your __setitem__ to cache whatever they find. OK. >>> If you come up with a nice "command map" object that lets you do >>> something like: >>> setup( >>> cmdclass = CommandMap( >>> # configuration stuff here >>> ), >>> ... >>> ) >>> then it would make a nice addition to setuptools for people that need >>> it. >> >> >> OK, I'll look into that. I'm guessing it'll be something like >> CommandMap([eggs]), and maybe some other options, though I can't think >> of any at the moment. Maybe also reading .egg-info/command_map.txt if >> not given any list of eggs -- I'd like to keep options for putting >> metadata into .egg-info/*.txt files instead of setup.py. > > > Yecch. When I first started doing egg stuff for setuptools, I thought > it'd be nice to just edit files in .egg-info, and then I quickly got > sick of it. IMO, setup.py should be a one-stop-shop. It's easier to > explain to setup authors, too. I can understand not wanting to go to > the trouble of implementing an egg_info "writer" plugin, but trying to > show people how to actually use these things, I think your users are a > lot better off if all they have to do is pass in a setup keyword or two. Putting things in setup.py means that those things are pretty opaque for tools. For instance, when generating packages setup.py is a point of contention when you are creating a package that uses more than one framework. If the generator was able to handle the specific files like entry_points.txt with knowledge of what those files meant, most contention could be avoided. requires.txt is another thing that could be usefully manipulated with tools; for instance, if you had a tool that did commits to two projects at once, and updated the second project to require on the after-commit version of the first project. So it's not the writer plugin that bothers me at all, though I haven't looked at the details of that. It's that Python source is opaque. Anyway, another thing occurred to me. In addition to providing plain egg files, I'd like to allow egg+entry_point name to be a possible specifier. It would be nice if we had a convention for how to specify both at once. I'm using spec#entry_point_name in Paste, which works fine for me. >> I think the features for a simple two-level command framework would be >> fairly simple: >> >> * Register commands somehow (entry points, I suppose). Commands have >> a name and aliases. >> * Commands can be called to run them, with all arguments after the >> command name. >> * Commands have a help() method or something like that, that returns a >> string help message. Probably two forms -- a long-form help, and >> short-form. Maybe only the short form? >> * The framework provides "cmd -h", "cmd help", "cmd -h subcommand", >> and "cmd help subcommand". The last two might just call "cmd >> subcommand -h" internally. >> * I suppose the framework needs to be given some global help message. >> * It should be possible to hide commands or aliases from help. E.g., >> commands that represent an internal interface, or aliases that are >> only provided for backward compatibility. >> * Maybe a standard way to activate specific eggs, e.g., "cmd >> --require='Foo==1.0' ...". Maybe also a way of controlling the >> PYTHONPATH, since setting environmental variables is not equally easy >> in all environments. >> >> Everything else should be possible to build from there with other >> sub-frameworks. > > > You left out configuration. Configuration...? I don't see where that fits in? Or, at least, I figured that configuration would be figured out by each subcommand, and they'd share command-line options for that by convention. I did forget --version, though, which should be implemented in the parent command. >> Self-referencing entry point loading would be nice. Then, for >> instance, you could do: >> >> entry_points=""" >> [whatever.this.group.is] >> sql-setup = sqlobject.commands:setup_command >> """ >> >> If setup_command is called with the package's Distribution object, >> then it can read config files or whatever from the egg, and you don't >> have to create little bits of glue all over the place. >> >> Hmm... putting information into the entry point name is problematic >> here, because SQLObject would give a set of commands which would be a >> bit tedious to enumerate, and it would be better if you could >> introduce the whole set of commands with one entry point. Also, it >> doesn't resolve the problem of aliases. Though it does give you an >> opportunity to resolve naming conflicts by renaming a command. > > > Sorry, I don't really understand what you're trying to do here. Are you > talking about adding something to "nest", or...? Well, to be more specific, here's what I'd like for SQLObject: * A setup.py command, like "setup.py sql-record". This would take the current database schema as defined by the source, and write that out somewhere into the project's source (kept in the source as a historical record of all possible schemas the app has had.) * The setup.py command has to figure out where to find SQLObject classes in the project, by some configuration. I'd rather that configuration not go in setup.cfg, because it is used later... * The SQLObject-using-application would get several subcommands: - create - drop - show-sql - status - upgrade * These commands would read some configuration from the original egg, to find the history of schemas, and to find the module that holds the current SQLObject classes. So, particularly in the second case, adding SQLObject to your application introduces a set of commands, and those commands can all run off the same configuration, which they also share with the setup.py command. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Wed Apr 19 23:18:16 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 19 Apr 2006 16:18:16 -0500 Subject: [Distutils] setuptools and site.py Message-ID: <4446A918.9000706@colorstudy.com> I'm trying to resolve a few issues in working-env.py (http://svn.colorstudy.com/home/ianb/working-env.py) and I'm running into some conflicts with setuptools. working-env.py basically works by creating another site.py, which doesn't add site-packages and other directories that I don't care for. Without setuptools this works pretty well. But because of changes in setuptools, added right around the time I wrote working-env.py, setuptools now wants to write its own site.py and doesn't like mine. One option is to integrate setuptools' changes into my own customized site.py, and then add __boot to the top to fake setuptools out. I think that would be bothersome. Another option would be if setuptools looked for something ahead of site. Like, say, siteoverride. Then I could put my siteoverride in place instead of site.py, let setuptools have site.py, and avoid the system site.py entirely. Also, and this is related to a request Jim made a while ago, I'm monkeypatching setuptools currently to change the way it creates scripts, and it would be nice if this wasn't necessary. Or was easier. The monkeypatch is basically changing the script generated so that it sets sys.path before importing site. Here's what I'm doing (in setuptools/__init__): import setuptools.command.easy_install as easy_install def get_script_header(script_text, executable=easy_install.sys_executable): from distutils.command.build_scripts import first_line_re first, rest = (script_text+'\n').split('\n',1) match = first_line_re.match(first) options = '' if match: script_text = rest options = match.group(1) or '' if options: options = ' '+options if options.find('-S') == -1: options += ' -S' shbang = "#!%(executable)s%(options)s\n" % locals() shbang += ("import sys, os\n" "join, dirname = os.path.join, os.path.dirname\n" "lib_dir = join(dirname(dirname(__file__)), 'lib', 'python%s.%s' % tuple(sys.version_info[:2]))\n" "sys.path.insert(0, lib_dir)\n" "import site\n") return shbang easy_install.get_script_header = get_script_header -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From cwmoad at gmail.com Thu Apr 20 20:19:21 2006 From: cwmoad at gmail.com (Charlie Moad) Date: Thu, 20 Apr 2006 14:19:21 -0400 Subject: [Distutils] Upgrading eggs from custom url Message-ID: <6382066a0604201119pe1ad9d3p705e3bb1b626a421@mail.gmail.com> I am using setuptools to maintain some plugins I wrote for a python based protein viewer. I would like to have another plugin that simply checks for updates for the currently installed plugins. Code may help, so basically I would like to run: from setuptools.command import easy_install easy_install.main('-U MyPlugin'.split()) This would work fine if I registered my plugins on Pypi, but just trust me that I shouldn't be pushing this junk up there yet. I saw the "dependency_links" option, and that is close to what I want. However, I would also like easy_install to check for an update to "MyPlugin" there as well. When issuing an "--upgrade" option, can the "download_url" be checked automatically, or is there an existing way I can get a handle on that "download_url" variable to pass to easy_install? Any advice would be appreciated. Thanks, Charlie From bear42 at code-bear.com Fri Apr 21 05:45:51 2006 From: bear42 at code-bear.com (bear) Date: Thu, 20 Apr 2006 23:45:51 -0400 Subject: [Distutils] Upgrading eggs from custom url In-Reply-To: <6382066a0604201119pe1ad9d3p705e3bb1b626a421@mail.gmail.com> References: <6382066a0604201119pe1ad9d3p705e3bb1b626a421@mail.gmail.com> Message-ID: <4448556F.9040809@code-bear.com> Charlie Moad wrote: > When issuing an "--upgrade" option, can the "download_url" be checked > automatically, or is there an existing way I can get a handle on that > "download_url" variable to pass to easy_install? Any advice would be > appreciated. > Check out the --find-links option: easy_install -f http://some_url eggname http://peak.telecommunity.com/DevCenter/EasyInstall#command-line-options From cwmoad at gmail.com Fri Apr 21 13:47:07 2006 From: cwmoad at gmail.com (Charlie Moad) Date: Fri, 21 Apr 2006 07:47:07 -0400 Subject: [Distutils] Upgrading eggs from custom url In-Reply-To: <4448556F.9040809@code-bear.com> References: <6382066a0604201119pe1ad9d3p705e3bb1b626a421@mail.gmail.com> <4448556F.9040809@code-bear.com> Message-ID: <6382066a0604210447q9f34098y8f6231d936f1d7ac@mail.gmail.com> On 4/20/06, bear wrote: > > Charlie Moad wrote: > > When issuing an "--upgrade" option, can the "download_url" be checked > > automatically, or is there an existing way I can get a handle on that > > "download_url" variable to pass to easy_install? Any advice would be > > appreciated. > > > Check out the --find-links option: > > easy_install -f http://some_url eggname > > http://peak.telecommunity.com/DevCenter/EasyInstall#command-line-options I know I can pass that, but it is a matter of introspecting an installed egg to find its download_url. From pje at telecommunity.com Fri Apr 21 18:28:26 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 21 Apr 2006 12:28:26 -0400 Subject: [Distutils] Upgrading eggs from custom url In-Reply-To: <6382066a0604210447q9f34098y8f6231d936f1d7ac@mail.gmail.com > References: <4448556F.9040809@code-bear.com> <6382066a0604201119pe1ad9d3p705e3bb1b626a421@mail.gmail.com> <4448556F.9040809@code-bear.com> Message-ID: <5.1.1.6.0.20060421122543.04439340@mail.telecommunity.com> At 07:47 AM 4/21/2006 -0400, Charlie Moad wrote: >On 4/20/06, bear wrote: > > > > Charlie Moad wrote: > > > When issuing an "--upgrade" option, can the "download_url" be checked > > > automatically, or is there an existing way I can get a handle on that > > > "download_url" variable to pass to easy_install? Any advice would be > > > appreciated. > > > > > Check out the --find-links option: > > > > easy_install -f http://some_url eggname > > > > http://peak.telecommunity.com/DevCenter/EasyInstall#command-line-options > >I know I can pass that, but it is a matter of introspecting an >installed egg to find its download_url. Eggs don't carry download URLs, they only carry dependency links, and only if the author specified them. If what you're saying is that you want to be able to check the page you originally got the egg from for updates, there isn't currently any way to do that, except to effectively maintain your own package index. It would probably be a better idea to gently encourage the package's author to run "setup.py register" on a regular basis to keep their PyPI listing up-to-date, since that helps everybody else, too. From philippe at fluendo.com Mon Apr 24 17:46:21 2006 From: philippe at fluendo.com (Philippe Normand) Date: Mon, 24 Apr 2006 17:46:21 +0200 (CEST) Subject: [Distutils] Problem importing pylirc egg Message-ID: Hi, i've easy_installed pylirc from sourceforge as an Egg. It's correctly compiled/installed on my system. But i can't import it, though it's found by pkg_resources: >>> import pylirc Traceback (most recent call last): File "", line 1, in ? ImportError: No module named pylirc >>> import pkg_resources >>> pkg_resources.require('pylirc') [pylirc 0.0.5 (/home/phil/lib/python2.4/pylirc-0.0.5-py2.4-linux-i686.egg)] What am i supposed to do with that Requirement object so as to get pylirc module in my namespace ? Philippe From ianb at colorstudy.com Mon Apr 24 18:24:24 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 24 Apr 2006 11:24:24 -0500 Subject: [Distutils] Problem importing pylirc egg In-Reply-To: References: Message-ID: <444CFBB8.9020700@colorstudy.com> Philippe Normand wrote: > Hi, > > i've easy_installed pylirc from sourceforge as an Egg. It's correctly > compiled/installed on my system. But i can't import it, though it's > found by pkg_resources: > > >>>>import pylirc > > Traceback (most recent call last): > File "", line 1, in ? > ImportError: No module named pylirc > >>>>import pkg_resources >>>>pkg_resources.require('pylirc') > > [pylirc 0.0.5 > (/home/phil/lib/python2.4/pylirc-0.0.5-py2.4-linux-i686.egg)] > > What am i supposed to do with that Requirement object so as to get > pylirc module in my namespace ? You have to require and then import: import pkg_resources pkg_resources.require('pylirc') import pylirc Also, if you list pylirc as a requirement in your setup.py of your pylirc-using-project and then require your own package, this will make sure pylirc is also required (along with any other requirements you put in setup.py). -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From philippe at fluendo.com Mon Apr 24 20:37:22 2006 From: philippe at fluendo.com (Philippe Normand) Date: Mon, 24 Apr 2006 20:37:22 +0200 (CEST) Subject: [Distutils] Problem importing pylirc egg In-Reply-To: <444CFBB8.9020700@colorstudy.com> Message-ID: <3NFDAmfW.1145903842.4045960.phil@localhost> Le 24/4/2006, "Ian Bicking" a ?crit: >Philippe Normand wrote: >> Hi, >> >> i've easy_installed pylirc from sourceforge as an Egg. It's correctly >> compiled/installed on my system. But i can't import it, though it's >> found by pkg_resources: >> >> >>>>>import pylirc >> >> Traceback (most recent call last): >> File "", line 1, in ? >> ImportError: No module named pylirc >> >>>>>import pkg_resources >>>>>pkg_resources.require('pylirc') >> >> [pylirc 0.0.5 >> (/home/phil/lib/python2.4/pylirc-0.0.5-py2.4-linux-i686.egg)] >> >> What am i supposed to do with that Requirement object so as to get >> pylirc module in my namespace ? > >You have to require and then import: > >import pkg_resources >pkg_resources.require('pylirc') >import pylirc > Even after requiring the package, i get an ImportError ;( >>> import pkg_resources >>> pkg_resources.require('pylirc') [pylirc 0.0.5 (/home/phil/lib/python2.4/pylirc-0.0.5-py2.4-linux-i686.egg)] >>> import pylirc Traceback (most recent call last): File "", line 1, in ? ImportError: No module named pylirc >>> >Also, if you list pylirc as a requirement in your setup.py of your >pylirc-using-project and then require your own package, this will make >sure pylirc is also required (along with any other requirements you put >in setup.py). > Ok, that's cool. Philippe From ianb at colorstudy.com Mon Apr 24 20:43:09 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 24 Apr 2006 13:43:09 -0500 Subject: [Distutils] Problem importing pylirc egg In-Reply-To: <3NFDAmfW.1145903842.4045960.phil@localhost> References: <3NFDAmfW.1145903842.4045960.phil@localhost> Message-ID: <444D1C3D.5080506@colorstudy.com> Philippe Normand wrote: > Le 24/4/2006, "Ian Bicking" a ?crit: > > >>Philippe Normand wrote: >> >>>Hi, >>> >>>i've easy_installed pylirc from sourceforge as an Egg. It's correctly >>>compiled/installed on my system. But i can't import it, though it's >>>found by pkg_resources: >>> >>> >>> >>>>>>import pylirc >>> >>>Traceback (most recent call last): >>>File "", line 1, in ? >>>ImportError: No module named pylirc >>> >>> >>>>>>import pkg_resources >>>>>>pkg_resources.require('pylirc') >>> >>>[pylirc 0.0.5 >>>(/home/phil/lib/python2.4/pylirc-0.0.5-py2.4-linux-i686.egg)] >>> >>>What am i supposed to do with that Requirement object so as to get >>>pylirc module in my namespace ? >> >>You have to require and then import: >> >>import pkg_resources >>pkg_resources.require('pylirc') >>import pylirc >> > > > Even after requiring the package, i get an ImportError ;( > > >>>>import pkg_resources >>>>pkg_resources.require('pylirc') > > [pylirc 0.0.5 > (/home/phil/lib/python2.4/pylirc-0.0.5-py2.4-linux-i686.egg)] What's in /home/phil/lib/python2.4/pylirc-0.0.5-py2.4-linux-i686.egg ? What does your sys.path look like after you do the require()? That path should be on your sys.path after require(), and it should contain pylirc/__init__.py (or be a zip file that includes pylirc/__init__.py). -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Mon Apr 24 21:03:05 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 24 Apr 2006 15:03:05 -0400 Subject: [Distutils] Problem importing pylirc egg In-Reply-To: Message-ID: <5.1.1.6.0.20060424150104.03757448@mail.telecommunity.com> At 05:46 PM 4/24/2006 +0200, Philippe Normand wrote: >Hi, > >i've easy_installed pylirc from sourceforge as an Egg. It's correctly >compiled/installed on my system. But i can't import it, though it's >found by pkg_resources: > > >>> import pylirc >Traceback (most recent call last): >File "", line 1, in ? >ImportError: No module named pylirc > >>> import pkg_resources > >>> pkg_resources.require('pylirc') >[pylirc 0.0.5 >(/home/phil/lib/python2.4/pylirc-0.0.5-py2.4-linux-i686.egg)] > >What am i supposed to do with that Requirement object so as to get >pylirc module in my namespace ? Try "import pylircmodule" - it looks like this may be a bug in building compressed eggs; if an extension module in a zipfile ends with 'module', the bdist_egg command probably isn't stripping the word 'module' from the loader name. If that works, try reinstalling the egg using -Z (or --always-unzip), and see if you can just use "import pylirc". In that case, it is indeed a bug in bdist_egg. From pje at telecommunity.com Mon Apr 24 21:03:50 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 24 Apr 2006 15:03:50 -0400 Subject: [Distutils] Problem importing pylirc egg In-Reply-To: <444D1C3D.5080506@colorstudy.com> References: <3NFDAmfW.1145903842.4045960.phil@localhost> <3NFDAmfW.1145903842.4045960.phil@localhost> Message-ID: <5.1.1.6.0.20060424150311.038fc780@mail.telecommunity.com> At 01:43 PM 4/24/2006 -0500, Ian Bicking wrote: > That path >should be on your sys.path after require(), and it should contain >pylirc/__init__.py (or be a zip file that includes pylirc/__init__.py). Actually, pylirc should be a single C extension module called 'pylircmodule.so'. From philippe at fluendo.com Mon Apr 24 21:22:43 2006 From: philippe at fluendo.com (Philippe Normand) Date: Mon, 24 Apr 2006 21:22:43 +0200 (CEST) Subject: [Distutils] Problem importing pylirc egg In-Reply-To: <444D1C3D.5080506@colorstudy.com> Message-ID: Le 24/4/2006, "Ian Bicking" a ?crit: > >What's in /home/phil/lib/python2.4/pylirc-0.0.5-py2.4-linux-i686.egg ? >What does your sys.path look like after you do the require()? That path >should be on your sys.path after require(), and it should contain >pylirc/__init__.py (or be a zip file that includes pylirc/__init__.py). > sys.path contains full path to the egg, even without requiring pylirc. The .egg is a zip file containing: Archive: pylirc-0.0.5-py2.4-linux-i686.egg inflating: pylircmodule.so inflating: pylircmodule.py inflating: pylircmodule.pyc inflating: EGG-INFO/PKG-INFO inflating: EGG-INFO/top_level.txt inflating: EGG-INFO/SOURCES.txt inflating: EGG-INFO/native_libs.txt inflating: EGG-INFO/zip-safe pylircmodule.py isn't part of the "official" pylirc distribution. Seems like it was generated by setuptools. But if there's a .so with the same name, it will confuse things, no ? That generated file contains: def __bootstrap__(): global __bootstrap__, __loader__, __file__ import sys, pkg_resources, imp __file__ = pkg_resources.resource_filename(__name__,'pylircmodule.so') del __bootstrap__, __loader__ imp.load_dynamic(__name__,__file__) __bootstrap__() Philippe From philippe at fluendo.com Mon Apr 24 21:30:04 2006 From: philippe at fluendo.com (Philippe Normand) Date: Mon, 24 Apr 2006 21:30:04 +0200 (CEST) Subject: [Distutils] Problem importing pylirc egg In-Reply-To: <5.1.1.6.0.20060424150104.03757448@mail.telecommunity.com> Message-ID: Le 24/4/2006, "Phillip J. Eby" a ?crit: > >Try "import pylircmodule" - it looks like this may be a bug in building >compressed eggs; if an extension module in a zipfile ends with 'module', >the bdist_egg command probably isn't stripping the word 'module' from the >loader name. > >If that works, try reinstalling the egg using -Z (or --always-unzip), and >see if you can just use "import pylirc". In that case, it is indeed a bug >in bdist_egg. > Gotcha ! :) I tried importing pylircmodule: >>> import pylircmodule Traceback (most recent call last): File "", line 1, in ? File "build/bdist.linux-i686/egg/pylircmodule.py", line 7, in ? File "build/bdist.linux-i686/egg/pylircmodule.py", line 6, in __bootstrap__ ImportError: dynamic module does not define init function (initpylircmodule) Re-installed the Egg as you advised, renammed pylircmodule.py to pylirc.py: >>> import pylirc >>> pylirc.__file__ '/home/phil/lib/python2.4/pylirc-0.0.5-py2.4-linux-i686.egg/pylircmodule.so' So it seems to be a bug ;) Philippe From pje at telecommunity.com Mon Apr 24 21:34:17 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 24 Apr 2006 15:34:17 -0400 Subject: [Distutils] Problem importing pylirc egg In-Reply-To: References: <444D1C3D.5080506@colorstudy.com> Message-ID: <5.1.1.6.0.20060424153219.0427a5d0@mail.telecommunity.com> At 09:22 PM 4/24/2006 +0200, Philippe Normand wrote: >Le 24/4/2006, "Ian Bicking" a ?crit: > > > >What's in /home/phil/lib/python2.4/pylirc-0.0.5-py2.4-linux-i686.egg ? > >What does your sys.path look like after you do the require()? That path > >should be on your sys.path after require(), and it should contain > >pylirc/__init__.py (or be a zip file that includes pylirc/__init__.py). > > > >sys.path contains full path to the egg, even without requiring pylirc. > >The .egg is a zip file containing: > >Archive: pylirc-0.0.5-py2.4-linux-i686.egg > inflating: pylircmodule.so > inflating: pylircmodule.py > inflating: pylircmodule.pyc > inflating: EGG-INFO/PKG-INFO > inflating: EGG-INFO/top_level.txt > inflating: EGG-INFO/SOURCES.txt > inflating: EGG-INFO/native_libs.txt > inflating: EGG-INFO/zip-safe > >pylircmodule.py isn't part of the "official" pylirc distribution. >Seems like it was generated by setuptools. But if there's a .so with >the same name, it will confuse things, no ? No. Python doesn't import .so files in a zipfile (such as a .egg file), and when the file is extracted, .so files take precedence over .py files, so the .py is ignored in that case. The problem here is that Python's import machinery ignores the word 'module' in the .so file name, but not in the .py file name. It looks like I need to change bdist_egg to handle this, and possibly some other code in setuptools as well. In the meantime, you should be able to 'import pylircmodule' and have it work, or re-install it using -Z (--always-unzip) so that it is installed as a directory instead of a zipfile. From pje at telecommunity.com Mon Apr 24 21:40:36 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 24 Apr 2006 15:40:36 -0400 Subject: [Distutils] Problem importing pylirc egg In-Reply-To: References: <5.1.1.6.0.20060424150104.03757448@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060424154004.021715e0@mail.telecommunity.com> At 09:30 PM 4/24/2006 +0200, Philippe Normand wrote: >Re-installed the Egg as you advised, renammed pylircmodule.py to >pylirc.py: Actually, it doesn't matter if you rename it or not, it should work when extracted. It's definitely a bug, and added to my list to fix before the 0.6b1 release. From pje at telecommunity.com Wed Apr 26 00:49:43 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 25 Apr 2006 18:49:43 -0400 Subject: [Distutils] Internal documentation for egg formats now available Message-ID: <5.1.1.6.0.20060425184505.0361ca30@mail.telecommunity.com> To satisfy requests on Python-Dev, I've just added some internal "developer's" documentation to setuptools. You don't need to read it unless you are working on setuptools itself, want to assist in troubleshooting a problem with your installation, or want to understand how setuptools may affect your packaging of libraries that use it (i.e., you're building RPMs or debs or what-have-you for packages that use setuptools, or any Python 2.5 package). The docs can be found at http://svn.python.org/projects/sandbox/trunk/setuptools/doc/formats.txt (source text), or http://peak.telecommunity.com/DevCenter/EggFormats (HTML-formatted version). Included are summary descriptions of the formats of all of the standard metadata produced by setuptools, along with pointers to the existing manuals that describe the syntax used for representing requirements, entry points, etc. as text. The .egg, .egg-info, and .egg-link formats and layouts are also specified, along with the filename syntax used to embed project/version/Python version/platform metadata. Last, but not least, there are detailed explanations of how resources (such as C extensions) are extracted on-the-fly and cached, how C extensions get imported from zipfiles, and how EasyInstall works around the limitations of Python's default sys.path initialization. Please remember, however, that a lot of this stuff is purely internal details and subject to change. Most of the information is also available to you directly via the "pkg_resources" API, so you probably don't care about the actual file formats in such cases anyway. From joe.gregorio at gmail.com Wed Apr 26 21:39:22 2006 From: joe.gregorio at gmail.com (Joe Gregorio) Date: Wed, 26 Apr 2006 15:39:22 -0400 Subject: [Distutils] setuptools, PYTHONPATH and suExec Message-ID: <3f1451f50604261239q6d887797sbaf539308ef305c9@mail.gmail.com> I recently installed 'paste' via setuptools to a local directory. I don't have root on this system so I had to use the --prefix option. From the command line everything works fine: $ python Python 2.4 (#1, May 19 2005, 08:18:31) [GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-49)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import paste >>> But when I try to import paste from a .CGI script it fails. The problem seems to be PYTHONPATH, or the lack-there-of. I have a PYTHONPATH in my shell and that seems to be the reason paste works from the shell. In the .CGI script I don't have PYTHONPATH and the import of paste fails. I tried manipulating sys.path at runtime from within the CGI, but I believe that is too late and by the time my script starts running the wrong (non- PYTHONPATH) site.py has already been read. I tried adding PYTHONPATH to the environment by adding SetEnv PYTHONPATH /blah to the .htaccess, but I am running under suExec which appears to only pass in white-listed environment variables. Is there a way to get this to work or am I out of luck? Thanks, -joe -- Joe Gregorio http://bitworking.org From ianb at colorstudy.com Wed Apr 26 21:48:11 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 26 Apr 2006 14:48:11 -0500 Subject: [Distutils] setuptools, PYTHONPATH and suExec In-Reply-To: <3f1451f50604261239q6d887797sbaf539308ef305c9@mail.gmail.com> References: <3f1451f50604261239q6d887797sbaf539308ef305c9@mail.gmail.com> Message-ID: <444FCE7B.1060500@colorstudy.com> Joe Gregorio wrote: > Is there a way to get this to work or am I out of luck? You might try workingenv: http://svn.colorstudy.com/home/ianb/workingenv If you use it, scripts installed via setuptools/easy_install will adjust their path automatically. In a CGI script you can do: #!/usr/bin/env python -S import sys sys.path.insert(0, 'path/to/lib/python') import site ... other stuff ... I'm not 100% sure how well that #! line will work on all operating systems; I haven't tested this out much on different systems. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Wed Apr 26 22:01:21 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 26 Apr 2006 16:01:21 -0400 Subject: [Distutils] setuptools, PYTHONPATH and suExec In-Reply-To: <3f1451f50604261239q6d887797sbaf539308ef305c9@mail.gmail.co m> Message-ID: <5.1.1.6.0.20060426155315.0412dd30@mail.telecommunity.com> At 03:39 PM 4/26/2006 -0400, Joe Gregorio wrote: >The problem seems to be PYTHONPATH, or the lack-there-of. >I have a PYTHONPATH in my shell and that seems to >be the reason paste works from the shell. In the .CGI script >I don't have PYTHONPATH and the import of paste fails. > >I tried manipulating sys.path at runtime from within the CGI, >but I believe that is too late >and by the time my script starts running the wrong (non- >PYTHONPATH) site.py has already been read. ... >Is there a way to get this to work or am I out of luck? Try this: #!/bin/sh """:" PYTHONPATH=whatever exec /path/to/python $0 ${1+"$@"} """ The trick here is that Python sees those three lines as a triple-quoted docstring, but the shell sees the first line as a no-op, and then bails out after the 'exec', so it doesn't read the rest of the file. The ${} magic is shell quoting stuff to make sure that all arguments are properly escaped; I don't pretend to understand how it works. :) From pje at telecommunity.com Wed Apr 26 22:09:30 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 26 Apr 2006 16:09:30 -0400 Subject: [Distutils] setuptools, PYTHONPATH and suExec In-Reply-To: <444FCE7B.1060500@colorstudy.com> References: <3f1451f50604261239q6d887797sbaf539308ef305c9@mail.gmail.com> <3f1451f50604261239q6d887797sbaf539308ef305c9@mail.gmail.com> Message-ID: <5.1.1.6.0.20060426160335.04286980@mail.telecommunity.com> At 02:48 PM 4/26/2006 -0500, Ian Bicking wrote: > #!/usr/bin/env python -S > import sys > sys.path.insert(0, 'path/to/lib/python') > import site > ... other stuff ... > >I'm not 100% sure how well that #! line will work on all operating >systems; I haven't tested this out much on different systems. That's a BSDism, it won't work on Linux. But this does: #!/path/to/python -S import os, sys paths = ["wherever"] os.environ["PYTHONPATH"] = os.pathsep.join(paths) sys.path[0:0] = paths import site And this approach doesn't need workingenv either; Joe's plain setuptools arrangement should work with this. I'd have suggested this first, except that I forgot that you could use -S and then import site afterward. From joe.gregorio at gmail.com Wed Apr 26 22:17:29 2006 From: joe.gregorio at gmail.com (Joe Gregorio) Date: Wed, 26 Apr 2006 16:17:29 -0400 Subject: [Distutils] setuptools, PYTHONPATH and suExec In-Reply-To: <5.1.1.6.0.20060426160335.04286980@mail.telecommunity.com> References: <3f1451f50604261239q6d887797sbaf539308ef305c9@mail.gmail.com> <444FCE7B.1060500@colorstudy.com> <5.1.1.6.0.20060426160335.04286980@mail.telecommunity.com> Message-ID: <3f1451f50604261317h7f0fb21bwe1af2717bd1bcd6c@mail.gmail.com> On 4/26/06, Phillip J. Eby wrote: > But this does: > > #!/path/to/python -S > import os, sys > paths = ["wherever"] > os.environ["PYTHONPATH"] = os.pathsep.join(paths) > sys.path[0:0] = paths > import site > > And this approach doesn't need workingenv either; Joe's plain setuptools > arrangement should work with this. > > I'd have suggested this first, except that I forgot that you could use -S > and then import site afterward. That works like a charm, thanks! -joe -- Joe Gregorio http://bitworking.org From philippe at fluendo.com Wed Apr 26 22:35:28 2006 From: philippe at fluendo.com (Philippe Normand) Date: Wed, 26 Apr 2006 22:35:28 +0200 (CEST) Subject: [Distutils] Setuptools / Python2.5 status ? Message-ID: Hi, I've read in PEP 356 that setuptools in the completed features list of Python2.5. Then i've read a mail from Phillip telling setuptools in now in Python trunk : http://mail.python.org/pipermail/distutils-sig/2006-April/006240.html But PEP 356 mentions that bdist_egg won't be part of Python2.5, so i'm a bit confused. If i want to distribute software using Eggs for Python2.5, do the final user or admin still needs to install setuptools himself ? If bdist_egg wouldn't be part of Python2.5, it would be impossible for developers to build Eggs, is that right ? So he'd still need to install setuptools yet ? Ok there are usefull scripts like working-env or virtual-python, but i apprehend them better in the developer context than from the final user point of view. So i'd like to be sure that requiring Python2.5 for the final user would make both final user and developer happy ;-) BTW this is probably off-topic, but does anybody has news about alpha2 ? PEP 356 says it should have been released yesterday and i've seen nothing so far. Philippe From pje at telecommunity.com Wed Apr 26 22:57:55 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 26 Apr 2006 16:57:55 -0400 Subject: [Distutils] Setuptools / Python2.5 status ? In-Reply-To: Message-ID: <5.1.1.6.0.20060426165259.0442cc58@mail.telecommunity.com> At 10:35 PM 4/26/2006 +0200, Philippe Normand wrote: >I've read in PEP 356 that setuptools in the completed features list of >Python2.5. Then i've read a mail from Phillip telling setuptools in now >in Python trunk : >http://mail.python.org/pipermail/distutils-sig/2006-April/006240.html > >But PEP 356 mentions that bdist_egg won't be part of Python2.5, so i'm >a bit confused. Things are in flux at the moment; there were some objections to setuptools' inclusion and for the moment it has been pulled from the 2.5 trunk. There will be a PEP and a re-review for inclusion plans in 2.5. >If i want to distribute software using Eggs for Python2.5, do the final >user or admin still needs to install setuptools himself ? Currently, yes. However, 2.5 is "egg-ready" in various other respects; for example 'pydoc' works correctly with zipped eggs now, and the traceback module should be able to display source code lines for them. Also, packages installed via the distutils under Python 2.5 will include a .egg-info file so that setuptools will be able to detect their presence, and will not reinstall them if a suitable version is already present. >If bdist_egg wouldn't be part of Python2.5, it would be impossible for >developers to build Eggs, is that right ? So he'd still need to install >setuptools yet ? This is correct. At the moment, installation of setuptools is still required. This may be revisited again by 2.5 alpha 3. However, since the addition met with some very strong (if terribly belated) opposition, there will be a PEP first. >BTW this is probably off-topic, but does anybody has news about alpha2 ? >PEP 356 says it should have been released yesterday and i've seen >nothing so far. I believe the SVN freeze for cutting the release is about to begin. I would expect it to be ready by the weekend. From john.pye at student.unsw.edu.au Thu Apr 27 07:38:08 2006 From: john.pye at student.unsw.edu.au (John Pye) Date: Thu, 27 Apr 2006 15:38:08 +1000 Subject: [Distutils] Create 'bdist' installer from already-compiled SWIG DLL/SO Message-ID: <445058C0.505@student.unsw.edu.au> Hi all, I have a C++ library that I recently wrapped to allow it it be used from Python. I have used SCons as my build tool, and I want to continue using that. Now that I have completed my python language bindings, I want to work out a sensible way to create an installable version of this code. I want to use the python 'distutils' as that seems to be preferable (in the case of Python programs) to using something like NSIS. How can I use distutils to package up an *already-built* wrapper .py and .dll file? I don't have any other files that I need to distribute. Everything in the distutils documentation assumes that you will use Python to compile from sources. But the SWIG support in distutils is not complete enough (and anyway, I want to use SCons for the compiling stage). Here is my current effort, but it doesn't handle the 'dll' correctly (and isn't cross-platform, but that's OK): #!/usr/bin/env python from distutils.core import setup, Extension setup(name='freesteam' , version='0.6' , py_modules=['freesteam'] , ext_modules=[Extension('_freesteam',['_freesteam.dll'])] ) Any suggestions would be great. Cheers JP -- John Pye Department of Mechanical and Manufacturing Engineering University of New South Wales, Sydney, Australia http://pye.dyndns.org/ From cmlenz at gmx.de Thu Apr 27 14:50:10 2006 From: cmlenz at gmx.de (Christopher Lenz) Date: Thu, 27 Apr 2006 14:50:10 +0200 Subject: [Distutils] Bug in parsing entry points with extras spec? Message-ID: <4B0C3BBD-2824-4AF7-82EB-3410E9117E33@gmx.de> Hi all, I'm trying to specify an extra for an entry point, as follows: setup( ... extras_require = { 'DNS': ['dnspython>=1.3.5'] }, entry_points = { 'trac.plugins': [ 'spamfilter = tracspamfilter.api', 'spamfilter.adapters = tracspamfilter.adapters', 'spamfilter.filters.akismet = tracspamfilter.filters.akismet', 'spamfilter.filters.ip_blacklist = tracspamfilter.filters.ip_blacklist [DNS]' ] }, ... ) egg_info then fails with the message: ValueError: ('Invalid module name', 'tracspamfilter.filters.ip_blacklist ') Obviously the space between the module and the extras spec is messing up the parsing somehow, so I remove the space. Now, this works *only the first time* (i.e. before the entry_points.txt file) gets written out. Apparently setuptools also puts a space between the module name and the extras :-P Using setuptools 0.6a10... couldn't find anything related in the check-in messages. Am I missing something obvious here? Or isn't anyone actually using that particular feature yet? Anyway, replacing "value.lstrip()" with "value.strip()" in EntryPoint.parse() appears to fix this. Cheers, Chris -- Christopher Lenz cmlenz at gmx.de http://www.cmlenz.net/ From philippe at fluendo.com Thu Apr 27 14:55:58 2006 From: philippe at fluendo.com (Philippe Normand) Date: Thu, 27 Apr 2006 14:55:58 +0200 (CEST) Subject: [Distutils] Bug in parsing entry points with extras spec? In-Reply-To: <4B0C3BBD-2824-4AF7-82EB-3410E9117E33@gmx.de> Message-ID: Le 27/4/2006, "Christopher Lenz" a ?crit: >Hi all, > >I'm trying to specify an extra for an entry point, as follows: > > setup( > ... > extras_require = { > 'DNS': ['dnspython>=1.3.5'] > }, > entry_points = { > 'trac.plugins': [ > 'spamfilter = tracspamfilter.api', > 'spamfilter.adapters = tracspamfilter.adapters', > 'spamfilter.filters.akismet = >tracspamfilter.filters.akismet', > 'spamfilter.filters.ip_blacklist = >tracspamfilter.filters.ip_blacklist [DNS]' > ] > }, > ... > ) And if you try something like: entry_points=""" # ... spamfilter.filters.ip_blacklist = tracspamfilter.filters.ip_blacklist [DNS] """ ? > >egg_info then fails with the message: > > ValueError: ('Invalid module name', >'tracspamfilter.filters.ip_blacklist ') > >Obviously the space between the module and the extras spec is messing >up the parsing somehow, so I remove the space. Now, this works *only >the first time* (i.e. before the entry_points.txt file) gets written >out. Apparently setuptools also puts a space between the module name >and the extras :-P > >Using setuptools 0.6a10... couldn't find anything related in the >check-in messages. > >Am I missing something obvious here? Or isn't anyone actually using >that particular feature yet? > I've used that feature, but using the string based syntax instead of the dictionary. And it worked. >Anyway, replacing "value.lstrip()" with "value.strip()" in >EntryPoint.parse() appears to fix this. > From cmlenz at gmx.de Thu Apr 27 15:39:31 2006 From: cmlenz at gmx.de (Christopher Lenz) Date: Thu, 27 Apr 2006 15:39:31 +0200 Subject: [Distutils] Bug in parsing entry points with extras spec? In-Reply-To: References: Message-ID: Am 27.04.2006 um 14:55 schrieb Philippe Normand: > And if you try something like: > > entry_points=""" > # ... > spamfilter.filters.ip_blacklist = > tracspamfilter.filters.ip_blacklist > [DNS] > > """ > > ? That works, but again only without a space between the module name and the extras. Which should still be fixed, I suppose. Thanks! Chris -- Christopher Lenz cmlenz at gmx.de http://www.cmlenz.net/ From pje at telecommunity.com Thu Apr 27 20:09:34 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 27 Apr 2006 14:09:34 -0400 Subject: [Distutils] Bug in parsing entry points with extras spec? In-Reply-To: <4B0C3BBD-2824-4AF7-82EB-3410E9117E33@gmx.de> Message-ID: <5.1.1.6.0.20060427140827.03a52278@mail.telecommunity.com> At 02:50 PM 4/27/2006 +0200, Christopher Lenz wrote: >Obviously the space between the module and the extras spec is messing >up the parsing somehow, so I remove the space. Now, this works *only >the first time* (i.e. before the entry_points.txt file) gets written >out. Apparently setuptools also puts a space between the module name >and the extras :-P > >Using setuptools 0.6a10... couldn't find anything related in the >check-in messages. > >Am I missing something obvious here? Or isn't anyone actually using >that particular feature yet? The parse tests didn't include the case of "modulename [extras]", only "modulename:attrs [extras]". I've fixed the tests and the code now in both the 0.7 trunk and the 0.6 stable branch. Thanks for the bug report! From collinw at gmail.com Thu Apr 27 23:09:50 2006 From: collinw at gmail.com (Collin Winter) Date: Thu, 27 Apr 2006 17:09:50 -0400 Subject: [Distutils] easy_install question Message-ID: <43aa6ff70604271409o349fb64clb3db570ad1387a3a@mail.gmail.com> I've been testing the latest release of my functional package[1] to see that it works with easy_install. When I do "easy_install functional", it shows that it looks at PyPI, can't find anything appropriate, then goes on to the project's website, eventually downloading and installing functional-Py-0.6.tar.gz. I imagine that easy_install is getting confused by the seven files listed on the PyPI page; there are two versions of the package -- one in C and one in pure Python -- with tarballs and eggs for each version. Is there any way that I can mark a particular file (in an easy_install-understandable way) as, "if you don't know what you need, download this"? Should I use the "Download URL" field for this? Thanks, Collin Winter [1] - http://cheeseshop.python.org/pypi/functional From pje at telecommunity.com Fri Apr 28 00:19:50 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 27 Apr 2006 18:19:50 -0400 Subject: [Distutils] easy_install question In-Reply-To: <43aa6ff70604271409o349fb64clb3db570ad1387a3a@mail.gmail.co m> Message-ID: <5.1.1.6.0.20060427181722.036f8008@mail.telecommunity.com> At 05:09 PM 4/27/2006 -0400, Collin Winter wrote: >I've been testing the latest release of my functional package[1] to >see that it works with easy_install. When I do "easy_install >functional", it shows that it looks at PyPI, can't find anything >appropriate, then goes on to the project's website, eventually >downloading and installing functional-Py-0.6.tar.gz. > >I imagine that easy_install is getting confused by the seven files >listed on the PyPI page; there are two versions of the package -- one >in C and one in pure Python -- with tarballs and eggs for each >version. Is there any way that I can mark a particular file (in an >easy_install-understandable way) as, "if you don't know what you need, >download this"? Should I use the "Download URL" field for this? The problem is that you've uploaded distributions for two different projects: one named "functional-C", and the other named "functional-Py". Since neither one is "functional", EasyInstall keeps looking. From collinw at gmail.com Fri Apr 28 00:47:19 2006 From: collinw at gmail.com (Collin Winter) Date: Thu, 27 Apr 2006 18:47:19 -0400 Subject: [Distutils] easy_install question In-Reply-To: <5.1.1.6.0.20060427181722.036f8008@mail.telecommunity.com> References: <5.1.1.6.0.20060427181722.036f8008@mail.telecommunity.com> Message-ID: <43aa6ff70604271547h1983e611jf352a514cf67cb0a@mail.gmail.com> On 4/27/06, Phillip J. Eby wrote: > The problem is that you've uploaded distributions for two different > projects: one named "functional-C", and the other named > "functional-Py". Since neither one is "functional", EasyInstall keeps looking. Would EasyInstall prefer names like functional-0.6-Py* instead? Even with the name change, though, my real question remains: how would EasyInstall decide between, say, functional-0.6-Py.tar.gz and functional-0.6-C.tar.gz (or similarly-named eggs)? Collin Winter From pje at telecommunity.com Fri Apr 28 01:38:54 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 27 Apr 2006 19:38:54 -0400 Subject: [Distutils] easy_install question In-Reply-To: <43aa6ff70604271547h1983e611jf352a514cf67cb0a@mail.gmail.co m> References: <5.1.1.6.0.20060427181722.036f8008@mail.telecommunity.com> <5.1.1.6.0.20060427181722.036f8008@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060427193249.01e2a488@mail.telecommunity.com> At 06:47 PM 4/27/2006 -0400, Collin Winter wrote: >On 4/27/06, Phillip J. Eby wrote: > > The problem is that you've uploaded distributions for two different > > projects: one named "functional-C", and the other named > > "functional-Py". Since neither one is "functional", EasyInstall keeps > looking. > >Would EasyInstall prefer names like functional-0.6-Py* instead? Technically, that would be a version number: "0.6-Py", and it would mean the "Py post-release of version 0.6". "0.6-C", on the other hand, would mean a "release candidate of version 0.6". >Even with the name change, though, my real question remains: how would >EasyInstall decide between, say, functional-0.6-Py.tar.gz and >functional-0.6-C.tar.gz (or similarly-named eggs)? By the higher version number, which in this case would be 0.6-Py. There are two ways you can solve this problem: 1. Name the projects functional-C and functional-Py, and give each its own Cheeseshop listing (ala ElementTree/cElementTree), or 2. Do a source-only distribution that builds the C extension if and only if a C compiler is available. Call the single project "functional". #2 is hard to do at the moment, as neither the distutils nor setuptools support such a fallback position at present. I hope to add some kind of "this C extension is optional" support during the 0.7 development cycle.