From jeff at taupro.com Wed Oct 1 00:42:05 2008 From: jeff at taupro.com (Jeff Rush) Date: Tue, 30 Sep 2008 17:42:05 -0500 Subject: [Distutils] Just downloading an egg In-Reply-To: References: Message-ID: <48E2AB3D.7040009@taupro.com> Pascoe, S (Stephen) wrote: > I often just want to do with easy_install is download an egg from pypi without installing it. I've studied the easy_install documentation and never found a way to do it. Even giving it the "-d" option results in easy-install.pth being created and other unwanted stuff. > > Looking at the setuptools pydoc I worked out a way to do it: > >>>> import setuptools >>>> d = setuptools.Distribution() >>>> d.fetch_build_egg(requirement) > > Voila! the egg is downloaded into cwd. It even seems built an egg from a tarball. > > My question is, can I rely on this feature and is it the best way of doing what I want? I'd like to use it in my code and hope it stays. It would be ideal if I could do this through easy_install. Actually this command will download the egg and just the egg: easy_install -zmaxd . -N SQLObject or if you want the egg -and- its dependencies (as eggs too): easy_install -zmaxd . SQLObject -Jeff From pje at telecommunity.com Wed Oct 1 01:50:38 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 30 Sep 2008 19:50:38 -0400 Subject: [Distutils] [issue48] ignores --build-directory= option if argument is a local file In-Reply-To: <1222809682.7.0.213082672644.issue48@psf.upfronthosting.co. za> References: <1222809682.7.0.213082672644.issue48@psf.upfronthosting.co.za> <1222809682.7.0.213082672644.issue48@psf.upfronthosting.co.za> Message-ID: <20080930234930.61FED3A407A@sparrow.telecommunity.com> This is as-documented. Per the docs: """If a package is built from a source distribution or checkout, it will be extracted to a subdirectory of the specified directory.""" That is, --build-dir applies only to SVN checkouts (done by easy_install itself) and source distributions (i.e. sdist zipfiles and tarballs). From greg.ewing at canterbury.ac.nz Wed Oct 1 02:10:16 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 01 Oct 2008 12:10:16 +1200 Subject: [Distutils] [pyconuk] "just use debian" In-Reply-To: <1222780673.4166.55.camel@shizuru> References: <20080917171851.GA27261@logilab.fr> <20080919174433.GS8000@logilab.fr> <48D8C927.4060609@simplistix.co.uk> <20080926214116.GI9951@volans.logilab.fr> <48DDEC5C.7090601@ar.media.kyoto-u.ac.jp> <20080928173011.GA4852@volans.logilab.fr> <48E0C007.70702@ar.media.kyoto-u.ac.jp> <94bdd2610809290509l2ab7359ep7f4edfcd23eae4ee@mail.gmail.com> <20080930084253.GF4494@volans.logilab.fr> <94bdd2610809300505u1291ae6an3c44e2ce31d43e3f@mail.gmail.com> <1222780673.4166.55.camel@shizuru> Message-ID: <48E2BFE8.1050508@canterbury.ac.nz> Josselin Mouette wrote: > if you try to > build a package of baz, there is no way to express correctly that you > depend on python-bar (>= 1.3) or python-foo (>= 1.2). Seems to me that baz shouldn't have to say that -- all it should have to say is that it requires bar version 1.3. It's up to the package manager to know how to look inside packages to see what versions of other packages they contain, if such a thing is going to be allowed. Otherwise, whenever one package is moved inside another, then in order to take advantage of that, all other packages that use it would have to have their dependencies updated, which doesn't seem reasonable. I can't see the point of nesting packages like this anyway. If bar really is usable independently of foo, then why not just leave it in a separate tar file and let foo declare a dependency on it? They can be bundled together for convenience of manual distribution if desired, but when installed, such a bundle should be split out into separate packages as far as the package manager sees them. If they're being automatically retrieved from a repository, it makes more sense to keep them separate. There's no more to download that way, and there may be less, since you can just download the packages actually needed. -- Greg From greg.ewing at canterbury.ac.nz Wed Oct 1 02:40:35 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 01 Oct 2008 12:40:35 +1200 Subject: [Distutils] PEP for distutils In-Reply-To: <48E24897.8010800@colorstudy.com> References: <94bdd2610809280555p12c0e326r4a867bd3b67efbd9@mail.gmail.com> <48E23F2D.3000802@simplistix.co.uk> <20080930153822.GJ26878@phare.normalesup.org> <48E24897.8010800@colorstudy.com> Message-ID: <48E2C703.2030801@canterbury.ac.nz> Ian Bicking wrote: > FWIW, pyinstall can collect all the packages before installing any of > them. You do have to download all packages, though, as that's the only > way to get the metadata. This may be something to make sure is on the requirements list for a metatdata standard: Make sure there is a defined way of getting just the metadata from a repository without having to download the whole package. -- Greg From a.badger at gmail.com Wed Oct 1 04:18:59 2008 From: a.badger at gmail.com (Toshio Kuratomi) Date: Tue, 30 Sep 2008 19:18:59 -0700 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <48D9C6A7.1010409@colorstudy.com> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <2DF68F3D-6DAD-40A9-98DA-C28999A15E04@drinktomi.com> <48D9633D.6060407@snaplogic.org> <48D970BB.7070504@snaplogic.org> <48D9C6A7.1010409@colorstudy.com> Message-ID: <48E2DE13.2090309@gmail.com> Ian Bicking wrote: > Rick Warner wrote: >>> Actually, PyPI is replicated. See, for example, >>> http://download.zope.org/simple/. >>> >>> It may be that some of the mirrors should be better advertised. >> >> A half-hearted effort. at best, after the problems last year. When I >> configure a CPAN client (once per user) I create a list of replicas I >> want to search for any query from a list of hundreds of replicas >> distributed around the world. > > Can someone suggest the best way to search among repositories? For > instance, try to connect to one, then stop if it gives Connection > Refused? If it gives any unexpected error (5xx)? Timing out is a > common failure, and a pain in the butt, but I guess there's that too. > What does the CPAN client do? > > I don't know what CPAN does but Linux distributions have also solved this problem. We send out massive numbers of updates and new packages to users every day so we need a mirror network that works well. In Fedora we have a server that gives out a list of mirrors with GeoIP data used to try and assemble a list of mirrors near you (country, then continent (with special cases, for instance, for certain middle eastern countries that connect better to Europe than to Asia) and then global). This server gives the mirror list out (randomized among the close mirrors) and the client goes through the list, trying to retrieve package metadata. If it times out or otherwise fails, then it goes on to the next mirror until it gets data. (Note, some alternate clients are able to download from multiple servers at the same time if multiple packages are needed.) The mirrorlist server is a pretty neat application (https://fedorahosted.org/mirrormanager). It has a TurboGears front end that allows people to add a new mirror (https://admin.fedoraproject.org/mirrormanager) for public availability or restricted to a subset of IPs. It allows you to only mirror a subset of the whole content. And it has several methods of telling if the mirror is in sync or outdated. The latter is important to us for making sure we're giving out users the latest updates that we've shipped and ranges from a script that the mirror admin can run from their cron job to check the data available and report back to a process run on our servers to check that the mirrors have up to date content. The mirrorlist itself is cached and served from a mod_python script (soon to be mod_wsgi) for speed. You might also be interested in the way that we work with package metadata. In Fedora and many other rpm-based distributions (Some Debian-based distros talked about this as well but I don't know if it was ever implemented there) we create static xml files (and recently, sqlite dbs as well) that live on the mirrors. The client hits the mirror and downloads at least two of these files. The repomd.xml file describes the other files with checksums and is used to verify that the other metadata is up to date and whether anything has changed. The primary.xml file stores information that is generally what is needed for doing depsolving on the packages. Then we have several other xml files that collectively contain the complete metadata for the packages but is usually overkill... by separating htis stuff out, we save clients from having to download it in the common case. This stuff could provide some design ideas for constructing a pypi metadata repository and is documented here: http://createrepo.baseurl.org/ Note: the reason we went with static metadata rather than some sort of cgi script is that static data can be mirrored without the mirror being required to run anything beyond a simple rsync cron job. This makes finding mirrors much easier. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: OpenPGP digital signature URL: From Brian.Cameron at Sun.COM Wed Oct 1 08:12:20 2008 From: Brian.Cameron at Sun.COM (Brian Cameron) Date: Wed, 01 Oct 2008 01:12:20 -0500 Subject: [Distutils] Question about easy-install.pth Message-ID: <48E314C4.1020300@sun.com> I am having troubles working with setuptools on Solaris. The Solaris operating system normally installs modules as packages which contain binaries. This is unlike other Linux operating systems where, for exmaple, an RPM would download the source and build it on the user's machine when they install the RPM. So, to create packages on Solaris we normally install a module to a temporary directory such as /var/tmp/pkgbuild-foo/usr, package up the files that are built, and when the user installs the package these files are then installed to their system. However, we are having problems with figuring out how to properly create the /usr/lib/python2.4/site-packages/easy-install.pth file using our build system. What happens is that each package ends up with its own easy-install.pth file which only contains the information for that one module. Users can't install two such packages at the same time because it creates a file conflict, two packages can't install the same file. Solaris packages do have pre-install, post-install, pre-uninstall and post-uninstall scripts so that we could do something like avoid installing the file as a part of the package and instead generate it on-the-fly via scripting. However, it doesn't seem that setuptools provides any mechanisms for doing this easily. Though I'm not very familiar with setuptools or easy_install, so I hope that I'm missing something. Aside from writing our own code to manually manage adding and removing entries to/from this file, is there any way that setuptools allows you to manage this file when installing binaries to a system, or does setuptools assume the file only needs to be managed when you build a module? I'm hoping to avoid some hacky solution where we try and hack the easy-install.pth file by hand when users install or uninstall packages. Since users can install or uninstall any random combination of packages which may need an easy-install.pth file, I can imagine that it would get complicated to properly manage it via our own scripts. Or is there some other solution that might be useful in our situation that might avoid the need for multiple packages to need this file, or might it be possible for different modules to use differently named files instead of a commonly named easy-install.pth file. Brian From deron.meranda at gmail.com Wed Oct 1 09:22:32 2008 From: deron.meranda at gmail.com (Deron Meranda) Date: Wed, 1 Oct 2008 03:22:32 -0400 Subject: [Distutils] Question about easy-install.pth In-Reply-To: <48E314C4.1020300@sun.com> References: <48E314C4.1020300@sun.com> Message-ID: <5c06fa770810010022q28bbfia796c13e7b024c80@mail.gmail.com> On Wed, Oct 1, 2008 at 2:12 AM, Brian Cameron wrote: > I am having troubles working with setuptools on Solaris. The Solaris > operating system normally installs modules as packages which contain > binaries. This is unlike other Linux operating systems where, for > exmaple, an RPM would download the source and build it on the user's > machine when they install the RPM. Actually RPMs usually are "binary". You may be thinking SRPM files (source RPM). Also, not all Linux's use RPM. > So, to create packages on Solaris we normally install a module to > a temporary directory such as /var/tmp/pkgbuild-foo/usr, package up > the files that are built, and when the user installs the package > these files are then installed to their system. What you'll want to do is to package up the files by creating an egg archive instead. In its simplest form and egg file is basically just a zip file which contains all the python files as well as a small amount of meta data. You use setuptools to create these egg archives. Note that it may be easier to think of an egg file as a "binary" package, much like a tar file, and not as a source package. The best way to make an egg file is to first create a setup.py file which describes the python module you're packaging along with a bit of meta data. At the bare minimum it may be just: #!/usr/bin/env python from setuptools import setup setup( name="mymodule", version="1.0", description="My python module" ) although you'll probably want to add more meta data to it. Then you create an egg file for it using something like: python setup.py bdist_egg -d /tmp/eggfiles and you'll get a file like /tmp/eggfiles/mymodule-1.0-py2.4.egg > However, we are having problems with figuring out how to properly > create the /usr/lib/python2.4/site-packages/easy-install.pth file > using our build system. What happens is that each package ends up > with its own easy-install.pth file ... The easy-install.pth file should not be part of the "package" (and it won't be if you use egg files as your package). The easy-install.pth file is instead created or modified as a side effect of installing your package. The easy_install command does all the hard work for you. > Aside from writing our own code to manually manage adding and removing > entries to/from this file, is there any way that setuptools allows you > to manage this file when installing binaries to a system... > > I'm hoping to avoid some hacky solution where we try and hack the > easy-install.pth file... You definitely want to avoid hacking the easy-install.pth file if possible. It's format could potentially change some day. Also it can be tricky if you ever deal with a multiple version installation. Fortunately the easy_install command handles that file for you. Basically you create your "binary" package as an egg file, and then when installing your package you invoke easy_install to copy the egg file into the proper place and register it in the easy-install.pth file. If you have your egg file(s) in the local filesystem, you would typically install/upgrade it like: easy_install -U some-package.egg where -U says to install or upgrade if an older version was already present. Depending on how standalone you want it, you may also want to include the option, -H "" , which prevents easy_install from searching the net for the egg file. For completeness I should note that you can opt not to use setuptools or even distutils at all. You could then just bundle all your module files up as an ordinary tar or cpio archive and then later install it by just unarchiving them into the site-packages directory on a different server. You could also create a *.pth file if you need to (named after your module like "mymodule.pth"; not easy-install.pth). However you're on your own in terms of any sort of dependency processing or version conflicts. This is though probably the closest to your original intent of just wanting to archiving files post-install, and extracting them as-is on another server. -- Deron Meranda From joss at debian.org Wed Oct 1 10:11:47 2008 From: joss at debian.org (Josselin Mouette) Date: Wed, 01 Oct 2008 10:11:47 +0200 Subject: [Distutils] "just use debian" In-Reply-To: <48E2A0BB.6090300@enthought.com> References: <20080917171851.GA27261@logilab.fr> <20080930084253.GF4494@volans.logilab.fr> <94bdd2610809300505u1291ae6an3c44e2ce31d43e3f@mail.gmail.com> <1222780673.4166.55.camel@shizuru> <94bdd2610809300649q6bc9cdf3t84d5221d956bfc9d@mail.gmail.com> <1222784868.4166.88.camel@shizuru> <94bdd2610809300820g36a3f1bbpc2a04e03b43ee9c9@mail.gmail.com> <48E24765.1020908@simplistix.co.uk> <48E24BDA.2050104@simplistix.co.uk> <94bdd2610809300901o3fa62601q985784f01a8ad774@mail.gmail.com> <48E24FF7.9000300@simplistix.co.uk> <48E255B9.50806@colorstudy.com> <1222801361.32659.1.camel@shizuru> <48E2903F.9000307@enthought.com> <1222810334.32659.18.camel@shizuru> <48E2A0BB.6090300@enthought.com> Message-ID: <1222848707.7060.4.camel@tomoyo> Le mardi 30 septembre 2008 ? 16:57 -0500, Dave Peterson a ?crit : > But we already have a separation between project name and module names > that are contained within that project. We don't currently declare > dependencies on the module names but on the project name. i.e. a > dependency on HardJSON > 2.0 does not say anything about what modules > you're expecting to import or use, only that you expect to use version > 2 of a project called HardJSON. Were you suggesting that change? As I understand PEP 345, it proposes to introduce dependencies based on module names, not on project names. This is one of the key points where it is superior to setuptools; especially, if developers correctly change the module name when changing the API, the dependencies are clear and reliable. At the distribution level, we will end up changing the binary package name, but this is not something you have to worry about. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From david at ar.media.kyoto-u.ac.jp Wed Oct 1 10:24:34 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 01 Oct 2008 17:24:34 +0900 Subject: [Distutils] "just use debian" In-Reply-To: <1222848707.7060.4.camel@tomoyo> References: <20080917171851.GA27261@logilab.fr> <20080930084253.GF4494@volans.logilab.fr> <94bdd2610809300505u1291ae6an3c44e2ce31d43e3f@mail.gmail.com> <1222780673.4166.55.camel@shizuru> <94bdd2610809300649q6bc9cdf3t84d5221d956bfc9d@mail.gmail.com> <1222784868.4166.88.camel@shizuru> <94bdd2610809300820g36a3f1bbpc2a04e03b43ee9c9@mail.gmail.com> <48E24765.1020908@simplistix.co.uk> <48E24BDA.2050104@simplistix.co.uk> <94bdd2610809300901o3fa62601q985784f01a8ad774@mail.gmail.com> <48E24FF7.9000300@simplistix.co.uk> <48E255B9.50806@colorstudy.com> <1222801361.32659.1.camel@shizuru> <48E2903F.9000307@enthought.com> <1222810334.32659.18.camel@shizuru> <48E2A0BB.6090300@enthought.com> <1222848707.7060.4.camel@tomoyo> Message-ID: <48E333C2.9030706@ar.media.kyoto-u.ac.jp> Josselin Mouette wrote: > > As I understand PEP 345, it proposes to introduce dependencies based on > module names, not on project names. This is one of the key points where > it is superior to setuptools; especially, if developers correctly change > the module name when changing the API, the dependencies are clear and > reliable. I don't understand this. Why do you need to change the module name ? You need to be able to request a given version of the module, yes, but you don't need to change the name itself. Changing name is an implementation detail of the concept of retrieving a particular version, no ? I mean, when I build my C program against the glibc, I never have to change my compiler commands depending on which version of glibc I want. Internally, this is handled by versioned sonames, at the link and load stage, yes, but only because it is "easy" to do and doable for C, because the library itself is never referenced in the code; IOW, there is the link process in C and similar compiled languages which use the name changes. But other languages use something different; I don't claim any deep understanding of the scheme, but mono uses a totally different scheme to handle versioning: http://www.mono-project.com/Assemblies_and_the_GAC Does mono cause problems to debian ? I think this kind of system is much more adapted to python than the C model is, because C has this two steps thing that python does not have. Of couse, it is more complicated than changing names. But I fear that people wants their cake and eating it: you can't have well maintained packages and reliability without thinking about API/ABI issues. There has to be a balance between OS distributors need (mostly only one version) and other people needs who may need several parallel installations. I think setuptools and the whole eggs thing makes it well too easy to install several versions at the same time, making some people naively thinking it solves the dependency issues, without thinking that it replaces a kind of dll hell by a dependency hell. But OTOH, as a developer, I need to be able to develop my packages, distribute them, and breaking them from time to time before I reach 1.0. That's almost requirement of open source development, at least if you follow the release early/release often. cheers, David From joss at debian.org Wed Oct 1 10:58:22 2008 From: joss at debian.org (Josselin Mouette) Date: Wed, 01 Oct 2008 10:58:22 +0200 Subject: [Distutils] "just use debian" In-Reply-To: <48E333C2.9030706@ar.media.kyoto-u.ac.jp> References: <20080917171851.GA27261@logilab.fr> <20080930084253.GF4494@volans.logilab.fr> <94bdd2610809300505u1291ae6an3c44e2ce31d43e3f@mail.gmail.com> <1222780673.4166.55.camel@shizuru> <94bdd2610809300649q6bc9cdf3t84d5221d956bfc9d@mail.gmail.com> <1222784868.4166.88.camel@shizuru> <94bdd2610809300820g36a3f1bbpc2a04e03b43ee9c9@mail.gmail.com> <48E24765.1020908@simplistix.co.uk> <48E24BDA.2050104@simplistix.co.uk> <94bdd2610809300901o3fa62601q985784f01a8ad774@mail.gmail.com> <48E24FF7.9000300@simplistix.co.uk> <48E255B9.50806@colorstudy.com> <1222801361.32659.1.camel@shizuru> <48E2903F.9000307@enthought.com> <1222810334.32659.18.camel@shizuru> <48E2A0BB.6090300@enthought.com> <1222848707.7060.4.camel@tomoyo> <48E333C2.9030706@ar.media.kyoto-u.ac.jp> Message-ID: <1222851502.7060.18.camel@tomoyo> Le mercredi 01 octobre 2008 ? 17:24 +0900, David Cournapeau a ?crit : > I mean, when I build my C program against the glibc, I never have to > change my compiler commands depending on which version of glibc I want. Indeed, and the reason is that *functions never disappear from the glibc*. > Internally, this is handled by versioned sonames, at the link and load > stage, yes, but only because it is "easy" to do and doable for C, > because the library itself is never referenced in the code; IOW, there > is the link process in C and similar compiled languages which use the > name changes. But other languages use something different; I don't claim > any deep understanding of the scheme, but mono uses a totally different > scheme to handle versioning: > > http://www.mono-project.com/Assemblies_and_the_GAC > > Does mono cause problems to debian ? I think this kind of system is much > more adapted to python than the C model is, because C has this two steps > thing that python does not have. I don?t think Mono causes issues, but I?m pretty sure that allowing multiple versions like the GAC allows *will* causes issues. Not for purely-packaged things, where you can safely ignore those directory renames, but if you start mixing distributed packages and user-downloaded stuff, they will run into the same issues we have with setuptools. > Of couse, it is more complicated than changing names. But I fear that > people wants their cake and eating it: you can't have well maintained > packages and reliability without thinking about API/ABI issues. There > has to be a balance between OS distributors need (mostly only one > version) and other people needs who may need several parallel > installations. Changing name is the only thing that is available at the language level, currently. Everything else that comes on top of it is a pile of gross hacks. If you really want something better, with e.g. a name and an API version, you need to convince the core python developers to introduce such functionality in the interpreter itself. Something that allows to say "import foo with apiver 2 minver 2.1". > I think setuptools and the whole eggs thing makes it well too easy to > install several versions at the same time, making some people naively > thinking it solves the dependency issues, without thinking that it > replaces a kind of dll hell by a dependency hell. But OTOH, as a > developer, I need to be able to develop my packages, distribute them, > and breaking them from time to time before I reach 1.0. That's almost > requirement of open source development, at least if you follow the > release early/release often. C developers have already solved these kinds of issues, and they no reason why you can?t do the same: keep the modules private until they are deemed stable, introduce deprecation warnings long (and I mean two years, not two weeks) before API breaks, require other developers to use e.g. specific flags to access unstable API, etc. They are all compromises, because there is no silver bullet for dealing with these issues. And setuptools is not a silver bullet either, except for shooting yourself in the foot. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From david at ar.media.kyoto-u.ac.jp Wed Oct 1 10:53:37 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 01 Oct 2008 17:53:37 +0900 Subject: [Distutils] "just use debian" In-Reply-To: <1222851502.7060.18.camel@tomoyo> References: <20080917171851.GA27261@logilab.fr> <20080930084253.GF4494@volans.logilab.fr> <94bdd2610809300505u1291ae6an3c44e2ce31d43e3f@mail.gmail.com> <1222780673.4166.55.camel@shizuru> <94bdd2610809300649q6bc9cdf3t84d5221d956bfc9d@mail.gmail.com> <1222784868.4166.88.camel@shizuru> <94bdd2610809300820g36a3f1bbpc2a04e03b43ee9c9@mail.gmail.com> <48E24765.1020908@simplistix.co.uk> <48E24BDA.2050104@simplistix.co.uk> <94bdd2610809300901o3fa62601q985784f01a8ad774@mail.gmail.com> <48E24FF7.9000300@simplistix.co.uk> <48E255B9.50806@colorstudy.com> <1222801361.32659.1.camel@shizuru> <48E2903F.9000307@enthought.com> <1222810334.32659.18.camel@shizuru> <48E2A0BB.6090300@enthought.com> <1222848707.7060.4.camel@tomoyo> <48E333C2.9030706@ar.media.kyoto-u.ac.jp> <1222851502.7060.18.camel@tomoyo> Message-ID: <48E33A91.5000602@ar.media.kyoto-u.ac.jp> Josselin Mouette wrote: > > Indeed, and the reason is that *functions never disappear from the > glibc*. Yes and no. If you remove a function, you're indeed screwed, because you can't handle versioning in the header. But you can handle versioning in libraries at the link step, and file name of the library is an implementation detail of this versioning. > > I don?t think Mono causes issues, but I?m pretty sure that allowing > multiple versions like the GAC allows *will* causes issues. Not for > purely-packaged things, where you can safely ignore those directory > renames, but if you start mixing distributed packages and > user-downloaded stuff, they will run into the same issues we have with > setuptools. Please read the article carefully, it is not only about the GAC. It does handle the two conflicting issues: API stability installed globally vs easiness of deployment. That's why it is an interesting read IMHO: it addresses both issues. I don't think there is a single chance to see something as strong as C for python, because it would severely undermine the whole idea of the language used for prototyping. > > Changing name is the only thing that is available at the language level, > currently. Everything else that comes on top of it is a pile of gross > hacks. If you really want something better, with e.g. a name and an API > version, you need to convince the core python developers to introduce > such functionality in the interpreter itself. Something that allows to > say "import foo with apiver 2 minver 2.1". Yes, that's exactly what I am saying. But that's the only solution in the long term I can see. Setuptools and co will never be usable for robust deployment: it kind of works for simple cases, or for developers who know what they are doing. But it is inherently unable to handle more complicated cases. > > C developers have already solved these kinds of issues, and they no > reason why you can?t do the same: keep the modules private until they > are deemed stable, introduce deprecation warnings long (and I mean two > years, not two weeks) before API breaks, require other developers to use > e.g. specific flags to access unstable API, etc. Again, this is pipe-dream. You can do that for C because it is a dead language (dead in the sense not evolving). Python is not like that. Changing python philosophy has zero chance of success. How many languages appeared after C do like C ? I don't know many. cheers, David From ziade.tarek at gmail.com Wed Oct 1 12:34:07 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 1 Oct 2008 12:34:07 +0200 Subject: [Distutils] [zc.buildout] running in safe mode Message-ID: <94bdd2610810010334g3e3994bcw5cfa270de3ae73e7@mail.gmail.com> Hello I know it is a bad practice for a recipe to return some paths that contains important data in the install() method, because zc.buildout might remove them. Nevertheless, it happens from time to time that a developer lose some content because of a misconfiguration, or a zealous recipe. That is his responsability, and backups are done for that. But I think we can improve zc.buildout a bit for that: what about introducing a safe-mode where the developer get prompted everytime zc.buildout.rmtree is about to call shutil.rmtree ? The option could be set in [buildout] like this: [buildout] ... safe-mode = true ... and challenge the user when a tree is about to be delete. (it might be overkill for single files, and they are most of the time unimportant configuration files) This is a small change, and it would avoid running a backup everytime the .cfg are changed. because it happens all day long when you are developing. Regards Tarek -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From S.Pascoe at rl.ac.uk Wed Oct 1 13:35:48 2008 From: S.Pascoe at rl.ac.uk (Pascoe, S (Stephen)) Date: Wed, 1 Oct 2008 12:35:48 +0100 Subject: [Distutils] Just downloading an egg In-Reply-To: <48E2AB3D.7040009@taupro.com> References: <48E2AB3D.7040009@taupro.com> Message-ID: I feel foolish now. The bit I was always missing was "-m" which stops easy_install complaining about "." not being on PYTHONPATH. I really think this should be more obvious though. Something this basic could do with it's own option. It would help demystify what easy_install does. The basic use case of "Just download and install" is so intuitive but doing individual steps like "Just download" requires understanding how all these options interact. Yes, I see "-zmaxd" is mentioned in http://peak.telecommunity.com/DevCenter/EasyInstall but it isn't made to sound that useful. I'm a big fan of easy_install but I do have problems selling it to my co-developers sometimes. Little things like this would help. Cheers, Stephen. --- Stephen Pascoe +44 (0)1235 445980 British Atmospheric Data Centre Rutherford Appleton Laboratory -----Original Message----- From: Jeff Rush [mailto:jeff at taupro.com] Sent: 30 September 2008 23:42 To: Pascoe, S (Stephen) Cc: distutils-sig at python.org Subject: Re: [Distutils] Just downloading an egg Pascoe, S (Stephen) wrote: > I often just want to do with easy_install is download an egg from pypi without installing it. I've studied the easy_install documentation and never found a way to do it. Even giving it the "-d" option results in easy-install.pth being created and other unwanted stuff. > > Looking at the setuptools pydoc I worked out a way to do it: > >>>> import setuptools >>>> d = setuptools.Distribution() >>>> d.fetch_build_egg(requirement) > > Voila! the egg is downloaded into cwd. It even seems built an egg from a tarball. > > My question is, can I rely on this feature and is it the best way of doing what I want? I'd like to use it in my code and hope it stays. It would be ideal if I could do this through easy_install. Actually this command will download the egg and just the egg: easy_install -zmaxd . -N SQLObject or if you want the egg -and- its dependencies (as eggs too): easy_install -zmaxd . SQLObject -Jeff From ziade.tarek at gmail.com Wed Oct 1 14:10:48 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 1 Oct 2008 14:10:48 +0200 Subject: [Distutils] pre-PEP : Synthesis of previous threads, and irc talks + proposals Message-ID: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.com> Hello I have followed most of the threads from the past days, and we talked a lot on IRC with people from Fedora, Debian, Enthought, TG2 on possible enhancements While the other threads are continuing in deeper details, I would like to start a fresh thread were people don't have to re-read everything to be able to give their opinions on very precise points, This thread is focusing on shouting out the current problems and the solutions that can be adopted. I'd like to have "+1" and "-1" on each proposal, with at most one sentence. or fix a mistake if there is. That could help us speed up the work. let's try to keep this thread concise, if you want to discuss deeply on a problem, start another thread, and i'll follow it to fix my summary. The problems =========== 1/ the dependencies of a package are not expressed in the Require metadata of the package most of the time. adding a dependency to a module is not really done, developer add dependencies to packages. Furthermore, developer tend to use setuptools "install_requires" and "tests_require" arguments to express dependencies. So basically, you have to run an egg_info to get them, because the info files are generated by commands. 2/ the existence of PyPI had a side-effect: people tend to push the entire doc of the package in one single field (long_description) to display them at PyPI. The documentation of the package is not cleary pointed to others. 3/ the metadata infos cannot be expressed in a static file by the developer, because sometimes they are calculated by code. while this very permissive, that is how it works but they are tighted to argument passing to setup(). 4/ PyPI lacks of direct information about dependencies. In the meantime, the DOAP project is working on a way to express dependencies, but it is a work in progress. 5/ ideally, they should be one and only one version of a given package in an OS-based installation 6/ packagers would like to have more compatibility information to work out on security upgrades or version conflicts 7/ developers should be able to have more options when they define version dependencies in their packages, things like: A depends on B>=1.2 and B<=2.0 but with a preference to B 1.4 or "avoid B 1.7" they give tips to packagers ! 8/ the requires-python field is rarely used by people, so unless you try the package, you don't know when it is a source distribution, if it is going to run on various python versions. 9/ unless the developer has a strong comitment to an OS, he will never create and use a file that is located in /etc 10/ you can't possibily have a complete knowledge of the dependency graph and possible conflicts when you introduce a versioned dependency in your package. packages at given versions are known by some people to work well together or not in a set of versioned packages, Let's call this a "known good set" (KGS) - OS packager know and maintain the KGS for their distribution. - Web framework packagers does it for their application you don't. unless you work in a "KGS" environment. But if you want your package to be a regular python package at PyPI, packagers should be able to change its dependencies to make it fit their own KGS, and to build their knowledge on it. The developer dependencies infos is a tip and a help for a packager, not an enforcement. see [7] 11/ people should always upload the sdist version at PyPI, they don't do it always. otherwise it is a pain for packagers. Proposals ======== this is also a synthezis of what I hurd, and some elements I have added to respect the needs that were expressed. 0/ a lot of work can be done to clean distutils, no matter what is decided (another PEP is built for that) cleanning, removing old-style code, testing 1/ let's change the Python Metadata , in order to introduce a better dependency system, by - officialy introduce "install requires" and "test requires" metadata in there - mark "requires" as deprecated 2/ Let's move part of setuptools code in distutils, to respect those changes. 3/ let's create a simple convention : the metadata should be expressed in a python module called 'pkginfo.py' where each metadata is a variable. that can be used by setup.py and therefore by any tool that work with it, even if it does not run a setup.py command. This is simpler, this is cleaner. you don't have to run some setup magic to read them. at least some magic introduces by commands 4/ let's change PyPI to make it work with the new metadata and to enforce a few things Enforcements: - a binary distribution cannot be uploaded if a source distrbution has not been previously provided for the version - the requires-python need to be present. : come on, you know what python versions your package work with ! New features: - we should be able to download the metadata of a package without downloading the package - PyPI should display the install and test dependencies in the UI - The XML-RPC should provide this new metadata as well. - a commenting system should allow developers and packagers to give more infos on a package at PyPI to make the work easier Open question ============ (please if you want to react on those, open another thread, with a clean cut, otherwise it is hard to follow directly) - what about the documentation ? can't we express it better in the Metadata ? I think we can structurize it a bit - what about the configuration ? can't we find a way to interact with a config ini-like file for instance and don't care if it is located at /etc/package.cfg or at /Volumes/..etc ? Regards Tarek -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From gherman at darwin.in-berlin.de Wed Oct 1 14:10:04 2008 From: gherman at darwin.in-berlin.de (Dinu Gherman) Date: Wed, 1 Oct 2008 14:10:04 +0200 Subject: [Distutils] Installing single module from src directory Message-ID: Hi, I've been using distutils for a while now, but today I'm running into what seems to be the "minimal strange issue". I want to in- stall a single Python module without anything else, no package around it, which resides in a source subdirectory of the main project directory. My layout looks like the following: Macintosh:Python dinu$ tree2.py -f mymodule mymodule/ | setup.py | src/ | | mymodule.py Now common sense says all I need to do is define the module in setup.py like this: Macintosh:mymodule dinu$ more setup.py from distutils.core import setup setup( py_modules = ["src/mymodule"], ) Then, distutils does the following when running setup.py: Macintosh:mymodule dinu$ py252 setup.py build running build running build_py creating build creating build/lib creating build/lib/src copying src/mymodule.py -> build/lib/src And of course, I get the following structure built: Macintosh:mymodule dinu$ tree2.py -f build build/ | lib/ | | src/ | | | mymodule.py And sure enough I don't want the "src" level. I can get rid of it, but only after moving mymodule.py into the the main project directory, and removing the "src/" in py_modules. Instead, what I would *really* like is some smart combination of setup parameters that would just do it. But so far, all my efforts in finding combinations of py_modules, packages, package_dir (dummy/fake package) and even package_data does not do the trick. Now I'm desperately looking for a distutils "Houdini" on this list. Regards, Dinu From setuptools at bugs.python.org Wed Oct 1 00:45:16 2008 From: setuptools at bugs.python.org (chris) Date: Tue, 30 Sep 2008 22:45:16 +0000 Subject: [Distutils] [issue49] [PATCH] Multiple dependency resolution In-Reply-To: <1222814716.18.0.900882745516.issue49@psf.upfronthosting.co.za> Message-ID: <1222814716.18.0.900882745516.issue49@psf.upfronthosting.co.za> New submission from chris : Given a set of packages and their dependencies, process all dependencies, find the intersection, and install dependencies based on the results. The current behavior is to find the best match for each individual package. This can result in breakages when different projects depend on different versions of a common project. ---------- files: merge.patch messages: 190 nosy: ccasey priority: feature status: unread title: [PATCH] Multiple dependency resolution Added file: http://bugs.python.org/setuptools/file24/merge.patch _______________________________________________ Setuptools tracker _______________________________________________ -------------- next part -------------- A non-text attachment was scrubbed... Name: merge.patch Type: text/x-patch Size: 34387 bytes Desc: not available URL: From ziade.tarek at gmail.com Wed Oct 1 14:26:02 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 1 Oct 2008 14:26:02 +0200 Subject: [Distutils] package information In-Reply-To: <20080930175150.GC11672@volans.logilab.fr> References: <20080930175150.GC11672@volans.logilab.fr> Message-ID: <94bdd2610810010526g4165257bi971997ddf04e355e@mail.gmail.com> On Tue, Sep 30, 2008 at 7:51 PM, Nicolas Chauvat wrote: > As a follow up to the thread named [Fwd: Re: Announcing distribute > project] where Tarek ended up proposing a package_info.py file or a > PackageInfo class: > > http://www.logilab.org/cgi-bin/hgwebdir.cgi/logilab/common/file/02993a85ee3c/__pkginfo__.py > That is exacly what I had in mind, I have renamed it pkginfo.py in my latest mail, but that's the spirit KISS definition at developer level, of metadata. > -- > Nicolas Chauvat > > logilab.fr - services en informatique scientifique et gestion de connaissances > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From fdrake at gmail.com Wed Oct 1 15:08:06 2008 From: fdrake at gmail.com (Fred Drake) Date: Wed, 1 Oct 2008 09:08:06 -0400 Subject: [Distutils] [Catalog-sig] pre-PEP : Synthesis of previous threads, and irc talks + proposals In-Reply-To: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.com> References: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.com> Message-ID: <9cee7ab80810010608uc761173t2937493d88a79ea5@mail.gmail.com> On Wed, Oct 1, 2008 at 8:10 AM, Tarek Ziad? wrote: > 8/ the requires-python field is rarely used by people, so unless you > try the package, you don't know when it is a source > distribution, if it is going to run on various python versions. What requires-python field? I don't see this documented for distutils or setuptools. -Fred -- Fred L. Drake, Jr. "Chaos is the score upon which reality is written." --Henry Miller From asmodai at in-nomine.org Wed Oct 1 15:16:09 2008 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Wed, 1 Oct 2008 15:16:09 +0200 Subject: [Distutils] pre-PEP : Synthesis of previous threads, and irc talks + proposals In-Reply-To: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.com> References: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.com> Message-ID: <20081001131609.GW30869@nexus.in-nomine.org> -On [20081001 14:10], Tarek Ziad? (ziade.tarek at gmail.com) wrote: >I have followed most of the threads from the past days, and we talked >a lot on IRC with people from Fedora, Debian, Enthought, TG2 on >possible enhancements Just a note: do not forget the BSD Unix systems when it comes to packaging and whatnot, it's quite a bit different from the Linux systems. Also, there's pkgsrc which spans multiple different platforms. -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B Vae victis! From ziade.tarek at gmail.com Wed Oct 1 15:21:35 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 1 Oct 2008 15:21:35 +0200 Subject: [Distutils] [Catalog-sig] pre-PEP : Synthesis of previous threads, and irc talks + proposals In-Reply-To: <9cee7ab80810010608uc761173t2937493d88a79ea5@mail.gmail.com> References: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.com> <9cee7ab80810010608uc761173t2937493d88a79ea5@mail.gmail.com> Message-ID: <94bdd2610810010621h6033e2d0l791e623e86ac8513@mail.gmail.com> On Wed, Oct 1, 2008 at 3:08 PM, Fred Drake wrote: > On Wed, Oct 1, 2008 at 8:10 AM, Tarek Ziad? wrote: >> 8/ the requires-python field is rarely used by people, so unless you >> try the package, you don't know when it is a source >> distribution, if it is going to run on various python versions. > > What requires-python field? > > I don't see this documented for distutils or setuptools. The one described here http://www.python.org/dev/peps/pep-0345/ in Metadata 1.2 'Requires-Python' So it can't be used by people at this time, that was a mistake. the new version of [8] could be: 8/ you don't know when it is a source distribution, if it is going to run on various python versions. PEP 345 defines it, but it is not yet implemented in distutils, either in setuptools. and in the proposal parts: 0/ ... + implement the metadata 1.2 from PEP 345, (besides the "requires" metadata) > > > -Fred > > -- > Fred L. Drake, Jr. > "Chaos is the score upon which reality is written." --Henry Miller > -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From ziade.tarek at gmail.com Wed Oct 1 15:24:38 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 1 Oct 2008 15:24:38 +0200 Subject: [Distutils] pre-PEP : Synthesis of previous threads, and irc talks + proposals In-Reply-To: <20081001131609.GW30869@nexus.in-nomine.org> References: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.com> <20081001131609.GW30869@nexus.in-nomine.org> Message-ID: <94bdd2610810010624t33fef42cs1268c05d63adf4d1@mail.gmail.com> On Wed, Oct 1, 2008 at 3:16 PM, Jeroen Ruigrok van der Werven wrote: > -On [20081001 14:10], Tarek Ziad? (ziade.tarek at gmail.com) wrote: >>I have followed most of the threads from the past days, and we talked >>a lot on IRC with people from Fedora, Debian, Enthought, TG2 on >>possible enhancements > > Just a note: do not forget the BSD Unix systems when it comes to packaging > and whatnot, it's quite a bit different from the Linux systems. Also, > there's pkgsrc which spans multiple different platforms. Yes indeed, a BSD packager could be helpfull in the discussion. Are you able to help us in this ? Tarek From asmodai at in-nomine.org Wed Oct 1 15:29:02 2008 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Wed, 1 Oct 2008 15:29:02 +0200 Subject: [Distutils] pre-PEP : Synthesis of previous threads, and irc talks + proposals In-Reply-To: <94bdd2610810010624t33fef42cs1268c05d63adf4d1@mail.gmail.com> References: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.com> <20081001131609.GW30869@nexus.in-nomine.org> <94bdd2610810010624t33fef42cs1268c05d63adf4d1@mail.gmail.com> Message-ID: <20081001132902.GX30869@nexus.in-nomine.org> -On [20081001 15:24], Tarek Ziad? (ziade.tarek at gmail.com) wrote: >Yes indeed, a BSD packager could be helpfull in the discussion. Are >you able to help us in this ? I used to be a FreeBSD and DragonFly BSD committer and I still use Python mostly on BSD systems. I am sure Trent Nelson could also help. I know Joerg Sonnenberger from pkgsrc/NetBSD quite well too. So all in all I should be able to. My time seems a bit more limited than yours though. :) -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B Dreams are like Angels, they keep bad at bay, bad at bay, Love is the Light, scaring Darkness away... From a.badger at gmail.com Wed Oct 1 16:26:30 2008 From: a.badger at gmail.com (Toshio Kuratomi) Date: Wed, 01 Oct 2008 07:26:30 -0700 Subject: [Distutils] Msgfmt in distutils? In-Reply-To: References: Message-ID: <48E38896.6040909@gmail.com> Andrew Price wrote: > Hi, > > Distutils has made my life easier as an python application author and packager > for some time but when it comes to generating and distributing message > catalogues for translations, things get unexpectedly tricky. > > It would be nice to be able to do something like, > > --------- > from distutils.core import setup > > setup(name="foo", > version="1.0", > description="A handy, translated application", > ... > po_files = [('share/locales', ['po/*.po'])] > ) > ---------- > > and have distutils do the right thing with the .po files at build time (generate > .mo files from them) and at install time (install them into > PREFIX/share/locales/LC_MESSAGES/, or wherever the distribution is configured to > put them). > > My big question is, are there plans to add msgfmt functionality into distutils? > Googling tells me there was some discussion about this back in 2003 but I can't > find any sign of progress since then. > This has been a big deal for some applications I work on. Our first cut was to add new Build and InstallData command classes. I'm not happy with how that works at all. Our next pass is going to be moving the build scripts to paver since that will make it easy to both declaratively specify what po files exist and writing the imperative task that actually installs them. http://www.blueskyonmars.com/projects/paver/ -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: OpenPGP digital signature URL: From asmodai at in-nomine.org Wed Oct 1 16:35:57 2008 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Wed, 1 Oct 2008 16:35:57 +0200 Subject: [Distutils] Msgfmt in distutils? In-Reply-To: <48E38896.6040909@gmail.com> References: <48E38896.6040909@gmail.com> Message-ID: <20081001143557.GY30869@nexus.in-nomine.org> -On [20081001 16:28], Toshio Kuratomi (a.badger at gmail.com) wrote: >> and have distutils do the right thing with the .po files at build time >> (generate .mo files from them) and at install time (install them into >> PREFIX/share/locales/LC_MESSAGES/, or wherever the distribution is >> configured to put them). [snip] >This has been a big deal for some applications I work on. Our first cut >was to add new Build and InstallData command classes. Actually with Babel (http://babel.edgewall.org/) that's all handled. -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B Cogito, ergo sum... From pje at telecommunity.com Wed Oct 1 19:29:32 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 01 Oct 2008 13:29:32 -0400 Subject: [Distutils] pre-PEP : Synthesis of previous threads, and irc talks + proposals In-Reply-To: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.co m> References: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.com> Message-ID: <20081001172824.B1E0B3A4072@sparrow.telecommunity.com> At 02:10 PM 10/1/2008 +0200, Tarek Ziad? wrote: >Proposals >======== > >this is also a synthezis of what I hurd, and some elements I have >added to respect the needs that were expressed. > >0/ a lot of work can be done to clean distutils, no matter what is >decided (another PEP is built for that) cleanning, removing old-style >code, testing > >1/ let's change the Python Metadata , in order to introduce a better >dependency system, by > > - officialy introduce "install requires" and "test requires" > metadata in there > - mark "requires" as deprecated > >2/ Let's move part of setuptools code in distutils, to respect those changes. > >3/ let's create a simple convention : the metadata should be expressed >in a python module called 'pkginfo.py' > where each metadata is a variable. > > that can be used by setup.py and therefore by any tool that work >with it, even if it does not run > a setup.py command. > > This is simpler, this is cleaner. you don't have to run some setup >magic to read them. > at least some magic introduces by commands I'm -1 on all of the above. I think we need a standard for tools interop (ala WSGI), not implementation tweaks for the existing tools. I also think that a concrete metadata format proposal is premature at this time; we've barely begun to gather -- let alone specify -- our requirements for that metadata. (Essentially, only version dependencies have been discussed, AFAICT.) There have been many people agreeing that the distutils are thoroughly broken and a new approach is needed; these proposals sound like minor tweaks to the existing infrastructure, rather than a way to get rid of it. So to me, the above doesn't seem like a synthesis of the threads that I've been reading. >4/ let's change PyPI to make it work with the new metadata and to >enforce a few things > >Enforcements: > - a binary distribution cannot be uploaded if a source distrbution >has not been previously provided for the version Note that this doesn't allow closed-source packages to be uploaded; thus it would need to be a warning, rather than a requirement. >New features: > - we should be able to download the metadata of a package without >downloading the package > - PyPI should display the install and test dependencies in the UI It could only do this for specific binaries, since dependencies can be dynamic. From pje at telecommunity.com Wed Oct 1 19:30:46 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 01 Oct 2008 13:30:46 -0400 Subject: [Distutils] Installing single module from src directory In-Reply-To: References: Message-ID: <20081001172937.7C9D73A4072@sparrow.telecommunity.com> At 02:10 PM 10/1/2008 +0200, Dinu Gherman wrote: >Hi, > >I've been using distutils for a while now, but today I'm running >into what seems to be the "minimal strange issue". I want to in- >stall a single Python module without anything else, no package >around it, which resides in a source subdirectory of the main >project directory. My layout looks like the following: > > Macintosh:Python dinu$ tree2.py -f mymodule > mymodule/ > | setup.py > | src/ > | | mymodule.py > >Now common sense says all I need to do is define the module in >setup.py like this: > > Macintosh:mymodule dinu$ more setup.py > from distutils.core import setup > > setup( > py_modules = ["src/mymodule"], > ) Actually, you want: setup( py_modules = ['mymodule'], package_dir = {'': 'src'} ) From pje at telecommunity.com Wed Oct 1 19:35:51 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 01 Oct 2008 13:35:51 -0400 Subject: [Distutils] Question about easy-install.pth In-Reply-To: <48E314C4.1020300@sun.com> References: <48E314C4.1020300@sun.com> Message-ID: <20081001173443.4013A3A40B0@sparrow.telecommunity.com> At 01:12 AM 10/1/2008 -0500, Brian Cameron wrote: >I am having troubles working with setuptools on Solaris. The >Solaris operating system normally installs modules as packages which contain >binaries. This is unlike other Linux operating systems where, for >exmaple, an RPM would download the source and build it on the user's >machine when they install the RPM. > >So, to create packages on Solaris we normally install a module to >a temporary directory such as /var/tmp/pkgbuild-foo/usr, package up >the files that are built, and when the user installs the package >these files are then installed to their system. When you do that, use "setup.py install --root=/var/tmp/pkgbuild-foo" to install the package, and you'll be good to go. That will automatically tell setuptools to install the package in a way that supports being installed like this; any .pth files generated will be named uniquely for that package, rather than using easy-install.pth. In fact, most packages won't generate a .pth file at all when installed this way. Note, too, that the same --root option is also safe for installing plain distutils packages; it's just that setuptools also uses it to enable a distutils-compatible (and packaging tool compatible) installation mode. From ziade.tarek at gmail.com Wed Oct 1 19:55:59 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 1 Oct 2008 19:55:59 +0200 Subject: [Distutils] pre-PEP : Synthesis of previous threads, and irc talks + proposals In-Reply-To: <20081001172824.B1E0B3A4072@sparrow.telecommunity.com> References: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.com> <20081001172824.B1E0B3A4072@sparrow.telecommunity.com> Message-ID: <94bdd2610810011055j69411c6fu6f417cf3cf930460@mail.gmail.com> On Wed, Oct 1, 2008 at 7:29 PM, Phillip J. Eby wrote: > I'm -1 on all of the above. I think we need a standard for tools interop > (ala WSGI), not implementation tweaks for the existing tools. I also think > that a concrete metadata format proposal is premature at this time; we've > barely begun to gather -- let alone specify -- our requirements for that > metadata. (Essentially, only version dependencies have been discussed, > AFAICT.) What are the other important points we need to discuss at this point in your opinion ? > > There have been many people agreeing that the distutils are thoroughly > broken and a new approach is needed; these proposals sound like minor > tweaks to the existing infrastructure, rather than a way to get rid of it. > So to me, the above doesn't seem like a synthesis of the threads that I've > been reading. > > >> 4/ let's change PyPI to make it work with the new metadata and to >> enforce a few things >> >> Enforcements: >> - a binary distribution cannot be uploaded if a source distrbution >> has not been previously provided for the version > > Note that this doesn't allow closed-source packages to be uploaded; thus it > would need to be a warning, rather than a requirement. > Right. do you agree it is something useful to do ? > >> New features: >> - we should be able to download the metadata of a package without >> downloading the package >> - PyPI should display the install and test dependencies in the UI > > It could only do this for specific binaries, since dependencies can be > dynamic. > > What dynamic means here ? the python module to static file process or more ? can you provide an example ? Regards Tarek -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From a.badger at gmail.com Wed Oct 1 20:00:46 2008 From: a.badger at gmail.com (Toshio Kuratomi) Date: Wed, 01 Oct 2008 11:00:46 -0700 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <48E24C28.8020304@simplistix.co.uk> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> Message-ID: <48E3BACE.9020207@gmail.com> You guys are fairly into you debate so hopefully I don't interject something that's already been gone over :-) Chris Withers wrote: > Matthias Klose wrote: >>>> Install debian and get back to productive tasks. >>> This is an almost troll-like answer. >>> See page 35 of the presentation. >> >> I disagree. You could think of "Packages are Pythons Plugins" (taken >> from page 35) as a troll-like statement as well. > > You're welcome to your (incorrect) opinion ;-) > Debian packages could just as easilly be seen as Debian's pluggins. > For a *very* loose definition of plugin, perhaps. But if you look at: http://en.wikipedia.org/wiki/Plugin the idea of Debian packages being plugins is a pretty far stretch. The idea of Packages being python plugins is less of a stretch but I'd call it an analogy. It's useful for looking on things in a new light but if we start designing a plugin interface and only viewing packages through that definition I think we'll be hindering ourselves. >>> - all the package management systems behave differently and expect >>> packages to be set up differently for them >> >> correct, but again they share common requirements. > > ...but all have different implementations. > The common requirements are more important than the varying implementations when thinking about the metadata and how flexible things need to be. When justifying the need for a separate python build tool and distribution format, realizing that there's different implementations is good. ie: we need to expose package naming, versioning, and dependencies to outside tools because they have a common need for that information on the one hand. We have to realize that there's a need for both run-from-egg and run-from-FHS-locations on the other. >> some people prefer to name this "stable releases" instead of >> "bitrot". > > I'll call bullshit on this one. The most common problem I have as a > happy Debian user and advocate when I go to try and get help for a > packaged application (I use packages because I perhaps mistakenly assume > this is the best way to get security-fixed softare), such as postfix, > postgres, and Zope if I was foolish enough to take that path, is "why > are toy using that ancient and buggy version of the software?!" shortly > before pointing out how all the issues I'm facing are solved in newer > (stable) releases. > > The problem is that first the application needs to be tested and > released by its community, then Debian needs to re-package, patch, > generally mess around with it, etc before it eventually gets a "Debian > release". It's bad enough with apps with huge support bases like > portgres, imagine trying to do this "properly" for the 4000-odd packages > on PyPI... > You're correct in the results you're seeing but not in the reason that it exists. There are many linux distributions and each has a different policy of how to update packages. The reason for the variety is that there's demand for both fast package updates and slow package updates. The Debian Stable, Red Hat Enterprise Linux, and other stable, enterprise-oriented distributions' aim is to provide a stable base on which people can build their applications and processes. A common misperception among developers who want faster cycles is that the base system is just a core of packages while things closer to the leaves of the dependency tree could be updated (ie: don't update the kernel; do update the python-sqlalchemy package). What's not seen is that these distributions are providing the base for so many people that updates that change the API/ABI/on-disk format/etc are likely to break *someone* out there. You want to be using one of these systems if you have deployed a major application that serves thousands of people and can afford little to no downtime because you can be more assured that any changes to the system are either changes that are overwhelmingly necessary and the API/ABI breakage has been reduced as much as possible or changes that you yourself have introduced. For system administrators it can also be frustrating due to knowing that there's been bug fixes that are not supposed to change backwards compatibility in newer upstream packages. The problem here is that we all know that all software has bugs. The risk with an update to a newer stable version of software is that the new software has bugs that are as bad or worse than the old one. The package maintainers have to evaluate how many changes have gone into the new version of the software and how big the current problem is and then apply the distribution's policy on updates to that. For a stable enterprise-oriented distro, it's often a case of "better the devil you know than the devil you don't". For a developer of software or someone deploying a new system (as opposed to someone who's had one deployed for several years before they hit a certain bug), this can be quite frustrating as you know that there are fixes and features in newer versions of the software. When you have the choice, then, you should use one of the other Linux distributions either whose focus is on staying closer to what upstream is shipping (I'd recommend this for developers) or one which has a stable policy but has released closer to the current date with newer packages. When you don't have a choice, you have to be prepared for the possibility that you will need to install the requirements for your app from another resource (this could be from another version that the distribution supports like installing debian backports or installing from source or installing an egg). Remember though, that sometimes the distribution will update a package for you if you just request it. It depends on the severity of what's broken currently, the risks involved with updating, and the distribution (and maintainer's) policies/perceptions of the risk vs reward. >> Speaking of extensions "maintained by the entity originating the >> python package": this much too often is a way of bitrot. is the >> shipped library up to date? does it have security fixes? how many >> duplicates are shipped in different extensions? does the library need >> to be shipped at all (because some os does ship it)? > > So what do you propose doing one projectA depends on version 1.0 of libC > and projectB depends on version 2.0 of libC? > This is a problem that is not new for distributions. Each one handles it slightly differently. For Fedora, we've decided the best course is to help upstream port to newer versions of the library. However, since this isn't always practical, we sometimes introduce compatibility packages which have the old version of libraries so older programs will continue to work. Having multiple versions not ideal as this is where bitrot sets in in earnest. If upstream for libC only supports version 2.0 and a security flaw comes out that affects both libC-1.0 and libC-2.0 then we have to fix libC-1.0 at the distribution level. This is more work for us to support something outdated. We'd much rather do work that has a future upstream by porting the application to the newer version. And the time to do that is *not* when there's a security flaw that has to be fixed yesterday. The exact wrong-thing to do (and prohibited in policy in most distributions) is for the applications to have their own copies of the libraries. When a security flaw comes out in that case, we'd have to: 1) hunt through the all the packages we ship to find any that are affected. 2) update the various versions in all of those packages which might mean we have to generate multiple different fixes. 3) rebuild those packages and force our users to redownload all of them. If we had separate library packages for the separate versions we'd: 1) know exactly which packages had to be fixed 2) Only have to apply fixes once to the versions that we were shipping 3) have our users only download the library packages as the applications will load the fixed version from the system. I can go on with other reasons why this is a bad idea and how to mitigate problems but if you're convinced already, I'll surrender the soapbox to someone else :-) >> Considering an extension interfacing a library >> shipped with the os, you do want to use this library, not add another >> copy. > > libxml2 seems to be agood example to use here... > > I guess on debian I'd need to likely install libxml2-dev before I could > install the lxml package... > Note: I'm a Fedora dev, not a Debian dev but the packaging techniques are similar in generalities. You should just be able to request that lxml be installed and it will automatically pull in libxml2. libxml2-dev shouldn't enter the picture as a python program that imports lxml won't need the C headers. (Unless you're talking about *building* lxml which is a separate problem.) > ...what about MacOS X? > > ...what about Windows? > Are you going to be distributing a separate version for MacOS X and Windows anyway since the norm is not to compile from source on those platforms? Then you're already at the point where you have multiple packages for different OS's. A source tarball for unix distributors and a binary zip/binhex/what have you for MacOSX and Windows. >> An upstream >> extension maintainer cannot provide this unless he builds this >> extension for every (os) distribution and maintains it during the os' >> lifecycle. > > ...or just says in the docs "hey, you need libxml2 for this, unless > you're on Windows, in which case the binary includes it". > >> - os distributors usually try to minimize the versions they include, >> trying to just ship one version. > > ...which is fair enough for the "system python", but many of us have a > collection of apps, some of which require Python 2.4, some Python 2.5, > and on top of each of those, different versions of different packages > for each app. > > In my case, I do source (alt-)installs of python rather than trusting > the broken stuff that ships with Debian and buildout to make sure I get > the right versions of the right packages for each project. > So this is fine to a certain extent. Pros: * Allows you to develop new applications using known good or latest versions of other software. * Allows you to deploy an app using newer-than system libraries on an otherwise stable-class distribution. Cons: * You become responsible for the code of all the components your installing. If there's a bug in your alt-install of lxml, you're the one that has to fix it rather than the linux distribution. * If you're distributing this so that everyone can use it, the os packagers are going to have to make sure that the code works with their versions and might have to do porting work. The first Con is the more important one for me. >> - setuptools has the narrow minded view of a python package being >> contained in a single directory, which doesn't fit well when you >> do have common locations for include or doc files. > > Python packages have no idea of "docs" or "includes", which is certainly > a deficiency. > +1. I know I've mentioned paver before but one of the things that it does right is making the declarative metadata extensible. Whereas you can't simply add a new piece of metadata to setup.py's setup() you can add a new Bunch() of metadata in a paver pavement.py file without any other code. This makes it easy to do the right thing and write code to operate on "docs", "includes", "locales", etc that you've defined declaratively in the metadata section. >> way packaging the python module with rpm or dpkg. E.g. namespace >> packages are a consequence how setuptools distributes and installs >> things. Why force this on everybody? > > being able to break a large lump (say zope.*) into seperate > distributions is a good idea, which setuptools implements very badly > using namespace packages... > >> A big win could be a modularized setuptools where you are able to only >> use the things you do want to use, e.g. >> >> - version specifications (not just the heuristics shipped with >> setuptools). > > not sure what you mean by this. > I'm not 100% certain of what Matthias means but there's several problems with seutptools usage of versions: 1) The heuristic encourages bad practices. Versions need to be parsed by computer programs (package managers, scripts that maintain repositories, etc). Not all of those are written in python. Having things other than letters and dots in version strings is problematic for these programs. For instance, here's something that setuptools versioning heuristics allow you to do: foo-1.0rc1 foo-1.0 foo-1.0post1 But here's how rpm would order it: foo-1.0 foo-1.0post1 foo-1.0rc1 In Fedora we have rules for puting non-numeric things in our release tag to work around this: version: 1.0 , release: 0.1.rc1 version: 1.0 , release: 1 version: 1.0 , release: 2.post1 This is not all inclusive, but you can see, we have to move the alpha portion of the version to the release to ensure that the upgrade path will move forward sensibly. 2) This is more important but much harder. Something that would really help everyone is having a way of versioning API/ABI. Right now you can specify that you depend on Foo >= 1.0 Foo <= 2.0. But the version numbers don't have meaning until the actual packages are released. If Foo-1.0 and Foo-1.1 don't have compatible API, your numbers are wrong. If Foo-1.0 is succeeded by Foo-2.0 with the same API your numbers are too restrictive. If you lock the versions to only what you've tested: Foo = 1.0 then you're going to have people and distributions that want to use the new version but can't. Some places have good versioning rules:: https://svn.enthought.com/enthought/wiki/EnthoughtVersionNumbers Other places say they have marketing departments that prevent that One possibility would be to have MyLib1-1.0, MyLib2-1.0, MyLib2-2.0, etc with the version for marketing included in the package name. Another idea would be to have API information stored in metadata but not in the package name. That way marketing can have a big party for MyLib-2.0 but the API metadata has API_Revision: 32. >> - specification of dependencies. >> >> - resource management > > ? > http://peak.telecommunity.com/DevCenter/PythonEggs#accessing-package-resources I have no love for how pkg_resources implements this (including the API) but the idea of retrieving data files, locales, config files, etc from an API is good. For packages to be coded that conform to the File Hierachy Standard on Linux, the API (and metadata) needs to be more flexible. We need to be able to mark locale, config, and data files in the metadata. The build/install tool needs to be able to install those into the filesystem in the proper places for a Linux distro, an egg, etc. and then we need to be able to call an API to retrieve the specific class of resources or a directory associated with them. use cases: * config files go to /etc on Linux and we'd want to retrieve the contents of /etc/configfile * generic, architecture-independent data files go under /usr/share/. We'd want to place them in or under /usr/share/$PACKAGENAME. Mostly we're going to want to retrieve the contents of a specific data file. * locale files go under /usr/share/locale/ (ex: /usr/share/locale/en_US/LC_MESSAGES/compiz.mo) We'll want to retrieve the directory '/usr/share/locale' for feeding to gettext. >> - a module system independent from any distribution specific stuff. > > ? I read this as "entry_points is a good feature". > >> - any other distribution specific stuff. > > ? > I think Matthias is trying to separate out the different services that setuptools provides so that they can be decoupled and worked on separately. So "other distribution specific stuff" would be things to do with distributing the results of your labors. eggs and pypi would fall under this. Matthias, if I'm wrong in any of this, please correct me :-). These are my perceptions due to them being the issues I have as a pakckger for a different distribution. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: OpenPGP digital signature URL: From a.badger at gmail.com Wed Oct 1 20:14:35 2008 From: a.badger at gmail.com (Toshio Kuratomi) Date: Wed, 01 Oct 2008 11:14:35 -0700 Subject: [Distutils] Question about easy-install.pth In-Reply-To: <48E314C4.1020300@sun.com> References: <48E314C4.1020300@sun.com> Message-ID: <48E3BE0B.90500@gmail.com> Brian Cameron wrote: > You're actually describing pretty much exactly how rpm works :-) As such, the Fedora Project has had to deal with many of the same issues in packaging python packages as you're running into. This pairs of pages may help you: http://fedoraproject.org/wiki/Packaging/Python http://fedoraproject.org/wiki/Packaging/Python/Eggs Note that we're only one Linux distribution. Other Linux distributions have had to tackle these issues in similar ways. I know that Debian, for instance, also has a comprehensive set of Guidelines which includes useful information on packaging python modules. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: OpenPGP digital signature URL: From a.badger at gmail.com Wed Oct 1 20:25:03 2008 From: a.badger at gmail.com (Toshio Kuratomi) Date: Wed, 01 Oct 2008 11:25:03 -0700 Subject: [Distutils] Msgfmt in distutils? In-Reply-To: <20081001143557.GY30869@nexus.in-nomine.org> References: <48E38896.6040909@gmail.com> <20081001143557.GY30869@nexus.in-nomine.org> Message-ID: <48E3C07F.1080605@gmail.com> Jeroen Ruigrok van der Werven wrote: > -On [20081001 16:28], Toshio Kuratomi (a.badger at gmail.com) wrote: >>> and have distutils do the right thing with the .po files at build time >>> (generate .mo files from them) and at install time (install them into >>> PREFIX/share/locales/LC_MESSAGES/, or wherever the distribution is >>> configured to put them). > > [snip] > >> This has been a big deal for some applications I work on. Our first cut >> was to add new Build and InstallData command classes. > > Actually with Babel (http://babel.edgewall.org/) that's all handled. > That's good to know. One of our Turbogears applications uses Babel and it definitely doesn't install to the right place. I'd love to fix it to take advantage of Babel' properly. Would you be kind enough to point me documentation on how to get Babel to install locale files? Looking at the babel website, I only see documentation up to building the message catalogs. If the install portion is integrated into setuptools is there something I might have to configure in setup() to tell babel/setuptools what directory to use? -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: OpenPGP digital signature URL: From pje at telecommunity.com Wed Oct 1 20:28:07 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 01 Oct 2008 14:28:07 -0400 Subject: [Distutils] pre-PEP : Synthesis of previous threads, and irc talks + proposals In-Reply-To: <94bdd2610810011055j69411c6fu6f417cf3cf930460@mail.gmail.co m> References: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.com> <20081001172824.B1E0B3A4072@sparrow.telecommunity.com> <94bdd2610810011055j69411c6fu6f417cf3cf930460@mail.gmail.com> Message-ID: <20081001182658.E96173A4072@sparrow.telecommunity.com> At 07:55 PM 10/1/2008 +0200, Tarek Ziad? wrote: >On Wed, Oct 1, 2008 at 7:29 PM, Phillip J. Eby wrote: > > I'm -1 on all of the above. I think we need a standard for tools interop > > (ala WSGI), not implementation tweaks for the existing tools. I also think > > that a concrete metadata format proposal is premature at this time; we've > > barely begun to gather -- let alone specify -- our requirements for that > > metadata. (Essentially, only version dependencies have been discussed, > > AFAICT.) > >What are the other important points we need to discuss at this point >in your opinion ? What information needs to be conveyed by a "build" tool to an "install" tool, and vice versa. For example, an install tool needs to know what files are documentation, which are sample data, and what is part of the library proper, as well as what things are scripts (and in what language those scripts are written, e.g. Python, shell, batch, etc.). Some install tools need to know about icons, menus, registry info, cron jobs, etc. (These are perhaps more properly the domain of applications than libraries, but I'm going to assume that these things are in scope.) The way that this information is communicated needs to be extensible, so that optional metadata for debs and msi's and rpm's and whathaveyou can be incorporated, without needing to modify the standard -- especially if the APIs for reading and writing this data are in the stdlib. There needs to be a way for install tools to ask a source package to "configure" itself, possibly specifying options (and a way for it to find out what those options are, to be able to present them to the user). Then there needs to be a way for the install tool to ask the package to build itself with the configured options, and a standard for how the build tool(s) communicate errors or other issues back to the install tools. There needs to be a way, of course, for the package to specify what build tools it needs in order to be built, and for those to plug in to the (again stdlib-contained) build API. There needs to be a better API for querying C configuration and compiler details, that's separate from the distutils "ccompiler" stuff. Last, but not least, there needs to be a definition of core build and install tools to be both an example/reference implementation of the standard, and to provide the basics needed by the core. I think I've mentioned all of these previously in the thread. I also think that as a matter of technicalities, these things are not difficult to achieve... but if it they are only achieved *technically*, then the result is a failure, not a success. In order for the *real* goal to be achieved (i.e., a flourishing build/install system for Python), widespread participation and buy-in is required. If the OS people or the big package people (e.g. Zope Corp., Enthought) or the distutils aficionados are left out, then the result won't get used. I think the best way to ensure that nobody is left out, is to get them to participate in the design of a standard that ensures that *they* will be able to control their destiny, by creating their own build plugins and/or install tools... or at least having a robust choice of alternatives. We need a consensus "de jure" standard (ala WSGI), rather than just an ongoing "de facto" standard (ala distutils/setuptools), or we're not making any substantial progress, just handing the reins over to somebody else. > >> Enforcements: > >> - a binary distribution cannot be uploaded if a source distrbution > >> has not been previously provided for the version > > > > Note that this doesn't allow closed-source packages to be uploaded; thus it > > would need to be a warning, rather than a requirement. > > > >Right. do you agree it is something useful to do ? Sure. > >> New features: > >> - we should be able to download the metadata of a package without > >> downloading the package > >> - PyPI should display the install and test dependencies in the UI > > > > It could only do this for specific binaries, since dependencies can be > > dynamic. > > > > > >What dynamic means here ? the python module to static file process or more ? >can you provide an example ? Dependencies can be platform-specific as well as python-version specific. If you want ctypes, you would depend on it in python 2.4 but not in 2.5. Similarly, if on some platform a certain library is required to implement a feature that is natively accessible on other platforms. In these cases, you would have logic in setup.py to detect this and choose the appropriate dependencies. From pje at telecommunity.com Wed Oct 1 20:39:03 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 01 Oct 2008 14:39:03 -0400 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <48E3BACE.9020207@gmail.com> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> Message-ID: <20081001183754.84BCE3A4072@sparrow.telecommunity.com> At 11:00 AM 10/1/2008 -0700, Toshio Kuratomi wrote: >I have no love for how pkg_resources implements this (including the API) >but the idea of retrieving data files, locales, config files, etc from >an API is good. For packages to be coded that conform to the File >Hierachy Standard on Linux, the API (and metadata) needs to be more >flexible. There's some confusion here. pkg_resources implements *resource* management and *metadata* management... NOT "file management". Resource files and metadata are no more "data" in the FHS sense than static data segments in a .so file are; they are simply a more convenient way of including such data than having a giant base64 string or something like that hardcoded into the program itself. There is thus no relevance to the FHS and absolutely no reason for them to live anywhere except within the Python packages they are a part of. > We need to be able to mark locale, config, and data files in >the metadata. Sure... and having a standard for specifying that kind of application/system-level install stuff is great; it's just entirely outside the scope of what eggs are for. To be clear, I mean here that a "file" (as opposed to a resource) is something that the user is expected to be able to read or copy, or modify. (Whereas a resource is something that is entirely internal to a library, and metadata is information *about* the library itself.) > The build/install tool needs to be able to install those >into the filesystem in the proper places for a Linux distro, an egg, >etc. and then we need to be able to call an API to retrieve the >specific class of resources or a directory associated with them. Agreed... assuming of course that we're keeping a clear distinction between static resources+metadata and actual "data" (e.g. configuration) files. From Brian.Cameron at Sun.COM Wed Oct 1 20:46:01 2008 From: Brian.Cameron at Sun.COM (Brian Cameron) Date: Wed, 01 Oct 2008 13:46:01 -0500 Subject: [Distutils] Question about easy-install.pth In-Reply-To: <20081001173443.4013A3A40B0@sparrow.telecommunity.com> References: <48E314C4.1020300@sun.com> <20081001173443.4013A3A40B0@sparrow.telecommunity.com> Message-ID: <48E3C569.5060906@sun.com> Phillip & Others: Thanks so much for taking the time to answer my questions, I am a newbie to setuptools so I apologize if I was just being thick. Phillip, your suggested solution of using --root solved my problem. I was previously using --prefix instead, but using --root creates the packages more in the way that work well with Solaris packaging. Brian > When you do that, use "setup.py install --root=/var/tmp/pkgbuild-foo" to > install the package, and you'll be good to go. That will automatically > tell setuptools to install the package in a way that supports being > installed like this; any .pth files generated will be named uniquely for > that package, rather than using easy-install.pth. In fact, most > packages won't generate a .pth file at all when installed this way. > > Note, too, that the same --root option is also safe for installing plain > distutils packages; it's just that setuptools also uses it to enable a > distutils-compatible (and packaging tool compatible) installation mode. > From asmodai at in-nomine.org Wed Oct 1 20:49:54 2008 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Wed, 1 Oct 2008 20:49:54 +0200 Subject: [Distutils] Msgfmt in distutils? In-Reply-To: <48E3C07F.1080605@gmail.com> References: <48E38896.6040909@gmail.com> <20081001143557.GY30869@nexus.in-nomine.org> <48E3C07F.1080605@gmail.com> Message-ID: <20081001184954.GZ30869@nexus.in-nomine.org> -On [20081001 20:26], Toshio Kuratomi (a.badger at gmail.com) wrote: >That's good to know. One of our Turbogears applications uses Babel and >it definitely doesn't install to the right place. I'd love to fix it to >take advantage of Babel' properly. Would you be kind enough to point me >documentation on how to get Babel to install locale files? Leaving aside the extractors for now, as far as I know either adding Babel to the requires or adding an import babel should suffice. You should get, when you do python setup.py --help-commands, a few options for catalog management. A simple compile_catalog (perhaps with the force flag) should create .mo files, then follow up with a normal build and install. Currently the locale catalog files are packaged inside the eggs. Feature requests and all that all welcome at the Babel site. :) -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B Forging the Future from the Timeless Stone... From andy at andrewprice.me.uk Wed Oct 1 21:01:10 2008 From: andy at andrewprice.me.uk (Andrew Price) Date: Wed, 01 Oct 2008 20:01:10 +0100 Subject: [Distutils] Msgfmt in distutils? In-Reply-To: <20081001143557.GY30869@nexus.in-nomine.org> References: <48E38896.6040909@gmail.com> <20081001143557.GY30869@nexus.in-nomine.org> Message-ID: Hi, On 01/10/08 15:35, Jeroen Ruigrok van der Werven wrote: > -On [20081001 16:28], Toshio Kuratomi (a.badger at gmail.com) wrote: >> This has been a big deal for some applications I work on. Our first cut >> was to add new Build and InstallData command classes. > > Actually with Babel (http://babel.edgewall.org/) that's all handled. I came across Babel during my search for a solution before writing my original post. But for relatively small projects such as mine, it seems overkill to depend on something as large as Babel just for installing message catalogues when distutils already does everything else I want to do, hence my original query. I've since added some code to my setup.py which adds a build_mo command and adds .mo installation to the install command. It'll probably need a bit more development for systems with which I'm not familiar but it's doing what I need for now. It still would be "killer" for this functionality to be handled in distutils. Regards, -- Andy Price From gael.varoquaux at normalesup.org Wed Oct 1 21:05:59 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 1 Oct 2008 21:05:59 +0200 Subject: [Distutils] pre-PEP : Synthesis of previous threads, and irc talks + proposals In-Reply-To: <20081001182658.E96173A4072@sparrow.telecommunity.com> References: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.com> <20081001172824.B1E0B3A4072@sparrow.telecommunity.com> <94bdd2610810011055j69411c6fu6f417cf3cf930460@mail.gmail.com> <20081001182658.E96173A4072@sparrow.telecommunity.com> Message-ID: <20081001190559.GB585@phare.normalesup.org> On Wed, Oct 01, 2008 at 02:28:07PM -0400, Phillip J. Eby wrote: > In order for the *real* goal to be achieved (i.e., a flourishing > build/install system for Python), widespread participation and buy-in is > required. If the OS people or the big package people (e.g. Zope Corp., > Enthought) or the distutils aficionados are left out, then the result won't > get used. > > [...] > > We need a consensus "de jure" standard (ala WSGI), rather than just an > ongoing "de facto" standard (ala distutils/setuptools), or we're not making > any substantial progress, just handing the reins over to somebody else. Nice words. I like very much the sound of this discussion. Ga?l From pje at telecommunity.com Wed Oct 1 21:25:21 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 01 Oct 2008 15:25:21 -0400 Subject: [Distutils] Question about easy-install.pth In-Reply-To: References: <48E314C4.1020300@sun.com> <20081001173443.4013A3A40B0@sparrow.telecommunity.com> Message-ID: <20081001192413.B2EEB3A4072@sparrow.telecommunity.com> At 02:15 PM 10/1/2008 -0500, chris wrote: >Is it just me, or does --root work the same way >--single-version-externally-managed does, just with the addition of >the directory? --root automatically triggers --single-version-externally-managed, in addition to distutils' normal handling of --root. From joss at debian.org Wed Oct 1 21:36:10 2008 From: joss at debian.org (Josselin Mouette) Date: Wed, 01 Oct 2008 21:36:10 +0200 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <48E3BACE.9020207@gmail.com> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> Message-ID: <1222889771.7060.50.camel@tomoyo> Le mercredi 01 octobre 2008 ? 11:00 -0700, Toshio Kuratomi a ?crit : > 1) The heuristic encourages bad practices. Versions need to be parsed > by computer programs (package managers, scripts that maintain > repositories, etc). Not all of those are written in python. Having > things other than letters and dots in version strings is problematic for > these programs. For instance, here's something that setuptools > versioning heuristics allow you to do: > > foo-1.0rc1 > foo-1.0 > foo-1.0post1 > > But here's how rpm would order it: > foo-1.0 > foo-1.0post1 > foo-1.0rc1 > > In Fedora we have rules for puting non-numeric things in our release tag > to work around this: > > version: 1.0 , release: 0.1.rc1 > version: 1.0 , release: 1 > version: 1.0 , release: 2.post1 > > This is not all inclusive, but you can see, we have to move the alpha > portion of the version to the release to ensure that the upgrade path > will move forward sensibly. In Debian, we have the ~ character to handle such versions easily: foo_1.0~rc1 << foo_1.0 << foo_1.0post1 In all cases, this is not something that matters much to us since the version of the tarball in Debian does not need to be the same as that of the upstream one; for example, 1.0rc1 will always be renamed to 1.0~rc1. > 2) This is more important but much harder. Something that would really > help everyone is having a way of versioning API/ABI. Right now you can > specify that you depend on Foo >= 1.0 Foo <= 2.0. But the version > numbers don't have meaning until the actual packages are released. If > Foo-1.0 and Foo-1.1 don't have compatible API, your numbers are wrong. > If Foo-1.0 is succeeded by Foo-2.0 with the same API your numbers are > too restrictive. If you lock the versions to only what you've tested: > Foo = 1.0 then you're going to have people and distributions that want > to use the new version but can't. Some places have good versioning rules:: > https://svn.enthought.com/enthought/wiki/EnthoughtVersionNumbers > > Other places say they have marketing departments that prevent that One > possibility would be to have MyLib1-1.0, MyLib2-1.0, MyLib2-2.0, etc > with the version for marketing included in the package name. > > Another idea would be to have API information stored in metadata but not > in the package name. That way marketing can have a big party for > MyLib-2.0 but the API metadata has API_Revision: 32. Fully agreed. We need to be able to specify an API version one way or the other. The simple way is to set naming rules that postfix the marketing name with such an API version, but I don?t think developers are going to follow that, so it needs a more formalized way to be specified. > I have no love for how pkg_resources implements this (including the API) > but the idea of retrieving data files, locales, config files, etc from > an API is good. For packages to be coded that conform to the File > Hierachy Standard on Linux, the API (and metadata) needs to be more > flexible. We need to be able to mark locale, config, and data files in > the metadata. The build/install tool needs to be able to install those > into the filesystem in the proper places for a Linux distro, an egg, > etc. and then we need to be able to call an API to retrieve the > specific class of resources or a directory associated with them. Amen to that. Currently, we find all kind of data and configuration files right into the python package directories, and this must really stop. The python-specific package management tools wouldn?t need to be so complex if you only found python modules in the python modules directory. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From joss at debian.org Wed Oct 1 21:40:32 2008 From: joss at debian.org (Josselin Mouette) Date: Wed, 01 Oct 2008 21:40:32 +0200 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <20081001183754.84BCE3A4072@sparrow.telecommunity.com> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> Message-ID: <1222890032.7060.55.camel@tomoyo> Le mercredi 01 octobre 2008 ? 14:39 -0400, Phillip J. Eby a ?crit : > > We need to be able to mark locale, config, and data files in > >the metadata. > > Sure... and having a standard for specifying that kind of > application/system-level install stuff is great; it's just entirely > outside the scope of what eggs are for. I don?t follow you. If the library needs these files to work, you definitely want to ship them, whether it is as their FHS locations in a package, or in the egg. > To be clear, I mean here that a "file" (as opposed to a resource) is > something that the user is expected to be able to read or copy, or > modify. (Whereas a resource is something that is entirely internal > to a library, and metadata is information *about* the library itself.) It?s not as simple as that. Python is not the only thing out there, and there are many times where your resources need to be shipped in existing formats, in files that land at specific places. For example icons go in /usr/share/icons, locale files in .mo format in /usr/share/locale, etc. -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From pjenvey at underboss.org Wed Oct 1 21:43:32 2008 From: pjenvey at underboss.org (Philip Jenvey) Date: Wed, 1 Oct 2008 12:43:32 -0700 Subject: [Distutils] Msgfmt in distutils? In-Reply-To: <48E3C07F.1080605@gmail.com> References: <48E38896.6040909@gmail.com> <20081001143557.GY30869@nexus.in-nomine.org> <48E3C07F.1080605@gmail.com> Message-ID: <8CCEF1EF-7B9F-4B55-B3CA-D548B3A494B4@underboss.org> On Oct 1, 2008, at 11:25 AM, Toshio Kuratomi wrote: > Jeroen Ruigrok van der Werven wrote: >> -On [20081001 16:28], Toshio Kuratomi (a.badger at gmail.com) wrote: >>>> and have distutils do the right thing with the .po files at build >>>> time >>>> (generate .mo files from them) and at install time (install them >>>> into >>>> PREFIX/share/locales/LC_MESSAGES/, or wherever the distribution is >>>> configured to put them). >> >> [snip] >> >>> This has been a big deal for some applications I work on. Our >>> first cut >>> was to add new Build and InstallData command classes. >> >> Actually with Babel (http://babel.edgewall.org/) that's all handled. >> > That's good to know. One of our Turbogears applications uses Babel > and > it definitely doesn't install to the right place. I'd love to fix > it to > take advantage of Babel' properly. Would you be kind enough to > point me > documentation on how to get Babel to install locale files? Looking at > the babel website, I only see documentation up to building the message > catalogs. If the install portion is integrated into setuptools is > there > something I might have to configure in setup() to tell babel/ > setuptools > what directory to use? Once you have Babel generating .mo files, all you'll need is a package_data entry for them, e.g.: package_data={'foo': ['i18n/*/LC_MESSAGES/*.mo']}, then the catalogs will make it into the final sdist/egg and be included during an installation. -- Philip Jenvey From a.badger at gmail.com Wed Oct 1 21:55:54 2008 From: a.badger at gmail.com (Toshio Kuratomi) Date: Wed, 01 Oct 2008 12:55:54 -0700 Subject: [Distutils] Msgfmt in distutils? In-Reply-To: <20081001184954.GZ30869@nexus.in-nomine.org> References: <48E38896.6040909@gmail.com> <20081001143557.GY30869@nexus.in-nomine.org> <48E3C07F.1080605@gmail.com> <20081001184954.GZ30869@nexus.in-nomine.org> Message-ID: <48E3D5CA.6010904@gmail.com> Jeroen Ruigrok van der Werven wrote: > -On [20081001 20:26], Toshio Kuratomi (a.badger at gmail.com) wrote: >> That's good to know. One of our Turbogears applications uses Babel and >> it definitely doesn't install to the right place. I'd love to fix it to >> take advantage of Babel' properly. Would you be kind enough to point me >> documentation on how to get Babel to install locale files? > > Leaving aside the extractors for now, as far as I know either adding Babel > to the requires or adding an import babel should suffice. > > You should get, when you do python setup.py --help-commands, a few options > for catalog management. > > A simple compile_catalog (perhaps with the force flag) should create .mo > files, then follow up with a normal build and install. Currently the locale > catalog files are packaged inside the eggs. Ah. I think you missed this from the original poster:: >and at install time (install them into > PREFIX/share/locales/LC_MESSAGES/, or wherever the distribution is configured to > put them). As for what I'm seeing... It doesn't look like my locales are being installed at all if I just rely on babel. Perhaps you are putting the mo files into package data normally and then the standard install_data is putting it in the egg? This definitely isn't what we want and is why I'm so gung-ho about moving over to paver. > > Feature requests and all that all welcome at the Babel site. :) > I suppose it depends on whether installation of .mo files is part of Babel's goal. Since I only saw information about extracting messages, creating, building, and maintaining catalogs I didn't know if that was part of what babel was aiming for. Thanks! -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: OpenPGP digital signature URL: From pje at telecommunity.com Wed Oct 1 22:57:43 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 01 Oct 2008 16:57:43 -0400 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <1222890032.7060.55.camel@tomoyo> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <1222890032.7060.55.camel@tomoyo> Message-ID: <20081001205645.A025C3A4072@sparrow.telecommunity.com> At 09:40 PM 10/1/2008 +0200, Josselin Mouette wrote: >Le mercredi 01 octobre 2008 ? 14:39 -0400, Phillip J. Eby a ??crit : > > > We need to be able to mark locale, config, and data files in > > >the metadata. > > > > Sure... and having a standard for specifying that kind of > > application/system-level install stuff is great; it's just entirely > > outside the scope of what eggs are for. > >I don???t follow you. If the library needs these files to work, you >definitely want to ship them, whether it is as their FHS locations in a >package, or in the egg. Egg files aren't an all-purpose distribution format; they were designed for application plugins, and for libraries needed to support application plugins. As such, they're self-contained and weren't designed for application-level installation support, such as documentation, configuration or data files, icons, etc. As has been pointed out, these are deficiencies of .egg files wrt the full spectrum of library and application installation needs, which is why I'm pushing for us working on an installation metadata standard that can accommodate these other needs that the .egg layout isn't really suited for. My main point about the resources is simply that it's a needless complication to physically separate static data needed by a library at runtime, based solely on its file extension, in cases where only that library will be reading that file, and the file's contents are constant for that version of the library. To put it another way, if some interpretation of the FHS makes a distinction between two files encoding the same data, one named foo.bar and foo.py, where the only difference between the two is the internal encoding of the data, then that interpretation of the FHS is not based on any real requirement, AFAICT. Of course, for documentation, application icons, and suchlike, the data *will* be read by things other than the library itself, and so a standardized location is appropriate. The .egg format was designed primarily to support resources read only by the package in question, and secondarily to support metadata needed by applications or libraries that the package "plugs in" to. It was not originally intended to be an general-purpose system package installation format. > > To be clear, I mean here that a "file" (as opposed to a resource) is > > something that the user is expected to be able to read or copy, or > > modify. (Whereas a resource is something that is entirely internal > > to a library, and metadata is information *about* the library itself.) > >It???s not as simple as that. Python is not the only thing out there, and >there are many times where your resources need to be shipped in existing >formats, in files that land at specific places. For example icons go >in /usr/share/icons, locale files in .mo format in /usr/share/locale, >etc. And docs need to go in /usr/share/doc, I presume. But these aren't necessarily "resources" in the way I'm defining the term. Some of them *could* be, perhaps. Others aren't. To be clear, what I'm trying to say is that it is a perfectly valid use case for a Python package author to have static data contained within their Python package directory layout for purposes of accessing that data as if it were code, but without having to go to the trouble of converting it to a .py file (and possibly having to extract it back out at runtime). This usage of "data" files isn't in conflict with the FHS, as I understand it. But I also understand that there are other kinds of "data" files which *don't* fall under that use case, and which it is desirable to install to shared locations. We need to support both. From a.badger at gmail.com Thu Oct 2 00:14:40 2008 From: a.badger at gmail.com (Toshio Kuratomi) Date: Wed, 01 Oct 2008 15:14:40 -0700 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <20081001183754.84BCE3A4072@sparrow.telecommunity.com> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> Message-ID: <48E3F650.6080309@gmail.com> Phillip J. Eby wrote: > At 11:00 AM 10/1/2008 -0700, Toshio Kuratomi wrote: >> I have no love for how pkg_resources implements this (including the API) >> but the idea of retrieving data files, locales, config files, etc from >> an API is good. For packages to be coded that conform to the File >> Hierachy Standard on Linux, the API (and metadata) needs to be more >> flexible. > > There's some confusion here. pkg_resources implements *resource* > management and *metadata* management... NOT "file management". > > Resource files and metadata are no more "data" in the FHS sense than > static data segments in a .so file are; they are simply a more > convenient way of including such data than having a giant base64 string > or something like that hardcoded into the program itself. There is thus > no relevance to the FHS and absolutely no reason for them to live > anywhere except within the Python packages they are a part of. > If we can agree on a definition of resource files there's a case to be made here. One of the problems, though, is that people use pkg_resources for things that are data. Now there could be two reasons for that: 1) Developers are abusing pkg_resources. 2) Linux distributions disagree with you on what consitutes data vs a resource. Let's discuss the definition of resource vs data below (since you made a good start at it) and we can see which of these it is. > >> We need to be able to mark locale, config, and data files in >> the metadata. > > Sure... and having a standard for specifying that kind of > application/system-level install stuff is great; it's just entirely > outside the scope of what eggs are for. > > To be clear, I mean here that a "file" (as opposed to a resource) is > something that the user is expected to be able to read or copy, or > modify. (Whereas a resource is something that is entirely internal to a > library, and metadata is information *about* the library itself.) > metadata, I haven't even begun to think about yet. I personally don't see a huge need to shift it around on the filesystem but someone who's thought about it longer might find reasons that it belongs in some other place. resources, as I said needs to be defined. You're saying here that a resource is something internal to the library. A "file" is something that a user can read, copy, or modify. In a typical TurboGears app, there's the following things to be found inside of the app's directory in site-packages: config/{app.cfg,__init__.py,log.cfg} - These could go in /etc/ as their configuration. However, I've tried to stress to upstream that only things that configure the TurboGears framework for use with their app should go in these files (default templating language, identity controller). When those things are true, I can see this as being an "internal resource". If upstream can't get their act together, it's config. locale/{message catalogs for various languages} -- These are binary files that contain strings that the user may see when a message is given. These, I think are data. templates/*html -- These are templates that the application fills in with variables intermixed with short bits of code. These are on the border between code and data. The user sees them in a modified form. The app sometimes executes pieces of them before the user sees them. Some template languages create python byte code from the templates, others load them and write into them every time. None of them can be executed on their own. All of them have to be loaded by a call to parse them from a piece of python code in another file. None of them are directly called or invoked. My leaning is that these are data. static/{javascript,css,images} -- These are things that are definitely never executed. They are served by the webserver verbatim when their URL is called. These are certainly data. (Note: I don't believe these are referenced using the resources API, just via URL.) So... do you agree on which of these are data and which are resources? Do you have an idea on how we can prevent application and framework writers from misusing the resources API to load things that are data? > >> The build/install tool needs to be able to install those >> into the filesystem in the proper places for a Linux distro, an egg, >> etc. and then we need to be able to call an API to retrieve the >> specific class of resources or a directory associated with them. > > Agreed... assuming of course that we're keeping a clear distinction > between static resources+metadata and actual "data" (e.g. configuration) > files. > > . The definition and distinction is important. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: OpenPGP digital signature URL: From a.badger at gmail.com Thu Oct 2 00:53:19 2008 From: a.badger at gmail.com (Toshio Kuratomi) Date: Wed, 01 Oct 2008 15:53:19 -0700 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <20081001205645.A025C3A4072@sparrow.telecommunity.com> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <1222890032.7060.55.camel@tomoyo> <20081001205645.A025C3A4072@sparrow.telecommunity.com> Message-ID: <48E3FF5F.2010207@gmail.com> Phillip J. Eby wrote: > At 09:40 PM 10/1/2008 +0200, Josselin Mouette wrote: >> Le mercredi 01 octobre 2008 ? 14:39 -0400, Phillip J. Eby a ??crit : >> > > We need to be able to mark locale, config, and data files in >> > >the metadata. >> > >> > Sure... and having a standard for specifying that kind of >> > application/system-level install stuff is great; it's just entirely >> > outside the scope of what eggs are for. >> >> I don???t follow you. If the library needs these files to work, you >> definitely want to ship them, whether it is as their FHS locations in a >> package, or in the egg. > > Egg files aren't an all-purpose distribution format; they were designed > for application plugins, and for libraries needed to support application > plugins. As such, they're self-contained and weren't designed for > application-level installation support, such as documentation, > configuration or data files, icons, etc. > > As has been pointed out, these are deficiencies of .egg files wrt the > full spectrum of library and application installation needs, which is > why I'm pushing for us working on an installation metadata standard that > can accommodate these other needs that the .egg layout isn't really > suited for. > We need to get the list of problems up somewhere on the wiki so that people can check that the evolving standard doesn't fall into the same pitfalls. After all, people are using the egg and pkg_resources API for just this purpose today with some happy about it and others not so much. > My main point about the resources is simply that it's a needless > complication to physically separate static data needed by a library at > runtime, based solely on its file extension, in cases where only that > library will be reading that file, and the file's contents are constant > for that version of the library. > > To put it another way, if some interpretation of the FHS makes a > distinction between two files encoding the same data, one named foo.bar > and foo.py, where the only difference between the two is the internal > encoding of the data, then that interpretation of the FHS is not based > on any real requirement, AFAICT. > Actually, file encoding is one major criteria in the FHS. However, it's probably not in the manner you're thinking of :-) Files which are architecture dependent generally need to be separated from files which are architecture independent. Since text files and binary data which has a standard byte-oriented format are generally what's used as data these days it's the major reason that data files usually go in /usr/share while libraries/binaries go in /usr/lib and /usr/bin. This is dues to the range of computers that architecture dependent vs architecture independent data can be shared with. Of course, part of python's site-packages on Linux systems violates this rule as python can split architecture dependent and architecture independent packages from one another. I know that some distributions have debated moving the architecture independent portion of site-packages to /usr/share although I don't know if any have (Josselin, has Debian done this?) The idea of moving is not straight forward because of 1) compatibility with unpackaged software and 2) /usr/share is seen in two lights: the place for architecture independent files and the place for data; /usr/lib is seen in two lights: the place for architecture dependent non-executables and the place for code whose instructions are run by executables. > Of course, for documentation, application icons, and suchlike, the data > *will* be read by things other than the library itself, and so a > standardized location is appropriate. The .egg format was designed > primarily to support resources read only by the package in question, and > secondarily to support metadata needed by applications or libraries that > the package "plugs in" to. It was not originally intended to be an > general-purpose system package installation format. > . Despite this design, it's presently being used for that. So we need to figure out what to do about it. > >> > To be clear, I mean here that a "file" (as opposed to a resource) is >> > something that the user is expected to be able to read or copy, or >> > modify. (Whereas a resource is something that is entirely internal >> > to a library, and metadata is information *about* the library itself.) >> >> It???s not as simple as that. Python is not the only thing out there, and >> there are many times where your resources need to be shipped in existing >> formats, in files that land at specific places. For example icons go >> in /usr/share/icons, locale files in .mo format in /usr/share/locale, >> etc. > > And docs need to go in /usr/share/doc, I presume. docs are special in the packaging world on several accounts. Generally the packager has to collect at least some of the docs themselves (as things like LICENSE.txt aren't normally included in a doc install but are important for distributions to package.) rpm, at least provides a macro to make it easy for the packager to mark files and directories from the source tree as documentation which rpm will put in the appropriate directory itself. So packagers often use an upstream's build scripts to build the docs, but usually install the docs using the package tool's facilities. Additionally, there's a difference between docs which the program uses (for instance for online help) and docs which the end user would have to navigate the filesystem and invoke a viewer themselves to read. The former is data, the latter is docs. > But these aren't > necessarily "resources" in the way I'm defining the term. Some of them > *could* be, perhaps. Others aren't. > > To be clear, what I'm trying to say is that it is a perfectly valid use > case for a Python package author to have static data contained within > their Python package directory layout for purposes of accessing that > data as if it were code, but without having to go to the trouble of > converting it to a .py file (and possibly having to extract it back out > at runtime). This usage of "data" files isn't in conflict with the FHS, > as I understand it. > > But I also understand that there are other kinds of "data" files which > *don't* fall under that use case, and which it is desirable to install > to shared locations. We need to support both. > Possibly. We could definitely throw out the first case (resources) and just have a data category and the FHS would be fine. Whether there's a case for resources depends on their definition. the test of "Could be put in a python file and extracted" doesn't fly. I could convert all my images to .xpm and put them in python files. But that's a lot of work. And the moment I take them back out to separate .xpm files, they would definitely belong in /usr/share. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: OpenPGP digital signature URL: From dpeterson at enthought.com Thu Oct 2 01:02:41 2008 From: dpeterson at enthought.com (Dave Peterson) Date: Wed, 01 Oct 2008 18:02:41 -0500 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <48E3BACE.9020207@gmail.com> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> Message-ID: <48E40191.7070401@enthought.com> Toshio Kuratomi wrote: > Another idea would be to have API information stored in metadata but not > in the package name. That way marketing can have a big party for > MyLib-2.0 but the API metadata has API_Revision: 32. > +1. (I think this idea has been mentioned before too.) We should definitely put a requirement for an API version number in the meta-data PEP. Or should it be an ABI number? -- Dave From dpeterson at enthought.com Thu Oct 2 01:16:17 2008 From: dpeterson at enthought.com (Dave Peterson) Date: Wed, 01 Oct 2008 18:16:17 -0500 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <1222890032.7060.55.camel@tomoyo> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <1222890032.7060.55.camel@tomoyo> Message-ID: <48E404C1.6020102@enthought.com> Josselin Mouette wrote: > Le mercredi 01 octobre 2008 ? 14:39 -0400, Phillip J. Eby a ?crit : > >> To be clear, I mean here that a "file" (as opposed to a resource) is >> something that the user is expected to be able to read or copy, or >> modify. (Whereas a resource is something that is entirely internal >> to a library, and metadata is information *about* the library itself.) >> > > It?s not as simple as that. Python is not the only thing out there, and > there are many times where your resources need to be shipped in existing > formats, in files that land at specific places. For example icons go > in /usr/share/icons, locale files in .mo format in /usr/share/locale, > etc. > Perhaps I'm putting words into PJE's mouth but it seems to me that if distributions want to put things in widely differing places from where the developer had them (in a single tree), then we're going to need a Python standard library API (implemented per platform/OS that defines those places!) for the code to find these resources/files. Certainly the expectation shouldn't be on developers to have to handle all the possible different locations OSes are going to put things? -- Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.badger at gmail.com Thu Oct 2 01:28:09 2008 From: a.badger at gmail.com (Toshio Kuratomi) Date: Wed, 01 Oct 2008 16:28:09 -0700 Subject: [Distutils] Msgfmt in distutils? In-Reply-To: <8CCEF1EF-7B9F-4B55-B3CA-D548B3A494B4@underboss.org> References: <48E38896.6040909@gmail.com> <20081001143557.GY30869@nexus.in-nomine.org> <48E3C07F.1080605@gmail.com> <8CCEF1EF-7B9F-4B55-B3CA-D548B3A494B4@underboss.org> Message-ID: <48E40789.2070601@gmail.com> Philip Jenvey wrote: > > On Oct 1, 2008, at 11:25 AM, Toshio Kuratomi wrote: > >> Jeroen Ruigrok van der Werven wrote: >>> -On [20081001 16:28], Toshio Kuratomi (a.badger at gmail.com) wrote: >>>>> and have distutils do the right thing with the .po files at build time >>>>> (generate .mo files from them) and at install time (install them into >>>>> PREFIX/share/locales/LC_MESSAGES/, or wherever the distribution is >>>>> configured to put them). >>> >>> [snip] >>> >>>> This has been a big deal for some applications I work on. Our first >>>> cut >>>> was to add new Build and InstallData command classes. >>> >>> Actually with Babel (http://babel.edgewall.org/) that's all handled. >>> >> That's good to know. One of our Turbogears applications uses Babel and >> it definitely doesn't install to the right place. I'd love to fix it to >> take advantage of Babel' properly. Would you be kind enough to point me >> documentation on how to get Babel to install locale files? Looking at >> the babel website, I only see documentation up to building the message >> catalogs. If the install portion is integrated into setuptools is there >> something I might have to configure in setup() to tell babel/setuptools >> what directory to use? > > Once you have Babel generating .mo files, all you'll need is a > package_data entry for them, e.g.: > > package_data={'foo': ['i18n/*/LC_MESSAGES/*.mo']}, > > then the catalogs will make it into the final sdist/egg and be included > during an installation. > Thanks! This isn't quite what I was asking, though. the orignal poster was asking how to install the catalogs into /usr/share/locale, the proper directory on a Linux system. I thouoght babel was able to do that but it seems babel currently just handles the creation and maintenance of the message catalogs. Which is a huge thing! I just was hoping to get rid of my ugly code to move the catalogs into the system directory. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: OpenPGP digital signature URL: From a.badger at gmail.com Thu Oct 2 01:33:01 2008 From: a.badger at gmail.com (Toshio Kuratomi) Date: Wed, 01 Oct 2008 16:33:01 -0700 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <48E404C1.6020102@enthought.com> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <1222890032.7060.55.camel@tomoyo> <48E404C1.6020102@enthought.com> Message-ID: <48E408AD.5090404@gmail.com> Dave Peterson wrote: > Josselin Mouette wrote: >> Le mercredi 01 octobre 2008 ? 14:39 -0400, Phillip J. Eby a ?crit : >> >>> To be clear, I mean here that a "file" (as opposed to a resource) is >>> something that the user is expected to be able to read or copy, or >>> modify. (Whereas a resource is something that is entirely internal >>> to a library, and metadata is information *about* the library itself.) >>> >> >> It?s not as simple as that. Python is not the only thing out there, and >> there are many times where your resources need to be shipped in existing >> formats, in files that land at specific places. For example icons go >> in /usr/share/icons, locale files in .mo format in /usr/share/locale, >> etc. >> > > Perhaps I'm putting words into PJE's mouth but it seems to me that if > distributions want to put things in widely differing places from where > the developer had them (in a single tree), then we're going to need a > Python standard library API (implemented per platform/OS that defines > those places!) for the code to find these resources/files. Certainly > the expectation shouldn't be on developers to have to handle all the > possible different locations OSes are going to put things? > The code should be general. We just need to have a configuration file that has the defaults for the OS-architecture the library is installed on. This shouldn't be much of a problem, distutils already holds information like that (for instance the value of distutils.sysconfig.get_python_lib() and get_python_lib(1).) -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: OpenPGP digital signature URL: From pje at telecommunity.com Thu Oct 2 01:59:16 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 01 Oct 2008 19:59:16 -0400 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <48E3F650.6080309@gmail.com> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> Message-ID: <20081001235808.85D453A4072@sparrow.telecommunity.com> At 03:14 PM 10/1/2008 -0700, Toshio Kuratomi wrote: >resources, as I said needs to be defined. You're saying here that a >resource is something internal to the library. A "file" is something >that a user can read, copy, or modify. I should probably clarify that I mean "unmediated by the program"... which is why I disagree regarding message catalogs. They're not user-modifiable and there's nothing you can usefully do with them outside the program that uses them. Of course, a default message catalog for someone to use to *create* translations from might be another story... >In a typical TurboGears app, there's the following things to be found >inside of the app's directory in site-packages: > >config/{app.cfg,__init__.py,log.cfg} - These could go in /etc/ as their >configuration. However, I've tried to stress to upstream that only >things that configure the TurboGears framework for use with their app >should go in these files (default templating language, identity >controller). When those things are true, I can see this as being an >"internal resource". If upstream can't get their act together, it's config. Agreed. >locale/{message catalogs for various languages} -- These are binary >files that contain strings that the user may see when a message is >given. These, I think are data. I lean the other way, since they're not editable. Keep in mind that some platforms have no "locale" directories as such, and thus trying to support multiple locations thereof is a burden for cross-platform app developers, vs. simply treating them as internal resources. This is especially true for plugin-oriented architectures that want to distribute localizations as plugins, with one plugin being able to supply localization for another. >templates/*html -- These are templates that the application fills in >with variables intermixed with short bits of code. These are on the >border between code and data. The user sees them in a modified form. >The app sometimes executes pieces of them before the user sees them. >Some template languages create python byte code from the templates, >others load them and write into them every time. None of them can be >executed on their own. All of them have to be loaded by a call to parse >them from a piece of python code in another file. None of them are >directly called or invoked. My leaning is that these are data. If you follow this logic (create bytecode from it, can't run w/out interp or compiler), then any .py file is *also* data; i.e., this Does Not Follow. >static/{javascript,css,images} -- These are things that are definitely >never executed. They are served by the webserver verbatim when their >URL is called. These are certainly data. (Note: I don't believe these >are referenced using the resources API, just via URL.) Uh... that would depend entirely on the library or application. But if they're static and the user's got no business tinkering with them, then it's a resource. >So... do you agree on which of these are data and which are resources? >Do you have an idea on how we can prevent application and framework >writers from misusing the resources API to load things that are data? Apparently not. The alternative I would suggest is that under the new standard, an install tool should be allowed to relocate any non-Python files, and all access has to go through a resource API. The install tool would then have to be responsible for putting some kind of forwarding information in the package directory to tell the resource API where it squirrelled the file(s) off to. Then we can avoid all this angels-on-a-pin argument and the distros can Have It Their Way[tm]. I'd have preferred to avoid that complexity, but if the two of us can't agree then there's no way on earth to get a community consensus. Btw, pkg_resources' concept of "metadata" would also need to be relocatable, since e.g. the "EggTranslations" package uses that metadata to store localizations of image resources and message catalogs. (Other uses of the metadata files also inlcude scripts, dependencies, version info, etc.) Hopefully, those folks who want relocation ability will chip in with code, docs, testing, etc. for the feature at some point. From greg.ewing at canterbury.ac.nz Thu Oct 2 02:36:40 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 02 Oct 2008 12:36:40 +1200 Subject: [Distutils] [pyconuk] "just use debian" In-Reply-To: <1222789811.4166.114.camel@shizuru> References: <20080917171851.GA27261@logilab.fr> <48DDEC5C.7090601@ar.media.kyoto-u.ac.jp> <20080928173011.GA4852@volans.logilab.fr> <48E0C007.70702@ar.media.kyoto-u.ac.jp> <94bdd2610809290509l2ab7359ep7f4edfcd23eae4ee@mail.gmail.com> <20080930084253.GF4494@volans.logilab.fr> <94bdd2610809300505u1291ae6an3c44e2ce31d43e3f@mail.gmail.com> <1222780673.4166.55.camel@shizuru> <94bdd2610809300649q6bc9cdf3t84d5221d956bfc9d@mail.gmail.com> <1222784868.4166.88.camel@shizuru> <94bdd2610809300820g36a3f1bbpc2a04e03b43ee9c9@mail.gmail.com> <1222789811.4166.114.camel@shizuru> Message-ID: <48E41798.6050107@canterbury.ac.nz> Josselin Mouette wrote: > Le mardi 30 septembre 2008 ? 17:20 +0200, Tarek Ziad? a ?crit : > > > I mean, if you change a public API of your package , you *have* to > > change its name ? > > Yes, this is the requirement for C libraries, and we try to enforce it > as well for other languages. Things are somewhat different in C, because the filename of the .so isn't something you refer to in the source code. Applying the same thing to Python would require the version to be specified every time the module is mentioned in an import statement. -- Greg From ziade.tarek at gmail.com Thu Oct 2 03:10:59 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 2 Oct 2008 03:10:59 +0200 Subject: [Distutils] pre-PEP : Synthesis of previous threads, and irc talks + proposals In-Reply-To: <20081001182658.E96173A4072@sparrow.telecommunity.com> References: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.com> <20081001172824.B1E0B3A4072@sparrow.telecommunity.com> <94bdd2610810011055j69411c6fu6f417cf3cf930460@mail.gmail.com> <20081001182658.E96173A4072@sparrow.telecommunity.com> Message-ID: <94bdd2610810011810h3ea48e6cmb8d223cc24e1cf85@mail.gmail.com> On Wed, Oct 1, 2008 at 8:28 PM, Phillip J. Eby wrote: > At 07:55 PM 10/1/2008 +0200, Tarek Ziad? wrote: >> >> On Wed, Oct 1, 2008 at 7:29 PM, Phillip J. Eby >> wrote: >> > I'm -1 on all of the above. I think we need a standard for tools >> > interop >> > (ala WSGI), not implementation tweaks for the existing tools. I also >> > think >> > that a concrete metadata format proposal is premature at this time; >> > we've >> > barely begun to gather -- let alone specify -- our requirements for that >> > metadata. (Essentially, only version dependencies have been discussed, >> > AFAICT.) >> >> What are the other important points we need to discuss at this point >> in your opinion ? > > What information needs to be conveyed by a "build" tool to an "install" > tool, and vice versa. > > For example, an install tool needs to know what files are documentation, > which are sample data, and what is part of the library proper, as well as > what things are scripts (and in what language those scripts are written, > e.g. Python, shell, batch, etc.). Some install tools need to know about > icons, menus, registry info, cron jobs, etc. (These are perhaps more > properly the domain of applications than libraries, but I'm going to assume > that these things are in scope.) > > The way that this information is communicated needs to be extensible, so > that optional metadata for debs and msi's and rpm's and whathaveyou can be > incorporated, without needing to modify the standard -- especially if the > APIs for reading and writing this data are in the stdlib. So basically with this system, this would mean that an ini-file in my package would be marked as a configuration file for the installer API, and that third party tools would decide what to do with this file. This would mean that my program would have to access to that file through an API as well to get back to the ini-file in the system. how would it work ? > I think I've mentioned all of these previously in the thread. I also think > that as a matter of technicalities, these things are not difficult to > achieve... but if it they are only achieved *technically*, then the result > is a failure, not a success. > > In order for the *real* goal to be achieved (i.e., a flourishing > build/install system for Python), widespread participation and buy-in is > required. If the OS people or the big package people (e.g. Zope Corp., > Enthought) or the distutils aficionados are left out, then the result won't > get used. Well yes, that is basically what we are trying to build since a few days, but threads are not linear so people cant' keep up and jump in like that. So maybe you mentioned that idea before in the thread, but if we want people to buy the idea, it should be put in the wiki imho, even prematurely, and built it there, starting from a small set of points. I mean, you said earlier that it is premature to write a concrete document but it's hard to follow in threads the ideas proposed. That was the idea of my early proposal : start a synthetic list of problems and for each problem possible solutions. Then discuss them in the ML and make the right one raise. That is just my 2 cents on how the discussions go > > I think the best way to ensure that nobody is left out, is to get them to > participate in the design of a standard that ensures that *they* will be > able to control their destiny, by creating their own build plugins and/or > install tools... or at least having a robust choice of alternatives. > > We need a consensus "de jure" standard (ala WSGI), rather than just an > ongoing "de facto" standard (ala distutils/setuptools), or we're not making > any substantial progress, just handing the reins over to somebody else. +10 > > >> >> Enforcements: >> >> - a binary distribution cannot be uploaded if a source distrbution >> >> has not been previously provided for the version >> > >> > Note that this doesn't allow closed-source packages to be uploaded; thus >> > it >> > would need to be a warning, rather than a requirement. >> > >> >> Right. do you agree it is something useful to do ? > > Sure. Ok so maybe this is *one* problem we can already solve at PyPI with a patch. >> >> - PyPI should display the install and test dependencies in the UI >> > >> > It could only do this for specific binaries, since dependencies can be >> > dynamic. >> > >> > >> >> What dynamic means here ? the python module to static file process or more >> ? >> can you provide an example ? > > Dependencies can be platform-specific as well as python-version specific. > If you want ctypes, you would depend on it in python 2.4 but not in 2.5. > Similarly, if on some platform a certain library is required to implement a > feature that is natively accessible on other platforms. In these cases, you > would have logic in setup.py to detect this and choose the appropriate > dependencies. ok, right it is not possible with what we have now. If each dependency was marked as platform-specific or python-version specific in the metadata when it is necessary, we would know them all without calling extra detection code to build them. I hate the idea of dynamic metadata in fact. I can't express precisely why at that point. > > -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From a.badger at gmail.com Thu Oct 2 04:14:38 2008 From: a.badger at gmail.com (Toshio Kuratomi) Date: Wed, 01 Oct 2008 19:14:38 -0700 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <20081001235808.85D453A4072@sparrow.telecommunity.com> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> <20081001235808.85D453A4072@sparrow.telecommunity.com> Message-ID: <48E42E8E.5090305@gmail.com> Phillip J. Eby wrote: > At 03:14 PM 10/1/2008 -0700, Toshio Kuratomi wrote: >> resources, as I said needs to be defined. You're saying here that a >> resource is something internal to the library. A "file" is something >> that a user can read, copy, or modify. > > I should probably clarify that I mean "unmediated by the program"... > which is why I disagree regarding message catalogs. They're not > user-modifiable and there's nothing you can usefully do with them > outside the program that uses them. Of course, a default message > catalog for someone to use to *create* translations from might be > another story... > this is what I was afraid of. This is definitely not a definition of resource-only that has meaning for Linux distributions. None of the data in /usr/share is user-modifiable (a tiny bit of it is copiable for the user to then edit the copy) and although a good fraction of it is usable outside the program that uses it, a much larger portion is taken up with things that are used by one program. I could go through the examples below and tell why Linux distributions feel the way they do but I don't think it's necessary. Whether they're data or resources, the files need to be relocatable. And they need to be accessed via an API for that to work. So as long as we're agreed that these have to be included in the egg on some platforms and in the filesystem on others then I think we know what needs to be done. [...] >> So... do you agree on which of these are data and which are resources? >> Do you have an idea on how we can prevent application and framework >> writers from misusing the resources API to load things that are data? > > Apparently not. The alternative I would suggest is that under the new > standard, an install tool should be allowed to relocate any non-Python > files, and all access has to go through a resource API. The install > tool would then have to be responsible for putting some kind of > forwarding information in the package directory to tell the resource API > where it squirrelled the file(s) off to. Then we can avoid all this > angels-on-a-pin argument and the distros can Have It Their Way[tm]. > In terms of implementation I'd much rather see something less centered on the egg being the right way and the filesystem being a secondary concern. We should have metadata that tells us where the types of resources come from. When a package is installed on Linux the metadata could point locales at file:///usr/share/locale. When on Windows egg:locale (Perhaps the uninstalled case would use this too... that depends on how the egg structure and metadata evolves.) A question we'd have to decide is whether this particular metadata is something that should be defined globally or per package. Or globally with a chance for packages to override it. > I'd have preferred to avoid that complexity, but if the two of us can't > agree then there's no way on earth to get a community consensus. > > Btw, pkg_resources' concept of "metadata" would also need to be > relocatable, since e.g. the "EggTranslations" package uses that metadata > to store localizations of image resources and message catalogs. (Other > uses of the metadata files also inlcude scripts, dependencies, version > info, etc.) > Actually, we should decide whether we want to support that kind of thing within the egg metadata at all. The other things we've been talking about belonging in the metadata are simple key value pairs. EggTranslations uses the metadata area as a data store. (Or in your definition, a resource store). This breaks with the definition of what metadata is. Translations don't store information about a package, they store alternate views of data within the package. While the simple key value pairings can be located in either setuptools .egg-info directories or python-2.5+ distutils .egg-info files, the data store in EggTranslations can only be placed in directories. Having a data store/resource store API would be more appropriate for the kinds of things that EggTranslation is doing. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: OpenPGP digital signature URL: From greg.ewing at canterbury.ac.nz Thu Oct 2 05:33:24 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 02 Oct 2008 15:33:24 +1200 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <48E42E8E.5090305@gmail.com> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> <20081001235808.85D453A4072@sparrow.telecommunity.com> <48E42E8E.5090305@gmail.com> Message-ID: <48E44104.9000904@canterbury.ac.nz> Toshio Kuratomi wrote: > this is what I was afraid of. This is definitely not a definition > of resource-only that has meaning for Linux distributions. None of the > data in /usr/share is user-modifiable In that case it must be there because it's architecture-independent, right? But by that criterion, all .py files should be in /usr/share, too. Also all shell scripts, Perl code, awk/sed scripts, etc, etc. Does the FHS specify that? -- Greg From a.badger at gmail.com Thu Oct 2 05:42:44 2008 From: a.badger at gmail.com (Toshio Kuratomi) Date: Wed, 01 Oct 2008 20:42:44 -0700 Subject: [Distutils] "just use debian" In-Reply-To: <48E33A91.5000602@ar.media.kyoto-u.ac.jp> References: <20080917171851.GA27261@logilab.fr> <20080930084253.GF4494@volans.logilab.fr> <94bdd2610809300505u1291ae6an3c44e2ce31d43e3f@mail.gmail.com> <1222780673.4166.55.camel@shizuru> <94bdd2610809300649q6bc9cdf3t84d5221d956bfc9d@mail.gmail.com> <1222784868.4166.88.camel@shizuru> <94bdd2610809300820g36a3f1bbpc2a04e03b43ee9c9@mail.gmail.com> <48E24765.1020908@simplistix.co.uk> <48E24BDA.2050104@simplistix.co.uk> <94bdd2610809300901o3fa62601q985784f01a8ad774@mail.gmail.com> <48E24FF7.9000300@simplistix.co.uk> <48E255B9.50806@colorstudy.com> <1222801361.32659.1.camel@shizuru> <48E2903F.9000307@enthought.com> <1222810334.32659.18.camel@shizuru> <48E2A0BB.6090300@enthought.com> <1222848707.7060.4.camel@tomoyo> <48E333C2.9030706@ar.media.kyoto-u.ac.jp> <1222851502.7060.18.camel@tomoyo> <48E33A91.5000602@ar.media.kyoto-u.ac.jp> Message-ID: <48E44334.4040405@gmail.com> David Cournapeau wrote: > Josselin Mouette wrote: >> Indeed, and the reason is that *functions never disappear from the >> glibc*. > > Yes and no. If you remove a function, you're indeed screwed, because you > can't handle versioning in the header. But you can handle versioning in > libraries at the link step, and file name of the library is an > implementation detail of this versioning. > I'm not 100% certain but I think that Josselin is speaking of glibc in particular here and you're speaking of c libraries in general. >> I don?t think Mono causes issues, but I?m pretty sure that allowing >> multiple versions like the GAC allows *will* causes issues. Not for >> purely-packaged things, where you can safely ignore those directory >> renames, but if you start mixing distributed packages and >> user-downloaded stuff, they will run into the same issues we have with >> setuptools. > > Please read the article carefully, it is not only about the GAC. It does > handle the two conflicting issues: API stability installed globally vs > easiness of deployment. That's why it is an interesting read IMHO: it > addresses both issues. I don't think there is a single chance to see > something as strong as C for python, because it would severely undermine > the whole idea of the language used for prototyping. > Mono is absolutely horrid in this regard. Those who care about mono (not our most persuasive speakers, I'm afraid) have asked upstream to stop making that a best practice for Mono applications. I've said before that ideally a Linux distribution only wants one version of a library. In Fedora we're willing to have compat packages that hold old API versions if we must but by and large we would rather help upstream apps port their applications forward than to have compatibility packages. This is because upstream for the library will always be focusing on the newer versions of the libraries, not the older versions. If applications stay stuck on older versions, we end up having to support libraries by ourselves with no upstream to help us with the old version. As much as I'd rather not have compat packages, having private versions of third party libraries as advocated in that Mono document is worse. The primary problem is security. If a distro allows application to have their own private copies of libraries and a security flaw is discovered we're going to hate life. We'll have to: 1) Find what packages include that library. Unlike when the link goes to a system installed library, this will not cause a dependency between packages. So we can't just query the package metadata to find out what packages are affected. 2) Fix all the versions in all the packages. Because each package includes its own version of the library, there are multiple versions of the library in these packages. If we're unlucky the change will be conceptual and we'll have to fix lots of different looking code. 3) Push rebuilds of all the fixed packages out that our users have to download. There's PR involved here: Security fix to Library Foo vs Security Fix to Library Foo, Application, Bar, Baz, [...] Zod. There's also the burden for the users to download the packages. Compare this with having to fix a set of compat library packages that's not included in other applications: 1) Find all the libraries in the affected set. It will probably be enough to look by package name since these will be like: python-foo1-1.0, python-foo2-2.2, python-foo-3.0 2) Fix the library (probably with help from upstream) and the compat-libraries (maybe with upstream help or maybe on our own). 3) Push rebuilds of the library packages for our users to download. Another concern is licensing. Anytime a package includes other, third party modules, the licensing situation becomes more complex. Are the licensing terms of any of the works being violated? How do we have to list the licensing terms in the package? Are licensing terms for all the packages available? is everything open source? (believe it or not, we do find non-OSS stuff in third party directories when we audit these bundled packages :-( Another concern is not giving back to upstream. Once a package starts including its own, private copies of a library it becomes more and more tempting for the package to make bug fixes and enhancements on its own copy. This has two serious problems: 1) It becomes harder to port the application forward to a new version because this is no longer what upstream has shipped at any time. 2) The changes may not get back to upstream at all. Those bug fixes and feature enhancements may end up being only part of this package, even though the whole community would benefit. Another concern is build scripts that become tied to building with/installing the private versions. Distributions have policies on inclusion of third party libraries in another application. Sometimes upstream has a reason to include a copy of a library for compatibility on Windows or for customers who aren't going to get it from their distribution. In these cases, if the distribution can give a command to the build scripts to not build or install the compat library, all is still well. However, using system libraries often bitrots in an upstream's build scripts because they start caring more about the pegged library version they control than about making things work with system libraries. This makes more work for the packager. Another concern is shipping of prebuilt-binaries. Just before Fedora 9 came out we had to go through our Mono packages, get rid of some, and do extensive work on the build scripts of others. This was because some packagers hadn't been watching their packages very well and upstream had started shipping prebuilt versions of the third party modules they required. For upstream, they were making things easier for their end-users. For us, we had to assure that everything was built from source that was auditable and available if needed (for instance, if one of those third party libraries had a security flaw). Another case is upstream reluctance to take patches to forward port the application. Unfortunately, when upstreams start including their own, known good versions of libraries inside their packages, they sometimes become reluctant to forward port to a new version of the library. For them, it's moving from a version that they know about to an untested version from upstream. For distributions, which have a vested interest in helping upstreams forward port so they have only one version of a library to maintain in the distro, this is an impediment to helping upstream by providing patches to forward port. These are some of the reasons that packaging Mono applications is something I personally avoid in Fedora :-) Please, do not go down this road with python. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: OpenPGP digital signature URL: From pje at telecommunity.com Thu Oct 2 06:10:01 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 02 Oct 2008 00:10:01 -0400 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <48E42E8E.5090305@gmail.com> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> <20081001235808.85D453A4072@sparrow.telecommunity.com> <48E42E8E.5090305@gmail.com> Message-ID: <20081002041137.355233A40B0@sparrow.telecommunity.com> At 07:14 PM 10/1/2008 -0700, Toshio Kuratomi wrote: >In terms of implementation I'd much rather see something less centered >on the egg being the right way and the filesystem being a secondary >concern. Eggs don't have anything to do with it; in Python, it's simply common sense to put static resources next to the code that uses them, if you want to "write once, run anywhere". And given Python's strength as an interactive development language with no "build" step, having to *install* your data files somewhere else on the system to use them isn't a *feature* -- not for a developer, anyway. And our hypothetical de-jure standard won't replace the de-facto standard unless it's adopted by developers... and it won't be adopted if it makes their lives harder without a compensating benefit. For the developer, FHS support is a cost, not a benefit, and only relevant to a subset of platforms, so the spec should make it as transparent for them as possible, if they don't have an interest in explicit support for it. By the STASCTAP principle (Simple Things Are Simple, Complex Things Are Possible), it should be possible for distros to relocate, and simple for developers not to care about it. > We should have metadata that tells us where the types of >resources come from. When a package is installed on Linux the metadata >could point locales at file:///usr/share/locale. When on Windows >egg:locale (Perhaps the uninstalled case would use this too... that >depends on how the egg structure and metadata evolves.) > >A question we'd have to decide is whether this particular metadata is >something that should be defined globally or per package. Or globally >with a chance for packages to override it. I think install tools should handle it and keep it out of developers' hair. We should of course distinguish configuration and other writable data from static data, not to mention documentation. Any other file-related info is going to have to be optional, if that. I don't really think it's a good idea to ask developers to fill in information they don't understand. A developer who works entirely on Windows, for example, is not going to have a clue what to specify for FHS stuff, and they absolutely shouldn't have to if all they're doing is including some static data. Even today, there exist Python developers who don't use the distutils to distribute their packages, so anything that makes it even more difficult than it is today, isn't going to be a viable standard. The closer we can get in ease of use to just tarring up a directory, the more viable it'll be. (That's one reason, btw, why setuptools offers revision control support and find_packages() for automating discovery of what to include.) > > I'd have preferred to avoid that complexity, but if the two of us can't > > agree then there's no way on earth to get a community consensus. > > > > Btw, pkg_resources' concept of "metadata" would also need to be > > relocatable, since e.g. the "EggTranslations" package uses that metadata > > to store localizations of image resources and message catalogs. (Other > > uses of the metadata files also inlcude scripts, dependencies, version > > info, etc.) > > >Actually, we should decide whether we want to support that kind of thing >within the egg metadata at all. The other things we've been talking >about belonging in the metadata are simple key value pairs. >EggTranslations uses the metadata area as a data store. (Or in your >definition, a resource store). This breaks with the definition of what >metadata is. Translations don't store information about a package, they >store alternate views of data within the package. I was actually somewhat incorrect in my statement about the distinction between pkg_resources "metadata" and "resources"; "metadata" is really "data that goes with the distribution, not with a specific package within the distribution". Only some of this data is "about" the distribution; the rest is data "with" or "of" the distribution. (This is a slight API wart, but the use case exists nonetheless.) Meanwhile, regarding the proposed key-value pairs system, I don't see how that works; "extras" dependency information and entry points are a bit more structured than just key-value pairs; both are currently represented as .ini-like files with arbitrary section names. I suppose you could squash those entire files into values in some sort of key-value system, but that seems a bit hairy to me. In particular, setuptools design choice for separate metadata files is that many of these things don't need to be loaded at the same time. Also, PKG-INFO-style metadata can contain rather large blobs of text that aren't needed or useful at runtime. Entry points and extras are mostly runtime metadata, with the occasional bit of build or install usage. From a.badger at gmail.com Thu Oct 2 06:33:52 2008 From: a.badger at gmail.com (Toshio Kuratomi) Date: Wed, 01 Oct 2008 21:33:52 -0700 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <48E44104.9000904@canterbury.ac.nz> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> <20081001235808.85D453A4072@sparrow.telecommunity.com> <48E42E8E.5090305@gmail.com> <48E44104.9000904@canterbury.ac.nz> Message-ID: <48E44F30.20908@gmail.com> Greg Ewing wrote: > Toshio Kuratomi wrote: > >> this is what I was afraid of. This is definitely not a definition >> of resource-only that has meaning for Linux distributions. None of the >> data in /usr/share is user-modifiable > > In that case it must be there because it's architecture-independent, > right? > ...That doesn't follow from what I said, but it's true :-) > But by that criterion, all .py files should be in /usr/share, too. > I mentioned in a different post that this has been considered by several distributions. Note that not all .py files can be shifted due to the way python parses modules. But certainly modules which are pure python could be moved. Reasons that Fedora hasn't done this are: 1) Historical: .py files have been in /usr/lib/python2.5/site-packages for a long time. 2) Compatibility with third parties: Unfortunately not everyone uses distutils. If we shifted the location to /usr/share and users installed those packages into /usr/lib it would fail. 3) /usr/share has two purposes/criteria[1]_: architecture independent and datafiles. /usr/lib has two criteria[2]_: architecture independent and libraries. With .py{,c,o} we have both architecture indepedence and a library. So the criteria is in conflict with each other. There may be more reasons, I'm in the /usr/share camp but not so much that I'll keep bringing it up when there's no new arguments to give. Note that Debian has done a lot of neat things with python source recent(ish). Josselin, Matthias, and some of the other Debian devs could tell us if .py files get installed to /usr/share there. .. _[1]: http://www.pathname.com/fhs/pub/fhs-2.3.html#USRSHAREARCHITECTUREINDEPENDENTDATA .. [2]_: http://www.pathname.com/fhs/pub/fhs-2.3.html#USRLIBLIBRARIESFORPROGRAMMINGANDPA > Also all shell scripts, Perl code, awk/sed scripts, etc, etc. Things that are directly executable belong in a bin directory. There are next to no shell script libraries, just scripts. Perl, awk, sed, etc *scripts* end up in /bin as well. To my knowledge perl doesn't support the split architecture independent library location/architecture dependent library location that python does so everything goes into /usr/lib. Mono assemblies do not because of a pair of limitations of the mono vm. java jars go in /usr/share. The m4 macros that autoconf/automake use go there as well. Programs that are written in python but don't want to expose their internals to the outside world have their code under /usr/share. We make php apps do the same. Perl is probably the same although I haven't looked at an actual multi-file perl program in.... well, I don't remember when so I don't know. > Does the FHS specify that? > The FHS sets out certain rules and criteria. Linux vendors have interpreted them and sometimes the standard is updated due to either current practice or clarification of former practice. I don't believe that FHS "specifies" that .py files go in /usr/lib or /usr/share. The rules state things like "architecture independent data file" which is why there's some grey area for /usr/lib/python's .py files. Note that although I'm happy to talk about the FHS here, I'm not involved with creating the standard. I'm also only one packager from one distro. So I'm happy to help answer questions about the FHS and how Fedora interprets it but am not in any better position to change it than any of you. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: OpenPGP digital signature URL: From ben+python at benfinney.id.au Thu Oct 2 07:02:13 2008 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 02 Oct 2008 15:02:13 +1000 Subject: [Distutils] "Python Package Management Sucks" References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> <20081001235808.85D453A4072@sparrow.telecommunity.com> <48E42E8E.5090305@gmail.com> <48E44104.9000904@canterbury.ac.nz> <48E44F30.20908@gmail.com> Message-ID: <87ej2zpp2i.fsf@benfinney.id.au> Toshio Kuratomi writes: > 3) /usr/share has two purposes/criteria[1]_: architecture > independent and datafiles. /usr/lib has two criteria[2]_: > architecture independent and libraries. With .py{,c,o} we have both > architecture indepedence and a library. So the criteria is in > conflict with each other. I think you mean: * /usr/share has two criteria: architecture independent and data files * /usr/lib has two criteria: architecture dependent and libraries and that Python library files are both architecture independent and libraries, thus neither of those fit perfectly. -- \ ?A free press is one where it's okay to state the conclusion | `\ you're led to by the evidence.? ?Bill Moyers | _o__) | Ben Finney From floris.bruynooghe at gmail.com Thu Oct 2 10:25:33 2008 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Thu, 2 Oct 2008 09:25:33 +0100 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <48E44F30.20908@gmail.com> References: <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> <20081001235808.85D453A4072@sparrow.telecommunity.com> <48E42E8E.5090305@gmail.com> <48E44104.9000904@canterbury.ac.nz> <48E44F30.20908@gmail.com> Message-ID: <20081002082533.GA11432@laurie.devork> On Wed, Oct 01, 2008 at 09:33:52PM -0700, Toshio Kuratomi wrote: > Note that Debian has done a lot of neat things with python source > recent(ish). Josselin, Matthias, and some of the other Debian devs > could tell us if .py files get installed to /usr/share there. Currently the two helper tools install the .py (and .egg-info) files somewhere into /usr/share. One tool places the .pyc files in /usr/lib while the other places them in /var/lib (and uses a .pth file to get that directory on sys.path), both have symlinks to the real .py next too the .pyc files. This way the same .py file can be shared between more then one version of python. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From david at ar.media.kyoto-u.ac.jp Thu Oct 2 12:34:20 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 02 Oct 2008 19:34:20 +0900 Subject: [Distutils] "just use debian" In-Reply-To: <48E44334.4040405@gmail.com> References: <20080917171851.GA27261@logilab.fr> <20080930084253.GF4494@volans.logilab.fr> <94bdd2610809300505u1291ae6an3c44e2ce31d43e3f@mail.gmail.com> <1222780673.4166.55.camel@shizuru> <94bdd2610809300649q6bc9cdf3t84d5221d956bfc9d@mail.gmail.com> <1222784868.4166.88.camel@shizuru> <94bdd2610809300820g36a3f1bbpc2a04e03b43ee9c9@mail.gmail.com> <48E24765.1020908@simplistix.co.uk> <48E24BDA.2050104@simplistix.co.uk> <94bdd2610809300901o3fa62601q985784f01a8ad774@mail.gmail.com> <48E24FF7.9000300@simplistix.co.uk> <48E255B9.50806@colorstudy.com> <1222801361.32659.1.camel@shizuru> <48E2903F.9000307@enthought.com> <1222810334.32659.18.camel@shizuru> <48E2A0BB.6090300@enthought.com> <1222848707.7060.4.camel@tomoyo> <48E333C2.9030706@ar.media.kyoto-u.ac.jp> <1222851502.7060.18.camel@tomoyo> <48E33A91.5000602@ar.media.kyoto-u.ac.jp> <48E44334.4040405@gmail.com> Message-ID: <48E4A3AC.8030909@ar.media.kyoto-u.ac.jp> Toshio Kuratomi wrote: > I'm not 100% certain but I think that Josselin is speaking of glibc in > particular here and you're speaking of c libraries in general. Maybe, but I don't see how this change the point: when you change the soname of a library, it has 0 impact on the source code of the software which links against it. That's not true in python. > > > I've said before that ideally a Linux distribution only wants one > version of a library. And ideally, developers do not want to care about versioning issues :) There is a middle ground to find; up to now, I think python distutils and co did not care at all about this, but you can not ask to move 180 degrees and do only as linux vendors want. I understand the reasons why OS vendors want to avoid distributing multiple versions as much as possible. But that's just not realistic in every case. Even in C-derived languages, this is sometimes a PITA. I could give you examples where distributions screwed up the packaging badly and hurt some projects I am involved with. But that would be unfair and besides the point (we all screw up); the point is that there are some practical reasons for sometimes including private copies. Because for example in Fortran's case, there is this huge mess of gfortran and g77 not being ABI compatible; there are examples in C++, and even in C. You also can't impose every software to follow distributions time-schedule. Reasons for single version for OS vendors are valid; but so are the ones to have multiple versions. I think compat modules would cover most needsl the problem is that python does not have a mechanism to request a particular version of a module. But wouldn't this help OS vendors to have such a mechanism (to decrease the burden of compat versions ?) > In Fedora we're willing to have compat packages > that hold old API versions if we must but by and large we would rather > help upstream apps port their applications forward than to have > compatibility packages. Yes, but here again the C comparison breaks. Some people use python as a "tool", not so much as a "programming language". Their applications are scripts, or softwares for experiment, that are not released, because they can't open source it, or simply because it has no use for anyone else. You can't port that. cheers, David From david at ar.media.kyoto-u.ac.jp Thu Oct 2 12:40:52 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 02 Oct 2008 19:40:52 +0900 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <20081002041137.355233A40B0@sparrow.telecommunity.com> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> <20081001235808.85D453A4072@sparrow.telecommunity.com> <48E42E8E.5090305@gmail.com> <20081002041137.355233A40B0@sparrow.telecommunity.com> Message-ID: <48E4A534.7010501@ar.media.kyoto-u.ac.jp> Phillip J. Eby wrote: > > I think install tools should handle it and keep it out of developers' > hair. We should of course distinguish configuration and other > writable data from static data, not to mention documentation. Any > other file-related info is going to have to be optional, if that. I > don't really think it's a good idea to ask developers to fill in > information they don't understand. A developer who works entirely on > Windows, for example, is not going to have a clue what to specify for > FHS stuff, and they absolutely shouldn't have to if all they're doing > is including some static data. I may be missing something, but why should the developer even care about FHS ? We should not standardize what goes where, but the kind of data needed to be installed (doc, etc...), and then have different (pluggable) implementations to put those where they should. Autotools does it this way, for example: you have pkg_data, etc... and every one of them can be changed. The proof that this is flexible is that fact that something like GoboLinux (which is a bit like what Mac OS X handles their files) is possible at all from the same sources. http://www.gobolinux.org/?page=doc/articles/compile I don't see the need for reinventing anything here: just start from the same concepts as autotools, modify it to handle non unix softwares (if it is even needed), and that's it. cheers, David From joss at debian.org Thu Oct 2 13:05:09 2008 From: joss at debian.org (Josselin Mouette) Date: Thu, 02 Oct 2008 13:05:09 +0200 Subject: [Distutils] [pyconuk] "just use debian" In-Reply-To: <48E41798.6050107@canterbury.ac.nz> References: <20080917171851.GA27261@logilab.fr> <48DDEC5C.7090601@ar.media.kyoto-u.ac.jp> <20080928173011.GA4852@volans.logilab.fr> <48E0C007.70702@ar.media.kyoto-u.ac.jp> <94bdd2610809290509l2ab7359ep7f4edfcd23eae4ee@mail.gmail.com> <20080930084253.GF4494@volans.logilab.fr> <94bdd2610809300505u1291ae6an3c44e2ce31d43e3f@mail.gmail.com> <1222780673.4166.55.camel@shizuru> <94bdd2610809300649q6bc9cdf3t84d5221d956bfc9d@mail.gmail.com> <1222784868.4166.88.camel@shizuru> <94bdd2610809300820g36a3f1bbpc2a04e03b43ee9c9@mail.gmail.com> <1222789811.4166.114.camel@shizuru> <48E41798.6050107@canterbury.ac.nz> Message-ID: <1222945509.6971.8.camel@shizuru> Le jeudi 02 octobre 2008 ? 12:36 +1200, Greg Ewing a ?crit : > Josselin Mouette wrote: > > Le mardi 30 septembre 2008 ? 17:20 +0200, Tarek Ziad? a ?crit : > > > > > I mean, if you change a public API of your package , you *have* to > > > change its name ? > > > > Yes, this is the requirement for C libraries, and we try to enforce it > > as well for other languages. > > Things are somewhat different in C, because the filename of > the .so isn't something you refer to in the source code. > > Applying the same thing to Python would require the version > to be specified every time the module is mentioned in an > import statement. There are no ABI issues with Python. Only when the API changes, software using it is broken. When the API changes incompatibly in C, the problem is the same. It is solved elegantly by pkg-config: the two versions can have a different (versioned) directory for the includes and a different library name, so you only change the pkg-config calls or the -I and -l build options. I wish we had a similar mechanism in Python, allowing to select the API version in some way. However, since there is no build step, you need to make it happen somewhere in the code. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From joss at debian.org Thu Oct 2 13:20:45 2008 From: joss at debian.org (Josselin Mouette) Date: Thu, 02 Oct 2008 13:20:45 +0200 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <20081001235808.85D453A4072@sparrow.telecommunity.com> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> <20081001235808.85D453A4072@sparrow.telecommunity.com> Message-ID: <1222946445.6971.21.camel@shizuru> Le mercredi 01 octobre 2008 ? 19:59 -0400, Phillip J. Eby a ?crit : > >locale/{message catalogs for various languages} -- These are binary > >files that contain strings that the user may see when a message is > >given. These, I think are data. > > I lean the other way, since they're not editable. Locale data should be shipped in standard form, in /usr/share/locale. Again, Python is not the only thing you need to think of. Not only, if you don?t use standard gettext catalogs, you are breaking translation tools, but if you don?t ship them at the standard location, you are breaking tools like localepurge (or anything that cleans up the filesystem based on standard file locations). > Keep in mind that > some platforms have no "locale" directories as such, and thus trying > to support multiple locations thereof is a burden for cross-platform > app developers, vs. simply treating them as internal resources. No, you are making the burden heavier for Linux platforms by trying to be more portable. There is no reason why you can?t define a locale directory on platforms where it does not exist, but when it exists a standards document defining where files go, you must follow it. Any time you don?t follow it, we consider it a serious breakage and we have to patch the code anyway. > >templates/*html -- These are templates that the application fills in > >with variables intermixed with short bits of code. [snip] > >My leaning is that these are data. > > If you follow this logic (create bytecode from it, can't run w/out > interp or compiler), then any .py file is *also* data; i.e., this > Does Not Follow. Sorry, but things don?t work this way. Anything that is *not* a .py file should not land in the python module directories. This is utter abuse of a loophole of the implementation, and I can?t think of another language that allows that. You will not find anything else than perl modules in the perl modules directories. You will not find anything else than C# modules in the Mono modules directory. And so on. > >static/{javascript,css,images} -- These are things that are definitely > >never executed. They are served by the webserver verbatim when their > >URL is called. These are certainly data. (Note: I don't believe these > >are referenced using the resources API, just via URL.) > > Uh... that would depend entirely on the library or application. But > if they're static and the user's got no business tinkering with them, > then it's a resource. No, again it is insane to ship them as resources. These are files that need to be accessed directly by the webserver, and as such they need to be shipped in a data directory, not in a python modules directory. > >So... do you agree on which of these are data and which are resources? > >Do you have an idea on how we can prevent application and framework > >writers from misusing the resources API to load things that are data? > > Apparently not. The alternative I would suggest is that under the > new standard, an install tool should be allowed to relocate any > non-Python files, and all access has to go through a resource > API. The install tool would then have to be responsible for putting > some kind of forwarding information in the package directory to tell > the resource API where it squirrelled the file(s) off to. Then we > can avoid all this angels-on-a-pin argument and the distros can Have > It Their Way[tm]. I don?t understand why you want to make it so complicated. All you need is a way to specify directories where different kinds of files land and a simple API to retrieve the file names/contents. Then, you can ship the files at the place you like in eggs, and we can ship the files at the standard places in our packages. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From asmodai at in-nomine.org Thu Oct 2 13:25:54 2008 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Thu, 2 Oct 2008 13:25:54 +0200 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <1222946445.6971.21.camel@shizuru> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> <20081001235808.85D453A4072@sparrow.telecommunity.com> <1222946445.6971.21.camel@shizuru> Message-ID: <20081002112554.GC30869@nexus.in-nomine.org> -On [20081002 13:20], Josselin Mouette (joss at debian.org) wrote: >Locale data should be shipped in standard form, in /usr/share/locale. >Again, Python is not the only thing you need to think of. Linux is not the only thing to think of by stating /usr/share/locale. :P I guess ${PREFIX}/share/locale would be better. But still, for .mo files I seriously do not see a problem if they're within an egg or something similar. Imagine also the possibility of providing specific language translations for a project as an egg (kind of like a plugin). -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B Whispering winds in moonlit wood, a totem oak once golden stood... From joss at debian.org Thu Oct 2 13:26:24 2008 From: joss at debian.org (Josselin Mouette) Date: Thu, 02 Oct 2008 13:26:24 +0200 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <48E44104.9000904@canterbury.ac.nz> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> <20081001235808.85D453A4072@sparrow.telecommunity.com> <48E42E8E.5090305@gmail.com> <48E44104.9000904@canterbury.ac.nz> Message-ID: <1222946784.6971.26.camel@shizuru> Le jeudi 02 octobre 2008 ? 15:33 +1200, Greg Ewing a ?crit : > Toshio Kuratomi wrote: > > > this is what I was afraid of. This is definitely not a definition > > of resource-only that has meaning for Linux distributions. None of the > > data in /usr/share is user-modifiable > > In that case it must be there because it's architecture-independent, > right? > > But by that criterion, all .py files should be in /usr/share, too. > > Also all shell scripts, Perl code, awk/sed scripts, etc, etc. > Does the FHS specify that? The FHS is not that precise, but in Debian, we do: * move architecture-independent perl modules to /usr/share/perl5 * move architecture-independent python modules to /usr/share/{python-support,pyshared} * when possible, move shell scripts (except those in /usr/bin) from /usr/libexec or /usr/lib to /usr/share/$package There?s no rocket science in that. But I wish we could make things as simple for Python modules as they are for Perl ones. But Perl does not allow messing with module namespaces with .pth files, nor shipping resource files and whatnot in the perl module directory. Only perl modules are allowed there. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From joss at debian.org Thu Oct 2 13:29:03 2008 From: joss at debian.org (Josselin Mouette) Date: Thu, 02 Oct 2008 13:29:03 +0200 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <20081002112554.GC30869@nexus.in-nomine.org> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> <20081001235808.85D453A4072@sparrow.telecommunity.com> <1222946445.6971.21.camel@shizuru> <20081002112554.GC30869@nexus.in-nomine.org> Message-ID: <1222946943.6971.29.camel@shizuru> Le jeudi 02 octobre 2008 ? 13:25 +0200, Jeroen Ruigrok van der Werven a ?crit : > -On [20081002 13:20], Josselin Mouette (joss at debian.org) wrote: > >Locale data should be shipped in standard form, in /usr/share/locale. > >Again, Python is not the only thing you need to think of. > > Linux is not the only thing to think of by stating /usr/share/locale. :P > I guess ${PREFIX}/share/locale would be better. But still, for .mo files I > seriously do not see a problem if they're within an egg or something > similar. Imagine also the possibility of providing specific language > translations for a project as an egg (kind of like a plugin). There?s definitely no problem with shipping them in an egg, as long as it is possible to ship them at standard locations. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From david at ar.media.kyoto-u.ac.jp Thu Oct 2 13:17:04 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Thu, 02 Oct 2008 20:17:04 +0900 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <1222946445.6971.21.camel@shizuru> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> <20081001235808.85D453A4072@sparrow.telecommunity.com> <1222946445.6971.21.camel@shizuru> Message-ID: <48E4ADB0.8000207@ar.media.kyoto-u.ac.jp> Josselin Mouette wrote: > > I don?t understand why you want to make it so complicated. All you need > is a way to specify directories where different kinds of files land and > a simple API to retrieve the file names/contents. Then, you can ship the > files at the place you like in eggs, and we can ship the files at the > standard places in our packages. > Yes, I don't understand all this complication and concepts either. I have not seen any reason not to do like autoconf. It is simple, obvious, has no burden for the developer, has been used by thousand of softwares for more than one decade, and is extremely flexible. The only thing needed in addition to autoconf is a python API to retrieve the location of each kind of files (python, data, doc, etc...) so that the package itself can find it independently of the way it was installed. cheers, David From asmodai at in-nomine.org Thu Oct 2 13:35:10 2008 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Thu, 2 Oct 2008 13:35:10 +0200 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <1222946943.6971.29.camel@shizuru> References: <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> <20081001235808.85D453A4072@sparrow.telecommunity.com> <1222946445.6971.21.camel@shizuru> <20081002112554.GC30869@nexus.in-nomine.org> <1222946943.6971.29.camel@shizuru> Message-ID: <20081002113510.GD30869@nexus.in-nomine.org> -On [20081002 13:29], Josselin Mouette (joss at debian.org) wrote: >There?s definitely no problem with shipping them in an egg, as long as >it is possible to ship them at standard locations. Standard according to whom though? Contrary to many Linux systems, for example, the BSDs tend to install Python files to /usr/local or /usr/pkg (if using pkgsrc) and not /usr. I'm sure /opt is also still widely used. Furthermore, if by standard you mean */share/locale/XX/LC_MESSAGES/, what good would an egg do at that place? -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B Reality is an illusion, grimmer. The dreamlands are like masks within masks, and Time has no dominion beyond the Shroud... From joss at debian.org Thu Oct 2 13:38:45 2008 From: joss at debian.org (Josselin Mouette) Date: Thu, 02 Oct 2008 13:38:45 +0200 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <20081002113510.GD30869@nexus.in-nomine.org> References: <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> <20081001235808.85D453A4072@sparrow.telecommunity.com> <1222946445.6971.21.camel@shizuru> <20081002112554.GC30869@nexus.in-nomine.org> <1222946943.6971.29.camel@shizuru> <20081002113510.GD30869@nexus.in-nomine.org> Message-ID: <1222947525.6971.33.camel@shizuru> Le jeudi 02 octobre 2008 ? 13:35 +0200, Jeroen Ruigrok van der Werven a ?crit : > -On [20081002 13:29], Josselin Mouette (joss at debian.org) wrote: > >There?s definitely no problem with shipping them in an egg, as long as > >it is possible to ship them at standard locations. > > Standard according to whom though? > > Contrary to many Linux systems, for example, the BSDs tend to install Python > files to /usr/local or /usr/pkg (if using pkgsrc) and not /usr. I'm sure > /opt is also still widely used. That?s not a problem. The only thing that is needed is to be able to select the path at installation time, just like we do with autoconf. BTW, I?m all for making /usr/local the default, since /usr being the default leads to manual installation overwriting packages instead of going to /usr/local. > Furthermore, if by standard you mean */share/locale/XX/LC_MESSAGES/, what > good would an egg do at that place? In the egg distribution, they can go at another place. Again, not a problem as long as we have a flag somewhere that can install them to the place we like. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From pje at telecommunity.com Thu Oct 2 15:44:28 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 02 Oct 2008 09:44:28 -0400 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <48E4A534.7010501@ar.media.kyoto-u.ac.jp> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> <20081001235808.85D453A4072@sparrow.telecommunity.com> <48E42E8E.5090305@gmail.com> <20081002041137.355233A40B0@sparrow.telecommunity.com> <48E4A534.7010501@ar.media.kyoto-u.ac.jp> Message-ID: <20081002134714.7B5143A40B0@sparrow.telecommunity.com> At 07:40 PM 10/2/2008 +0900, David Cournapeau wrote: >Phillip J. Eby wrote: > > > > I think install tools should handle it and keep it out of developers' > > hair. We should of course distinguish configuration and other > > writable data from static data, not to mention documentation. Any > > other file-related info is going to have to be optional, if that. I > > don't really think it's a good idea to ask developers to fill in > > information they don't understand. A developer who works entirely on > > Windows, for example, is not going to have a clue what to specify for > > FHS stuff, and they absolutely shouldn't have to if all they're doing > > is including some static data. > >I may be missing something, but why should the developer even care about >FHS ? We should not standardize what goes where, but the kind of data >needed to be installed (doc, etc...), and then have different >(pluggable) implementations to put those where they should. Yep - that's precisely what I've proposed. We just need to define those categories in such a way as to minimize the burden on a Python developer. Where data files are concerned, however, a developer should only need to distinguish between read-only, read-write, and samples, because any finer-grained distinction that relies on platform-specific concepts (like locale directories) is probably going to be error-prone. From pje at telecommunity.com Thu Oct 2 16:05:49 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 02 Oct 2008 10:05:49 -0400 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <1222946445.6971.21.camel@shizuru> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> <20081001235808.85D453A4072@sparrow.telecommunity.com> <1222946445.6971.21.camel@shizuru> Message-ID: <20081002140440.929FF3A407A@sparrow.telecommunity.com> At 01:20 PM 10/2/2008 +0200, Josselin Mouette wrote: >Le mercredi 01 octobre 2008 ? 19:59 -0400, Phillip J. Eby a ??crit : > > >locale/{message catalogs for various languages} -- These are binary > > >files that contain strings that the user may see when a message is > > >given. These, I think are data. > > > > I lean the other way, since they're not editable. > >Locale data should be shipped in standard form, in /usr/share/locale. >Again, Python is not the only thing you need to think of. Not only, if >you don???t use standard gettext catalogs, you are breaking translation >tools, but if you don???t ship them at the standard location, you are >breaking tools like localepurge (or anything that cleans up the >filesystem based on standard file locations). > > > Keep in mind that > > some platforms have no "locale" directories as such, and thus trying > > to support multiple locations thereof is a burden for cross-platform > > app developers, vs. simply treating them as internal resources. > >No, you are making the burden heavier for Linux platforms by trying to >be more portable. There is no reason why you can???t define a locale >directory on platforms where it does not exist, but when it exists a >standards document defining where files go, you must follow it. Any time >you don???t follow it, we consider it a serious breakage and we have to >patch the code anyway. We are defining an *extensible* standard by which developers and build tools will be able to tell what files in a Python project's distribution directories are what "kind" of files and how they should be handled on a given platform, and a way for installation tools to let the installed project access any files that the tool relocates. That means that: 1. An FHS-friendly installation tool will be able to put data files with .mo/.po extensions wherever it pleases, and 2. An optional "locale info" metadata namespace could be defined by and for such tools, to allow more precise specification regarding these files, if they can't be identified automatically. (Similar to how there will need to be optional metadata namespaces for things like icons and menus for Windows, Mac, and other desktops.) Is this "making the burden heavier for linux platforms"? Well, it is for creating the installation tool, I suppose. And if you can't detect what needs to be done automatically, you'll have to encourage upstream add the information to their specs. However, that will be true whether the information is "optional" or "required". And non-linux platforms will certainly have their own extensions to deal with as well (e.g. registry on Windows, "system" DLLs, exe manifests, etc.). So I think it's reasonable to put platform-specific burdens on the installation tools for those platforms - the whole point being to not have *one* installation tool team that's got to mentally juggle all platforms' requirements. Defining the *spec* is going to be tough enough as it is. >Sorry, but things don???t work this way. Anything that is *not* a .py file >should not land in the python module directories. This is utter abuse of >a loophole of the implementation, and I can???t think of another language >that allows that. You will not find anything else than perl modules in >the perl modules directories. You will not find anything else than C# >modules in the Mono modules directory. And so on. I don't see how this is remotely relevant to how Python works or how people use (and have used) Python over the decades. Data files in package directories is something I've seen in Python since, oh, 1997 or so. If it were a mistake for Python, I think somebody would've noticed by now. In any case, "should" is irrelevant; these packages exist in the field and if the only reason not to have them is to not annoy Linux packagers, there's not much that can be done about it. I could even agree with you, and it would still make no difference: you'd have to get basically *everyone* to agree with you, and it's just not gonna happen. > > Apparently not. The alternative I would suggest is that under the > > new standard, an install tool should be allowed to relocate any > > non-Python files, and all access has to go through a resource > > API. The install tool would then have to be responsible for putting > > some kind of forwarding information in the package directory to tell > > the resource API where it squirrelled the file(s) off to. Then we > > can avoid all this angels-on-a-pin argument and the distros can Have > > It Their Way[tm]. > >I don???t understand why you want to make it so complicated. All you need >is a way to specify directories where different kinds of files land and >a simple API to retrieve the file names/contents. Then, you can ship the >files at the place you like in eggs, and we can ship the files at the >standard places in our packages. How is that not *exactly* what I said in the paragraph above yours? All I added was that "some kind of forwarding information" be used to implement the "simple API to retrieve the file names/contents". In the case of Linux, of course, symlinks would be an ideal "kind of forwarding information" (at least from a Python developer POV), since it would allow most existing programs to work with no changes -- even ones that don't use the pkg_resources API. From ben+python at benfinney.id.au Thu Oct 2 16:20:19 2008 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 03 Oct 2008 00:20:19 +1000 Subject: [Distutils] "Python Package Management Sucks" References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> <20081001235808.85D453A4072@sparrow.telecommunity.com> <1222946445.6971.21.camel@shizuru> <48E4ADB0.8000207@ar.media.kyoto-u.ac.jp> Message-ID: <8763oboz8c.fsf@benfinney.id.au> David Cournapeau writes: > Yes, I don't understand all this complication and concepts either. I > have not seen any reason not to do like autoconf. It is simple, > obvious, has no burden for the developer, has been used by thousand > of softwares for more than one decade, and is extremely flexible. I'm not sure exactly what you mean by ?do like autoconf?. Can you please describe exactly what behaviour you are envisaging, so that we don't all have a different interpretation of what ?like autoconf? means? -- \ ?Broken promises don't upset me. I just think, why did they | `\ believe me?? ?Jack Handey | _o__) | Ben Finney From gael.varoquaux at normalesup.org Thu Oct 2 18:02:17 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 2 Oct 2008 18:02:17 +0200 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <48E44F30.20908@gmail.com> References: <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> <20081001235808.85D453A4072@sparrow.telecommunity.com> <48E42E8E.5090305@gmail.com> <48E44104.9000904@canterbury.ac.nz> <48E44F30.20908@gmail.com> Message-ID: <20081002160217.GE13534@phare.normalesup.org> On Wed, Oct 01, 2008 at 09:33:52PM -0700, Toshio Kuratomi wrote: > Note that Debian has done a lot of neat things with python source > recent(ish). Josselin, Matthias, and some of the other Debian devs > could tell us if .py files get installed to /usr/share there. One of the current options in Debian (python-central) involves installing the .py files to /usr/share/pycentral/package_name and symlinking them in the site-package of each Python version the package is known to work with. You have thus separating of binary and source code. Ga?l From gael.varoquaux at normalesup.org Thu Oct 2 18:04:17 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 2 Oct 2008 18:04:17 +0200 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <1222946445.6971.21.camel@shizuru> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> <20081001235808.85D453A4072@sparrow.telecommunity.com> <1222946445.6971.21.camel@shizuru> Message-ID: <20081002160417.GF13534@phare.normalesup.org> On Thu, Oct 02, 2008 at 01:20:45PM +0200, Josselin Mouette wrote: > Sorry, but things don?t work this way. Anything that is *not* a .py file > should not land in the python module directories. This is utter abuse of > a loophole of the implementation, and I can?t think of another language > that allows that. You will not find anything else than perl modules in > the perl modules directories. You will not find anything else than C# > modules in the Mono modules directory. And so on. What about .so of compiled modules? Ga?l From a.badger at gmail.com Thu Oct 2 19:08:01 2008 From: a.badger at gmail.com (Toshio Kuratomi) Date: Thu, 02 Oct 2008 10:08:01 -0700 Subject: [Distutils] "just use debian" In-Reply-To: <48E4A3AC.8030909@ar.media.kyoto-u.ac.jp> References: <20080917171851.GA27261@logilab.fr> <20080930084253.GF4494@volans.logilab.fr> <94bdd2610809300505u1291ae6an3c44e2ce31d43e3f@mail.gmail.com> <1222780673.4166.55.camel@shizuru> <94bdd2610809300649q6bc9cdf3t84d5221d956bfc9d@mail.gmail.com> <1222784868.4166.88.camel@shizuru> <94bdd2610809300820g36a3f1bbpc2a04e03b43ee9c9@mail.gmail.com> <48E24765.1020908@simplistix.co.uk> <48E24BDA.2050104@simplistix.co.uk> <94bdd2610809300901o3fa62601q985784f01a8ad774@mail.gmail.com> <48E24FF7.9000300@simplistix.co.uk> <48E255B9.50806@colorstudy.com> <1222801361.32659.1.camel@shizuru> <48E2903F.9000307@enthought.com> <1222810334.32659.18.camel@shizuru> <48E2A0BB.6090300@enthought.com> <1222848707.7060.4.camel@tomoyo> <48E333C2.9030706@ar.media.kyoto-u.ac.jp> <1222851502.7060.18.camel@tomoyo> <48E33A91.5000602@ar.media.kyoto-u.ac.jp> <48E44334.4040405@gmail.com> <48E4A3AC.8030909@ar.media.kyoto-u.ac.jp> Message-ID: <48E4FFF1.4040504@gmail.com> David Cournapeau wrote: > Toshio Kuratomi wrote: >> I'm not 100% certain but I think that Josselin is speaking of glibc in >> particular here and you're speaking of c libraries in general. > > Maybe, but I don't see how this change the point: when you change the > soname of a library, it has 0 impact on the source code of the software > which links against it. That's not true in python. > . I just noticed that you guys seemed to be speaking past each other and wanted to point it out. >> >> I've said before that ideally a Linux distribution only wants one >> version of a library. > > And ideally, developers do not want to care about versioning issues :) > There is a middle ground to find; up to now, I think python distutils > and co did not care at all about this, but you can not ask to move 180 > degrees and do only as linux vendors want. I understand the reasons why > OS vendors want to avoid distributing multiple versions as much as > possible. But that's just not realistic in every case. > > Even in C-derived languages, this is sometimes a PITA. I could give you > examples where distributions screwed up the packaging badly and hurt > some projects I am involved with. But that would be unfair and besides > the point (we all screw up); the point is that there are some practical > reasons for sometimes including private copies. Because for example in > Fortran's case, there is this huge mess of gfortran and g77 not being > ABI compatible; there are examples in C++, and even in C. You also can't > impose every software to follow distributions time-schedule. > > Reasons for single version for OS vendors are valid; but so are the ones > to have multiple versions. I think compat modules would cover most > needsl the problem is that python does not have a mechanism to request a > particular version of a module. But wouldn't this help OS vendors to > have such a mechanism (to decrease the burden of compat versions ?) > Very true! Which is why say single package versions are ideal for Linux distributions. Ideal being one of those ever striven for, never achieved goals and Linux distributions being the demographic whose opinion I'm pretending to represent :-) I can definitely understand the need to develop packages with different versions of python packages than a system might have. (I develop software as well as package it.) So where the concerns intersect is when you go to distribute your package. To have your package run in as many places as possible, you want to guarantee the correct versions of libraries are installed in those places. OTOH, in order to get Linux distributions interested in packaging your project you really must not use your own private copies of those libraries. (BTW, Josselin seems to be saying something different is true on Debian but I had posted this question to the cross-distro distributions-list freedesktop.org two weeks ago after dealing with it in the banshee package and people seemed to agree that it was not proper packaging. I'll have to ask for clarification here. Perhaps I phrased my question poorly on that list :-) Anyhow... the problems I outlined in my mail are the reasons that packagers have a wtf moment when they untar a source tarball and find that there's fifteen other upstream packages included. Ways that this can be remedied: 1) Have a source tarball that's separate from binary distribution. The binary distribution can contain the third party modules while the source tarball just contains your code. Distro packagers will love you for this because it means the source tarball is clean and they can just g to work packaging it. 2) If you must distribute the source tarball with third party modules, make sure your build scripts work with the installed system packages instead of the modules you are including. This lets a packager build and install just your code and ignore the rest. 3) Make sure you document how to do this. Good packagers read the README. If you have to rm -rf THIRD_PARTY-DIR prior to building to just build and install your code, mention that. 4) make sure your package works with vanilla upstream versions of the third party modules. It's tempting to fix things in your local copies of modules. If at all possible don't. If that's not possible, make sure upstream has incorporated the patch and make a note in the README -- using a patched version of Foo-x.y project. The patch is in their svn as of DATE. patch is located in myproject/foo-x.y/patches. Doing this means that the distribution packager of your package can take your patch to the packager of Foo and ask that the patch be incorporated there. >> In Fedora we're willing to have compat packages >> that hold old API versions if we must but by and large we would rather >> help upstream apps port their applications forward than to have >> compatibility packages. > > Yes, but here again the C comparison breaks. Some people use python as a > "tool", not so much as a "programming language". Their applications are > scripts, or softwares for experiment, that are not released, because > they can't open source it, or simply because it has no use for anyone > else. You can't port that. > If complaints in Fedora are any indication this happens with C as well ;-). If by you, you mean me or a distribution you're right. Of course, the "can't port that" doesn't apply to the person with the script. The solution at the distro level for the non-OSS software in deployment scenario is not to run a distribution with a limited lifespan. If you need something to run reliably on python2.4 for years, run RHEL5 or CentOS or Debian stable. They won't change the major components without reason and you'll be able to run just your changes (more recent packages, etc) on top. There is no solution at the distro level for experimental, in development stuff. But I think that using eggs, workingenv, etc is a fine solution for the development case. scripts and small, local software is a problem. "I have a script to download por^Wfiles from the internet. You started shipping py3k and now urllib is busted!" Debian has solved one portion of this by shipping different versions of python that are parallel installable. Fedora has solved a different portion by shipping compat packages (using setuptools for parallel install) when there's a need. If the software is small the best answer may be to port. If the software is intricate, the best answer may be to use workingenv or something else from the development case. I think it's important to note that I see different use cases for using distribution packages and using local solutions like workingenv. Local solutions let you use more current (or less current) versions of things. They let you experiment and give you the option to refuse to port your local scripts. Supporting people overriding what's installed on their OS distro is important for doing these things. But distro packaging serves a very useful purpose as well. It lets people experiment with your work who are just looking for something that can get a specific job done. It lets developers worry about something other than security fixes to packages which they depend on. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: OpenPGP digital signature URL: From a.badger at gmail.com Thu Oct 2 19:33:48 2008 From: a.badger at gmail.com (Toshio Kuratomi) Date: Thu, 02 Oct 2008 10:33:48 -0700 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <20081002041137.355233A40B0@sparrow.telecommunity.com> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> <20081001235808.85D453A4072@sparrow.telecommunity.com> <48E42E8E.5090305@gmail.com> <20081002041137.355233A40B0@sparrow.telecommunity.com> Message-ID: <48E505FC.8060102@gmail.com> Phillip J. Eby wrote: > At 07:14 PM 10/1/2008 -0700, Toshio Kuratomi wrote: >> In terms of implementation I'd much rather see something less centered >> on the egg being the right way and the filesystem being a secondary >> concern. > > Eggs don't have anything to do with it; in Python, it's simply common > sense to put static resources next to the code that uses them, if you > want to "write once, run anywhere". And given Python's strength as an > interactive development language with no "build" step, having to > *install* your data files somewhere else on the system to use them isn't > a *feature* -- not for a developer, anyway. > You're arguing about the developers point of view on something that's hidden behind an API. You've already made it so that the developer cannot just reference the file on the filesystem because the egg may be zipped. So for the developer there's no change here. I'm saying that there's no need to have a hardcoded path to lookup the information at and then make the install tool place "forwarding information" there to send the package somewhere else. We have metadata. We should use it. > And our hypothetical de-jure standard won't replace the de-facto > standard unless it's adopted by developers... and it won't be adopted > if it makes their lives harder without a compensating benefit. For the > developer, FHS support is a cost, not a benefit, and only relevant to a > subset of platforms, so the spec should make it as transparent for them > as possible, if they don't have an interest in explicit support for it. > By the STASCTAP principle (Simple Things Are Simple, Complex Things Are > Possible), it should be possible for distros to relocate, and simple for > developers not to care about it. > It's both a cost and a benefit. The cost is having to use an API which they have to use anyway due to eggs possibly being zip files. The benefit is getting their code packaged by Linux distributors quicker and getting more contributors as a result of the exposure. > >> We should have metadata that tells us where the types of >> resources come from. When a package is installed on Linux the metadata >> could point locales at file:///usr/share/locale. When on Windows >> egg:locale (Perhaps the uninstalled case would use this too... that >> depends on how the egg structure and metadata evolves.) >> >> A question we'd have to decide is whether this particular metadata is >> something that should be defined globally or per package. Or globally >> with a chance for packages to override it. > > I think install tools should handle it and keep it out of developers' > hair. We should of course distinguish configuration and other writable > data from static data, not to mention documentation. Any other > file-related info is going to have to be optional, if that. I don't > really think it's a good idea to ask developers to fill in information > they don't understand. A developer who works entirely on Windows, for > example, is not going to have a clue what to specify for FHS stuff, and > they absolutely shouldn't have to if all they're doing is including some > static data. > Needing to have some information about the files you ship is inevitable. Documentation is a good example. man pages, License.txt, gnome help files, windows help files, API docs, sphinx docs, etc each have to be installed in different places, some with requirements to register the files so the system knows they exist. All the knowledge about what to do with these files should be placed in the tool. But the knowledge of what type to mark a given file with will have to lay with the developer. > Even today, there exist Python developers who don't use the distutils to > distribute their packages, so anything that makes it even more difficult > than it is today, isn't going to be a viable standard. The closer we > can get in ease of use to just tarring up a directory, the more viable > it'll be. (That's one reason, btw, why setuptools offers revision > control support and find_packages() for automating discovery of what to > include.) > Actually, as a person who distributes upstream packages which don't use distutils and is exposed to others, I'd say that the shortcomings in terms of where to install files and how to reference the files after install is one of the reasons that distutils is not used. Are there other reasons? Sure. But this is definitely one of the reasons. > >> > I'd have preferred to avoid that complexity, but if the two of us can't >> > agree then there's no way on earth to get a community consensus. >> > >> > Btw, pkg_resources' concept of "metadata" would also need to be >> > relocatable, since e.g. the "EggTranslations" package uses that >> metadata >> > to store localizations of image resources and message catalogs. (Other >> > uses of the metadata files also inlcude scripts, dependencies, version >> > info, etc.) >> > >> Actually, we should decide whether we want to support that kind of thing >> within the egg metadata at all. The other things we've been talking >> about belonging in the metadata are simple key value pairs. >> EggTranslations uses the metadata area as a data store. (Or in your >> definition, a resource store). This breaks with the definition of what >> metadata is. Translations don't store information about a package, they >> store alternate views of data within the package. > > I was actually somewhat incorrect in my statement about the distinction > between pkg_resources "metadata" and "resources"; "metadata" is really > "data that goes with the distribution, not with a specific package > within the distribution". Only some of this data is "about" the > distribution; the rest is data "with" or "of" the distribution. (This > is a slight API wart, but the use case exists nonetheless.) > > Meanwhile, regarding the proposed key-value pairs system, I don't see > how that works; "extras" dependency information and entry points are a > bit more structured than just key-value pairs; both are currently > represented as .ini-like files with arbitrary section names. I suppose > you could squash those entire files into values in some sort of > key-value system, but that seems a bit hairy to me. In particular, > setuptools design choice for separate metadata files is that many of > these things don't need to be loaded at the same time. Also, > PKG-INFO-style metadata can contain rather large blobs of text that > aren't needed or useful at runtime. Entry points and extras are mostly > runtime metadata, with the occasional bit of build or install usage. > Structured, yes. Structure and optimizations to how you lookup the data is good. But there is a difference between using metadata to save and lookup configuration and using metadata to save and lookup data (like locale files). You wouldn't save data into gconf or the Windows Registry for instance (at least, not if you don't expect people to make fun of you *cough*evolution*cough*). OTOH if it's not really a metadata store vs a resource store but instead a package store vs a distribution store we need to decide if we really want to have both. Someone pointed out earlier that Side note: the fact that someone wrote EggTranslations speaks of a need for people to be able to access the per-package data store across packages. Let's fix that and work with EggTranslations to rewrite its backend to use a proper storage. (Looking at the EggTranslations documentation, it might even be a proper place for getting ideas and help with designing the API for a public data store.) -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: OpenPGP digital signature URL: From joss at debian.org Thu Oct 2 19:47:39 2008 From: joss at debian.org (Josselin Mouette) Date: Thu, 02 Oct 2008 19:47:39 +0200 Subject: [Distutils] "just use debian" In-Reply-To: <48E4FFF1.4040504@gmail.com> References: <20080917171851.GA27261@logilab.fr> <20080930084253.GF4494@volans.logilab.fr> <94bdd2610809300505u1291ae6an3c44e2ce31d43e3f@mail.gmail.com> <1222780673.4166.55.camel@shizuru> <94bdd2610809300649q6bc9cdf3t84d5221d956bfc9d@mail.gmail.com> <1222784868.4166.88.camel@shizuru> <94bdd2610809300820g36a3f1bbpc2a04e03b43ee9c9@mail.gmail.com> <48E24765.1020908@simplistix.co.uk> <48E24BDA.2050104@simplistix.co.uk> <94bdd2610809300901o3fa62601q985784f01a8ad774@mail.gmail.com> <48E24FF7.9000300@simplistix.co.uk> <48E255B9.50806@colorstudy.com> <1222801361.32659.1.camel@shizuru> <48E2903F.9000307@enthought.com> <1222810334.32659.18.camel@shizuru> <48E2A0BB.6090300@enthought.com> <1222848707.7060.4.camel@tomoyo> <48E333C2.9030706@ar.media.kyoto-u.ac.jp> <1222851502.7060.18.camel@tomoyo> <48E33A91.5000602@ar.media.kyoto-u.ac.jp> <48E44334.4040405@gmail.com> <48E4A3AC.8030909@ar.media.kyoto-u.ac.jp> <48E4FFF1.4040504@gmail.com> Message-ID: <1222969659.6971.102.camel@shizuru> Le jeudi 02 octobre 2008 ? 10:08 -0700, Toshio Kuratomi a ?crit : > So where the concerns intersect is when you go to distribute your > package. To have your package run in as many places as possible, you > want to guarantee the correct versions of libraries are installed in > those places. OTOH, in order to get Linux distributions interested in > packaging your project you really must not use your own private copies > of those libraries. > > (BTW, Josselin seems to be saying something different is true on Debian > but I had posted this question to the cross-distro distributions-list > freedesktop.org two weeks ago after dealing with it in the banshee > package and people seemed to agree that it was not proper packaging. > I'll have to ask for clarification here. Perhaps I phrased my question > poorly on that list :-) I?m not sure I understand what point you think is different on Debian. We ship several versions of Python at once, but we do not ship several versions of a module unless absolutely necessary. > 1) Have a source tarball that's separate from binary distribution. The > binary distribution can contain the third party modules while the source > tarball just contains your code. Distro packagers will love you for > this because it means the source tarball is clean and they can just g to > work packaging it. Full ACK. Repackaging is a pain. > 4) make sure your package works with vanilla upstream versions of the > third party modules. It's tempting to fix things in your local copies > of modules. If at all possible don't. If that's not possible, make > sure upstream has incorporated the patch and make a note in the README > -- using a patched version of Foo-x.y project. The patch is in their > svn as of DATE. patch is located in myproject/foo-x.y/patches. Doing > this means that the distribution packager of your package can take your > patch to the packager of Foo and ask that the patch be incorporated there. Amen. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From pje at telecommunity.com Thu Oct 2 19:57:58 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 02 Oct 2008 13:57:58 -0400 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <48E505FC.8060102@gmail.com> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> <20081001235808.85D453A4072@sparrow.telecommunity.com> <48E42E8E.5090305@gmail.com> <20081002041137.355233A40B0@sparrow.telecommunity.com> <48E505FC.8060102@gmail.com> Message-ID: <20081002175648.11C3B3A409C@sparrow.telecommunity.com> At 10:33 AM 10/2/2008 -0700, Toshio Kuratomi wrote: >OTOH if it's not really a metadata store vs a resource store but instead >a package store vs a distribution store we need to decide if we really >want to have both. Someone pointed out earlier that > >Side note: the fact that someone wrote EggTranslations speaks of a need >for people to be able to access the per-package data store across >packages. Let's fix that and work with EggTranslations to rewrite its >backend to use a proper storage. (Looking at the EggTranslations >documentation, it might even be a proper place for getting ideas and >help with designing the API for a public data store.) If we want to have something that can be adopted with any speed, we're going to have to rule out of scope anything that forces people to change their *runtime* code (vs. packaging code). Even setuptools doesn't require that people use the API; it detects when a program is probably reading stuff directly and installs packages unzipped in that case. So, let's focus the discussion towards ways to make it possible for people to declare what they're *already* doing; otherwise, we are just adding to the switching cost for the new system, which needs to be kept as low as possible. The new system has to be more attractive to developers in the general case, in order to overcome the cost of switching, so adding *new* switching costs is to be avoided at all... costs. :) From nicolas.chauvat at logilab.fr Thu Oct 2 20:08:42 2008 From: nicolas.chauvat at logilab.fr (Nicolas Chauvat) Date: Thu, 2 Oct 2008 20:08:42 +0200 Subject: [Distutils] "just use debian" In-Reply-To: <48E4FFF1.4040504@gmail.com> References: <48E2903F.9000307@enthought.com> <1222810334.32659.18.camel@shizuru> <48E2A0BB.6090300@enthought.com> <1222848707.7060.4.camel@tomoyo> <48E333C2.9030706@ar.media.kyoto-u.ac.jp> <1222851502.7060.18.camel@tomoyo> <48E33A91.5000602@ar.media.kyoto-u.ac.jp> <48E44334.4040405@gmail.com> <48E4A3AC.8030909@ar.media.kyoto-u.ac.jp> <48E4FFF1.4040504@gmail.com> Message-ID: <20081002180842.GA9247@volans.logilab.fr> On Thu, Oct 02, 2008 at 10:08:01AM -0700, Toshio Kuratomi wrote: > 4) make sure your package works with vanilla upstream versions of the > third party modules. It's tempting to fix things in your local copies > of modules. If at all possible don't. If that's not possible, make > sure upstream has incorporated the patch and make a note in the README > -- using a patched version of Foo-x.y project. The patch is in their > svn as of DATE. patch is located in myproject/foo-x.y/patches. Doing > this means that the distribution packager of your package can take your > patch to the packager of Foo and ask that the patch be incorporated there. Mercurial's patch queues can be of great help for this. http://www.selenic.com/mercurial/wiki/index.cgi/MqExtension > development stuff. But I think that using eggs, workingenv, etc is a > fine solution for the development case. Someone told me about http://0install.net/ but I have not tested it and do not know how good/bad it is. -- Nicolas Chauvat logilab.fr - services en informatique scientifique et gestion de connaissances From floris.bruynooghe at gmail.com Thu Oct 2 21:08:04 2008 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Thu, 2 Oct 2008 20:08:04 +0100 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <20081002140440.929FF3A407A@sparrow.telecommunity.com> References: <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> <20081001235808.85D453A4072@sparrow.telecommunity.com> <1222946445.6971.21.camel@shizuru> <20081002140440.929FF3A407A@sparrow.telecommunity.com> Message-ID: <20081002190804.GA7702@laurie.devork> On Thu, Oct 02, 2008 at 10:05:49AM -0400, Phillip J. Eby wrote: > I don't see how this is remotely relevant to how Python works or how > people use (and have used) Python over the decades. Data files in > package directories is something I've seen in Python since, oh, 1997 or > so. If it were a mistake for Python, I think somebody would've noticed > by now. Well, maybe someone just did notice! I don't see why you disagree so strongly with this as it is entirely in line with providing separate source tree locations for differents types of data which you are in favour of. Specifiying that this would be the recommended way and providing an API to deal with it doesn't mean you forbit the old way. You'd still be able to dig around with __file__. Even if any GNU/Linux packagers will try and convince upstreams not to do so. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From pje at telecommunity.com Thu Oct 2 21:49:43 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 02 Oct 2008 15:49:43 -0400 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <20081002190804.GA7702@laurie.devork> References: <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> <20081001235808.85D453A4072@sparrow.telecommunity.com> <1222946445.6971.21.camel@shizuru> <20081002140440.929FF3A407A@sparrow.telecommunity.com> <20081002190804.GA7702@laurie.devork> Message-ID: <20081002194834.790E83A409C@sparrow.telecommunity.com> At 08:08 PM 10/2/2008 +0100, Floris Bruynooghe wrote: >On Thu, Oct 02, 2008 at 10:05:49AM -0400, Phillip J. Eby wrote: > > I don't see how this is remotely relevant to how Python works or how > > people use (and have used) Python over the decades. Data files in > > package directories is something I've seen in Python since, oh, 1997 or > > so. If it were a mistake for Python, I think somebody would've noticed > > by now. > >Well, maybe someone just did notice! I don't see why you disagree so >strongly with this as it is entirely in line with providing separate >source tree locations for differents types of data which you are in >favour of. > >Specifiying that this would be the recommended way and providing an >API to deal with it doesn't mean you forbit the old way. You'd still >be able to dig around with __file__. Even if any GNU/Linux packagers >will try and convince upstreams not to do so. We're actually in "violent agreement" here regarding what is to be done at a practical level. Linux packagers want it one way, developers want it another, and both can have what they want... unless they want the other guys to agree their way is the "right" way of doing it. ;-) That is, the Linux packagers appear to be upset and offended by the mere idea that anyone should consider the Python way to be correct, but it's not reasonable to expect people to change their minds, any more than it's reasonable to exepct the Linux packagers to agree the Python way is the right way and that everything they've been doing is wrong. (And if you respond to this idea by thinking that it would be because the Linux packagers are *right*, then you're simply illustrating my point.) Now, on a practical level, if what you are trying to say is that everybody should access all data strictly through some new API created for that purpose, then that's a non-starter for practical reasons, rather than ideological ones. Even setuptools doesn't *require* that people use an API to access their static data, and does its darnedest to support that use case. And, if you want a new spec to actually *replace* setuptools, then you're going to have to be very careful about what setuptools features are dropped, or the spec is unlikely to go from "de jure" to "de facto". From jim at zope.com Fri Oct 3 00:09:15 2008 From: jim at zope.com (Jim Fulton) Date: Thu, 2 Oct 2008 18:09:15 -0400 Subject: [Distutils] [zc.buildout] running in safe mode In-Reply-To: <94bdd2610810010334g3e3994bcw5cfa270de3ae73e7@mail.gmail.com> References: <94bdd2610810010334g3e3994bcw5cfa270de3ae73e7@mail.gmail.com> Message-ID: <78EFD40E-0971-4705-BDD9-5064DF8DC880@zope.com> On Oct 1, 2008, at 6:34 AM, Tarek Ziad? wrote: > Hello > > I know it is a bad practice for a recipe to return some paths that > contains important data in the install() method, > because zc.buildout might remove them. > > Nevertheless, it happens from time to time that a developer lose some > content because of a misconfiguration, > or a zealous recipe. That is his responsability, and backups are > done for that. I don't think backups are the right approach. It's a mistake to have recipes manage precious data. If you really really really think that's a good idea, then the recipe should at least manage uninstall and move precious data aside, rather than remove it. I don't think it is really the user's problem is a recipe misbehaves by allowing precious data to be removed. > But I think we can improve zc.buildout a bit for that: > > what about introducing a safe-mode where the developer get prompted > everytime zc.buildout.rmtree is about to call > shutil.rmtree ? > > The option could be set in [buildout] like this: > > [buildout] > ... > safe-mode = true > ... > > and challenge the user when a tree is about to be delete. (it might be > overkill for single files, > and they are most of the time unimportant configuration files) > > This is a small change, and it would avoid running a backup everytime > the .cfg are changed. because > it happens all day long when you are developing. I suspect this would be so annoying to use that no one would use it. I think it's probably easier to fix the broken recipes. I also think calling this "safe" mode is misleading. I could live with an option named something like "prompt-before-removing-files-or-directories". A better option might be something like "move-aside-on-uninstall", which would move files or directories aside rather than deleting them. I still think it would be better to just fix the broken recipes. Jim -- Jim Fulton Zope Corporation From ianb at colorstudy.com Fri Oct 3 00:15:10 2008 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 02 Oct 2008 17:15:10 -0500 Subject: [Distutils] [zc.buildout] running in safe mode In-Reply-To: <78EFD40E-0971-4705-BDD9-5064DF8DC880@zope.com> References: <94bdd2610810010334g3e3994bcw5cfa270de3ae73e7@mail.gmail.com> <78EFD40E-0971-4705-BDD9-5064DF8DC880@zope.com> Message-ID: <48E547EE.6080007@colorstudy.com> Jim Fulton wrote: >> I know it is a bad practice for a recipe to return some paths that >> contains important data in the install() method, >> because zc.buildout might remove them. >> >> Nevertheless, it happens from time to time that a developer lose some >> content because of a misconfiguration, >> or a zealous recipe. That is his responsability, and backups are done >> for that. > > I don't think backups are the right approach. It's a mistake to have > recipes manage precious data. If you really really really think that's > a good idea, then the recipe should at least manage uninstall and move > precious data aside, rather than remove it. > > I don't think it is really the user's problem is a recipe misbehaves by > allowing precious data to be removed. I'll note fassembler uses a file abstraction layer so that its recipes are safe by default: https://svn.openplans.org/svn/fassembler/trunk/fassembler/filemaker.py I think buildout would be a lot more humane if it took the same approach. -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From jim at zope.com Fri Oct 3 00:26:52 2008 From: jim at zope.com (Jim Fulton) Date: Thu, 2 Oct 2008 18:26:52 -0400 Subject: [Distutils] [zc.buildout] running in safe mode In-Reply-To: <48E547EE.6080007@colorstudy.com> References: <94bdd2610810010334g3e3994bcw5cfa270de3ae73e7@mail.gmail.com> <78EFD40E-0971-4705-BDD9-5064DF8DC880@zope.com> <48E547EE.6080007@colorstudy.com> Message-ID: <07F59784-77C8-4988-B64C-5573F7432302@zope.com> On Oct 2, 2008, at 6:15 PM, Ian Bicking wrote: > Jim Fulton wrote: >>> I know it is a bad practice for a recipe to return some paths that >>> contains important data in the install() method, >>> because zc.buildout might remove them. >>> >>> Nevertheless, it happens from time to time that a developer lose >>> some >>> content because of a misconfiguration, >>> or a zealous recipe. That is his responsability, and backups are >>> done for that. >> I don't think backups are the right approach. It's a mistake to >> have recipes manage precious data. If you really really really >> think that's a good idea, then the recipe should at least manage >> uninstall and move precious data aside, rather than remove it. >> I don't think it is really the user's problem is a recipe >> misbehaves by allowing precious data to be removed. > > I'll note fassembler uses a file abstraction layer so that its > recipes are safe by default: https://svn.openplans.org/svn/fassembler/trunk/fassembler/filemaker.py > > I think buildout would be a lot more humane if it took the same > approach. I'd be interested to know what you mean by this, but I'm not willing to read that source to find out. Can you be a little more specific? Jim -- Jim Fulton Zope Corporation From ziade.tarek at gmail.com Fri Oct 3 00:32:05 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 3 Oct 2008 00:32:05 +0200 Subject: [Distutils] [zc.buildout] running in safe mode In-Reply-To: <78EFD40E-0971-4705-BDD9-5064DF8DC880@zope.com> References: <94bdd2610810010334g3e3994bcw5cfa270de3ae73e7@mail.gmail.com> <78EFD40E-0971-4705-BDD9-5064DF8DC880@zope.com> Message-ID: <94bdd2610810021532n6ac8b9c3w70cc8f5f54b5d724@mail.gmail.com> On Fri, Oct 3, 2008 at 12:09 AM, Jim Fulton wrote: > > A better option might be something like "move-aside-on-uninstall", which > would move files or directories aside rather than deleting them. ok why not, > > I still think it would be better to just fix the broken recipes. I agree it is the solution, but we tend not to be able to control all the recipes, and zc.buildout just calls out rmtree in some cases, wich is rather violent imho. ++ Tarek -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From ianb at colorstudy.com Fri Oct 3 00:37:05 2008 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 02 Oct 2008 17:37:05 -0500 Subject: [Distutils] [zc.buildout] running in safe mode In-Reply-To: <07F59784-77C8-4988-B64C-5573F7432302@zope.com> References: <94bdd2610810010334g3e3994bcw5cfa270de3ae73e7@mail.gmail.com> <78EFD40E-0971-4705-BDD9-5064DF8DC880@zope.com> <48E547EE.6080007@colorstudy.com> <07F59784-77C8-4988-B64C-5573F7432302@zope.com> Message-ID: <48E54D11.4030807@colorstudy.com> Jim Fulton wrote: > > On Oct 2, 2008, at 6:15 PM, Ian Bicking wrote: > >> Jim Fulton wrote: >>>> I know it is a bad practice for a recipe to return some paths that >>>> contains important data in the install() method, >>>> because zc.buildout might remove them. >>>> >>>> Nevertheless, it happens from time to time that a developer lose some >>>> content because of a misconfiguration, >>>> or a zealous recipe. That is his responsability, and backups are >>>> done for that. >>> I don't think backups are the right approach. It's a mistake to have >>> recipes manage precious data. If you really really really think >>> that's a good idea, then the recipe should at least manage uninstall >>> and move precious data aside, rather than remove it. >>> I don't think it is really the user's problem is a recipe misbehaves >>> by allowing precious data to be removed. >> >> I'll note fassembler uses a file abstraction layer so that its recipes >> are safe by default: >> https://svn.openplans.org/svn/fassembler/trunk/fassembler/filemaker.py >> >> I think buildout would be a lot more humane if it took the same approach. > > > I'd be interested to know what you mean by this, but I'm not willing to > read that source to find out. > > Can you be a little more specific? Instead of using open(), etc, to write files, there's an instance of Maker which holds some of the settings (--interactive, --simulate, a base directory). Then you do all your file operations like: maker.ensure_file('path/to/file.txt', content) If that file exists with different content then the user gets asked about what to do. It also logs all the writing, shows diffs, can make backups, etc. You can force overwriting, but that's a keyword argument that defaults to False, so only if you actually have good reason to overwrite files (without asking) then that's fine, but you will start developing the easy way, which is to be safe about this stuff. -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From ziade.tarek at gmail.com Fri Oct 3 00:38:56 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 3 Oct 2008 00:38:56 +0200 Subject: [Distutils] [zc.buildout] running in safe mode In-Reply-To: <48E54D11.4030807@colorstudy.com> References: <94bdd2610810010334g3e3994bcw5cfa270de3ae73e7@mail.gmail.com> <78EFD40E-0971-4705-BDD9-5064DF8DC880@zope.com> <48E547EE.6080007@colorstudy.com> <07F59784-77C8-4988-B64C-5573F7432302@zope.com> <48E54D11.4030807@colorstudy.com> Message-ID: <94bdd2610810021538v7fa34209v515675b49d9f04a8@mail.gmail.com> On Fri, Oct 3, 2008 at 12:37 AM, Ian Bicking wrote: >> Can you be a little more specific? > > Instead of using open(), etc, to write files, there's an instance of Maker > which holds some of the settings (--interactive, --simulate, a base > directory). Then you do all your file operations like: > > maker.ensure_file('path/to/file.txt', content) > > If that file exists with different content then the user gets asked about > what to do. That is basically what I wanted to introduce From greg.ewing at canterbury.ac.nz Fri Oct 3 04:04:27 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 03 Oct 2008 15:04:27 +1300 Subject: [Distutils] AMP, anyone? (Re: "just use debian") In-Reply-To: <48E4A3AC.8030909@ar.media.kyoto-u.ac.jp> References: <20080917171851.GA27261@logilab.fr> <20080930084253.GF4494@volans.logilab.fr> <94bdd2610809300505u1291ae6an3c44e2ce31d43e3f@mail.gmail.com> <1222780673.4166.55.camel@shizuru> <94bdd2610809300649q6bc9cdf3t84d5221d956bfc9d@mail.gmail.com> <1222784868.4166.88.camel@shizuru> <94bdd2610809300820g36a3f1bbpc2a04e03b43ee9c9@mail.gmail.com> <48E24765.1020908@simplistix.co.uk> <48E24BDA.2050104@simplistix.co.uk> <94bdd2610809300901o3fa62601q985784f01a8ad774@mail.gmail.com> <48E24FF7.9000300@simplistix.co.uk> <48E255B9.50806@colorstudy.com> <1222801361.32659.1.camel@shizuru> <48E2903F.9000307@enthought.com> <1222810334.32659.18.camel@shizuru> <48E2A0BB.6090300@enthought.com> <1222848707.7060.4.camel@tomoyo> <48E333C2.9030706@ar.media.kyoto-u.ac.jp> <1222851502.7060.18.camel@tomoyo> <48E33A91.5000602@ar.media.kyoto-u.ac.jp> <48E44334.4040405@gmail.com> <48E4A3AC.8030909@ar.media.kyoto-u.ac.jp> Message-ID: <48E57DAB.2020806@canterbury.ac.nz> David Cournapeau wrote: > when you change the > soname of a library, it has 0 impact on the source code of the software > which links against it. That's not true in python. In the Eiffel world, there's a thing called an ACE, which stands for Assembly of Classes in Eiffel. It's a way of specifying a mapping between class names used internally in the code and where to get them from in the environment. I think Python could do with something similar for managing version issues. Perhaps we could study how ACEs work and see if we can use any ideas from there. We could call it an AMP -- Assembly of Modules in Python. -- Greg From greg.ewing at canterbury.ac.nz Fri Oct 3 04:14:45 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 03 Oct 2008 15:14:45 +1300 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <20081002113510.GD30869@nexus.in-nomine.org> References: <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> <20081001235808.85D453A4072@sparrow.telecommunity.com> <1222946445.6971.21.camel@shizuru> <20081002112554.GC30869@nexus.in-nomine.org> <1222946943.6971.29.camel@shizuru> <20081002113510.GD30869@nexus.in-nomine.org> Message-ID: <48E58015.80509@canterbury.ac.nz> Jeroen Ruigrok van der Werven wrote: > -On [20081002 13:29], Josselin Mouette (joss at debian.org) wrote: >> There?s definitely no problem with shipping them in an egg, as long as >> it is possible to ship them at standard locations. > > Standard according to whom though? I think the idea is to *define* a standard location within the egg. An install tool is then free to extract them out of the egg and put them somewhere else if it wants. There should also be an API for finding them (rather than assuming they're still inside the egg) so that the code that uses them can still find them if they've been moved somewhere platform-specific. -- Greg From greg.ewing at canterbury.ac.nz Fri Oct 3 04:25:00 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 03 Oct 2008 15:25:00 +1300 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <20081002134714.7B5143A40B0@sparrow.telecommunity.com> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> <20081001235808.85D453A4072@sparrow.telecommunity.com> <48E42E8E.5090305@gmail.com> <20081002041137.355233A40B0@sparrow.telecommunity.com> <48E4A534.7010501@ar.media.kyoto-u.ac.jp> <20081002134714.7B5143A40B0@sparrow.telecommunity.com> Message-ID: <48E5827C.4000606@canterbury.ac.nz> Phillip J. Eby wrote: > Where data files are concerned, however, a developer should only need to > distinguish between read-only, read-write, and samples, because any > finer-grained distinction that relies on platform-specific concepts > (like locale directories) is probably going to be error-prone. However, if we don't allow for such distinctions, then people who e.g. are passionate about their locale files ending up in the right place on Linux aren't going to be happy. -- Greg From greg.ewing at canterbury.ac.nz Fri Oct 3 04:57:23 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 03 Oct 2008 15:57:23 +1300 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <20081002160417.GF13534@phare.normalesup.org> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> <20081001235808.85D453A4072@sparrow.telecommunity.com> <1222946445.6971.21.camel@shizuru> <20081002160417.GF13534@phare.normalesup.org> Message-ID: <48E58A13.2080601@canterbury.ac.nz> On Thu, Oct 02, 2008 at 01:20:45PM +0200, Josselin Mouette wrote: > Sorry, but things don?t work this way. Anything that is *not* a .py file > should not land in the python module directories. So where *should* they go, on platforms where there is no defined place for such things? And how can the code that uses them find them without platform-specific knowledge? -- Greg From pje at telecommunity.com Fri Oct 3 05:11:26 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 02 Oct 2008 23:11:26 -0400 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <48E5827C.4000606@canterbury.ac.nz> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> <20081001235808.85D453A4072@sparrow.telecommunity.com> <48E42E8E.5090305@gmail.com> <20081002041137.355233A40B0@sparrow.telecommunity.com> <48E4A534.7010501@ar.media.kyoto-u.ac.jp> <20081002134714.7B5143A40B0@sparrow.telecommunity.com> <48E5827C.4000606@canterbury.ac.nz> Message-ID: <20081003031025.ACA413A4072@sparrow.telecommunity.com> At 03:25 PM 10/3/2008 +1300, Greg Ewing wrote: >Phillip J. Eby wrote: > >>Where data files are concerned, however, a developer should only >>need to distinguish between read-only, read-write, and samples, >>because any finer-grained distinction that relies on >>platform-specific concepts (like locale directories) is probably >>going to be error-prone. > >However, if we don't allow for such distinctions, then people >who e.g. are passionate about their locale files ending up >in the right place on Linux aren't going to be happy. Yep. As I've said, it should be possible to define optional extensions that define these sorts of things. They're needed for e.g. icons, Windows registry entries, etc. anyway. From david at ar.media.kyoto-u.ac.jp Fri Oct 3 06:37:02 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 03 Oct 2008 13:37:02 +0900 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <8763oboz8c.fsf@benfinney.id.au> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> <20081001235808.85D453A4072@sparrow.telecommunity.com> <1222946445.6971.21.camel@shizuru> <48E4ADB0.8000207@ar.media.kyoto-u.ac.jp> <8763oboz8c.fsf@benfinney.id.au> Message-ID: <48E5A16E.6040400@ar.media.kyoto-u.ac.jp> Ben Finney wrote: > > I'm not sure exactly what you mean by ?do like autoconf?. Can you > please describe exactly what behaviour you are envisaging, so that we > don't all have a different interpretation of what ?like autoconf? > means ? Not autoconf, but the whole autotools suite (I think you need to use automake at least to do what I have in mind). The idea is deceptively simple: when you use ./configure, it gives you many options wrt paths: --bindir=DIR user executables [EPREFIX/bin] --sbindir=DIR system admin executables [EPREFIX/sbin] --libexecdir=DIR program executables [EPREFIX/libexec] --sysconfdir=DIR read-only single-machine data [PREFIX/etc] --sharedstatedir=DIR modifiable architecture-independent data [PREFIX/com] --localstatedir=DIR modifiable single-machine data [PREFIX/var] --libdir=DIR object code libraries [EPREFIX/lib] --includedir=DIR C header files [PREFIX/include] --oldincludedir=DIR C header files for non-gcc [/usr/include] --datarootdir=DIR read-only arch.-independent data root [PREFIX/share] --datadir=DIR read-only architecture-independent data [DATAROOTDIR] --infodir=DIR info documentation [DATAROOTDIR/info] --localedir=DIR locale-dependent data [DATAROOTDIR/locale] --mandir=DIR man documentation [DATAROOTDIR/man] --docdir=DIR documentation root [DATAROOTDIR/doc/libsndfile] --htmldir=DIR html documentation [DOCDIR] --dvidir=DIR dvi documentation [DOCDIR] --pdfdir=DIR pdf documentation [DOCDIR] --psdir=DIR ps documentation [DOCDIR] By default, they are laid out more or less like the FHS (and again, the detail do not matter: you can change them), BUT, it is extremely flexible. The only burden for the developer is to say which files go in which category, which he has to do anyway. You could think that because it is autoconf, it is not flexible for other OSes (windows), but with this, you can have extremely different ways of packaging, which are 100 % against the FHS spirit. I gave the gobolinux example, which does package every software in its own directory, a bit like a super-stow if you know stow. The thing is, I am sure the majority of softwares developers never think about this for their own software, still it is possible: it really come for free for the developer. And for the distributor, it is extremely easy (and self-documented). You could have different defaults on different platforms (windows comes to mind). cheers, David From david at ar.media.kyoto-u.ac.jp Fri Oct 3 07:01:10 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Fri, 03 Oct 2008 14:01:10 +0900 Subject: [Distutils] AMP, anyone? (Re: "just use debian") In-Reply-To: <48E57DAB.2020806@canterbury.ac.nz> References: <20080917171851.GA27261@logilab.fr> <94bdd2610809300505u1291ae6an3c44e2ce31d43e3f@mail.gmail.com> <1222780673.4166.55.camel@shizuru> <94bdd2610809300649q6bc9cdf3t84d5221d956bfc9d@mail.gmail.com> <1222784868.4166.88.camel@shizuru> <94bdd2610809300820g36a3f1bbpc2a04e03b43ee9c9@mail.gmail.com> <48E24765.1020908@simplistix.co.uk> <48E24BDA.2050104@simplistix.co.uk> <94bdd2610809300901o3fa62601q985784f01a8ad774@mail.gmail.com> <48E24FF7.9000300@simplistix.co.uk> <48E255B9.50806@colorstudy.com> <1222801361.32659.1.camel@shizuru> <48E2903F.9000307@enthought.com> <1222810334.32659.18.camel@shizuru> <48E2A0BB.6090300@enthought.com> <1222848707.7060.4.camel@tomoyo> <48E333C2.9030706@ar.media.kyoto-u.ac.jp> <1222851502.7060.18.camel@tomoyo> <48E33A91.5000602@ar.media.kyoto-u.ac.jp> <48E44334.4040405@gmail.com> <48E4A3AC.8030909@ar.media.kyoto-u.ac.jp> <48E57DAB.2020806@canterbury.ac.nz> Message-ID: <48E5A716.1000405@ar.media.kyoto-u.ac.jp> Greg Ewing wrote: > I think Python could do with something similar for managing > version issues. Perhaps we could study how ACEs work and see > if we can use any ideas from there. Interesting. From your description, it does sound a bit like the GAC for MONO/.Net I was mentioning earlier: http://www.mono-project.com/Assemblies_and_the_GAC I know Bertrand Meyer say mostly good things about .net technologies, and Eiffel softwares went a lot toward so .net technologies, so I would not be surprised if it were somewhat similar. Do you have a link toward this. I could not find much info with google on this Assembly Classes for Eiffel. cheers, David From joss at debian.org Fri Oct 3 14:42:58 2008 From: joss at debian.org (Josselin Mouette) Date: Fri, 03 Oct 2008 14:42:58 +0200 Subject: [Distutils] "Python Package Management Sucks" In-Reply-To: <48E58A13.2080601@canterbury.ac.nz> References: <48D00408.50705@simplistix.co.uk> <20080917171851.GA27261@logilab.fr> <48D8C701.6070707@simplistix.co.uk> <18650.28513.573244.313581@gargle.gargle.HOWL> <48E24C28.8020304@simplistix.co.uk> <48E3BACE.9020207@gmail.com> <20081001183754.84BCE3A4072@sparrow.telecommunity.com> <48E3F650.6080309@gmail.com> <20081001235808.85D453A4072@sparrow.telecommunity.com> <1222946445.6971.21.camel@shizuru> <20081002160417.GF13534@phare.normalesup.org> <48E58A13.2080601@canterbury.ac.nz> Message-ID: <1223037778.4257.23.camel@shizuru> Le vendredi 03 octobre 2008 ? 15:57 +1300, Greg Ewing a ?crit : > On Thu, Oct 02, 2008 at 01:20:45PM +0200, Josselin Mouette wrote: > > Sorry, but things don?t work this way. Anything that is *not* a .py file > > should not land in the python module directories. > > So where *should* they go, on platforms where there is > no defined place for such things? If there is no standard place, you can simply define one. > And how can the code > that uses them find them without platform-specific > knowledge? With a simple API to locate them. In fact, you don?t even need an API. If you distribute a python program using autoconf, you can simply say: mydata = file (@datadir@ "/blah/blahblah.dat",'r') call the file blah.py.in, and add it to AC_CONFIG_FILES. Autoconf will detect the correct directory depending on the platform. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From jim at zope.com Fri Oct 3 14:57:21 2008 From: jim at zope.com (Jim Fulton) Date: Fri, 3 Oct 2008 08:57:21 -0400 Subject: [Distutils] [zc.buildout] running in safe mode In-Reply-To: <48E54D11.4030807@colorstudy.com> References: <94bdd2610810010334g3e3994bcw5cfa270de3ae73e7@mail.gmail.com> <78EFD40E-0971-4705-BDD9-5064DF8DC880@zope.com> <48E547EE.6080007@colorstudy.com> <07F59784-77C8-4988-B64C-5573F7432302@zope.com> <48E54D11.4030807@colorstudy.com> Message-ID: <0F2F1416-ED7A-4CBF-B5B8-CF70A7EBA83E@zope.com> On Oct 2, 2008, at 6:37 PM, Ian Bicking wrote: > Jim Fulton wrote: >> On Oct 2, 2008, at 6:15 PM, Ian Bicking wrote: >>> Jim Fulton wrote: >>>>> I know it is a bad practice for a recipe to return some paths that >>>>> contains important data in the install() method, >>>>> because zc.buildout might remove them. >>>>> >>>>> Nevertheless, it happens from time to time that a developer lose >>>>> some >>>>> content because of a misconfiguration, >>>>> or a zealous recipe. That is his responsability, and backups are >>>>> done for that. >>>> I don't think backups are the right approach. It's a mistake to >>>> have recipes manage precious data. If you really really really >>>> think that's a good idea, then the recipe should at least manage >>>> uninstall and move precious data aside, rather than remove it. >>>> I don't think it is really the user's problem is a recipe >>>> misbehaves by allowing precious data to be removed. >>> >>> I'll note fassembler uses a file abstraction layer so that its >>> recipes are safe by default: https://svn.openplans.org/svn/fassembler/trunk/fassembler/filemaker.py >>> >>> I think buildout would be a lot more humane if it took the same >>> approach. >> I'd be interested to know what you mean by this, but I'm not >> willing to read that source to find out. >> Can you be a little more specific? > > Instead of using open(), etc, to write files, there's an instance of > Maker which holds some of the settings (--interactive, --simulate, a > base directory). Then you do all your file operations like: > > maker.ensure_file('path/to/file.txt', content) > > If that file exists with different content then the user gets asked > about what to do. It also logs all the writing, shows diffs, can > make backups, etc. You can force overwriting, but that's a keyword > argument that defaults to False, so only if you actually have good > reason to overwrite files (without asking) then that's fine, but you > will start developing the easy way, which is to be safe about this > stuff. In a system in which most data is managed automatically, asking the user before doing anything that might remove or overwrite data is, in my experience, counterproductive. It's like a security system that constantly asks for permission do do things, training users to hit an "OK" button very quickly. In a previous version of buildout, it worked the way you and Tarek suggest. It asked users before performing any action that caused a part to be uninstalled. This was extremely annoying. I finally just started piping the output of the yes command into it. Again, I can live with people adding an option that causes buildout to prompt before removing files or directories (or maybe just uninstalling parts that would cause it to remove files or directories). I know that I wouldn't use the option myself. Jim -- Jim Fulton Zope Corporation From ziade.tarek at gmail.com Fri Oct 3 15:40:59 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 3 Oct 2008 15:40:59 +0200 Subject: [Distutils] [zc.buildout] running in safe mode In-Reply-To: <0F2F1416-ED7A-4CBF-B5B8-CF70A7EBA83E@zope.com> References: <94bdd2610810010334g3e3994bcw5cfa270de3ae73e7@mail.gmail.com> <78EFD40E-0971-4705-BDD9-5064DF8DC880@zope.com> <48E547EE.6080007@colorstudy.com> <07F59784-77C8-4988-B64C-5573F7432302@zope.com> <48E54D11.4030807@colorstudy.com> <0F2F1416-ED7A-4CBF-B5B8-CF70A7EBA83E@zope.com> Message-ID: <94bdd2610810030640p25a47639y973351b374f60dc@mail.gmail.com> On Fri, Oct 3, 2008 at 2:57 PM, Jim Fulton wrote: > In a system in which most data is managed automatically, asking the user > before doing anything that might remove or overwrite data is, in my > experience, counterproductive. It's like a security system that constantly > asks for permission do do things, training users to hit an "OK" button very > quickly. > > In a previous version of buildout, it worked the way you and Tarek suggest. > It asked users before performing any action that caused a part to be > uninstalled. This was extremely annoying. I finally just started piping > the output of the yes command into it. > > Again, I can live with people adding an option that causes buildout to > prompt before removing files or directories (or maybe just uninstalling > parts that would cause it to remove files or directories). I know that I > wouldn't use the option myself. Ok. To reduce the noise, I think we should work with a list of folders or files we would like to 'protect', with a glob-style pattern, relative to the buildout folder, or absolute. For instance if I work with a zope application I would probably do something like that: [buildout] ... prompt-before-delete = var ... Or maybe: [buildout] ... prompt-before-delete = var/filestorage/*.fs ... Tarek -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From chris at simplistix.co.uk Fri Oct 3 16:49:23 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 03 Oct 2008 15:49:23 +0100 Subject: [Distutils] [zc.buildout] running in safe mode In-Reply-To: <94bdd2610810030640p25a47639y973351b374f60dc@mail.gmail.com> References: <94bdd2610810010334g3e3994bcw5cfa270de3ae73e7@mail.gmail.com> <78EFD40E-0971-4705-BDD9-5064DF8DC880@zope.com> <48E547EE.6080007@colorstudy.com> <07F59784-77C8-4988-B64C-5573F7432302@zope.com> <48E54D11.4030807@colorstudy.com> <0F2F1416-ED7A-4CBF-B5B8-CF70A7EBA83E@zope.com> <94bdd2610810030640p25a47639y973351b374f60dc@mail.gmail.com> Message-ID: <48E630F3.9090604@simplistix.co.uk> Tarek Ziad? wrote: > [buildout] > ... > prompt-before-delete = > var > ... > > Or maybe: > > [buildout] > ... > prompt-before-delete = > var/filestorage/*.fs I like this idea :-) Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Fri Oct 3 17:01:44 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 03 Oct 2008 16:01:44 +0100 Subject: [Distutils] [Catalog-sig] PEP for distutils In-Reply-To: <20080930175201.106863A409C@sparrow.telecommunity.com> References: <94bdd2610809280555p12c0e326r4a867bd3b67efbd9@mail.gmail.com> <48E23F2D.3000802@simplistix.co.uk> <20080930153822.GJ26878@phare.normalesup.org> <48E24897.8010800@colorstudy.com> <20080930162559.GA11804@amk-desktop.matrixgroup.net> <20080930175201.106863A409C@sparrow.telecommunity.com> Message-ID: <48E633D8.1090808@simplistix.co.uk> Phillip J. Eby wrote: > Nope. And it can't possibly do so, unless it contains dependency data > for every possible variation of the package. For example, a package > might dynamically declare dependency on ctypes, depending on whether > you're installing it for Python 2.4 or Python 2.5. (Dependencies can > also be platform-specific and build-option-specific, as well as > Python-version-specific.) Ug. No-one actually does this do they? man, setup.py both sucks and blows at the same time :-( Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Fri Oct 3 17:06:36 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 03 Oct 2008 16:06:36 +0100 Subject: [Distutils] "just use debian" In-Reply-To: <1222791462.4166.134.camel@shizuru> References: <20080917171851.GA27261@logilab.fr> <48DDEC5C.7090601@ar.media.kyoto-u.ac.jp> <20080928173011.GA4852@volans.logilab.fr> <48E0C007.70702@ar.media.kyoto-u.ac.jp> <94bdd2610809290509l2ab7359ep7f4edfcd23eae4ee@mail.gmail.com> <20080930084253.GF4494@volans.logilab.fr> <94bdd2610809300505u1291ae6an3c44e2ce31d43e3f@mail.gmail.com> <1222780673.4166.55.camel@shizuru> <94bdd2610809300649q6bc9cdf3t84d5221d956bfc9d@mail.gmail.com> <1222784868.4166.88.camel@shizuru> <94bdd2610809300820g36a3f1bbpc2a04e03b43ee9c9@mail.gmail.com> <48E24765.1020908@simplistix.co.uk> <1222790492.4166.127.camel@shizuru> <48E24EFF.3010707@simplistix.co.uk> <1222791462.4166.134.camel@shizuru> Message-ID: <48E634FC.1060101@simplistix.co.uk> Josselin Mouette wrote: > This information is not accessible directly at import time. Two questions as answers: - why does it need to be? - why not? > If you want > to rely on it to check the API compatibility, you?ll end up doing the > horrible things pygtk and gst-python did. And believe me, that will not > be helpful. It's unfortunately not a black'n'white thing... ...but if you require a particular API, just define the appropriate dependency in setup.py. What's the problem with that? > Indeed, but if we change the package name without changing the file > name, we have to make both packages conflict with each other. Why? > This works > for distribution packages, What do you mean by "distribution package"? > but it doesn?t help for third-party addons, Why not? > and it can make things complicated if two packages need different APIs. If two packages need two different versions of a third package, then your project is in trouble, api differences or not... > I showed already two examples: versioned provides and exact > dependencies. I don't know what either of these terms means I'm afraid :-S cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Fri Oct 3 17:14:22 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 03 Oct 2008 16:14:22 +0100 Subject: [Distutils] specifying dependencies In-Reply-To: <48E255B9.50806@colorstudy.com> References: <20080917171851.GA27261@logilab.fr> <20080930084253.GF4494@volans.logilab.fr> <94bdd2610809300505u1291ae6an3c44e2ce31d43e3f@mail.gmail.com> <1222780673.4166.55.camel@shizuru> <94bdd2610809300649q6bc9cdf3t84d5221d956bfc9d@mail.gmail.com> <1222784868.4166.88.camel@shizuru> <94bdd2610809300820g36a3f1bbpc2a04e03b43ee9c9@mail.gmail.com> <48E24765.1020908@simplistix.co.uk> <48E24BDA.2050104@simplistix.co.uk> <94bdd2610809300901o3fa62601q985784f01a8ad774@mail.gmail.com> <48E24FF7.9000300@simplistix.co.uk> <48E255B9.50806@colorstudy.com> Message-ID: <48E636CE.4000703@simplistix.co.uk> Ian Bicking wrote: > Say I have a package that represents an application. We'll call it > FooBlog. I release version 1.0. It uses the Turplango web framework > (1.5 at the time of release) and the Storchalmy ORM (0.4), and Turplango > uses HardJSON (1.2.1). > > I want my version 1.0 to keep working. So, I figure I'll add the > dependencies: > > Turplango==1.5 > Storchalmy==0.4 Why? I would have suggested: Turplango>=1.5,<2.0 Storchalmy==>=0.4,<0.5 > Then HardJSON 2.0 is released, and Turplango only required > HardJSON>=1.2, so new installations start installing HardJSON 2.0. But > my app happens not to be compatible with that library, and so it's > broken. > OK... so, I could add HardJSON==1.2.1 in my requirements. Not could, should, in fact must. Relying on a dependency provided by library you're using is suicide. Again, I'd suggest: HardJSON >=1.2.1,<1.3 > But then a small bug fix, HardJSON 1.2.2 comes out, that fixes a > security bug. Turplango releases version 1.5.1 that requires > HardJSON>=1.2.2. I now have have to update FooBlog to require both > Turplango==1.5.1 and HardJSON==1.2.2. Not if you'd followed my advice above. > Later on, I decide that Turplango 1.6 fixes some important bugs, and I > want to try it with my app. I can install Turplango 1.6, but I can't > start my app because I'll get a version conflict. Not if you'd followed my advice above. > So to even experiment > with a new version of the app, I have to check out FooBlog, update > setup.py, reinstall (setup.py develop) the package, and then I can start > using it. Right, you're developing FooBlog by changing the software it uses, so it seems natural enough to have to edit FooBlog code. You don't have to check those edits into your SCM ;-) > But if I've made other hard requirements of packages like > HardJSON, I'll have to update all those too. Yes, that's true, and why I recommeded what I did. That said, if you're paranoid enough to specify the exact versions (there's nothing wrong with this ;-) ) then it should be no surprise that you need to edit them... > more than 3 libraries involved. I now think it is best to only use > version requirements to express known conflicts. Or likely sources of known conflicts, such as major version increases, which is why I suggested what I did above... > For future versions of > packages you can't really know if they will cause conflicts until they > are released. Right, which is why consistency is version numbering for backwards incompatible changes is important. Myself, I stick pretty rigidly to: x.y.z: z = no api change y = new apis added x = old apis changed or removed cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Fri Oct 3 17:18:48 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 03 Oct 2008 16:18:48 +0100 Subject: [Distutils] [Catalog-sig] pre-PEP : Synthesis of previous threads, and irc talks + proposals In-Reply-To: <20081001172824.B1E0B3A4072@sparrow.telecommunity.com> References: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.com> <20081001172824.B1E0B3A4072@sparrow.telecommunity.com> Message-ID: <48E637D8.9070004@simplistix.co.uk> Phillip J. Eby wrote: > I think we need a standard for tools > interop (ala WSGI), not implementation tweaks for the existing tools. Agreed. >> 4/ let's change PyPI to make it work with the new metadata and to >> enforce a few things >> >> Enforcements: >> - a binary distribution cannot be uploaded if a source distrbution >> has not been previously provided for the version > > Note that this doesn't allow closed-source packages to be uploaded; thus > it would need to be a warning, rather than a requirement. This is an important point. We can't assume any one repository will have all needed packages. > It could only do this for specific binaries, since dependencies can be > dynamic. They should not be dynamic :-( Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From ianb at colorstudy.com Fri Oct 3 17:30:38 2008 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 03 Oct 2008 10:30:38 -0500 Subject: [Distutils] specifying dependencies In-Reply-To: <48E636CE.4000703@simplistix.co.uk> References: <20080917171851.GA27261@logilab.fr> <20080930084253.GF4494@volans.logilab.fr> <94bdd2610809300505u1291ae6an3c44e2ce31d43e3f@mail.gmail.com> <1222780673.4166.55.camel@shizuru> <94bdd2610809300649q6bc9cdf3t84d5221d956bfc9d@mail.gmail.com> <1222784868.4166.88.camel@shizuru> <94bdd2610809300820g36a3f1bbpc2a04e03b43ee9c9@mail.gmail.com> <48E24765.1020908@simplistix.co.uk> <48E24BDA.2050104@simplistix.co.uk> <94bdd2610809300901o3fa62601q985784f01a8ad774@mail.gmail.com> <48E24FF7.9000300@simplistix.co.uk> <48E255B9.50806@colorstudy.com> <48E636CE.4000703@simplistix.co.uk> Message-ID: <48E63A9E.1050404@colorstudy.com> Chris Withers wrote: > Ian Bicking wrote: >> Say I have a package that represents an application. We'll call it >> FooBlog. I release version 1.0. It uses the Turplango web framework >> (1.5 at the time of release) and the Storchalmy ORM (0.4), and >> Turplango uses HardJSON (1.2.1). >> >> I want my version 1.0 to keep working. So, I figure I'll add the >> dependencies: >> >> Turplango==1.5 >> Storchalmy==0.4 > > Why? > > I would have suggested: > > Turplango>=1.5,<2.0 > Storchalmy==>=0.4,<0.5 Then when Turplango 1.6 comes out it'll break my code. >> Then HardJSON 2.0 is released, and Turplango only required >> HardJSON>=1.2, so new installations start installing HardJSON 2.0. >> But my app happens not to be compatible with that library, and so it's >> broken. > > > OK... so, I could add HardJSON==1.2.1 in my requirements. > > Not could, should, in fact must. Relying on a dependency provided by > library you're using is suicide. > > Again, I'd suggest: > > HardJSON >=1.2.1,<1.3 What does 1.3 mean? You imply there is a disciplined use of a versioning pattern, and that every release is a guarantee that the versioning has been properly declared. There isn't a common understanding of versions, and it's common that conflicts are released unintentionally. >> But then a small bug fix, HardJSON 1.2.2 comes out, that fixes a >> security bug. Turplango releases version 1.5.1 that requires >> HardJSON>=1.2.2. I now have have to update FooBlog to require both >> Turplango==1.5.1 and HardJSON==1.2.2. > > Not if you'd followed my advice above. OK, change that to "a small bug fix comes out as HardJSON 1.3", and the same problems follow. I don't know what the nature of future releases will be. >> Later on, I decide that Turplango 1.6 fixes some important bugs, and I >> want to try it with my app. I can install Turplango 1.6, but I can't >> start my app because I'll get a version conflict. > > Not if you'd followed my advice above. Now you've introduced an entirely different requirement -- for some reason I am supposed to have known that HardJSON 1.3 would break my code, but only Turplango 2.0 would cause a conflict. And Turplango 1.6 wouldn't >> So to even experiment with a new version of the app, I have to check >> out FooBlog, update setup.py, reinstall (setup.py develop) the >> package, and then I can start using it. > > Right, you're developing FooBlog by changing the software it uses, so it > seems natural enough to have to edit FooBlog code. You don't have to > check those edits into your SCM ;-) > >> But if I've made other hard requirements of packages like HardJSON, >> I'll have to update all those too. > > Yes, that's true, and why I recommeded what I did. That said, if you're > paranoid enough to specify the exact versions (there's nothing wrong > with this ;-) ) then it should be no surprise that you need to edit them... It's not surprising, it's just very annoying. >> more than 3 libraries involved. I now think it is best to only use >> version requirements to express known conflicts. > > Or likely sources of known conflicts, such as major version increases, > which is why I suggested what I did above... You presume you can predict likely sources of known conflicts in software that doesn't exist yet. This is simply not true. >> For future versions of packages you can't really know if they will >> cause conflicts until they are released. > > Right, which is why consistency is version numbering for backwards > incompatible changes is important. There is no single concept of what backward compatibility even is. You can off something that fixes my specific example, using knowledge that would not be available to you at the time when you were using the code. That doesn't really prove anything -- I could also come up with conflicts that would break any example you could provide. There's no version change so minor that it can't break anything, and there's no version change so major that you should end up with a cascading set of updates that only change dependency information just to accommodate it. -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From joss at debian.org Fri Oct 3 17:32:28 2008 From: joss at debian.org (Josselin Mouette) Date: Fri, 03 Oct 2008 17:32:28 +0200 Subject: [Distutils] "just use debian" In-Reply-To: <48E634FC.1060101@simplistix.co.uk> References: <20080917171851.GA27261@logilab.fr> <48DDEC5C.7090601@ar.media.kyoto-u.ac.jp> <20080928173011.GA4852@volans.logilab.fr> <48E0C007.70702@ar.media.kyoto-u.ac.jp> <94bdd2610809290509l2ab7359ep7f4edfcd23eae4ee@mail.gmail.com> <20080930084253.GF4494@volans.logilab.fr> <94bdd2610809300505u1291ae6an3c44e2ce31d43e3f@mail.gmail.com> <1222780673.4166.55.camel@shizuru> <94bdd2610809300649q6bc9cdf3t84d5221d956bfc9d@mail.gmail.com> <1222784868.4166.88.camel@shizuru> <94bdd2610809300820g36a3f1bbpc2a04e03b43ee9c9@mail.gmail.com> <48E24765.1020908@simplistix.co.uk> <1222790492.4166.127.camel@shizuru> <48E24EFF.3010707@simplistix.co.uk> <1222791462.4166.134.camel@shizuru> <48E634FC.1060101@simplistix.co.uk> Message-ID: <1223047948.4257.65.camel@shizuru> Le vendredi 03 octobre 2008 ? 16:06 +0100, Chris Withers a ?crit : > Josselin Mouette wrote: > > This information is not accessible directly at import time. > > Two questions as answers: > > - why does it need to be? It does not strictly need to be, it?s just more convenient ; especially since you generally have to be compatible with several APIs and have compatibility code for each of them. Note that the reduced complexity of removing the old API is by far compensated by the added complexity of such compatibility code. > - why not? Currently there is simply no way to express the API version of a module. > > If you want > > to rely on it to check the API compatibility, you?ll end up doing the > > horrible things pygtk and gst-python did. And believe me, that will not > > be helpful. > > It's unfortunately not a black'n'white thing... > > ...but if you require a particular API, just define the appropriate > dependency in setup.py. What's the problem with that? There are at least two problems with that: * you cannot reliably translate this to package dependencies (even when doing it by hand as we do currently); * it is not possible to install two different APIs for the same module name on the same system. > > Indeed, but if we change the package name without changing the file > > name, we have to make both packages conflict with each other. > > Why? Because otherwise the package manager will choke when trying to install both of them together. > > This works > > for distribution packages, > > What do you mean by "distribution package"? I mean a .deb or a .rpm. > > but it doesn?t help for third-party addons, > > Why not? Because you can say in the package metadata: "Conflicts: python-foo", but you cannot express "this package will break local installations of module foo". The user will just break his python installation without knowing why. > > and it can make things complicated if two packages need different APIs. > > If two packages need two different versions of a third package, then > your project is in trouble, api differences or not... Your project is in trouble because of the total absence of API management. Otherwise there?s no reason why you couldn?t have some dependencies use one API and some other dependencies an other one. But the problem is worse than that: you know, there?s not only one application installed on a system. If one project needs one version and the other project another, incompatible version, you cannot install both on the same system. > > I showed already two examples: versioned provides and exact > > dependencies. > > I don't know what either of these terms means I'm afraid :-S Versioned provides mean "module bar at version 1.2 provides functionality of module foo at version 1.0". Exact dependencies mean "module bar at version 1.2 requires module foo at version 1.0 *and only this version*." Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From pje at telecommunity.com Fri Oct 3 17:59:14 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 03 Oct 2008 11:59:14 -0400 Subject: [Distutils] [Catalog-sig] pre-PEP : Synthesis of previous threads, and irc talks + proposals In-Reply-To: <48E637D8.9070004@simplistix.co.uk> References: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.com> <20081001172824.B1E0B3A4072@sparrow.telecommunity.com> <48E637D8.9070004@simplistix.co.uk> Message-ID: <20081003155805.15F023A40D9@sparrow.telecommunity.com> At 04:18 PM 10/3/2008 +0100, Chris Withers wrote: >Phillip J. Eby wrote: >>It could only do this for specific binaries, since dependencies can >>be dynamic. > >They should not be dynamic :-( Too bad. They are, because they have to be in order to support more than one platform and/or Python version from the same source base. From ianb at colorstudy.com Fri Oct 3 18:12:06 2008 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 03 Oct 2008 11:12:06 -0500 Subject: [Distutils] [zc.buildout] running in safe mode In-Reply-To: <0F2F1416-ED7A-4CBF-B5B8-CF70A7EBA83E@zope.com> References: <94bdd2610810010334g3e3994bcw5cfa270de3ae73e7@mail.gmail.com> <78EFD40E-0971-4705-BDD9-5064DF8DC880@zope.com> <48E547EE.6080007@colorstudy.com> <07F59784-77C8-4988-B64C-5573F7432302@zope.com> <48E54D11.4030807@colorstudy.com> <0F2F1416-ED7A-4CBF-B5B8-CF70A7EBA83E@zope.com> Message-ID: <48E64456.5030702@colorstudy.com> Jim Fulton wrote: >> Instead of using open(), etc, to write files, there's an instance of >> Maker which holds some of the settings (--interactive, --simulate, a >> base directory). Then you do all your file operations like: >> >> maker.ensure_file('path/to/file.txt', content) >> >> If that file exists with different content then the user gets asked >> about what to do. It also logs all the writing, shows diffs, can make >> backups, etc. You can force overwriting, but that's a keyword >> argument that defaults to False, so only if you actually have good >> reason to overwrite files (without asking) then that's fine, but you >> will start developing the easy way, which is to be safe about this stuff. > > In a system in which most data is managed automatically, asking the user > before doing anything that might remove or overwrite data is, in my > experience, counterproductive. It's like a security system that > constantly asks for permission do do things, training users to hit an > "OK" button very quickly. > > In a previous version of buildout, it worked the way you and Tarek > suggest. It asked users before performing any action that caused a part > to be uninstalled. This was extremely annoying. I finally just started > piping the output of the yes command into it. > > Again, I can live with people adding an option that causes buildout to > prompt before removing files or directories (or maybe just uninstalling > parts that would cause it to remove files or directories). I know that I > wouldn't use the option myself. You can also treat that nuisance like a bug and resolve the problem. I think fassembler has mostly done that, so that only really interesting problems are noted. We also switched to a system inspired by gentoo ports, where these queries are deferred until the end of the build. Yes, it can be annoying, but it doesn't have to be annoying. I'd rather start at a slightly annoying situation and try to decrease that problem, than start with a periodically infuriating situation. -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From pje at telecommunity.com Fri Oct 3 18:35:11 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 03 Oct 2008 12:35:11 -0400 Subject: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification Message-ID: <20081003163402.638A93A407A@sparrow.telecommunity.com> I'm thinking about putting together a pre-PEP for a "Build Utilities, Installation Locations, & Distribution Standards" (BUILDS) specification. But first, I want to throw out a few ideas to test the waters, and to give a general idea of what the first PEP would cover. The basic idea for the first PEP is to: 1. Give an overview of the current situation (problems w/distutils and setuptools, mainly, but also some of the successes) 2. Comment on some lessons learned from the WSGI definition process and WSGI in the field, and how they can be applied to the BUILDS process 3. Lay out high-level goals for the BUILDS project, such as: * distributing responsibility/authority for build tools development * adding extensibility to installation processes, * providing a 100% open playing field for build & install tools, * interoperability with the existing "egg" infrastructure, and * interoperability (but not 100% backward-compatibility) with distutils * allowing an incremental transition to the new standard 4. Lay out *non-goals* for the BUILDS project, such as trying to get developers to become system packagers or doing anything that requires them to change the runtime contents of their projects (as opposed to merely porting their setup.py, setup.cfg, etc.), defining and implementing the "perfect" system, etc. 5. Define rigorous terminology to be used for discussion of requirements and design, including such terms as "project", "release", "distribution", "system package", "installed distribution", etc. (This is incredibly important, because the discussions we're having now are already having Tower-of-Babel confusions.) 6. Sketch an overall design concept (build libraries in the stdlib establishing a Python API for invoking build tools, that in turn build an installation manifest to be used by installation tools), but without specifying actual APIs, manifest format, or a full enumeration of release metadata. 7. Present a vision/plan for how migration can occur from current tools. 8. Set the scope for the PEPs that should follow, on the installation manifest format, build architecture, build tool API, compiler/config infrastructure, etc. Whew. As you can see, just defining the problem adequately is a big job, and it may take a while to get even this first PEP right. So, I'd like some feedback, if anybody has some ideas about what else needs to be added to this. I also don't want to end up writing all of the PEPs, or else it may be a year or so before they all get done. ;-) I also think we're going to want a working group for this, or maybe multiple working groups, and it might be best not to use the general distutils-SIG for discussion past the first PEP, to allow people to filter threads better. Anyway. Thoughts, comments, volunteers, anyone? From nicolas.chauvat at logilab.fr Fri Oct 3 18:46:16 2008 From: nicolas.chauvat at logilab.fr (Nicolas Chauvat) Date: Fri, 3 Oct 2008 18:46:16 +0200 Subject: [Distutils] specifying dependencies In-Reply-To: <48E636CE.4000703@simplistix.co.uk> References: <94bdd2610809300649q6bc9cdf3t84d5221d956bfc9d@mail.gmail.com> <1222784868.4166.88.camel@shizuru> <94bdd2610809300820g36a3f1bbpc2a04e03b43ee9c9@mail.gmail.com> <48E24765.1020908@simplistix.co.uk> <48E24BDA.2050104@simplistix.co.uk> <94bdd2610809300901o3fa62601q985784f01a8ad774@mail.gmail.com> <48E24FF7.9000300@simplistix.co.uk> <48E255B9.50806@colorstudy.com> <48E636CE.4000703@simplistix.co.uk> Message-ID: <20081003164616.GA9504@volans.logilab.fr> On Fri, Oct 03, 2008 at 04:14:22PM +0100, Chris Withers wrote: > Myself, I stick pretty rigidly to: > > x.y.z: > > z = no api change > > y = new apis added > > x = old apis changed or removed That's our policy at www.logilab.org -- Nicolas Chauvat logilab.fr - services en informatique scientifique et gestion de connaissances From tarek.ziade at ingeniweb.com Fri Oct 3 19:00:10 2008 From: tarek.ziade at ingeniweb.com (Tarek Ziade) Date: Fri, 3 Oct 2008 19:00:10 +0200 Subject: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification In-Reply-To: <20081003163402.638A93A407A@sparrow.telecommunity.com> References: <20081003163402.638A93A407A@sparrow.telecommunity.com> Message-ID: Phillip, You said it was too early, 2 days ago when I sent a similar (less detailed) mail where I expressed the need to start writing things down in Python Wiki . But this work has already started, by Thoshio, me, and some others. mail. http://wiki.python.org/moin/Distribute/Functionality for instance. or here for the overview, http://wiki.python.org/moin/Distribute we are also hanging in #distutils Can't you join ? and help on it ? We where waiting for that in fact.. Also , I am -1 on having another Mailing List for that. Distutils is the place to be imho. Since main talks are about that. And the working group is there : people that are talking for now a few weeks, a bit before you joined the discussions. Tarek 2008/10/3 Phillip J. Eby > I'm thinking about putting together a pre-PEP for a "Build Utilities, > Installation Locations, & Distribution Standards" (BUILDS) specification. > But first, I want to throw out a few ideas to test the waters, and to give > a general idea of what the first PEP would cover. > > The basic idea for the first PEP is to: > > 1. Give an overview of the current situation (problems w/distutils and > setuptools, mainly, but also some of the successes) > > 2. Comment on some lessons learned from the WSGI definition process and > WSGI in the field, and how they can be applied to the BUILDS process > > 3. Lay out high-level goals for the BUILDS project, such as: > > * distributing responsibility/authority for build tools development > * adding extensibility to installation processes, > * providing a 100% open playing field for build & install tools, > * interoperability with the existing "egg" infrastructure, and > * interoperability (but not 100% backward-compatibility) with distutils > * allowing an incremental transition to the new standard > > 4. Lay out *non-goals* for the BUILDS project, such as trying to get > developers to become system packagers or doing anything that requires them > to change the runtime contents of their projects (as opposed to merely > porting their setup.py, setup.cfg, etc.), defining and implementing the > "perfect" system, etc. > > 5. Define rigorous terminology to be used for discussion of requirements > and design, including such terms as "project", "release", "distribution", > "system package", "installed distribution", etc. (This is incredibly > important, because the discussions we're having now are already having > Tower-of-Babel confusions.) > > 6. Sketch an overall design concept (build libraries in the stdlib > establishing a Python API for invoking build tools, that in turn build an > installation manifest to be used by installation tools), but without > specifying actual APIs, manifest format, or a full enumeration of release > metadata. > > 7. Present a vision/plan for how migration can occur from current tools. > > 8. Set the scope for the PEPs that should follow, on the installation > manifest format, build architecture, build tool API, compiler/config > infrastructure, etc. > > Whew. As you can see, just defining the problem adequately is a big job, > and it may take a while to get even this first PEP right. So, I'd like some > feedback, if anybody has some ideas about what else needs to be added to > this. > > I also don't want to end up writing all of the PEPs, or else it may be a > year or so before they all get done. ;-) I also think we're going to want > a working group for this, or maybe multiple working groups, and it might be > best not to use the general distutils-SIG for discussion past the first PEP, > to allow people to filter threads better. > > Anyway. Thoughts, comments, volunteers, anyone? > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -- Tarek Ziad? - Directeur Technique INGENIWEB (TM) - SAS 50000 Euros - RC B 438 725 632 Bureaux de la Colline - 1 rue Royale - B?timent D - 9?me ?tage 92210 Saint Cloud - France Phone : 01.78.15.24.00 / Fax : 01 46 02 44 04 http://www.ingeniweb.com - une soci?t? du groupe Alter Way -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Fri Oct 3 20:03:36 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 03 Oct 2008 14:03:36 -0400 Subject: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification In-Reply-To: References: <20081003163402.638A93A407A@sparrow.telecommunity.com> Message-ID: <20081003180228.B13373A4045@sparrow.telecommunity.com> At 07:00 PM 10/3/2008 +0200, Tarek Ziade wrote: >Phillip, > >You said it was too early, 2 days ago when I sent a similar (less >detailed) mail where I expressed the need to start writing things >down in Python Wiki . I understood you to be specifying implementation designs, not creating a charter for requirements gathering. And it is definitely still too early for that. >But this work has already started, by Thoshio, me, and some others. mail. > >http://wiki.python.org/moin/Distribute/Functionality >for instance. >or here for the overview, >http://wiki.python.org/moin/Distribute > >we are also hanging in #distutils > >Can't you join ? and help on it ? What is it you think I'm doing with the email you replied to? ;-) From greg.ewing at canterbury.ac.nz Sat Oct 4 00:50:16 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 04 Oct 2008 10:50:16 +1200 Subject: [Distutils] AMP, anyone? (Re: "just use debian") In-Reply-To: <48E5A716.1000405@ar.media.kyoto-u.ac.jp> References: <20080917171851.GA27261@logilab.fr> <1222780673.4166.55.camel@shizuru> <94bdd2610809300649q6bc9cdf3t84d5221d956bfc9d@mail.gmail.com> <1222784868.4166.88.camel@shizuru> <94bdd2610809300820g36a3f1bbpc2a04e03b43ee9c9@mail.gmail.com> <48E24765.1020908@simplistix.co.uk> <48E24BDA.2050104@simplistix.co.uk> <94bdd2610809300901o3fa62601q985784f01a8ad774@mail.gmail.com> <48E24FF7.9000300@simplistix.co.uk> <48E255B9.50806@colorstudy.com> <1222801361.32659.1.camel@shizuru> <48E2903F.9000307@enthought.com> <1222810334.32659.18.camel@shizuru> <48E2A0BB.6090300@enthought.com> <1222848707.7060.4.camel@tomoyo> <48E333C2.9030706@ar.media.kyoto-u.ac.jp> <1222851502.7060.18.camel@tomoyo> <48E33A91.5000602@ar.media.kyoto-u.ac.jp> <48E44334.4040405@gmail.com> <48E4A3AC.8030909@ar.media.kyoto-u.ac.jp> <48E57DAB.2020806@canterbury.ac.nz> <48E5A716.1000405@ar.media.kyoto-u.ac.jp> Message-ID: <48E6A1A8.1070904@canterbury.ac.nz> David Cournapeau wrote: > Do you have a link toward > this. I could not find much info with google on this Assembly Classes > for Eiffel. It seems to be difficult to find any in-depth information about it on the web. There's a summary of the syntax here: http://archive.eiffel.com/nice/language/page.html#HDR199 although it doesn't give much idea of what it all means. There's an example of one here: http://archive.eiffel.com/doc/online/eiffel50/intro/language/invitation-15.html There's some better info here on the GNU SmartEiffel site: http://smarteiffel.loria.fr/wiki/en/index.php/ACE -- Greg From eric at trueblade.com Sat Oct 4 01:38:03 2008 From: eric at trueblade.com (Eric Smith) Date: Fri, 03 Oct 2008 19:38:03 -0400 Subject: [Distutils] What's the difference between --index-url and --find-links? Message-ID: <48E6ACDB.6030000@trueblade.com> I understand that --index-url is single valued, and --find-links is multi-valued. But what's the semantic difference between them? From my reading, it sounds like they both point to pages that contain links to eggs (or other distributions). What I'm really trying to do is set up my own server, with my own collection of eggs. I want to configure easy_install (via a config file or command line) to only look at my collection of eggs, and never the Cheeseshop or anywhere else. Do I do this with just --index-url? Or with --find-links, and if so, how do I disable the index? Thanks. I searched the archives, but I couldn't find anything (these are very common search terms!). Eric. From ziade.tarek at gmail.com Sat Oct 4 02:10:36 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 4 Oct 2008 02:10:36 +0200 Subject: [Distutils] What's the difference between --index-url and --find-links? In-Reply-To: <48E6ACDB.6030000@trueblade.com> References: <48E6ACDB.6030000@trueblade.com> Message-ID: <94bdd2610810031710p19324b6anb3728e998833c8e8@mail.gmail.com> On Sat, Oct 4, 2008 at 1:38 AM, Eric Smith wrote: > I understand that --index-url is single valued, and --find-links is > multi-valued. But what's the semantic difference between them? From my > reading, it sounds like they both point to pages that contain links to eggs > (or other distributions). The index url contains an index page of packages, like http://pypi.python.org/simple/ as defined here http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api When setuptools or easy_install are used, they load the index and look for packages using the INDEX/PACKAGE_NAME/ base for the url. find-links will let you add extra links for setuptools to look at, so basically you can add any place that contains a flat list of distributions. Notice that there's a patch that has been proposed so setuptools can handle several indexes. > > What I'm really trying to do is set up my own server, with my own collection > of eggs. I want to configure easy_install (via a config file or command > line) to only look at my collection of eggs, and never the Cheeseshop or > anywhere else. Do I do this with just --index-url? Or with --find-links, and > if so, how do I disable the index? > You could build your own index, using static html files, or use one of the pypi compatible server applications out there, like EggBasket or PloneSoftwareCenter. to make sure you don't use anything else than your server (even if some packages are trying to access other places with dependency links) you can use the --allow-hosts option to give a white list of urls patterns. > Thanks. I searched the archives, but I couldn't find anything (these are > very common search terms!). > > Eric. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From eric at trueblade.com Sat Oct 4 03:05:52 2008 From: eric at trueblade.com (Eric Smith) Date: Fri, 03 Oct 2008 21:05:52 -0400 Subject: [Distutils] What's the difference between --index-url and --find-links? In-Reply-To: <94bdd2610810031710p19324b6anb3728e998833c8e8@mail.gmail.com> References: <48E6ACDB.6030000@trueblade.com> <94bdd2610810031710p19324b6anb3728e998833c8e8@mail.gmail.com> Message-ID: <48E6C170.8030904@trueblade.com> Tarek Ziad? wrote: > On Sat, Oct 4, 2008 at 1:38 AM, Eric Smith wrote: >> I understand that --index-url is single valued, and --find-links is >> multi-valued. But what's the semantic difference between them? From my >> reading, it sounds like they both point to pages that contain links to eggs >> (or other distributions). > > The index url contains an index page of packages, like > http://pypi.python.org/simple/ > as defined here > http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api > > When setuptools or easy_install are used, they load the index and look > for packages > using the INDEX/PACKAGE_NAME/ base for the url. > > find-links will let you add extra links for setuptools to look at, so > basically you > can add any place that contains a flat list of distributions. > > Notice that there's a patch that has been proposed so setuptools can handle > several indexes. > >> What I'm really trying to do is set up my own server, with my own collection >> of eggs. I want to configure easy_install (via a config file or command >> line) to only look at my collection of eggs, and never the Cheeseshop or >> anywhere else. Do I do this with just --index-url? Or with --find-links, and >> if so, how do I disable the index? >> > > You could build your own index, using static html files, or use one > of the pypi compatible server applications out there, like EggBasket > or PloneSoftwareCenter. > > to make sure you don't use anything else than your server (even if some packages > are trying to access other places with dependency links) you can use > the --allow-hosts > option to give a white list of urls patterns. Awesome. Thanks for the help. Eric. From bretthoerner at gmail.com Sat Oct 4 18:59:07 2008 From: bretthoerner at gmail.com (Brett Hoerner) Date: Sat, 4 Oct 2008 11:59:07 -0500 Subject: [Distutils] virtualenv + python 2.6 (setuptools 2.6 egg) Message-ID: Ian Bicking wrote: > Yes, the missing egg is causing the problem. ez_setup.py doesn't seem > to use a tarball fallback (I'm not sure why). Phillip - can there > either be a 2.6 egg uploaded, or have ez_setup.py install the tarball > when the egg is missing? Any news on this front? 2.6 is out but virtualenv is still broken because it can't fallback to the tarball. Thanks! Brett From pje at telecommunity.com Sat Oct 4 20:34:55 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 04 Oct 2008 14:34:55 -0400 Subject: [Distutils] virtualenv + python 2.6 (setuptools 2.6 egg) In-Reply-To: References: Message-ID: <20081004183346.634EC3A4045@sparrow.telecommunity.com> At 11:59 AM 10/4/2008 -0500, Brett Hoerner wrote: >Ian Bicking wrote: > > Yes, the missing egg is causing the problem. ez_setup.py doesn't seem > > to use a tarball fallback (I'm not sure why). Phillip - can there > > either be a 2.6 egg uploaded, or have ez_setup.py install the tarball > > when the egg is missing? > >Any news on this front? 2.6 is out but virtualenv is still broken >because it can't fallback to the tarball. Ian uploaded the egg a couple days ago. >Thanks! >Brett >_______________________________________________ >Distutils-SIG maillist - Distutils-SIG at python.org >http://mail.python.org/mailman/listinfo/distutils-sig From bretthoerner at gmail.com Sat Oct 4 20:39:23 2008 From: bretthoerner at gmail.com (Brett Hoerner) Date: Sat, 4 Oct 2008 13:39:23 -0500 Subject: [Distutils] virtualenv + python 2.6 (setuptools 2.6 egg) In-Reply-To: <20081004183346.634EC3A4045@sparrow.telecommunity.com> References: <20081004183346.634EC3A4045@sparrow.telecommunity.com> Message-ID: On Sat, Oct 4, 2008 at 1:34 PM, Phillip J. Eby wrote: > Ian uploaded the egg a couple days ago. Sorry, I've been reading up recently but I'm still a bit unfamiliar with PyPI, etc. Does it take a while for the new egg to show up on PyPI? I'm running virtualenv 1.3 (latest on PyPI) on Python 2.6 - and it's failing as of Oct 4. I don't see any 2.6 eggs on http://pypi.python.org/pypi/setuptools I don't mean to poke anyone - I can wait if that's all it takes now - I just want to make sure it isn't something on my end that I need to fix? Thanks, Brett From bretthoerner at gmail.com Sat Oct 4 20:41:01 2008 From: bretthoerner at gmail.com (Brett Hoerner) Date: Sat, 4 Oct 2008 13:41:01 -0500 Subject: [Distutils] virtualenv + python 2.6 (setuptools 2.6 egg) In-Reply-To: References: <20081004183346.634EC3A4045@sparrow.telecommunity.com> Message-ID: On Sat, Oct 4, 2008 at 1:39 PM, Brett Hoerner wrote: > I don't mean to poke anyone - I can wait if that's all it takes now - > I just want to make sure it isn't something on my end that I need to > fix? And right after I send this I realized that there is a 2.6 egg on PyPI. virtualenv 1.3 just wants to find setuptools-0.6c8 and not c9. I guess this needs a new virtualenv point release? Anyways, thanks. I'll take it to the virtualenv list. Brett From ianb at colorstudy.com Sat Oct 4 20:58:14 2008 From: ianb at colorstudy.com (Ian Bicking) Date: Sat, 04 Oct 2008 13:58:14 -0500 Subject: [Distutils] virtualenv + python 2.6 (setuptools 2.6 egg) In-Reply-To: References: <20081004183346.634EC3A4045@sparrow.telecommunity.com> Message-ID: <48E7BCC6.7010506@colorstudy.com> Brett Hoerner wrote: > On Sat, Oct 4, 2008 at 1:39 PM, Brett Hoerner wrote: >> I don't mean to poke anyone - I can wait if that's all it takes now - >> I just want to make sure it isn't something on my end that I need to >> fix? > > And right after I send this I realized that there is a 2.6 egg on > PyPI. virtualenv 1.3 just wants to find setuptools-0.6c8 and not c9. > I guess this needs a new virtualenv point release? > > Anyways, thanks. I'll take it to the virtualenv list. For now use virtualenv trunk, which has several 2.6 fixes: http://svn.colorstudy.com/virtualenv/trunk/virtualenv.py -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From zooko at zooko.com Sun Oct 5 18:04:20 2008 From: zooko at zooko.com (zooko) Date: Sun, 5 Oct 2008 10:04:20 -0600 Subject: [Distutils] pre-PEP : Synthesis of previous threads, and irc talks + proposals In-Reply-To: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.com> References: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.com> Message-ID: <8E57AB2B-4C1C-49F4-9FF8-78F57509395E@zooko.com> Thanks for the synthesis, Tarek. I have some experience using current Python packaging in the field -- the Tahoe project [1] -- and so I would like to throw in what I know of what is currently working and what is currently needed and what isn't a big deal to me. This doesn't, of course, mean that other people might value things that I don't, but at least the following opinions of mine are won from hard experience. On Oct 1, 2008, at 6:10 AM, Tarek Ziad? wrote: > 1/ the dependencies of a package are not expressed in the Require > metadata of the package most of the time. +2 -- This is the biggest problem. The dependencies are not expressed *anywhere* in the metadata of the package most of the time. We need a de jure and de facto way to express dependencies so that developers will actually write them down. > Furthermore, developer tend to use setuptools "install_requires" > and "tests_require" arguments to express dependencies. > > So basically, you have to run an egg_info to get them, because the > info files are generated by commands. +0 -- I can see how this could be done better, but it isn't a pressing problem us. The current mechanism to get that dependency information at build/develop/install time works okay. > 2/ the existence of PyPI had a side-effect: people tend to push the > entire doc of the package in one single field (long_description) > to display them at PyPI. The documentation of the package is not > cleary pointed to others. +0 -- I would like more structured docs because then I could patch stdeb [2] to put docs into /usr/share/docs/$PACKAGE on Debian. But it isn't a pressing problem for us (we currently kludge around that issue). > 3/ the metadata infos cannot be expressed in a static file by the > developer, because sometimes they are calculated by code. > while this very permissive, that is how it works but they are > tighted to argument passing to setup(). +0 -- I totally agree that a static, separate, declarative file containing just data and no code would be a nicer way to do this. But the current way is working for us. > 4/ PyPI lacks of direct information about dependencies. +? -- I don't know. It sounds like it would be a big improvement, but the current mechanism of discovering dependencies by downloading distributions and executing their setup.py's seems to be working. > 5/ ideally, they should be one and only one version of a given package > in an OS-based installation -1 -- This is the strong preference of the folks who package software for OSes -- Debian, Fedora, etc. -- but it is not necessarily the choice of the users who use their OSes. It is best for the Python packaging standards to be agnostic towards this, or at least to support both this desideratum and its opposite. > 6/ packagers would like to have more compatibility information to work > out on security upgrades or version conflicts > 7/ developers should be able to have more options when they define > version dependencies in their packages, things like: > A depends on B>=1.2 and B<=2.0 but with a preference to B 1.4 > or "avoid B 1.7" > > they give tips to packagers ! +0 -- If we try to do better than Debian and Fedora already do then this risks being a science project -- i.e. something that will take a few years and might or might not pan out. If we try to just ape them and learn from their decade's worth of mistakes then this is probably doable. > The developer dependencies infos is a tip and a help for a packager, > not an enforcement. see [7] +1 -- In around 95% of the cases that I've seen, the developer's dependencies info was good enough. But, people have to be able to do something about the other 5%, so they have to be able to override developer-provided dependency information with their own. Obviously they can do this by patching or runtime-patching or maintaining their own branch, but we should specify a standard, principled way to do it instead. > 11/ people should always upload the sdist version at PyPI, they > don't do it always. otherwise it is a pain for packagers. +1 -- sdist format should be encouraged. > 1/ let's change the Python Metadata , in order to introduce a better > dependency system, by > > - officialy introduce "install requires" and "test requires" > metadata in there > - mark "requires" as deprecated +1 > 2/ Let's move part of setuptools code in distutils, to respect > those changes. +1 > 3/ let's create a simple convention : the metadata should be expressed > in a python module called 'pkginfo.py' > where each metadata is a variable. > > that can be used by setup.py and therefore by any tool that work > with it, even if it does not run > a setup.py command. > > This is simpler, this is cleaner. you don't have to run some setup > magic to read them. > at least some magic introduces by commands Uh... I thought the idea was to *not* have arbitrary Python code executed in this part? How about a flat file that people can reliably parse with, say, "grep", to learn about metadata. > - a binary distribution cannot be uploaded if a source distrbution > has not been previously provided for the version > - the requires-python need to be present. : come on, you know what > python versions your package work with ! +1 > - we should be able to download the metadata of a package without > downloading the package > - PyPI should display the install and test dependencies in the UI > - The XML-RPC should provide this new metadata as well. > - a commenting system should allow developers and packagers to > give more infos on a package at PyPI > to make the work easier +1 Regards, Zooko [1] http://allmydata.org/trac/tahoe [2] http://stdeb.python-hosting.com/ --- http://allmydata.org -- Tahoe, the Least-Authority Filesystem http://allmydata.com -- back up all your files for $5/month From zooko at zooko.com Sun Oct 5 18:13:37 2008 From: zooko at zooko.com (zooko) Date: Sun, 5 Oct 2008 10:13:37 -0600 Subject: [Distutils] pre-PEP : Synthesis of previous threads, and irc talks + proposals In-Reply-To: <94bdd2610810011810h3ea48e6cmb8d223cc24e1cf85@mail.gmail.com> References: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.com> <20081001172824.B1E0B3A4072@sparrow.telecommunity.com> <94bdd2610810011055j69411c6fu6f417cf3cf930460@mail.gmail.com> <20081001182658.E96173A4072@sparrow.telecommunity.com> <94bdd2610810011810h3ea48e6cmb8d223cc24e1cf85@mail.gmail.com> Message-ID: On Oct 1, 2008, at 19:10 PM, Tarek Ziad? wrote: > I hate the idea of dynamic metadata in fact. I can't express precisely > why at that point. Me too and me too. Perhaps it would help to distinguish between requiring a certain functionality and requiring a specific codebase which implements that functionality. For example: distribution A requires the functionality of ctypes. That part is statically, declaratively always true. However, distribution A doesn't necessarily require a *distribution* named "ctypes". If you are running on Python 2.6, then that functionality is already present. If there is a new distribution out there named "new_ctypes" which provides the same functionality and the same interface but is a completely different code base, then the presence of "new_ctypes" satisfies distribution A's requirements. The former question is simple, static, and declarative. The latter question isn't. In most cases there is only one implementation of a given interface, so we make do by equating the interface with the implementation. I wonder how Debian and Fedora handle this sort of issue? Regards, Zooko --- http://allmydata.org -- Tahoe, the Least-Authority Filesystem http://allmydata.com -- back up all your files for $5/month From setuptools at bugs.python.org Sun Oct 5 21:06:57 2008 From: setuptools at bugs.python.org (Bryan Deter) Date: Sun, 05 Oct 2008 19:06:57 +0000 Subject: [Distutils] [issue50] Add ability use proxies In-Reply-To: <1223233617.96.0.277604744209.issue50@psf.upfronthosting.co.za> Message-ID: <1223233617.96.0.277604744209.issue50@psf.upfronthosting.co.za> New submission from Bryan Deter : It would be very helpful to have the ability to have setuptools route through a proxy. This would make using it in controlled environments significantly easier. ---------- messages: 197 nosy: deterb priority: wish status: unread title: Add ability use proxies _______________________________________________ Setuptools tracker _______________________________________________ From lists at zopyx.com Sun Oct 5 21:12:10 2008 From: lists at zopyx.com (Andreas Jung) Date: Sun, 05 Oct 2008 21:12:10 +0200 Subject: [Distutils] [issue50] Add ability use proxies In-Reply-To: <1223233617.96.0.277604744209.issue50@psf.upfronthosting.co.za> References: <1223233617.96.0.277604744209.issue50@psf.upfronthosting.co.za> Message-ID: --On 5. Oktober 2008 19:06:57 +0000 Bryan Deter wrote: > > New submission from Bryan Deter : > > It would be very helpful to have the ability to have setuptools route > through a proxy. This would make using it in controlled environments > significantly easier. I would expect that setuptools respects the http_proxy (check the spelling with the Python library) environment variable. -aj -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available URL: From a.badger at gmail.com Mon Oct 6 07:27:18 2008 From: a.badger at gmail.com (Toshio Kuratomi) Date: Sun, 05 Oct 2008 22:27:18 -0700 Subject: [Distutils] pre-PEP : Synthesis of previous threads, and irc talks + proposals In-Reply-To: References: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.com> <20081001172824.B1E0B3A4072@sparrow.telecommunity.com> <94bdd2610810011055j69411c6fu6f417cf3cf930460@mail.gmail.com> <20081001182658.E96173A4072@sparrow.telecommunity.com> <94bdd2610810011810h3ea48e6cmb8d223cc24e1cf85@mail.gmail.com> Message-ID: <48E9A1B6.2050607@gmail.com> zooko wrote: > On Oct 1, 2008, at 19:10 PM, Tarek Ziad? wrote: > >> I hate the idea of dynamic metadata in fact. I can't express precisely >> why at that point. > > Me too and me too. > > Perhaps it would help to distinguish between requiring a certain > functionality and requiring a specific codebase which implements that > functionality. > > For example: distribution A requires the functionality of ctypes. That > part is statically, declaratively always true. > > However, distribution A doesn't necessarily require a *distribution* > named "ctypes". If you are running on Python 2.6, then that > functionality is already present. If there is a new distribution out > there named "new_ctypes" which provides the same functionality and the > same interface but is a completely different code base, then the > presence of "new_ctypes" satisfies distribution A's requirements. > > The former question is simple, static, and declarative. The latter > question isn't. > > In most cases there is only one implementation of a given interface, so > we make do by equating the interface with the implementation. > > I wonder how Debian and Fedora handle this sort of issue? > With python modules we just require one thing providing the interface. Let's say that elementtree was merged into python-2.5. And let's say that we got python-2.5 as the default python in Fedora 7. Since we only have one version of python in any release of Fedora we do something like this: Require: python %if 0%{?fedora} < 7 Require: python-elementtree %endif We are thinking of enhancing what dependency information we Require and Provide (the problem being... we want to do this automatically.) If we get that working, we could do things like: Require: python(elementtree) and in Fedora 6, python-elementtree would have: Provide: python(elementtree) whereas Fedora 7+, the python package would have: Provide: python(elementtree) Note that this information is not as easy to get to as the metadata provided by eggs so we're still trying to come up with a script that will generate this data automatically. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: OpenPGP digital signature URL: From anand.prabhakar.patil at gmail.com Mon Oct 6 13:41:52 2008 From: anand.prabhakar.patil at gmail.com (Anand Patil) Date: Mon, 6 Oct 2008 12:41:52 +0100 Subject: [Distutils] setup.py develop not working, setuptools 0.6c9 Message-ID: <2bc7a5a50810060441o1753a1d3he6c8019cf84bf7f1@mail.gmail.com> Hello, I get the following error with setuptools 0.6c9, updated today with easy_install: sihpc03:pymc anand$ python setup.py develop usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: setup.py --help [cmd1 cmd2 ...] or: setup.py --help-commands or: setup.py cmd --help error: invalid command 'develop' sihpc03:pymc anand$ I'm using: Python 2.5.2 Stackless 3.1b3 060516 (python-2.52:61022, Feb 27 2008, 16:52:03) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin Please copy me directly in any reply. Thank you, Anand Patil -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Mon Oct 6 16:15:07 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 06 Oct 2008 10:15:07 -0400 Subject: [Distutils] setup.py develop not working, setuptools 0.6c9 In-Reply-To: <2bc7a5a50810060441o1753a1d3he6c8019cf84bf7f1@mail.gmail.co m> References: <2bc7a5a50810060441o1753a1d3he6c8019cf84bf7f1@mail.gmail.com> Message-ID: <20081006141356.A8D873A4114@sparrow.telecommunity.com> At 12:41 PM 10/6/2008 +0100, Anand Patil wrote: >Hello, > >I get the following error with setuptools 0.6c9, updated today with >easy_install: > >sihpc03:pymc anand$ python setup.py develop >usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] > or: setup.py --help [cmd1 cmd2 ...] > or: setup.py --help-commands > or: setup.py cmd --help > >error: invalid command 'develop' >sihpc03:pymc anand$ > >I'm using: > >Python 2.5.2 Stackless 3.1b3 060516 (python-2.52:61022, Feb 27 2008, >16:52:03) >[GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin > >Please copy me directly in any reply. Does your setup.py import setuptools? From anand.prabhakar.patil at gmail.com Mon Oct 6 16:24:50 2008 From: anand.prabhakar.patil at gmail.com (Anand Patil) Date: Mon, 6 Oct 2008 15:24:50 +0100 Subject: [Distutils] setup.py develop not working, setuptools 0.6c9 In-Reply-To: <20081006141356.A8D873A4114@sparrow.telecommunity.com> References: <2bc7a5a50810060441o1753a1d3he6c8019cf84bf7f1@mail.gmail.com> <20081006141356.A8D873A4114@sparrow.telecommunity.com> Message-ID: <2bc7a5a50810060724l9bc84b4nbbeec31cfa154e46@mail.gmail.com> Aw jeez, that was it. Thanks for responding. Anand On Mon, Oct 6, 2008 at 3:15 PM, Phillip J. Eby wrote: > At 12:41 PM 10/6/2008 +0100, Anand Patil wrote: > >> Hello, >> >> I get the following error with setuptools 0.6c9, updated today with >> easy_install: >> >> sihpc03:pymc anand$ python setup.py develop >> usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] >> or: setup.py --help [cmd1 cmd2 ...] >> or: setup.py --help-commands >> or: setup.py cmd --help >> >> error: invalid command 'develop' >> sihpc03:pymc anand$ >> >> I'm using: >> >> Python 2.5.2 Stackless 3.1b3 060516 (python-2.52:61022, Feb 27 2008, >> 16:52:03) >> [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin >> >> Please copy me directly in any reply. >> > > Does your setup.py import setuptools? > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From floris.bruynooghe at gmail.com Mon Oct 6 19:17:14 2008 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Mon, 6 Oct 2008 18:17:14 +0100 Subject: [Distutils] pre-PEP : Synthesis of previous threads, and irc talks + proposals In-Reply-To: References: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.com> <20081001172824.B1E0B3A4072@sparrow.telecommunity.com> <94bdd2610810011055j69411c6fu6f417cf3cf930460@mail.gmail.com> <20081001182658.E96173A4072@sparrow.telecommunity.com> <94bdd2610810011810h3ea48e6cmb8d223cc24e1cf85@mail.gmail.com> Message-ID: <20081006171714.GA4698@laurie.devork> On Sun, Oct 05, 2008 at 10:13:37AM -0600, zooko wrote: > On Oct 1, 2008, at 19:10 PM, Tarek Ziad? wrote: > For example: distribution A requires the functionality of ctypes. That > part is statically, declaratively always true. > > However, distribution A doesn't necessarily require a *distribution* > named "ctypes". If you are running on Python 2.6, then that > functionality is already present. If there is a new distribution out > there named "new_ctypes" which provides the same functionality and the > same interface but is a completely different code base, then the > presence of "new_ctypes" satisfies distribution A's requirements. > > The former question is simple, static, and declarative. The latter > question isn't. > > In most cases there is only one implementation of a given interface, so > we make do by equating the interface with the implementation. > > I wonder how Debian and Fedora handle this sort of issue? I think Josselin mentioned this already for Debian's case, but basically when python2.4 was default packages did say they wanted the python-elementtree package. So when python2.5 was packaged it said "Provides: python-elementree" and all packages depending on python-elementree will now be happy with just python2.5 installed. It's slightly more complicated then that in reality, the full Provides of python2.5 is: Provides: python2.5-celementtree, python2.5-cjkcodecs, python2.5-ctypes, python2.5-elementtree, python2.5-plistlib, python2.5-wsgiref The Debian infrastructure has various ways of helping packages cope with the versioned package names in the above (and it was all still very much a mess in python2.4 times). Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From chris.mahan at gmail.com Mon Oct 6 21:05:27 2008 From: chris.mahan at gmail.com (Chris Mahan) Date: Mon, 6 Oct 2008 12:05:27 -0700 Subject: [Distutils] Availability of setuptools installer for python2.6? Message-ID: <4d9b53db0810061205w75a14a4ewf388d8bd5dc0f62a@mail.gmail.com> Phillip, At http://pypi.python.org/pypi/setuptools there is no MS windows installer for python 2.6. Do you have an idea of when that might be available? Thanks. -- Chris Mahan chris.mahan at gmail.com cell 818.943.1850 -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Mon Oct 6 22:43:30 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 06 Oct 2008 16:43:30 -0400 Subject: [Distutils] Availability of setuptools installer for python2.6? In-Reply-To: <4d9b53db0810061205w75a14a4ewf388d8bd5dc0f62a@mail.gmail.co m> References: <4d9b53db0810061205w75a14a4ewf388d8bd5dc0f62a@mail.gmail.com> Message-ID: <20081006204222.486FF3A4045@sparrow.telecommunity.com> At 12:05 PM 10/6/2008 -0700, Chris Mahan wrote: >Phillip, > >At >http://pypi.python.org/pypi/setuptools >there is no MS windows installer for python 2.6. > >Do you have an idea of when that might be available? No, it'll likely be a while before I'm doing anything with 2.6. Ian Bicking now has privs for the setuptools PyPI entry, so perhaps he could upload one. (There actually isn't anything Windows-specific about it; all the .exe installers that I upload are actually built on a Linux machine.) From floris.bruynooghe at gmail.com Mon Oct 6 23:25:11 2008 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Mon, 6 Oct 2008 22:25:11 +0100 Subject: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification In-Reply-To: <20081003163402.638A93A407A@sparrow.telecommunity.com> References: <20081003163402.638A93A407A@sparrow.telecommunity.com> Message-ID: <20081006212511.GC17624@laurie.devork> On Fri, Oct 03, 2008 at 12:35:11PM -0400, Phillip J. Eby wrote: > I'm thinking about putting together a pre-PEP for a "Build Utilities, > Installation Locations, & Distribution Standards" (BUILDS) > specification. Hehe, clever name! > The basic idea for the first PEP is to: [...] > 2. Comment on some lessons learned from the WSGI definition process and > WSGI in the field, and how they can be applied to the BUILDS process That will be interesting to read. Not being a web developer I have no idea how WSGI came to be and what exactly it does. [...] > 4. Lay out *non-goals* for the BUILDS project, [...] > or doing anything that requires > them to change the runtime contents of their projects (as opposed to > merely porting their setup.py, setup.cfg, etc.) Does this mean actively avoiding an API that would allow developers to access certain types of data files (I'm thinking of the discussion about locale data and not putting anything else but .py/.pyc/.pyo files in packages) or merely making sure the existing way (of shipping data files in packages and finding them by os.path and __file__) keeps working? > 5. Define rigorous terminology to be used for discussion of requirements > and design, including such terms as "project", "release", "distribution", > "system package", "installed distribution", etc. (This is incredibly > important, because the discussions we're having now are already having > Tower-of-Babel confusions.) Yay, please make sure the terminology finally allows us to talk about packages in the .deb and .rpm meaning too. The current distutils terminology doesn't and it really doesn't help. > I also think we're going to > want a working group for this, or maybe multiple working groups, and it > might be best not to use the general distutils-SIG for discussion past > the first PEP, to allow people to filter threads better. Hmm, this all seems quite on topic for distutils-sig. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From p.f.moore at gmail.com Mon Oct 6 23:25:46 2008 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 6 Oct 2008 22:25:46 +0100 Subject: [Distutils] Availability of setuptools installer for python2.6? In-Reply-To: <20081006204222.486FF3A4045@sparrow.telecommunity.com> References: <4d9b53db0810061205w75a14a4ewf388d8bd5dc0f62a@mail.gmail.com> <20081006204222.486FF3A4045@sparrow.telecommunity.com> Message-ID: <79990c6b0810061425j11b0a4d7tf51b0e8d856a9497@mail.gmail.com> 2008/10/6 Phillip J. Eby : > At 12:05 PM 10/6/2008 -0700, Chris Mahan wrote: >> >> Phillip, >> >> At >> http://pypi.python.org/pypi/setuptools >> there is no MS windows installer for python 2.6. >> >> Do you have an idea of when that might be available? > > No, it'll likely be a while before I'm doing anything with 2.6. Ian Bicking > now has privs for the setuptools PyPI entry, so perhaps he could upload one. > (There actually isn't anything Windows-specific about it; all the .exe > installers that I upload are actually built on a Linux machine.) This is a really frustrating aspect of setuptools, that pure-Python packages produce version-specific installers. I'm sure I've seen an explanation of why this is necessary somewhere before, but I can't recall precisely where (and I don't really have time to wade through all the setuptools documentation to see if it's in there - it wasn't obvious from reading the contents). Can anyone give me a quick pointer to the explanation? Thanks, Paul From robert.kern at gmail.com Tue Oct 7 00:43:26 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 06 Oct 2008 17:43:26 -0500 Subject: [Distutils] Availability of setuptools installer for python2.6? In-Reply-To: <79990c6b0810061425j11b0a4d7tf51b0e8d856a9497@mail.gmail.com> References: <4d9b53db0810061205w75a14a4ewf388d8bd5dc0f62a@mail.gmail.com> <20081006204222.486FF3A4045@sparrow.telecommunity.com> <79990c6b0810061425j11b0a4d7tf51b0e8d856a9497@mail.gmail.com> Message-ID: Paul Moore wrote: > 2008/10/6 Phillip J. Eby : >> At 12:05 PM 10/6/2008 -0700, Chris Mahan wrote: >>> Phillip, >>> >>> At >>> http://pypi.python.org/pypi/setuptools >>> there is no MS windows installer for python 2.6. >>> >>> Do you have an idea of when that might be available? >> No, it'll likely be a while before I'm doing anything with 2.6. Ian Bicking >> now has privs for the setuptools PyPI entry, so perhaps he could upload one. >> (There actually isn't anything Windows-specific about it; all the .exe >> installers that I upload are actually built on a Linux machine.) > > This is a really frustrating aspect of setuptools, that pure-Python > packages produce version-specific installers. I'm sure I've seen an > explanation of why this is necessary somewhere before, but I can't > recall precisely where (and I don't really have time to wade through > all the setuptools documentation to see if it's in there - it wasn't > obvious from reading the contents). Can anyone give me a quick pointer > to the explanation? .pyc files are minor-version-specific. Eggs contain .pyc files. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From zooko at zooko.com Tue Oct 7 01:19:54 2008 From: zooko at zooko.com (zooko) Date: Mon, 6 Oct 2008 17:19:54 -0600 Subject: [Distutils] Availability of setuptools installer for python2.6? In-Reply-To: References: <4d9b53db0810061205w75a14a4ewf388d8bd5dc0f62a@mail.gmail.com> <20081006204222.486FF3A4045@sparrow.telecommunity.com> <79990c6b0810061425j11b0a4d7tf51b0e8d856a9497@mail.gmail.com> Message-ID: <4DA33247-7EAD-4215-B470-A1FE8BC38527@zooko.com> On Oct 6, 2008, at 16:43 PM, Robert Kern wrote: > >> This is a really frustrating aspect of setuptools, that pure-Python >> packages produce version-specific installers. ... > .pyc files are minor-version-specific. Eggs contain .pyc files. That's one of the advantages that .py's have over .pyc's in eggs. For example, Tahoe ships a bundled version of the setuptools bootstrap egg in which I just manually deleted the .pyc's and renamed it from "setuptools-0.6c8-py2.5.egg" to "setuptools-0.6c8.egg": http://allmydata.org/trac/tahoe/browser/misc/dependencies?rev=2943 Perhaps this will cause some problems for us (what are those .exe's in there for, anyway? cli.exe and gui.exe), but it has worked fine so far. See also: http://allmydata.org/trac/setuptools/ticket/5 # binary eggs should come with .py files by default, rather than .pyc files Note that this URL is to an interim issue tracker that I set up a while back, before the official setuptools issue tracker [1] was set up. So don't post any comments to that ticket -- instead if you think it is an issue then post a new ticket on the official setuptools tracker and/or the new "distribute" project... Hey! Where's the issue tracker for the new "distribute" project? Regards, Zooko [1] http://bugs.python.org/setuptools --- http://allmydata.org -- Tahoe, the Least-Authority Filesystem http://allmydata.com -- back up all your files for $5/month From pje at telecommunity.com Tue Oct 7 03:18:17 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 06 Oct 2008 21:18:17 -0400 Subject: [Distutils] Availability of setuptools installer for python2.6? In-Reply-To: <79990c6b0810061425j11b0a4d7tf51b0e8d856a9497@mail.gmail.co m> References: <4d9b53db0810061205w75a14a4ewf388d8bd5dc0f62a@mail.gmail.com> <20081006204222.486FF3A4045@sparrow.telecommunity.com> <79990c6b0810061425j11b0a4d7tf51b0e8d856a9497@mail.gmail.com> Message-ID: <20081007011708.203CE3A4045@sparrow.telecommunity.com> At 10:25 PM 10/6/2008 +0100, Paul Moore wrote: >2008/10/6 Phillip J. Eby : > > At 12:05 PM 10/6/2008 -0700, Chris Mahan wrote: > >> > >> Phillip, > >> > >> At > >> > http://pypi.python.org/pypi/setuptools > >> there is no MS windows installer for python 2.6. > >> > >> Do you have an idea of when that might be available? > > > > No, it'll likely be a while before I'm doing anything with > 2.6. Ian Bicking > > now has privs for the setuptools PyPI entry, so perhaps he could > upload one. > > (There actually isn't anything Windows-specific about it; all the .exe > > installers that I upload are actually built on a Linux machine.) > >This is a really frustrating aspect of setuptools, that pure-Python >packages produce version-specific installers. Actually, that's not setuptools' fault in this case; I specifically make the .exe's version-specific because they have different contents. Different versions of Python include different distutils commands, and setuptools needs to install different things. So even though it's "pure" Python (ha!) it is still Python-version specific. >I'm sure I've seen an >explanation of why this is necessary somewhere before, but I can't >recall precisely where (and I don't really have time to wade through >all the setuptools documentation to see if it's in there - it wasn't >obvious from reading the contents). Can anyone give me a quick pointer >to the explanation? Eggs contain bytecode, and bytecode is Python version-specific. From pje at telecommunity.com Tue Oct 7 03:33:23 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 06 Oct 2008 21:33:23 -0400 Subject: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification In-Reply-To: <20081006212511.GC17624@laurie.devork> References: <20081003163402.638A93A407A@sparrow.telecommunity.com> <20081006212511.GC17624@laurie.devork> Message-ID: <20081007013212.02F773A4045@sparrow.telecommunity.com> At 10:25 PM 10/6/2008 +0100, Floris Bruynooghe wrote: >On Fri, Oct 03, 2008 at 12:35:11PM -0400, Phillip J. Eby wrote: > > I'm thinking about putting together a pre-PEP for a "Build Utilities, > > Installation Locations, & Distribution Standards" (BUILDS) > > specification. > >Hehe, clever name! > > > The basic idea for the first PEP is to: >[...] > > 2. Comment on some lessons learned from the WSGI definition process and > > WSGI in the field, and how they can be applied to the BUILDS process > >That will be interesting to read. Not being a web developer I have no >idea how WSGI came to be and what exactly it does. WSGI came about because I was frustrated with the non-progress of the Web-SIG in creating standard request and response objects; it was quickly obvious to me that it would never go anywhere, due to socio/psych/economic considerations. I proposed it in order to break the logjam by having a standard that more or less allowed everybody to do whatever they wanted and still be able to co-operate... by arranging things so that the costs and benefits of adoption were balanced for all parties. In the case of BUILDS, I propose to do the same: define a standard whose cost/benefit ratios are ideally balanced for each participant. This does not, by the way, mean that everybody ends up with the same cost/benefit ratio; it simply means that the cost/benefit ratios are best for those people whose participation is most required for the standard to be widely adopted. You can see that this is also what I did in the design of easy_install and setuptools, except that in that effort I only considered developers and users, not system packagers. I propose to rectify that with BUILDS, although developers are still the most important audience a new standard must satisfy, in the sense that they require the best cost/benefit ratio for widespread adoption to occur. (And this is of course in system packagers' best interest, since widespread adoption of the standard should make their lives easier, as long as the standard provides an improvement over the status quo for them.) There are of course some other differences between this effort and those of WSGI and setuptools, but that's the 30-second summary. :) >[...] > > 4. Lay out *non-goals* for the BUILDS project, >[...] > > or doing anything that requires > > them to change the runtime contents of their projects (as opposed to > > merely porting their setup.py, setup.cfg, etc.) > >Does this mean actively avoiding an API that would allow developers to >access certain types of data files (I'm thinking of the discussion >about locale data and not putting anything else but .py/.pyc/.pyo >files in packages) or merely making sure the existing way (of shipping >data files in packages and finding them by os.path and __file__) keeps >working? I would actively avoid it for a "BUILDS 1.0" spec, because on any platform where the people building installation tools care about relocating these files, symlinks are available, so both sides can be happy without needing a new API. That is, unless I have misunderstood Josselin and Toshio, I understand symlinks to currently be an acceptable compromise. (For example, Debian uses them to relocate .pyc files currently.) > > 5. Define rigorous terminology to be used for discussion of requirements > > and design, including such terms as "project", "release", "distribution", > > "system package", "installed distribution", etc. (This is incredibly > > important, because the discussions we're having now are already having > > Tower-of-Babel confusions.) > >Yay, please make sure the terminology finally allows us to talk about >packages in the .deb and .rpm meaning too. The term I use for such packages is "system packages", btw. From joss at debian.org Tue Oct 7 09:57:56 2008 From: joss at debian.org (Josselin Mouette) Date: Tue, 07 Oct 2008 09:57:56 +0200 Subject: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification In-Reply-To: <20081007013212.02F773A4045@sparrow.telecommunity.com> References: <20081003163402.638A93A407A@sparrow.telecommunity.com> <20081006212511.GC17624@laurie.devork> <20081007013212.02F773A4045@sparrow.telecommunity.com> Message-ID: <1223366277.5063.18.camel@tomoyo> Le lundi 06 octobre 2008 ? 21:33 -0400, Phillip J. Eby a ?crit : > >Does this mean actively avoiding an API that would allow developers to > >access certain types of data files (I'm thinking of the discussion > >about locale data and not putting anything else but .py/.pyc/.pyo > >files in packages) or merely making sure the existing way (of shipping > >data files in packages and finding them by os.path and __file__) keeps > >working? > > I would actively avoid it for a "BUILDS 1.0" spec, because on any > platform where the people building installation tools care about > relocating these files, symlinks are available, so both sides can be > happy without needing a new API. I fail to see what this would bring over the current situation, then. > That is, unless I have misunderstood Josselin and Toshio, I > understand symlinks to currently be an acceptable compromise. (For > example, Debian uses them to relocate .pyc files currently.) Symlinks are a real pain to handle. We can use them transparently for .pyc files, but if we want to relocate data files to some other directories, currently it has to be done by hand, and this is why most maintainers don?t do it. Furthermore, you seem to be unaware of the amount of abuse that was produced by the mere idea of using __file__ to locate data. It is not only about loading a few image files, let me show you some cases. Case 1: pastescript. This module ships templates of python modules, that are named with the .py extension but are not usable as is from the interpreter, and shipped in the modules directory. How in the world can our tools know that they shouldn?t be byte-compiled and that they should be relocated to somewhere in /usr/share/$package? Case 2: twisted. Plugins, consisting of a few .py files, are shipped in a plugins/ subdirectory, and its content is dynamically added to sys.path after some sanity checks (which fail when you add namespace packages). The reason is to allow plugins for several incompatible Twisted versions to be installed in the *same* directory. When there are solutions as simple as versioning the plugin directory, I can?t understand how people can think of (and implement) such broken solutions. Case 3: GStreamer. The modules installation directory is detected at compile time and then hardcoded in the modules themselves so that some subdirectories can be added to sys.path upon loading. In the end, the modules are not relocatable. I keep wondering what Python module developers will invent next to complicate packaging even more. Frankly, we?d lose much less hair if such things were simply impossible. At least, it should be documented that such practice is wrong, so that we can actually consider it buggy and make our tools simpler. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From p.f.moore at gmail.com Tue Oct 7 10:58:24 2008 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 7 Oct 2008 09:58:24 +0100 Subject: [Distutils] Availability of setuptools installer for python2.6? In-Reply-To: <20081007011708.203CE3A4045@sparrow.telecommunity.com> References: <4d9b53db0810061205w75a14a4ewf388d8bd5dc0f62a@mail.gmail.com> <20081006204222.486FF3A4045@sparrow.telecommunity.com> <79990c6b0810061425j11b0a4d7tf51b0e8d856a9497@mail.gmail.com> <20081007011708.203CE3A4045@sparrow.telecommunity.com> Message-ID: <79990c6b0810070158v76353a01x1c71c89ba02eeab5@mail.gmail.com> 2008/10/7 Phillip J. Eby : >> This is a really frustrating aspect of setuptools, that pure-Python >> packages produce version-specific installers. > > Actually, that's not setuptools' fault in this case; I specifically make the > .exe's version-specific because they have different contents. Different > versions of Python include different distutils commands, and setuptools > needs to install different things. So even though it's "pure" Python (ha!) > it is still Python-version specific. Not sure I follow this. I see this in bdist_wininst installers, so distutils commands shouldn't be relevant (?) But I'll freely admit I'm naive over this stuff, so if I need to be enlightened, please do so! > Eggs contain bytecode, and bytecode is Python version-specific. bdist_wininst recompiles bytecode at install-time, so that's not relevant for me. I can see it would be for eggs, but I'm talking about installers, here. Hmm, I just went looking for a specific example, and I see that the one I thought I remembered, Genshi, includes some C code I'd missed. Maybe I'm misinterpreting what I thought I'd seen. But I do wish people would continue to distribute bdist_wininst installers for pure python packages, rather than just version-specific .egg files. Never mind. We've done this discussion to death in the past. Let's just say that the move from bdist_wininst to eggs in some areas, is making an early switch to Python 2.6 harder for me on Windows than the equivalent switch to 2.5 was... Paul. From p.f.moore at gmail.com Tue Oct 7 11:04:19 2008 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 7 Oct 2008 10:04:19 +0100 Subject: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification In-Reply-To: <20081007013212.02F773A4045@sparrow.telecommunity.com> References: <20081003163402.638A93A407A@sparrow.telecommunity.com> <20081006212511.GC17624@laurie.devork> <20081007013212.02F773A4045@sparrow.telecommunity.com> Message-ID: <79990c6b0810070204r4b2fb216la1daa69e36ba78f0@mail.gmail.com> 2008/10/7 Phillip J. Eby : > In the case of BUILDS, I propose to do the same: define a standard whose > cost/benefit ratios are ideally balanced for each participant. This does > not, by the way, mean that everybody ends up with the same cost/benefit > ratio; it simply means that the cost/benefit ratios are best for those > people whose participation is most required for the standard to be widely > adopted. > > You can see that this is also what I did in the design of easy_install and > setuptools, except that in that effort I only considered developers and > users, not system packagers. I'd argue (you may differ) that the most significant area where you missed the mark on user benefits with easy_install and setuptools is the lack of easy *uninstall* and easy *list* options. Most of the issues I hear from users about setuptools (filtered by my prejudices, admittedly) is that there's no management options (which brings in the system packagers, and their concerns). Can I suggest that this be included somehow in the new spec, so that metadata is available to make wtiting uninstallers and listers as easy as writing installers? Paul. From skip at pobox.com Tue Oct 7 11:41:30 2008 From: skip at pobox.com (skip at pobox.com) Date: Tue, 7 Oct 2008 04:41:30 -0500 Subject: [Distutils] Downloading betas? Message-ID: <18667.11978.960682.426847@montanaro-dyndns-org.local> I have been using ZODB 3.6.0 for SpamBayes for awhile. I noticed today that ZODB 3.8.0 is available from PyPI, so I ran easy_install -U ZODB3 It determined that 3.8.1b8 was the "best" match. Why didn't it stop with 3.8.0 instead of installing a version which was clearly marked as beta? Is there a way to tell it, "don't do that"? Thx, Skip From exarkun at divmod.com Tue Oct 7 14:01:28 2008 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Tue, 7 Oct 2008 08:01:28 -0400 Subject: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification In-Reply-To: <1223366277.5063.18.camel@tomoyo> Message-ID: <20081007120128.29191.432847177.divmod.quotient.35085@ohm> On Tue, 07 Oct 2008 09:57:56 +0200, Josselin Mouette wrote: > >Case 2: twisted. Plugins, consisting of a few .py files, are shipped in >a plugins/ subdirectory, and its content is dynamically added to >sys.path after some sanity checks (which fail when you add namespace >packages). The reason is to allow plugins for several incompatible >Twisted versions to be installed in the *same* directory. When there are >solutions as simple as versioning the plugin directory, I can?t >understand how people can think of (and implement) such broken >solutions. Hi Josselin, The facts you have presented above are not correct. Jean-Paul From floris.bruynooghe at gmail.com Tue Oct 7 14:12:03 2008 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Tue, 7 Oct 2008 13:12:03 +0100 Subject: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification In-Reply-To: <20081007120128.29191.432847177.divmod.quotient.35085@ohm> References: <1223366277.5063.18.camel@tomoyo> <20081007120128.29191.432847177.divmod.quotient.35085@ohm> Message-ID: <20081007121203.GA7418@laurie.devork> On Tue, Oct 07, 2008 at 08:01:28AM -0400, Jean-Paul Calderone wrote: > On Tue, 07 Oct 2008 09:57:56 +0200, Josselin Mouette wrote: >> >> Case 2: twisted. Plugins, consisting of a few .py files, are shipped in >> a plugins/ subdirectory, and its content is dynamically added to >> sys.path after some sanity checks (which fail when you add namespace >> packages). The reason is to allow plugins for several incompatible >> Twisted versions to be installed in the *same* directory. When there are >> solutions as simple as versioning the plugin directory, I can?t >> understand how people can think of (and implement) such broken >> solutions. > > The facts you have presented above are not correct. It would be helpful if you could correct them in that case. Cheers Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From exarkun at divmod.com Tue Oct 7 14:58:40 2008 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Tue, 7 Oct 2008 08:58:40 -0400 Subject: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification In-Reply-To: <20081007121203.GA7418@laurie.devork> Message-ID: <20081007125840.29191.1303604134.divmod.quotient.35103@ohm> On Tue, 7 Oct 2008 13:12:03 +0100, Floris Bruynooghe wrote: >On Tue, Oct 07, 2008 at 08:01:28AM -0400, Jean-Paul Calderone wrote: >> On Tue, 07 Oct 2008 09:57:56 +0200, Josselin Mouette wrote: >>> >>> Case 2: twisted. Plugins, consisting of a few .py files, are shipped in >>> a plugins/ subdirectory, and its content is dynamically added to >>> sys.path after some sanity checks (which fail when you add namespace >>> packages). The reason is to allow plugins for several incompatible >>> Twisted versions to be installed in the *same* directory. When there are >>> solutions as simple as versioning the plugin directory, I can?t >>> understand how people can think of (and implement) such broken >>> solutions. >> >> The facts you have presented above are not correct. > >It would be helpful if you could correct them in that case. > I don't think the details of the plugin system are relevant to the topic under discussion here. The installation requirements are not unusual for the most part - that a directory full of .py files be copied to the install location, just as any package would be. The one unusual thing is that non-Twisted package X might want to copy files into twisted/plugins/ (if it is providing plugins). For details, see the Twisted plugin documentation: http://twistedmatrix.com/projects/core/documentation/howto/plugin.html Or this discussion in the Debian issue tracker: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=474630 I would be happy to answer any questions not addressed in the documentation, but perhaps off-list if they are not directly relevant to distribution topics. Jean-Paul From jim at zope.com Tue Oct 7 14:58:28 2008 From: jim at zope.com (Jim Fulton) Date: Tue, 7 Oct 2008 08:58:28 -0400 Subject: [Distutils] Downloading betas? In-Reply-To: <18667.11978.960682.426847@montanaro-dyndns-org.local> References: <18667.11978.960682.426847@montanaro-dyndns-org.local> Message-ID: <9F7DAD75-1BA2-4657-B17E-365CC32BD48F@zope.com> On Oct 7, 2008, at 5:41 AM, skip at pobox.com wrote: > > I have been using ZODB 3.6.0 for SpamBayes for awhile. I noticed > today that > ZODB 3.8.0 is available from PyPI, so I ran > > easy_install -U ZODB3 > > It determined that 3.8.1b8 was the "best" match. Why didn't it stop > with > 3.8.0 instead of installing a version which was clearly marked as > beta? Is > there a way to tell it, "don't do that"? No. Note that buildout has a prefer-final option to prefer final releases over non-final releases. Jim -- Jim Fulton Zope Corporation From joss at debian.org Tue Oct 7 15:19:57 2008 From: joss at debian.org (Josselin Mouette) Date: Tue, 07 Oct 2008 15:19:57 +0200 Subject: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification In-Reply-To: <20081007125840.29191.1303604134.divmod.quotient.35103@ohm> References: <20081007125840.29191.1303604134.divmod.quotient.35103@ohm> Message-ID: <1223385597.5063.73.camel@tomoyo> Le mardi 07 octobre 2008 ? 08:58 -0400, Jean-Paul Calderone a ?crit : > I don't think the details of the plugin system are relevant to the topic > under discussion here. The installation requirements are not unusual for > the most part - that a directory full of .py files be copied to the install > location, just as any package would be. The one unusual thing is that > non-Twisted package X might want to copy files into twisted/plugins/ (if it > is providing plugins). This is not unusual, but other frameworks that do the same don?t break when you add namespace packages. In fact, the very fact that you can?t add namespace packages shows that you are abusing the python modules directory: you are shipping a twisted/plugins/foo/foo.py file which is not meant to be available as a twisted.plugins.foo.foo module. The ability to install several incompatible versions of Twisted on the system is the reason I was given by glyph; if there is another reason, I?d be happy to hear the explanation, but the result is the same: we had to code a special case for Twisted. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From exarkun at divmod.com Tue Oct 7 16:02:58 2008 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Tue, 7 Oct 2008 10:02:58 -0400 Subject: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification In-Reply-To: <1223385597.5063.73.camel@tomoyo> Message-ID: <20081007140258.29191.1111906319.divmod.quotient.35125@ohm> On Tue, 07 Oct 2008 15:19:57 +0200, Josselin Mouette wrote: >Le mardi 07 octobre 2008 ? 08:58 -0400, Jean-Paul Calderone a ?crit : >> I don't think the details of the plugin system are relevant to the topic >> under discussion here. The installation requirements are not unusual for >> the most part - that a directory full of .py files be copied to the install >> location, just as any package would be. The one unusual thing is that >> non-Twisted package X might want to copy files into twisted/plugins/ (if it >> is providing plugins). > >This is not unusual, but other frameworks that do the same don?t break >when you add namespace packages. In fact, the very fact that you can?t >add namespace packages shows that you are abusing the python modules >directory: you are shipping a twisted/plugins/foo/foo.py file which is >not meant to be available as a twisted.plugins.foo.foo module. I think you meant "shipping a twisted/plugins/foo.py file which is not meant to be available as a twisted.plugins.foo module", since as far as I know, no one is putting packages (as opposed to modules) into the twisted.plugins package (and I don't think it works, either). >The ability to install several incompatible versions of Twisted on the >system is the reason I was given by glyph; if there is another reason, >I?d be happy to hear the explanation, but the result is the same: we had >to code a special case for Twisted. > Above correction aside, your understanding of the system is still flawed (I am fully aware that this is likely due to our failure to document the system sufficiently, so please don't interpret that as an attack). I think that you also misunderstood the explanation that Glyph provided. In your earlier message, you wrote this: > The reason is to allow plugins for several incompatible Twisted versions > to be installed in the *same* directory. However, no such feature is offered by the Twisted plugin system. :) The issue which arose when Debian packaged Twisted and (for example) a new version of Nevow which provides a plugin to Twisted is that Nevow's plugin was installed to a location which was not the location intended or expected by the Twisted developers or the Nevow developers. The expectation of the Twisted developers was that the installed directory structure for Twisted files would look like this: /usr/lib/python2.5/site-packages/twisted/ /usr/lib/python2.5/site-packages/twisted/__init__.py /usr/lib/python2.5/site-packages/twisted/plugins/ /usr/lib/python2.5/site-packages/twisted/plugins/__init__.py (obviously this is only part of what a Twisted installation would look like) The expectations of the Nevow developers was that a file included in Nevow, nevow_widget.py, would be installed to /usr/lib/python2.5/site-packages/twisted/plugins/nevow_widget.py On Debian, difficulties were encountered because, while Twisted was installed more or less to the expected location, the file in Nevow was instead installed to /var/lib/python-support/python2.5/twisted/plugins/nevow_widget.py and the infrastructure for installing the file to this location also creates /var/lib/python-support/python2.5/twisted/ /var/lib/python-support/python2.5/twisted/__init__.py /var/lib/python-support/python2.5/twisted/plugins/ /var/lib/python-support/python2.5/twisted/plugins/__init__.py This is problematic because there are now two Twisted packages in the module search path. Python's module system can't do anything with the second one (which one is second depends on the order of sys.path). If the version created for nevow_widget.py came first, the Twisted package itself would basically be completely broken. Fortunately, it doesn't, so that doesn't happen. However, since nevow_widget.py is in the second version, the only way it can be imported is if some extra help is supplied to the import system. In previous releases of Twisted, this help was given by twisted/plugins/ __init__.py. It included an extremely primitive implementation of the concept of namespace packages (_extremely_ primitive). It would find all directories named "twisted/plugins" which were in the module search path and add them to its own __path__. The purpose of this feature wasn't to support installations, however. The purpose of this feature was as a developer convenience. It allows developers to edit plugins in a non- installed location (for example, their home directory) and then use them without copying them to the installed Twisted's twisted/plugins/ directory. In a more recent release of Twisted, this feature was tweaked slightly. The reason for the change is that there was a significant bug in the old implementation of namespace packages. If a developer was using their own copy of Twisted (for example, an SVN checkout in their home directory) instead of the installed version, the namespace package implementation would still load plugins from the *installed* version of Twisted. In the best possible case, this gave you two copies of every plugin. In the worst case, the installed version of Twisted wasn't compatible with the other version of Twisted and the plugins would raise exceptions. This is the failure condition which I believe you misinterpreted as meaning we were trying to support any kind of versioning features. We're not. We're only trying to not run code that the user (/developer) doesn't want run. So, to fix this, we tweaked the namespace package setup code so that it will skip any "twisted/plugins/" directory which contains an "__init__.py", since that is a relatively strong hint that it's a real package, not one of the developer-convenience plugin locations (which are not real packages). Debian's Nevow package then broke, since Debian installed an __init__.py when installing Nevow's nevow_widget.py file. The ideal fix, from my perspective as a Twisted developer, is to install the nevow_widget.py file into /usr/lib/python2.5/site-packages/twisted/plugins/ along with all the other Twisted plugin files (perhaps using a symlink, I can't think of any reason why that would make a difference, particularly since all the other .py files in that area are symlinks). In other words, Debian took advantage of a feature which was intended to be used only as a convenience for development and was never intended to assist installation. Have I succeeded in explaining why the Twisted plugin system isn't making any unusual requirements of the installation system nor using the Python module system in an unusual way? Can I clarify any parts of this further? Can I expect to see any improvements to the Debian packaging of Twisted and projects which supply plugins to Twisted? :) Jean-Paul From ziade.tarek at gmail.com Tue Oct 7 16:07:26 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 7 Oct 2008 10:07:26 -0400 Subject: [Distutils] pre-PEP : Synthesis of previous threads, and irc talks + proposals In-Reply-To: <8E57AB2B-4C1C-49F4-9FF8-78F57509395E@zooko.com> References: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.com> <8E57AB2B-4C1C-49F4-9FF8-78F57509395E@zooko.com> Message-ID: <94bdd2610810070707k5fee06eayd2d1086da9459213@mail.gmail.com> On Sun, Oct 5, 2008 at 12:04 PM, zooko wrote: >> 5/ ideally, they should be one and only one version of a given package >> in an OS-based installation > > -1 -- This is the strong preference of the folks who package software for > OSes -- Debian, Fedora, etc. -- but it is not necessarily the choice of the > users who use their OSes. It is best for the Python packaging standards to > be agnostic towards this, or at least to support both this desideratum and > its opposite. I can see this as an exponentional problem for packagers, but let's think about it: - How you would handle several version of the same package in Python then ? - How each application would pick the right version ? - How would you decide which version is the one by default ? That is the core of the problem. The -m feature of setuptools is nice, but it activates one version at a time, and this is globlal to Python unless each application is handling the version switch, wich is pretty heavy. A programmable sys.path ? Tarek. From joss at debian.org Tue Oct 7 17:11:19 2008 From: joss at debian.org (Josselin Mouette) Date: Tue, 07 Oct 2008 17:11:19 +0200 Subject: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification In-Reply-To: <20081007140258.29191.1111906319.divmod.quotient.35125@ohm> References: <20081007140258.29191.1111906319.divmod.quotient.35125@ohm> Message-ID: <1223392279.5063.123.camel@tomoyo> Le mardi 07 octobre 2008 ? 10:02 -0400, Jean-Paul Calderone a ?crit : > The expectations of the Nevow developers was that a file included in Nevow, > nevow_widget.py, would be installed to > > /usr/lib/python2.5/site-packages/twisted/plugins/nevow_widget.py This expectation is wrong. You?re shipping it as a Python module, the only thing you can expect is to be able to import it. This wouldn?t happen if you were shipping plugins in a specific, private, plugins directory. > The ideal fix, from my perspective as a Twisted developer, is to install > the nevow_widget.py file into > > /usr/lib/python2.5/site-packages/twisted/plugins/ As explained on the debian-python list (see http://lists.debian.org/debian-python/2008/05/msg00032.html), there are strong reasons for not putting managed modules in the same directory as modules included in the Python distribution. So you can?t expect modules to be in a specific directory (but this shouldn?t be a problem since these are not modules, hmmm?). (I have some ideas for keeping all files at the same place and moving only .pyc files to /var, but it requires patching the interpreter and the Debian Python maintainer disagrees, so we?ll have to stick to the current situation for now.) > along with all the other Twisted plugin files (perhaps using a symlink, I > can't think of any reason why that would make a difference, particularly > since all the other .py files in that area are symlinks). > > In other words, Debian took advantage of a feature which was intended to be > used only as a convenience for development and was never intended to assist > installation. The fact that we use a .pth to specify the different module hierarchy is an implementation detail but is indeed abusing this feature; we should be shipping a modified setup.py instead. However this has nothing to do with the inability of Twisted to cope with multiple module paths. Let?s take another example that is not Debian specific: what if I want to install a plugin to /usr/local? The fact that you are abusing the python modules directory forces me to install it to /usr/local/python2.5/site-packages/twisted/plugins instead of e.g. /usr/local/share/twisted/plugins, where a normal application or library would be looking for its plugins. And there, I have to deal with namespaces and other python module specificities that should be irrelevant for plugins. In the beginning, Debian only expected the Python modules directory to contain, well, Python modules. It seems even that is too much to expect, since module developers are taking advantage of a set of features (module file location) that were initially meant for introspection. > Have I succeeded in explaining why the Twisted plugin system isn't making > any unusual requirements of the installation system nor using the Python > module system in an unusual way? Not really. You have succeeded in explaining what is the exact nature of the unusual requirements you make, though. Thanks for that, as it is much clearer than the original explanations I received. > Can I expect to see any improvements to the Debian packaging of Twisted and > projects which supply plugins to Twisted? :) Until you fix your plugin system, we are bound to treat Twisted packages as a special case. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From fdrake at gmail.com Tue Oct 7 17:37:52 2008 From: fdrake at gmail.com (Fred Drake) Date: Tue, 7 Oct 2008 11:37:52 -0400 Subject: [Distutils] [Catalog-sig] Distutils and PyPI : P4-Sprint in D.C. In-Reply-To: <94bdd2610810070735g1e4a93d6n45beda83dd8c158c@mail.gmail.com> References: <94bdd2610810070735g1e4a93d6n45beda83dd8c158c@mail.gmail.com> Message-ID: <9cee7ab80810070837q486414a3v8dadd697e7892d09@mail.gmail.com> On Tue, Oct 7, 2008 at 10:35 AM, Tarek Ziad? wrote: > We are going to have a P4-sprint (pre-pre-pre-PEP sprint) in D.C. > during the Plone Conference. Very cool. Given the venue, should people expect that they're welcome in person even if not associated with the Plone conference? I don't know if I'll be able to steal some time from my family, but there's a possibility. -Fred -- Fred L. Drake, Jr. "Chaos is the score upon which reality is written." --Henry Miller From exarkun at divmod.com Tue Oct 7 17:44:06 2008 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Tue, 7 Oct 2008 11:44:06 -0400 Subject: [Distutils] Twisted packaging in Debian (was Re: Pre-pre-PEP: Requirements for the Python BUILDS Specification) In-Reply-To: <1223392279.5063.123.camel@tomoyo> Message-ID: <20081007154406.29191.1974198569.divmod.quotient.35156@ohm> On Tue, 07 Oct 2008 17:11:19 +0200, Josselin Mouette wrote: >Le mardi 07 octobre 2008 ? 10:02 -0400, Jean-Paul Calderone a ?crit : >> The expectations of the Nevow developers was that a file included in Nevow, >> nevow_widget.py, would be installed to >> >> /usr/lib/python2.5/site-packages/twisted/plugins/nevow_widget.py > >This expectation is wrong. You?re shipping it as a Python module, the >only thing you can expect is to be able to import it. > Leaving aside whether or not I think that is a reasonable restriction to impose, I don't see the relevance. The only reason the module can be imported on Debian now is because of the special measures included (by upstream developers, not by Debian) in twisted/plugins/__init__.py. I can't understand how this could be part of Debian's Python packaging policy, so perhaps I'm missing something very fundamental. > [snip - "specific, private, plugins directory" which I may not understand, > but doubt is relevant] >> The ideal fix, from my perspective as a Twisted developer, is to install >> the nevow_widget.py file into >> >> /usr/lib/python2.5/site-packages/twisted/plugins/ > >As explained on the debian-python list (see >http://lists.debian.org/debian-python/2008/05/msg00032.html), there are >strong reasons for not putting managed modules in the same directory as >modules included in the Python distribution. So you can?t expect modules >to be in a specific directory (but this shouldn?t be a problem since >these are not modules, hmmm?). They're modules. They get imported. That makes them modules, right? exarkun at charm:~$ python Python 2.5.2 (r252:60911, Jul 31 2008, 17:28:52) [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import twisted.plugins.nevow_widget >>> Do we have the same definition of "module"? I chose the phrase "The ideal fix, from my perspective as a Twisted developer" very intentionally. If the file ends up somewhere else but it's still an importable module, fine - if, from a Debian packager's perspective that's better, fine. I don't believe that the current situation is better from a Debian packager's perspective, though, since it involves carrying a patch that reverts an upstream bugfix. > [snip - .pyc files in /var - not directly relevant] > [snip - use of .pth file - not directly relevant] > >However this has nothing to do with the inability of Twisted to cope >with multiple module paths. Let?s take another example that is not >Debian specific: what if I want to install a plugin to /usr/local? The >fact that you are abusing the python modules directory forces me to ^^^^^^^^^^^^ >install it to /usr/local/python2.5/site-packages/twisted/plugins instead ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >of e.g. /usr/local/share/twisted/plugins, where a normal application or ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >library would be looking for its plugins. And there, I have to deal with >namespaces and other python module specificities that should be >irrelevant for plugins. No, this is wrong. Install it to /usr/local/share/ if you want. It will work just fine. Why do you think it won't? I'd rather you _didn't_, at least not without a couple other changes, and I don't understand why you would _want_ to, but it's possible. > >In the beginning, Debian only expected the Python modules directory to >contain, well, Python modules. It seems even that is too much to expect, >since module developers are taking advantage of a set of features >(module file location) that were initially meant for introspection. I assume you're talking about the "Python modules directory" in the install target (not, say, in an upstream VCS). Twisted plugins are modules, so Debian's expectations (be they valid or otherwise, I make no judgement) aren't violated by them (oops, except for dropin.cache files, but we haven't really talked about those yet and I don't think they're actually contentious). >> Have I succeeded in explaining why the Twisted plugin system isn't making >> any unusual requirements of the installation system nor using the Python >> module system in an unusual way? > >Not really. You have succeeded in explaining what is the exact nature of >the unusual requirements you make, though. Thanks for that, as it is >much clearer than the original explanations I received. I'm glad this was clearer. I'm not sure I've yet succeeded in entirely explaining the system. I hope this follow up goes a little bit further toward that goal. > [snip - I can't expect to see any improvements yet] Jean-Paul From ziade.tarek at gmail.com Tue Oct 7 17:44:29 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 7 Oct 2008 11:44:29 -0400 Subject: [Distutils] [Catalog-sig] Distutils and PyPI : P4-Sprint in D.C. In-Reply-To: <9cee7ab80810070837q486414a3v8dadd697e7892d09@mail.gmail.com> References: <94bdd2610810070735g1e4a93d6n45beda83dd8c158c@mail.gmail.com> <9cee7ab80810070837q486414a3v8dadd697e7892d09@mail.gmail.com> Message-ID: <94bdd2610810070844t7622c326ib9aba78a18e1bc5c@mail.gmail.com> On Tue, Oct 7, 2008 at 11:37 AM, Fred Drake wrote: > On Tue, Oct 7, 2008 at 10:35 AM, Tarek Ziad? wrote: >> We are going to have a P4-sprint (pre-pre-pre-PEP sprint) in D.C. >> during the Plone Conference. > > Very cool. > > Given the venue, should people expect that they're welcome in person > even if not associated with the Plone conference? Yes there's a huge space, everyone is welcome ! > > I don't know if I'll be able to steal some time from my family, but > there's a possibility. > That would be awesome ! Tarek > > -Fred > > -- > Fred L. Drake, Jr. > "Chaos is the score upon which reality is written." --Henry Miller > -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From joss at debian.org Tue Oct 7 18:26:57 2008 From: joss at debian.org (Josselin Mouette) Date: Tue, 07 Oct 2008 18:26:57 +0200 Subject: [Distutils] Twisted packaging in Debian (was Re: Pre-pre-PEP: Requirements for the Python BUILDS Specification) In-Reply-To: <20081007154406.29191.1974198569.divmod.quotient.35156@ohm> References: <20081007154406.29191.1974198569.divmod.quotient.35156@ohm> Message-ID: <1223396817.5063.161.camel@tomoyo> Le mardi 07 octobre 2008 ? 11:44 -0400, Jean-Paul Calderone a ?crit : > They're modules. They get imported. That makes them modules, right? Fine. I hadn?t understood that they are actually meant to be imported as is (which makes you right in your assertions that the explanations weren?t completely clear to me yet). If they are modules, meant to be imported, they actually have their place in the modules directory, so please disregard what I?ve said about private directories. So, if they are modules, they sometimes need namespace packages to be imported properly. However you intentionally do not import modules that have associated namespace packages, which is the exact opposite of what we expect for other modules. Let me comment on your previous email with that in mind. In previous releases of Twisted, this help was given by twisted/plugins/ __init__.py. It included an extremely primitive implementation of the concept of namespace packages (_extremely_ primitive). It would find all directories named "twisted/plugins" which were in the module search path and add them to its own __path__. The purpose of this feature wasn't to support installations, however. The purpose of this feature was as a developer convenience. It allows developers to edit plugins in a non- installed location (for example, their home directory) and then use them without copying them to the installed Twisted's twisted/plugins/ directory. OK, I get that now. You need to do that because importing plugins.something in the twisted/ directory will only look for subdirectories of the same installation path. In a more recent release of Twisted, this feature was tweaked slightly. The reason for the change is that there was a significant bug in the old implementation of namespace packages. If a developer was using their own copy of Twisted (for example, an SVN checkout in their home directory) instead of the installed version, the namespace package implementation would still load plugins from the *installed* version of Twisted. In the best possible case, this gave you two copies of every plugin. In the worst case, the installed version of Twisted wasn't compatible with the other version of Twisted and the plugins would raise exceptions. About this issue, I still think you need to version your plugins directory if the plugins are not compatible across versions. Otherwise - and this is something that will hit distributions hard - you may install a an incompatible version of, say, Nevow, and get your installation fucked up. However, in the end, I think you?ve been hit by an issue that was only sporadically met until now: managing files belonging to the same module hierarchy with two different tools doesn?t always work. And this is something that we need to manage inside Debian (which is more a political problem than a technical one). > Do we have the same definition of "module"? I think so. > I chose the phrase "The ideal fix, from my perspective as a Twisted > developer" very intentionally. If the file ends up somewhere else > but it's still an importable module, fine - if, from a Debian packager's > perspective that's better, fine. I don't believe that the current > situation is better from a Debian packager's perspective, though, since > it involves carrying a patch that reverts an upstream bugfix. In the end, I don?t think this patch was added as there is another workaround. Either way, it may break again if you change the way you look for plugins, so you are right to expect that they land in the same directory. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From pje at telecommunity.com Tue Oct 7 19:47:24 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 07 Oct 2008 13:47:24 -0400 Subject: [Distutils] Availability of setuptools installer for python2.6? In-Reply-To: <79990c6b0810070158v76353a01x1c71c89ba02eeab5@mail.gmail.co m> References: <4d9b53db0810061205w75a14a4ewf388d8bd5dc0f62a@mail.gmail.com> <20081006204222.486FF3A4045@sparrow.telecommunity.com> <79990c6b0810061425j11b0a4d7tf51b0e8d856a9497@mail.gmail.com> <20081007011708.203CE3A4045@sparrow.telecommunity.com> <79990c6b0810070158v76353a01x1c71c89ba02eeab5@mail.gmail.com> Message-ID: <20081007174613.95DC23A4045@sparrow.telecommunity.com> At 09:58 AM 10/7/2008 +0100, Paul Moore wrote: >2008/10/7 Phillip J. Eby : > >> This is a really frustrating aspect of setuptools, that pure-Python > >> packages produce version-specific installers. > > > > Actually, that's not setuptools' fault in this case; I > specifically make the > > .exe's version-specific because they have different contents. Different > > versions of Python include different distutils commands, and setuptools > > needs to install different things. So even though it's "pure" Python (ha!) > > it is still Python-version specific. > >Not sure I follow this. I see this in bdist_wininst installers, so >distutils commands shouldn't be relevant (?) I'm saying that setuptools' own bdist_wininst installer is version specific because setuptools itself includes different code (and different script names, e.g. "easy_install-2.x") depending on the Python version. So, even though setuptools is "pure Python", its bdist_wininst files are version-specific. This does not have anything to do with packages that simply *use* setuptools; if you make a bdist_wininst of some non-setuptools package, it will not be version specific unless you include C code. From pje at telecommunity.com Tue Oct 7 20:30:45 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 07 Oct 2008 14:30:45 -0400 Subject: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification In-Reply-To: <79990c6b0810070204r4b2fb216la1daa69e36ba78f0@mail.gmail.co m> References: <20081003163402.638A93A407A@sparrow.telecommunity.com> <20081006212511.GC17624@laurie.devork> <20081007013212.02F773A4045@sparrow.telecommunity.com> <79990c6b0810070204r4b2fb216la1daa69e36ba78f0@mail.gmail.com> Message-ID: <20081007182934.40C923A4045@sparrow.telecommunity.com> At 10:04 AM 10/7/2008 +0100, Paul Moore wrote: >2008/10/7 Phillip J. Eby : > > In the case of BUILDS, I propose to do the same: define a standard whose > > cost/benefit ratios are ideally balanced for each participant. This does > > not, by the way, mean that everybody ends up with the same cost/benefit > > ratio; it simply means that the cost/benefit ratios are best for those > > people whose participation is most required for the standard to be widely > > adopted. > > > > You can see that this is also what I did in the design of easy_install and > > setuptools, except that in that effort I only considered developers and > > users, not system packagers. > >I'd argue (you may differ) that the most significant area where you >missed the mark on user benefits with easy_install and setuptools is >the lack of easy *uninstall* and easy *list* options. Well, I'd certainly agree that those features are desirable. But I didn't "miss the mark", in that those features were not part of my mark. ;-) The goal was to get widespread adoption by developers, and thus the primary target audience was developers. Any features that attracted users were there only insofar as the benefit to users would specifically drive adoption by developers. And that means that features that only matter *after* you have things installed (i.e., easy uninstall and list) were of relatively little importance in the initial feature set. It was only necessary to provide the *possibility* of uninstall and list features, and then allow others to scratch the resulting itches. Now, for BUILDS, the situation is different. Just as setuptools' competitor was distutils, BUILDS' competitor is setuptools. So, BUILDS has to offer developers a switching benefit. And, unlike the situation before, now developers are much more likely to have a lot of packages they'd like to list or uninstall, so such features have considerably more value now, than they did then, which makes them worth putting some effort into. Also, since that switching benefit is largely going to come from having better management tools, the management tools need to be easy to build (so that people will build them). However, since system packagers are experiencing pain right now wrt Python packages, there is at least some benefit to their participating in the process. Back when setuptools was developed, people from the distros weren't hanging out here in the distutils-sig... or at least, they never answered any of my calls for interested parties. > Most of the >issues I hear from users about setuptools (filtered by my prejudices, >admittedly) is that there's no management options (which brings in the >system packagers, and their concerns). > >Can I suggest that this be included somehow in the new spec, so that >metadata is available to make wtiting uninstallers and listers as easy >as writing installers? Since the single most important part of BUILDS (and the part that should be done first) is an installation manifest for packaging tools to know what files are of what kind and where, it should be easy for uninstallers to work. Regarding listers, there are already a number of them available in the Cheeseshop, and if you're working with Python 2.5+ (and not using an older distro that deletes .egg-info files), they should also work with distutils-installed packages. The idea behind the installation manifest and metadata (the "IL" and "DS" of BUILDS) is that it should be possible for anyone to write their own management tools for specific scenarios, and not have to rely on a bunch of implementation-specific and not-very-well-documented stuff. (It should also be possible, given the "IL" part, to make a distutils and/or setuptools command that generates this data from a sufficiently well-behaved setup.py.) From pje at telecommunity.com Tue Oct 7 20:40:05 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 07 Oct 2008 14:40:05 -0400 Subject: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification In-Reply-To: <1223366277.5063.18.camel@tomoyo> References: <20081003163402.638A93A407A@sparrow.telecommunity.com> <20081006212511.GC17624@laurie.devork> <20081007013212.02F773A4045@sparrow.telecommunity.com> <1223366277.5063.18.camel@tomoyo> Message-ID: <20081007183853.8E7243A4045@sparrow.telecommunity.com> At 09:57 AM 10/7/2008 +0200, Josselin Mouette wrote: >Le lundi 06 octobre 2008 ? 21:33 -0400, Phillip J. Eby a ??crit : > > >Does this mean actively avoiding an API that would allow developers to > > >access certain types of data files (I'm thinking of the discussion > > >about locale data and not putting anything else but .py/.pyc/.pyo > > >files in packages) or merely making sure the existing way (of shipping > > >data files in packages and finding them by os.path and __file__) keeps > > >working? > > > > I would actively avoid it for a "BUILDS 1.0" spec, because on any > > platform where the people building installation tools care about > > relocating these files, symlinks are available, so both sides can be > > happy without needing a new API. > >I fail to see what this would bring over the current situation, then. > > > That is, unless I have misunderstood Josselin and Toshio, I > > understand symlinks to currently be an acceptable compromise. (For > > example, Debian uses them to relocate .pyc files currently.) > >Symlinks are a real pain to handle. We can use them transparently >for .pyc files, but if we want to relocate data files to some other >directories, currently it has to be done by hand, and this is why most >maintainers don???t do it. Which is why the idea for the BUILDS spec to include a way for automated tools to do it, so that you won't have to do it manually. Or are you saying that that isn't an improvement over the current situation? Yes, it's true that I'm saying that developers should not be *required* to add the extra data to their packages, but that they should be *able* to, and if it is trivial to add the extra data, most should accept patches or respond to requests to do so. Right now, it's not even *possible* for them to do so, however. Also, I wasn't aware of those peculiarities regarding Twisted and GStreamer -- neither one uses setuptools. But certainly they should make interesting test cases for any proposed standard, wrt to finding ways to satisfy both parties. Please understand that I'm not saying we must be 100% backward compatible with 100% of the packages; we just need to be compatible enough with enough of the packages for network effects to drive the adoption of the rest. At the same time, the people I'd most like to see on the PEP team from the developer-user side would definitely include folks from Twisted, Numpy, Enthought, and Zope, as they are the folks who have most stressed the distutils to the limits and beyond. Just getting you and them in the "same room" so to speak seems to already be producing some benefits. From pje at telecommunity.com Tue Oct 7 20:42:24 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 07 Oct 2008 14:42:24 -0400 Subject: [Distutils] [Catalog-sig] pre-PEP : Synthesis of previous threads, and irc talks + proposals In-Reply-To: <94bdd2610810070707k5fee06eayd2d1086da9459213@mail.gmail.co m> References: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.com> <8E57AB2B-4C1C-49F4-9FF8-78F57509395E@zooko.com> <94bdd2610810070707k5fee06eayd2d1086da9459213@mail.gmail.com> Message-ID: <20081007184113.54CF03A4045@sparrow.telecommunity.com> At 10:07 AM 10/7/2008 -0400, Tarek Ziad? wrote: >The -m feature of setuptools is nice, but it activates one version at >a time, and >this is globlal to Python unless each application is handling the >version switch, >wich is pretty heavy. With or without the -m switch, scripts installed by setuptools will find the version they are specified to use, without the user needing to do anything. So, you can have a default version of an egg (used by the interpreter and non-setuptools scripts), and then some non-default versions that are used by scripts. zc.buildout and virtualenv also have their own ways of accomplishing the same thing, e.g., by hardcoding paths in an installed script. From ziade.tarek at gmail.com Tue Oct 7 16:35:49 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 7 Oct 2008 10:35:49 -0400 Subject: [Distutils] Distutils and PyPI : P4-Sprint in D.C. Message-ID: <94bdd2610810070735g1e4a93d6n45beda83dd8c158c@mail.gmail.com> Hey all, We are going to have a P4-sprint (pre-pre-pre-PEP sprint) in D.C. during the Plone Conference. The idea is to try to bring the discussions that have been going on in the mailing lists into the next stage. Please join us if you are interested, even if you are not in D.C. http://www.openplans.org/projects/plone-conference-2008-dc/distribute Regards Tarek -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From joss at debian.org Tue Oct 7 20:58:51 2008 From: joss at debian.org (Josselin Mouette) Date: Tue, 07 Oct 2008 20:58:51 +0200 Subject: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification In-Reply-To: <20081007183853.8E7243A4045@sparrow.telecommunity.com> References: <20081003163402.638A93A407A@sparrow.telecommunity.com> <20081006212511.GC17624@laurie.devork> <20081007013212.02F773A4045@sparrow.telecommunity.com> <1223366277.5063.18.camel@tomoyo> <20081007183853.8E7243A4045@sparrow.telecommunity.com> Message-ID: <1223405931.5063.190.camel@tomoyo> Le mardi 07 octobre 2008 ? 14:40 -0400, Phillip J. Eby a ?crit : > At 09:57 AM 10/7/2008 +0200, Josselin Mouette wrote: > >Symlinks are a real pain to handle. We can use them transparently > >for .pyc files, but if we want to relocate data files to some other > >directories, currently it has to be done by hand, and this is why most > >maintainers don???t do it. > > Which is why the idea for the BUILDS spec to include a way for > automated tools to do it, so that you won't have to do it manually. If you write a tool to do that, why not make it simply move files properly and let the code locate them, instead of adding yet another layer on top of the existing stuff? The tool will not be more complicated this way. > Or are you saying that that isn't an improvement over the current situation? It is. I didn?t understand you wanted to automate the symlinks creation. It indeed means less burden on the maintainers, but it would be a shame to keep the same mess in binary packages, while having a tool to do that would allow to make things cleaner. > Yes, it's true that I'm saying that developers should not be > *required* to add the extra data to their packages, but that they > should be *able* to, and if it is trivial to add the extra data, most > should accept patches or respond to requests to do so. > > Right now, it's not even *possible* for them to do so, however. It is possible - if they use autoconf instead. BTW, I would consider it a good approach to try bringing BUILDS on par with the autotools capabilities. These tools have serious drawbacks, but they were written with the ability to work for distributors in mind. > At the same time, the people I'd most like to see on the PEP team > from the developer-user side would definitely include folks from > Twisted, Numpy, Enthought, and Zope, as they are the folks who have > most stressed the distutils to the limits and beyond. Just getting > you and them in the "same room" so to speak seems to already be > producing some benefits. Looks like a sane approach. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From ziade.tarek at gmail.com Tue Oct 7 20:58:00 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 7 Oct 2008 14:58:00 -0400 Subject: [Distutils] [Catalog-sig] pre-PEP : Synthesis of previous threads, and irc talks + proposals In-Reply-To: <20081007184113.54CF03A4045@sparrow.telecommunity.com> References: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.com> <8E57AB2B-4C1C-49F4-9FF8-78F57509395E@zooko.com> <94bdd2610810070707k5fee06eayd2d1086da9459213@mail.gmail.com> <20081007184113.54CF03A4045@sparrow.telecommunity.com> Message-ID: <94bdd2610810071158u624007eft4593b253f1887be0@mail.gmail.com> On Tue, Oct 7, 2008 at 2:42 PM, Phillip J. Eby wrote: > At 10:07 AM 10/7/2008 -0400, Tarek Ziad? wrote: >> >> The -m feature of setuptools is nice, but it activates one version at >> a time, and >> this is globlal to Python unless each application is handling the >> version switch, >> wich is pretty heavy. > > With or without the -m switch, scripts installed by setuptools will find the > version they are specified to use, without the user needing to do anything. > So, you can have a default version of an egg (used by the interpreter and > non-setuptools scripts), and then some non-default versions that are used by > scripts. > > zc.buildout and virtualenv also have their own ways of accomplishing the > same thing, e.g., by hardcoding paths in an installed script. in a plain python setup, If foo 1.2 is the default, and a package wants use foo 1.4, it needs to specifically call pkg_resources.require() in the code, to activate it in sys.path before importing "foo" in the code. Since each package can list with setuptools its dependencies with versions in install_requires, how hard would it be to automatically call the right "require()" calls when the package is used ? Tarek -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From ianb at colorstudy.com Tue Oct 7 21:07:35 2008 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 07 Oct 2008 14:07:35 -0500 Subject: [Distutils] [Catalog-sig] pre-PEP : Synthesis of previous threads, and irc talks + proposals In-Reply-To: <94bdd2610810071158u624007eft4593b253f1887be0@mail.gmail.com> References: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.com> <8E57AB2B-4C1C-49F4-9FF8-78F57509395E@zooko.com> <94bdd2610810070707k5fee06eayd2d1086da9459213@mail.gmail.com> <20081007184113.54CF03A4045@sparrow.telecommunity.com> <94bdd2610810071158u624007eft4593b253f1887be0@mail.gmail.com> Message-ID: <48EBB377.7030800@colorstudy.com> Tarek Ziad? wrote: > On Tue, Oct 7, 2008 at 2:42 PM, Phillip J. Eby wrote: >> At 10:07 AM 10/7/2008 -0400, Tarek Ziad? wrote: >>> The -m feature of setuptools is nice, but it activates one version at >>> a time, and >>> this is globlal to Python unless each application is handling the >>> version switch, >>> wich is pretty heavy. >> With or without the -m switch, scripts installed by setuptools will find the >> version they are specified to use, without the user needing to do anything. >> So, you can have a default version of an egg (used by the interpreter and >> non-setuptools scripts), and then some non-default versions that are used by >> scripts. >> >> zc.buildout and virtualenv also have their own ways of accomplishing the >> same thing, e.g., by hardcoding paths in an installed script. > > in a plain python setup, > > If foo 1.2 is the default, and a package wants use foo 1.4, > it needs to specifically call pkg_resources.require() in the code, to > activate it in sys.path > before importing "foo" in the code. > > Since each package can list with setuptools its dependencies with > versions in install_requires, > how hard would it be to automatically call the right "require()" > calls when the package is used ? require() is recursive, so as long as the original script is explicitly loaded (e.g., from a binary script, or something that loads eggs/entry points) then the proper versions will be loaded. Though as far as I know, pkg_resources won't remove other versions of the egg from the path, so it only works if there are no active versions of the eggs. Which isn't how many people install packages, so this feature of require() doesn't get used for much of anything (at least that I've seen). I'll also note that the require in setuptools-generated scripts causes pretty frequent problems for people, all to support this multi-version feature that no one really uses. An example of an easy way to cause the problem, if you do: "python setup.py develop; svn up; python setup.py egg_info" it'll break any scripts, or if you install a script in an unusual location, or use $PYTHONPATH but don't set $PATH so that you get an unexpected script that doesn't match your libraries -- since pyinstall is using --single-version-externally-managed, I kind of wish I could easily turn off the require() as well (I could monkeypatch setuptools to remove it, but I've been burned by going down that path before). -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From pje at telecommunity.com Tue Oct 7 22:27:40 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 07 Oct 2008 16:27:40 -0400 Subject: [Distutils] [Catalog-sig] pre-PEP : Synthesis of previous threads, and irc talks + proposals In-Reply-To: <94bdd2610810071158u624007eft4593b253f1887be0@mail.gmail.co m> References: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.com> <8E57AB2B-4C1C-49F4-9FF8-78F57509395E@zooko.com> <94bdd2610810070707k5fee06eayd2d1086da9459213@mail.gmail.com> <20081007184113.54CF03A4045@sparrow.telecommunity.com> <94bdd2610810071158u624007eft4593b253f1887be0@mail.gmail.com> Message-ID: <20081007202630.266D23A4045@sparrow.telecommunity.com> At 02:58 PM 10/7/2008 -0400, Tarek Ziad? wrote: >On Tue, Oct 7, 2008 at 2:42 PM, Phillip J. Eby wrote: > > At 10:07 AM 10/7/2008 -0400, Tarek Ziad? wrote: > >> > >> The -m feature of setuptools is nice, but it activates one version at > >> a time, and > >> this is globlal to Python unless each application is handling the > >> version switch, > >> wich is pretty heavy. > > > > With or without the -m switch, scripts installed by setuptools > will find the > > version they are specified to use, without the user needing to do anything. > > So, you can have a default version of an egg (used by the interpreter and > > non-setuptools scripts), and then some non-default versions that > are used by > > scripts. > > > > zc.buildout and virtualenv also have their own ways of accomplishing the > > same thing, e.g., by hardcoding paths in an installed script. > >in a plain python setup, > >If foo 1.2 is the default, and a package wants use foo 1.4, >it needs to specifically call pkg_resources.require() in the code, to >activate it in sys.path >before importing "foo" in the code. You can't un-default the default, actually. If there's a default, it can't be replaced once pkg_resources has been imported. >Since each package can list with setuptools its dependencies with >versions in install_requires, >how hard would it be to automatically call the right "require()" >calls when the package is used ? This is already done by setuptools-generated scripts. Same for zc.buildout and virtualenv, they just do it differently. From pje at telecommunity.com Tue Oct 7 22:51:57 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 07 Oct 2008 16:51:57 -0400 Subject: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification In-Reply-To: <1223405931.5063.190.camel@tomoyo> References: <20081003163402.638A93A407A@sparrow.telecommunity.com> <20081006212511.GC17624@laurie.devork> <20081007013212.02F773A4045@sparrow.telecommunity.com> <1223366277.5063.18.camel@tomoyo> <20081007183853.8E7243A4045@sparrow.telecommunity.com> <1223405931.5063.190.camel@tomoyo> Message-ID: <20081007205047.805C03A4045@sparrow.telecommunity.com> At 08:58 PM 10/7/2008 +0200, Josselin Mouette wrote: >Le mardi 07 octobre 2008 ? 14:40 -0400, Phillip J. Eby a ??crit : > > At 09:57 AM 10/7/2008 +0200, Josselin Mouette wrote: > > >Symlinks are a real pain to handle. We can use them transparently > > >for .pyc files, but if we want to relocate data files to some other > > >directories, currently it has to be done by hand, and this is why most > > >maintainers don????t do it. >. > > > > Which is why the idea for the BUILDS spec to include a way for > > automated tools to do it, so that you won't have to do it manually. > >If you write a tool to do that, why not make it simply move files >properly and let the code locate them, instead of adding yet another >layer on top of the existing stuff? The tool will not be more >complicated this way. I'm not sure I follow you. What I'm proposing is that: 1. A BUILDS "manifest" for a project should list the files included in the project by what "kind" of file they are, as well as by inter-file relationships (e.g., "these 3 files are part of Python package 'foo.bar' within this project, and they are static data used at runtime but not read or written"). 2. BUILDS-compatible install tools will be free to install those files wherever they like (or create .debs, .rpms, etc. that do), and use symlinks so that the installed project can find them in the places that the developer wanted to find them. Is this what you're asking for? Because it's what I've been trying to say we should do. > > Or are you saying that that isn't an improvement over the current > situation? > >It is. I didn???t understand you wanted to automate the symlinks creation. >It indeed means less burden on the maintainers, but it would be a shame >to keep the same mess in binary packages, while having a tool to do that >would allow to make things cleaner. I'm not sure I understand you here, either. I'm saying that a BUILDS-compatible install tools for Debian and RPM systems should create system packages with the stuff wherever the OS people want it to go, *and* symlinks so the Python code can find it. And that this should be an automated process, to the extent possible/practical. For example, if Debian needs metadata that's not in the core BUILDS metadata, it should be possible within BUILDS to include it, so that it can be contributed upstream. Likewise, it should be possible for a BUILDS install tool to use a third-party supplement to get that data, so that when you're initially porting a project, you could write a file with the additional data while doing the initial port, then submit that file upstream. > > Yes, it's true that I'm saying that developers should not be > > *required* to add the extra data to their packages, but that they > > should be *able* to, and if it is trivial to add the extra data, most > > should accept patches or respond to requests to do so. > > > > Right now, it's not even *possible* for them to do so, however. > >It is possible - if they use autoconf instead. Sure, but who's going to convince them to do *that*? ;-) Less flippantly, our goal here is to have something that has a better cost/benefit ratio for adoption by a Python developer who has not seen any pressing need for autoconf in their life to date. And if somebody wants to make a BUILDS tool to generate autoconf sources from a project manifest, then great! >BTW, I would consider it a good approach to try bringing BUILDS on par >with the autotools capabilities. These tools have serious drawbacks, but >they were written with the ability to work for distributors in mind. Sure. I like stealing designs and requirements definition from mature projects, even if you might argue that I chose poor role models for setuptools. (Java and OSGi.) However, since autotools compatibility isn't a goal in itself, I think I'll leave it to those people who have interest/experience to propose what specific autotools features they'd like to see in BUILDS, and the use cases they have for them. If I were to just skim the docs trying to figure that out, I might think that something cool in it is actually useful, or vice versa. ;-) So, it's better here I think to hear from experienced users such as yourself. From p.f.moore at gmail.com Tue Oct 7 23:16:53 2008 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 7 Oct 2008 22:16:53 +0100 Subject: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification In-Reply-To: <20081007182934.40C923A4045@sparrow.telecommunity.com> References: <20081003163402.638A93A407A@sparrow.telecommunity.com> <20081006212511.GC17624@laurie.devork> <20081007013212.02F773A4045@sparrow.telecommunity.com> <79990c6b0810070204r4b2fb216la1daa69e36ba78f0@mail.gmail.com> <20081007182934.40C923A4045@sparrow.telecommunity.com> Message-ID: <79990c6b0810071416m3f841e27ve4d2880e4ccc810d@mail.gmail.com> 2008/10/7 Phillip J. Eby : > At 10:04 AM 10/7/2008 +0100, Paul Moore wrote: >> >> 2008/10/7 Phillip J. Eby : >> > You can see that this is also what I did in the design of easy_install >> > and >> > setuptools, except that in that effort I only considered developers and >> > users, not system packagers. >> >> I'd argue (you may differ) that the most significant area where you >> missed the mark on user benefits with easy_install and setuptools is >> the lack of easy *uninstall* and easy *list* options. > > Well, I'd certainly agree that those features are desirable. But I didn't > "miss the mark", in that those features were not part of my mark. ;-) The > goal was to get widespread adoption by developers, and thus the primary > target audience was developers. Any features that attracted users were > there only insofar as the benefit to users would specifically drive adoption > by developers. OK, I was assuming that when you said "I only considered developers and users", you were implying that you were aiming to provide for users benefits (and hence, the user benefit of package management - uninstall and list - was important). As you say, though, your key aim was to drive adoption, so making *installation* easy for users is important, but making *uninstall* easy isn't, as it's not something related to adoption - rather the opposite :-) Of course, as a user, my perspectives are different :-) And as a user who is increasingly resistant to setuptools-enabled packages precisely because they don't address user-level issues, I suspect that catering for user level requirements (in addition to system packagers', and the already covered developers) is more relevant this time round. My feeling, by the way, is that "system packagers" are the more relevant group on Linux/Unix (where most users install Python modules via system packages, or else they are developers) whereas on Windows, users are the more relevant group (as there isn't really a significant system packager role in the first place). Whether this, coupled with the notoriously difficult task of getting Windows and Linux/Unix users to understand each other's perspectives, is an important fact, only time will tell. Thanks for taking the time to clarify for me. Paul. From ben+python at benfinney.id.au Tue Oct 7 23:44:35 2008 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 08 Oct 2008 08:44:35 +1100 Subject: [Distutils] Twisted packaging in Debian References: <20081007154406.29191.1974198569.divmod.quotient.35156@ohm> <1223396817.5063.161.camel@tomoyo> Message-ID: <87ej2sjd18.fsf@benfinney.id.au> Josselin Mouette writes: > About this issue, I still think you need to version your plugins > directory if the plugins are not compatible across versions. You've referred to this a few times now without explaining exactly what you expect the developer to do. So, just in case: I presume you mean that, in the case where a program's plugins are not compatible with plugins for previous versions of the same program, the developer should always seek (and attempt to install) plugins for version N in a differently-named directory from plugins for version M and version P. That is, that a directory expected to contain plugins for version N should contain *only* plugins for version N, and never for other versions. Yes? -- \ ?Pinky, are you pondering what I'm pondering?? ?I think so, | `\ Brain, but isn't that why they invented tube socks?? ?_Pinky | _o__) and The Brain_ | Ben Finney From ben+python at benfinney.id.au Tue Oct 7 23:47:40 2008 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 08 Oct 2008 08:47:40 +1100 Subject: [Distutils] Simultaneous multi-version support (was: [Catalog-sig] pre-PEP : Synthesis of previous threads, and irc talks + proposals) References: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.com> <8E57AB2B-4C1C-49F4-9FF8-78F57509395E@zooko.com> <94bdd2610810070707k5fee06eayd2d1086da9459213@mail.gmail.com> <20081007184113.54CF03A4045@sparrow.telecommunity.com> <94bdd2610810071158u624007eft4593b253f1887be0@mail.gmail.com> <48EBB377.7030800@colorstudy.com> Message-ID: <87abdgjcw3.fsf_-_@benfinney.id.au> Ian Bicking writes: > I'll also note that the require in setuptools-generated scripts causes > pretty frequent problems for people, all to support this multi-version > feature that no one really uses. I agree heartily that it seems to cause more trouble than it's worth ? for my assessment of its worth, anyway. Is it true that ?no-one? (to some epsilon value) actually uses this feature? -- \ ?The best ad-libs are rehearsed.? ?Graham Kennedy | `\ | _o__) | Ben Finney From robert.kern at gmail.com Tue Oct 7 23:58:42 2008 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 07 Oct 2008 16:58:42 -0500 Subject: [Distutils] Simultaneous multi-version support In-Reply-To: <87abdgjcw3.fsf_-_@benfinney.id.au> References: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.com> <8E57AB2B-4C1C-49F4-9FF8-78F57509395E@zooko.com> <94bdd2610810070707k5fee06eayd2d1086da9459213@mail.gmail.com> <20081007184113.54CF03A4045@sparrow.telecommunity.com> <94bdd2610810071158u624007eft4593b253f1887be0@mail.gmail.com> <48EBB377.7030800@colorstudy.com> <87abdgjcw3.fsf_-_@benfinney.id.au> Message-ID: Ben Finney wrote: > Ian Bicking writes: > >> I'll also note that the require in setuptools-generated scripts causes >> pretty frequent problems for people, all to support this multi-version >> feature that no one really uses. > > I agree heartily that it seems to cause more trouble than it's worth ? > for my assessment of its worth, anyway. Is it true that ?no-one? (to > some epsilon value) actually uses this feature? There is one person on enthought-dev who does this for everything. He says it keeps him honest about his dependencies. And consequently keeps us at Enthought honest about ours. I typically have multiple versions of things installed, and switch between them, but I never use pkg_resources.require() to do so. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From joss at debian.org Wed Oct 8 00:03:16 2008 From: joss at debian.org (Josselin Mouette) Date: Wed, 08 Oct 2008 00:03:16 +0200 Subject: [Distutils] Twisted packaging in Debian In-Reply-To: <87ej2sjd18.fsf@benfinney.id.au> References: <20081007154406.29191.1974198569.divmod.quotient.35156@ohm> <1223396817.5063.161.camel@tomoyo> <87ej2sjd18.fsf@benfinney.id.au> Message-ID: <1223416996.5063.193.camel@tomoyo> Le mercredi 08 octobre 2008 ? 08:44 +1100, Ben Finney a ?crit : > Josselin Mouette writes: > > > About this issue, I still think you need to version your plugins > > directory if the plugins are not compatible across versions. > > You've referred to this a few times now without explaining exactly > what you expect the developer to do. So, just in case: > > I presume you mean that, in the case where a program's plugins are not > compatible with plugins for previous versions of the same program, the > developer should always seek (and attempt to install) plugins for > version N in a differently-named directory from plugins for version M > and version P. > > That is, that a directory expected to contain plugins for version N > should contain *only* plugins for version N, and never for other > versions. Yes. Sorry for omitting this detailed explanation. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From joss at debian.org Wed Oct 8 00:43:39 2008 From: joss at debian.org (Josselin Mouette) Date: Wed, 08 Oct 2008 00:43:39 +0200 Subject: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification In-Reply-To: <20081007205047.805C03A4045@sparrow.telecommunity.com> References: <20081003163402.638A93A407A@sparrow.telecommunity.com> <20081006212511.GC17624@laurie.devork> <20081007013212.02F773A4045@sparrow.telecommunity.com> <1223366277.5063.18.camel@tomoyo> <20081007183853.8E7243A4045@sparrow.telecommunity.com> <1223405931.5063.190.camel@tomoyo> <20081007205047.805C03A4045@sparrow.telecommunity.com> Message-ID: <1223419419.5063.231.camel@tomoyo> Le mardi 07 octobre 2008 ? 16:51 -0400, Phillip J. Eby a ?crit : > >If you write a tool to do that, why not make it simply move files > >properly and let the code locate them, instead of adding yet another > >layer on top of the existing stuff? The tool will not be more > >complicated this way. > > I'm not sure I follow you. What I'm proposing is that: > > 1. A BUILDS "manifest" for a project should list the files included > in the project by what "kind" of file they are, as well as by > inter-file relationships (e.g., "these 3 files are part of Python > package 'foo.bar' within this project, and they are static data used > at runtime but not read or written"). > > 2. BUILDS-compatible install tools will be free to install those > files wherever they like (or create .debs, .rpms, etc. that do), and > use symlinks so that the installed project can find them in the > places that the developer wanted to find them. > > Is this what you're asking for? Because it's what I've been trying > to say we should do. What I am afraid of is that, by adding just another layer, you will introduce new problems while fixing existing ones. Currently we already have a hard time maintaining a symlink farm, and adding a second symlink farm on top of it is not going to make things more understandable (currently, package maintainers already have a hard time understanding how to deal with Python modules) nor more reliable. The first kind of issues I can think of is that dpkg handles symlinks to directories very badly, so you?re likely to run into issues that have to be dealt with by hand by maintainers. I haven?t thought about it much, but I have the feeling that other things will break. In the end, if you are designing a new packaging specification and a new tool, I think you?d better take this as an occasion to do the right thing instead of adding this new layer. > For example, if Debian needs metadata that's not in the core BUILDS > metadata, it should be possible within BUILDS to include it, so that > it can be contributed upstream. Likewise, it should be possible for > a BUILDS install tool to use a third-party supplement to get that > data, so that when you're initially porting a project, you could > write a file with the additional data while doing the initial port, > then submit that file upstream. If there is a need for Debian-specific metadata, it will go in the debian/ directory. It looks interesting to integrate it upstream only if it is relevant for other distributors. > Less flippantly, our goal here is to have something that has a better > cost/benefit ratio for adoption by a Python developer who has not > seen any pressing need for autoconf in their life to date. And if > somebody wants to make a BUILDS tool to generate autoconf sources > from a project manifest, then great! No, no. Please don?t :) But if you can express in the project manifest the same flexibility as with autoconf (or even the tenth of it, we don?t need *that* much flexibility), that should be enough. > >BTW, I would consider it a good approach to try bringing BUILDS on par > >with the autotools capabilities. These tools have serious drawbacks, but > >they were written with the ability to work for distributors in mind. > > Sure. I like stealing designs and requirements definition from > mature projects, even if you might argue that I chose poor role > models for setuptools. (Java and OSGi.) > > However, since autotools compatibility isn't a goal in itself, I > think I'll leave it to those people who have interest/experience to > propose what specific autotools features they'd like to see in > BUILDS, and the use cases they have for them. If I were to just skim > the docs trying to figure that out, I might think that something cool > in it is actually useful, or vice versa. ;-) So, it's better here I > think to hear from experienced users such as yourself. There is no use in being compatible with the autotools, the point is about stealing good ideas. The first thing that comes to mind, and that we already discussed at large, is the ability to specify at install time where data, documentation, manual pages and so on have to go. Another thing that I consider a good design (when expunged of the useless stuff in it) is the config.h file. It is a file that is generated at configure time and which contains macros you may need inside your code: package version, installation directories, enabled features, etc. What some Python projects using autoconf are doing, and I?d like to see in BUILDS, is the generation of a config.py file at installation time. It would be useful to specify install-time stuff like installation directories, optional features and installation parameters. For example it could contain: version = "1.2.3" builddate = "Tue 7 Oct 2008, 23:12 +0100" templates_dir = "/usr/share/foobar/templates" has_libamazing = True # _foobar.so is built with libamazing support page_footer = "Powered by foobar" For Python-only packages, you can easily ship a default config.py file that works without installation: version = get_version_from_changelog () templates_dir = os.path.join(os.path.basedir(__file__),"templates") page_footer = "Foobar (development version)" By doing such a thing, you are guaranteed to obtain easily the data you need from within the Python application: import config template = file(os.path.join(config.templates_dir), "foo.template") I may be biased because I?m used to work like this, but I find this easier for both the developer and the distributor. In my personal experience, it is also more reliable. Another thing from the autotools that I?d like to see is seamless integration with pkg-config, for modules with C extensions. Bonus points for being able to use not only compilation flags and installation paths, but also macros you need to use inside the code (they could go in config.py). I?ll tell you if I can think of other things. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From pje at telecommunity.com Wed Oct 8 01:25:10 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 07 Oct 2008 19:25:10 -0400 Subject: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification In-Reply-To: <1223419419.5063.231.camel@tomoyo> References: <20081003163402.638A93A407A@sparrow.telecommunity.com> <20081006212511.GC17624@laurie.devork> <20081007013212.02F773A4045@sparrow.telecommunity.com> <1223366277.5063.18.camel@tomoyo> <20081007183853.8E7243A4045@sparrow.telecommunity.com> <1223405931.5063.190.camel@tomoyo> <20081007205047.805C03A4045@sparrow.telecommunity.com> <1223419419.5063.231.camel@tomoyo> Message-ID: <20081007232358.0A1153A4045@sparrow.telecommunity.com> At 12:43 AM 10/8/2008 +0200, Josselin Mouette wrote: >Le mardi 07 octobre 2008 ? 16:51 -0400, Phillip J. Eby a ??crit : > > >If you write a tool to do that, why not make it simply move files > > >properly and let the code locate them, instead of adding yet another > > >layer on top of the existing stuff? The tool will not be more > > >complicated this way. > > > > I'm not sure I follow you. What I'm proposing is that: > > > > 1. A BUILDS "manifest" for a project should list the files included > > in the project by what "kind" of file they are, as well as by > > inter-file relationships (e.g., "these 3 files are part of Python > > package 'foo.bar' within this project, and they are static data used > > at runtime but not read or written"). > > > > 2. BUILDS-compatible install tools will be free to install those > > files wherever they like (or create .debs, .rpms, etc. that do), and > > use symlinks so that the installed project can find them in the > > places that the developer wanted to find them. > > > > Is this what you're asking for? Because it's what I've been trying > > to say we should do. > >What I am afraid of is that, by adding just another layer, you will >introduce new problems while fixing existing ones. Currently we already >have a hard time maintaining a symlink farm, and adding a second symlink >farm on top of it is not going to make things more understandable >(currently, package maintainers already have a hard time understanding >how to deal with Python modules) nor more reliable. I guess I'm confused a bit here, since the idea is not to have maintainers need to understand anything but how to be able to tell upstream, "hey, you didn't configure file X correctly". At least, that's what I'd hope would be all that's needed. :-) >The first kind of issues I can think of is that dpkg handles symlinks to >directories very badly, so you???re likely to run into issues that have to >be dealt with by hand by maintainers. I haven???t thought about it much, >but I have the feeling that other things will break. Actually, until I saw the Twisted plugins directory use case, I hadn't actually seen any use cases for symlinks to directories that couldn't be handled by using a directory of symlinks instead. Are there any other Python packages you know of that would need a symlink to a directory? >In the end, if you are designing a new packaging specification and a new >tool, The BUILDS manifest is neither packaging nor a tool; it's just information about a bunch of files. ;-) We don't want to design *a* tool, we want to make it possible for tools to be written. Currently, there are a whole bunch of installation and maintenance tools out there built on top of setuptools, but they aren't necessarily very compatible with each other, and building tools that approach things very differently from setuptools is hard. The idea of a specification is to allow installation tools to be built that apply different policies on different platforms, not to build The One True Tool[tm]. In other words, the goal isn't to "do the right thing", but to allow various useful things to be done. > > For example, if Debian needs metadata that's not in the core BUILDS > > metadata, it should be possible within BUILDS to include it, so that > > it can be contributed upstream. Likewise, it should be possible for > > a BUILDS install tool to use a third-party supplement to get that > > data, so that when you're initially porting a project, you could > > write a file with the additional data while doing the initial port, > > then submit that file upstream. > >If there is a need for Debian-specific metadata, it will go in the >debian/ directory. It looks interesting to integrate it upstream only if >it is relevant for other distributors. If I have to explicitly mark my locale files for Debian to be happy, then that *is* Debian-specific metadata from my perspective as a cross-platform developer. That is, I didn't say anything about metadata that was unique to Debian, just metadata needed by a Debian BUILDS installer to be happy, as opposed to a generic BUILDS installer that doesn't care about the FHS. (For example, an Enthought installer for Windows that's also installing its own Python interpreter might want to consume BUILDS but it sure as heck wouldn't be following FHS standards; it'd have its own.) >There is no use in being compatible with the autotools, the point is >about stealing good ideas. > >The first thing that comes to mind, and that we already discussed at >large, is the ability to specify at install time where data, >documentation, manual pages and so on have to go. The idea of the BUILDS "install locations" (IL) manifest is just that it lists the contents of the project distribution, and describes the logical locations that things go in. So, part of the BUILDS spec would be what categories exist for the above. It would then be the responsibility of people building BUILDS-consuming install tools to define the actual locations. I say it that way mainly because I'm not volunteering my time to actually build such an install tool, nor am I necessarily volunteering to build *any* tools. No sense in replacing me as the setuptools bottleneck to be the bottleneck on new tools. The whole idea is to get system packagers and major developers together to agree on how to communicate installation requirements to installation tools, and then have everybody go back and update their tools to generate or consume installation data in accordance with the spec. Of course, it's likely that rather than everybody creating their own install tools, there will probably be some grouping and alliances along platform and/or use-case lines. For the stdlib, I expect there will be a simple installer with configurable locations for installing things, maybe with some ability to be extended to handle more complex policies. There would probably also need to be BUILDS-to-wininst, and BUILDS-to-msi converters. >What some Python projects using autoconf are doing, and I???d like to see >in BUILDS, is the generation of a config.py file at installation time. >It would be useful to specify install-time stuff like installation >directories, optional features and installation parameters. > >For example it could contain: > version = "1.2.3" > builddate = "Tue 7 Oct 2008, 23:12 +0100" > templates_dir = "/usr/share/foobar/templates" > has_libamazing = True # _foobar.so is built with libamazing support > page_footer = "Powered by foobar" > >For Python-only packages, you can easily ship a default config.py file >that works without installation: > version = get_version_from_changelog () > templates_dir = > os.path.join(os.path.basedir(__file__),"templates") > page_footer = "Foobar (development version)" > >By doing such a thing, you are guaranteed to obtain easily the data you >need from within the Python application: > import config > template = file(os.path.join(config.templates_dir), > "foo.template") > >I may be biased because I???m used to work like this, but I find this >easier for both the developer and the distributor. In my personal >experience, it is also more reliable. It looks like a great idea, and I'm all for recommending people use it, or something like it. I'm also fine with BUILDS having support for some sort of "install-time editing" like this. I just don't want it to be a requirement that everybody change their ways. Not everybody even uses pkg_resources' API for accessing data files yet, after all. In my experience with setuptools adoption, anything that makes you have to change something outside of the packaging part is a significant barrier. People wonder if they can rely on it, whether it will lock them in to something, they need to do testing, etc. Sure, these things aren't REALLY that big of a hassle, but the psychological effect is to create hesitation. But if you can move your code without changing your code, it creates a sense of safety, and provides incremental rewards for incremental efforts. Thus, people who first moved to setuptools without needing to change their actual package code, were then much more likely to adopt the API later. The same will be true for this. >Another thing from the autotools that I???d like to see is seamless >integration with pkg-config, for modules with C extensions. Bonus points >for being able to use not only compilation flags and installation paths, >but also macros you need to use inside the code (they could go in >config.py). Sorry, what is pkg-config? From exarkun at divmod.com Wed Oct 8 03:59:46 2008 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Tue, 7 Oct 2008 21:59:46 -0400 Subject: [Distutils] Twisted packaging in Debian In-Reply-To: <1223416996.5063.193.camel@tomoyo> Message-ID: <20081008015946.29191.566446680.divmod.quotient.35393@ohm> On Wed, 08 Oct 2008 00:03:16 +0200, Josselin Mouette wrote: >Le mercredi 08 octobre 2008 ? 08:44 +1100, Ben Finney a ?crit : >> Josselin Mouette writes: >> >> > About this issue, I still think you need to version your plugins >> > directory if the plugins are not compatible across versions. >> >> You've referred to this a few times now without explaining exactly >> what you expect the developer to do. So, just in case: >> >> I presume you mean that, in the case where a program's plugins are not >> compatible with plugins for previous versions of the same program, the >> developer should always seek (and attempt to install) plugins for >> version N in a differently-named directory from plugins for version M >> and version P. >> >> That is, that a directory expected to contain plugins for version N >> should contain *only* plugins for version N, and never for other >> versions. > >Yes. Sorry for omitting this detailed explanation. What if we consider the scenario without two different versions? The plugin duplication issue happens if you have a site-wide installation of Twisted X.Y and an extra (eg, per-user) copy of Twisted X.Y, the same version, explicitly in PYTHONPATH. Versioning the plugin directory wouldn't help here, because both copies would appear to be the right version for whichever instance of the install gets used. It might be possible to avoid duplication by inspecting the names carefully and avoiding loading anything that looks like it was already loaded. But I can still think of problem cases - the site-wide installation has plugins X, Y, and Z and my personal installation has plugins A, B, and C, and I use my personal installation with the express intent of not having X, Y, and Z made available. If the site-wide plugins directory is considered, then I get all 6 - not what I wanted. Jean-Paul From david at ar.media.kyoto-u.ac.jp Wed Oct 8 04:45:39 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 08 Oct 2008 11:45:39 +0900 Subject: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification In-Reply-To: <20081007232358.0A1153A4045@sparrow.telecommunity.com> References: <20081003163402.638A93A407A@sparrow.telecommunity.com> <20081006212511.GC17624@laurie.devork> <20081007013212.02F773A4045@sparrow.telecommunity.com> <1223366277.5063.18.camel@tomoyo> <20081007183853.8E7243A4045@sparrow.telecommunity.com> <1223405931.5063.190.camel@tomoyo> <20081007205047.805C03A4045@sparrow.telecommunity.com> <1223419419.5063.231.camel@tomoyo> <20081007232358.0A1153A4045@sparrow.telecommunity.com> Message-ID: <48EC1ED3.7020001@ar.media.kyoto-u.ac.jp> Phillip J. Eby wrote: > > Sorry, what is pkg-config? http://pkg-config.freedesktop.org/wiki/ David From david at ar.media.kyoto-u.ac.jp Wed Oct 8 04:58:20 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 08 Oct 2008 11:58:20 +0900 Subject: [Distutils] pre-PEP : Synthesis of previous threads, and irc talks + proposals In-Reply-To: <94bdd2610810070707k5fee06eayd2d1086da9459213@mail.gmail.com> References: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.com> <8E57AB2B-4C1C-49F4-9FF8-78F57509395E@zooko.com> <94bdd2610810070707k5fee06eayd2d1086da9459213@mail.gmail.com> Message-ID: <48EC21CC.3080709@ar.media.kyoto-u.ac.jp> Tarek Ziad? wrote: > On Sun, Oct 5, 2008 at 12:04 PM, zooko wrote: >>> 5/ ideally, they should be one and only one version of a given package >>> in an OS-based installation >> -1 -- This is the strong preference of the folks who package software for >> OSes -- Debian, Fedora, etc. -- but it is not necessarily the choice of the >> users who use their OSes. It is best for the Python packaging standards to >> be agnostic towards this, or at least to support both this desideratum and >> its opposite. It is not just a strong preference of the Linux guys, it is proper software engineering. It is fine for developers to have several versions installed at the same time, but we should really discourage developers to depend on the possibility to deploy several versions of the same package, at least in the site-packages owned by the system. The problem is not just for linux guys: by enabling everyone to deploy several versions, you encourage people not to care about API compatibility, and then quikcly, in the dependencies, you will have depends on foo >= 1.2 (foo < 1.3), which quickly means it will fail because a package A depends on B and C, and B depends on foo 1.2 and C 1.3. By encouraging multiple versions, you are encouraging this kind of failures all the time. If people want to try several versions, there is virtual env and co (or just installing several python interpreters). It should be a *developer only* convenience as much as possible. We can have a system to control imported versions, but without support from python interpreter, it will be unreliable, and keeps breaking as it does now. cheers, David From joss at debian.org Wed Oct 8 11:41:38 2008 From: joss at debian.org (Josselin Mouette) Date: Wed, 08 Oct 2008 11:41:38 +0200 Subject: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification In-Reply-To: <20081007232358.0A1153A4045@sparrow.telecommunity.com> References: <20081003163402.638A93A407A@sparrow.telecommunity.com> <20081006212511.GC17624@laurie.devork> <20081007013212.02F773A4045@sparrow.telecommunity.com> <1223366277.5063.18.camel@tomoyo> <20081007183853.8E7243A4045@sparrow.telecommunity.com> <1223405931.5063.190.camel@tomoyo> <20081007205047.805C03A4045@sparrow.telecommunity.com> <1223419419.5063.231.camel@tomoyo> <20081007232358.0A1153A4045@sparrow.telecommunity.com> Message-ID: <1223458898.4207.17.camel@shizuru> Le mardi 07 octobre 2008 ? 19:25 -0400, Phillip J. Eby a ?crit : > At 12:43 AM 10/8/2008 +0200, Josselin Mouette wrote: > >What I am afraid of is that, by adding just another layer, you will > >introduce new problems while fixing existing ones. Currently we already > >have a hard time maintaining a symlink farm, and adding a second symlink > >farm on top of it is not going to make things more understandable > >(currently, package maintainers already have a hard time understanding > >how to deal with Python modules) nor more reliable. > > I guess I'm confused a bit here, since the idea is not to have > maintainers need to understand anything but how to be able to tell > upstream, "hey, you didn't configure file X correctly". At least, > that's what I'd hope would be all that's needed. :-) First, it is wrong to expect maintainers to not understand what they do. It is a guaranteed recipe for failure. Second, it is not realistic to assume that all the maintainer will do is forward bugs to the upstream developers, and that upstream developers will fix it in the minute ? especially for packaging issues, which developers generally disregard. The maintainer often needs to patch the sources or the installation scripts so that the software installs properly, and to do so he needs to understand what he does. > >The first kind of issues I can think of is that dpkg handles symlinks to > >directories very badly, so you???re likely to run into issues that have to > >be dealt with by hand by maintainers. I haven???t thought about it much, > >but I have the feeling that other things will break. > > Actually, until I saw the Twisted plugins directory use case, I > hadn't actually seen any use cases for symlinks to directories that > couldn't be handled by using a directory of symlinks instead. Are > there any other Python packages you know of that would need a symlink > to a directory? Yes, you can arrange for not meeting this specific issue. What I?m saying is not that such issues don?t have workarounds, I?m saying that we will discover new ones, again and again, as long as the only thing we do to fix each issue is add another carpet to hide the dust. > The idea of the BUILDS "install locations" (IL) manifest is just that > it lists the contents of the project distribution, and describes the > logical locations that things go in. So, part of the BUILDS spec > would be what categories exist for the above. If the only thing it does is allowing to move files and symlink them, I frankly doubt the benefits will outperform the cost of maintaining this new symlink farm. > >What some Python projects using autoconf are doing, and I???d like to > see > >in BUILDS, is the generation of a config.py file at installation time. > >It would be useful to specify install-time stuff like installation > >directories, optional features and installation parameters. > > It looks like a great idea, and I'm all for recommending people use > it, or something like it. I'm also fine with BUILDS having support > for some sort of "install-time editing" like this. I just don't want > it to be a requirement that everybody change their ways. Not > everybody even uses pkg_resources' API for accessing data files yet, after all. You don?t have to make it a requirement. If you agree with the idea of such a feature, a good approach might be to: * always generate this file; * provide the possibility to easily install it; * let developers gradually start using it directly in the code. > >Another thing from the autotools that I???d like to see is seamless > >integration with pkg-config, for modules with C extensions. Bonus points > >for being able to use not only compilation flags and installation paths, > >but also macros you need to use inside the code (they could go in > >config.py). > > Sorry, what is pkg-config? pkg-config is becoming the de-facto standard for providing information about installed development libraries. With it you can easily: * obtain compile-time parameters for dependent programs (CFLAGS and LIBS); * require a range of versions for a number of libraries you depend on; * obtain any parameter that the developer wanted you to know (for example, the name of a dbus service you need to start, the path to a helper utility, or the installation path for plugins); * with the automake integration, enable programs or features based on the availability of dependencies. For a Python example: you want to know where to install your definition files for GNOME Python bindings? Just use: pkg-config --variable=codegendir pygtk-2.0 Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From zooko at zooko.com Wed Oct 8 15:15:08 2008 From: zooko at zooko.com (zooko) Date: Wed, 8 Oct 2008 07:15:08 -0600 Subject: [Distutils] Simultaneous multi-version support In-Reply-To: References: <94bdd2610810010510w43abf97bkf19f0324520e3dfe@mail.gmail.com> <8E57AB2B-4C1C-49F4-9FF8-78F57509395E@zooko.com> <94bdd2610810070707k5fee06eayd2d1086da9459213@mail.gmail.com> <20081007184113.54CF03A4045@sparrow.telecommunity.com> <94bdd2610810071158u624007eft4593b253f1887be0@mail.gmail.com> <48EBB377.7030800@colorstudy.com> <87abdgjcw3.fsf_-_@benfinney.id.au> Message-ID: <4CA944A8-FD07-4157-BB2E-4F0D5D674066@zooko.com> We use pkg_resources.require() in Tahoe solely in order to get better and earlier error messages in the case of missing or wrong-version dependencies: http://allmydata.org/trac/tahoe/browser/_auto_deps.py?rev=2968#L22 """ The purpose of this function is to raise a pkg_resources exception if any of the requirements can't be imported. This is just to give earlier and more explicit error messages, as opposed to waiting until the source code tries to import some module from one of these packages and gets an ImportError. This function gets called from src/allmydata/__init__.py . """ We are considering experimenting with the multi-version feature of eggs, but haven't tried it yet. pkg_resources.require() is not particularly problematic for us as far as I recall. Regards, Zooko --- http://allmydata.org -- Tahoe, the Least-Authority Filesystem http://allmydata.com -- back up all your files for $5/month From chris at simplistix.co.uk Fri Oct 10 17:13:39 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 10 Oct 2008 16:13:39 +0100 Subject: [Distutils] specifying dependencies In-Reply-To: <48E63A9E.1050404@colorstudy.com> References: <20080917171851.GA27261@logilab.fr> <20080930084253.GF4494@volans.logilab.fr> <94bdd2610809300505u1291ae6an3c44e2ce31d43e3f@mail.gmail.com> <1222780673.4166.55.camel@shizuru> <94bdd2610809300649q6bc9cdf3t84d5221d956bfc9d@mail.gmail.com> <1222784868.4166.88.camel@shizuru> <94bdd2610809300820g36a3f1bbpc2a04e03b43ee9c9@mail.gmail.com> <48E24765.1020908@simplistix.co.uk> <48E24BDA.2050104@simplistix.co.uk> <94bdd2610809300901o3fa62601q985784f01a8ad774@mail.gmail.com> <48E24FF7.9000300@simplistix.co.uk> <48E255B9.50806@colorstudy.com> <48E636CE.4000703@simplistix.co.uk> <48E63A9E.1050404@colorstudy.com> Message-ID: <48EF7123.4080506@simplistix.co.uk> Ian Bicking wrote: > Chris Withers wrote: >> Ian Bicking wrote: >>> Say I have a package that represents an application. We'll call it >>> FooBlog. I release version 1.0. It uses the Turplango web framework >>> (1.5 at the time of release) and the Storchalmy ORM (0.4), and >>> Turplango uses HardJSON (1.2.1). >>> >>> I want my version 1.0 to keep working. So, I figure I'll add the >>> dependencies: >>> >>> Turplango==1.5 >>> Storchalmy==0.4 >> >> Why? >> >> I would have suggested: >> >> Turplango>=1.5,<2.0 >> Storchalmy==>=0.4,<0.5 > > Then when Turplango 1.6 comes out it'll break my code. I'm assuming that you, as a consumer of Turplango, understand the versioning structure of Turplango. Based on the above, my model assumed that <2.0 would be api-compatible with 1.x. If that's not the case, adjust the dependencies as necessary. >> Not could, should, in fact must. Relying on a dependency provided by >> library you're using is suicide. >> >> Again, I'd suggest: >> >> HardJSON >=1.2.1,<1.3 > > What does 1.3 mean? You imply there is a disciplined use of a > versioning pattern, I think for each usable library, there *is* a versioning pattern. If it's extremely unstable, that *should* push users away from the library. > and that every release is a guarantee that the > versioning has been properly declared. This comes under stability. Shit software is shit software whether its because it contains tonnes of bugs or because it doesn't specify its dependencies properly. > There isn't a common > understanding of versions, ...within a project, there generally is, which is all that's required here. > and it's common that conflicts are released > unintentionally. Well, if people have drummed into them how important accurate version dependencies are, then this won't happen... >>> But then a small bug fix, HardJSON 1.2.2 comes out, that fixes a >>> security bug. Turplango releases version 1.5.1 that requires >>> HardJSON>=1.2.2. I now have have to update FooBlog to require both >>> Turplango==1.5.1 and HardJSON==1.2.2. >> >> Not if you'd followed my advice above. > > OK, change that to "a small bug fix comes out as HardJSON 1.3", and the > same problems follow. I don't know what the nature of future releases > will be. See previous comments on the versioning structure used by a library. >>> Later on, I decide that Turplango 1.6 fixes some important bugs, and >>> I want to try it with my app. I can install Turplango 1.6, but I >>> can't start my app because I'll get a version conflict. >> >> Not if you'd followed my advice above. > > Now you've introduced an entirely different requirement -- for some > reason I am supposed to have known that HardJSON 1.3 would break my > code, but only Turplango 2.0 would cause a conflict. And Turplango 1.6 > wouldn't You're trying to make something out of nothing here. If the version dependencies are specified in setup.py or some kind of KGS they still have to be specified correctly. If they are not, you're screwed... >>> But if I've made other hard requirements of packages like HardJSON, >>> I'll have to update all those too. >> >> Yes, that's true, and why I recommeded what I did. That said, if >> you're paranoid enough to specify the exact versions (there's nothing >> wrong with this ;-) ) then it should be no surprise that you need to >> edit them... > > It's not surprising, it's just very annoying. Well then, only use libraries which properly specify their version dependencies and fix those that don't and you have no problem or annoyance. >> Or likely sources of known conflicts, such as major version increases, >> which is why I suggested what I did above... > > You presume you can predict likely sources of known conflicts in > software that doesn't exist yet. This is simply not true. Indeed, but I'm damned sure I can tell you what version ranges of *existing* software should be api compatible. >> Right, which is why consistency is version numbering for backwards >> incompatible changes is important. > > There is no single concept of what backward compatibility even is. There doesn't have to be one single concept, just that each library has to have its own understanding of this so that consumers of that library can express their requirements properly. > You can off something that fixes my specific example, using knowledge > that would not be available to you at the time when you were using the > code. That doesn't really prove anything -- I could also come up with > conflicts that would break any example you could provide. There's no > version change so minor that it can't break anything, and there's no > version change so major that you should end up with a cascading set of > updates that only change dependency information just to accommodate it. Well, if you want to be this negative about it, then you can lock down versions. No-ones stopping you and current tools such as buildout support this. Personally, I just don't thinl it should be necessary... Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From ianb at colorstudy.com Fri Oct 10 20:34:30 2008 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 10 Oct 2008 14:34:30 -0400 Subject: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification In-Reply-To: <79990c6b0810071416m3f841e27ve4d2880e4ccc810d@mail.gmail.com> References: <20081003163402.638A93A407A@sparrow.telecommunity.com> <20081006212511.GC17624@laurie.devork> <20081007013212.02F773A4045@sparrow.telecommunity.com> <79990c6b0810070204r4b2fb216la1daa69e36ba78f0@mail.gmail.com> <20081007182934.40C923A4045@sparrow.telecommunity.com> <79990c6b0810071416m3f841e27ve4d2880e4ccc810d@mail.gmail.com> Message-ID: <48EFA036.6000402@colorstudy.com> Paul Moore wrote: > My feeling, by the way, is that "system packagers" are the more > relevant group on Linux/Unix (where most users install Python modules > via system packages, or else they are developers) I think this is part of why I don't understand the system packager perspective. Developers shouldn't use system packages, it just doesn't make any sense to have that intermediation. Users don't use Python modules, they use applications. Users only care that their applications work, that they can install applications without unnecessary conflicts, that the applications don't break based on unintentional environment changes (e.g., the value of PYTHONPATH). Packagers seem to care a great deal about having applications share libraries on the packaging level, but this is for their own accounting, there's no reason for users to care (except for the too-small-to-matter issue of disk space). Also, packagers seem to jump the gun on this library sharing, as they are concerned about libraries when one (or often zero!) applications depend on the library. Some widely used libraries seem reasonable, but for every widely used library there are a dozen or more niche libraries. Users also don't care about /usr/share or /usr/lib -- the only thing *I* ever care about is /usr/share/doc, /usr/bin, /etc, and maybe a man page. -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From p.f.moore at gmail.com Fri Oct 10 21:27:37 2008 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 10 Oct 2008 20:27:37 +0100 Subject: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification In-Reply-To: <48EFA036.6000402@colorstudy.com> References: <20081003163402.638A93A407A@sparrow.telecommunity.com> <20081006212511.GC17624@laurie.devork> <20081007013212.02F773A4045@sparrow.telecommunity.com> <79990c6b0810070204r4b2fb216la1daa69e36ba78f0@mail.gmail.com> <20081007182934.40C923A4045@sparrow.telecommunity.com> <79990c6b0810071416m3f841e27ve4d2880e4ccc810d@mail.gmail.com> <48EFA036.6000402@colorstudy.com> Message-ID: <79990c6b0810101227w6901c523g34164b5e3240c540@mail.gmail.com> 2008/10/10 Ian Bicking : > Paul Moore wrote: >> >> My feeling, by the way, is that "system packagers" are the more >> relevant group on Linux/Unix (where most users install Python modules >> via system packages, or else they are developers) > > I think this is part of why I don't understand the system packager > perspective. Developers shouldn't use system packages, it just doesn't make > any sense to have that intermediation. Users don't use Python modules, they > use applications. Users only care that their applications work, that they > can install applications without unnecessary conflicts, that the > applications don't break based on unintentional environment changes (e.g., > the value of PYTHONPATH). Interesting viewpoint (from my Windows developer/user POV :-)) >From my point of view, as a Windows developer, I want to use "system packages" (bdist_wininst installers) because then I get the benefits of having a listing of what I have installed, being able to quickly, easily and cleanly install and uninstall packages to "try them out" etc. I have the same environment for my one-off scripts as for my development environment, which again I feel is good as I'm using a consistent, familiar, toolset. As a user, I want my applications packaged with py2exe, and independent of the "system Python" - which for me is the development environment, and therefore not stable enough to rely on for apps! And as a developer, py2exe cleanly picks out those bits of my development environment which are needed in the delivered application, so I don't have to worry about installing "too much" before I package my application up. This probably reflects the fact that per-user, or any other sort of private, installations of Python, are the exception rather than the norm on Windows, whereas on Unix, building a custom Python to do your development with (either from scratch, or with things like virtualenv) is easy and sensible. I guess the more we see how "the other half" lives (without the flamewars :-)), the better we'll be able to compromise or provide options to help everyone. (Skipped the comments on system packaging, as you've confirmed what I thought, which is that it's very much a Unix phenomenon, and I'm not qualified to have an opinion...) Paul. From marius at pov.lt Fri Oct 10 21:56:02 2008 From: marius at pov.lt (Marius Gedminas) Date: Fri, 10 Oct 2008 22:56:02 +0300 Subject: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification In-Reply-To: <48EFA036.6000402@colorstudy.com> References: <20081003163402.638A93A407A@sparrow.telecommunity.com> <20081006212511.GC17624@laurie.devork> <20081007013212.02F773A4045@sparrow.telecommunity.com> <79990c6b0810070204r4b2fb216la1daa69e36ba78f0@mail.gmail.com> <20081007182934.40C923A4045@sparrow.telecommunity.com> <79990c6b0810071416m3f841e27ve4d2880e4ccc810d@mail.gmail.com> <48EFA036.6000402@colorstudy.com> Message-ID: <20081010195601.GA8232@fridge.pov.lt> On Fri, Oct 10, 2008 at 02:34:30PM -0400, Ian Bicking wrote: > Paul Moore wrote: > >My feeling, by the way, is that "system packagers" are the more > >relevant group on Linux/Unix (where most users install Python modules > >via system packages, or else they are developers) > > I think this is part of why I don't understand the system packager > perspective. Developers shouldn't use system packages, it just doesn't > make any sense to have that intermediation. This only applies to packages you're developing, not those you're using to develop your app/library/whatever. Why I, as a developer, prefer system packages: * easier installation * easier upgrades * I don't have to keep track of security updates myself In fact those are the same reasons why I prefer system packages as a user. There are downsides (e.g. you usually don't get the latest version, and sometimes distributions apply broken patches). Marius Gedminas -- You'll find creativity working hand in hand with engineering. It will feel strange and you might feel like things are out of control. Relax - they are. -- Richard Gabriel on software http://www.dreamsongs.net/LessonsFromNothing.html -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From floris.bruynooghe at gmail.com Fri Oct 10 22:10:25 2008 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Fri, 10 Oct 2008 21:10:25 +0100 Subject: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification In-Reply-To: <20081010195601.GA8232@fridge.pov.lt> References: <20081003163402.638A93A407A@sparrow.telecommunity.com> <20081006212511.GC17624@laurie.devork> <20081007013212.02F773A4045@sparrow.telecommunity.com> <79990c6b0810070204r4b2fb216la1daa69e36ba78f0@mail.gmail.com> <20081007182934.40C923A4045@sparrow.telecommunity.com> <79990c6b0810071416m3f841e27ve4d2880e4ccc810d@mail.gmail.com> <48EFA036.6000402@colorstudy.com> <20081010195601.GA8232@fridge.pov.lt> Message-ID: <20081010201025.GA19580@laurie.devork> On Fri, Oct 10, 2008 at 10:56:02PM +0300, Marius Gedminas wrote: > On Fri, Oct 10, 2008 at 02:34:30PM -0400, Ian Bicking wrote: > > Paul Moore wrote: > > >My feeling, by the way, is that "system packagers" are the more > > >relevant group on Linux/Unix (where most users install Python modules > > >via system packages, or else they are developers) > > > > I think this is part of why I don't understand the system packager > > perspective. Developers shouldn't use system packages, it just doesn't > > make any sense to have that intermediation. > > This only applies to packages you're developing, not those you're using > to develop your app/library/whatever. > > Why I, as a developer, prefer system packages: > > * easier installation > * easier upgrades > * I don't have to keep track of security updates myself I'd like to add to this: making myself develop against system packages will make sure my work will be easy to package up and hence deploy. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From david at ar.media.kyoto-u.ac.jp Sat Oct 11 11:33:38 2008 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Sat, 11 Oct 2008 18:33:38 +0900 Subject: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification In-Reply-To: <48EFA036.6000402@colorstudy.com> References: <20081003163402.638A93A407A@sparrow.telecommunity.com> <20081006212511.GC17624@laurie.devork> <20081007013212.02F773A4045@sparrow.telecommunity.com> <79990c6b0810070204r4b2fb216la1daa69e36ba78f0@mail.gmail.com> <20081007182934.40C923A4045@sparrow.telecommunity.com> <79990c6b0810071416m3f841e27ve4d2880e4ccc810d@mail.gmail.com> <48EFA036.6000402@colorstudy.com> Message-ID: <48F072F2.9020002@ar.media.kyoto-u.ac.jp> Ian Bicking wrote: > I think this is part of why I don't understand the system packager > perspective. Developers shouldn't use system packages, it just > doesn't make any sense to have that intermediation. I can't see why you would think that. Of course it makes sense to have that intermediation, for packages which are not directly relevant to you. I am more than happy to use packages myself for almost everything but the things I am directly working on (and maybe the layer just below). > Users don't use Python modules, they use applications. What is an application ? I mean, people who use numpy/scipy, they use a module. People who are scons users, they use a software which is just one module + some scripts. The application / library difference is blurry in python. > Users only care that their applications work, that they can install > applications without unnecessary conflicts, that the applications > don't break based on unintentional environment changes (e.g., the > value of PYTHONPATH). Yes, of course. But this is obviously linked to deployment issues. Then, the argument is that when deploying things, multiple / concurrent modules is a headache. Api versioning and stability is not a linux packagers crazy idea > > Packagers seem to care a great deal about having applications share > libraries on the packaging level, but this is for their own > accounting, there's no reason for users to care (except for the > too-small-to-matter issue of disk space). Did you read carefully what was written by the packagers ? They almost never mention disk space (which does matter on some systems BTW; not everybody uses a desktop or a server). They mention security, they mention maintanability. Both those are made much more complicated when you enable multiple / concurrent versions. > Also, packagers seem to jump the gun on this library sharing, as > they are concerned about libraries when one (or often zero!) > applications depend on the library. Some widely used libraries seem > reasonable, but for every widely used library there are a dozen or > more niche libraries. Those niche libraries are not always packaged. If a library is used by only one package, it is quite likely that it is included in the package itself. cheers, David From gael.varoquaux at normalesup.org Sat Oct 11 12:21:48 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sat, 11 Oct 2008 12:21:48 +0200 Subject: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification In-Reply-To: <48EFA036.6000402@colorstudy.com> References: <20081003163402.638A93A407A@sparrow.telecommunity.com> <20081006212511.GC17624@laurie.devork> <20081007013212.02F773A4045@sparrow.telecommunity.com> <79990c6b0810070204r4b2fb216la1daa69e36ba78f0@mail.gmail.com> <20081007182934.40C923A4045@sparrow.telecommunity.com> <79990c6b0810071416m3f841e27ve4d2880e4ccc810d@mail.gmail.com> <48EFA036.6000402@colorstudy.com> Message-ID: <20081011102148.GA22415@phare.normalesup.org> On Fri, Oct 10, 2008 at 02:34:30PM -0400, Ian Bicking wrote: > Developers shouldn't use system packages, it just doesn't > make any sense to have that intermediation. I don't agree. I work as a developer/scientist in a lab. I develop algorithms that sometimes will never be used on another box than mine (that's when they are bad). We happen to be standardised on a Linux distro that is a bit old, and with little packages. I can tell you it really annoys me to have to install manually a package each time I want to use it. The pure-Python ones, with no other dependencies are not hard, easy-install does the trick, but these actually represent a small fraction of the important ones (wxpython, numpy, pytables, matplotlib, elementtree, pyobjects, cython, ets, pyvtk). So I lose a lot of time installing packages on my box. But the real nightmare begins when I want to share the algorithms and applications I developed. Across our institute we have NFS-shared drives on which we have to compile all the dependencies, for all three platforms we support. And that's only for distributing software in the institute, when we want to distribute outside we end up shipping big binary blobs with everything, from our own version of Python, to numpy, wxPython, ATLAS (that's bad). If only we could rely on package managers to provide us with base packages (and API-stability of these packages to be sure they work). > Users don't use Python modules, they use applications. Users only care > that their applications work, that they can install applications > without unnecessary conflicts, that the applications don't break based > on unintentional environment changes (e.g., the value of PYTHONPATH). That is true, as long as you ship "the world", ie everything you'll ever need for your extensible application, including the c libraries that go behind. This means putting a huge burden on your development team, and if you think about out, this is exactly the work of a distribution. So you are duplicating this effort. I addition, one of the great things about Python, is that it is easy to read and to code. Advanced users will want to write plugins, extend there programs, and programs should be written for this: "Hey, I have this cool application that does brain segmentation, and visualization of the tissues, and I have this cool other app that has an algorithm to identify tumorous tissues, I want to plug them together, and because the APIs are well-written, it should be trivial... Darn, the two apps have confined Pythons, and they can't import from each-others. No big deal, I'll unbundle the whole thing... Darn, one has been build with UCS-2, the other one with UCS-4, and there C libraries are incompatible... So now the easy task has become a long effort of building the two apps, and I can't easily share my work." I can hardly believe this happens only in the scientific computing world. I have fought (and I still fighting) to deploy a Django-based application. It didn't get along with the server's apache version. Ga?l From ziade.tarek at gmail.com Sat Oct 11 19:56:27 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 11 Oct 2008 13:56:27 -0400 Subject: [Distutils] distribute D.C. sprint tasks Message-ID: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> Here's the lists of tasks we are going to work on. They are simple. - PyPI : write a patch to enforce (or display a warning) the source distribution to be uploaded. so if a binary distribution or a zipped egg is uploaded we are sure we provide the source as well. - Documentation: write a glossary for the distutils/setuptools/Pypi terminology on python.org wiki - PyPI mirroring: write a PEP to implement a mirroring protocol, where mirrors can register at PyPI. Then when a package is uploaded, mirrors will be ping through RPC so they know they can eventually get synced. - setuptools: finish the patch for the multiple index support, with a CPAN-like mechanism on the client side, with a socket timeout managment - - distutils: code cleaning: better test coverage, remove logging, etc. Tarek -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From ziade.tarek at gmail.com Sun Oct 12 00:24:04 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 11 Oct 2008 18:24:04 -0400 Subject: [Distutils] distribute D.C. sprint tasks In-Reply-To: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> Message-ID: <94bdd2610810111524w4cba4f70n4766fdb0b95fe73e@mail.gmail.com> Day 1 is almost over, We worked on two elements so far: the mirroring thing, and the terminology one these are early drafts, http://wiki.python.org/moin/PythonPackagingTerminology http://wiki.python.org/moin/PEP_374 please comment Tarek On Sat, Oct 11, 2008 at 1:56 PM, Tarek Ziad? wrote: > Here's the lists of tasks we are going to work on. They are simple. > > - PyPI : write a patch to enforce (or display a warning) the source > distribution to be uploaded. so if a binary distribution or a zipped > egg is uploaded > we are sure we provide the source as well. > > - Documentation: write a glossary for the distutils/setuptools/Pypi > terminology on python.org wiki > > - PyPI mirroring: write a PEP to implement a mirroring protocol, where > mirrors can register at PyPI. Then when a package is uploaded, mirrors > will be ping through RPC > so they know they can eventually get synced. > > - setuptools: finish the patch for the multiple index support, with a > CPAN-like mechanism on the client side, with a socket timeout > managment - > > - distutils: code cleaning: better test coverage, remove logging, etc. > > > Tarek > > -- > Tarek Ziad? | Association AfPy | www.afpy.org > Blog FR | http://programmation-python.org > Blog EN | http://tarekziade.wordpress.com/ > -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From lists at zopyx.com Sun Oct 12 01:40:28 2008 From: lists at zopyx.com (Andreas Jung) Date: Sat, 11 Oct 2008 19:40:28 -0400 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <94bdd2610810111524w4cba4f70n4766fdb0b95fe73e@mail.gmail.com> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <94bdd2610810111524w4cba4f70n4766fdb0b95fe73e@mail.gmail.com> Message-ID: <48F1396C.8030907@zopyx.com> On 11.10.2008 18:24 Uhr, Tarek Ziad? wrote: > Day 1 is almost over, > > We worked on two elements so far: the mirroring thing, and the terminology one > > these are early drafts, > > http://wiki.python.org/moin/PythonPackagingTerminology > http://wiki.python.org/moin/PEP_374 > > I think we should also investigate how other repositories like CPAN or the Ruby world deals with mirroring. A notification mechanism appears fragile to me. I believe that the mirrors should remain "dumb" in order to keep the complete system simple and solid - less moving parts are better than more. I really have a bad feeling about the approach as described in your PEP. Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 316 bytes Desc: not available URL: From kteague at bawx.ca Sun Oct 12 02:10:42 2008 From: kteague at bawx.ca (Kevin Wheat Teague) Date: Sat, 11 Oct 2008 17:10:42 -0700 Subject: [Distutils] distribute D.C. sprint tasks Message-ID: <20081012001042.GA3861@bawx.ca> I'm (physically) at the sprint, and am working on documentation aspects. As a starting point for this I've begun to compile a list of terminology relevant to Python packaging and have put these terms and definitions on the Python wiki: http://wiki.python.org/moin/PythonPackagingTerminology In order to make it easier to make it easier for people to chip in their opinions on different terms, I've pasted the contents of the wiki page into this email. This way people can send reply to this mail to the distutils-sig list and I'll aggregate the discussion over time into the wiki page. I'd also like to expand the scope of terms to include anything in the Python packaging ecosystem, so please feel free to send in your own terms and definitions. However, only a sub-set of these terms may eventually be included in a possible list of terms used in a BUILDS PEP. Terminology Status ------------------ Currently, this is a **very** rough list of terms and definitions used in the Python packaging ecosystem and currently exists as a very loose first draft. More terms may be added (and a few terms might be spurious and can be removed). The definitions of terms has largely been culled from the Distutils and Setuptools/PEAK web sites. These term and definitions are intended as a starting point for discussion. Module ------ The basic unit of code reusability in Python. A block of code imported by some other code. Pure Python module ------------------ A module written in Python and contained in a single .py file (and possibly associated .pyc and/or .pyo files). Sometimes referred to as a "pure module." Package ------- A module that contains other modules. Typically contained in a directory in the filesystem and distinguished from other directories by the presence of a __init__.py file. * To-Do: Update this definition to reflect namespace packages? Root Package ------------ The root of the hierarchy of packages. (This isn? distribution). The directory where setup.py exists. Generally setup.py will be run from this directory. Distutils --------- Package included in the Python Standard Library for installing, building and distributing Python code. Metadata for Python Software Packages ------------------------------------- Metadata is data about the contents of a Python package. The format and fields specified in version 1.0 is detailed in PEP 241 (http://www.python.org/dev/peps/pep-0241/), and support for working with these fields was included in the distutils package which was added in Python 2.1. Version 1.1 additiones were detailed in PEP 314 (http://www.python.org/dev/peps/pep-0314/). This updated the fields to include 'Download-URL', 'Requires', 'Provides' and 'Obsoletes' fields. Support was added to the distutils package for this in Python 2.5. The simple dependency information fields for a distribution is generally not used, as the specific module requirements can be dynamic depending on the platform and installation context. Version 1.2 was proposed in PEP 345 (http://www.python.org/dev/peps/pep-0345/), but is still in the draft status and has not been approved not support for these fields implemented. Setuptools ---------- Setuptools is a collection of enhancements to the Python distutils (for Python 2.3.5 and up on most platforms; 64-bit platforms require a minimum of Python 2.4) that allow you to more easily build and distribute Python packages, especially ones that have dependencies on other packages. easy_install ------------ Easy Install is a python module (easy_install) bundled with setuptools that lets you automatically download, build, install, and manage Python packages. pkg_resources ------------- The pkg_resources module distributed with setuptools provides an API for Python libraries to access their resource files, and for extensible applications and frameworks to automatically discover plugins. It also provides runtime support for using C extensions that are inside zipfile-format eggs, support for merging packages that have separately-distributed modules or subpackages, and APIs for managing Python's current "working set" of active packages. Eggs ---- The Egg PEAK page definition: Eggs are a way of bundling additional information with a Python project, that allows the project's dependencies to be checked and satisfied at runtime, as well as allowing projects to provide plugins for other projects. There are several binary formats that embody eggs, but the most common is '.egg' zipfile format, because it's a convenient one for distributing projects. All of the formats support including package-specific data, project-wide metadata, C extensions, and Python code. The PkgResources PEAK page definition: Eggs are pluggable distributions in one of the three formats currently supported by pkg_resources. There are built eggs, development eggs, and egg links. Built eggs are directories or zipfiles whose name ends with .egg and follows the egg naming conventions, and contain an EGG-INFO subdirectory (zipped or otherwise). Development eggs are normal directories of Python code with one or more ProjectName.egg-info subdirectories. And egg links are *.egg-link files that contain the name of a built or development egg, to support symbolic linking on platforms that do not have native symbolic links. * To-Do: There is a lot of confusion as to what an "egg" is. Some believe it refers to the additional metadata, others believe it is the binary format (the .egg file). e.g. is a source distribution that uses setuptools an egg? Built Egg --------- A zipped file or directory whose name ends with .egg and follows the egg naming conventions, and contains an EGG-INFO subdirectory. Built eggs can contain binary data specific to a target platform. * To-Do: some call these "binary eggs", clarify between a binary egg and a built egg? Development Egg --------------- Normal directories of Python code with one or more ProjectName.egg-info subdirectories. * To-Do: is there a difference between a "source egg" and a "development egg"? Egg Link -------- *.egg-link files that contain the name of a built or development egg, to support symbolic linking on platforms that do not have native symbolic links. Project ------- A library, framework, script, plugin, application, or collection of data or other resources, or some combination thereof. Projects are assumed to have "relatively unique" names, e.g. names registered with PyPI. Release ------- A snapshot of a project at a particular point in time, denoted by a version identifier. Distribution ------------ A file or files that represent a particular release. Importable Distribution ----------------------- A file or directory that, if placed on sys.path, allows Python to import any modules contained within it. Pluggable Distribution ---------------------- An importable distribution whose filename unambiguously identifies its release (i.e. project and version), and whose contents unamabiguously specify what releases of other projects will satisfy its runtime requirements. Extra ----- An "extra" is an optional feature of a release, that may impose additional runtime requirements. For example, if docutils PDF support required a PDF support library to be present, docutils could define its PDF support as an "extra", and list what other project releases need to be available in order to provide it. Environment ----------- A collection of distributions potentially available for importing, but not necessarily active. More than one distribution (i.e. release version) for a given project may be present in an environment. * To-Do: A shared egg cache can be specified in Buildout by using the 'eggs-directory' option. This is often informally referred to as an "egg cache". * To-Do: Recommendations for where a "global egg cache" could possibly live within different operating systems? Working Set ----------- A collection of distributions available for importing. That is distributions that are on the on the sys.path. At most one distribution (release version) of a given project may be present in a working set, as otherwise there would be ambiguity as to what to import. Working sets include all distributions available for importing, not just the sub-set of distributions which have actually been imported. System Package -------------- A package provided by in a format native to the operating system. e.g. rpm or dpkg file. Installed Distribution ---------------------- A distribution which is available for import without explicitly modifying the sys.path. An installed distribution is * To-Do: clarify this ... ? Namespace Package ----------------- A namespace package is a package that only contains other packages and modules, with no direct contents of its own. Such packages can be split across multiple, separately-packaged distributions. Buildout -------- Tool that provides support for creating applications, especially Python applications. It provides tools for assembling applications from multiple parts, Python or otherwise. An application may actually contain multiple programs, processes, and configuration settings. Buildout is commonly used to install and manage working sets of Python distributions in the egg format. * To-Do: clarify the project name, since some refer to Buildout as "zc.buildout", the name of the Python package. VirtualEnv ---------- Tool to create isolated Python environments. * To-Do: The term environment differs between setuptools and virtualenv. setup.py -------- The name of the Python module in the root package which is used to dynamically generate the metadata. PyPI ---- The Python Package Index, a central repository of Python software. PyPI is organized by projects, each of which has a unique name on PyPI. A project can have multiple releases, and each release can contain mutliple distributions. Typically a source distribution is the preferred format for release, but distributions in a build egg format are also possible, especially to make it easier to install software without requiring a working compiler on the target system. Known Good Set -------------- A working set which has been tested to work together, e.g. integration tests between assorted package dependencies have been run. From ziade.tarek at gmail.com Sun Oct 12 03:56:51 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 11 Oct 2008 21:56:51 -0400 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <48F1396C.8030907@zopyx.com> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <94bdd2610810111524w4cba4f70n4766fdb0b95fe73e@mail.gmail.com> <48F1396C.8030907@zopyx.com> Message-ID: <94bdd2610810111856j4dbefcabra43dc590cb556e0d@mail.gmail.com> On Sat, Oct 11, 2008 at 7:40 PM, Andreas Jung wrote: > On 11.10.2008 18:24 Uhr, Tarek Ziad? wrote: >> >> Day 1 is almost over, >> >> We worked on two elements so far: the mirroring thing, and the terminology >> one >> >> these are early drafts, >> >> http://wiki.python.org/moin/PythonPackagingTerminology >> http://wiki.python.org/moin/PEP_374 >> >> > > I think we should also investigate how other repositories like CPAN > or the Ruby world deals with mirroring. or rather linux http://www.mail-archive.com/distutils-sig at python.org/msg05791.html > A notification mechanism appears > fragile to me. I believe that the mirrors should remain "dumb" > in order to keep the complete system simple and solid -` Well at some point you need a protocol, otherwise your mirror is this "dumb" thing we cannot garantee to be a reliable thing. > less moving parts > are better than more. I really have a bad feeling about the approach as > described in your PEP. well, I think we need more than a rsync script at some point. > > Andreas > -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From lists at zopyx.com Sun Oct 12 09:57:00 2008 From: lists at zopyx.com (Andreas Jung) Date: Sun, 12 Oct 2008 03:57:00 -0400 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <94bdd2610810111856j4dbefcabra43dc590cb556e0d@mail.gmail.com> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <94bdd2610810111524w4cba4f70n4766fdb0b95fe73e@mail.gmail.com> <48F1396C.8030907@zopyx.com> <94bdd2610810111856j4dbefcabra43dc590cb556e0d@mail.gmail.com> Message-ID: <48F1ADCC.5050602@zopyx.com> On 11.10.2008 21:56 Uhr, Tarek Ziad? wrote: > On Sat, Oct 11, 2008 at 7:40 PM, Andreas Jung wrote: >> On 11.10.2008 18:24 Uhr, Tarek Ziad? wrote: >>> Day 1 is almost over, >>> >>> We worked on two elements so far: the mirroring thing, and the terminology >>> one >>> >>> these are early drafts, >>> >>> http://wiki.python.org/moin/PythonPackagingTerminology >>> http://wiki.python.org/moin/PEP_374 >>> >>> >> I think we should also investigate how other repositories like CPAN >> or the Ruby world deals with mirroring. > > or rather linux > > http://www.mail-archive.com/distutils-sig at python.org/msg05791.html > > >> A notification mechanism appears >> fragile to me. I believe that the mirrors should remain "dumb" >> in order to keep the complete system simple and solid -` > > Well at some point you need a protocol, otherwise your mirror > is this "dumb" thing we cannot garantee to be a reliable thing. > >> less moving parts >> are better than more. I really have a bad feeling about the approach as >> described in your PEP. > > well, I think we need more than a rsync script at some point. The question is if you want a push or pull mechanism. z3c.pypimirror implements the pull implementation and performs an incremental update in the sense of rsync in a reliable way. Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 316 bytes Desc: not available URL: From sdouche at gmail.com Sun Oct 12 14:40:05 2008 From: sdouche at gmail.com (Sebastien Douche) Date: Sun, 12 Oct 2008 14:40:05 +0200 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <48F1ADCC.5050602@zopyx.com> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <94bdd2610810111524w4cba4f70n4766fdb0b95fe73e@mail.gmail.com> <48F1396C.8030907@zopyx.com> <94bdd2610810111856j4dbefcabra43dc590cb556e0d@mail.gmail.com> <48F1ADCC.5050602@zopyx.com> Message-ID: <5e1183fa0810120540g6e902699hfe7b177b57532f9c@mail.gmail.com> On Sun, Oct 12, 2008 at 09:57, Andreas Jung wrote: > The question is if you want a push or pull mechanism. z3c.pypimirror > implements the pull implementation and performs an incremental update > in the sense of rsync in a reliable way. Can you speak more on incremental update ? I use z3c;pypimirror and it needs 2 hours to make a complete update. 2008-10-12 12:01:43,671 DEBUG Statistics 2008-10-12 12:01:43,672 DEBUG ---------- 2008-10-12 12:01:43,672 DEBUG Found (cached): 17626 2008-10-12 12:01:43,672 DEBUG Stored (downloaded): 1306 2008-10-12 12:01:43,673 DEBUG Not found (404): 35 2008-10-12 12:01:43,673 DEBUG Invalid packages: 1 2008-10-12 12:01:43,673 DEBUG Invalid URLs: 265 2008-10-12 12:01:43,673 DEBUG Runtime: 120m21s -- Seb From ziade.tarek at gmail.com Sun Oct 12 14:55:44 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 12 Oct 2008 08:55:44 -0400 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <48F1ADCC.5050602@zopyx.com> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <94bdd2610810111524w4cba4f70n4766fdb0b95fe73e@mail.gmail.com> <48F1396C.8030907@zopyx.com> <94bdd2610810111856j4dbefcabra43dc590cb556e0d@mail.gmail.com> <48F1ADCC.5050602@zopyx.com> Message-ID: <94bdd2610810120555q5ee21b59q595efffed1aff2ed@mail.gmail.com> On Sun, Oct 12, 2008 at 3:57 AM, Andreas Jung wrote: >> well, I think we need more than a rsync script at some point. > > The question is if you want a push or pull mechanism. z3c.pypimirror > implements the pull implementation and performs an incremental update > in the sense of rsync in a reliable way. You don't want a push mechanism, but you want a way to list the mirrors at PyPI, their states, and know if they are good mirrors or not. That is what a ping mechanism provides. So the point is not about being able to provide a reliable "copy of files program", I can use for that a "wget --mirror" or "rsync -r" and I don't need to write a program for this. On the other hand maybe the XML-RPC thing is a little heavy but we sure need to know how "fresh" a mirror is. Maybe if the last-modified header is maintained on the mirror this could be enough for PyPI and other third-party application to know about it. -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From ziade.tarek at gmail.com Sun Oct 12 14:58:02 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 12 Oct 2008 08:58:02 -0400 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <5e1183fa0810120540g6e902699hfe7b177b57532f9c@mail.gmail.com> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <94bdd2610810111524w4cba4f70n4766fdb0b95fe73e@mail.gmail.com> <48F1396C.8030907@zopyx.com> <94bdd2610810111856j4dbefcabra43dc590cb556e0d@mail.gmail.com> <48F1ADCC.5050602@zopyx.com> <5e1183fa0810120540g6e902699hfe7b177b57532f9c@mail.gmail.com> Message-ID: <94bdd2610810120558v31a75fc2ga9823e92f906bede@mail.gmail.com> On Sun, Oct 12, 2008 at 8:40 AM, Sebastien Douche wrote: > On Sun, Oct 12, 2008 at 09:57, Andreas Jung wrote: >> The question is if you want a push or pull mechanism. z3c.pypimirror >> implements the pull implementation and performs an incremental update >> in the sense of rsync in a reliable way. > > Can you speak more on incremental update ? I use z3c;pypimirror and it > needs 2 hours to make a complete update. we use wget --mirror, and it is working quite good, an upgrade is around 10 minutes IIRC but I think rsync is the best way to go for a mirror. From ziade.tarek at gmail.com Sun Oct 12 18:05:20 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 12 Oct 2008 12:05:20 -0400 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <48F1396C.8030907@zopyx.com> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <94bdd2610810111524w4cba4f70n4766fdb0b95fe73e@mail.gmail.com> <48F1396C.8030907@zopyx.com> Message-ID: <94bdd2610810120905i2af1c221gc6075d7fe4b573ce@mail.gmail.com> We removed the RPC thing, and added a freshness date principle, to make thing simpler. see http://wiki.python.org/moin/PEP_374 On Sat, Oct 11, 2008 at 7:40 PM, Andreas Jung wrote: > On 11.10.2008 18:24 Uhr, Tarek Ziad? wrote: >> >> Day 1 is almost over, >> >> We worked on two elements so far: the mirroring thing, and the terminology >> one >> >> these are early drafts, >> >> http://wiki.python.org/moin/PythonPackagingTerminology >> http://wiki.python.org/moin/PEP_374 >> >> > > I think we should also investigate how other repositories like CPAN > or the Ruby world deals with mirroring. A notification mechanism appears > fragile to me. I believe that the mirrors should remain "dumb" > in order to keep the complete system simple and solid - less moving parts > are better than more. I really have a bad feeling about the approach as > described in your PEP. > > Andreas > -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From martin at v.loewis.de Sun Oct 12 19:47:39 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 12 Oct 2008 19:47:39 +0200 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <48F1396C.8030907@zopyx.com> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <94bdd2610810111524w4cba4f70n4766fdb0b95fe73e@mail.gmail.com> <48F1396C.8030907@zopyx.com> Message-ID: <48F2383B.7020103@v.loewis.de> > I think we should also investigate how other repositories like CPAN > or the Ruby world deals with mirroring. A notification mechanism appears > fragile to me. I believe that the mirrors should remain "dumb" > in order to keep the complete system simple and solid - less moving > parts are better than more. I really have a bad feeling about the > approach as described in your PEP. I'm also skeptical about that. I don't think this callback solves any specific problem. Regards, Martin From martin at v.loewis.de Sun Oct 12 19:48:43 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 12 Oct 2008 19:48:43 +0200 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <5e1183fa0810120540g6e902699hfe7b177b57532f9c@mail.gmail.com> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <94bdd2610810111524w4cba4f70n4766fdb0b95fe73e@mail.gmail.com> <48F1396C.8030907@zopyx.com> <94bdd2610810111856j4dbefcabra43dc590cb556e0d@mail.gmail.com> <48F1ADCC.5050602@zopyx.com> <5e1183fa0810120540g6e902699hfe7b177b57532f9c@mail.gmail.com> Message-ID: <48F2387B.8060709@v.loewis.de> > Can you speak more on incremental update ? What would you like to know? incremental update should be very easy to implement for a mirror tool, with no additional changes to PyPI. Regards, Martin From martin at v.loewis.de Sun Oct 12 19:50:15 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 12 Oct 2008 19:50:15 +0200 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <94bdd2610810120558v31a75fc2ga9823e92f906bede@mail.gmail.com> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <94bdd2610810111524w4cba4f70n4766fdb0b95fe73e@mail.gmail.com> <48F1396C.8030907@zopyx.com> <94bdd2610810111856j4dbefcabra43dc590cb556e0d@mail.gmail.com> <48F1ADCC.5050602@zopyx.com> <5e1183fa0810120540g6e902699hfe7b177b57532f9c@mail.gmail.com> <94bdd2610810120558v31a75fc2ga9823e92f906bede@mail.gmail.com> Message-ID: <48F238D7.7050600@v.loewis.de> > we use wget --mirror, and it is working quite good, an upgrade is > around 10 minutes IIRC > but I think rsync is the best way to go for a mirror. I disagree. The best way to do the mirroring is with the specific protocol explicitly designed to do mirroring, namely to look at the changelog. PLEASE DONT USE RSYNC OR WGET TO MIRROR PYPI. PLEASE! Regards, Martin From ziade.tarek at gmail.com Sun Oct 12 19:51:42 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 12 Oct 2008 13:51:42 -0400 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <48F238D7.7050600@v.loewis.de> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <94bdd2610810111524w4cba4f70n4766fdb0b95fe73e@mail.gmail.com> <48F1396C.8030907@zopyx.com> <94bdd2610810111856j4dbefcabra43dc590cb556e0d@mail.gmail.com> <48F1ADCC.5050602@zopyx.com> <5e1183fa0810120540g6e902699hfe7b177b57532f9c@mail.gmail.com> <94bdd2610810120558v31a75fc2ga9823e92f906bede@mail.gmail.com> <48F238D7.7050600@v.loewis.de> Message-ID: <94bdd2610810121051m40f4114ai8a9730d958bf7d50@mail.gmail.com> On Sun, Oct 12, 2008 at 1:50 PM, "Martin v. L?wis" wrote: >> we use wget --mirror, and it is working quite good, an upgrade is >> around 10 minutes IIRC >> but I think rsync is the best way to go for a mirror. > > I disagree. The best way to do the mirroring is with the specific > protocol explicitly designed to do mirroring, namely to look at > the changelog. > > PLEASE DONT USE RSYNC OR WGET TO MIRROR PYPI. PLEASE! could you explain why is that a problem ? > > Regards, > Martin > -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From martin at v.loewis.de Sun Oct 12 19:53:45 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 12 Oct 2008 19:53:45 +0200 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <94bdd2610810120555q5ee21b59q595efffed1aff2ed@mail.gmail.com> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <94bdd2610810111524w4cba4f70n4766fdb0b95fe73e@mail.gmail.com> <48F1396C.8030907@zopyx.com> <94bdd2610810111856j4dbefcabra43dc590cb556e0d@mail.gmail.com> <48F1ADCC.5050602@zopyx.com> <94bdd2610810120555q5ee21b59q595efffed1aff2ed@mail.gmail.com> Message-ID: <48F239A9.2040609@v.loewis.de> > That is what a ping mechanism provides. Hmm. If the mirror provided a file "last-changed", it would be very easy to find out whether the mirror is still running. Mirrors not providing that file could be ignored. > I can use for that a "wget --mirror" or "rsync -r" and I don't need to > write a program for this. If more people start mirroring PyPI through wget or rsync, I need to ban specific IP addresses. For the moment, please consider my request to stop mirroring PyPI with wget, and to write (or use) a real mirroring tool. Regards, Martin From martin at v.loewis.de Sun Oct 12 20:12:33 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 12 Oct 2008 20:12:33 +0200 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <94bdd2610810121051m40f4114ai8a9730d958bf7d50@mail.gmail.com> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <94bdd2610810111524w4cba4f70n4766fdb0b95fe73e@mail.gmail.com> <48F1396C.8030907@zopyx.com> <94bdd2610810111856j4dbefcabra43dc590cb556e0d@mail.gmail.com> <48F1ADCC.5050602@zopyx.com> <5e1183fa0810120540g6e902699hfe7b177b57532f9c@mail.gmail.com> <94bdd2610810120558v31a75fc2ga9823e92f906bede@mail.gmail.com> <48F238D7.7050600@v.loewis.de> <94bdd2610810121051m40f4114ai8a9730d958bf7d50@mail.gmail.com> Message-ID: <48F23E11.30801@v.loewis.de> > could you explain why is that a problem ? It produces significant load on the master. If you look at the web stats, e.g for September: http://pypi.python.org/webstats/usage_200809.html you see that there had been 5671455 hits, or 41%, of accesses through wget. The problem with wget mirroring is that it needs to read *many* pages, to find out the *few* changes. FWIW, it's also the case that 4940769 hits originate from France. Could it be that you are alone responsible for 40% of the traffic on PyPI? Regards, Martin From ziade.tarek at gmail.com Sun Oct 12 20:16:17 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 12 Oct 2008 14:16:17 -0400 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <48F239A9.2040609@v.loewis.de> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <94bdd2610810111524w4cba4f70n4766fdb0b95fe73e@mail.gmail.com> <48F1396C.8030907@zopyx.com> <94bdd2610810111856j4dbefcabra43dc590cb556e0d@mail.gmail.com> <48F1ADCC.5050602@zopyx.com> <94bdd2610810120555q5ee21b59q595efffed1aff2ed@mail.gmail.com> <48F239A9.2040609@v.loewis.de> Message-ID: <94bdd2610810121116x3b7f7a6bw7cf0298600d33cba@mail.gmail.com> On Sun, Oct 12, 2008 at 1:53 PM, "Martin v. L?wis" wrote: >> That is what a ping mechanism provides. > > Hmm. If the mirror provided a file "last-changed", it would be very > easy to find out whether the mirror is still running. Mirrors not > providing that file could be ignored. Right, please take a look at my last version http://wiki.python.org/moin/PEP_374 it tries to go in that direction From ziade.tarek at gmail.com Sun Oct 12 20:32:25 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 12 Oct 2008 14:32:25 -0400 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <48F23E11.30801@v.loewis.de> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <94bdd2610810111524w4cba4f70n4766fdb0b95fe73e@mail.gmail.com> <48F1396C.8030907@zopyx.com> <94bdd2610810111856j4dbefcabra43dc590cb556e0d@mail.gmail.com> <48F1ADCC.5050602@zopyx.com> <5e1183fa0810120540g6e902699hfe7b177b57532f9c@mail.gmail.com> <94bdd2610810120558v31a75fc2ga9823e92f906bede@mail.gmail.com> <48F238D7.7050600@v.loewis.de> <94bdd2610810121051m40f4114ai8a9730d958bf7d50@mail.gmail.com> <48F23E11.30801@v.loewis.de> Message-ID: <94bdd2610810121132o34c51df8gb3acac84fa283502@mail.gmail.com> On Sun, Oct 12, 2008 at 2:12 PM, "Martin v. L?wis" wrote: >> could you explain why is that a problem ? > > It produces significant load on the master. If you look at the web > stats, e.g for September: > > http://pypi.python.org/webstats/usage_200809.html > > you see that there had been 5671455 hits, or 41%, of accesses through > wget. > > The problem with wget mirroring is that it needs to read *many* > pages, to find out the *few* changes. Sure, > FWIW, it's also the case that 4940769 hits originate from > France. Could it be that you are alone responsible for 40% of > the traffic on PyPI? > Yes, I am the only Python developer in France. That's me. Just kidding :) France has a lot of python/plone developers that triggers buildouts every day, so I am pretty sure the mirrors don't make the whole traffic in PyPI. we could probably do things better though. Here's my proposal: + see if we can locate the mirrors, so for instance, if i register a "Paris mirror" people will eventually go there because it is the nearest location for them. (? la CPAN) + create a new user agent for mirroring tools Regards Tarek > Regards, > Martin > -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From martin at v.loewis.de Sun Oct 12 20:34:27 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 12 Oct 2008 20:34:27 +0200 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <94bdd2610810121116x3b7f7a6bw7cf0298600d33cba@mail.gmail.com> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <94bdd2610810111524w4cba4f70n4766fdb0b95fe73e@mail.gmail.com> <48F1396C.8030907@zopyx.com> <94bdd2610810111856j4dbefcabra43dc590cb556e0d@mail.gmail.com> <48F1ADCC.5050602@zopyx.com> <94bdd2610810120555q5ee21b59q595efffed1aff2ed@mail.gmail.com> <48F239A9.2040609@v.loewis.de> <94bdd2610810121116x3b7f7a6bw7cf0298600d33cba@mail.gmail.com> Message-ID: <48F24333.5070008@v.loewis.de> > Right, please take a look at my last version http://wiki.python.org/moin/PEP_374 > it tries to go in that direction For such an infrastructure (which apparently intends to mirror the files as well), I insist that a propagation of download counters is made mandatory. The only mirrors that can be excused from that are private ones. Regards, Martin From martin at v.loewis.de Sun Oct 12 20:49:26 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 12 Oct 2008 20:49:26 +0200 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <94bdd2610810121132o34c51df8gb3acac84fa283502@mail.gmail.com> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <94bdd2610810111524w4cba4f70n4766fdb0b95fe73e@mail.gmail.com> <48F1396C.8030907@zopyx.com> <94bdd2610810111856j4dbefcabra43dc590cb556e0d@mail.gmail.com> <48F1ADCC.5050602@zopyx.com> <5e1183fa0810120540g6e902699hfe7b177b57532f9c@mail.gmail.com> <94bdd2610810120558v31a75fc2ga9823e92f906bede@mail.gmail.com> <48F238D7.7050600@v.loewis.de> <94bdd2610810121051m40f4114ai8a9730d958bf7d50@mail.gmail.com> <48F23E11.30801@v.loewis.de> <94bdd2610810121132o34c51df8gb3acac84fa283502@mail.gmail.com> Message-ID: <48F246B6.8000707@v.loewis.de> >> FWIW, it's also the case that 4940769 hits originate from >> France. Could it be that you are alone responsible for 40% of >> the traffic on PyPI? >> > > Yes, I am the only Python developer in France. That's me. > > Just kidding :) > > France has a lot of python/plone developers that triggers buildouts every day, > so I am pretty sure the mirrors don't make the whole traffic in PyPI. Hmm. Yesterday, there were 199250 accesses to PyPI through wget. Of those, 169971 requests came from a single address (from Dedibox in France), 28966 requests from a second one (from Sakura in Japan). So it *is* wget mirrors that make the whole traffic in PyPI. Regards, Martin From ziade.tarek at gmail.com Sun Oct 12 21:12:51 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 12 Oct 2008 15:12:51 -0400 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <48F24333.5070008@v.loewis.de> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <94bdd2610810111524w4cba4f70n4766fdb0b95fe73e@mail.gmail.com> <48F1396C.8030907@zopyx.com> <94bdd2610810111856j4dbefcabra43dc590cb556e0d@mail.gmail.com> <48F1ADCC.5050602@zopyx.com> <94bdd2610810120555q5ee21b59q595efffed1aff2ed@mail.gmail.com> <48F239A9.2040609@v.loewis.de> <94bdd2610810121116x3b7f7a6bw7cf0298600d33cba@mail.gmail.com> <48F24333.5070008@v.loewis.de> Message-ID: <94bdd2610810121212xb132054y1e43f87e034fd1c2@mail.gmail.com> On Sun, Oct 12, 2008 at 2:34 PM, "Martin v. L?wis" wrote: >> Right, please take a look at my last version http://wiki.python.org/moin/PEP_374 >> it tries to go in that direction > > For such an infrastructure (which apparently intends to mirror the files > as well), I insist that a propagation of download counters is made > mandatory. But how do you want to display them ? Do you want to display the grand total on PyPI ? In that cas each mirror should provide a counter page, > The only mirrors that can be excused from that are private > ones. > > Regards, > Martin > -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From ziade.tarek at gmail.com Sun Oct 12 21:21:02 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 12 Oct 2008 15:21:02 -0400 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <48F246B6.8000707@v.loewis.de> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <94bdd2610810111856j4dbefcabra43dc590cb556e0d@mail.gmail.com> <48F1ADCC.5050602@zopyx.com> <5e1183fa0810120540g6e902699hfe7b177b57532f9c@mail.gmail.com> <94bdd2610810120558v31a75fc2ga9823e92f906bede@mail.gmail.com> <48F238D7.7050600@v.loewis.de> <94bdd2610810121051m40f4114ai8a9730d958bf7d50@mail.gmail.com> <48F23E11.30801@v.loewis.de> <94bdd2610810121132o34c51df8gb3acac84fa283502@mail.gmail.com> <48F246B6.8000707@v.loewis.de> Message-ID: <94bdd2610810121221se32d293yd51e51b516fd3811@mail.gmail.com> On Sun, Oct 12, 2008 at 2:49 PM, "Martin v. L?wis" wrote: >>> FWIW, it's also the case that 4940769 hits originate from >>> France. Could it be that you are alone responsible for 40% of >>> the traffic on PyPI? >>> >> >> Yes, I am the only Python developer in France. That's me. >> >> Just kidding :) >> >> France has a lot of python/plone developers that triggers buildouts every day, >> so I am pretty sure the mirrors don't make the whole traffic in PyPI. > > Hmm. Yesterday, there were 199250 accesses to PyPI through wget. > Of those, 169971 requests came from a single address (from Dedibox in > France), 28966 requests from a second one (from Sakura in Japan). yes that is us, we have two mirror here, one smart one (proxy with the minimum amount on call over PyPI) and the wget one, the second one is the wget. but the mirror option use a file listing and a timestamping mecanism, so the file or pages are downloaded only if they have changed. Basically only headers are read. I'll shut it off anyway, we have the smart collective.proxy at work now, and will eventually switch to the wget one to the pypimirror one if we provide an "official" full mirror > > So it *is* wget mirrors that make the whole traffic in PyPI. > > Regards, > Martin > -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From martin at v.loewis.de Sun Oct 12 21:23:44 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 12 Oct 2008 21:23:44 +0200 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <94bdd2610810121212xb132054y1e43f87e034fd1c2@mail.gmail.com> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <94bdd2610810111524w4cba4f70n4766fdb0b95fe73e@mail.gmail.com> <48F1396C.8030907@zopyx.com> <94bdd2610810111856j4dbefcabra43dc590cb556e0d@mail.gmail.com> <48F1ADCC.5050602@zopyx.com> <94bdd2610810120555q5ee21b59q595efffed1aff2ed@mail.gmail.com> <48F239A9.2040609@v.loewis.de> <94bdd2610810121116x3b7f7a6bw7cf0298600d33cba@mail.gmail.com> <48F24333.5070008@v.loewis.de> <94bdd2610810121212xb132054y1e43f87e034fd1c2@mail.gmail.com> Message-ID: <48F24EC0.601@v.loewis.de> > But how do you want to display them ? Do you want to display the grand total > on PyPI ? Yes, exactly so. > In that cas each mirror should provide a counter page, Why that? Shouldn't the mirrors then also display the grand total? Regards, Martin From ziade.tarek at gmail.com Sun Oct 12 21:34:20 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 12 Oct 2008 15:34:20 -0400 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <48F24EC0.601@v.loewis.de> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <48F1396C.8030907@zopyx.com> <94bdd2610810111856j4dbefcabra43dc590cb556e0d@mail.gmail.com> <48F1ADCC.5050602@zopyx.com> <94bdd2610810120555q5ee21b59q595efffed1aff2ed@mail.gmail.com> <48F239A9.2040609@v.loewis.de> <94bdd2610810121116x3b7f7a6bw7cf0298600d33cba@mail.gmail.com> <48F24333.5070008@v.loewis.de> <94bdd2610810121212xb132054y1e43f87e034fd1c2@mail.gmail.com> <48F24EC0.601@v.loewis.de> Message-ID: <94bdd2610810121234l133cac7bsb6377a87f44ae8f9@mail.gmail.com> On Sun, Oct 12, 2008 at 3:23 PM, "Martin v. L?wis" wrote: >> But how do you want to display them ? Do you want to display the grand total >> on PyPI ? > > Yes, exactly so. > >> In that cas each mirror should provide a counter page, > > Why that? Shouldn't the mirrors then also display the grand total? ok then we'll try to think about some solutions for that, > > Regards, > Martin > -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From ziade.tarek at gmail.com Sun Oct 12 22:08:27 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 12 Oct 2008 16:08:27 -0400 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <48F24EC0.601@v.loewis.de> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <48F1396C.8030907@zopyx.com> <94bdd2610810111856j4dbefcabra43dc590cb556e0d@mail.gmail.com> <48F1ADCC.5050602@zopyx.com> <94bdd2610810120555q5ee21b59q595efffed1aff2ed@mail.gmail.com> <48F239A9.2040609@v.loewis.de> <94bdd2610810121116x3b7f7a6bw7cf0298600d33cba@mail.gmail.com> <48F24333.5070008@v.loewis.de> <94bdd2610810121212xb132054y1e43f87e034fd1c2@mail.gmail.com> <48F24EC0.601@v.loewis.de> Message-ID: <94bdd2610810121308oe29debdy98a917c3a72fc211@mail.gmail.com> On Sun, Oct 12, 2008 at 3:23 PM, "Martin v. L?wis" wrote: >> But how do you want to display them ? Do you want to display the grand total >> on PyPI ? > > Yes, exactly so. > >> In that cas each mirror should provide a counter page, > > Why that? Shouldn't the mirrors then also display the grand total? how do you collect them in PyPI ? via Apache logs ? > > Regards, > Martin > -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From martin at v.loewis.de Sun Oct 12 22:32:51 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 12 Oct 2008 22:32:51 +0200 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <94bdd2610810121308oe29debdy98a917c3a72fc211@mail.gmail.com> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <48F1396C.8030907@zopyx.com> <94bdd2610810111856j4dbefcabra43dc590cb556e0d@mail.gmail.com> <48F1ADCC.5050602@zopyx.com> <94bdd2610810120555q5ee21b59q595efffed1aff2ed@mail.gmail.com> <48F239A9.2040609@v.loewis.de> <94bdd2610810121116x3b7f7a6bw7cf0298600d33cba@mail.gmail.com> <48F24333.5070008@v.loewis.de> <94bdd2610810121212xb132054y1e43f87e034fd1c2@mail.gmail.com> <48F24EC0.601@v.loewis.de> <94bdd2610810121308oe29debdy98a917c3a72fc211@mail.gmail.com> Message-ID: <48F25EF3.60509@v.loewis.de> > how do you collect them in PyPI ? via Apache logs ? Exactly. It's in tools/apache_count.py Regards, Martin From zopyxfilter at googlemail.com Sun Oct 12 23:33:17 2008 From: zopyxfilter at googlemail.com (zopyxfilter at googlemail.com) Date: Sun, 12 Oct 2008 17:33:17 -0400 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <48F2387B.8060709@v.loewis.de> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <94bdd2610810111524w4cba4f70n4766fdb0b95fe73e@mail.gmail.com> <48F1396C.8030907@zopyx.com> <94bdd2610810111856j4dbefcabra43dc590cb556e0d@mail.gmail.com> <48F1ADCC.5050602@zopyx.com> <5e1183fa0810120540g6e902699hfe7b177b57532f9c@mail.gmail.com> <48F2387B.8060709@v.loewis.de> Message-ID: <48F26D1D.4010901@gmail.com> On 12.10.2008 13:48 Uhr, Martin v. L?wis wrote: >> Can you speak more on incremental update ? >> > > What would you like to know? incremental update should be very > easy to implement for a mirror tool, with no additional changes > to PyPI. Our z3c.pypimirror already performs an incremental update based on the information available from the index.html page of the simple index and the available md5 hashes. Works like a charm... Andreas From martin at v.loewis.de Sun Oct 12 23:47:00 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 12 Oct 2008 23:47:00 +0200 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <48F26D1D.4010901@gmail.com> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <94bdd2610810111524w4cba4f70n4766fdb0b95fe73e@mail.gmail.com> <48F1396C.8030907@zopyx.com> <94bdd2610810111856j4dbefcabra43dc590cb556e0d@mail.gmail.com> <48F1ADCC.5050602@zopyx.com> <5e1183fa0810120540g6e902699hfe7b177b57532f9c@mail.gmail.com> <48F2387B.8060709@v.loewis.de> <48F26D1D.4010901@gmail.com> Message-ID: <48F27054.1060307@v.loewis.de> > Our z3c.pypimirror already performs an incremental update based on > the information available from the index.html page of the simple > index and the available md5 hashes. Works like a charm... So how does it find out when a release gets made? Regards, Martin From zopyxfilter at googlemail.com Sun Oct 12 23:52:29 2008 From: zopyxfilter at googlemail.com (zopyxfilter at googlemail.com) Date: Sun, 12 Oct 2008 17:52:29 -0400 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <48F27054.1060307@v.loewis.de> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <94bdd2610810111524w4cba4f70n4766fdb0b95fe73e@mail.gmail.com> <48F1396C.8030907@zopyx.com> <94bdd2610810111856j4dbefcabra43dc590cb556e0d@mail.gmail.com> <48F1ADCC.5050602@zopyx.com> <5e1183fa0810120540g6e902699hfe7b177b57532f9c@mail.gmail.com> <48F2387B.8060709@v.loewis.de> <48F26D1D.4010901@gmail.com> <48F27054.1060307@v.loewis.de> Message-ID: <48F2719D.9050409@gmail.com> On 12.10.2008 17:47 Uhr, Martin v. L?wis wrote: >> Our z3c.pypimirror already performs an incremental update based on >> the information available from the index.html page of the simple >> index and the available md5 hashes. Works like a charm... >> > > So how does it find out when a release gets made? > What do you mean by that? Andreas From martin at v.loewis.de Mon Oct 13 00:18:35 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 13 Oct 2008 00:18:35 +0200 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <48F2719D.9050409@gmail.com> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <94bdd2610810111524w4cba4f70n4766fdb0b95fe73e@mail.gmail.com> <48F1396C.8030907@zopyx.com> <94bdd2610810111856j4dbefcabra43dc590cb556e0d@mail.gmail.com> <48F1ADCC.5050602@zopyx.com> <5e1183fa0810120540g6e902699hfe7b177b57532f9c@mail.gmail.com> <48F2387B.8060709@v.loewis.de> <48F26D1D.4010901@gmail.com> <48F27054.1060307@v.loewis.de> <48F2719D.9050409@gmail.com> Message-ID: <48F277BB.4060105@v.loewis.de> >>> Our z3c.pypimirror already performs an incremental update based on >>> the information available from the index.html page of the simple >>> index and the available md5 hashes. Works like a charm... >>> >> >> So how does it find out when a release gets made? >> > > What do you mean by that? If you only look at http://pypi.python.org/simple/ then you have no way of find out out what changed. So "the information available from the index.html page of the simple index" is not actually suitable for building incremental mirroring. What you describe is not possible. I just looked at the z3c.pypimirror source, and found that it isn't really incremental: Whenever it mirrors, it looks at *all* index.html pages, of each an every package (all 4900 of them, except when you restrict the mirror). It then only downloads any new files that may have been added/deleted, and it *is* incremental wrt. files. IIUC, it is *not* incremental wrt. the package index itself. Please correct me if I'm wrong (and please correct z3c.pypimirror if I'm not :-) Can you please set a specific useragent header, to find out what amount of traffic pypimirror produces? Currently, urllib accounts for 17% of the requests, excluding requests made through urllib by setuptools (which is a separate 18%). It's probably not all of them through pypimirror, but of the 64626 requests made through urllib yesterday, 41671 originated from zopyx.com. For real incremental mirroring, you should retrieve the changelog, and access only those package pages that have actually changed since the last time you ran the mirror (successfully). Regards, Martin From zopyxfilter at googlemail.com Mon Oct 13 14:10:24 2008 From: zopyxfilter at googlemail.com (zopyxfilter at googlemail.com) Date: Mon, 13 Oct 2008 08:10:24 -0400 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <48F277BB.4060105@v.loewis.de> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <94bdd2610810111524w4cba4f70n4766fdb0b95fe73e@mail.gmail.com> <48F1396C.8030907@zopyx.com> <94bdd2610810111856j4dbefcabra43dc590cb556e0d@mail.gmail.com> <48F1ADCC.5050602@zopyx.com> <5e1183fa0810120540g6e902699hfe7b177b57532f9c@mail.gmail.com> <48F2387B.8060709@v.loewis.de> <48F26D1D.4010901@gmail.com> <48F27054.1060307@v.loewis.de> <48F2719D.9050409@gmail.com> <48F277BB.4060105@v.loewis.de> Message-ID: <48F33AB0.9010405@gmail.com> On 12.10.2008 18:18 Uhr, Martin v. L?wis wrote: >>>> Our z3c.pypimirror already performs an incremental update based on >>>> the information available from the index.html page of the simple >>>> index and the available md5 hashes. Works like a charm... >>>> >>>> >>> So how does it find out when a release gets made? >>> >>> >> What do you mean by that? >> > > If you only look at > > http://pypi.python.org/simple/ > > then you have no way of find out out what changed. So "the information > available from the index.html page of the simple index" is not actually > suitable for building incremental mirroring. What you describe is not > possible. > > I just looked at the z3c.pypimirror source, and found that it isn't > really incremental: Whenever it mirrors, it looks at *all* index.html > pages, of each an every package (all 4900 of them, except when you > restrict the mirror). It then only downloads any new files that may > have been added/deleted, and it *is* incremental wrt. files. IIUC, > it is *not* incremental wrt. the package index itself. > > Please correct me if I'm wrong (and please correct z3c.pypimirror > if I'm not :-) > Good suggestion. I think we can take the changelog into account easily. Having to check this with Daniel Kraft, the original author of the package. > Can you please set a specific useragent header, to find out what > amount of traffic pypimirror produces? Currently, urllib accounts for > 17% of the requests, excluding requests made through urllib by > setuptools (which is a separate 18%). It's probably not all of them > through pypimirror, but of the 64626 requests made through urllib > yesterday, 41671 originated from zopyx.com. > > Should not be a problem. > For real incremental mirroring, you should retrieve the changelog, > and access only those package pages that have actually changed since > the last time you ran the mirror (successfully). See above. Andreas From ziade.tarek at gmail.com Mon Oct 13 16:35:11 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 13 Oct 2008 10:35:11 -0400 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <48F25EF3.60509@v.loewis.de> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <48F1ADCC.5050602@zopyx.com> <94bdd2610810120555q5ee21b59q595efffed1aff2ed@mail.gmail.com> <48F239A9.2040609@v.loewis.de> <94bdd2610810121116x3b7f7a6bw7cf0298600d33cba@mail.gmail.com> <48F24333.5070008@v.loewis.de> <94bdd2610810121212xb132054y1e43f87e034fd1c2@mail.gmail.com> <48F24EC0.601@v.loewis.de> <94bdd2610810121308oe29debdy98a917c3a72fc211@mail.gmail.com> <48F25EF3.60509@v.loewis.de> Message-ID: <94bdd2610810130735r779745cdr70fd5e0afcfcdbc0@mail.gmail.com> On Sun, Oct 12, 2008 at 4:32 PM, "Martin v. L?wis" wrote: >> how do you collect them in PyPI ? via Apache logs ? > > Exactly. It's in tools/apache_count.py How often do you run it ? I guess a daily update is enough for the grand total ? Anyway, so the mirrors should be able to reuse this script for their own internal count as well, and PyPI would need to provide a way for the mirror to report them, and to get back the grand count. but i wouldn't want to make apache mandatory for the mirrors. What about this: 1/ each mirror maintain simple text-based stats pages, with the local count, reachable from an url (/local_stats) 2/ PyPI modifies its script so it injects its apache count + the registered mirrors local counts 3/ PyPI maintains a simple text stats page, with the grand count (/stats) one stat page represents one day, and the stats are presented in folders that represents the year and the month So the stats from october the 11th will be reachable at: .../local_stats/2008/10/11 The stat page can referer to the packages using a PACKAGE_NAME/FILE = HITS syntax: iw.recipe.fss/iw.recipe.fss-0.2.1.tar.gz = 123 foo.bar/foo.bar-0.3.tar.gz = 12 ... This is a fairly simple structure any mirroring tool can create, and we could provide a simple python script that generates it from the Apache logs, Regards Tarek > > Regards, > Martin > -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From p.f.moore at gmail.com Mon Oct 13 22:10:08 2008 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 13 Oct 2008 21:10:08 +0100 Subject: [Distutils] Setuptools Windows binary for Python 2.6 Message-ID: <79990c6b0810131310k632e3804idef9d7c43d2646aa@mail.gmail.com> I see that there's a Win32 egg for setuptools for Python 2.6, but no bdist_wininst style installer. Will a bdist_wininst installer be provided? When is it likely to be available? Paul. From santagada at gmail.com Mon Oct 13 21:13:59 2008 From: santagada at gmail.com (Leonardo Santagada) Date: Mon, 13 Oct 2008 17:13:59 -0200 Subject: [Distutils] Implementing the reporthook on setuptools Message-ID: Sidnei gave me the idea to implement a progress status while downloading packages with setuptools. There is an empty reporthook on package_index.py. If I understood it correctly we could override it to show a progress report while it is downloading a package. The problem is that setuptools don't print directly to the stdout but uses a logging infrastructure (the one from distutils) to print. This way it is impossible for example to print a '.' at each block received (like wget does). I want to know if you guys think this is interesting and if it is, if anyone has any idea on how to do it? -- Leonardo Santagada santagada at gmail.com From martin at v.loewis.de Mon Oct 13 23:35:34 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 13 Oct 2008 23:35:34 +0200 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <94bdd2610810130735r779745cdr70fd5e0afcfcdbc0@mail.gmail.com> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <48F1ADCC.5050602@zopyx.com> <94bdd2610810120555q5ee21b59q595efffed1aff2ed@mail.gmail.com> <48F239A9.2040609@v.loewis.de> <94bdd2610810121116x3b7f7a6bw7cf0298600d33cba@mail.gmail.com> <48F24333.5070008@v.loewis.de> <94bdd2610810121212xb132054y1e43f87e034fd1c2@mail.gmail.com> <48F24EC0.601@v.loewis.de> <94bdd2610810121308oe29debdy98a917c3a72fc211@mail.gmail.com> <48F25EF3.60509@v.loewis.de> <94bdd2610810130735r779745cdr70fd5e0afcfcdbc0@mail.gmail.com> Message-ID: <48F3BF26.4050905@v.loewis.de> > How often do you run it ? I guess a daily update is enough for the grand total ? I think it still runs daily. There was one complaint about that, but the user could accept that as a policy after understanding what happened (he thought the feature was broken as there was no immediate update). > 1/ each mirror maintain simple text-based stats pages, with the local > count, reachable from an url (/local_stats) > 2/ PyPI modifies its script so it injects its apache count + the > registered mirrors local counts > 3/ PyPI maintains a simple text stats page, with the grand count (/stats) Sounds fine to me. Expect that to become a long file, though, with one line per file (roughly 20000 files with at least one download). > one stat page represents one day, and the stats are presented in > folders that represents the year and the month I wonder whether it might be easier to have a single file, with the totals for that server. > iw.recipe.fss/iw.recipe.fss-0.2.1.tar.gz = 123 > foo.bar/foo.bar-0.3.tar.gz = 12 I would drop the "=" in that syntax. Regards, Martin From tarek.ziade at ingeniweb.com Mon Oct 13 23:58:57 2008 From: tarek.ziade at ingeniweb.com (Tarek Ziade) Date: Mon, 13 Oct 2008 17:58:57 -0400 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <48F3BF26.4050905@v.loewis.de> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <48F239A9.2040609@v.loewis.de> <94bdd2610810121116x3b7f7a6bw7cf0298600d33cba@mail.gmail.com> <48F24333.5070008@v.loewis.de> <94bdd2610810121212xb132054y1e43f87e034fd1c2@mail.gmail.com> <48F24EC0.601@v.loewis.de> <94bdd2610810121308oe29debdy98a917c3a72fc211@mail.gmail.com> <48F25EF3.60509@v.loewis.de> <94bdd2610810130735r779745cdr70fd5e0afcfcdbc0@mail.gmail.com> <48F3BF26.4050905@v.loewis.de> Message-ID: 2008/10/13 "Martin v. L?wis" > > How often do you run it ? I guess a daily update is enough for the grand > total ? > > I think it still runs daily. There was one complaint about that, but the > user could accept that as a policy after understanding what happened (he > thought the feature was broken as there was no immediate update). > > > 1/ each mirror maintain simple text-based stats pages, with the local > > count, reachable from an url (/local_stats) > > 2/ PyPI modifies its script so it injects its apache count + the > > registered mirrors local counts > > 3/ PyPI maintains a simple text stats page, with the grand count > (/stats) > > Sounds fine to me. Expect that to become a long file, though, with one > line per file (roughly 20000 files with at least one download). Maybe we could use one subfolder per alphabet letter, like what is done in packages/ at PyPI that would lower it down to roughly 1000 items per pages, > > > one stat page represents one day, and the stats are presented in > > folders that represents the year and the month > > I wonder whether it might be easier to have a single file, with the > totals for that server. You would need to specify a timestamp for each single download though, to make sure PyPI knows which hits to count, depending on the last date it checked the mirror. if we have 1000 downloads per day, that's a huge file after a while > > > > iw.recipe.fss/iw.recipe.fss-0.2.1.tar.gz = 123 > > foo.bar/foo.bar-0.3.tar.gz = 12 > > I would drop the "=" in that syntax. > Ok I'll upgrade the proposal, reflecting these infos > Regards, > Martin > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -- Tarek Ziad? - Directeur Technique INGENIWEB (TM) - SAS 50000 Euros - RC B 438 725 632 Bureaux de la Colline - 1 rue Royale - B?timent D - 9?me ?tage 92210 Saint Cloud - France Phone : 01.78.15.24.00 / Fax : 01 46 02 44 04 http://www.ingeniweb.com - une soci?t? du groupe Alter Way -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Tue Oct 14 00:16:56 2008 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 14 Oct 2008 00:16:56 +0200 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <48F239A9.2040609@v.loewis.de> <94bdd2610810121116x3b7f7a6bw7cf0298600d33cba@mail.gmail.com> <48F24333.5070008@v.loewis.de> <94bdd2610810121212xb132054y1e43f87e034fd1c2@mail.gmail.com> <48F24EC0.601@v.loewis.de> <94bdd2610810121308oe29debdy98a917c3a72fc211@mail.gmail.com> <48F25EF3.60509@v.loewis.de> <94bdd2610810130735r779745cdr70fd5e0afcfcdbc0@mail.gmail.com> <48F3BF26.4050905@v.loewis.de> Message-ID: <48F3C8D8.4020804@v.loewis.de> > Maybe we could use one subfolder per alphabet letter, Would that simplify anything? PyPI uses one directory per letter to reduce the number of files in a single directory, in case ext3 doesn't deal with large directories well. For the stats, the "large directories" argument wouldn't count. OTOH, if you do have separate pages per letter, the master server would still need to download all individual files. Having them split into chunks just increases the load, rather than reducing it. > You would need to specify a timestamp for each single download though, > to make sure PyPI > knows which hits to count, depending on the last date it checked the > mirror. No. It would just compute the grand total from scratch each time. Regards, Martin From ziade.tarek at gmail.com Tue Oct 14 02:34:00 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 13 Oct 2008 20:34:00 -0400 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <48F3C8D8.4020804@v.loewis.de> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <48F24333.5070008@v.loewis.de> <94bdd2610810121212xb132054y1e43f87e034fd1c2@mail.gmail.com> <48F24EC0.601@v.loewis.de> <94bdd2610810121308oe29debdy98a917c3a72fc211@mail.gmail.com> <48F25EF3.60509@v.loewis.de> <94bdd2610810130735r779745cdr70fd5e0afcfcdbc0@mail.gmail.com> <48F3BF26.4050905@v.loewis.de> <48F3C8D8.4020804@v.loewis.de> Message-ID: <94bdd2610810131734k59be8e2al71db3e142e9244c9@mail.gmail.com> On Mon, Oct 13, 2008 at 6:16 PM, "Martin v. L?wis" wrote: >> Maybe we could use one subfolder per alphabet letter, > > Would that simplify anything? > > PyPI uses one directory per letter to reduce the number of files in a > single directory, in case ext3 doesn't deal with large directories well. > For the stats, the "large directories" argument wouldn't count. > > OTOH, if you do have separate pages per letter, the master server would > still need to download all individual files. Having them split into > chunks just increases the load, rather than reducing it. Yes I thaught you were concerned by the size of that file, rather by the number of calls PyPI would need to perform. > > >> You would need to specify a timestamp for each single download though, >> to make sure PyPI >> knows which hits to count, depending on the last date it checked the >> mirror. > > No. It would just compute the grand total from scratch each time. > ok OTHO you would lose an interesting info: how downloads evolve in time. As a packager, I can see some interesting use cases. For example when foo 2.0 gets out, I can watch foo 1.0 downloads decrease and foo 2.0 raise. (if not make sure i have promoted 2.0 correctly) People would be able to generate interesting statistics tools from there. This would be possible of course only if PyPI provides the same timestamped pages for the grand total. This leads to another point we did not discuss yet: it would be interesting to keep the user-agent info in the mirrors, and make sure all automatic-package-grabbing softwares out there have there own user agent id For instance, knowing that 90% of the downloads of a given package where done by zc.buildout is interesting. IIRC, we cannot know it right now, and I could work on zc.buildout side for that, because it uses the setuptools user agent id Regards Tarek > > Regards, > Martin > -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From martin at v.loewis.de Tue Oct 14 06:59:54 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 14 Oct 2008 06:59:54 +0200 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <94bdd2610810131734k59be8e2al71db3e142e9244c9@mail.gmail.com> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <48F24333.5070008@v.loewis.de> <94bdd2610810121212xb132054y1e43f87e034fd1c2@mail.gmail.com> <48F24EC0.601@v.loewis.de> <94bdd2610810121308oe29debdy98a917c3a72fc211@mail.gmail.com> <48F25EF3.60509@v.loewis.de> <94bdd2610810130735r779745cdr70fd5e0afcfcdbc0@mail.gmail.com> <48F3BF26.4050905@v.loewis.de> <48F3C8D8.4020804@v.loewis.de> <94bdd2610810131734k59be8e2al71db3e142e9244c9@mail.gmail.com> Message-ID: <48F4274A.9060601@v.loewis.de> > OTHO you would lose an interesting info: how downloads evolve in > time. Users interested in that could produce that information themselves, though: they query the download stats once a day, and compute the first derivative. > For instance, knowing that 90% of the downloads of a given package > where done by zc.buildout is interesting. IIRC, we cannot know it > right now, and I could work on zc.buildout side for that, because it > uses the setuptools user agent id In principle, it would be fine with me to track that, as long as I don't need to preserve the the complete log files. So would the stats file then be of the form package,filename,useragent,count ? (commas in the useragent replaced with semicolons) Regards, Martin From ziade.tarek at gmail.com Tue Oct 14 14:02:05 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 14 Oct 2008 14:02:05 +0200 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <48F4274A.9060601@v.loewis.de> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <48F24EC0.601@v.loewis.de> <94bdd2610810121308oe29debdy98a917c3a72fc211@mail.gmail.com> <48F25EF3.60509@v.loewis.de> <94bdd2610810130735r779745cdr70fd5e0afcfcdbc0@mail.gmail.com> <48F3BF26.4050905@v.loewis.de> <48F3C8D8.4020804@v.loewis.de> <94bdd2610810131734k59be8e2al71db3e142e9244c9@mail.gmail.com> <48F4274A.9060601@v.loewis.de> Message-ID: <94bdd2610810140502k6632e640o8e1466c872c72223@mail.gmail.com> On Tue, Oct 14, 2008 at 12:59 AM, "Martin v. L?wis" wrote: >> OTHO you would lose an interesting info: how downloads evolve in >> time. > > Users interested in that could produce that information themselves, > though: they query the download stats once a day, and compute the > first derivative. ok > >> For instance, knowing that 90% of the downloads of a given package >> where done by zc.buildout is interesting. IIRC, we cannot know it >> right now, and I could work on zc.buildout side for that, because it >> uses the setuptools user agent id > > In principle, it would be fine with me to track that, as long as I > don't need to preserve the the complete log files. > > So would the stats file then be of the form > > package,filename,useragent,count > > ? (commas in the useragent replaced with semicolons) > sounds good, i'll write that down in the proposal as well last points I can think of, that we have discussed at the sprint: - are non open source licensed packages alllowed at PyPI ? - wouldn't it make sense for open source package to force a sdist upload before any other kind of distribution (this is a feature claimed by many people in fact, as binary distribution obsfuscate things and make it hard to install if it's not the same version, and if it was not intended by the packager) Regards > Regards, > Martin > > > -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From tseaver at palladion.com Tue Oct 14 14:03:30 2008 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 14 Oct 2008 08:03:30 -0400 Subject: [Distutils] distribute D.C. sprint tasks In-Reply-To: <48F4274A.9060601@v.loewis.de> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <48F24333.5070008@v.loewis.de> <94bdd2610810121212xb132054y1e43f87e034fd1c2@mail.gmail.com> <48F24EC0.601@v.loewis.de> <94bdd2610810121308oe29debdy98a917c3a72fc211@mail.gmail.com> <48F25EF3.60509@v.loewis.de> <94bdd2610810130735r779745cdr70fd5e0afcfcdbc0@mail.gmail.com> <48F3BF26.4050905@v.loewis.de> <48F3C8D8.4020804@v.loewis.de> <94bdd2610810131734k59be8e2al71db3e142e9244c9@mail.gmail.com> <48F4274A.9060601@v.loewis.de> Message-ID: <48F48A92.6010106@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Martin v. L?wis wrote: >> OTHO you would lose an interesting info: how downloads evolve in >> time. > > Users interested in that could produce that information themselves, > though: they query the download stats once a day, and compute the > first derivative. > >> For instance, knowing that 90% of the downloads of a given package >> where done by zc.buildout is interesting. IIRC, we cannot know it >> right now, and I could work on zc.buildout side for that, because it >> uses the setuptools user agent id > > In principle, it would be fine with me to track that, as long as I > don't need to preserve the the complete log files. > > So would the stats file then be of the form > > package,filename,useragent,count > > ? (commas in the useragent replaced with semicolons) Why not just use the csv module for that, and let it handle the escaping? Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFI9IqS+gerLs4ltQ4RAnNPAKDUZ+TuDjTt8yL4ncf78DCeSYiXsACfYjB2 p/l1QhaPSovWJHMdL+JcqrU= =4ClS -----END PGP SIGNATURE----- From martin at v.loewis.de Tue Oct 14 19:52:52 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 14 Oct 2008 19:52:52 +0200 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <94bdd2610810140502k6632e640o8e1466c872c72223@mail.gmail.com> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <48F24EC0.601@v.loewis.de> <94bdd2610810121308oe29debdy98a917c3a72fc211@mail.gmail.com> <48F25EF3.60509@v.loewis.de> <94bdd2610810130735r779745cdr70fd5e0afcfcdbc0@mail.gmail.com> <48F3BF26.4050905@v.loewis.de> <48F3C8D8.4020804@v.loewis.de> <94bdd2610810131734k59be8e2al71db3e142e9244c9@mail.gmail.com> <48F4274A.9060601@v.loewis.de> <94bdd2610810140502k6632e640o8e1466c872c72223@mail.gmail.com> Message-ID: <48F4DC74.4010308@v.loewis.de> > - are non open source licensed packages alllowed at PyPI ? Sure! There is no censorship applied in PyPI, except for content completely unrelated to Python. The only exception is when the package owner is unresponsive, and somebody else wants to take over the package. We need some procedure to formalize this case. > - wouldn't it make sense for open source package to force a sdist > upload before any other kind of distribution (this is a feature > claimed by many people in fact, as binary distribution obsfuscate > things and make it hard to install if it's not the same version, and > if it was not intended by the packager) I don't want to assert quality control to the packages. If they don't upload anything, fine. If they upload broken packages, fine. If they supply invalid URLs, fine. If they mistype their email addresses, names, or licensing terms, fine. If they fail to provide source code, fine. Users should contact the authors and report problems with the registration if they find any. They sometimes mistake the PyPI tracker as tracking problems with the packages, but that happens rarely. Perhaps we can provide a form to submit a message to the package owner, to be used when everything else fails (such form would require a PyPI account for the sender, too). Regards, Martin From martin at v.loewis.de Tue Oct 14 21:21:58 2008 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 14 Oct 2008 21:21:58 +0200 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <48F24EC0.601@v.loewis.de> <94bdd2610810121308oe29debdy98a917c3a72fc211@mail.gmail.com> <48F25EF3.60509@v.loewis.de> <94bdd2610810130735r779745cdr70fd5e0afcfcdbc0@mail.gmail.com> <48F3BF26.4050905@v.loewis.de> <48F3C8D8.4020804@v.loewis.de> <94bdd2610810131734k59be8e2al71db3e142e9244c9@mail.gmail.com> <48F4274A.9060601@v.loewis.de> <94bdd2610810140502k6632e640o8e1466c872c72223@mail.gmail.com> <48F4DC74.4010308@v.loewis.de> Message-ID: <48F4F156.1050909@v.loewis.de> > If you are going to formalize such procedures, it may also be worth > thinking about the case of someone uploading code that they do not have > rights to. It's not very likely, but the response often must be swift, > at least in some jurisdictions. The potential for mirrors, and thus the > need for coordinated action, makes it even more important to formalize a > procedure, IMO. The procedure for such a case is fairly obvious: when the true copyright holder requests removal (e.g. through the bug tracker, or by email to the PSF, which appears as the responsible entity of the website), the files would be deleted immediately, and the uploader would be requested to clarify. Regards, Martin From chris at simplistix.co.uk Wed Oct 15 15:05:07 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 15 Oct 2008 14:05:07 +0100 Subject: [Distutils] developing a package and two packages it depends on Message-ID: <48F5EA83.2070801@simplistix.co.uk> Hi All, I have a package that has a setup.py containing the following: setup( name='xlutils', ... install_requires=[ 'xlrd', 'xlwt', ], ) I'm using buildout to setup and development and testing environment for this project so I have a buildout.cfg containing the following: [buildout] develop = . parts = test py [py] recipe = zc.recipe.egg eggs = xlutils interpreter = py [test] recipe = zc.recipe.testrunner eggs = xlutils How can I get the above to use a checkout of xlrd and xlwt's trunk? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From marius at pov.lt Wed Oct 15 16:19:33 2008 From: marius at pov.lt (Marius Gedminas) Date: Wed, 15 Oct 2008 17:19:33 +0300 Subject: [Distutils] developing a package and two packages it depends on In-Reply-To: <48F5EA83.2070801@simplistix.co.uk> References: <48F5EA83.2070801@simplistix.co.uk> Message-ID: <20081015141933.GA17850@fridge.pov.lt> On Wed, Oct 15, 2008 at 02:05:07PM +0100, Chris Withers wrote: > I have a package that has a setup.py containing the following: > > setup( > name='xlutils', > ... > install_requires=[ > 'xlrd', > 'xlwt', > ], > ) > > I'm using buildout to setup and development and testing environment for > this project so I have a buildout.cfg containing the following: > > [buildout] > develop = . > parts = test py > > [py] > recipe = zc.recipe.egg > eggs = xlutils > interpreter = py > > [test] > recipe = zc.recipe.testrunner > eggs = xlutils > > How can I get the above to use a checkout of xlrd and xlwt's trunk? Use svn:externals to check them out inside your tree, then include those directories in the 'develop' line in buildout.cfg. (And if you want your users, i.e., people who easy_install xlutils from PyPI, to get trunk checkouts from xlrd and xlwt, then you want the wrong thing and should reconsider.) Marius Gedminas -- We don't really understand it, so we'll give it to the programmers. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From chris at simplistix.co.uk Wed Oct 15 17:45:47 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 15 Oct 2008 16:45:47 +0100 Subject: [Distutils] developing a package and two packages it depends on In-Reply-To: <20081015141933.GA17850@fridge.pov.lt> References: <48F5EA83.2070801@simplistix.co.uk> <20081015141933.GA17850@fridge.pov.lt> Message-ID: <48F6102B.5060905@simplistix.co.uk> Marius Gedminas wrote: > Use svn:externals to check them out inside your tree, then include those > directories in the 'develop' line in buildout.cfg. Okay, but how then do I check the library works with the released versions of the xlrd and xlwt packages? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From ziade.tarek at gmail.com Wed Oct 15 18:18:23 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 15 Oct 2008 18:18:23 +0200 Subject: [Distutils] developing a package and two packages it depends on In-Reply-To: <48F6102B.5060905@simplistix.co.uk> References: <48F5EA83.2070801@simplistix.co.uk> <20081015141933.GA17850@fridge.pov.lt> <48F6102B.5060905@simplistix.co.uk> Message-ID: <94bdd2610810150918x1b655fffo3a51cb80f20d1635@mail.gmail.com> On Wed, Oct 15, 2008 at 5:45 PM, Chris Withers wrote: > Marius Gedminas wrote: >> >> Use svn:externals to check them out inside your tree, then include those >> directories in the 'develop' line in buildout.cfg. > > Okay, but how then do I check the library works with the released versions > of the xlrd and xlwt packages? it's a development and release process matter. Most people have two .cfg files : - one for development (dev.cfg) - one for production (prod.cfg) where you don't leave any develop option of course. You can also have another cfg for your developments, that points to the released versions of the third party packages if needed in your development process. And all those cfg extends buildout.cfg. Regards > > cheers, > > Chris > > -- > Simplistix - Content Management, Zope & Python Consulting > - http://www.simplistix.co.uk > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From m.pricejones at gmail.com Thu Oct 16 19:25:24 2008 From: m.pricejones at gmail.com (Michael Jones) Date: Thu, 16 Oct 2008 18:25:24 +0100 Subject: [Distutils] Separate source and build folders Message-ID: <8c8775e00810161025w2f5b5bd0gc91aab3e999eb999@mail.gmail.com> Hi, I've got a directory structure that looks like: ./setup.py ./src ./src/dn where dn is a package containing my code. My setup.py looks like this: ------- from setuptools import setup, find_packages setup( name = "dnpkg", version = "0.1.7", packages = ['dn', 'dn.pkg', 'dn.pkg.interface'], package_dir = { "": "src"}, namespace_packages = ['dn'], install_requires = ['SQLAlchemy>=0.4.2p3'], entry_points = { 'console_scripts': [ 'dnpkg.test = dn.pkg.interface.commandline:main', 'dnpkg.testweb = dn.pkg.interface.web:cgi', ] }, package_data = { 'dn.pkg' : [ 'templates/user/*.mako', 'templates/show/*.mako', 'templates/project/*.mako', 'templates/*.mako' ], } ) ------- As indicated I have two console scripts. This setup works fine the console scripts are installed as desired. However I would like to have a separate build directory so the layout matches my other (non-python) projects. So I created a build directory, put my setup.py in there and so I have: ./build ./build/setup.py ./src ./src/dn And I changed the package_dir line in setup.py to "../src" instead of "src". Now it appears ok, running "python2.5 setup.py bdist_egg" in the build directory does much the same as in the old directory structure. But when I do an install, I no longer get the lines: Installing dnpkg.test script to Installing dnpkg.testweb script to and the scripts don't appear where they should. What am I missing? I've been reading the docs but can't figure out where I'm going wrong. Any help would be much appreciated, Michael From a.badger at gmail.com Fri Oct 17 04:01:34 2008 From: a.badger at gmail.com (Toshio Kuratomi) Date: Thu, 16 Oct 2008 19:01:34 -0700 Subject: [Distutils] Pre-pre-PEP: Requirements for the Python BUILDS Specification In-Reply-To: <48EFA036.6000402@colorstudy.com> References: <20081003163402.638A93A407A@sparrow.telecommunity.com> <20081006212511.GC17624@laurie.devork> <20081007013212.02F773A4045@sparrow.telecommunity.com> <79990c6b0810070204r4b2fb216la1daa69e36ba78f0@mail.gmail.com> <20081007182934.40C923A4045@sparrow.telecommunity.com> <79990c6b0810071416m3f841e27ve4d2880e4ccc810d@mail.gmail.com> <48EFA036.6000402@colorstudy.com> Message-ID: <48F7F1FE.7080509@gmail.com> Ian Bicking wrote: > Paul Moore wrote: >> My feeling, by the way, is that "system packagers" are the more >> relevant group on Linux/Unix (where most users install Python modules >> via system packages, or else they are developers) > > I think this is part of why I don't understand the system packager > perspective. Developers shouldn't use system packages, it just doesn't > make any sense to have that intermediation. Users don't use Python > modules, they use applications. Users only care that their applications > work, that they can install applications without unnecessary conflicts, > that the applications don't break based on unintentional environment > changes (e.g., the value of PYTHONPATH). > > Packagers seem to care a great deal about having applications share > libraries on the packaging level, but this is for their own accounting, > there's no reason for users to care (except for the too-small-to-matter > issue of disk space). Also, packagers seem to jump the gun on this > library sharing, as they are concerned about libraries when one (or > often zero!) applications depend on the library. Some widely used > libraries seem reasonable, but for every widely used library there are a > dozen or more niche libraries. There is a great deal of incentive for everything to use system libraries. take formencode-1.0 vs formencode-1.0.1 for instance. If every TurboGears application and whatnot that used formencode were using their own copy of formencode-1.0 and then the discovery that formencode-1.0 wasn't doing chained validators correctly came out, the end users have to go about finding every copy of formencode and updating it. If there is only a single formencode library on the system and all the TurboGears apps are using it, that's a single package that needs to be updated. > Users also don't care about /usr/share > or /usr/lib -- the only thing *I* ever care about is /usr/share/doc, > /usr/bin, /etc, and maybe a man page. > They do when user == system administrator. Ask someone who manages Linux systems at a large University if they care about the FHS and the answer will be yes. The various distinctions in the FHS between writable vs nonwritable, architecture dependent vs architecture independent, etc allow system administrators to maintain a large number of systems with less pain. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: OpenPGP digital signature URL: From a.badger at gmail.com Fri Oct 17 06:47:34 2008 From: a.badger at gmail.com (Toshio Kuratomi) Date: Thu, 16 Oct 2008 21:47:34 -0700 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <48F24333.5070008@v.loewis.de> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <94bdd2610810111524w4cba4f70n4766fdb0b95fe73e@mail.gmail.com> <48F1396C.8030907@zopyx.com> <94bdd2610810111856j4dbefcabra43dc590cb556e0d@mail.gmail.com> <48F1ADCC.5050602@zopyx.com> <94bdd2610810120555q5ee21b59q595efffed1aff2ed@mail.gmail.com> <48F239A9.2040609@v.loewis.de> <94bdd2610810121116x3b7f7a6bw7cf0298600d33cba@mail.gmail.com> <48F24333.5070008@v.loewis.de> Message-ID: <48F818E6.7020101@gmail.com> Martin v. L?wis wrote: >> Right, please take a look at my last version http://wiki.python.org/moin/PEP_374 >> it tries to go in that direction > > For such an infrastructure (which apparently intends to mirror the files > as well), I insist that a propagation of download counters is made > mandatory. The only mirrors that can be excused from that are private > ones. This may not apply to pypi as the sites you get to mirror you may be different but Linux distributions have found that it is easier to get mirrors if the mirror admins can run as little custom stuff as possible. ie: If they can retrieve content from the master mirror via a simple rsync cron job that they write they are happiest. We have found other ways to generate statistics regarding download in these cases (for instance, based upon how many calls to retrieve the mirrorlist or how many calls for specific packages via the mirror redirector). As I say, whether this is a problem for you will depend on the willingness of the sites that are mirroring you to run your scripts with code that you've written rather than themselves. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: OpenPGP digital signature URL: From techtonik at gmail.com Fri Oct 17 11:32:29 2008 From: techtonik at gmail.com (anatoly techtonik) Date: Fri, 17 Oct 2008 12:32:29 +0300 Subject: [Distutils] [patch] executable Scripts on windows Message-ID: Hello, I would like to discuss a patch that adds executable .bat files for scripts installed with distutils. As a windows user with several Python versions installed I often create shortcuts for Python applications that were installed using distutils. But I also have to navigate to Scripts/ directory and mess with script files. Common problems: 1. python script doesn't have .py extension 2. script is actually a shell script 3. script is executed with wrong Python version The first problem is clear. Files without extension are not executable on windows. If distutils makes scripts executable on posix, why shouldn't it do the same on windows? The second problem should be addressed to package maintainer in the first place. The third one is also quite common - distutils fixes scripts and inserts correct Python path in shebang even on windows platform, but it makes little sense, because the script is executed with the only interpreter that was associated with .py extensions. To solve problems 1 and 3 I added patch that adds .bat file to execute the script with correct python version. As a positive side effect - the application can be executed through "Run" menu. The patch is posted into http://bugs.python.org/issue4015 and I would like to know your opinion about it. -- --anatoly t. From a.badger at gmail.com Fri Oct 17 19:32:35 2008 From: a.badger at gmail.com (Toshio Kuratomi) Date: Fri, 17 Oct 2008 10:32:35 -0700 Subject: [Distutils] Symlinks vs API -- question for developers Message-ID: <48F8CC33.7090106@gmail.com> So I have a question for all the developers on this list. Philip thinks that using symlinks will drive adoption better than an API to access package data. I think an API will have better adoption than a symlink hack. But the real question is what do people who maintain packages think? Since Philip's given his reasoning, here's mine: 1) Philip says that with symlinks distributions will likely have to submit patches to the build scripts to tag various files as belonging to certain categories. If you, as an upstream are going to accept a patch to your build scripts to place files in a different place wouldn't you also accept a patch to your source code to use a well defined API to pull files from a different source? This is a distribution's bread and butter and if there's a small, useful, well-liked, standard API for accessing data files you will start receiving patches from distributions that want to help you help them. 2) Symlinks cannot be used universally. Although it might not be common to want an FHS style install in such an environment, it isn't unheard of. At one time in the distant past I had to use cygwin so I know that while this may be a corner case, it does exist. 3) The primary argument for symlinks is that symlinks are compatible with __file__. But this compatibility comes at a cost -- symlinks can't do anything extra. In a different subthread Philip argues that setuptools provides more than distutils and that's why people switch and that the next generation tool needs to provide even more than setuptools. Symlinks cannot do that. 4) In contrast an API can do more: It can deal with writable files. On Unix, persistent, per user storage would go in the user's home directory, on other OS's it would go somewhere else. This is abstractable using an API at runtime but not using symlinks at install time. 5) cross package data. Using __file__ to detect file location is inherently not suitable for crossing package boundaries. Egg Translations would not be able to use a symlink based backend to do its work for this reason. 6) zipped eggs. These require an API. So moving to symlinks is actually a regression. 7) Philip says that the reason pkg_resources does not see widespread adoption is that the developer cost of using an API is too high compared to __file__. I don't believe that the difference between file and API is that great. An example of using an API could be something like this: Symlinks:: import os icondirectory = os.path.join(os.path.basename(__file__), 'icons') API:: import pkgdata icondirectory = pkgdata.resource(pkg='setuptools', \ category='icon', resource='setuptools.png') Instead I think the data handling portion of pkg_resources is not more widely adopted for these reasons: * pkg_resources's package handling is painful for the not-infrequent corner cases. So people who have encountered the problems with require() not overriding a default or not selecting the proper version when multiple packages specify overlapping version ranges already have a negative impression of the library before they even get to the data handling portion. * pkg_resources does too much: loading libraries by version really has nothing to do with loading data for use by a library. This is a drawback because people think of and promote pkg_resources as a way to enable easy_install rather than a way to enable abstraction of data location. * The only benefit (at least, being promoted in the documentation) is to allow zipped eggs to work. Distributions have no reason to create zipped eggs so they have no reason to submit patches to upstream to support the pkg_resources api. * Distributions, further, don't want to install all-in-one egg directories on the system. The pkg_resources API just gets in the way of doing things correctly in a distribution. I've had to patch code to not use pkg_resources if data is installed in the FHS mandated areas. Far from encouraging distributions to send patches upstream to make modules use pkg_resources this makes distributions actively discourage upstreams from using it. * The API isn't flexible enough. EggTranslations places its data within the metadata store of eggs instead of within the data store. This is because the metadata is able to be read outside of the package in which it is included while the package data can only be accessed from within the package. 8) To a distribution, symlinks are just a hack. We use them for things like php web apps when the web application is hardcoded to accept only one path for things (like the writable state files being intermixed with the program code). Managing a symlink farm is not something distributions are going to get excited over so adoption by distributions that this is the way to work with files won't happen until upstreams move on their own. Further, since the install tool is being proposed as a separate project from the metadata to mark files, the expectation is that the distributions are going to want to write an install tool that manages this symlink farm. For that to happen, you have to get distributions to be much more than simply neutral about the idea of symlinks, you have to have them enthused enough about using symlinks that they are willing to spend time writing a tool to do it. So once again, I think this boils down to these questions: if we have a small library whose sole purpose is to abstract a data store so you can find out where a particular non-code file lives on this system will you use it? If a distribution packager sends you a patch so the data files are marked correctly and the code can retrieve their location instead of hardcoding an offset against __file__ will you commit it? -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: OpenPGP digital signature URL: From barry at python.org Fri Oct 17 19:48:34 2008 From: barry at python.org (Barry Warsaw) Date: Fri, 17 Oct 2008 13:48:34 -0400 Subject: [Distutils] Symlinks vs API -- question for developers In-Reply-To: <48F8CC33.7090106@gmail.com> References: <48F8CC33.7090106@gmail.com> Message-ID: <4D65936D-3885-4855-8BFF-761C21E5F4DB@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Oct 17, 2008, at 1:32 PM, Toshio Kuratomi wrote: > 7) Philip says that the reason pkg_resources does not see widespread > adoption is that the developer cost of using an API is too high > compared > to __file__. I don't believe that the difference between file and API > is that great. An example of using an API could be something like > this: > > Symlinks:: > import os > icondirectory = os.path.join(os.path.basename(__file__), 'icons') s/basename/dirname/ I think. > > > API:: > import pkgdata > icondirectory = pkgdata.resource(pkg='setuptools', \ > category='icon', resource='setuptools.png') Having tried to be religious about using pkg_resources instead of __file__ in all my new code, I tend to agree that the API cost is not that high. I don't particularly like the verbosity of the names chosen, but I actually like not having to use the __file__ idiom. > * Distributions, further, don't want to install all-in-one egg > directories on the system. The pkg_resources API just gets in the way > of doing things correctly in a distribution. I've had to patch code > to > not use pkg_resources if data is installed in the FHS mandated areas. > Far from encouraging distributions to send patches upstream to make > modules use pkg_resources this makes distributions actively discourage > upstreams from using it. I hadn't thought of this, but yes, this is a serious negative. > So once again, I think this boils down to these questions: if we > have a > small library whose sole purpose is to abstract a data store so you > can > find out where a particular non-code file lives on this system will > you > use it? I would. I apologize for not having followed the discussion that closely, but as an application developer, I would really like an API that hides all the location nonsense from me. As familiar as __file__ is, it's a fragile hack. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSPjP8nEjvBPtnXfVAQKOeQP/eJywpz1CUxJhUD9NhUj68rpoHbato8W4 fP2ZNRmKOSGUmtaj9hM1vduMoCszCN/vz8fX+gGZFu9ySkWyQfO5Q6Hh/kBrKSRN IzVYcd3lbV+e63+twk3Ht4gSX8j2iWnt375976kFgvmMc2iB7zn0r/TDblMIqvxV NUUi3d3zaPs= =5yrx -----END PGP SIGNATURE----- From ianb at colorstudy.com Fri Oct 17 22:49:05 2008 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 17 Oct 2008 16:49:05 -0400 Subject: [Distutils] Symlinks vs API -- question for developers In-Reply-To: <48F8CC33.7090106@gmail.com> References: <48F8CC33.7090106@gmail.com> Message-ID: <48F8FA41.2030309@colorstudy.com> Toshio Kuratomi wrote: > So I have a question for all the developers on this list. Philip thinks > that using symlinks will drive adoption better than an API to access > package data. I think an API will have better adoption than a symlink > hack. But the real question is what do people who maintain packages > think? Since Philip's given his reasoning, here's mine: > > 1) Philip says that with symlinks distributions will likely have to > submit patches to the build scripts to tag various files as belonging to > certain categories. If you, as an upstream are going to accept a patch > to your build scripts to place files in a different place wouldn't you > also accept a patch to your source code to use a well defined API to > pull files from a different source? This is a distribution's bread and > butter and if there's a small, useful, well-liked, standard API for > accessing data files you will start receiving patches from distributions > that want to help you help them. Annotating my files is extremely unlike to break code, so I am more likely to accept a patch that does that. > 2) Symlinks cannot be used universally. Although it might not be common > to want an FHS style install in such an environment, it isn't unheard > of. At one time in the distant past I had to use cygwin so I know that > while this may be a corner case, it does exist. > > 3) The primary argument for symlinks is that symlinks are compatible > with __file__. But this compatibility comes at a cost -- symlinks can't > do anything extra. In a different subthread Philip argues that > setuptools provides more than distutils and that's why people switch and > that the next generation tool needs to provide even more than > setuptools. Symlinks cannot do that. As a library writer I have no motivation to do any of this. New features do drive adoption more quickly than simple cleanup, but only features that would help me as a developer in some way (including making it easier to support users). A new API wouldn't help me, and might hurt as it means more conventions to communicate to other developers. Also I'd have to debug problems with the resource loading, which be nothing but frustration. I hate platform issues, and moving files around just means there's more platform issues I'd be exposed to. Nothing platform-specific is of any interest to me as a developer -- unfortunately such problems come up often, but I don't want to go looking for new platform issues. > 4) In contrast an API can do more: It can deal with writable files. On > Unix, persistent, per user storage would go in the user's home > directory, on other OS's it would go somewhere else. This is > abstractable using an API at runtime but not using symlinks at install time. Writable stuff is quite different, IMHO. An API for writable files might be useful, but there's no current conventions around it, and I would expect that API to be entirely different from a resource API. > 5) cross package data. Using __file__ to detect file location is > inherently not suitable for crossing package boundaries. Egg > Translations would not be able to use a symlink based backend to do its > work for this reason. You'll need to explain further, as I am unclear of the problem with __file__ in this context. For instance, couldn't you symlink somepackage/translations/ to /usr/share/lang/somepackage ? > 6) zipped eggs. These require an API. So moving to symlinks is > actually a regression. True. > 7) Philip says that the reason pkg_resources does not see widespread > adoption is that the developer cost of using an API is too high compared > to __file__. I don't believe that the difference between file and API > is that great. An example of using an API could be something like this: > > Symlinks:: > import os > icondirectory = os.path.join(os.path.basename(__file__), 'icons') > > API:: > import pkgdata > icondirectory = pkgdata.resource(pkg='setuptools', \ > category='icon', resource='setuptools.png') > > Instead I think the data handling portion of pkg_resources is not more > widely adopted for these reasons: Just personally, it's entirely laziness on my part; I can't remember the signatures for the resource stuff, so I write what I most immediately remember. I think the Distro/package ambiguity also confuses me. > * pkg_resources's package handling is painful for the not-infrequent > corner cases. So people who have encountered the problems with > require() not overriding a default or not selecting the proper version > when multiple packages specify overlapping version ranges already have a > negative impression of the library before they even get to the data > handling portion. > > * pkg_resources does too much: loading libraries by version really has > nothing to do with loading data for use by a library. This is a > drawback because people think of and promote pkg_resources as a way to > enable easy_install rather than a way to enable abstraction of data > location. > > * The only benefit (at least, being promoted in the documentation) is to > allow zipped eggs to work. Distributions have no reason to create > zipped eggs so they have no reason to submit patches to upstream to > support the pkg_resources api. > > * Distributions, further, don't want to install all-in-one egg > directories on the system. The pkg_resources API just gets in the way > of doing things correctly in a distribution. I've had to patch code to > not use pkg_resources if data is installed in the FHS mandated areas. > Far from encouraging distributions to send patches upstream to make > modules use pkg_resources this makes distributions actively discourage > upstreams from using it. > > * The API isn't flexible enough. EggTranslations places its data within > the metadata store of eggs instead of within the data store. This is > because the metadata is able to be read outside of the package in which > it is included while the package data can only be accessed from within > the package. > > > 8) To a distribution, symlinks are just a hack. We use them for things > like php web apps when the web application is hardcoded to accept only > one path for things (like the writable state files being intermixed with > the program code). Managing a symlink farm is not something > distributions are going to get excited over so adoption by distributions > that this is the way to work with files won't happen until upstreams > move on their own. > > Further, since the install tool is being proposed as a separate project > from the metadata to mark files, the expectation is that the > distributions are going to want to write an install tool that manages > this symlink farm. For that to happen, you have to get distributions to > be much more than simply neutral about the idea of symlinks, you have to > have them enthused enough about using symlinks that they are willing to > spend time writing a tool to do it. > > > So once again, I think this boils down to these questions: if we have a > small library whose sole purpose is to abstract a data store so you can > find out where a particular non-code file lives on this system will you > use it? Realistically, no. > If a distribution packager sends you a patch so the data files > are marked correctly and the code can retrieve their location instead of > hardcoding an offset against __file__ will you commit it? If it adds a dependency and an abstraction that isn't obvious, then no, I would not commit it. Just marking the files is fine, because it has no impact on other code. -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From p.f.moore at gmail.com Fri Oct 17 23:25:09 2008 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 17 Oct 2008 22:25:09 +0100 Subject: [Distutils] Symlinks vs API -- question for developers In-Reply-To: <4D65936D-3885-4855-8BFF-761C21E5F4DB@python.org> References: <48F8CC33.7090106@gmail.com> <4D65936D-3885-4855-8BFF-761C21E5F4DB@python.org> Message-ID: <79990c6b0810171425n47dd3455v2fb3832d98b93ff4@mail.gmail.com> 2008/10/17 Barry Warsaw : >> So once again, I think this boils down to these questions: if we have a >> small library whose sole purpose is to abstract a data store so you can >> find out where a particular non-code file lives on this system will you >> use it? > > I would. I apologize for not having followed the discussion that closely, > but as an application developer, I would really like an API that hides all > the location nonsense from me. As familiar as __file__ is, it's a fragile > hack. I'd like an API, as well. It's probably the only truly cross-platform approach. Having said that, a key question is, what precisely is needed here? Python 2.6 has pkgutil.get_data, which abstracts the idea of grabbing the content of a file. There's not much else that can realistically be supported by fully general PEP 302 style loaders, so you have to start either (1) requiring additional functionality from loaders, or (2) restricting usage to filesystems, and losing the whole concept of a "loader protocol" which PEP 302 provided. (And that way lies incompatibilities with py2exe, which is very common on Windows, and zipped eggs, as well as possibly other more obscure cases). I'd fully support the development of an API for data access, but I'd suggest that it should take the form of a PEP extending PEP 302, plus an implementation in core Python (pkgutil is the obvious place), rather than being restricted purely to an external module like setuptools. Of course, having an external implementation for use in older versions of Python which don't have the API in core, would be fine, but the aim should be the core. (Otherwise, it's adding a dependency to projects that otherwise don't need it). Paul. From greg.ewing at canterbury.ac.nz Sat Oct 18 02:45:15 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 18 Oct 2008 13:45:15 +1300 Subject: [Distutils] Symlinks vs API -- question for developers In-Reply-To: <48F8CC33.7090106@gmail.com> References: <48F8CC33.7090106@gmail.com> Message-ID: <48F9319B.6010605@canterbury.ac.nz> Toshio Kuratomi wrote: > So once again, I think this boils down to these questions: if we have a > small library whose sole purpose is to abstract a data store so you can > find out where a particular non-code file lives on this system will you > use it? As part of the stdlib? Yes, I'd use it. -- Greg From martin at v.loewis.de Sat Oct 18 15:00:48 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 18 Oct 2008 15:00:48 +0200 Subject: [Distutils] Classifiers for Python versions Message-ID: <48F9DE00.3000404@v.loewis.de> As suggested in issue 2169549, I have now added a number of trove classifiers to PyPI to denote support for certain Python versions, namely Programming Language :: Python :: 2 Programming Language :: Python :: 2.3 Programming Language :: Python :: 2.4 Programming Language :: Python :: 2.5 Programming Language :: Python :: 2.6 Programming Language :: Python :: 2.7 Programming Language :: Python :: 3 Programming Language :: Python :: 3.0 Programming Language :: Python :: 3.1 With these, packages can indicate that they support 2.x (but not 3.x), or that they have been tested/written for specific Python releases. As a side effect, packages can now expressly state that they support Python 3000. Regards, Martin From james at mansionfamily.plus.com Sat Oct 18 16:27:51 2008 From: james at mansionfamily.plus.com (James Mansion) Date: Sat, 18 Oct 2008 15:27:51 +0100 Subject: [Distutils] Proxy server with easy_install (Python noob) Message-ID: <48F9F267.4070302@mansionfamily.plus.com> Bit of a Python noob I'm afraid. I'd like to install Sphinx with easy_install, but its timing out. I'm running XP sp3. This is the usual symptom of trying to use software that hasn't noticed that it needs to use a proxy server to get out of my network. Searches seem to come up with implementations of a proxy server in Pythn (and installation thereof) rather than a way to tell easy_install or installutils etc where my squid server is. Can anyone help? Thanks in advance James From lists at zopyx.com Sat Oct 18 15:52:38 2008 From: lists at zopyx.com (Andreas Jung) Date: Sat, 18 Oct 2008 15:52:38 +0200 Subject: [Distutils] Proxy server with easy_install (Python noob) In-Reply-To: <48F9F267.4070302@mansionfamily.plus.com> References: <48F9F267.4070302@mansionfamily.plus.com> Message-ID: <48F9EA26.4040902@zopyx.com> On 18.10.2008 16:27 Uhr, James Mansion wrote: > Bit of a Python noob I'm afraid. > > I'd like to install Sphinx with easy_install, but its timing out. > > I'm running XP sp3. > > This is the usual symptom of trying to use software that hasn't noticed > that it needs to use a proxy server to get out of my network. Set the 'http_proxy_ environment variable. Check with the urllib2 module documentation. -aj -------------- next part -------------- A non-text attachment was scrubbed... Name: lists.vcf Type: text/x-vcard Size: 316 bytes Desc: not available URL: From fdrake at gmail.com Sat Oct 18 17:17:40 2008 From: fdrake at gmail.com (Fred Drake) Date: Sat, 18 Oct 2008 11:17:40 -0400 Subject: [Distutils] [Catalog-sig] Classifiers for Python versions In-Reply-To: <48F9DE00.3000404@v.loewis.de> References: <48F9DE00.3000404@v.loewis.de> Message-ID: <9cee7ab80810180817v78c99910g4b0c0f7663782406@mail.gmail.com> On Sat, Oct 18, 2008 at 9:00 AM, "Martin v. L?wis" wrote: > With these, packages can indicate that they support 2.x > (but not 3.x), or that they have been tested/written for > specific Python releases. > > As a side effect, packages can now expressly state that they > support Python 3000. Thanks, Martin! This is good news. -Fred -- Fred L. Drake, Jr. "Chaos is the score upon which reality is written." --Henry Miller From garrcoop at cisco.com Sun Oct 19 04:14:11 2008 From: garrcoop at cisco.com (Garrett Cooper (garrcoop)) Date: Sat, 18 Oct 2008 19:14:11 -0700 Subject: [Distutils] Potential issue with multiple easy_install instances and single easy_install.pth Message-ID: Hi Python folks, As part of a build system I work with, my group installs multiple Python packages via source using easy_install. One such issue I've seen before in the past is that when using multiple easy_install instances (via multiple make jobs), the last instance that opened up easy_install.pth records its changes; the file should contain entries for all packages installed by easy_install. I was wondering if this was a known issue / caveat and/or whether or not anyone was possibly working on rectifying the problem. One workaround to this issue I could think of is possibly using a single .pth file per egg, as Python by default opens up all .pth files at startup in site-packages using a *.pth glob, but the only issue is open file descriptors and wasted cluster space with a number of small files... Either that or easy_install could (more properly) use a database backend for storing all package registries, which would not only simplify adding the .egg entries, but also simplify deleting .egg entries as well (a feature missing from easy_install and I believe from setup.py as well). Thanks! -Garrett From pje at telecommunity.com Mon Oct 20 01:28:01 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 19 Oct 2008 19:28:01 -0400 Subject: [Distutils] Symlinks vs API -- question for developers In-Reply-To: <48F8CC33.7090106@gmail.com> References: <48F8CC33.7090106@gmail.com> Message-ID: <20081019232644.BBBE93A408C@sparrow.telecommunity.com> At 10:32 AM 10/17/2008 -0700, Toshio Kuratomi wrote: >So I have a question for all the developers on this list. Philip thinks >that using symlinks will drive adoption better than an API to access >package data. I think an API will have better adoption than a symlink >hack. But the real question is what do people who maintain packages >think? Since Philip's given his reasoning, here's mine: > >1) Philip says that with symlinks distributions will likely have to >submit patches to the build scripts to tag various files as belonging to >certain categories. If you, as an upstream are going to accept a patch >to your build scripts to place files in a different place wouldn't you >also accept a patch to your source code to use a well defined API to >pull files from a different source? This is a distribution's bread and >butter and if there's a small, useful, well-liked, standard API for >accessing data files you will start receiving patches from distributions >that want to help you help them. I'll leave this to the developers, but please note that the real historical answer to this question is "no", or at least "not in the current release". Keep in mind that most "yeses" you get to this question will really mean, "when I can get around to understanding the API and testing it and have time to put it in a new release" -- while the "yeses" for adding spec metadata are more likely to mean, "yes, I'll check it in right now if it looks correct". >2) Symlinks cannot be used universally. Although it might not be common >to want an FHS style install in such an environment, it isn't unheard >of. At one time in the distant past I had to use cygwin so I know that >while this may be a corner case, it does exist. Cygwin does symlinks, actually. >3) The primary argument for symlinks is that symlinks are compatible >with __file__. But this compatibility comes at a cost -- symlinks can't >do anything extra. In a different subthread Philip argues that >setuptools provides more than distutils and that's why people switch and >that the next generation tool needs to provide even more than >setuptools. Symlinks cannot do that. I think Ian's already said this, but the API itself has to do something more, and so far nobody's proposed an API that does anything "more" than what setuptools does in this area, from the developer point of view. (Except for the request that such an API be in the stdlib and thus avoid an extra dependency... but that of course introduces yet another implementation delay, if it means a new release of Python.) >4) In contrast an API can do more: It can deal with writable files. On >Unix, persistent, per user storage would go in the user's home >directory, on other OS's it would go somewhere else. This is >abstractable using an API at runtime but not using symlinks at install time. This is all well and good, but it's actually quite orthogonal to most uses of __file__ and resources today. >5) cross package data. Using __file__ to detect file location is >inherently not suitable for crossing package boundaries. Egg >Translations would not be able to use a symlink based backend to do its >work for this reason. EggTranslations doesn't use __file__, it uses the API, so I don't see how this relates. >6) zipped eggs. These require an API. So moving to symlinks is >actually a regression. As I mentioned earlier, setuptools marks eggs that use __file__ as needing to be installed unzipped, so it's not a regression; it's simply providing the same level of compatibility that setuptools does. It's requiring the use of an API that's a regression wrt developer-side features. >7) Philip says that the reason pkg_resources does not see widespread >adoption is that the developer cost of using an API is too high compared >to __file__. I don't believe that the difference between file and API >is that great. It isn't; it's the *switching* cost that's high, and that's the cost that needs to be minimized in order to drive adoption quickly. >[snip] I'll just note that the bullets I'm skipping are mostly irrelevant to the issue at hand: i.e., switching cost of using *any* API, AND switching cost for the developers who *are* using pkg_resources presently. Let's not forget that second group of people, because the fact they are using the API shows they are likely early adopters. Make it too hard for them to switch, and you might not have any early adopters left for the new thing. ;-) >* The API isn't flexible enough. EggTranslations places its data within >the metadata store of eggs instead of within the data store. This is >because the metadata is able to be read outside of the package in which >it is included while the package data can only be accessed from within >the package. Actually, this is incorrect. EggTranslations' use of project-level data is so that it's not necessary to include a Python module in the egg, just to have a place to put the data. Access from other packages hasn't got anything to do with it. >8) To a distribution, symlinks are just a hack. We use them for things >like php web apps when the web application is hardcoded to accept only >one path for things (like the writable state files being intermixed with >the program code). Managing a symlink farm is not something >distributions are going to get excited over so adoption by distributions >that this is the way to work with files won't happen until upstreams >move on their own. We need to distinguish between "providing the ability to have a low-cost transition" and "the recommended True Way". IOW, symlinks and an API are not mutually exclusive; I'm just pointing out that if an API is required, the transition of packages to the new standard will occur *only as quickly as the slowest upstream dependency*. If the developer of A depends on B, and B hasn't transitioned yet, then A can't transition. >Further, since the install tool is being proposed as a separate project >from the metadata to mark files, the expectation is that the >distributions are going to want to write an install tool that manages >this symlink farm. For that to happen, you have to get distributions to >be much more than simply neutral about the idea of symlinks, you have to >have them enthused enough about using symlinks that they are willing to >spend time writing a tool to do it. Well, the question is whether they prefer to have a long, drawn out transition or not. Maybe they don't care about that part, but my assumption was that a replacement for setuptools/easy_install in this space was desired sooner rather than later. If that's the case, then making it possible for packages to transition without changing their runtime code is a must-have. >So once again, I think this boils down to these questions: if we have a >small library whose sole purpose is to abstract a data store so you can >find out where a particular non-code file lives on this system will you >use it? If a distribution packager sends you a patch so the data files >are marked correctly and the code can retrieve their location instead of >hardcoding an offset against __file__ will you commit it? I think the answer to both questions is "yes... eventually... if the API is in the stdlib for all Python versions I'm targeting and everybody else is doing it." Which is why *requiring* it for transition will prevent the distros from seeing benefits from a new standard for quite some time. Conversely, if the patch for installation metadata is separated from patches to code, I would expect a *much* faster uptake of the metadata patches. And, once having accepted the metadata patch, a developer is actually more likely to take the second step willingly, than if required to do both at once. (See "Influence" by Cialdini.) To be 100% clear (I hope): I have no objection to an API. It's unequivocally a good idea, and *should* be part of BUILDS. *Requiring* it, on the other hand, is unequivocally a *bad* idea, if you want adoption sooner rather than later. Now, if you want to establish a transition timetable for phasing out __file__ usage, deprecation, etc., based on when the API will be available in the stdlib etc., publicize and bless that schedule, etc... again, these are all good ideas. The ONLY thing I object to is requiring it up front from day 1, because then we're just shooting off a giant foot-gun wrt adoption. From pje at telecommunity.com Mon Oct 20 01:30:56 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 19 Oct 2008 19:30:56 -0400 Subject: [Distutils] Potential issue with multiple easy_install instances and single easy_install.pth In-Reply-To: References: Message-ID: <20081019232939.8D0783A408C@sparrow.telecommunity.com> At 07:14 PM 10/18/2008 -0700, Garrett Cooper (garrcoop) wrote: >Hi Python folks, > As part of a build system I work with, my group installs >multiple Python packages via source using easy_install. One such issue >I've seen before in the past is that when using multiple easy_install >instances (via multiple make jobs), the last instance that opened up >easy_install.pth records its changes; the file should contain entries >for all packages installed by easy_install. easy_install doesn't support simultaneous parallel installations to the same target directory, and there are no plans at the moment to add that support. Note, however, that if you use the -m (--multi-version) option, then easy_install will not save the .pth file unless there was already a default version of the target package present. However, if you begin by deleting the .pth file altogether, then using -m will avoid creating or updating it. Of course, the downside of -m is that you will not be able to access the installed packages except via setuptools-built scripts or by using explicit require() calls. From garrcoop at cisco.com Mon Oct 20 01:44:34 2008 From: garrcoop at cisco.com (Garrett Cooper (garrcoop)) Date: Sun, 19 Oct 2008 16:44:34 -0700 Subject: [Distutils] Potential issue with multiple easy_install instances and single easy_install.pth In-Reply-To: <20081019232939.8D0783A408C@sparrow.telecommunity.com> References: <20081019232939.8D0783A408C@sparrow.telecommunity.com> Message-ID: > -----Original Message----- > From: Phillip J. Eby [mailto:pje at telecommunity.com] > Sent: Sunday, October 19, 2008 4:31 PM > To: Garrett Cooper (garrcoop); distutils-sig at python.org > Subject: Re: [Distutils] Potential issue with multiple > easy_install instances and single easy_install.pth > > At 07:14 PM 10/18/2008 -0700, Garrett Cooper (garrcoop) wrote: > >Hi Python folks, > > As part of a build system I work with, my group installs > >multiple Python packages via source using easy_install. One > such issue > >I've seen before in the past is that when using multiple > easy_install > >instances (via multiple make jobs), the last instance that opened up > >easy_install.pth records its changes; the file should > contain entries > >for all packages installed by easy_install. > > easy_install doesn't support simultaneous parallel > installations to the same target directory, and there are no > plans at the moment to add that support. > > Note, however, that if you use the -m (--multi-version) > option, then easy_install will not save the .pth file unless > there was already a default version of the target package > present. However, if you begin by deleting the .pth file > altogether, then using -m will avoid creating or updating it. > > Of course, the downside of -m is that you will not be able to > access the installed packages except via setuptools-built > scripts or by using explicit require() calls. Phillip, Do you or anyone else know where would I need to look into the distutils source to implement this enhancement? I forsee using Python's version of flock, with possibly the use of a simple semaphore like-system. Thanks for your quick feedback, -Garrett From pje at telecommunity.com Mon Oct 20 05:35:13 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 19 Oct 2008 23:35:13 -0400 Subject: [Distutils] Potential issue with multiple easy_install instances and single easy_install.pth In-Reply-To: References: <20081019232939.8D0783A408C@sparrow.telecommunity.com> Message-ID: <20081020033358.9D0EA3A408C@sparrow.telecommunity.com> At 04:44 PM 10/19/2008 -0700, Garrett Cooper (garrcoop) wrote: > > -----Original Message----- > > From: Phillip J. Eby [mailto:pje at telecommunity.com] > > Sent: Sunday, October 19, 2008 4:31 PM > > To: Garrett Cooper (garrcoop); distutils-sig at python.org > > Subject: Re: [Distutils] Potential issue with multiple > > easy_install instances and single easy_install.pth > > > > At 07:14 PM 10/18/2008 -0700, Garrett Cooper (garrcoop) wrote: > > >Hi Python folks, > > > As part of a build system I work with, my group installs > > >multiple Python packages via source using easy_install. One > > such issue > > >I've seen before in the past is that when using multiple > > easy_install > > >instances (via multiple make jobs), the last instance that opened up > > >easy_install.pth records its changes; the file should > > contain entries > > >for all packages installed by easy_install. > > > > easy_install doesn't support simultaneous parallel > > installations to the same target directory, and there are no > > plans at the moment to add that support. > > > > Note, however, that if you use the -m (--multi-version) > > option, then easy_install will not save the .pth file unless > > there was already a default version of the target package > > present. However, if you begin by deleting the .pth file > > altogether, then using -m will avoid creating or updating it. > > > > Of course, the downside of -m is that you will not be able to > > access the installed packages except via setuptools-built > > scripts or by using explicit require() calls. > >Phillip, > Do you or anyone else know where would I need to look into the >distutils source to implement this enhancement? It's setuptools, not distutils, and the module in question is setuptools.command.easy_install. >I forsee using Python's >version of flock, with possibly the use of a simple semaphore >like-system. Note that you can probably also do this in your makefile, or in a script wrapper around easy_install. For that matter, you can create an easy_install subclass that does this, and then register it under a new command name and script. See the setuptools manual for more info. From joss at debian.org Mon Oct 20 12:45:54 2008 From: joss at debian.org (Josselin Mouette) Date: Mon, 20 Oct 2008 12:45:54 +0200 Subject: [Distutils] Symlinks vs API -- question for developers In-Reply-To: <20081019232644.BBBE93A408C@sparrow.telecommunity.com> References: <48F8CC33.7090106@gmail.com> <20081019232644.BBBE93A408C@sparrow.telecommunity.com> Message-ID: <1224499554.4133.50.camel@shizuru> Le dimanche 19 octobre 2008 ? 19:28 -0400, Phillip J. Eby a ?crit : > >3) The primary argument for symlinks is that symlinks are compatible > >with __file__. But this compatibility comes at a cost -- symlinks can't > >do anything extra. From my point of view, compatibility with __file__ is something that is not desired. We should discourage developers from using such a hack instead of going on supporting it. Yes, there is a cost for this migration. There is always a cost for developing properly, but it is a win in the long term. > I think Ian's already said this, but the API itself has to do > something more, and so far nobody's proposed an API that does > anything "more" than what setuptools does in this area, from the > developer point of view. Part of the solution is to introduce semantics in this API. Merely tagging files only allows to move them to a handful of usable locations (e.g. /usr/share/$package, /usr/share/doc/$package, /usr/share/pixmaps, /etc/$package). However, depending on the file type and what you want to use for, it may be useful in many more locations, and they will not depend on a mere tag. For example, when you use gettext, the thing you express is ?get translations for the current locale?, not ?load the LC_MESSAGES/fr/blah.po file?. Similarly for GTK+, the API allows to retrieve an icon at the size of a given widget, the file location being completely abstracted. As such, the solution is probably to develop an API allowing to access data in an abstract way, and one that can integrate properly with existing tools relying on file locations. Let?s take the icon example. You first need to define what is an icon file and where it goes. It could be defined in a file looking like this: hicolor scalable 16x16 22x22 ... apps ${prefix}/icons/${theme}/${size}/${category} Then, you need to flag your icons; this is what you already proposed, in a more sophisticated way. foo.png: icon size="48x48" category="apps" foo.svg: icon size="scalable" category="apps" Finally, in your code: * if you already use GTK+, you only have to retrieve the icon named "foo"; * otherwise, you need to write a helper function based on the API, which will allow to write things like this: location=builds.get_path("icon", "foo.png", category="apps", size="48x48") I don?t think any other build system is going this far; which means it is not necessary to go this far, but it could be a way to get developers to abandon their hacks. > >4) In contrast an API can do more: It can deal with writable files. On > >Unix, persistent, per user storage would go in the user's home > >directory, on other OS's it would go somewhere else. This is > >abstractable using an API at runtime but not using symlinks at install time. > > This is all well and good, but it's actually quite orthogonal to most > uses of __file__ and resources today. Not entirely orthogonal, since it goes in the same direction: a common base to implement standard FHS and freedesktop.org locations. If all Python applications start using standard home and configuration directories, it?s a lot of burden that goes away for sysadmins and users. > >5) cross package data. Using __file__ to detect file location is > >inherently not suitable for crossing package boundaries. Egg > >Translations would not be able to use a symlink based backend to do its > >work for this reason. > > EggTranslations doesn't use __file__, it uses the API, so I don't see > how this relates. I think this was just an example. There will be many cases where you need to extend an existing package?s resources with your own. Using __file__ doesn?t allow that and will require that you pass full paths instead of using an abstract API to access the data. > It's requiring the use of an API that's a regression wrt > developer-side features. Generally, I call the deprecation of a hack in favor of a clean API an improvement, not a regression. But that may just be your own terminology. > >7) Philip says that the reason pkg_resources does not see widespread > >adoption is that the developer cost of using an API is too high compared > >to __file__. I don't believe that the difference between file and API > >is that great. > > It isn't; it's the *switching* cost that's high, and that's the cost > that needs to be minimized in order to drive adoption quickly. You shouldn?t forget another part of the issue. If the only thing the new specification allows is to make symbolic links to a few pre-defined locations without any flexibility, it is very unlikely that distributions will take the time to write any patches to it. Without anyone to drive this feature forward, it will not be adopted at all. It is not only a question of making the migration easy, but also on making people want to push this migration. > >8) To a distribution, symlinks are just a hack. We use them for things > >like php web apps when the web application is hardcoded to accept only > >one path for things (like the writable state files being intermixed with > >the program code). Managing a symlink farm is not something > >distributions are going to get excited over so adoption by distributions > >that this is the way to work with files won't happen until upstreams > >move on their own. > > We need to distinguish between "providing the ability to have a > low-cost transition" and "the recommended True Way". If no one makes the low-cost transition in the end, you are going to lose your time trying to set it up. > >Further, since the install tool is being proposed as a separate project > >from the metadata to mark files, the expectation is that the > >distributions are going to want to write an install tool that manages > >this symlink farm. For that to happen, you have to get distributions to > >be much more than simply neutral about the idea of symlinks, you have to > >have them enthused enough about using symlinks that they are willing to > >spend time writing a tool to do it. > > Well, the question is whether they prefer to have a long, drawn out > transition or not. Maybe they don't care about that part, but my > assumption was that a replacement for setuptools/easy_install in this > space was desired sooner rather than later. Currently I think we?d be more reluctant to symlinks than enthusiastic about them. We already maintain symlink farms to deal with the insanity of Python modules, moving files to a place that is more neutral for the FHS (/usr/share/pysomething). The urgency is far behind us since Python module developers didn?t see it at all at that time. > Now, if you want to establish a transition timetable for phasing out > __file__ usage, deprecation, etc., based on when the API will be > available in the stdlib etc., publicize and bless that schedule, > etc... again, these are all good ideas. > > The ONLY thing I object to is requiring it up front from day 1, > because then we're just shooting off a giant foot-gun wrt adoption. No one is asking that. BTW deprecating __file__ is far from easy, since there are valid uses for it, and it?s not trivial to distinguish them. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From zooko at zooko.com Mon Oct 20 14:36:32 2008 From: zooko at zooko.com (zooko) Date: Mon, 20 Oct 2008 06:36:32 -0600 Subject: [Distutils] Symlinks vs API -- question for developers In-Reply-To: <48F8CC33.7090106@gmail.com> References: <48F8CC33.7090106@gmail.com> Message-ID: <3AB22696-7716-4AB3-BFE6-3E8061E17518@zooko.com> On Oct 17, 2008, at 11:32 AM, Toshio Kuratomi wrote: > So once again, I think this boils down to these questions: if we > have a > small library whose sole purpose is to abstract a data store so you > can > find out where a particular non-code file lives on this system will > you > use it? Well, since I'm already using pkg_resources then I'm obviously willing to use such an API. Whether I'm willing to switch to a different API from the one that I already have and that already works for all of my needs -- pkg_resources -- would be a different question. By the way, I don't understand that hack using "__file__" that was mentioned as an alternative, so from my perspective as a poor, dumb, working developer, that __file__ thing is more confusing than the pkg_resources API is. Regards, Zooko --- http://allmydata.org -- Tahoe, the Least-Authority Filesystem http://allmydata.com -- back up all your files for $10/month From greg.ewing at canterbury.ac.nz Tue Oct 21 01:18:21 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 21 Oct 2008 12:18:21 +1300 Subject: [Distutils] Symlinks vs API -- question for developers In-Reply-To: <1224499554.4133.50.camel@shizuru> References: <48F8CC33.7090106@gmail.com> <20081019232644.BBBE93A408C@sparrow.telecommunity.com> <1224499554.4133.50.camel@shizuru> Message-ID: <48FD11BD.5080701@canterbury.ac.nz> Josselin Mouette wrote: > Then, you need to flag your icons; this is what you already proposed, in > a more sophisticated way. > foo.png: icon size="48x48" category="apps" > foo.svg: icon size="scalable" category="apps" This smells like overdesign to me. Unless you want to allow for the possibility of e.g. putting your 48x48 icons in one location and icons of other sizes in a different location -- which seems rather unlikely -- then distinguishing between resources at this level of detail is not something the API we're considering should be concerned with. IMO all we need is a single parameter: resource.read(modulename, kind, path) where 'kind' is one of a well-known set of tags that correspond to ways that *packagers* may want to organize resources. If applications want to categorize their resources in more detail, it's up to them to provide a mapping from their own categorization scheme to this one. -- Greg From joss at debian.org Tue Oct 21 14:19:38 2008 From: joss at debian.org (Josselin Mouette) Date: Tue, 21 Oct 2008 14:19:38 +0200 Subject: [Distutils] Symlinks vs API -- question for developers In-Reply-To: <48FD11BD.5080701@canterbury.ac.nz> References: <48F8CC33.7090106@gmail.com> <20081019232644.BBBE93A408C@sparrow.telecommunity.com> <1224499554.4133.50.camel@shizuru> <48FD11BD.5080701@canterbury.ac.nz> Message-ID: <1224591578.4292.40.camel@shizuru> Le mardi 21 octobre 2008 ? 12:18 +1300, Greg Ewing a ?crit : > Josselin Mouette wrote: > > > Then, you need to flag your icons; this is what you already proposed, in > > a more sophisticated way. > > foo.png: icon size="48x48" category="apps" > > foo.svg: icon size="scalable" category="apps" > > This smells like overdesign to me. Unless you want to allow > for the possibility of e.g. putting your 48x48 icons in > one location and icons of other sizes in a different > location -- which seems rather unlikely -- then > distinguishing between resources at this level of detail > is not something the API we're considering should be > concerned with. Unless you want to be able to follow existing standards. http://standards.freedesktop.org/icon-theme-spec/icon-theme-spec-latest.html#directory_layout The same goes for localisation tiles, for which there is a similar directory scheme to follow. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Ceci est une partie de message num?riquement sign?e URL: From fadhley.salim at uk.calyon.com Tue Oct 21 14:28:48 2008 From: fadhley.salim at uk.calyon.com (Fadhley Salim) Date: Tue, 21 Oct 2008 13:28:48 +0100 Subject: [Distutils] Building a Win32/Python2.4 multi-version egg for Numpy References: <48F8CC33.7090106@gmail.com><20081019232644.BBBE93A408C@sparrow.telecommunity.com><1224499554.4133.50.camel@shizuru> <48FD11BD.5080701@canterbury.ac.nz> Message-ID: <7F347D91614EBC48AA7540A2A76D3BB204483DF0@MXCU10MX1.MSX.CIB> Is there an easy way to make a multi-version egg from the Numpy source code (supplied from sourceforge). Ideally I'd like to create an automated process that allows me to quickly make a Win32 egg any time a new release of Numpy comes out so that my team can test it and adjust our project's dependancies as they see fit. Initially I'd like to make multiversion eggs for both Numpy 1.1 and 1.2 so that my team can get started on testing this library. Sal -------------- next part -------------- This email does not create a legal relationship between any member of the Cr?dit Agricole group and the recipient or constitute investment advice. The content of this email should not be copied or disclosed (in whole or part) to any other person. It may contain information which is confidential, privileged or otherwise protected from disclosure. If you are not the intended recipient, you should notify us and delete it from your system. Emails may be monitored, are not secure and may be amended, destroyed or contain viruses and in communicating with us such conditions are accepted. Any content which does not relate to business matters is not endorsed by us. Calyon is authorised by the Commission Bancaire in France and regulated by the Financial Services Authority for the conduct of business in the United Kingdom. Calyon is incorporated in France with limited liability and registered in England & Wales. Registration number: FC008194. Registered office: Broadwalk House, 5 Appold Street, London, EC2A 2DA. From chris at simplistix.co.uk Tue Oct 21 14:59:53 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Tue, 21 Oct 2008 13:59:53 +0100 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <48F246B6.8000707@v.loewis.de> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <94bdd2610810111524w4cba4f70n4766fdb0b95fe73e@mail.gmail.com> <48F1396C.8030907@zopyx.com> <94bdd2610810111856j4dbefcabra43dc590cb556e0d@mail.gmail.com> <48F1ADCC.5050602@zopyx.com> <5e1183fa0810120540g6e902699hfe7b177b57532f9c@mail.gmail.com> <94bdd2610810120558v31a75fc2ga9823e92f906bede@mail.gmail.com> <48F238D7.7050600@v.loewis.de> <94bdd2610810121051m40f4114ai8a9730d958bf7d50@mail.gmail.com> <48F23E11.30801@v.loewis.de> <94bdd2610810121132o34c51df8gb3acac84fa283502@mail.gmail.com> <48F246B6.8000707@v.loewis.de> Message-ID: <48FDD249.8070309@simplistix.co.uk> Martin v. L?wis wrote: > Hmm. Yesterday, there were 199250 accesses to PyPI through wget. > Of those, 169971 requests came from a single address (from Dedibox in > France), 28966 requests from a second one (from Sakura in Japan). > > So it *is* wget mirrors that make the whole traffic in PyPI. If it were me, I'd just IP firewall the offendors. There's not need for this kind of behaviour if there's an acceptable mirror protocol available... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From ziade.tarek at gmail.com Tue Oct 21 15:06:18 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 21 Oct 2008 15:06:18 +0200 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <48FDD249.8070309@simplistix.co.uk> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <48F1ADCC.5050602@zopyx.com> <5e1183fa0810120540g6e902699hfe7b177b57532f9c@mail.gmail.com> <94bdd2610810120558v31a75fc2ga9823e92f906bede@mail.gmail.com> <48F238D7.7050600@v.loewis.de> <94bdd2610810121051m40f4114ai8a9730d958bf7d50@mail.gmail.com> <48F23E11.30801@v.loewis.de> <94bdd2610810121132o34c51df8gb3acac84fa283502@mail.gmail.com> <48F246B6.8000707@v.loewis.de> <48FDD249.8070309@simplistix.co.uk> Message-ID: <94bdd2610810210606p5533a1f3mbb6b37f02da85d8d@mail.gmail.com> On Tue, Oct 21, 2008 at 2:59 PM, Chris Withers wrote: > Martin v. L?wis wrote: >> >> Hmm. Yesterday, there were 199250 accesses to PyPI through wget. >> Of those, 169971 requests came from a single address (from Dedibox in >> France), 28966 requests from a second one (from Sakura in Japan). >> >> So it *is* wget mirrors that make the whole traffic in PyPI. > > If it were me, I'd just IP firewall the offendors. There's not need for this > kind of behaviour if there's an acceptable mirror protocol available... Well not yet... but the PEP should be finished sometimes this week, > > cheers, > > Chris > > -- > Simplistix - Content Management, Zope & Python Consulting > - http://www.simplistix.co.uk > -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From stephen.pascoe at stfc.ac.uk Tue Oct 21 18:21:24 2008 From: stephen.pascoe at stfc.ac.uk (Pascoe, S (Stephen)) Date: Tue, 21 Oct 2008 17:21:24 +0100 Subject: [Distutils] [zc.buildout] use of pkg_resources Message-ID: I'm intrigued ... Why the scripts buildout produces don't use pkg_resources. They all add each egg to sys.path manually and import script entry points manually. It would seem more consistent to use pkg_resources.working_set.find_plugins() and pkg_resources.working_set.run_script() I can imagine there might be a good reason, such as making absolutely sure the correct egg is at the top of sys.path. This is another example of the TIMTOWTDI malaise that is rife with anything to do with setuptools. Thanks, Stephen. --- Stephen Pascoe +44 (0)1235 445980 British Atmospheric Data Centre Rutherford Appleton Laboratory -- Scanned by iCritical for STFC. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim at zope.com Tue Oct 21 18:36:29 2008 From: jim at zope.com (Jim Fulton) Date: Tue, 21 Oct 2008 12:36:29 -0400 Subject: [Distutils] [zc.buildout] use of pkg_resources In-Reply-To: References: Message-ID: <9A597F88-BDFC-4C3A-A3E2-9636B63B433A@zope.com> On Oct 21, 2008, at 12:21 PM, Pascoe, S (Stephen) wrote: > I'm intrigued ... > > Why the scripts buildout produces don't use pkg_resources. They all > add each egg to sys.path manually and import script entry points > manually. It would seem more consistent to use > pkg_resources.working_set.find_plugins() and > pkg_resources.working_set.run_script() > > I can imagine there might be a good reason, such as making > absolutely sure the correct egg is at the top of sys.path. Yes, among other reasons, such as: - Not using system Python, - Not requiring a special python environment to be set when not using a system Python, - Allowing different scripts to use different package versions, as needed by the script. > This is another example of the TIMTOWTDI malaise that is rife with > anything to do with setuptools. The "it" is different in the case of buildout. Buildout is solving a somewhat different problem than easy_install. Jim -- Jim Fulton Zope Corporation From ianb at colorstudy.com Tue Oct 21 20:11:05 2008 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 21 Oct 2008 14:11:05 -0400 Subject: [Distutils] Symlinks vs API -- question for developers In-Reply-To: <20081019232644.BBBE93A408C@sparrow.telecommunity.com> References: <48F8CC33.7090106@gmail.com> <20081019232644.BBBE93A408C@sparrow.telecommunity.com> Message-ID: <48FE1B39.4010103@colorstudy.com> Phillip J. Eby wrote: > I think Ian's already said this, but the API itself has to do something > more, and so far nobody's proposed an API that does anything "more" than > what setuptools does in this area, from the developer point of view. > (Except for the request that such an API be in the stdlib and thus avoid > an extra dependency... but that of course introduces yet another > implementation delay, if it means a new release of Python.) It's probably a bit easier than waiting for a release of Python -- if it's in a PEP, and will be in a release of Python, then the library will be blessed and people will pick it up much more quickly. Realistically most library developers would need to add the package as a requirement for some time, since they won't stop supporting older versions of Python that don't have that package. -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From martin at v.loewis.de Tue Oct 21 21:42:17 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 21 Oct 2008 21:42:17 +0200 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <94bdd2610810210606p5533a1f3mbb6b37f02da85d8d@mail.gmail.com> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <48F1ADCC.5050602@zopyx.com> <5e1183fa0810120540g6e902699hfe7b177b57532f9c@mail.gmail.com> <94bdd2610810120558v31a75fc2ga9823e92f906bede@mail.gmail.com> <48F238D7.7050600@v.loewis.de> <94bdd2610810121051m40f4114ai8a9730d958bf7d50@mail.gmail.com> <48F23E11.30801@v.loewis.de> <94bdd2610810121132o34c51df8gb3acac84fa283502@mail.gmail.com> <48F246B6.8000707@v.loewis.de> <48FDD249.8070309@simplistix.co.uk> <94bdd2610810210606p5533a1f3mbb6b37f02da85d8d@mail.gmail.com> Message-ID: <48FE3099.8020501@v.loewis.de> >> If it were me, I'd just IP firewall the offendors. There's not need for this >> kind of behaviour if there's an acceptable mirror protocol available... > > Well not yet... but the PEP should be finished sometimes this week, There was an acceptable mirror protocol available for more than a year now, irrespective of any PEP you are working on. Claiming that only with the PEP, true mirroring becomes possible, disregards the work that Jim Fulton and I put into coming up with a workable solution. Regards, Martin From ziade.tarek at gmail.com Wed Oct 22 00:10:43 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 22 Oct 2008 00:10:43 +0200 Subject: [Distutils] [Catalog-sig] distribute D.C. sprint tasks In-Reply-To: <48FE3099.8020501@v.loewis.de> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <94bdd2610810120558v31a75fc2ga9823e92f906bede@mail.gmail.com> <48F238D7.7050600@v.loewis.de> <94bdd2610810121051m40f4114ai8a9730d958bf7d50@mail.gmail.com> <48F23E11.30801@v.loewis.de> <94bdd2610810121132o34c51df8gb3acac84fa283502@mail.gmail.com> <48F246B6.8000707@v.loewis.de> <48FDD249.8070309@simplistix.co.uk> <94bdd2610810210606p5533a1f3mbb6b37f02da85d8d@mail.gmail.com> <48FE3099.8020501@v.loewis.de> Message-ID: <94bdd2610810211510i166fe697x1b368ebda4bad74e@mail.gmail.com> On Tue, Oct 21, 2008 at 9:42 PM, "Martin v. L?wis" wrote: >>> If it were me, I'd just IP firewall the offendors. There's not need for this >>> kind of behaviour if there's an acceptable mirror protocol available... >> >> Well not yet... but the PEP should be finished sometimes this week, > > There was an acceptable mirror protocol available for more than a year > now, irrespective of any PEP you are working on. Claiming that only with > the PEP, true mirroring becomes possible, disregards the work that Jim > Fulton and I put into coming up with a workable solution. Right, sorry about that. The PEP works on complementary needs (like client-side failover on mirrors) and will also try to summarize in a section how mirrors and client apps should behave for every aspect, reffering to all previous works on that, since they have already provided solutions for an optimal access to pypi. Tarek -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From greg.ewing at canterbury.ac.nz Wed Oct 22 01:50:11 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 22 Oct 2008 12:50:11 +1300 Subject: [Distutils] Symlinks vs API -- question for developers In-Reply-To: <94bdd2610810211510i166fe697x1b368ebda4bad74e@mail.gmail.com> References: <94bdd2610810111056k2e277cc0t4a1e1cbe88fcf841@mail.gmail.com> <94bdd2610810120558v31a75fc2ga9823e92f906bede@mail.gmail.com> <48F238D7.7050600@v.loewis.de> <94bdd2610810121051m40f4114ai8a9730d958bf7d50@mail.gmail.com> <48F23E11.30801@v.loewis.de> <94bdd2610810121132o34c51df8gb3acac84fa283502@mail.gmail.com> <48F246B6.8000707@v.loewis.de> <48FDD249.8070309@simplistix.co.uk> <94bdd2610810210606p5533a1f3mbb6b37f02da85d8d@mail.gmail.com> <48FE3099.8020501@v.loewis.de> <94bdd2610810211510i166fe697x1b368ebda4bad74e@mail.gmail.com> Message-ID: <48FE6AB3.3020203@canterbury.ac.nz> Josselin Mouette: > Unless you want to be able to follow existing standards. > > http://standards.freedesktop.org/icon-theme-spec/icon-theme-spec-latest.html#directory_layout All you need for that is a category called "icons". I don't see anything there about putting different sizes of icon in different places on different platforms. Once you're in the icons location, the path from there to the flavour of icon you want is fixed by that spec. So the way you reference it is using something like. resource.read(module, "icon", "%s/48x48/apps/foo.png" % theme) -- Greg From rasky at develer.com Fri Oct 24 18:00:03 2008 From: rasky at develer.com (Giovanni Bajo) Date: Fri, 24 Oct 2008 18:00:03 +0200 Subject: [Distutils] compiler-specific extra_compiler_args Message-ID: <4901F103.10808@develer.com> Hello, using vanilla distutils with vanilla Python 2.5, I would like to pass different extra_compiler_args to the setup() call based on the compiler I'm using. In fact, if I'm using GCC (whether on Windows, Linux or Mac), I have a few options to specify; if I'm using MSVC (on Windows), I have others. This looks like a basic need, but I couldn't find a way to achieve it. Most existing scripts in the wild seem to simply do a check on the platform, which neglects the fact that one can specify --compiler=mingw32 to use GCC MinGW to build on Windows. Any suggestion on how to achieve this? -- Giovanni Bajo Develer S.r.l. http://www.develer.com From justizin at vongogo.net Sat Oct 25 07:08:38 2008 From: justizin at vongogo.net (Justin Ryan) Date: Fri, 24 Oct 2008 22:08:38 -0700 Subject: [Distutils] zc.buildout, recipe downloading dev eggs Message-ID: Howdy! I apologize if this isn't the place to ask about buildout, but it is listed as the support address in PyPI. I use zc.buildout, as many Zope / Plone and other Python developers do nowadays to help in deployment of our code. We are abandoning SVN as version control, and for now are using BZR with a shell script to pull externals from SVN. As BZR is a Python DSCM, we want to use it api-wise in a buildout recipe, as an alternative to svn:externals, as BZR's externals-like feature is a bit slow in coming, and also a recipe could support a variety of SCM by URI. My concern is that, of course, buildout expects dev-eggs to be in src/ when its' run, though dev products and other bits can come as dependencies of eggs and via recipes. What I want is to be able to download dev eggs in a recipe, which is applied before the eggs are sought. I'm glad to help, but expect that approaching folks with setuptools and esp buildout expertise, via e-mail, as a group, might be a good start. I'd be glad to help document how to do this and what situations it benefits. Best, and TIA! J -- Robert Half - "When one teaches, two learn." -------------- next part -------------- An HTML attachment was scrubbed... URL: From setuptools at bugs.python.org Sat Oct 25 14:55:02 2008 From: setuptools at bugs.python.org (Zooko O'Whielacronx) Date: Sat, 25 Oct 2008 12:55:02 +0000 Subject: [Distutils] [issue51] "Please rename it back to setuptools-0.6c9-py2.5.egg and try again." In-Reply-To: <1224939302.43.0.865493902798.issue51@psf.upfronthosting.co.za> Message-ID: <1224939302.43.0.865493902798.issue51@psf.upfronthosting.co.za> New submission from Zooko O'Whielacronx : If you follow the install instructions on http://pypi.python.org/pypi/setuptools/0.6c9#cygwin-mac-os-x-linux-other with Safari on Mac OS X, you will probably accidentally rename the egg to setuptools-0.6c9-py2.5.egg.sh, since Safari does that for you automatically without really pointing out what it is doing. Then if you run it, it will say "./setuptools-0.6c9-py2.5.egg.sh is not the correct name for this egg file. Please rename it back to setuptools-0.6c9-py2.5.egg and try again.". Why does it matter what the name is? I ask because I don't want to advise my Macintosh users to follow the instructions at http://pypi.python.org/pypi/setuptools/0.6c9#cygwin-mac-os-x-linux-other if it is going to lead to these sorts of issues for them, and nor do I want to explain to them how to download a file without letting Safari rename it. ---------- messages: 200 nosy: zooko priority: bug status: unread title: "Please rename it back to setuptools-0.6c9-py2.5.egg and try again." _______________________________________________ Setuptools tracker _______________________________________________ From jim at zope.com Sat Oct 25 17:20:40 2008 From: jim at zope.com (Jim Fulton) Date: Sat, 25 Oct 2008 11:20:40 -0400 Subject: [Distutils] zc.buildout, recipe downloading dev eggs In-Reply-To: References: Message-ID: On Oct 25, 2008, at 1:08 AM, Justin Ryan wrote: ... > I apologize if this isn't the place to ask about buildout, but it is > listed as the support address in PyPI. This is the right place. ... > My concern is that, of course, buildout expects dev-eggs to be in > src/ when its' run, No it doesn't. buildout is based on setuptools, which is based on distutils and builds on any mechanisms they provide, which are very flexible. Use of src directories is a common and generally convenient convention. > though dev products and other bits can come as dependencies of eggs > and via recipes. > > What I want is to be able to download dev eggs in a recipe, which is > applied before the eggs are sought. I'm not sure what you're saying. You can certainly create develop eggs in a recipe. Buildout "prefers" develop eggs over non-develop eggs, so it will use a develop egg rather than a regular egg if both satisfy given requirements, even develop eggs with lower version numbers. Note that this policy sometimes gets buildout into trouble since system installs often have the same type as develop eggs in setuptools. I'm considering having buildout only prefer develop eggs found in it's develop-eggs directory. I'm also, belatedly, planning to add an option to buildout to ignore site packages, which would also avoid this problem. Jim -- Jim Fulton Zope Corporation From justizin at vongogo.net Sat Oct 25 19:55:40 2008 From: justizin at vongogo.net (Justin Ryan) Date: Sat, 25 Oct 2008 10:55:40 -0700 Subject: [Distutils] zc.buildout, recipe downloading dev eggs In-Reply-To: References: Message-ID: On Sat, Oct 25, 2008 at 8:20 AM, Jim Fulton wrote: > > On Oct 25, 2008, at 1:08 AM, Justin Ryan wrote: > ... > >> I apologize if this isn't the place to ask about buildout, but it is >> listed as the support address in PyPI. >> > > This is the right place. > Awesome. :) > > ... > > My concern is that, of course, buildout expects dev-eggs to be in src/ >> when its' run, >> > > No it doesn't. buildout is based on setuptools, which is based on > distutils and builds on any mechanisms they provide, which are very > flexible. Use of src directories is a common and generally convenient > convention. > A great point. I meant to say, it expects them to be accessible before any parts are built out, AFAIK. > > > though dev products and other bits can come as dependencies of eggs and >> via recipes. >> >> What I want is to be able to download dev eggs in a recipe, which is >> applied before the eggs are sought. >> > > I'm not sure what you're saying. > > You can certainly create develop eggs in a recipe. Buildout "prefers" > develop eggs over non-develop eggs, so it will use a develop egg rather than > a regular egg if both satisfy given requirements, even develop eggs with > lower version numbers. > It seems that I would have to run buildout twice for the develop eggs to be available, if they are downloaded in a recipe, but perhaps I'm working on an incorrect assumption. After sending this message, I started wondering if a plugin would be more appropriate than a recipe, using a recipe just seems simpler at the outset because there are existing recipes to start with, at least for SVN. > > Note that this policy sometimes gets buildout into trouble since system > installs often have the same type as develop eggs in setuptools. I'm > considering having buildout only prefer develop eggs found in it's > develop-eggs directory. I'm also, belatedly, planning to add an option to > buildout to ignore site packages, which would also avoid this problem. > Seems to me that virtualenv is the answer to this problem, but it's not perfect. -- Robert Half - "When one teaches, two learn." -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim at zope.com Sat Oct 25 20:05:14 2008 From: jim at zope.com (Jim Fulton) Date: Sat, 25 Oct 2008 14:05:14 -0400 Subject: [Distutils] zc.buildout, recipe downloading dev eggs In-Reply-To: References: Message-ID: On Oct 25, 2008, at 1:55 PM, Justin Ryan wrote: > My concern is that, of course, buildout expects dev-eggs to be in > src/ when its' run, > > No it doesn't. buildout is based on setuptools, which is based on > distutils and builds on any mechanisms they provide, which are very > flexible. Use of src directories is a common and generally > convenient convention. > > A great point. I meant to say, it expects them to be accessible > before any parts are built out, AFAIK. No. buildout doesn't care when develop eggs are created. If you use the buildout develop option to create develop eggs, then they are created early. If you use the zc.recipe.egg:develop recipe, or some other recipe that creates develop eggs, then you can create them later. > It seems that I would have to run buildout twice for the develop > eggs to be available, if they are downloaded in a recipe, but > perhaps I'm working on an incorrect assumption. You are. See above. > After sending this message, I started wondering if a plugin would be > more appropriate than a recipe, using a recipe just seems simpler at > the outset because there are existing recipes to start with, at > least for SVN. Recipes work find for your use case afaict. > Note that this policy sometimes gets buildout into trouble since > system installs often have the same type as develop eggs in > setuptools. I'm considering having buildout only prefer develop > eggs found in it's develop-eggs directory. I'm also, belatedly, > planning to add an option to buildout to ignore site packages, which > would also avoid this problem. > > Seems to me that virtualenv is the answer to this problem, but it's > not perfect. It would be wildly simpler to just get buildout to ignore site- packages. :) Jim -- Jim Fulton Zope Corporation From justizin at vongogo.net Sat Oct 25 21:05:07 2008 From: justizin at vongogo.net (Justin Ryan) Date: Sat, 25 Oct 2008 12:05:07 -0700 Subject: [Distutils] zc.buildout, recipe downloading dev eggs In-Reply-To: References: Message-ID: On Sat, Oct 25, 2008 at 11:05 AM, Jim Fulton wrote: > > On Oct 25, 2008, at 1:55 PM, Justin Ryan wrote: > >> My concern is that, of course, buildout expects dev-eggs to be in src/ >> when its' run, >> >> No it doesn't. buildout is based on setuptools, which is based on >> distutils and builds on any mechanisms they provide, which are very >> flexible. Use of src directories is a common and generally convenient >> convention. >> >> A great point. I meant to say, it expects them to be accessible before >> any parts are built out, AFAIK. >> > > No. buildout doesn't care when develop eggs are created. If you use the > buildout develop option to create develop eggs, then they are created early. > If you use the zc.recipe.egg:develop recipe, or some other recipe that > creates develop eggs, then you can create them later. > Ah, fantastic, I came to an understanding of this last week or so, but some example usage I found led me to believe it wouldn't work. Perfect! Thanks. > > > It seems that I would have to run buildout twice for the develop eggs to >> be available, if they are downloaded in a recipe, but perhaps I'm working on >> an incorrect assumption. >> > > You are. See above. > > After sending this message, I started wondering if a plugin would be more >> appropriate than a recipe, using a recipe just seems simpler at the outset >> because there are existing recipes to start with, at least for SVN. >> > > Recipes work find for your use case afaict. > w00t! :) > > > Note that this policy sometimes gets buildout into trouble since system >> installs often have the same type as develop eggs in setuptools. I'm >> considering having buildout only prefer develop eggs found in it's >> develop-eggs directory. I'm also, belatedly, planning to add an option to >> buildout to ignore site packages, which would also avoid this problem. >> >> Seems to me that virtualenv is the answer to this problem, but it's not >> perfect. >> > > > It would be wildly simpler to just get buildout to ignore site-packages. :) > If you say so, I believe it. Looking forward to new func! :) Thanks again! J -- Robert Half - "When one teaches, two learn." -------------- next part -------------- An HTML attachment was scrubbed... URL: From justizin at vongogo.net Sun Oct 26 02:22:38 2008 From: justizin at vongogo.net (Justin Ryan) Date: Sat, 25 Oct 2008 17:22:38 -0700 Subject: [Distutils] BadStatusLine ( was Re: zc.buildout, recipe downloading dev eggs ) Message-ID: Hi again.. :) >> Recipes work find for your use case afaict. >> > > So, I tried removing the lines which pointed at my various parts' eggs and develop options collectively in the [buildout] section, and now I get a BadStatusLine from within setuptools trying to install my dev egg. Speaking of which, I ran into this BadStatusLine issue with some other eggs, it seems that setuptools could more gracefull deal with the situation and be more informative, but perhaps it appears simpler than it is. Best, and TIA! J -- Robert Half - "When one teaches, two learn." -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim at zope.com Sun Oct 26 14:12:56 2008 From: jim at zope.com (Jim Fulton) Date: Sun, 26 Oct 2008 09:12:56 -0400 Subject: [Distutils] BadStatusLine ( was Re: zc.buildout, recipe downloading dev eggs ) In-Reply-To: References: Message-ID: On Oct 25, 2008, at 8:22 PM, Justin Ryan wrote: > Hi again.. :) > > > Recipes work find for your use case afaict. > > > So, I tried removing the lines which pointed at my various parts' > eggs and develop options collectively in the [buildout] section, and > now I get a BadStatusLine from within setuptools trying to install > my dev egg. > > Speaking of which, I ran into this BadStatusLine issue with some > other eggs, it seems that setuptools could more gracefull deal with > the situation and be more informative, but perhaps it appears > simpler than it is. If you expect any sort of response, you'll need to provide more information. Jim -- Jim Fulton Zope Corporation From justizin at vongogo.net Sun Oct 26 19:37:06 2008 From: justizin at vongogo.net (Justin Ryan) Date: Sun, 26 Oct 2008 11:37:06 -0700 Subject: [Distutils] BadStatusLine ( was Re: zc.buildout, recipe downloading dev eggs ) In-Reply-To: References: Message-ID: On Sun, Oct 26, 2008 at 6:12 AM, Jim Fulton wrote: > > On Oct 25, 2008, at 8:22 PM, Justin Ryan wrote: > > Hi again.. :) >> >> >> Recipes work find for your use case afaict. >> >> >> So, I tried removing the lines which pointed at my various parts' eggs and >> develop options collectively in the [buildout] section, and now I get a >> BadStatusLine from within setuptools trying to install my dev egg. >> >> Speaking of which, I ran into this BadStatusLine issue with some other >> eggs, it seems that setuptools could more gracefull deal with the situation >> and be more informative, but perhaps it appears simpler than it is. >> > > > If you expect any sort of response, you'll need to provide more > information. > I'm a bit confused at the situation myself, am working on a simple repro case. I can get it to fail, but not to throw BadStatusLine from within urllib just yet. I wouldn't dare ask you to untangle our production buildout, so I'll continue trying to come up with a better case and question. Whatever the cause, it seems that setuptools, or buildout, someone is not doing their job of catching exceptions from urllib. The exception seems to raise from within setuptools, but I've only seen this set of circumstances arise with buildout, so perhaps there's simply an integration issue on one side or the other. This is annoying enough without adding the fact that this happens against develop eggs which urrllib shouldn't be invoked for, but that's related to the previous thread, which I'll respond to in context. :) -- Robert Half - "When one teaches, two learn." -------------- next part -------------- An HTML attachment was scrubbed... URL: From justizin at vongogo.net Sun Oct 26 20:04:40 2008 From: justizin at vongogo.net (Justin Ryan) Date: Sun, 26 Oct 2008 12:04:40 -0700 Subject: [Distutils] zc.buildout, recipe downloading dev eggs In-Reply-To: References: Message-ID: > On Sat, Oct 25, 2008 at 11:05 AM, Jim Fulton wrote: >> >> No. buildout doesn't care when develop eggs are created. If you use the buildout develop option to create develop eggs, then they are created early. If you use the zc.recipe.egg:develop recipe, or some other recipe that creates develop eggs, then you can create them later. > > Ah, fantastic, I came to an understanding of this last week or so, but some example usage I found led me to believe it wouldn't work. > > Perfect! Thanks. > OK, I gave this a shot in a simple case, a buildout you can find here: http://dev.vongogo.net/bzr/repos/buildouts/getpaid.buildout/ The develop option in getpaid.cfg seems to be ignored entirely, I don't see any messages like "We have a develop egg", and it fails to find a distribution for yoma.batching, which is in the develop list, though it finds others which are available in PyPI. If I'm working on another false assumption, or parsed something from you or the documentation wrong, please correct. I just want to be able to deploy in this style. :) Best, J -- Robert Half - "When one teaches, two learn." From ziade.tarek at gmail.com Sun Oct 26 23:01:06 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 26 Oct 2008 23:01:06 +0100 Subject: [Distutils] fail-over and index merging for tools like setuptools and pyinstall Message-ID: <94bdd2610810261501p51e7dfa2ya6de9d8e56a2a824@mail.gmail.com> Hi ! Sorry for the crosspost, but this mail concerns both lists, There are two points that need to be discussed to finish the work on the PyPI proposal started in D.C. (http://wiki.python.org/PEP_374): - the fail-over mechanism - merging several indexes *Fail over* PyPI will provide a static page that lists all its mirrors. Each line of this file describes a mirror. It provides the root url, followed by the relative url of: - the index : the root of the package index - the last-modified page : a static text file that gives the date of the last sync. - the local stats page: a static text file that gives the number of downloads of a file, per package, per user-agent - the global stats page, calculated by pypi that gives the grand total of all downloads (the sum of PyPI local stats + mirrors local stats) - the mirrors page, that lists all mirrors For example: http://example.com/pypi,index,last-modified,local-stats,stats,mirrors http://example2.com/pypi,index,last-modified,local-stats,stats,mirrors (see the proposal doc for more info) This mirror list says for example that a mirror is available at http://example.com/pypi/index, and that its last modified date is available at http://example.com/pypi/last-modified. On client side it means that it is possible to list mirrors of a given package index to implement a fail-over mechanism. Moreover, it makes it possible to select the nearest mirror. *Merging several indexes* Besides fail over, another thing needs to be implemented on client side: being able to use different indexes. This is an obvious missing feature: we don't want to push in PyPI all our customers package. In the meantime we do want to use tools like distutils, setuptools etc., the same way with any kind of package. So using private package indexes easily besides PyPI is needed. It is now possible in Python 2.6 with the new .pypirc file to define several indexes. >From there softwares like PloneSoftwareCenter allows developers to work with other indexes than PyPI. But tools like setuptools need to evolve the same way. Each one of this index can have its own mirrors, as defined previously, but the client needs to combine all the different index, into a "super" index. This can be implemented by working with a sorted list of index. When a client is looking for a package, it can look in each index and pick the first package that fits. Any comments ? Cheers Tarek -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From martin at v.loewis.de Sun Oct 26 23:10:03 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 26 Oct 2008 23:10:03 +0100 Subject: [Distutils] [Catalog-sig] fail-over and index merging for tools like setuptools and pyinstall In-Reply-To: <94bdd2610810261501p51e7dfa2ya6de9d8e56a2a824@mail.gmail.com> References: <94bdd2610810261501p51e7dfa2ya6de9d8e56a2a824@mail.gmail.com> Message-ID: <4904EABB.7070709@v.loewis.de> > Any comments ? Who will implement all that? Regards, Martin From ziade.tarek at gmail.com Sun Oct 26 23:14:59 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 26 Oct 2008 23:14:59 +0100 Subject: [Distutils] [Catalog-sig] fail-over and index merging for tools like setuptools and pyinstall In-Reply-To: <4904EABB.7070709@v.loewis.de> References: <94bdd2610810261501p51e7dfa2ya6de9d8e56a2a824@mail.gmail.com> <4904EABB.7070709@v.loewis.de> Message-ID: <94bdd2610810261514j6da146d2uc3be1cc95c1a212f@mail.gmail.com> On Sun, Oct 26, 2008 at 11:10 PM, "Martin v. L?wis" wrote: >> Any comments ? > > Who will implement all that? > I am willing to do it. I started already to write some patch for setuptools, submited for review in the tracker, I can also write the patches for PyPI, since I have the code and a database dump, I can also organize sprints if some other people want to help implementing that. > Regards, > Martin > -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From benji at zope.com Mon Oct 27 13:29:03 2008 From: benji at zope.com (Benji York) Date: Mon, 27 Oct 2008 08:29:03 -0400 Subject: [Distutils] BadStatusLine ( was Re: zc.buildout, recipe downloading dev eggs ) In-Reply-To: References: Message-ID: On Sun, Oct 26, 2008 at 2:37 PM, Justin Ryan wrote: > > I'm a bit confused at the situation myself, am working on a simple repro > case. I can get it to fail, but not to throw BadStatusLine from within > urllib just yet. I've not been following this thread closely, but wanted to mention the -D switch to buildout. From --help: Debug errors. If an error occurs, then the post-mortem debugger will be started. This is especially useful for debuging recipe problems. -- Benji York Senior Software Engineer Zope Corporation From justizin at vongogo.net Mon Oct 27 16:59:21 2008 From: justizin at vongogo.net (Justin Ryan) Date: Mon, 27 Oct 2008 08:59:21 -0700 Subject: [Distutils] BadStatusLine ( was Re: zc.buildout, recipe downloading dev eggs ) In-Reply-To: References: Message-ID: On Mon, Oct 27, 2008 at 5:29 AM, Benji York wrote: > On Sun, Oct 26, 2008 at 2:37 PM, Justin Ryan wrote: >> >> I'm a bit confused at the situation myself, am working on a simple repro >> case. I can get it to fail, but not to throw BadStatusLine from within >> urllib just yet. > > I've not been following this thread closely, but wanted to mention the -D > switch to buildout. From --help: > > Debug errors. If an error occurs, then the post-mortem debugger > will be started. This is especially useful for debuging recipe > problems. Thanks, Benji, I don't doubt that will be useful! :) -- Robert Half - "When one teaches, two learn." From cipherzero at gmail.com Tue Oct 28 15:52:10 2008 From: cipherzero at gmail.com (Jake Swenson) Date: Tue, 28 Oct 2008 07:52:10 -0700 Subject: [Distutils] Improper permissions when a restrictive usmask is set Message-ID: <286e71640810280752w4b04063bq63f8f343b6301e21@mail.gmail.com> Hi, i recently have been having trouble with easy_install. I have my usmask set to 027 and when installing packages with easyinstall they would only work while in a root account. i now know that the problem is because the packages that get installed are not installed accessable by normal users. i can correct this by setting the o+rx flags on the installed files. is there a chance that this can be fixed? thanks. -- Cipher -------------- next part -------------- An HTML attachment was scrubbed... URL: From ianb at colorstudy.com Tue Oct 28 18:33:41 2008 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 28 Oct 2008 12:33:41 -0500 Subject: [Distutils] pyinstall renamed to pip Message-ID: <49074CF5.8090208@colorstudy.com> I've renamed pyinstall to pip (last renaming, I promise). It now uses commands like "pip install something". This will make it easier to add new commands in the future, with entirely different option signatures. New site: http://pip.openplans.org -- Ian Bicking : ianb at colorstudy.com : http://blog.ianbicking.org From piyush at piyushverma.net Wed Oct 29 00:39:13 2008 From: piyush at piyushverma.net (Piyush Verma) Date: Wed, 29 Oct 2008 05:09:13 +0530 (IST) Subject: [Distutils] A Bug with Setuptools.. Sdist Message-ID: <949168.36480.qm@web5003.biz.mail.in2.yahoo.com> Well... there seems a bug in setuptools. In commands/sdist.py There is a variable log accesed in Line 98: file "/usr/lib/python2.5/site-packages/setuptools/command/sdist.py", line=98, in the method entries_finder log.warn("unrecognized .svn/entries format in %s", dirname) NameError: global name 'log' is not defined SO at the top of the file please use : from distutils import log -- command/sdist.py 2008-10-20 16:02:09.000000000 +0530 +++ sdist_old.py 2008-10-20 16:03:45.000000000 +0530 @@ -1,6 +1,5 @@ from distutils.command.sdist import sdist as _sdist from distutils.util import convert_path =2Dfrom distutils import log import os, re, sys, pkg_resources -------------- next part -------------- An HTML attachment was scrubbed... URL: From pje at telecommunity.com Wed Oct 29 00:53:19 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 28 Oct 2008 19:53:19 -0400 Subject: [Distutils] A Bug with Setuptools.. Sdist In-Reply-To: <949168.36480.qm@web5003.biz.mail.in2.yahoo.com> References: <949168.36480.qm@web5003.biz.mail.in2.yahoo.com> Message-ID: <20081028235202.09BAD3A405B@sparrow.telecommunity.com> You're using 0.6c8 or older; upgrade to 0.6c9. At 05:09 AM 10/29/2008 +0530, Piyush Verma wrote: >Well... there seems a bug in setuptools. >In commands/sdist.py > >There is a variable log accesed in Line 98: >file "/usr/lib/python2.5/site-packages/setuptools/command/sdist.py", >line=98, >in the method entries_finder > log.warn("unrecognized .svn/entries format in %s", dirname) >NameError: global name 'log' is not defined > >SO at the top of the file please use : > >from distutils import log > >-- command/sdist.py 2008-10-20 16:02:09.000000000 +0530 >+++ sdist_old.py 2008-10-20 16:03:45.000000000 +0530 >@@ -1,6 +1,5 @@ >from distutils.command.sdist import sdist as _sdist >from distutils.util import convert_path >=2Dfrom distutils import log >import os, re, sys, pkg_resources >_______________________________________________ >Distutils-SIG maillist - Distutils-SIG at python.org >http://mail.python.org/mailman/listinfo/distutils-sig From skip at pobox.com Wed Oct 29 17:40:46 2008 From: skip at pobox.com (skip at pobox.com) Date: Wed, 29 Oct 2008 11:40:46 -0500 Subject: [Distutils] eggs and import order Message-ID: <18696.37390.390734.624742@montanaro-dyndns-org.local> Way back when I installed SQLalchemy 0.3.3 in such a way that it wound up in .../site-packages/easy-install.pth: ... ./SQLAlchemy-0.3.3-py2.4.egg ... A colleague installed a later version in a different directory, referenced that in PYTHONPATH but still got the old version: udesktop116% PYTHONPATH=/opt/tradelink/research/site-packages.beta python -c 'import sqlalchemy ; print sqlalchemy.__file__' /opt/app/g++lib6/python-2.4/lib/python2.4/site-packages/SQLAlchemy-0.3.3-py2.4.egg/sqlalchemy/__init__.pyc udesktop116% ls /opt/tradelink/research/site-packages.beta BeautifulSoup.py _mssql.so pylab.py Cython configobj.py pylab.pyc IPython dateutil pymssql.py Jinja-1.2-py2.4-solaris-2.10-i86pc.egg docutils-0.4-py2.4.egg pymssql.pyc PIL easy-install.pth pyparsing.py PIL.pth enthought pytz Pygments-0.10-py2.4.egg fpconst.py pyximport Pyrex fpconst.pyc research SOAPpy lib scipy SQLAlchemy-0.5.0rc3dev_r5205-py2.4.egg matplotlib timeseries Sphinx-0.3-py2.4.egg mpl_toolkits tlsec Symbide-0.3.1-py2.4.egg numpy __init__.py pyExcelerator Looking at sys.path shows why: PYTHONPATH=/opt/tradelink/research/site-packages.beta python -c 'import sys ; print sys.path' ['', '/opt/app/g++lib6/python-2.4/lib/python2.4/site-packages/zope.interface-3.3.0-py2.4-solaris-2.10-i86pc.egg', '/opt/app/g++lib6/python-2.4/lib/python2.4/site-packages/pycrypto-2.0.1-py2.4-solaris-2.10-i86pc.egg', '/opt/app/g++lib6/python-2.4/lib/python2.4/site-packages/magnitude-0.9.3-py2.4.egg', '/opt/app/g++lib6/python-2.4/lib/python2.4/site-packages/setuptools-0.6c7-py2.4.egg', '/opt/app/g++lib6/python-2.4/lib/python2.4/site-packages/paramiko-1.7.2-py2.4.egg', '/opt/app/g++lib6/python-2.4/lib/python2.4/site-packages/processing-0.52-py2.4-solaris-2.10-i86pc.egg', '/opt/app/g++lib6/python-2.4/lib/python2.4/site-packages/pp-1.5.4-py2.4.egg', '/opt/app/g++lib6/python-2.4/lib/python2.4/site-packages/greenlet-0.1-py2.4-solaris-2.10-i86pc.egg', '/opt/app/g++lib6/python-2.4/lib/python2.4/site-packages/MDP-2.3-py2.4.egg', '/opt/app/g++lib6/python-2.4/lib/python2.4/site-packages/hashlib-20060408a-py2.4-solaris-2.10-i86pc.egg', '/opt/app/g++lib6/python-2.4/lib/python2.4/site-packages/nose-0.10.3-py2.4.egg', '/opt/app/g++lib6/python-2.4/lib/python2.4/site-packages/coverage-2.85-py2.4.egg', '/opt/app/g++lib6/python-2.4/lib/python2.4/site-packages/daemon-1.0.1-py2.4.egg', '/opt/app/g++lib6/python-2.4/lib/python2.4/site-packages/SQLAlchemy-0.3.3-py2.4.egg', '/opt/tradelink/research/site-packages.beta', '/opt/app/g++lib6/python-2.4/lib/python24.zip', '/opt/app/g++lib6/python-2.4/lib/python2.4', '/opt/app/g++lib6/python-2.4/lib/python2.4/plat-sunos5', '/opt/app/g++lib6/python-2.4/lib/python2.4/lib-tk', '/opt/app/g++lib6/python-2.4/lib/python2.4/lib-dynload', '/opt/app/g++lib6/python-2.4/lib/python2.4/site-packages', '/opt/app/g++lib6/python-2.4/lib/python2.4/site-packages/3rdParty', '/opt/app/g++lib6/python-2.4/lib/python2.4/site-packages/3rdParty/Numeric'] The beta site-packages directory does indeed appear ahead of the normal site-packages, but the contents of easy-install.pth are all ahead of that. In my opinion they should be inserted right before the directory in which their .pth file was found. Why is this not the case? In case it's a setuptools/easy_install thing, I'm using setuptools 0.6c7 with Python 2.4.5. Thanks, Skip From pje at telecommunity.com Wed Oct 29 17:55:23 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 29 Oct 2008 12:55:23 -0400 Subject: [Distutils] eggs and import order In-Reply-To: <18696.37390.390734.624742@montanaro-dyndns-org.local> References: <18696.37390.390734.624742@montanaro-dyndns-org.local> Message-ID: <20081029165404.9B3523A40DF@sparrow.telecommunity.com> At 11:40 AM 10/29/2008 -0500, skip at pobox.com wrote: >Way back when I installed SQLalchemy 0.3.3 in such a way that it wound up in >.../site-packages/easy-install.pth: > > ... > ./SQLAlchemy-0.3.3-py2.4.egg > ... > >A colleague installed a later version in a different directory, referenced >that in PYTHONPATH but still got the old version: > > udesktop116% > PYTHONPATH=/opt/tradelink/research/site-packages.beta python -c > 'import sqlalchemy ; print sqlalchemy.__file__' > >/opt/app/g++lib6/python-2.4/lib/python2.4/site-packages/SQLAlchemy-0.3.3-py2.4.egg/sqlalchemy/__init__.pyc > udesktop116% ls /opt/tradelink/research/site-packages.beta > BeautifulSoup.py _mssql.so pylab.py > Cython configobj.py pylab.pyc > IPython dateutil > pymssql.py > Jinja-1.2-py2.4-solaris-2.10-i86pc.egg docutils-0.4-py2.4.egg > pymssql.pyc > PIL easy-install.pth > pyparsing.py > PIL.pth enthought pytz > Pygments-0.10-py2.4.egg fpconst.py pyximport > Pyrex fpconst.pyc research > SOAPpy lib scipy > SQLAlchemy-0.5.0rc3dev_r5205-py2.4.egg matplotlib > timeseries > Sphinx-0.3-py2.4.egg mpl_toolkits tlsec > Symbide-0.3.1-py2.4.egg numpy > __init__.py pyExcelerator There's a special 'site.py' missing from this directory. You can fix that by doing this: cd /opt/tradelink/research/site-packages.beta PYTHONPATH=. easy_install -d . SQLAlchemy-0.5.0rc3dev_r5205-py2.4.egg My guess is that this directory was created by copying a site-packages directory from somewhere, or by easy_installing to it with the --site-dirs option. Either would have prevented easy_install from creating the site.py that's necessary to make this work. Without the special site.py, Python doesn't pick this directory up as a "site" directory, and doesn't process PYTHONPATH eggs properly. Note that you must use easy_install as described above to do this; you can't just copy Python's site.py, as it's not the same file. The one easy_install puts there forces .pth processing to occur on PYTHONPATH, and then invokes the original Python site.py. From skip at pobox.com Wed Oct 29 19:03:51 2008 From: skip at pobox.com (skip at pobox.com) Date: Wed, 29 Oct 2008 13:03:51 -0500 Subject: [Distutils] eggs and import order In-Reply-To: <20081029165404.9B3523A40DF@sparrow.telecommunity.com> References: <18696.37390.390734.624742@montanaro-dyndns-org.local> <20081029165404.9B3523A40DF@sparrow.telecommunity.com> Message-ID: <18696.42375.350847.399153@montanaro-dyndns-org.local> Phillip> There's a special 'site.py' missing from this directory. You Phillip> can fix that by doing this: ... Thanks for the quick response. We'll look into it. Skip From cipherzero at gmail.com Wed Oct 29 23:14:20 2008 From: cipherzero at gmail.com (Jake Swenson) Date: Wed, 29 Oct 2008 15:14:20 -0700 Subject: [Distutils] Improper permissions when a restrictive usmask is set In-Reply-To: <286e71640810280752w4b04063bq63f8f343b6301e21@mail.gmail.com> References: <286e71640810280752w4b04063bq63f8f343b6301e21@mail.gmail.com> Message-ID: <286e71640810291514p2fdfd6f6oce52feac478331a@mail.gmail.com> So, any ideas? or am i the only one that has a restrictive umask?... On Tue, Oct 28, 2008 at 7:52 AM, Jake Swenson wrote: > Hi, i recently have been having trouble with easy_install. > I have my usmask set to 027 > and when installing packages with easyinstall they would only work while in > a root account. > i now know that the problem is because the packages that get installed are > not installed accessable by normal users. > i can correct this by setting the o+rx flags on the installed files. > is there a chance that this can be fixed? > > thanks. > > -- > Cipher > -- Cipher -------------- next part -------------- An HTML attachment was scrubbed... URL: From justizin at vongogo.net Thu Oct 30 05:34:51 2008 From: justizin at vongogo.net (Justin Ryan) Date: Wed, 29 Oct 2008 21:34:51 -0700 Subject: [Distutils] zc.buildout, recipe downloading dev eggs In-Reply-To: References: Message-ID: On closer inspection, the zc.recipe.egg documentation doesn't make any mention of this feature, though I think it's a great idea and hats off to the author if it was originally intended to work that way but doesn't, or maybe broke at some point because a contributor didn't understand the feature. If there's some work to be done, I'm happy to help. I really want this style of buildout usage to be possible. Let's work together! Our Foundation's goals line up with Zope extremeley well! :) Peace, J On Sun, Oct 26, 2008 at 12:04 PM, Justin Ryan wrote: >> On Sat, Oct 25, 2008 at 11:05 AM, Jim Fulton wrote: > > >>> >>> No. buildout doesn't care when develop eggs are created. If you use the buildout develop option to create develop eggs, then they are created early. If you use the zc.recipe.egg:develop recipe, or some other recipe that creates develop eggs, then you can create them later. >> >> Ah, fantastic, I came to an understanding of this last week or so, but some example usage I found led me to believe it wouldn't work. >> >> Perfect! Thanks. >> > > > OK, I gave this a shot in a simple case, a buildout you can find here: > > http://dev.vongogo.net/bzr/repos/buildouts/getpaid.buildout/ > > The develop option in getpaid.cfg seems to be ignored entirely, I > don't see any messages like "We have a develop egg", and it fails to > find a distribution for yoma.batching, which is in the develop list, > though it finds others which are available in PyPI. > > If I'm working on another false assumption, or parsed something from > you or the documentation wrong, please correct. I just want to be > able to deploy in this style. :) > > Best, > > J > > -- > Robert Half - "When one teaches, two learn." > -- Robert Half - "When one teaches, two learn." From setuptools at bugs.python.org Fri Oct 31 16:53:47 2008 From: setuptools at bugs.python.org (Miki Tebebeka) Date: Fri, 31 Oct 2008 15:53:47 +0000 Subject: [Distutils] [issue52] Missing import in command/sdist.py In-Reply-To: <1225468427.48.0.85003055753.issue52@psf.upfronthosting.co.za> Message-ID: <1225468427.48.0.85003055753.issue52@psf.upfronthosting.co.za> New submission from Miki Tebebeka : In line 99 of commands/sdist.py there is a call to "log.warn", however "log" is not imported. IMO there should be a line from distutils import log somewhere at the beginning. ---------- messages: 202 nosy: tebeka priority: bug status: unread title: Missing import in command/sdist.py _______________________________________________ Setuptools tracker _______________________________________________