From richardjones at optushome.com.au Mon May 2 00:29:46 2005 From: richardjones at optushome.com.au (Richard Jones) Date: Mon, 2 May 2005 08:29:46 +1000 Subject: [Catalog-sig] New PEP, pre-submission RFC In-Reply-To: <20050429095933.GA25977@mylene.ghaering.de> References: <200504291943.32344.richardjones@optushome.com.au> <20050429095933.GA25977@mylene.ghaering.de> Message-ID: <200505020829.50038.richardjones@optushome.com.au> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Fri, 29 Apr 2005 07:59 pm, Gerhard Haering wrote: > For those who have names with characters not in ASCII, is that possible? > do we just define that all text is UTF-8? At the moment, the web interface / database is all UTF-8, so it therefore assumes that submissions are UTF-8. I guess we should actually document that :) The one weak link of the current UTF-8-ness is the setup.py file, as you've identified. In most cases that'll be the default encoding for Python code (whatever that is, I should look it up ;) but in some cases it'll be different. Regardless, we really need to UTF-8 encode the submission of the setup.py-derived metadata to the webserver so there's no incorrect decoding done at its end. > Not that I really need it, but would this be the right time to think > about optional description fields for other languages, too, like > > Description-fr, Description-de for French and German ones? Eeek :) Richard -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) iD8DBQFCdVhdrGisBEHG6TARAvQIAJwNDP6CaPRPggyEsDcJsMspeKg4MACeOKzo yU2bfpcOJFWE7YIYW3zCzGY= =47L7 -----END PGP SIGNATURE----- From richardjones at optushome.com.au Mon May 2 00:42:50 2005 From: richardjones at optushome.com.au (Richard Jones) Date: Mon, 2 May 2005 08:42:50 +1000 Subject: [Catalog-sig] "implementation" of PEP 262 as an academic project In-Reply-To: <42731018.5060700@acm.org> References: <1114735502.510591.6360@f14g2000cwb.googlegroups.com> <4272CE91.5080908@acm.org> <42731018.5060700@acm.org> Message-ID: <200505020842.50769.richardjones@optushome.com.au> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Sat, 30 Apr 2005 02:56 pm, Maurice LING wrote: > > I've just read PEP 262 last night and finds that it does more or less > > describes what I have in mind. However, I am not sure if there is every > > need for such a descriptive database file or something slimmer, like > > Fink's .info files will suffice. You should make yourself familiar with the latest metadata PEP 314 (specifies metadata version 1.1). I'm also currently working on metadata version 1.2, which allows specification of python version requirements, and external software requirements. > > Implementing the API specified in PEP 262 is desirable. > > > > What I am thinking is this, > > > > 1. when user specify a package to install, the package's .info file will > > be looked up in 'pkginfo' directory (in PEP 262, it is the INSTALLDB > > directory that holds all these .info files). > > > > 2. to the system, only 3 things are crucial: where to get the package? > > what packages the package needs? how to install? These 3 things are the > > real critical parts of .info file, the rest are information and metadata. You might need to prompt the user for handling external dependencies here. > > Please tell me what you think... Sounds good to me. I'd like to see the PEP (261) updated with the latest state of distutils (ie. metadata, the package index), your ideas and re-posted. Richard -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) iD8DBQFCdVtqrGisBEHG6TARAsxFAJ9906N9GwODpcoNqEcNqXi37tcKiwCeMNu3 5dmJVm97e8ZrsjxHwAcbwnM= =hhwI -----END PGP SIGNATURE----- From mauriceling at acm.org Mon May 2 12:15:58 2005 From: mauriceling at acm.org (Maurice Ling) Date: Mon, 02 May 2005 20:15:58 +1000 Subject: [Catalog-sig] "implementation" of PEP 262 as an academic project Message-ID: <4275FDDE.8010006@acm.org> Thank you for your pointers, Richard. So now there are PEP 262 PEP 314 to take note of. Are there any relevant PEPs towards this purpose? Please kindly tell me. Thanks Cheers Maurice -------------- next part -------------- A non-text attachment was scrubbed... Name: mauriceling.vcf Type: text/x-vcard Size: 334 bytes Desc: not available Url : http://mail.python.org/pipermail/catalog-sig/attachments/20050502/478cf37d/mauriceling.vcf From richardjones at optusnet.com.au Tue May 3 09:02:05 2005 From: richardjones at optusnet.com.au (richardjones@optusnet.com.au) Date: Tue, 03 May 2005 17:02:05 +1000 Subject: [Catalog-sig] (no subject) Message-ID: <200505030702.j437258A025261@mail28.syd.optusnet.com.au> An embedded and charset-unspecified text was scrubbed... Name: not available Url: http://mail.python.org/pipermail/catalog-sig/attachments/20050503/c89d7233/attachment.ksh From ianb at colorstudy.com Tue May 3 09:54:17 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 03 May 2005 02:54:17 -0500 Subject: [Catalog-sig] (no subject) In-Reply-To: <200505030702.j437258A025261@mail28.syd.optusnet.com.au> References: <200505030702.j437258A025261@mail28.syd.optusnet.com.au> Message-ID: <42772E29.90204@colorstudy.com> richardjones at optusnet.com.au wrote: > Another point that I'd like to make is that I believe we should avoid reimplementing > RPM, DEB, fink, whatever. Or at least we should not ignore them. We can produce > meta-data that supports those packaging formats. Thus a user has the opportunity to > install Python software using the existing package management mechanisms installed > in their system, rather than something new and independent just for Python. The > latest PEP I've proposed adds some meta-data that makes the DEB-alike packagers more > happy, and should also help out RPM packagers. > > Distutils already does a fairly good job of handling the actual installation of > Python software -- it compiles things, can install data files, with a little extra > effort other stuff can be done too. It generates RPMs, DEBs, Windows Installers. > There's a desire from the Ubuntu packagers to add a doc_files option to the setup() > args, but that's for another discussion. > > Anyway, I'm rambling. My point is clear though: I believe we should avoid developing > Yet Another Packaging System just for the sake of it. And certainly we should play > well with others where possible. OTOH, there is a place for distutils installs even in the presence of native packaging. Because Python doesn't have versioned imports, in many situations (at least the kind of stuff I do) a system-wide installation isn't appropriate. And most native packagers don't do well at localized installations. > I guess we need to ask what it is that our database of installed Python modules will > achieve that the existing packaging systems don't. Of course, I don't know what a database would achieve exactly in this case either, except perhaps in the presence of versioned imports. Which I guess Eggs provide, in a way. But then Eggs mix it up too, since wouldn't be "installed" either, they just get put in the right place, and so there's no chance for a database to recognize them. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From mauriceling at acm.org Tue May 3 11:20:37 2005 From: mauriceling at acm.org (Maurice Ling) Date: Tue, 03 May 2005 19:20:37 +1000 Subject: [Catalog-sig] (no subject) In-Reply-To: <42772E29.90204@colorstudy.com> References: <200505030702.j437258A025261@mail28.syd.optusnet.com.au> <42772E29.90204@colorstudy.com> Message-ID: <42774265.6000006@acm.org> Ian Bicking wrote: > richardjones at optusnet.com.au wrote: > >> Another point that I'd like to make is that I believe we should avoid >> reimplementing >> RPM, DEB, fink, whatever. Or at least we should not ignore them. We >> can produce >> meta-data that supports those packaging formats. Thus a user has the >> opportunity to >> install Python software using the existing package management >> mechanisms installed >> in their system, rather than something new and independent just for >> Python. The >> latest PEP I've proposed adds some meta-data that makes the DEB-alike >> packagers more >> happy, and should also help out RPM packagers. >> >> Distutils already does a fairly good job of handling the actual >> installation of >> Python software -- it compiles things, can install data files, with a >> little extra >> effort other stuff can be done too. It generates RPMs, DEBs, Windows >> Installers. >> There's a desire from the Ubuntu packagers to add a doc_files option >> to the setup() >> args, but that's for another discussion. >> >> Anyway, I'm rambling. My point is clear though: I believe we should >> avoid developing >> Yet Another Packaging System just for the sake of it. And certainly >> we should play >> well with others where possible. > > > OTOH, there is a place for distutils installs even in the presence of > native packaging. Because Python doesn't have versioned imports, in > many situations (at least the kind of stuff I do) a system-wide > installation isn't appropriate. And most native packagers don't do > well at localized installations. > >> I guess we need to ask what it is that our database of installed >> Python modules will >> achieve that the existing packaging systems don't. > > > Of course, I don't know what a database would achieve exactly in this > case either, except perhaps in the presence of versioned imports. > Which I guess Eggs provide, in a way. But then Eggs mix it up too, > since wouldn't be "installed" either, they just get put in the right > place, and so there's no chance for a database to recognize them. > Hi, Thank for all your pointers. I see my proposed work to solve one very fundamental issue. (Please take a look at the thread "bytecode non-backcompatibility" started in April 2005 in python-list) Given that (1) there can be multiple versions of Python installed in a system, (2) each version maintains their own site-packages directory and (3) if C modules are installed, they are not compatible with other versions of Python... Imagine a system administrator who had installed 50 libraries and their dependencies in site-package of Python 2.3 and now has to do it for Python 2.4, can we make his life better? Building on that, if say I distribute a software which requires 5 3rd-party libraries. Currently, I can only tell the user which libraries are necessary and hopefully they don't get too frustrated (imagine if each of the 5 libraries depends on 2 libraries each). If my work is around (PEP 262 and 314), I can then ask the user to just install ONE system and in my software distribution, calls that one system to install all the necessary libraries. I've not done anything substantial yet.... still feeling around my way. Cheers Maurice -- Maurice Han Tong LING, BSc(Hons)(Biochem), AdvDipComp, CPT, SSN, FIFA, MASBMB, MAMBIS, MACM Doctor of Philosophy (Science) Candidate, The University of Melbourne mobile: +61 4 22781753, +65 96669233 mailing address: Department of Zoology, The University of Melbourne Royal Parade, Parkville, Victoria 3010, Australia residence: 9/41 Dover Street, Flemington, Victoria 3031, Australia resume: http://maurice.vodien.com/maurice_resume.pdf www: http://www.geocities.com/beldin79/ -------------- next part -------------- A non-text attachment was scrubbed... Name: mauriceling.vcf Type: text/x-vcard Size: 334 bytes Desc: not available Url : http://mail.python.org/pipermail/catalog-sig/attachments/20050503/2a6166cb/mauriceling.vcf From richardjones at optushome.com.au Tue May 3 11:58:34 2005 From: richardjones at optushome.com.au (Richard Jones) Date: Tue, 3 May 2005 19:58:34 +1000 Subject: [Catalog-sig] (no subject) In-Reply-To: <42774265.6000006@acm.org> References: <200505030702.j437258A025261@mail28.syd.optusnet.com.au> <42772E29.90204@colorstudy.com> <42774265.6000006@acm.org> Message-ID: <200505031958.37920.richardjones@optushome.com.au> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tue, 3 May 2005 07:20 pm, Maurice Ling wrote: > Given that (1) there can be multiple versions of Python installed in a > system, (2) each version maintains their own site-packages directory and > (3) if C modules are installed, they are not compatible with other > versions of Python... Imagine a system administrator who had installed > 50 libraries and their dependencies in site-package of Python 2.3 and > now has to do it for Python 2.4, can we make his life better? If you could solve that *just* for pure-python packages, you'd already be a hero :) More response later (probably tomorrow). Richard -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) iD8DBQFCd0tNrGisBEHG6TARAqFFAJ9lDxeFNjVzGNfcFHnAbCtxbmdKHwCeK2t4 4mL4uGGw6CoWXht+x+NiWXU= =TPNR -----END PGP SIGNATURE----- From martin at v.loewis.de Thu May 5 14:19:43 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 05 May 2005 14:19:43 +0200 Subject: [Catalog-sig] (no subject) In-Reply-To: <42774265.6000006@acm.org> References: <200505030702.j437258A025261@mail28.syd.optusnet.com.au> <42772E29.90204@colorstudy.com> <42774265.6000006@acm.org> Message-ID: <427A0F5F.4080009@v.loewis.de> Maurice Ling wrote: > Given that (1) there can be multiple versions of Python installed in a > system, (2) each version maintains their own site-packages directory and > (3) if C modules are installed, they are not compatible with other > versions of Python... Imagine a system administrator who had installed > 50 libraries and their dependencies in site-package of Python 2.3 and > now has to do it for Python 2.4, can we make his life better? I'd like to point out that this is a task that the "native" package format can solve. For example, in Debian, when I use Debian packages to install the 50 libraries, the current Debian Python policy manages to update all the libraries from Python 2.3 to Python 2.4, when the "official" Debian Python version becomes 2.4. This is achieved by providing, for a library "foo", several packages: python-foo, python2.2-foo, python2.3-foo, python2.4-foo. As a user, I install python-foo, which currently depends on python2.3-foo. When /usr/bin/python becomes 2.4, a new version of python-foo will be released which depends on python2.4-foo. Just by updating all packages I have installed, it automatically installs python2.4-foo as well. Of course, if I don't want to wait for Debian to make /usr/bin/python Python 2.4, I easily upgrade all libraries myself: dpkg --get-selections|grep python2.3-|sed s/2.3/2.4/|dpkg --set-selections will arrange to select all 2.4 libraries for which I have 2.3 versions installed. So on Debian, I don't need any further support from distutils. Regards, Martin From mauriceling at acm.org Thu May 5 14:57:27 2005 From: mauriceling at acm.org (Maurice Ling) Date: Thu, 05 May 2005 22:57:27 +1000 Subject: [Catalog-sig] implementing PEP 262 as academic work In-Reply-To: <427A0F5F.4080009@v.loewis.de> References: <200505030702.j437258A025261@mail28.syd.optusnet.com.au> <42772E29.90204@colorstudy.com> <42774265.6000006@acm.org> <427A0F5F.4080009@v.loewis.de> Message-ID: <427A1837.3060109@acm.org> Thanks for your pointers, Martin. From your point of view, I do agree that you do not need more intervention from disutils. In fact, if the package maintainers so desire, you do not even need disutils at all. So the question here is, is PEP 262 really needed at all? I can see that Martin or anyone using Debian may not need it. However, at where I am (MacOSX with Fink), I need it. What is the current plans now? Cheers Maurice >Maurice Ling wrote: > > >>Given that (1) there can be multiple versions of Python installed in a >>system, (2) each version maintains their own site-packages directory and >>(3) if C modules are installed, they are not compatible with other >>versions of Python... Imagine a system administrator who had installed >>50 libraries and their dependencies in site-package of Python 2.3 and >>now has to do it for Python 2.4, can we make his life better? >> >> > >I'd like to point out that this is a task that the "native" package >format can solve. For example, in Debian, when I use Debian packages >to install the 50 libraries, the current Debian Python policy manages >to update all the libraries from Python 2.3 to Python 2.4, when >the "official" Debian Python version becomes 2.4. > >This is achieved by providing, for a library "foo", several packages: >python-foo, python2.2-foo, python2.3-foo, python2.4-foo. As a user, >I install python-foo, which currently depends on python2.3-foo. When >/usr/bin/python becomes 2.4, a new version of python-foo will be >released which depends on python2.4-foo. Just by updating all packages >I have installed, it automatically installs python2.4-foo as well. > >Of course, if I don't want to wait for Debian to make /usr/bin/python >Python 2.4, I easily upgrade all libraries myself: > >dpkg --get-selections|grep python2.3-|sed s/2.3/2.4/|dpkg --set-selections > >will arrange to select all 2.4 libraries for which I have 2.3 >versions installed. > >So on Debian, I don't need any further support from distutils. > >Regards, >Martin > > > -- Maurice Han Tong LING, BSc(Hons)(Biochem), AdvDipComp, CPT, SSN, FIFA, MASBMB, MAMBIS, MACM Doctor of Philosophy (Science) Candidate, The University of Melbourne mobile: +61 4 22781753, +65 96669233 mailing address: Department of Zoology, The University of Melbourne Royal Parade, Parkville, Victoria 3010, Australia residence: 9/41 Dover Street, Flemington, Victoria 3031, Australia resume: http://maurice.vodien.com/maurice_resume.pdf www: http://www.geocities.com/beldin79/ -------------- next part -------------- A non-text attachment was scrubbed... Name: mauriceling.vcf Type: text/x-vcard Size: 334 bytes Desc: not available Url : http://mail.python.org/pipermail/catalog-sig/attachments/20050505/83cd7984/mauriceling.vcf From bob at redivi.com Thu May 5 16:04:43 2005 From: bob at redivi.com (Bob Ippolito) Date: Thu, 5 May 2005 10:04:43 -0400 Subject: [Catalog-sig] implementing PEP 262 as academic work In-Reply-To: <427A1837.3060109@acm.org> References: <200505030702.j437258A025261@mail28.syd.optusnet.com.au> <42772E29.90204@colorstudy.com> <42774265.6000006@acm.org> <427A0F5F.4080009@v.loewis.de> <427A1837.3060109@acm.org> Message-ID: <7D0BC443-0438-43D0-835B-EE9B6B5BD155@redivi.com> On May 5, 2005, at 8:57 AM, Maurice Ling wrote: >> Maurice Ling wrote: >> >> >>> Given that (1) there can be multiple versions of Python installed >>> in a >>> system, (2) each version maintains their own site-packages >>> directory and >>> (3) if C modules are installed, they are not compatible with other >>> versions of Python... Imagine a system administrator who had >>> installed >>> 50 libraries and their dependencies in site-package of Python 2.3 >>> and >>> now has to do it for Python 2.4, can we make his life better? >> >> I'd like to point out that this is a task that the "native" package >> format can solve. For example, in Debian, when I use Debian packages >> to install the 50 libraries, the current Debian Python policy manages >> to update all the libraries from Python 2.3 to Python 2.4, when >> the "official" Debian Python version becomes 2.4. >> > From your point of view, I do agree that you do not need more > intervention from disutils. In fact, if the package maintainers so > desire, you do not even need disutils at all. So the question here > is, is PEP 262 really needed at all? I can see that Martin or > anyone using Debian may not need it. However, at where I am (MacOSX > with Fink), I need it. > Since Fink uses the same software that Debian does, why shouldn't the same upgrade pathway work? What did they screw up? They've screwed up other things, so I don't use or trust Fink, but I'm just curious... When using Mac OS X without the help of a "trusted third party" like Darwinports or Fink, or when using Windows (where there isn't such an option), there is a most definitely a use case for PEP 262. Additionally, the Python world generally moves a hell of a lot faster than the Debian world, so there's probably some reasons to have PEP 262 even in such an "ideal" world where dependencies (almost) "just work" without Python's help. -bob From ianb at colorstudy.com Thu May 5 19:38:04 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 05 May 2005 12:38:04 -0500 Subject: [Catalog-sig] (no subject) In-Reply-To: <427A0F5F.4080009@v.loewis.de> References: <200505030702.j437258A025261@mail28.syd.optusnet.com.au> <42772E29.90204@colorstudy.com> <42774265.6000006@acm.org> <427A0F5F.4080009@v.loewis.de> Message-ID: <427A59FC.1000302@colorstudy.com> Martin v. L?wis wrote: > Maurice Ling wrote: > >>Given that (1) there can be multiple versions of Python installed in a >>system, (2) each version maintains their own site-packages directory and >>(3) if C modules are installed, they are not compatible with other >>versions of Python... Imagine a system administrator who had installed >>50 libraries and their dependencies in site-package of Python 2.3 and >>now has to do it for Python 2.4, can we make his life better? > > > I'd like to point out that this is a task that the "native" package > format can solve. For example, in Debian, when I use Debian packages > to install the 50 libraries, the current Debian Python policy manages > to update all the libraries from Python 2.3 to Python 2.4, when > the "official" Debian Python version becomes 2.4. Just as a reminder, what Maurice talked about actually doesn't relate to my initial concern. Or, maybe I'll call it my distutils-love, where distutils does something currently that system packagers don't do. I'm not so worried about per-python-version installation of packages -- for the most part this has worked well for some time. It's the support of multiple *library* versions that keeps me using distutils, and actually keeps me from using native packages for most things, except those places where there's already a centralized system (e.g., I don't install more than one version of Postgres on a server, so I don't need more than one version of psycopg). Or things that are stable and practically equivalent to "standard" for me, like mxDateTime. But for most libraries I'm very reluctant to install them globally; I don't want to do monolithic upgrades of systems, I'd rather do incremental upgrades of specific applications, and handle any large upgrades through centralized tracking of the installations. In the C world this is more-or-less solved through a system of symlinks, e.g., libjpeg.so, libjpeg.so.9, libjpeg.so.9.1, etc. But we don't have that for Python. So if I have one application that needs SQLObject 0.5, and another that needs SQLObject 0.6, I just can't use any system package manager. At first I was thinking, well, that's because of my specific situation -- multi-client host machines, applications that go quiet for extended periods of time and shouldn't be disturbed, and my own tendency to use lots of in-development libraries (my own and other people's). But as I think about it, it seems like it applies to most situations. Subway, for instance, is built against a specific version of SQLObject. UltraGleeper is built against a different version (just to pick two projects I know of off the top of my head). Well, technically they used different package names for the different versions, but the general problem should be fairly obvious. Making the installation robust and easy for these kinds of things -- both applications and frameworks -- is what I think we are trying to do. But as long as we have unversioned package installation and imports, I think it's a bad idea to do system-wide installation most packages. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From mauriceling at acm.org Fri May 6 01:36:28 2005 From: mauriceling at acm.org (Maurice Ling) Date: Fri, 06 May 2005 09:36:28 +1000 Subject: [Catalog-sig] (no subject) In-Reply-To: References: <200505030702.j437258A025261@mail28.syd.optusnet.com.au> <42772E29.90204@colorstudy.com> <42774265.6000006@acm.org> <427A0F5F.4080009@v.loewis.de> <427A59FC.1000302@colorstudy.com> Message-ID: <427AADFC.9030100@acm.org> Bob Ippolito wrote: >> >> Just as a reminder, what Maurice talked about actually doesn't >> relate to >> my initial concern. Or, maybe I'll call it my distutils-love, where >> distutils does something currently that system packagers don't do. I'm >> not so worried about per-python-version installation of packages -- for >> the most part this has worked well for some time. It's the support of >> multiple *library* versions that keeps me using distutils, and actually >> keeps me from using native packages for most things, except those >> places >> where there's already a centralized system (e.g., I don't install more >> than one version of Postgres on a server, so I don't need more than one >> version of psycopg). Or things that are stable and practically >> equivalent to "standard" for me, like mxDateTime. >> >> But for most libraries I'm very reluctant to install them globally; I >> don't want to do monolithic upgrades of systems, I'd rather do >> incremental upgrades of specific applications, and handle any large >> upgrades through centralized tracking of the installations. >> >> In the C world this is more-or-less solved through a system of >> symlinks, >> e.g., libjpeg.so, libjpeg.so.9, libjpeg.so.9.1, etc. But we don't have >> that for Python. So if I have one application that needs SQLObject >> 0.5, >> and another that needs SQLObject 0.6, I just can't use any system >> package manager. >> >> At first I was thinking, well, that's because of my specific situation >> -- multi-client host machines, applications that go quiet for >> extended periods of time and shouldn't be disturbed, and my own >> tendency to use >> lots of in-development libraries (my own and other people's). But as I >> think about it, it seems like it applies to most situations. Subway, >> for instance, is built against a specific version of SQLObject. >> UltraGleeper is built against a different version (just to pick two >> projects I know of off the top of my head). Well, technically they >> used >> different package names for the different versions, but the general >> problem should be fairly obvious. Making the installation robust and >> easy for these kinds of things -- both applications and frameworks >> -- is >> what I think we are trying to do. But as long as we have unversioned >> package installation and imports, I think it's a bad idea to do >> system-wide installation most packages. > > > In other words, Python has "DLL hell" and there isn't a damned thing > that a system package manager can do about it other than say "sorry, > you can't install FooBar and CheeseFactory at the same time, because > FooBar needs BazWibble 1.0 and CheeseFactory needs BazWibble 2.0". > > Therefore, there needs to be some mechanism to install stuff to an > "app-packages" because it's not always possible or appropriate to put > everything in "site-packages". > > ... however, the referenced PEP does nothing to help this problem > really. > Yes, I agree that PEP 262 doesn't help to solve local packages (app-packages) as compared to global packages (site-packages). But it can be extended into that direction. Given that different applications may require different versions of a package, and disutils installs newer versions of the package over older versions (I'm not sure on this one, although the end results seems pretty much this way), there is a reason why this feature is useful. Thanks guys. Cheers Maurice -- Maurice Han Tong LING, BSc(Hons)(Biochem), AdvDipComp, CPT, SSN, FIFA, MASBMB, MAMBIS, MACM Doctor of Philosophy (Science) Candidate, The University of Melbourne mobile: +61 4 22781753, +65 96669233 mailing address: Department of Zoology, The University of Melbourne Royal Parade, Parkville, Victoria 3010, Australia residence: 9/41 Dover Street, Flemington, Victoria 3031, Australia resume: http://maurice.vodien.com/maurice_resume.pdf www: http://www.geocities.com/beldin79/ -------------- next part -------------- A non-text attachment was scrubbed... Name: mauriceling.vcf Type: text/x-vcard Size: 334 bytes Desc: not available Url : http://mail.python.org/pipermail/catalog-sig/attachments/20050506/4623bf1d/mauriceling.vcf From bob at redivi.com Thu May 5 20:37:42 2005 From: bob at redivi.com (Bob Ippolito) Date: Thu, 5 May 2005 14:37:42 -0400 Subject: [Catalog-sig] (no subject) In-Reply-To: <427A59FC.1000302@colorstudy.com> References: <200505030702.j437258A025261@mail28.syd.optusnet.com.au> <42772E29.90204@colorstudy.com> <42774265.6000006@acm.org> <427A0F5F.4080009@v.loewis.de> <427A59FC.1000302@colorstudy.com> Message-ID: On May 5, 2005, at 1:38 PM, Ian Bicking wrote: > Martin v. L?wis wrote: > >> Maurice Ling wrote: >> >> >>> Given that (1) there can be multiple versions of Python installed >>> in a >>> system, (2) each version maintains their own site-packages >>> directory and >>> (3) if C modules are installed, they are not compatible with other >>> versions of Python... Imagine a system administrator who had >>> installed >>> 50 libraries and their dependencies in site-package of Python 2.3 >>> and >>> now has to do it for Python 2.4, can we make his life better? >>> >> >> >> I'd like to point out that this is a task that the "native" package >> format can solve. For example, in Debian, when I use Debian packages >> to install the 50 libraries, the current Debian Python policy manages >> to update all the libraries from Python 2.3 to Python 2.4, when >> the "official" Debian Python version becomes 2.4. >> > > Just as a reminder, what Maurice talked about actually doesn't > relate to > my initial concern. Or, maybe I'll call it my distutils-love, where > distutils does something currently that system packagers don't do. > I'm > not so worried about per-python-version installation of packages -- > for > the most part this has worked well for some time. It's the support of > multiple *library* versions that keeps me using distutils, and > actually > keeps me from using native packages for most things, except those > places > where there's already a centralized system (e.g., I don't install more > than one version of Postgres on a server, so I don't need more than > one > version of psycopg). Or things that are stable and practically > equivalent to "standard" for me, like mxDateTime. > > But for most libraries I'm very reluctant to install them globally; I > don't want to do monolithic upgrades of systems, I'd rather do > incremental upgrades of specific applications, and handle any large > upgrades through centralized tracking of the installations. > > In the C world this is more-or-less solved through a system of > symlinks, > e.g., libjpeg.so, libjpeg.so.9, libjpeg.so.9.1, etc. But we don't > have > that for Python. So if I have one application that needs SQLObject > 0.5, > and another that needs SQLObject 0.6, I just can't use any system > package manager. > > At first I was thinking, well, that's because of my specific situation > -- multi-client host machines, applications that go quiet for > extended > periods of time and shouldn't be disturbed, and my own tendency to use > lots of in-development libraries (my own and other people's). But > as I > think about it, it seems like it applies to most situations. Subway, > for instance, is built against a specific version of SQLObject. > UltraGleeper is built against a different version (just to pick two > projects I know of off the top of my head). Well, technically they > used > different package names for the different versions, but the general > problem should be fairly obvious. Making the installation robust and > easy for these kinds of things -- both applications and frameworks > -- is > what I think we are trying to do. But as long as we have unversioned > package installation and imports, I think it's a bad idea to do > system-wide installation most packages. In other words, Python has "DLL hell" and there isn't a damned thing that a system package manager can do about it other than say "sorry, you can't install FooBar and CheeseFactory at the same time, because FooBar needs BazWibble 1.0 and CheeseFactory needs BazWibble 2.0". Therefore, there needs to be some mechanism to install stuff to an "app-packages" because it's not always possible or appropriate to put everything in "site-packages". ... however, the referenced PEP does nothing to help this problem really. -bob From janc13 at gmail.com Fri May 6 19:49:30 2005 From: janc13 at gmail.com (JanC) Date: Fri, 6 May 2005 19:49:30 +0200 Subject: [Catalog-sig] (no subject) In-Reply-To: <984838bf050506104419b3ce3@mail.gmail.com> References: <200505030702.j437258A025261@mail28.syd.optusnet.com.au> <42772E29.90204@colorstudy.com> <42774265.6000006@acm.org> <427A0F5F.4080009@v.loewis.de> <427A59FC.1000302@colorstudy.com> <984838bf050506104419b3ce3@mail.gmail.com> Message-ID: <984838bf0505061049504a790c@mail.gmail.com> Oops, sorry for the empty mail... :-/. On 5/6/05, JanC wrote: > On 5/5/05, Bob Ippolito wrote: > > > > In other words, Python has "DLL hell" and there isn't a damned thing > > that a system package manager can do about it other than say "sorry, > > you can't install FooBar and CheeseFactory at the same time, because > > FooBar needs BazWibble 1.0 and CheeseFactory needs BazWibble 2.0". > > > > Therefore, there needs to be some mechanism to install stuff to an > > "app-packages" because it's not always possible or appropriate to put > > everything in "site-packages". wxPython has solved this more or less with a mechanism that people allows to use the default version or a specific version.if they want, with something like this: import wxversion wxversion.select("2.4") import wx -- JanC From ianb at colorstudy.com Fri May 6 21:02:41 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 06 May 2005 14:02:41 -0500 Subject: [Catalog-sig] (no subject) In-Reply-To: <984838bf0505061049504a790c@mail.gmail.com> References: <200505030702.j437258A025261@mail28.syd.optusnet.com.au> <42772E29.90204@colorstudy.com> <42774265.6000006@acm.org> <427A0F5F.4080009@v.loewis.de> <427A59FC.1000302@colorstudy.com> <984838bf050506104419b3ce3@mail.gmail.com> <984838bf0505061049504a790c@mail.gmail.com> Message-ID: <427BBF51.90405@colorstudy.com> JanC wrote: >>On 5/5/05, Bob Ippolito wrote: >> >>>In other words, Python has "DLL hell" and there isn't a damned thing >>>that a system package manager can do about it other than say "sorry, >>>you can't install FooBar and CheeseFactory at the same time, because >>>FooBar needs BazWibble 1.0 and CheeseFactory needs BazWibble 2.0". >>> >>>Therefore, there needs to be some mechanism to install stuff to an >>>"app-packages" because it's not always possible or appropriate to put >>>everything in "site-packages". > > > wxPython has solved this more or less with a mechanism that people > allows to use the default version or a specific version.if they want, > with something like this: > > import wxversion > wxversion.select("2.4") > import wx Python Eggs also address this, like: require("Twisted-Internet>=2.0") import twisted.internet More on Eggs: http://peak.telecommunity.com/DevCenter/PythonEggs They also aren't really "installed", except at runtime by adding them to sys.path. I've only played around with them a little, but it's an interesting way to handle things. It feels like it resolves (through avoidance) a lot of issues of centralized installation. But this moves things into the scope of distutils-sig: http://www.python.org/sigs/distutils-sig/ -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Mon May 23 00:34:37 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 22 May 2005 17:34:37 -0500 Subject: [Catalog-sig] Back to RPC Message-ID: <429108FD.7080405@colorstudy.com> OK, I'm trying to make package installation easier for Paste, which comes back to PyPI pretty quickly, of course. I think all the information I really need is how to get an sdist package, or a checkout -- right now it's important to be to be able to install out of repositories, not just released versions. So, I think all the data is there now. I might search package_urls for the right release, packagetype 'sdist'. Though there isn't any place for a repository URL, and that gets a little vague -- in theory you really could have a per-version repository in the form of a branch. But that's not very useful information. But maybe that can be disambiguated with a URL "svn-trunk" or "cvs-trunk", which doesn't imply that the repository is version-specific, even if the database schema requires it be put there. From there it's all client-side work to install the package, so it doesn't involve PyPI. The only issue is giving URLs to packages that don't have URLs. It would be a lot easier from my perspective if I didn't have to go through the package author when URLs were missing, maybe with notification going back to the owner so they can adjust their setup.py file. But lets ignore that for a little while, though it's also very important to me, as I'd like to use this stuff sooner rather than later. Anyway, summarizing: * Agree and document release_urls.packagetype values. I'd like: * sdist * bdist * bdist_dumb * bdist_rpm * bdist_wininst * bdist_egg * svn-tag * svn-trunk * cvs-trunk (is there any agreed-upon format for CVS URLs? What about branches and tags?) Also, these really need to be documented somewhere on the pypi site, that's the only way agreeing on it means anything ;) * Add some functions to rpc.py: * package_releases(package_name): list of release versions, as strings * package_stable_version(package_name): packages.stable_version * package_urls(package_name, version): A list of {'url': url, 'packagetype': packagetype} * package_data(package_name, version): A dictionary that basically summarizes the releases table, plus release_classifiers. * providing_packages(specifier): A list of (name, version) from release_provides. * requiring_packages(specifier): A list of (name, version) from release_requires. I don't entirely understand "specifier" and "release_obsoletes". Are specifiers package names, or a slightly more abstract version of package names? Or something more structured? While it isn't important to me now, I think it would be good to change rpc.search to take a single dictionary argument, which it would pass in as the query spec. Probably the return value should also be cleaned up, as store.query_packages returns a kind of odd data structure. The adding URL thing is a bit more of a policy decision. Maybe we can just add a form to request a role, which will email the owner and give them a quick form (maybe just one link) to add the user. If I can get CVS commit access I can start working on some of these; a recent database dump would also be useful for testing. Incidentally, I notice the PyPI page is quite slow. Thoughts? I think I'm noticing that templates are loaded off disk for each request, which be really slow. Is PyPI running under mod_python now? -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From richardjones at optushome.com.au Mon May 23 01:45:51 2005 From: richardjones at optushome.com.au (Richard Jones) Date: Mon, 23 May 2005 09:45:51 +1000 Subject: [Catalog-sig] Back to RPC In-Reply-To: <429108FD.7080405@colorstudy.com> References: <429108FD.7080405@colorstudy.com> Message-ID: <200505230945.51529.richardjones@optushome.com.au> [Barry - SVN question for you buried down below] On Mon, 23 May 2005 08:34 am, Ian Bicking wrote: > I think all the information I really need is how to get an sdist > package, or a checkout -- right now it's important to be to be able to > install out of repositories, not just released versions. There's something that's not been on *my* radar :) Do we really want this in PyPI's database? Related: will people really want to *maintain* this stuff in PyPI's database? > So, I think all the data is there now. I might search package_urls for > the right release, packagetype 'sdist'. Though there isn't any place > for a repository URL, and that gets a little vague -- in theory you > really could have a per-version repository in the form of a branch. But > that's not very useful information. But maybe that can be disambiguated > with a URL "svn-trunk" or "cvs-trunk", which doesn't imply that the > repository is version-specific, even if the database schema requires it > be put there. Well, as soon as you start talking SVN *and* CVS you talking about totally different meta-data. For SVN, you've got just a URL (in my understanding of SVN's branching/tagging) and for CVS you've got a ... well, it's not quite a URL, and a tag name. And of course, this ignores the plethora of other revision-control repositories that people use. So *perhaps* you just want to mandate SVN and just deal with losing all the potential Sourceforge contributions? I'm not sure what it is you're actually trying to do, so I couldn't say :) > From there it's all client-side work to install the package, so it > doesn't involve PyPI. The only issue is giving URLs to packages that > don't have URLs. It would be a lot easier from my perspective if I > didn't have to go through the package author when URLs were missing, > maybe with notification going back to the owner so they can adjust their > setup.py file. But lets ignore that for a little while, though it's > also very important to me, as I'd like to use this stuff sooner rather > than later. URLs/uploaded files for all registered packages is something I don't think we'll ever achieve. > Anyway, summarizing: > > * Agree and document release_urls.packagetype values. I'd like: > * sdist > * bdist > * bdist_dumb > * bdist_rpm > * bdist_wininst > * bdist_egg > * svn-tag > * svn-trunk > * cvs-trunk (is there any agreed-upon format for CVS URLs? What about > branches and tags?) > > Also, these really need to be documented somewhere on the pypi site, > that's the only way agreeing on it means anything ;) They should be listed in the PEP (or non-PEP) design document you're writing (hint hint ). > * Add some functions to rpc.py: > * package_releases(package_name): list of release versions, as strings > * package_stable_version(package_name): packages.stable_version > * package_urls(package_name, version): A list of {'url': url, > 'packagetype': packagetype} > * package_data(package_name, version): A dictionary that basically > summarizes the releases table, plus release_classifiers. > * providing_packages(specifier): A list of (name, version) from > release_provides. > * requiring_packages(specifier): A list of (name, version) from > release_requires. Good list. > I don't entirely understand "specifier" and "release_obsoletes". Are > specifiers package names, or a slightly more abstract version of package > names? Or something more structured? See the PEP: http://www.python.org/peps/pep-0314.html > While it isn't important to me now, I think it would be good to change > rpc.search to take a single dictionary argument, which it would pass in > as the query spec. Probably the return value should also be cleaned up, > as store.query_packages returns a kind of odd data structure. Sounds fine to me. As I understand it, the code there is more of a proof-of-concept. > The adding URL thing is a bit more of a policy decision. Maybe we can > just add a form to request a role, which will email the owner and give > them a quick form (maybe just one link) to add the user. Er, so this is someone asking to have permission to add the URL on behalf of the package owner? Seems reasonable to me, I suppose. > If I can get CVS commit access I can start working on some of these; a > recent database dump would also be useful for testing. PyPI's just moved to svn.python.org, but I couldn't say more than that 'cos I don't know how it works. I've cc'ed Barry Warsaw in case he's not on this list. > Incidentally, I notice the PyPI page is quite slow. Thoughts? I think > I'm noticing that templates are loaded off disk for each request, which > be really slow. Is PyPI running under mod_python now? It's being moved right now. AMK's in the final stages of apache config. Something that concerns me about all this is the referencing of older versions of software. I can't help this niggling feeling I have of people down the track asking package developers to fix bugs in ancient branches of their code. And people on both sides of the equation getting snarky about it. Richard -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/catalog-sig/attachments/20050523/f287c438/attachment.pgp From ianb at colorstudy.com Mon May 23 02:54:11 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 22 May 2005 19:54:11 -0500 Subject: [Catalog-sig] Back to RPC In-Reply-To: <200505230945.51529.richardjones@optushome.com.au> References: <429108FD.7080405@colorstudy.com> <200505230945.51529.richardjones@optushome.com.au> Message-ID: <429129B3.4040207@colorstudy.com> Richard Jones wrote: > [Barry - SVN question for you buried down below] > > On Mon, 23 May 2005 08:34 am, Ian Bicking wrote: > >>I think all the information I really need is how to get an sdist >>package, or a checkout -- right now it's important to be to be able to >>install out of repositories, not just released versions. > > > There's something that's not been on *my* radar :) > > Do we really want this in PyPI's database? Related: will people really want to > *maintain* this stuff in PyPI's database? Well, it's a pretty stable URL for most projects, more stable than the download URL. Also, if we have tools that use the data, they *have* to be maintained, so they will be maintained ;) -- but it's also why I'm concerned that these things are correctable regardless of the presence or interest of the package "owner". There will be workarounds -- I'd rather those happen in PyPI than to build those into external tools and external annotations of the index. >>So, I think all the data is there now. I might search package_urls for >>the right release, packagetype 'sdist'. Though there isn't any place >>for a repository URL, and that gets a little vague -- in theory you >>really could have a per-version repository in the form of a branch. But >>that's not very useful information. But maybe that can be disambiguated >>with a URL "svn-trunk" or "cvs-trunk", which doesn't imply that the >>repository is version-specific, even if the database schema requires it >>be put there. > > > Well, as soon as you start talking SVN *and* CVS you talking about totally > different meta-data. For SVN, you've got just a URL (in my understanding of > SVN's branching/tagging) and for CVS you've got a ... well, it's not quite a > URL, and a tag name. Yeah, it's too bad there aren't real URLs for CVS. You could do pserver:hostname/path.../modulename, or something like that, but it's just making it up. > And of course, this ignores the plethora of other revision-control > repositories that people use. Yes, but it's a process. Maybe darcs-trunk could be added, and so on. Right now I'm actually only concerned about svn, because all the projects I care about installing automatically are in svn. I wouldn't be implementing anything else in my own client libraries. It's a problem for me, because I'm starting to maintain or be involved with several very interrelated projects that are most useful when used in concert, but have valid independent existances. So if I'm working with any one of them outside of a released version, there's a good chance all of them need to be installed from checkouts. (Incidentally, I'm not expecting to install any of these globally; I've pretty much given up on globally installing any but a handful of non-stdlib packages) > So *perhaps* you just want to mandate SVN and just deal with losing all the > potential Sourceforge contributions? I'm not sure what it is you're actually > trying to do, so I couldn't say :) Sure. It's a process; if there's demand (and code), then there's nothing stopping CVS repositories from having their access encoded into a single URI-like string. There's a lot of projects still in CVS, but not a lot of the immature projects where installing non-released versions is particularly useful. >>From there it's all client-side work to install the package, so it >>doesn't involve PyPI. The only issue is giving URLs to packages that >>don't have URLs. It would be a lot easier from my perspective if I >>didn't have to go through the package author when URLs were missing, >>maybe with notification going back to the owner so they can adjust their >>setup.py file. But lets ignore that for a little while, though it's >>also very important to me, as I'd like to use this stuff sooner rather >>than later. > > > URLs/uploaded files for all registered packages is something I don't think > we'll ever achieve. If we make it matter, that will help a lot. >>Anyway, summarizing: >> >>* Agree and document release_urls.packagetype values. I'd like: >> * sdist >> * bdist >> * bdist_dumb >> * bdist_rpm >> * bdist_wininst >> * bdist_egg >> * svn-tag >> * svn-trunk >> * cvs-trunk (is there any agreed-upon format for CVS URLs? What about >> branches and tags?) >> >>Also, these really need to be documented somewhere on the pypi site, >>that's the only way agreeing on it means anything ;) > > > They should be listed in the PEP (or non-PEP) design document you're writing > (hint hint ). Oh, sure, I guess I can do that. Does the PEP process really add much here, or would a document be sufficient? The presence of PyPI documentation in the PEP index is actually a bit unhelpful, compared to documentation sitting right in PyPI (and maybe besides distutil-related documentation). >>* Add some functions to rpc.py: >> * package_releases(package_name): list of release versions, as strings >> * package_stable_version(package_name): packages.stable_version >> * package_urls(package_name, version): A list of {'url': url, >> 'packagetype': packagetype} >> * package_data(package_name, version): A dictionary that basically >> summarizes the releases table, plus release_classifiers. >> * providing_packages(specifier): A list of (name, version) from >> release_provides. >> * requiring_packages(specifier): A list of (name, version) from >> release_requires. > > > Good list. > > > >>I don't entirely understand "specifier" and "release_obsoletes". Are >>specifiers package names, or a slightly more abstract version of package >>names? Or something more structured? > > > See the PEP: http://www.python.org/peps/pep-0314.html OK. I won't plan to do much with this information right now, but maybe if there's easy access to the information someone will take up the cause. >>While it isn't important to me now, I think it would be good to change >>rpc.search to take a single dictionary argument, which it would pass in >>as the query spec. Probably the return value should also be cleaned up, >>as store.query_packages returns a kind of odd data structure. > > > Sounds fine to me. As I understand it, the code there is more of a > proof-of-concept. Certainly, there's not a useful list of functions currently. >>The adding URL thing is a bit more of a policy decision. Maybe we can >>just add a form to request a role, which will email the owner and give >>them a quick form (maybe just one link) to add the user. > > > Er, so this is someone asking to have permission to add the URL on behalf of > the package owner? > > Seems reasonable to me, I suppose. Yes, something like that. Maybe just building a contact form into the package display (or linked off that) is sufficient, that would mail all owners and maintainers of a package (with language in the form to indicate that only metadata errors should be reported through that form). Later, maybe something could be added to record these issues, let the owners mark them "resolved", and put out whiney messages to owners if they don't do resolve them, and maybe alert "editors" eventually. But just a contact form is a start. >>If I can get CVS commit access I can start working on some of these; a >>recent database dump would also be useful for testing. > > > PyPI's just moved to svn.python.org, but I couldn't say more than that 'cos I > don't know how it works. I've cc'ed Barry Warsaw in case he's not on this > list. OK. Could I get CVS access too? (username: ianbicking) >>Incidentally, I notice the PyPI page is quite slow. Thoughts? I think >>I'm noticing that templates are loaded off disk for each request, which >>be really slow. Is PyPI running under mod_python now? > > > It's being moved right now. AMK's in the final stages of apache config. OK, then the speed doesn't really mean anything, and if it's CGI there's no useful caching of templates that can be done. But I can look at that too. > Something that concerns me about all this is the referencing of older versions > of software. I can't help this niggling feeling I have of people down the > track asking package developers to fix bugs in ancient branches of their > code. And people on both sides of the equation getting snarky about it. Well, you can't predict or solve all future problems ;) I assume most of the client tools will heavily prefer the "stable version" of the package. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From martin at v.loewis.de Mon May 23 07:19:50 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 23 May 2005 07:19:50 +0200 Subject: [Catalog-sig] Back to RPC In-Reply-To: <429129B3.4040207@colorstudy.com> References: <429108FD.7080405@colorstudy.com> <200505230945.51529.richardjones@optushome.com.au> <429129B3.4040207@colorstudy.com> Message-ID: <429167F6.7020400@v.loewis.de> Ian Bicking wrote: > Oh, sure, I guess I can do that. Does the PEP process really add much > here, or would a document be sufficient? It might be sufficient, provided it contains the same information that a PEP contains. I.e. I would like to see a problem statement, and a discussion of alternatives (based on the feedback that you get for the initial document). I find that your initial draft describes a very complex system, and I'm concerned that it is too complex for most users. It seems to be similar to the SF file release system (requiring me to describe all sorts of meta-information with choices from a set of options that don't help to describe *my* package), so I wonder whether this change would make PyPI less usable. Regards, Martin From ianb at colorstudy.com Mon May 23 08:29:59 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 23 May 2005 01:29:59 -0500 Subject: [Catalog-sig] Back to RPC In-Reply-To: <429167F6.7020400@v.loewis.de> References: <429108FD.7080405@colorstudy.com> <200505230945.51529.richardjones@optushome.com.au> <429129B3.4040207@colorstudy.com> <429167F6.7020400@v.loewis.de> Message-ID: <42917867.3000008@colorstudy.com> Martin v. L?wis wrote: > Ian Bicking wrote: > >>Oh, sure, I guess I can do that. Does the PEP process really add much >>here, or would a document be sufficient? > > > It might be sufficient, provided it contains the same information that > a PEP contains. I.e. I would like to see a problem statement, and > a discussion of alternatives (based on the feedback that you get for > the initial document). Sure, I'll try to do that. But like I said, it's something I'd like to use very soon, and I was actually in the middle of implementing it separately when I realized that it was closer to the scope of PyPI than I at first thought. I'd at least like to have a read-only devel copy of this stuff that I could experiment with. If it's in a branch or whatever, that's fine. If I have to host my own copy of PyPI for a while as we discuss the additions that'd even be fine. Whatever -- I just want to move forward on this, and discuss the good and bad of specific code; if it means I write code that's thrown away, that doesn't really bother me. OTOH, there's a lot of purely internal refactorings to PyPI that would make many of these things easier while also making PyPI easier to maintain, and forking PyPI to do apply those would be a total waste. > I find that your initial draft describes a very complex system, and > I'm concerned that it is too complex for most users. It seems to > be similar to the SF file release system (requiring me to describe > all sorts of meta-information with choices from a set of options > that don't help to describe *my* package), so I wonder whether this > change would make PyPI less usable. I don't see many added complexities at all. It comes down to a couple of things: * Document what packagetype's are. Pretty much a no-brainer; what can be wrong about documentation? * I think those types should include svn-trunk. I can spell out use cases if people like -- they have a lot to do with interdependent packages that track each other's changes, and making that more usable for package users. * Some public XML-RPC methods. I think the ones I spelled out are pretty obvious; not necessarily complete, but a proper start. * Some resolution process for updating and correcting metadata, probably just starting with a contact form. This is primarily about PyPI-the-webapp, which doesn't really need to be that formalized (compared to PyPI-the-package-database). packagetype already exists. I don't know what the interface is to set it, but it's there in the database. I haven't proposed any new fields (though an optional "title" field for package URLs might be nice, kind of like a title field to tags). And I don't think SF is a good example of anything. I can't imagine a worse designed user interface. So it's not fair to use that as a counter argument to anything ;) -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Wed May 25 08:36:20 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 25 May 2005 01:36:20 -0500 Subject: [Catalog-sig] PyPI enhancement doc Message-ID: <42941CE4.7070206@colorstudy.com> Here's my first go at a document about the enhancements I asked about earlier... Package Types ------------- PyPI includes this table:: CREATE TABLE release_urls ( name TEXT, version TEXT, url TEXT, packagetype TEXT, FOREIGN KEY (name, version) REFERENCES releases (name, version) ); The ``packagetype`` column has no particular type or constraint. I suggest that it contain these types taken from ``distutils``: sdist: a source distribution (tarball, zip file, etc.) bdist: a built (binary) distribution bdist_dumb: a "dumb" built distribution bdist_rpm: an RPM distribution bdist_wininst: an executable installer for MS Windows bdist_egg: a Python Egg In addition, I would like these possible values: svn_trunk: the URL of a subversion repository; a setup.py file should be found directly under this URL. This URL will not necessarily change for different revisions of the package. svn_tag: the URL of a subversion repository. A branch or tag for this specific version. Issues ~~~~~~ The binary distributions are all potentially platform specific, but (except for Windows) the platform isn't encoded into the packagetype. Should packagetype be extended to include information like Python version? Should it have the same fields as the release_files table? Potentially even more information is required to fully describe the package type. Another change would be to refactor the database, so that release_files didn't have python_version, packagetype, or md5_digest fields, but rather a third table (referenced by both release_files and package_urls) reference this package-metadata field. Potentially the last portion of the URL is itself self-describing. The files produced by the bdist* methods generally are, and as long as the URL doesn't obscure that filename it can be automatically determined by clients. The sdist type does not indicate what kind of archive is used -- most notable, both zip and tar are possible. (Does sdist create zip files on its own?) However, I think the client can determine the proper way to unpack the file after downloading; this aspect of files is self-describing. An svn trunk is not a package file. However, it is largely equivalent to an sdist file, though it must be downloaded in a different way; actually performing the download is a client concern, so it doesn't really effect this. XML-RPC methods --------------- This is an initial list of proposed methods for PyPI to support: package_releases(package_name): returns list of release versions, as strings, e.g., ['0.1', '0.2b', '0.2'], in chronological order. package_stable_version(package_name): returns packages.stable_version; the current stable version of the package. E.g., the string '0.3' package_urls(package_name, version): A list of {'url': url, 'packagetype': packagetype}, like [{'url': 'http://svn.pythonpaste.org/Paste/trunk', 'packagetype': 'svn_trunk'}, {'url': 'http://pythonpaste.org/Paste-0.1.tar.gz', 'packagetype': 'sdist'}] package_data(package_name, version): A dictionary that summarizes the releases table, plus release_classifiers. E.g.: {'name': 'OpenRelease', 'version': '0.1.2', 'author': 'Richard Harris', 'author_email': 'goosequill at users.sourceforge.net', 'maintainer': '', 'maintainer_email': '', 'homepage': 'http://open-release.sourceforge.net', 'download_url': 'http://prdownloads.sourceforge.net/projects/open-release/OpenRelease-0.1.2.tar.gz', 'description': """OpenRelease is a Python module which automates the packaging, release and announcement of Open Source software. The pack class creates packages, which are defined by packer classes, manages versioning, and brings up your notes and changelog in an editor. The release class uploads the package to SourceForge, releases it through QRS, announces it on freshmeat and (if appropriate) on pypi.""", 'license': 'GNU General Public License', 'platform': 'any', 'classifiers': [ 'Development Status :: 4 - Beta', 'Environment :: Console', 'Intended Audience :: Developers', 'License :: OSI Approved :: GNU General Public License (GPL)', 'Natural Language :: English', 'Operating System :: OS Independent', 'Programming Language :: Python', 'Topic :: Software Development'], 'summary': '', 'description_html': '', 'keywords': '', } All keys are required. None/NULL is translated to ''. Open issues: will emails be obscured? Is keywords turned into a list? search(field_specifiers, [operator='and']): field_specifiers is a dictionary of {fieldname: searchvalue}. Returns a list like [(name, version)] of matching non-hidden records. The search values are case-insensitive and match any substring. The second argument indicates if all the field specifiers are ANDed or ORed together. The value defaults to 'and' and is case-insensitive. I'm a little soft on these, since I don't know if specifiers and the necessary metadata is really ready: providing_packages(specifier): A list of (name, version) from release_provides. E.g., providing_packages('PageTemplate>=1.0') == [('zpt', '1.0')]. This will only return non-hidden packages. requiring_packages(specifier): A list of (name, version) from release_requires. Web UI Enhancements ------------------- Search should be available with a single field, that searches all of package name, summary, description/description_html, keywords. This should be on the front page. There should be a contact form attached to each package that will be emailed to any people on record as owners or maintainers, for reporting bad or missing links or other incorrect data. Anything submitted through that form will also be stored in the database, and the owner should mark it "resolved" when the issue is corrected. Until the issue is resolved, the content will show up on the package page (so other people can see the comment). All links in such emails should have rel="nofollow" (if HTML anchors are created at all). This should be clearly marked as being for problem reports only; adding package comments is another issue. From exarkun at divmod.com Wed May 25 13:43:28 2005 From: exarkun at divmod.com (Jp Calderone) Date: Wed, 25 May 2005 11:43:28 GMT Subject: [Catalog-sig] PyPI enhancement doc In-Reply-To: <42941CE4.7070206@colorstudy.com> Message-ID: <20050525114328.559.209089629.divmod.quotient.9347@ohm> On Wed, 25 May 2005 01:36:20 -0500, Ian Bicking wrote: >Here's my first go at a document about the enhancements I asked about >earlier... > > >Package Types >------------- > >PyPI includes this table:: > > [snip] > >In addition, I would like these possible values: > >svn_trunk: > the URL of a subversion repository; a setup.py file should be > found directly under this URL. This URL will not necessarily > change for different revisions of the package. >svn_tag: > the URL of a subversion repository. A branch or tag for this > specific version. > Why not just svn_url, taking on values like "svn://svn.twistedmatrix.com/svn/Twisted/tags/2.0.1/@13844" or "svn://svn.twistedmatrix.com/svn/Twisted/trunk/@HEAD"? Jp From barry at python.org Wed May 25 13:49:18 2005 From: barry at python.org (Barry Warsaw) Date: Wed, 25 May 2005 07:49:18 -0400 Subject: [Catalog-sig] Back to RPC In-Reply-To: <200505230945.51529.richardjones@optushome.com.au> References: <429108FD.7080405@colorstudy.com> <200505230945.51529.richardjones@optushome.com.au> Message-ID: <1117021758.16104.675.camel@presto.wooz.org> On Sun, 2005-05-22 at 19:45, Richard Jones wrote: > > If I can get CVS commit access I can start working on some of these; a > > recent database dump would also be useful for testing. > > PyPI's just moved to svn.python.org, but I couldn't say more than that 'cos I > don't know how it works. I've cc'ed Barry Warsaw in case he's not on this > list. (I'm not on this list.) We're working (slowly) on migrating all the python.org services that are currently under CVS to be controlled by Subversion. We've started with the various system files for the two new machines donated by XS4ALL, and Andrew has been using the same setup to configure the planet.python.org stuff. We're still working out the details of how access will work for www content updates, i.e. whether it will be svn+ssh or https, or both, and how the site changes will get pushed out, etc. Specifically for cheeseshop, Andrew's been talking lately about how to set that up for efficient mirroring. Unfortunately, I've been too busy with other things to spend much time on this, but Andrew's been doing a lot lately. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/catalog-sig/attachments/20050525/cfa5fe27/attachment.pgp From ianb at colorstudy.com Wed May 25 17:17:34 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 25 May 2005 10:17:34 -0500 Subject: [Catalog-sig] PyPI enhancement doc In-Reply-To: <20050525114328.559.209089629.divmod.quotient.9347@ohm> References: <20050525114328.559.209089629.divmod.quotient.9347@ohm> Message-ID: <4294970E.1020502@colorstudy.com> Jp Calderone wrote: > Why not just svn_url, taking on values like > "svn://svn.twistedmatrix.com/svn/Twisted/tags/2.0.1/@13844" or > "svn://svn.twistedmatrix.com/svn/Twisted/trunk/@HEAD"? I hadn't seen the @ syntax before. Does that work with svn natively, for all the protocols svn supports? Also, I don't really see a way to distinguish between the two. And /tags/2.0.1/@HEAD seems like a more appropriate URL for the tag, since the directory (not the revision number) indicates the release; in theory (or by convention) the revision and the tag are identical in svn, but a branch could have updates to HEAD that actually apply to the given revision. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From martin at v.loewis.de Wed May 25 23:02:10 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 25 May 2005 23:02:10 +0200 Subject: [Catalog-sig] PyPI enhancement doc In-Reply-To: <4294970E.1020502@colorstudy.com> References: <20050525114328.559.209089629.divmod.quotient.9347@ohm> <4294970E.1020502@colorstudy.com> Message-ID: <4294E7D2.3040908@v.loewis.de> Ian Bicking wrote: > I hadn't seen the @ syntax before. Does that work with svn natively, > for all the protocols svn supports? AFAICT, it does not support it at all. I can't see where Jp got this syntax from. Regards, Martin From martin at v.loewis.de Wed May 25 23:23:37 2005 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 25 May 2005 23:23:37 +0200 Subject: [Catalog-sig] PyPI enhancement doc In-Reply-To: <42941CE4.7070206@colorstudy.com> References: <42941CE4.7070206@colorstudy.com> Message-ID: <4294ECD9.5010804@v.loewis.de> Ian Bicking wrote: > The ``packagetype`` column has no particular type or constraint. I > suggest that it contain these types taken from ``distutils``: I see. I hadn't understood this earlier - documenting that seems fine to me. > The binary distributions are all potentially platform specific, but > (except for Windows) the platform isn't encoded into the packagetype. > Should packagetype be extended to include information like Python > version? It was excluded because it can be implied from the package file (distutils makes sure the filenames don't conflict). upload.py puts the platform in the comment. Standardizing on platform names is a tricky issue: Is "SuSE" the same platform as "Debian"? Is it the same platform as "SuSE Linux"? > Another change would be to refactor the database, so that > release_files didn't have python_version, packagetype, or md5_digest > fields, but rather a third table (referenced by both release_files and > package_urls) reference this package-metadata field. What would be the advantage of doing so? If it is normalization: I don't see any advantage in normalization at all. > XML-RPC methods > --------------- > > This is an initial list of proposed methods for PyPI to support: I think this whole thing is highly debatable. Why XML-RPC? Are HTML forms not good enough for you? Why not CORBA? etc. It appears to be a design principle of PyPI to not use XML-RPC if it can be avoided. I don't know where that principle comes from, but I think it needs to be followed if indeed it is a principle. If there is no such principle, I think all the current forms should be duplicated to simplify the code in the distutils commands. > Search should be available with a single field, that searches all of > package name, summary, description/description_html, keywords. This > should be on the front page. See http://sourceforge.net/tracker/index.php?func=detail&aid=989926&group_id=66150&atid=513503 In general, details of the UI and the database can be easily integrated as long as they don't affect current users and as long as they don't affect distutils or other programmatic interfaces. So contributions are welcome, although I understand this is meant to be solved by Postgres full text search. > There should be a contact form attached to each package that will be > emailed to any people on record as owners or maintainers, for > reporting bad or missing links or other incorrect data. Again, contributions are welcome. We should be careful not to implement another issue tracker here, though, so just sending an email might be enough for the moment (or else a read-only roundup installation, writable only to PyPI and the maintainer might be the right solution). As soon as tracking is available, people will start using it to report bugs on the actual software itself, which might defeat the bug reporting channels the maintainers try to establish. So perhaps a "report problems with this record to" field might be sufficient, with the feedback form only available if the field is not filled. In any case, if you have something that works, please submit a patch to sf.net/projects/pypi. Regards, Martin From ianb at colorstudy.com Wed May 25 23:59:06 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 25 May 2005 16:59:06 -0500 Subject: [Catalog-sig] PyPI enhancement doc In-Reply-To: <4294ECD9.5010804@v.loewis.de> References: <42941CE4.7070206@colorstudy.com> <4294ECD9.5010804@v.loewis.de> Message-ID: <4294F52A.9040904@colorstudy.com> Incidentally, in case I forget, I should remind the group I have this file from that last sprint that gives PyPI a WSGI frontend: http://lonelylion.com/chipy/pypi/pypi_wsgi.py It's pretty straight-forward. It might be useful in porting PyPI to mod_python, or some other WSGI server. Incidentally with Paste you could use the configuration: publish_app = 'pypi.pypi_wsgi:pypi_app("/path/to/base")' Martin v. L?wis wrote: >>The binary distributions are all potentially platform specific, but >>(except for Windows) the platform isn't encoded into the packagetype. >>Should packagetype be extended to include information like Python >>version? > > > It was excluded because it can be implied from the package file > (distutils makes sure the filenames don't conflict). upload.py > puts the platform in the comment. > > Standardizing on platform names is a tricky issue: Is "SuSE" the > same platform as "Debian"? Is it the same platform as "SuSE Linux"? Yes, I agree, it's tricky. Lets say that the URL must include as its basename the filename that distutils produces, so that this can be used to recreate that filename: base = url.split('/')[-1] base = base.split('?')[0] >>Another change would be to refactor the database, so that >>release_files didn't have python_version, packagetype, or md5_digest >>fields, but rather a third table (referenced by both release_files and >>package_urls) reference this package-metadata field. > > > What would be the advantage of doing so? If it is normalization: > I don't see any advantage in normalization at all. Because those fields can apply to other packages, even if they aren't stored in PyPI's file repository. Just for normalization's sake I wouldn't really care. And really, I don't actually care if the fields are copied either. But anyway, if we are expecting the filename to be self-describing then this mostly doesn't matter. Except perhaps an optional md5_digest. >>XML-RPC methods >>--------------- >> >>This is an initial list of proposed methods for PyPI to support: > > > I think this whole thing is highly debatable. Why XML-RPC? Are HTML > forms not good enough for you? Why not CORBA? etc. XML-RPC is already in PyPI (and live!), in rpc.py. It just doesn't have any useful functions right now. > It appears to be a design principle of PyPI to not use XML-RPC if > it can be avoided. I don't know where that principle comes from, > but I think it needs to be followed if indeed it is a principle. > > If there is no such principle, I think all the current forms should > be duplicated to simplify the code in the distutils commands. You mean the (somewhat awkward, if I remember) construction of POST requests in distutils? Yes, I think it would be nice to move those to XML-RPC. That so many old distutils exist means the forms would have to stay around for some time though :( Or does it; can we ask people to update at least for submission? Eh, probably not for some time. >>Search should be available with a single field, that searches all of >>package name, summary, description/description_html, keywords. This >>should be on the front page. > > > See > > http://sourceforge.net/tracker/index.php?func=detail&aid=989926&group_id=66150&atid=513503 > > In general, details of the UI and the database can be easily > integrated as long as they don't affect current users and > as long as they don't affect distutils or other programmatic > interfaces. > > So contributions are welcome, although I understand this is > meant to be solved by Postgres full text search. OK. I expect there's some sysadmin issues to enabling full text search in Postgres at this point as well. >>There should be a contact form attached to each package that will be >>emailed to any people on record as owners or maintainers, for >>reporting bad or missing links or other incorrect data. > > > Again, contributions are welcome. We should be careful not to implement > another issue tracker here, though, so just sending an email might be > enough for the moment (or else a read-only roundup installation, > writable only to PyPI and the maintainer might be the right solution). > As soon as tracking is available, people will start using it to > report bugs on the actual software itself, which might defeat > the bug reporting channels the maintainers try to establish. > > So perhaps a "report problems with this record to" field might > be sufficient, with the feedback form only available if the field > is not filled. Is there a missing negation there? You mean, only allow reporting problems if that field is filled in? I figure it's good enough to send mail to everyone who has sufficient permission to resolve the problem, another field shouldn't be necessary. I'd like to display the message as well, so that readers can see message until it is corrected; of course, leaving it forever just builds cruft. Hence the resolved/unresolved thing. > In any case, if you have something that works, please submit > a patch to sf.net/projects/pypi. Well, for now I'll try to do something with rpc.py, since there's not enough code their to require refactoring (at which point patches start to become more challenging). -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From richardjones at optushome.com.au Thu May 26 02:15:45 2005 From: richardjones at optushome.com.au (Richard Jones) Date: Thu, 26 May 2005 10:15:45 +1000 Subject: [Catalog-sig] PyPI development no longer on sf.net Message-ID: <200505261015.45374.richardjones@optushome.com.au> The PyPI CVS on sf.net is no longer the repository of PyPI code. It has been moved to svn.python.org. If you need access to that repository, please contact me and I'll see whether I can figure something out. Richard -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/catalog-sig/attachments/20050526/e58eab10/attachment.pgp From exarkun at divmod.com Thu May 26 04:01:47 2005 From: exarkun at divmod.com (Jp Calderone) Date: Thu, 26 May 2005 02:01:47 GMT Subject: [Catalog-sig] PyPI enhancement doc In-Reply-To: <4294970E.1020502@colorstudy.com> Message-ID: <20050526020147.559.2045491994.divmod.quotient.9868@ohm> On Wed, 25 May 2005 10:17:34 -0500, Ian Bicking wrote: >Jp Calderone wrote: >> Why not just svn_url, taking on values like >> "svn://svn.twistedmatrix.com/svn/Twisted/tags/2.0.1/@13844" or >> "svn://svn.twistedmatrix.com/svn/Twisted/trunk/@HEAD"? > >I hadn't seen the @ syntax before. Does that work with svn natively, >for all the protocols svn supports? > >Also, I don't really see a way to distinguish between the two. And >/tags/2.0.1/@HEAD seems like a more appropriate URL for the tag, since >the directory (not the revision number) indicates the release; in theory >(or by convention) the revision and the tag are identical in svn, but a >branch could have updates to HEAD that actually apply to the given revision. > Hmmm, my mistake, I think. I thought it was general svn URL syntax, but it seems to be a special form just for the diff and merge commands. Still, if you think the logically right thing to do is to always use @HEAD, then the whole @ bit can be omitted, and it could just be svn_url=".../trunk" and svn_url=".../tags/2.0.1/", since HEAD is the implied revision in those cases. The reason I prefer @REV when looking at tags is that tags can be deleted. In such cases, trying to look at them @HEAD does not work, but trying to look at them at @REV, where REV is a revision at which they existed, does. But that's moot since it seems it won't work :) Jp From richardjones at optushome.com.au Thu May 26 04:50:15 2005 From: richardjones at optushome.com.au (Richard Jones) Date: Thu, 26 May 2005 12:50:15 +1000 Subject: [Catalog-sig] Distutils projects page Message-ID: <200505261250.16115.richardjones@optushome.com.au> In an effort to try to understand what projects are out there trying to improve distutils / packaging / make the world a better place, I've put up a wiki page: http://wiki.python.org/moin/DistutilsProjects It contains all the projects I'm aware of. I'm not on the distutils SIG list, so I might have missed some. I did scan the archive for this year, and didn't see anything. Please visit the page and edit it for correctness - or include your project if it's not in there! Richard -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/catalog-sig/attachments/20050526/cad222d5/attachment-0001.pgp From richardjones at optushome.com.au Thu May 26 08:20:43 2005 From: richardjones at optushome.com.au (Richard Jones) Date: Thu, 26 May 2005 16:20:43 +1000 Subject: [Catalog-sig] PyPI enhancement doc In-Reply-To: <4294ECD9.5010804@v.loewis.de> References: <42941CE4.7070206@colorstudy.com> <4294ECD9.5010804@v.loewis.de> Message-ID: <200505261620.43695.richardjones@optushome.com.au> On Thu, 26 May 2005 07:23 am, Martin v. L?wis wrote: > It appears to be a design principle of PyPI to not use XML-RPC if > it can be avoided. I don't know where that principle comes from, > but I think it needs to be followed if indeed it is a principle. > > If there is no such principle, I think all the current forms should > be duplicated to simplify the code in the distutils commands. There is no such design principle :) I just found it easier to reuse the existing HTTP/HTML form interface when writing the "register" command. Now that people are requesting more complex interfaces, I see no problem with using XML-RPC for them. > > There should be a contact form attached to each package that will be > > emailed to any people on record as owners or maintainers, for > > reporting bad or missing links or other incorrect data. > > Again, contributions are welcome. We should be careful not to implement > another issue tracker here, though, so just sending an email might be > enough for the moment I think that the email would be quite sufficient. Richard -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/catalog-sig/attachments/20050526/dbb0d366/attachment.pgp From richardjones at optushome.com.au Thu May 26 08:26:39 2005 From: richardjones at optushome.com.au (Richard Jones) Date: Thu, 26 May 2005 16:26:39 +1000 Subject: [Catalog-sig] PyPI enhancement doc In-Reply-To: <42941CE4.7070206@colorstudy.com> References: <42941CE4.7070206@colorstudy.com> Message-ID: <200505261626.40111.richardjones@optushome.com.au> On Wed, 25 May 2005 04:36 pm, Ian Bicking wrote: > The sdist type does not indicate what kind of archive is used -- most > notable, both zip and tar are possible. (Does sdist create zip files > on its own?) As it turns out, this is the case. I didn't realise it until someone got all hot under the collar about the unix-centric PyPI... no, I'm getting sidetracked here. Anyway, yes, on Windows sdists are ZIP files. You can force sdist to produce a particular format with the --format switch. > However, I think the client can determine the proper way > to unpack the file after downloading; this aspect of files is > self-describing. The correct meta-type should be supplied to the client too (by Apache, the file server). > An svn trunk is not a package file. However, it is largely equivalent > to an sdist file, though it must be downloaded in a different way; > actually performing the download is a client concern, so it doesn't > really effect this. Are you advocating that release_files somehow point to SVN repos? That doesn't sound right, so I'll presume I've misunderstood you :) > I'm a little soft on these, since I don't know if specifiers and the > necessary metadata is really ready: > > providing_packages(specifier): > A list of (name, version) from release_provides. E.g., > providing_packages('PageTemplate>=1.0') == [('zpt', '1.0')]. This > will only return non-hidden packages. > > requiring_packages(specifier): > A list of (name, version) from release_requires. There's no code to actually perform this function, but the meta-data capability is there. We probably won't see much in the way of this stuff until Python2.5 becomes widespread in use (since additions to the meta-data require releases of Python). > Search should be available with a single field, that searches all of > package name, summary, description/description_html, keywords. This > should be on the front page. A laudable goal. TSearch2 might be able to do this --- I don't know enough about it. I've recently added Xapian support to Roundup --- and it's really, really easy to use. Richard -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/catalog-sig/attachments/20050526/885e79cc/attachment.pgp From ianb at colorstudy.com Thu May 26 08:45:11 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 26 May 2005 01:45:11 -0500 Subject: [Catalog-sig] PyPI enhancement doc In-Reply-To: <200505261626.40111.richardjones@optushome.com.au> References: <42941CE4.7070206@colorstudy.com> <200505261626.40111.richardjones@optushome.com.au> Message-ID: <42957077.2020305@colorstudy.com> Richard Jones wrote: >>However, I think the client can determine the proper way >>to unpack the file after downloading; this aspect of files is >>self-describing. > > > The correct meta-type should be supplied to the client too (by Apache, the > file server). I'd trust magic numbers first. But it doesn't matter -- this is all a client issue, not a PyPI issue. In general I think if the client can do it, then the client should do it. >>An svn trunk is not a package file. However, it is largely equivalent >>to an sdist file, though it must be downloaded in a different way; >>actually performing the download is a client concern, so it doesn't >>really effect this. > > > Are you advocating that release_files somehow point to SVN repos? That doesn't > sound right, so I'll presume I've misunderstood you :) I'm advocating package_urls point to a SVN repository, so that you can install from the svn trunk in addition to formal releases. >>Search should be available with a single field, that searches all of >>package name, summary, description/description_html, keywords. This >>should be on the front page. > > > A laudable goal. TSearch2 might be able to do this --- I don't know enough > about it. I've recently added Xapian support to Roundup --- and it's really, > really easy to use. It's also quite possibly fast enough to simply use OR and LIKE; I don't know if the data is large enough that it's an issue. I mean, we're just now moving off CGI, right... I think full text indexing is a relatively fine point of optimization in comparison ;) -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From fdrake at gmail.com Thu May 26 19:04:15 2005 From: fdrake at gmail.com (Fred Drake) Date: Thu, 26 May 2005 13:04:15 -0400 Subject: [Catalog-sig] Distutils projects page In-Reply-To: <200505261250.16115.richardjones@optushome.com.au> References: <200505261250.16115.richardjones@optushome.com.au> Message-ID: <9cee7ab805052610046c0d20e0@mail.gmail.com> On 5/25/05, Richard Jones wrote: > In an effort to try to understand what projects are out there trying to > improve distutils / packaging / make the world a better place, I've put up a > wiki page: > > http://wiki.python.org/moin/DistutilsProjects I couldn't edit this page; it looks like we're still having wiki problems on python.org. I was going to add a link to the zpkg pages: http://www.zope.org/Members/fdrake/zpkgtools/ The pages there need to be updated; I'll see about getting that done in the next couple of days. (I'm on travel now.) -Fred -- Fred L. Drake, Jr. From pje at telecommunity.com Sun May 29 06:57:14 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 29 May 2005 00:57:14 -0400 Subject: [Catalog-sig] EasyInstall - automated download/install/manage Python packages Message-ID: <5.1.1.6.0.20050529005708.02c461f0@mail.telecommunity.com> Just a heads-up: http://peak.telecommunity.com/DevCenter/EasyInstall I would like to encourage package authors to try using EasyInstall to download and build the packages they distribute, and to let me know if they encounter any issues. I'd like to make EasyInstall able to build as many Python packages "out of the box" as possible. I've also been updating the Python Eggs documentation: http://peak.telecommunity.com/DevCenter/PythonEggs But there is still a fair amount to write about the runtime API, and as you can see from the "Implementation Status" there are plenty of tasks still "available" with respect to getting the zip-based runtime as robust as it should be. From kenneth.m.mcdonald at gmail.com Tue May 31 03:08:51 2005 From: kenneth.m.mcdonald at gmail.com (Kenneth McDonald) Date: Mon, 30 May 2005 20:08:51 -0500 Subject: [Catalog-sig] Questions about Qt and wxWindows python packages Message-ID: If this is not an appropriate newsgroup for this type of posting, please let me know and (if possible) suggest an alternative. The reading I've done of some of the back articles suggests that python-relevant questions specific to any python package (whether distrubuted via something like distutils or not) are acceptable here, sorry if this is incorrect. I'd like to start using Python for some GUI programming again. I'd like to use Tk, but it seems to be dying a slow death, so it's probably time to look at wxPython and PythonQt. I'd appreciate an comments on the relevant merits of these two libraries, both with respect to Python and more generally. I know about the differences in licensing, that's not an issue. My most specific concerns are ease of use (which includes how much in the way of documentation or examples for using through Python are available, and how easy the Python libs are to use in each case); availability of a generally capable styled text widget, as one of the things I need to build is an editor, which needs to be able to handle different fonts, styles, tabs, and so forth; and ease of installation on a Macintosh, since that is my preferred dev platform, and installing things made for other UNIX variants can be a little wonky at times. Many thanks for any feedback you can give. Cheers, Ken From pje at telecommunity.com Tue May 31 03:14:09 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 30 May 2005 21:14:09 -0400 Subject: [Catalog-sig] [Distutils] EasyInstall 0.3a3 released; what about PyPI? (was Re: Initial auto-installation support) In-Reply-To: <5.1.1.6.0.20050530173012.021f2340@mail.telecommunity.com> References: <429B8133.5070007@colorstudy.com> Message-ID: <5.1.1.6.0.20050530204156.021f27f8@mail.telecommunity.com> >At 04:10 PM 5/30/2005 -0500, Ian Bicking wrote: > > But besides > >that, this should work now for any packages with a distutils install, so > >long as those packages are reasonably well behaved. Hrm... except > >setuptools 0.3a2 doesn't have SourceForge download support, but 0.3a3 > >does and I think PJE will release that soon. 0.3a3 is now released, with a new --build-dir option, sandboxing, more package workarounds, SourceForge mirror selection, and "installation reports". See: http://peak.telecommunity.com/DevCenter/EasyInstall#release-notes-change-history for more details. I'm thinking that adding automatic package location via PyPI is probably pretty doable now, by the way. My plan is to create a PackageFinder class (subclassing AvailableDistributions) whose obtain() method searches for the desired package on PyPI, keeping a cache of URLs it has already seen. (It would also accept a callback argument that it would use to create Installer objects when it needs to install packages.) The command-line tool (easy_install.main) would create a PackageFinder with an interactive installation callback, and in the main loop it would pass it to each new Installer instance. The Installer would then use it whenever it gets a non-file, non-URL command line option, and use it to resolve() such requests. The PackageFinder.obtain() method would go to the PyPI base URL followed by the desired distribution name, e.g. 'http://www.python.org/pypi/SQLObject', and then scrape the page to see if it is a multi-version page, or a single-version page. If it's multi-version, it would scrape the version links and select the highest-numbered version that meets all of your criteria. Once it has a single-version page, it would look for a download URL, and see if its filename is that of an archive (.egg, .tar, .tgz, etc.) or if the URL is for subversion. If so, we assume it's the right thing and invoke the callback to do the install. If not, then we follow the link anyway, and scrape for links to archives, checking versions when we get there if possible. If there's still nothing suitable (or there was no download URL), we apply the same procedure to the homepage URL. This should suffice to make a significant number of packages available from PyPI with autodownload, and packages with dependencies would also be downloaded, built, and installed. The hardest parts of this aren't in the screen-scraping per se; it's more in the heuristics for evaluating whether a specific URL is suitable for download. Many PyPI download URLs are of the form "foopackage-latest.tgz", so it's not possible to determine a usable version number from this, unless I special-case "latest" in the version parser -- which I guess I could do. We also probably need some kind of heuristic to determine which URLs are "better" to try, as we don't want to just run through the links in order. Hm. You know, what if as an interim step we had the command-line tool just launch a webbrowser pointing you to PyPI? Getting to a page for a suitable version is easy, so we could then let the user find the right download URL and then go back to paste it on the command line. That could be a nice interim addition, although it isn't much of a solution for packages with a lot of un-installed dependencies. You'd keep getting kicked back to the web browser a lot, and more to the point you'd have to keep restarting the tool. So, ultimately we really need a way to actually find the URLs. There are going to have to be new options for the tool, too. Like a way to set the PyPI URL to use, and a way to specify what sort of package revisions are acceptable (e.g. no alphas, no betas, no snapshots). From ianb at colorstudy.com Tue May 31 03:29:11 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 30 May 2005 20:29:11 -0500 Subject: [Catalog-sig] Questions about Qt and wxWindows python packages In-Reply-To: References: Message-ID: <429BBDE7.4030108@colorstudy.com> The list is really meant for questions about packaging. You can ask the general python-list at python.org -- but first you should read the archive (http://groups-beta.google.com/group/comp.lang.python), because this exact question gets asked there often. Kenneth McDonald wrote: > If this is not an appropriate newsgroup for this type of posting, > please let me know and (if possible) suggest an alternative. The > reading I've done of some of the back articles suggests that > python-relevant questions specific to any python package (whether > distrubuted via something like distutils or not) are acceptable here, > sorry if this is incorrect. > > I'd like to start using Python for some GUI programming again. I'd > like to use Tk, but it seems to be dying a slow death, so it's > probably time to look at wxPython and PythonQt. > > I'd appreciate an comments on the relevant merits of these two > libraries, both with respect to Python and more generally. I know > about the differences in licensing, that's not an issue. My most > specific concerns are ease of use (which includes how much in the way > of documentation or examples for using through Python are available, > and how easy the Python libs are to use in each case); availability of > a generally capable styled text widget, as one of the things I need to > build is an editor, which needs to be able to handle different fonts, > styles, tabs, and so forth; and ease of installation on a Macintosh, > since that is my preferred dev platform, and installing things made > for other UNIX variants can be a little wonky at times. > > Many thanks for any feedback you can give. From ianb at colorstudy.com Tue May 31 03:47:28 2005 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 30 May 2005 20:47:28 -0500 Subject: [Catalog-sig] [Distutils] EasyInstall 0.3a3 released; what about PyPI? (was Re: Initial auto-installation support) In-Reply-To: <5.1.1.6.0.20050530204156.021f27f8@mail.telecommunity.com> References: <429B8133.5070007@colorstudy.com> <5.1.1.6.0.20050530204156.021f27f8@mail.telecommunity.com> Message-ID: <429BC230.5040709@colorstudy.com> Phillip J. Eby wrote: > The PackageFinder.obtain() method would go to the PyPI base URL followed > by the desired distribution name, e.g. > 'http://www.python.org/pypi/SQLObject', and then scrape the page to see > if it is a multi-version page, or a single-version page. If it's > multi-version, it would scrape the version links and select the > highest-numbered version that meets all of your criteria. Really this would be very easy to add to PyPI as a set of xml-rpc calls, once the server move gets resolved. I could develop it now, except that the last pypi database dump I have is from before the move to Postgres, and there've been a number of changes since then, and I'd like some test data. I also have a checkout from CVS, but the SF project no longer lists CVS and svn.python.org doesn't have public access yet, so I can't point you to the code. But anyway, if someone on the new or old server can just run pg_dump on that database and email me the results (or a url to those results) that would be very helpful. Getting the data without screenscraping won't instantly give us all the necessary information. But it does contain good information about available versions, what the active version is, and per-version download URLs (which, if nothing else, could be compared against each other to detect non-version-specific URLs). > Hm. You know, what if as an interim step we had the command-line tool > just launch a webbrowser pointing you to PyPI? Getting to a page for a > suitable version is easy, so we could then let the user find the right > download URL and then go back to paste it on the command line. That > could be a nice interim addition, although it isn't much of a solution > for packages with a lot of un-installed dependencies. You'd keep > getting kicked back to the web browser a lot, and more to the point > you'd have to keep restarting the tool. So, ultimately we really need a > way to actually find the URLs. That's not a very satisfying experience -- the person might as well just download the file at that point. Even with accurate data from PyPI, it's still likely there will be multiple possible URLs. At that point, at least if you are going through the command line, displaying all the URLs (numbered) and asking the user would probably give the user enough information to choose. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Tue May 31 06:32:05 2005 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 31 May 2005 00:32:05 -0400 Subject: [Catalog-sig] [Distutils] EasyInstall 0.3a3 released; what about PyPI? (was Re: Initial auto-installation support) In-Reply-To: <429BC230.5040709@colorstudy.com> References: <5.1.1.6.0.20050530204156.021f27f8@mail.telecommunity.com> <429B8133.5070007@colorstudy.com> <5.1.1.6.0.20050530204156.021f27f8@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20050531001410.021f3aa0@mail.telecommunity.com> At 08:47 PM 5/30/2005 -0500, Ian Bicking wrote: >Getting the data without screenscraping won't instantly give us all the >necessary information. But it does contain good information about >available versions, what the active version is, and per-version download >URLs (which, if nothing else, could be compared against each other to >detect non-version-specific URLs). Right; but none of that helps with the real problem (from EasyInstall's perspective), which is that the current incarnation of PyPI doesn't list multiple download URLs for a single release of a specific package. For example, when I release PEAK or PyProtocols, I've been releasing sdists (in two formats) plus a bdist_wininst -- and in the future I'll probably drop the bdist_wininst in favor of eggs. But I can't put any of that info on PyPI, so I just link to my downloads directory - as do 25% of the packages I surveyed in a random sampling last week. In order to get at packages like those, a flexible screen scraper is a must. I agree that PyPI should have better handling of download URLs, but I'm in a lot better position to improve EasyInstall than PyPI. >>Hm. You know, what if as an interim step we had the command-line tool >>just launch a webbrowser pointing you to PyPI? Getting to a page for a >>suitable version is easy, so we could then let the user find the right >>download URL and then go back to paste it on the command line. That >>could be a nice interim addition, although it isn't much of a solution >>for packages with a lot of un-installed dependencies. You'd keep getting >>kicked back to the web browser a lot, and more to the point you'd have to >>keep restarting the tool. So, ultimately we really need a way to >>actually find the URLs. > >That's not a very satisfying experience -- the person might as well just >download the file at that point. Tastes differ, I suppose. I'd just right-click the link to copy it, and then alt-tab, ^R, space, ^K, space, shift-insert, ENTER. But then, I've been downloading a lot of packages this weekend, so that sequence is already in my muscle memory. :) Hm. Maybe somebody could create a Firefox extension that runs EasyInstall on a selected link. :) >Even with accurate data from PyPI, it's still likely there will be >multiple possible URLs. At that point, at least if you are going through >the command line, displaying all the URLs (numbered) and asking the user >would probably give the user enough information to choose. In which case you might as well be back in the web browser. I'm fine with options being available to fine-tune the selection process, but the criteria can and should be mechanically processed. After all unusable versions, platforms, and archive types are eliminated, the prioritization should be in descending version order, with same version archives sorted by archive type, eggs first, everything else second. (Since the eggs don't need to be built.) By the way, in all of this there's been no discussion about MD5 signatures or code signing. That's probably because I don't know a whole lot about that subject. :) But I'm certainly interested in hearing from those who do.