From amk@amk.ca Sun Oct 13 21:40:29 2002 From: amk@amk.ca (A.M. Kuchling) Date: Sun, 13 Oct 2002 16:40:29 -0400 Subject: [Catalog-sig] Withdrawing PEP 262 Message-ID: <20021013164029.A26682@nyman.amk.ca> PEP 262 is a proposal for a database of installed Python packages. I wrote it as a step toward implementing automatic installation of Python software. After playing with Fink on MacOS X a bit, I've decided to withdraw the PEP, because I now think that Python shouldn't invent its own database; package handling and dependency tracking should be the operating system's job. Accordingly, I'm marking PEP 262 as 'Status: Deferred', and will request that it be removed from consideration for Python 2.3. If someone else wants to work on it, feel free to take over the PEP and resurrect it. --amk From cliechti@gmx.net Sun Oct 13 22:55:56 2002 From: cliechti@gmx.net (Chris Liechti) Date: Sun, 13 Oct 2002 23:55:56 +0200 Subject: [Catalog-sig] Withdrawing PEP 262 In-Reply-To: <20021013164029.A26682@nyman.amk.ca> Message-ID: Am 13.10.2002 22:40:29, schrieb "A.M. Kuchling" : >PEP 262 is a proposal for a database of installed Python packages. I >wrote it as a step toward implementing automatic installation of >Python software. After playing with Fink on MacOS X a bit, I've >decided to withdraw the PEP, because I now think that Python shouldn't >invent its own database; package handling and dependency tracking >should be the operating system's job. i use Debian Linux, which has a great package management system too and it's very nice when all the dependencies are installed along with a python extension. i understand you're exitement. but... not all systems have a package management system, e.g. on windows rules chaos. what can we do there? and there are some many different packaging systems... the problem of different package management systems can be partly soved by distutils. there are some packaging types already there (but i miss deb). the other problem is that a extension creator need to make all those formats and provide the downloads. and how about dependecies to onther packages. it's impossible for an author to cope with all of them for all different systems. (simplest to do will be to create a package with no dependecies, that way a bit confort goes away but its more motivating for the maintainer to create different package downloads, so that they're at least there.) i admit that i'm less interested in an installed files database (within python) rather than in a good way to download modules from a repository. if there was a browser that lets one select the extensions one wishes and then just press OK, that would be great. (maybe that could even be called within a running python application to request the user to install an addon. run in a separete process to solve the problem with different GUI systems) i think it would be a pythonic solution when that browser would integrate with the systems package manager. as a fallback it could still use "setup.py install" (loosing remove capability but that's not too important as systems without a package manager have a mess anyway ;-) that would mean that there must be a repository of modules (at python.org?) with downloads of zip, rpm, deb, etc access through http and ftp and easy upload (for a quick growth). and maybe some hierarchical package naming system should be developed for that. what do you think of such an approach? chris From martin@v.loewis.de Mon Oct 14 05:27:18 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 14 Oct 2002 06:27:18 +0200 Subject: [Catalog-sig] Withdrawing PEP 262 In-Reply-To: References: Message-ID: Chris Liechti writes: > i use Debian Linux, which has a great package management system too > and it's very nice when all the dependencies are installed along with > a python extension. i understand you're exitement. > > but... not all systems have a package management system, e.g. on > windows rules chaos. That may be true in general, but on Windows, it is not. There is a database of installed packages, complete with uninstallation etc. In fact, I'm not aware of any modern system that does not have a packaging infrastructure. > i admit that i'm less interested in an installed files database > (within python) rather than in a good way to download modules from a > repository. That's an entire different issue. The catalog system, which causes this SIG's existence, is still an open problem. > i think it would be a pythonic solution when that browser would integrate > with the systems package manager. I agree, and I think this is Andrew's rationale for withdrawing the PEP. Regards, Martin From DavidA@ActiveState.com Mon Oct 14 05:52:21 2002 From: DavidA@ActiveState.com (David Ascher) Date: Sun, 13 Oct 2002 21:52:21 -0700 Subject: [Catalog-sig] Withdrawing PEP 262 References: Message-ID: <3DAA4D85.7010000@ActiveState.com> Martin v. Loewis wrote: >Chris Liechti writes: > > > >>i use Debian Linux, which has a great package management system too >>and it's very nice when all the dependencies are installed along with >>a python extension. i understand you're exitement. >> >>but... not all systems have a package management system, e.g. on >>windows rules chaos. >> >> > >That may be true in general, but on Windows, it is not. There is a >database of installed packages, complete with uninstallation etc. > In fact what I think is wrong with both Windows and many other systems' is that they tend to be 'one-level' systems -- it's hard to define a "subsystem" installation. It'd be nice if every python module installed didn't show up in the ever-growing list of programs under Add/Remove Programs. Or are there subtleties to the Windows Installer that I haven't heard of? --david From thomas.heller@ion-tof.com Mon Oct 14 10:43:09 2002 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Mon, 14 Oct 2002 11:43:09 +0200 Subject: [Catalog-sig] Withdrawing PEP 262 References: <3DAA4D85.7010000@ActiveState.com> Message-ID: <01d401c27366$1adfacc0$e000a8c0@thomasnotebook> From: "David Ascher" > Martin v. Loewis wrote: > > >Chris Liechti writes: > > > > > > > >>i use Debian Linux, which has a great package management system too > >>and it's very nice when all the dependencies are installed along with > >>a python extension. i understand you're exitement. > >> > >>but... not all systems have a package management system, e.g. on > >>windows rules chaos. > >> > >> > > > >That may be true in general, but on Windows, it is not. There is a > >database of installed packages, complete with uninstallation etc. Well, it's not really a database, it is just a listing of files and registry entries which must be removed at uninstallation time. > > > In fact what I think is wrong with both Windows and many other systems' > is that they tend to be 'one-level' systems -- it's hard to define a > "subsystem" installation. It'd be nice if every python module installed > didn't show up in the ever-growing list of programs under Add/Remove > Programs. Or are there subtleties to the Windows Installer that I > haven't heard of? No, currently not. It would be possible to change this, but is it really worth the effort? I've heard some complaints that the python root directory is populated with these xxx-wininst.log and Removexxx.exe files, but why bother? Thomas From AASterling@Hotpop.com Mon Oct 14 20:36:42 2002 From: AASterling@Hotpop.com (Aaron Sterling) Date: Mon, 14 Oct 2002 15:36:42 -0400 Subject: [Catalog-sig] Withdrawing PEP 262 In-Reply-To: Message-ID: 10/13/2002 5:55:56 PM, Chris Liechti wrote: >i admit that i'm less interested in an installed files database (within python) >rather than in a good way to download modules from a repository. That is my view as well, I also agree with you implication that for the second to be done well the first must be in place. >if there was a browser that lets one select the extensions one wishes and >then just press OK, that would be great. (maybe that could even be called >within a running python application to request the user to install an addon. >run in a separete process to solve the problem with different GUI systems) > >i think it would be a pythonic solution when that browser would integrate >with the systems package manager. as a fallback it could still use >"setup.py install" (loosing remove capability but that's not too important as >systems without a package manager have a mess anyway ;-) > >that would mean that there must be a repository of modules (at python.org?) Perhaps a good way to do it would be to have a table that can be easily downloaded at python.org. The table would consist of package name in one coloumn and the direct url/s (suitable for automatic download) of the resource as the other coloumn. If the desired module is not available from any of the given urls, their could be a repository of the more frequently used/standard library modules at the site to be downloaded as a last resort. The table could be included with the standard library and accessed on the website if the local copy were six months (I pulled it out of the air, it could be any age) old or older. It would also be accessed when the local copy did not contain the required module. It could then be stored as the new local copy until a new copy was required according to the specified material. This scheme would have the advantage of saving bandwidth on the python website. An alternate way to do it(and a much cooler way of doing it in my book), would be to select a port on the python.org site and attach a server to it for a very simple protocol that could be developed and implemented in about half an hour. The server would simply recieve the name of the desired package and dispatch the url from which it could be downloaded. This would have the advantage of allowing the server to keep track of requests for packages, thus allowing usage data to be collected(or not). This could be used to decide which packages to add to the standard library. Adding the most frequently used packages to the standard library would have the effect of cutting bandwidth use on the server. Like the first solution, the table would also be maintained locally and the main table at the website would only be accessed under conditions similiar to those in the first solution. It also goes without saying that the code to do these things would be a package so that Large organizations would be able to run their own package servers. ImportError could even be changed in such a manner that(at the administrator/users discretion for obvious security reasons) if a required package was not on the system, it could be downloaded and installed automatically. That would be cool(Is their a programming system that provides that functionality now?). Security concerns could be overcome to some degree by having automatic installation download exclusively from the python.org website. for either solution new submissions would be added by mailing the name of the package and the url to a specified mailbox at the website. A simple script could add them on a daily basis. I think that the first plan is definitely doable, I favor the second(I know my voice carries no weight). It goes without saying that somebody will think that the second solution is overkill, but is that the general concensus? If --any-- intererest whatsoever is expressed in either of these plans, I would be more then willing to further flesh them out or work with anyone on further fleshing them out, in fact I am fleshing the second one out at the moment. Aaron From cliechti@gmx.net Mon Oct 14 22:21:00 2002 From: cliechti@gmx.net (Chris Liechti) Date: Mon, 14 Oct 2002 23:21:00 +0200 Subject: [Catalog-sig] Withdrawing PEP 262 In-Reply-To: Message-ID: Am 14.10.2002 21:36:42, schrieb Aaron Sterling : >10/13/2002 5:55:56 PM, Chris Liechti wrote: >>i admit that i'm less interested in an installed files database (within python) >>rather than in a good way to download modules from a repository. > >That is my view as well, I also agree with you implication that for the second to be done >well the first must be in place. > >>if there was a browser that lets one select the extensions one wishes and >>then just press OK, that would be great. (maybe that could even be called >>within a running python application to request the user to install an addon. >>run in a separete process to solve the problem with different GUI systems) >> >>i think it would be a pythonic solution when that browser would integrate >>with the systems package manager. as a fallback it could still use >>"setup.py install" (loosing remove capability but that's not too important as >>systems without a package manager have a mess anyway ;-) >> >>that would mean that there must be a repository of modules (at python.org?) > >Perhaps a good way to do it would be to have a table that can be easily downloaded at >python.org. The table would consist of package name in one coloumn and the direct url/s >(suitable for automatic download) of the resource as the other coloumn. If the desired module >is not available from any of the given urls, their could be a repository of the more >frequently used/standard library modules at the site to be downloaded as a last resort. >The table could be included with the standard library and accessed on the website if the >local copy were six months (I pulled it out of the air, it could be any age) old or older. It >would also be accessed when the local copy did not contain the required module. It could then >be stored as the new local copy until a new copy was required according to the specified >material. This scheme would have the advantage of saving bandwidth on the python website. > >An alternate way to do it(and a much cooler way of doing it in my book), would be to select a >port on the python.org site and attach a server to it for a very simple protocol that could >be developed and implemented in about half an hour. The server would simply recieve the name >of the desired package and dispatch the url from which it could be downloaded. no no...reason: firewalls. there must be a way to access the list through a http proxy when no other protocols are passed trough a firewall (i have this situation where i work). because of that i'd say the priorites should be the following 1. http, 2. ftp, 3. custom protocol,if any. an another good reason: standards like a web server are easier to use. it's easy to convince the IT departement that they should open a directory for the module repository while it is likely to be very difficult to convince them to run a special server app. look how Debian is working. there is a Packages.tgz on the server with a list of the available stuff. and it can be accessed throught htpp and ftp, realy easy and just working. > This would >have the advantage of allowing the server to keep track of requests for packages, thus >allowing usage data to be collected(or not). This could be used to decide which packages to >add to the standard library. Adding the most frequently used packages to the standard library >would have the effect of cutting bandwidth use on the server. Like the first solution, the >table would also be maintained locally and the main table at the website would only be >accessed under conditions similiar to those in the first solution. It also goes without >saying that the code to do these things would be a package so that Large organizations would >be able to run their own package servers. ImportError could even be changed in such a manner >that(at the administrator/users discretion for obvious security reasons) if a required >package was not on the system, it could be downloaded and installed automatically. That would >be cool(Is their a programming system that provides that functionality now?). Security >concerns could be overcome to some degree by having automatic installation download >exclusively from the python.org website. sure that'd be cool, but that should be only allowed with signed packages and otherwise only with a BIG warning box... >for either solution new submissions would be added by mailing the name of the package and the >url to a specified mailbox at the website. A simple script could add them on a daily basis. i think there should be two categories. one trusted, where the submissions must be signed and go through some process where different people can look at it and a second easy to upload repository, so that the number of packagaes can quickly grow. the quality of the submissions will be lower in the second one but there might be some pearls... >I think that the first plan is definitely doable, I favor the second(I know my voice carries >no weight). It goes without saying that somebody will think that the second solution is >overkill, but is that the general concensus? > >If --any-- intererest whatsoever is expressed in either of these plans, I would be more then >willing to further flesh them out or work with anyone on further fleshing them out, in fact I >am fleshing the second one out at the moment. From AASterling@Hotpop.com Tue Oct 15 05:45:14 2002 From: AASterling@Hotpop.com (Aaron Sterling) Date: Tue, 15 Oct 2002 00:45:14 -0400 Subject: [Catalog-sig] Withdrawing PEP 262 In-Reply-To: Message-ID: 10/14/2002 5:21:00 PM, Chris Liechti wrote: >no no...reason: firewalls. >sure that'd be cool, but that should be only allowed with signed packages and otherwise only >with a BIG warning box... Oh...Yeah... Man the real world is a drag. Especially the security aspect of it. That being said, let me list some rough specifications for such a system(the sane one) and see if anyone agrees with any of them. 1) As Chris pointed out, the catalog should be divided into two levels, signed and unsigned. 2) The catalog(I propose calling it the Python Package Catalog, or PPC, at least for the purpose of this discussion) should be structured thusly. Three rfc822 fields: DATE, PPC-Signed, PPC-Unsigned. This divides the catalog into two subcatalogs. Each entry in a sub-catalog occupies one and only one line. This line takes the form . The package name is the name of the package, the package infofile URL is the URL of the package infofile(described in 4), the package website is the packages' website. The package summary is an eighty charachter summary of the package. This is for use in the browser. 3) Access to the primary PPC at python.org should be kept to a minimum to conserve bandwidth. This can be facilitated by keeping a local copy of the PPC on each machine. This carries with the extra benefit of redundency. When it is time to download a package, the Local PPC(LPPC) is consulted. If the DATE field in the LPPC is expired a fresh copy is downloaded from python.org. 4) This is a hack I think. It would be preferable to simply have one universal format for all packages on all platforms, that not being the case, we need some means of redirecting general requests for a package into a concrete URL for that specific platform. This can be accomplished with infofiles. Infofiles would be plaintext, either ASCII or LATIN-1. They would essentially be a catalog of the different distributions of the package for different platforms. Each platform for which that package is supported would have one and only one line given to it. This line would take the form . The platform is simply sys.platform so that the code is guaranteed to know what it is. The plaform specific URL is the url of the package for that specific platform. 5) In addition to the LPPC, there should be the IPPC (Installed PPC), this is almost the same format as the PPC, but contains only the subset of the PPC that is installed on the local system. The change in format is the substitution of a reference to the packages' local package database entry for the infofile reference. 6) The stucture of the local package database would be essentially the same as under PEP 262 There is obviously a lot of stuff that needs to be fleshed out, such as where do these files live and the UI for the Package Browser, to give just two examples. As a whole though, how does it hold up? Aaron From lucio@movilogic.com Tue Oct 15 14:24:13 2002 From: lucio@movilogic.com (Lucio Torre) Date: Tue, 15 Oct 2002 10:24:13 -0300 Subject: [Catalog-sig] Withdrawing PEP 262 References: Message-ID: <3DAC16FD.8030506@movilogic.com> Aaron, i a bit lost in time, i hope this is still relevant. comments below. Aaron Sterling wrote: >10/14/2002 5:21:00 PM, Chris Liechti wrote: > > > >>no no...reason: firewalls. >> >> > > > >>sure that'd be cool, but that should be only allowed with signed packages and >> >> >otherwise only > > >>with a BIG warning box... >> >> > >Oh...Yeah... Man the real world is a drag. Especially the security aspect of it. > >That being said, let me list some rough specifications for such a system(the sane one) >and see if anyone agrees with any of them. > > I can testify for the imposrtance of this issue!.. On the other hand, simplicity should also be considered a plus. > >1) As Chris pointed out, the catalog should be divided into two levels, signed and >unsigned. > >2) The catalog(I propose calling it the Python Package Catalog, or PPC, at least for >the purpose of this discussion) should be structured thusly. Three rfc822 fields: DATE, >PPC-Signed, PPC-Unsigned. This divides the catalog into two subcatalogs. Each entry in >a sub-catalog occupies one and only one line. This line takes the form > . The >package name is the name of the package, the package infofile URL is the URL of the >package infofile(described in 4), the package website is the packages' website. The >package summary is an eighty charachter summary of the package. This is for use in the >browser. > > Im sure i lost something, but where are dependencies? What about when you want to install this you should have that installed first? Thats a great feature. Also, being a debian fan, i recommend allowing each instalation to configure it sources. Python.org could provide a basic one, parnassus another. something like that. i dont find cerificates too important. i dont really know the authors of the code i download, and i dont think python.org should tell anyone about the security/autenticity of my code. >3) Access to the primary PPC at python.org should be kept to a minimum to conserve >bandwidth. This can be facilitated by keeping a local copy of the PPC on each machine. >This carries with the extra benefit of redundency. When it is time to download a >package, the Local PPC(LPPC) is consulted. If the DATE field in the LPPC is expired a >fresh copy is downloaded from python.org. > I like the 'apt-get update' way better. Why force update? New versions and incompatibilities always arise that should be considered. > >4) This is a hack I think. It would be preferable to simply have one universal format >for all packages on all platforms, that not being the case, we need some means of >redirecting general requests for a package into a concrete URL for that specific >platform. This can be accomplished with infofiles. Infofiles would be plaintext, either >ASCII or LATIN-1. They would essentially be a catalog of the different distributions of >the package for different platforms. Each platform for which that package is supported >would have one and only one line given to it. This line would take the form > . The platform is simply sys.platform so that the code >is guaranteed to know what it is. The plaform specific URL is the url of the package >for that specific platform. > > Here we also have issues with python versions. What if im still using python 2.7 and python 3000 is out? I should be able to select the packages that are compatible with python 2.7 >5) In addition to the LPPC, there should be the IPPC (Installed PPC), this is almost >the same format as the PPC, but contains only the subset of the PPC that is installed >on the local system. The change in format is the substitution of a reference to the >packages' local package database entry for the infofile reference. > >6) The stucture of the local package database would be essentially the same as under >PEP 262 > >There is obviously a lot of stuff that needs to be fleshed out, such as where do these >files live and the UI for the Package Browser, to give just two examples. As a whole >though, how does it hold up? > > The way i see it is like this: (sorry for the extra noise in this touchy issue) $INSTALLDB/Packages - A Database of Installed Python Packages (PEP 262) $INSTALLDB/Config/Sources.py - A sources list. Something like this: sources = { "python.org":{ "url":"http://python.org/", "desc":"Default source", } "parnasuss": { etc... } } As you can see, im implying the use of python source as the format of the configuration files. I think its better because: a- its more felixible b- we already have a parser c- everyone knows how to read it d- security issues with "pakcage info files" can be solved easily $INSTALLDB/Cache/$SOURCE.package.list List of packages. This file should be very similar to the sources.py file, and include the requiered information of a packe. I $INSTALLDB/Keys/* Public keys of people you trust. And usage should go like this: dsitutils update check every source for updates. perform a head on every url, download new versions. distutils install package [-s SOURCE] look for the package in cache. if its found in more than one source, complain and stop. require usage of 'source' check prerequisites. in they are not met, say so and stop. download. (http) check for signatures. install. etc. I think that we should keep these goals in mind: 1- No automation/intelligence until we are sure it can be done right. I dont want automatic dependency matching and endeless loops like dselect. 2- it should not force us to keep an up to date installation of python, but help us in doing so. 3- it should be easily wrapped. So i can use debian to keep my package database or make a sample inno setup tool for the same thing. just my 2c lucio > > From thomas.heller@ion-tof.com Wed Oct 16 18:43:40 2002 From: thomas.heller@ion-tof.com (Thomas Heller) Date: 16 Oct 2002 19:43:40 +0200 Subject: [Catalog-sig] Withdrawing PEP 262 In-Reply-To: <3DAA4D85.7010000@ActiveState.com> References: <3DAA4D85.7010000@ActiveState.com> Message-ID: David Ascher writes: > Martin v. Loewis wrote: > > >Chris Liechti writes: > > > > > > >>i use Debian Linux, which has a great package management system too > >>and it's very nice when all the dependencies are installed along with > >>a python extension. i understand you're exitement. > >> > >>but... not all systems have a package management system, e.g. on > >>windows rules chaos. > >> > > > > >That may be true in general, but on Windows, it is not. There is a > >database of installed packages, complete with uninstallation etc. > > I was wondering if it would make sense (now that PEP262 is no longer), to convert these *-wininst.log files somewhat more into package database components. Currently these files contain a list of files created, directories created, registry keys and values added to the system by the installation program. All this is used to cleanly remove the package from the system if it is no longer needed. Other info in these files is: - the windows installer exe pathname used to install the package - the name (including the version) of the package installed, for example Numeric-20.3 - the command line for removing this package (used by the add/remove programs Windows applet. This information could be used also by a python program to verify that all files are still present, to display the name and version, to remove the package, and maybe more. To extend the possibilities, would it make sense to include more information into these files, starting with the metadata info for the package at least, see PEP 242. > In fact what I think is wrong with both Windows and many other > systems' is that they tend to be 'one-level' systems -- it's hard to > define a "subsystem" installation. It'd be nice if every python > module installed didn't show up in the ever-growing list of programs > under Add/Remove Programs. Or are there subtleties to the Windows > Installer that I haven't heard of? > Given the above, it would be easy to write a slick command line or gui python app with a nice user interface for displaying, removing, verifying, and maybe sometime downloading and installing packages. Thomas From rjones@ekit-inc.com Tue Oct 22 08:32:45 2002 From: rjones@ekit-inc.com (Richard Jones) Date: Tue, 22 Oct 2002 17:32:45 +1000 Subject: [Catalog-sig] Distutils package index online Message-ID: <200210221732.45600.rjones@ekit-inc.com> So I bit the bullet on the train this morning (what an odd phrase.... ;) I have written a simple CGI and distutils extension that handles registration of distutils package metadata in a central index. The intention is that this system is run on python.org, mostly to lend it an air of officiality ;) It's simple, it's bare-bones, it works: http://mechanicalcat.net:8081/ I just figured we needed _something_ to get out there to start the ball rolling. And I'm serious about the python.org part... Richard From thomas.heller@ion-tof.com Tue Oct 22 13:47:34 2002 From: thomas.heller@ion-tof.com (Thomas Heller) Date: 22 Oct 2002 14:47:34 +0200 Subject: [Catalog-sig] RFC: pypan - a Python package manager Message-ID: I've hacked together a simple Python Package manager. It is available here: Suggestions for a better name are welcome... It is only tested with Python 2.2, but it should also run under 2.0 and 2.1. Currently works only under Windows, but I hope someone will join me and extend it to linux. For information, please look at the web page. --------------------- Here are the general concepts (which are mostly derived from ciphon): pypan can list the packages installed on the system (those installed from bdist_wininst packages). It does not maintain a database of installed packages, the list is created at runtime by using the information from the install log files (I assume Linux package formats like .rpm or .deb could provide the same information). See the 'list' command. The package repository is a simple directory tree on the server, although pypan can also use a local directory tree. At the top of the directory structure is a 'packages.xml' file which lists the contained files together with their PKG-INFO. pypan can handle source distributions built by distutils (.zip, .tar.gz), as well as binary windows distributions built with bdist_wininst. The 'packages.xml' file is built by a 'build_packagelist.py' script which is included. If a source distribution is installed, a windows installer is built on the fly and run. Unfortunately, these windows installers can not yet run in 'silent' mode, but I plan to add this later. For your convenience I've uploaded (additionally to pypan itself) several PyChecker distributions to the repository. This is alpha software - please take care when using it. Of course I am *much* interested in any feedback, bug reports, flames... Regards, Thomas From rjones@ekit-inc.com Wed Oct 23 02:09:55 2002 From: rjones@ekit-inc.com (Richard Jones) Date: Wed, 23 Oct 2002 11:09:55 +1000 Subject: [Catalog-sig] Trove information Message-ID: <200210231109.55429.rjones@ekit-inc.com> So why didn't Trove information make it into the distutils metadata? It came up numerous times on the mailing list, but didn't actually make it into the metadata set... Richard From martin@v.loewis.de Wed Oct 23 07:13:18 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 23 Oct 2002 08:13:18 +0200 Subject: [Catalog-sig] Trove information In-Reply-To: <200210231109.55429.rjones@ekit-inc.com> References: <200210231109.55429.rjones@ekit-inc.com> Message-ID: Richard Jones writes: > So why didn't Trove information make it into the distutils metadata? It came > up numerous times on the mailing list, but didn't actually make it into the > metadata set... Either because there was no agreement, or because nobody ever contributed patches (probably both). Regards, Martin From rjones@ekit-inc.com Wed Oct 23 07:24:06 2002 From: rjones@ekit-inc.com (Richard Jones) Date: Wed, 23 Oct 2002 16:24:06 +1000 Subject: [Catalog-sig] Online catalog now includes basic searching Message-ID: <200210231624.06412.rjones@ekit-inc.com> Again, proof-of-concept rather than fully-featured, but the online distutils catalog now has searching. That address again :) http://mechanicalcat.net:8081/ Richard From rjones@ekit-inc.com Wed Oct 23 07:24:44 2002 From: rjones@ekit-inc.com (Richard Jones) Date: Wed, 23 Oct 2002 16:24:44 +1000 Subject: [Catalog-sig] Trove information In-Reply-To: References: <200210231109.55429.rjones@ekit-inc.com> Message-ID: <200210231624.44848.rjones@ekit-inc.com> On Wed, 23 Oct 2002 4:13 pm, Martin v. Loewis wrote: > Richard Jones writes: > > So why didn't Trove information make it into the distutils metadata? It > > came up numerous times on the mailing list, but didn't actually make it > > into the metadata set... > > Either because there was no agreement, or because nobody ever > contributed patches (probably both). I've asked the same question on the distutils mailing list (where I should have asked in the first place) and will see. Richard From s-thapa-11@alumni.uchicago.edu Fri Oct 25 20:21:22 2002 From: s-thapa-11@alumni.uchicago.edu (Suchandra Thapa) Date: 25 Oct 2002 14:21:22 -0500 Subject: [Catalog-sig] RFC: pypan - a Python package manager In-Reply-To: <7kg64dfk.fsf@ion-tof.com> References: <1035306997.23711.63.camel@hepcat> <1035309484.23711.84.camel@hepcat> <1035565579.1571.61.camel@hepcat> <7kg64dfk.fsf@ion-tof.com> Message-ID: <1035573683.1575.92.camel@hepcat> --=-nDsWAenhgWjtl1pvJF/N Content-Type: multipart/alternative; boundary="=-4xzlh8jQ4jc2Y2Sk5TQt" --=-4xzlh8jQ4jc2Y2Sk5TQt Content-Type: text/plain Content-Transfer-Encoding: quoted-printable On Fri, 2002-10-25 at 12:48, Thomas Heller wrote: Shouldn't we take this thread to the catalog-sig, or at least parts of it? The discussion pypan vs. ciphon? I have the impression that some discussion also lets other people jump in... They see that something is happing. =20 Sure, I've cc'ed the message to the catalog-sig list. And also I'm still hacking on pypan, and would be interested in comments ;-) That's the rub. There have been several attempts to get this functionality for python, but my impression is that most of died because the authors didn't get much feedback and therefore gradually lost interest in continuing. > I'm still not sure what the best approach is however. I don't really > want to have functionality that is available only in some cases but > trying to offer a remove or uninstall command on all platforms would > require changes to distutils to track what files are being installed = (I > think). =20 =20 IMO we should try to avoid requiring changes to distutils, and this seems also what the withdrawn PEP 262 was about. =20 I agree. However, I'm not sure how else to make an uninstall command available without modifying the distutils source. Windows and rpm based systems are fine since we can use the uninstaller or the rpm command to remove modules. Providing a reliable method on other systems seems to be a more difficult problem. I originally considered just removing the appropriate directory in /usr/lib/pythonx.x/site-packages but that doesn't catch everything and may not even work. > Yeah, I've had more time recently so I've been working on ciphon here > and there. I usually get more time to work on ciphon in the weekends= so > I try to get things done then. The next items on my todo list are > allowing users to run ciphon from the command line (e.g. ciphon insta= ll > mymodule) and to allow http servers to be used as well. If you have = any > suggestions on other things that I should work on I'm open to > suggestions. The two changes I mentioned above probably won't take t= o > long to do. =20 Command line is a good idea: already present in pypan. Yeah, I didn't really think about it initially but after looking at the pypan sources and how it worked, it made a lot of sense to have that available. In pypan I used urllib - it handles FTP servers, HTTP servers, and local files "file:..." as well. Seems pretty universal to me. =20 I was thinking about going along the same lines. I'll need to check how well the urllib handles ftp downloads from Bernstein's anonftpd though. Another suggestion: sending bug reports via email doesn't work per default on windows - it's very uncommon to have an smtp server running on the machine you work on. If this feature is to stay, there should probably be a config option 'smtpserver'. I forgot to ask about how well that worked on Windows. I'll change the configuration files and the ciphon config command to allow the smtp server to be set. I think I'll keep it around since it seems like it would be useful for developers to be able to get an automated bug report with a stack trace emailed to the bugs mailing list. I'm planning on extending it a little further so that users can send a bug report manually and to give better reporting of errors. --=20 ------------------------------------------------------------------ Suchandra S. Thapa=20 s-thapa-11@alumni.uchicago.edu ------------------------------------------------------------------ --=-4xzlh8jQ4jc2Y2Sk5TQt Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable On Fri, 2002-10-25 at 12:48, Thomas Heller wrote: 
Shouldn't we take this thr=
ead to the catalog-sig, or at least
parts of it? The discussion pyp=
an vs. ciphon?
I have the impression that some=
 discussion also lets other people
jump in... They see that someth=
ing is happing.
Sure, I've cc'ed the message to the catalog-sig list. 
And also I'm still hacking=
 on pypan, and would be interested in
comments ;-)<=
/PRE>
That's the rub.  There have been several attempts to = get this functionality for python, but my impression is that most of died b= ecause the authors didn't get much feedback and therefore gradually lost in= terest in continuing. 
> I'm still not sure wh=
at the best approach is however.  I don't really
> want to have functionality=
 that is available only in some cases but
> trying to offer a remove o=
r uninstall command on all platforms would
> require changes to distuti=
ls to track what files are being installed (I
> think).  

IMO we should try to avoid requ=
iring changes to distutils, and this
seems also what the withdrawn P=
EP 262 was about.
I agree.  However, I'm not sure how else to make an u= ninstall command available without modifying the distutils source.  Wi= ndows and rpm based systems are fine since we can use the uninstaller or th= e rpm command to remove modules.  Providing a reliable method on other= systems seems to be a more difficult problem.    I original= ly considered just removing the appropriate directory in /usr/lib/pythonx.x= /site-packages but that doesn't  catch everything and may not even wor= k. 
> Yeah, I've had more t=
ime recently so I've been working on ciphon here
> and there.  I usually get =
more time to work on ciphon in the weekends so
> I try to get things done t=
hen.  The next items on my todo list are
> allowing users to run ciph=
on from the command line (e.g. ciphon install
> mymodule) and to allow htt=
p servers to be used as well.  If you have any
> suggestions on other thing=
s that I should work on I'm open to
> suggestions.  The two chan=
ges I mentioned above probably won't take to
> long to do.<=
/I>

Command line is a good idea: al=
ready present in pypan.
Yeah, I didn't really think about it initially but after l= ooking at the pypan sources and how it worked, it made a lot of sense to ha= ve that available. 
In pypan I used urllib - i=
t handles FTP servers, HTTP servers, and
local files "file:..."=
; as well. Seems pretty universal to me.
I was thinking about going along the same lines.  I'l= l need to check how well the urllib handles ftp downloads from Bernstein's = anonftpd though. 
Another suggestion: sendin=
g bug reports via email doesn't work per
default on windows - it's very =
uncommon to have an smtp server running
on the machine you work on. If =
this feature is to stay, there should
probably be a config option 'sm=
tpserver'.
I forgot to ask about how well that worked on Windows. I'l= l change the configuration files and the ciphon config command to allow the= smtp server to be set.  I think I'll keep it around since it seems li= ke it would be useful for developers to be able to get an automated bug rep= ort with a stack trace emailed to the bugs mailing list.  I'm planning= on extending it a little further so that users can send a bug report manua= lly and to give better reporting of errors. 
--=20
------------------------------------------------------------------

Suchandra S. Thapa=20
s-thapa-11@alumni.uchicago.edu

------------------------------------------------------------------
--=-4xzlh8jQ4jc2Y2Sk5TQt-- --=-nDsWAenhgWjtl1pvJF/N Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: For info see http://www.gnupg.org iEYEABECAAYFAj25mbIACgkQ6nShCjt5AZLuDQCfbJWe3tPDIejcmxiM07CiTzY3 MnUAn3+sF5aX2ZwBqAsR/Aqjsu+xkZbR =nkSz -----END PGP SIGNATURE----- --=-nDsWAenhgWjtl1pvJF/N-- From thomas.heller@ion-tof.com Fri Oct 25 20:55:57 2002 From: thomas.heller@ion-tof.com (Thomas Heller) Date: 25 Oct 2002 21:55:57 +0200 Subject: [Catalog-sig] RFC: pypan - a Python package manager In-Reply-To: <1035573683.1575.92.camel@hepcat> References: <1035306997.23711.63.camel@hepcat> <1035309484.23711.84.camel@hepcat> <1035565579.1571.61.camel@hepcat> <7kg64dfk.fsf@ion-tof.com> <1035573683.1575.92.camel@hepcat> Message-ID: Suchandra Thapa writes: > On Fri, 2002-10-25 at 12:48, Thomas Heller wrote: > > > Shouldn't we take this thread to the catalog-sig, or at least > parts of it? The discussion pypan vs. ciphon? > I have the impression that some discussion also lets other people > jump in... They see that something is happing. > > > Sure, I've cc'ed the message to the catalog-sig list. > > And also I'm still hacking on pypan, and would be interested in > comments ;-) > > That's the rub. There have been several attempts to get this > functionality for python, but my impression is that most of died because > the authors didn't get much feedback and therefore gradually lost > interest in continuing. The chicken and egg issue. It's difficult to reach the critical mass where the system is actually used. Question for other readers of this list (David, maybe): How's pyppm going? I don't hear much from it. > > > I'm still not sure what the best approach is however. I don't > > really want to have functionality that is available only in > > some cases but trying to offer a remove or uninstall command > > on all platforms would require changes to distutils to track > > what files are being installed (I think). > > IMO we should try to avoid requiring changes to distutils, and > this seems also what the withdrawn PEP 262 was about. > > > I agree. However, I'm not sure how else to make an uninstall > command available without modifying the distutils source. Windows > and rpm based systems are fine since we can use the uninstaller or > the rpm command to remove modules. Providing a reliable method on > other systems seems to be a more difficult problem. I originally > considered just removing the appropriate directory in > /usr/lib/pythonx.x/site-packages but that doesn't catch everything > and may not even work. Right - this cannot work. IMO we should concentrate first on those systems where a 'native package manager' is used. At least this is the approach I will continue in pypan (Hm, really a strange name. Think I'll change it to 'pypit' or PythonPit, PythonTrove. Difficult for a non-native english speaker ;-). Since I only use Windows - except sometimes, when managing my starship area - this is my main interest. But a 'remove' feature is really needed for a package manager, I think. > Another suggestion: sending bug reports via email doesn't work > per default on windows - it's very uncommon to have an smtp > server running on the machine you work on. If this feature is to > stay, there should probably be a config option 'smtpserver'. > > I forgot to ask about how well that worked on Windows. It works on windows if you change 'localhost' to a real smtp server. > I'll change the configuration files and the ciphon config command to > allow the smtp server to be set. I think I'll keep it around since > it seems like it would be useful for developers to be able to get an > automated bug report with a stack trace emailed to the bugs mailing > list. I'm planning on extending it a little further so that users > can send a bug report manually and to give better reporting of > errors. This is only useful if ciphon is in active use by real users. IMO for the developers it would be better to see the traceback directly. Maybe a CIPHON_DEBUG environment var or something like that? My usual work style is to catch exceptions in a sys.excepthook installed in my sitecustomize file - this recipe is in the Python Cookbook. This excepthook starts the debugger (pdb), and lets me inspect the state of the program immediately. By the way, there are a lot of unqualified try/except clauses in ciphon.py, which make debugging a pain IMO, and also sometimes mask programming errors. Thomas From DavidA@ActiveState.com Fri Oct 25 21:45:28 2002 From: DavidA@ActiveState.com (David Ascher) Date: Fri, 25 Oct 2002 13:45:28 -0700 Subject: [Catalog-sig] RFC: pypan - a Python package manager References: <1035306997.23711.63.camel@hepcat> <1035309484.23711.84.camel@hepcat> <1035565579.1571.61.camel@hepcat> <7kg64dfk.fsf@ion-tof.com> <1035573683.1575.92.camel@hepcat> Message-ID: <3DB9AD68.5010009@ActiveState.com> Thomas Heller wrote: > The chicken and egg issue. It's difficult to reach the critical mass > where the system is actually used. > > Question for other readers of this list (David, maybe): How's pyppm > going? I don't hear much from it. PyPPM is pretty stagnant. There are a few problems w/ PyPPM from where I'm sitting. 1) Unlike in the Perl world, we have to manage both the code repositories and the binary package building & distributing processes. In Perl, CPAN exists (ActiveState need do nothing there), and PPM is "relatively" easy (it's still hard, since many many modules fail to build on some OSes or their reqs aren't clear, or ...). It's a maintenance nightmare. I'm not convinced that PyPPM will live much longer. I'm very interested in alternatives that provide value to our users and that don't have the high maintenance cost. 2) Distutils, while better than nothing, is still very young. There are (lots of) bugs (in codepaths that most people don't touch), but mostly it's not really designed to do what we need it to do. People tend to write setup.py scripts for one particular goal ("setup.py install"), and extracting the relevant information to get "setup.py makeppd" to work takes a large amount of work for each package for each platform. I argued as hard as I could in distutils' early days that we needed a declarative syntax to describe packages rather than Python code, but that wasn't chosen. I still think I was right =). I realize that I could pick up the mantle of distutils and speak with code. However, changing such a fundamental aspect involves more work than it's worth to me right now. 3) The chicken/egg problem hits us hard -- the infrastructure isn't in place, and putting in infrastructure is expensive. Making it easier for Python users to install modules is hardly where ActiveState can expect to reap the largest rewards for its efforts in our larger picture. 4) The first implementation of PyPPM was sub-par. I'd rather rewrite it from scratch. 5) Distutils is "too" good in some ways. The windows installers, for example, solve the problem well enough for enough users (and, crucially, package authors), that there isn't a huge pressure on us to do it better/differently. As I've said before, I'm not convinced that Python modules belong at the same level of granularity as Python itself, but that's a second-order problem. Just a few thoughts. --david From s-thapa-11@alumni.uchicago.edu Fri Oct 25 22:45:09 2002 From: s-thapa-11@alumni.uchicago.edu (Suchandra Thapa) Date: 25 Oct 2002 16:45:09 -0500 Subject: [Catalog-sig] RFC: pypan - a Python package manager Message-ID: <1035582310.1575.126.camel@hepcat> --=-9eNyJx/Y+2Xf/DSv/lNS Content-Type: multipart/mixed; boundary="=-V29YUkY6rPi32loKGNgD" --=-V29YUkY6rPi32loKGNgD Content-Type: multipart/alternative; boundary="=-ORxt5hDhwgcniyLIBxRs" --=-ORxt5hDhwgcniyLIBxRs Content-Type: text/plain Content-Transfer-Encoding: quoted-printable On Fri, 2002-10-25 at 14:55, Thomas Heller wrote:=20 The chicken and egg issue. It's difficult to reach the critical mass where the system is actually used. It might be useful to see how CPAN got started. =20 But a 'remove' feature is really needed for a package manager, I think. Since, ciphon already differentiates rpm and win32 installs from I can run the uninstall=20 for those cases and do nothing in the general one (for now at least). > I'll change the configuration files and the ciphon config command to > allow the smtp server to be set. I think I'll keep it around since > it seems like it would be useful for developers to be able to get an > automated bug report with a stack trace emailed to the bugs mailing > list. I'm planning on extending it a little further so that users > can send a bug report manually and to give better reporting of > errors. =20 My usual work style is to catch exceptions in a sys.excepthook installe= d in my sitecustomize file - this recipe is in the Python Cookbook. This excepthook starts the debugger (pdb), and lets me inspect the state of the program immediately. I'll see if I can get something similar into the current code. By the way, there are a lot of unqualified try/except clauses in ciphon.py, which make debugging a pain IMO, and also sometimes mask programming errors. =20 There are a bit of places in ciphon where the exception handling is sub-par. That's something that I need to work on. --=20 ------------------------------------------------------------------ Suchandra S. Thapa=20 s-thapa-11@alumni.uchicago.edu ------------------------------------------------------------------ --=-ORxt5hDhwgcniyLIBxRs Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable On Fri, 2002-10-25 at 14:55, Thomas Heller wrote:=20 
The chicken and egg issue.=
 It's difficult to reach the critical mass
where the system is actually us=
ed.
It might be useful to see how CPAN got started.  
But a 'remove' feature is =
really needed for a package manager, I
think.
Since, ciphon already differentiates rpm and win32 install= s from I can run the uninstall
for those cases and do nothing in the general one (for now= at least). 
> I'll change the confi=
guration files and the ciphon config command to
> allow the smtp server to b=
e set.  I think I'll keep it around since
> it seems like it would be =
useful for developers to be able to get an
> automated bug report with =
a stack trace emailed to the bugs mailing
> list.  I'm planning on ext=
ending it a little further so that users
> can send a bug report manu=
ally and to give better reporting of
> errors.

My usual work style is to catch=
 exceptions in a sys.excepthook installed
in my sitecustomize file - this=
 recipe is in the Python Cookbook.
This excepthook starts the debu=
gger (pdb), and lets me inspect
the state of the program immedi=
ately.
I'll see if I can get something similar into the current c= ode. 
By the way, there are a lo=
t of unqualified try/except clauses in
ciphon.py, which make debugging=
 a pain IMO, and also sometimes mask
programming errors.
There are a bit of places in ciphon where the exception ha= ndling is sub-par.  That's something that I need to work on. 
--=20
------------------------------------------------------------------

Suchandra S. Thapa=20
s-thapa-11@alumni.uchicago.edu

------------------------------------------------------------------

--=-ORxt5hDhwgcniyLIBxRs-- --=-V29YUkY6rPi32loKGNgD Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part Content-Transfer-Encoding: base64 LS0tLS1CRUdJTiBQR1AgU0lHTkFUVVJFLS0tLS0KVmVyc2lvbjogR251UEcgdjEuMC42IChHTlUv TGludXgpCkNvbW1lbnQ6IEZvciBpbmZvIHNlZSBodHRwOi8vd3d3LmdudXBnLm9yZwoKaUVZRUFC RUNBQVlGQWoyNXV3OEFDZ2tRNm5TaENqdDVBWkpFNndDZlZidUFPc2Qya3RJYUNDVUNlZDFSck0z ZgpHTGtBb0lic0llSmJvUlJobDlwVlBhNnZ5V0YySmg4RQo9NGd3RAotLS0tLUVORCBQR1AgU0lH TkFUVVJFLS0tLS0K --=-V29YUkY6rPi32loKGNgD-- --=-9eNyJx/Y+2Xf/DSv/lNS Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: For info see http://www.gnupg.org iEYEABECAAYFAj25u2UACgkQ6nShCjt5AZJIQgCgkz5FRt9ZUnZKSyILSLBOi89o ZfUAoJjfR15bBRWw6ExgCQDW/4SKpc9n =dFRs -----END PGP SIGNATURE----- --=-9eNyJx/Y+2Xf/DSv/lNS-- From DavidA@ActiveState.com Fri Oct 25 22:53:57 2002 From: DavidA@ActiveState.com (David Ascher) Date: Fri, 25 Oct 2002 14:53:57 -0700 Subject: [Catalog-sig] RFC: pypan - a Python package manager References: <1035582310.1575.126.camel@hepcat> Message-ID: <3DB9BD75.5020405@ActiveState.com> Suchandra Thapa wrote: > On Fri, 2002-10-25 at 14:55, Thomas Heller wrote: > > /The chicken and egg issue. It's difficult to reach the critical mass/// > /where the system is actually used./// > > It might be useful to see how CPAN got started. The standard perl way. A huge amount of energy, a lot of people actively working together, and a lack of fear of hacks. CPAN is nothing but a replicated FTP server w/ a bit of authentication and a lot of library science =). There's almost 0 meta-data, no distribution model. It was also started at a time when there were no alternatives -- only a few people had access to permanent ftp sites or web sites from which to publish. --david From s-thapa-11@alumni.uchicago.edu Fri Oct 25 23:04:31 2002 From: s-thapa-11@alumni.uchicago.edu (Suchandra Thapa) Date: 25 Oct 2002 17:04:31 -0500 Subject: [Catalog-sig] RFC: pypan - a Python package manager In-Reply-To: <3DB9BD75.5020405@ActiveState.com> References: <1035582310.1575.126.camel@hepcat> <3DB9BD75.5020405@ActiveState.com> Message-ID: <1035583471.1574.144.camel@hepcat> --=-Jc7o67b+Sw67N3s9g8Jg Content-Type: multipart/alternative; boundary="=-8v1viCvHgfUO3Dn8lIYL" --=-8v1viCvHgfUO3Dn8lIYL Content-Type: text/plain Content-Transfer-Encoding: quoted-printable On Fri, 2002-10-25 at 16:53, David Ascher wrote: Suchandra Thapa wrote: > On Fri, 2002-10-25 at 14:55, Thomas Heller wrote: >=20 > /The chicken and egg issue. It's difficult to reach the critical mass= /// > /where the system is actually used./// >=20 > It might be useful to see how CPAN got started.=20 =20 The standard perl way. A huge amount of energy, a lot of people active= ly=20 working together, and a lack of fear of hacks. CPAN is nothing but a r= eplicated=20 FTP server w/ a bit of authentication and a lot of library science =3D)= . There's=20 almost 0 meta-data, no distribution model. It was also started at a ti= me when=20 there were no alternatives -- only a few people had access to permanent= ftp=20 sites or web sites from which to publish. Couldn't we do the same thing? Have package distribution done using http or ftp servers then add more sophisticated package handling done later. As long as we provide a place of updating the original package manager over the internet, we should be able to make changes in the distribution method somewhat seamless. =20 We could probably get a fairly good userbase for a package manager just by providing a few packages like PIL, numeric python, mxtools, and win32all. =20 --=20 ------------------------------------------------------------------ Suchandra S. Thapa=20 s-thapa-11@alumni.uchicago.edu ------------------------------------------------------------------ --=-8v1viCvHgfUO3Dn8lIYL Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable On Fri, 2002-10-25 at 16:53, David Ascher wrote: 
Suchandra Thapa wrote:
> On Fri, 2002-10-25 at 14:5=
5, Thomas Heller wrote:
> 
> /The chicken and egg issue=
. It's difficult to reach the critical mass///
> /where the system is actua=
lly used.///
> 
> It might be useful to see =
how CPAN got started. 

The standard perl way.  A huge =
amount of energy, a lot of people actively 
working together, and a lack of=
 fear of hacks.  CPAN is nothing but a replicated 
FTP server w/ a bit of authenti=
cation and a lot of library science =3D).  There's 
almost 0 meta-data, no distribu=
tion model.  It was also started at a time when 
there were no alternatives -- o=
nly a few people had access to permanent ftp 
sites or web sites from which t=
o publish.
Couldn't we do the same thing?  Have package distribu= tion done using http or ftp servers then add more sophisticated package han= dling done later.  As long as we provide a place of updating the origi= nal package manager over the internet, we should be able to make changes in= the distribution method somewhat seamless. 

We could probably get a fairly good userbase for a package= manager just by providing  a few packages like PIL, numeric python, m= xtools, and win32all.  
--=20
------------------------------------------------------------------

Suchandra S. Thapa=20
s-thapa-11@alumni.uchicago.edu

------------------------------------------------------------------
--=-8v1viCvHgfUO3Dn8lIYL-- --=-Jc7o67b+Sw67N3s9g8Jg Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: For info see http://www.gnupg.org iEYEABECAAYFAj25v+8ACgkQ6nShCjt5AZKb/ACggMDioy7dkzKr6Jb6XYP6kDTb RgwAni07K4d7p4a6Gwb8PFx2NFJMqHe+ =J+wi -----END PGP SIGNATURE----- --=-Jc7o67b+Sw67N3s9g8Jg-- From Chris.Barker@noaa.gov Fri Oct 25 23:35:16 2002 From: Chris.Barker@noaa.gov (Chris Barker) Date: Fri, 25 Oct 2002 15:35:16 -0700 Subject: [Catalog-sig] RFC: pypan - a Python package manager References: <1035582310.1575.126.camel@hepcat> <3DB9BD75.5020405@ActiveState.com> Message-ID: <3DB9C724.DDBA2A2C@noaa.gov> David Ascher wrote: > > /The chicken and egg issue. It's difficult to reach the critical mass/// > > /where the system is actually used./// > > > > It might be useful to see how CPAN got started. > > The standard perl way. A huge amount of energy, a lot of people actively > working together, and a lack of fear of hacks. CPAN is nothing but a replicated > FTP server w/ a bit of authentication and a lot of library science =). This history can be instructive. I've always thought that the community as a whole was kind of going about this backward. There is a lot of talk, and a fair bit of coding about the technology of a Python Module Catalog, but no one actually building the catalog. When I go to set up Python on a new machine (or upgrade). I have to go to a bunch of different web sites to get the packages I generally use, but for the most part, getting them to install has been pretty painless. Some I just copy, some I ./configure; make; make install, some I ./setup.py build, etc. On Windows, most of them come with installers. If I could just get them from one web site I'd be a whole lot happier right there. If they all installed the same way that would be great too. I'd like it even more if there was one installer that installed a "complete" system. There is no such thing, of course, but there are a limited number of packages that are in really general use (PIL, mxTools, Numeric, ...). All I would be wasting is some disk space and download bandwidth to download a pile of stuff in one fell swoop. If a particular set of packages where semi-officially known as CompletePython version 2.2 (or whatever), then when you wanted to distribute your code, you could just require CompletePython 2.2, and your users would have what they need. I guess what I'm getting at is an expanded "batteries included" philosophy. It is great that the standard library has as much as it does, but it's not enough. Trying to fold all that other stuff into the standard library wouldn't work, of course, but if we managed to establish a list of commonly used third party packages, each maintained by different folks, but obtainable from one source, we'd have a great start. I'm inspired by the old "Python on Linux" site (sorry, I can't remember who put that together). All I had to do was go there, download everything, and do an rpm -U *, and I had most of the stuff I needed. What it would take to get this going is someone to host the site, and a small group of folks to decide on a package list and start populating the site. As it grows, it would be useful to build the tools to automate the whole thing more, but we could get it started without any fancy technology. I proposed something like this on c.l.p a while back (1yr or so??). I got surprisingly little support, and I just couldn't do it alone, so it died. We really need a core group of a few folks (ideally one with experience in building packages/installers on each of the major platforms). What I envision is being able to go to www.completepython.org, click on version 2.2, click on Windows|OS_X|Linux and get a list of maybe 25-50 packages, any of which I could click on, or a "Download All" button. That's it. Maybe if that package list grows to 100s (or thousands) it would require more organization, but let's cross that bridge when we come to it. Yow! that rant got a lot longer than I had planned. Thanks for bearing with me, if you got this far. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov From DavidA@ActiveState.com Fri Oct 25 23:48:14 2002 From: DavidA@ActiveState.com (David Ascher) Date: Fri, 25 Oct 2002 15:48:14 -0700 Subject: [Catalog-sig] RFC: pypan - a Python package manager References: <1035582310.1575.126.camel@hepcat> <3DB9BD75.5020405@ActiveState.com> <3DB9C724.DDBA2A2C@noaa.gov> Message-ID: <3DB9CA2E.5070707@ActiveState.com> Chris Barker wrote: > David Ascher wrote: > > I've always thought that the community as a whole was kind of going > about this backward. There is a lot of talk, and a fair bit of coding > about the technology of a Python Module Catalog, but no one actually > building the catalog. That doesn't surprise me, if you consider why people do things in this community -- because they enjoy it. I'm just as guilty of that as anyone. I, like most of the community, like to design, code, and talk, but not really manage. It takes a person who likes to manage (with the coddling, arm-twisting, persistence and rigor that any kind of management implies) to get this job done. IMO, of course. --david From andy@agmweb.ca Fri Oct 25 23:56:40 2002 From: andy@agmweb.ca (Andy McKay) Date: Fri, 25 Oct 2002 15:56:40 -0700 Subject: [Catalog-sig] RFC: pypan - a Python package manager References: <1035582310.1575.126.camel@hepcat> <3DB9BD75.5020405@ActiveState.com> <3DB9C724.DDBA2A2C@noaa.gov> Message-ID: <000501c27c79$c910e590$89505018@cr582427a> > > The standard perl way. A huge amount of energy, a lot of people actively > > working together, and a lack of fear of hacks. CPAN is nothing but a replicated > > FTP server w/ a bit of authentication and a lot of library science =). > > This history can be instructive. I think thats part of the problem, the Perl thing started out as something quite simple. Many of the proposals have been over analysed with far too many features to the degree that people only get so far before time pressures or interest drops off. I've seen proposals that want things available by HTTP, XML-RPC, RSS, FTP, Web Services etc..., and cope with every different way imaginable of uploading a module. Although Zope.org may have (many) problems, but its simply a case of making a tar ball and put some info about it. Sure it doesn't have the automated installer, but as a simple repository it does a basic job. While I'm not saying its a good solution, it makes the point that a simple solution would really help as the first step. -- Andy McKay www.agmweb.ca From rjones@ekit-inc.com Sat Oct 26 00:31:26 2002 From: rjones@ekit-inc.com (Richard Jones) Date: Sat, 26 Oct 2002 09:31:26 +1000 Subject: [Catalog-sig] New proposal, with PEP Message-ID: <200210260931.26702.rjones@ekit-inc.com> [ I've now posted three messages on the subject of my online catalog effort - four if you count the message I sent to the distutils sig. None of those posts have generated any feedback to me. Not even "piss off, you're wasting our precious time". That's a bit damn disheartening. People have looked at and used the prototype though, so I suppose at least the posts weren't ignored. I'm persisting regardless, because I believe I've got a good idea, and I have friends who also believe I'm on to a good thing too. I also know there's a history of little support for these projects. I have now written a draft PEP which follows. Comments are welcome. Please note the specific limitations of scope in the Abstract. ] PEP: XXX Title: Distutils Enhancements Version: $Revision: 1.2 $ Last-Modified: $Date: 2002/10/25 07:08:28 $ Author: Richard Jones Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 24-Oct-2002 Python-Version: 2.3 Post-History: Abstract ======== This PEP proposes several extensions to the distutils packaging system [1]_. These enhancements include a central package index, tools for submitting package information to the index and extensions to the package metadata to include Trove [2]_ information. This PEP does not address either issues of package dependency, nor centralised storage of packages. Nor is it proposing a local database of packages as described in PEP 262 [6]_. Existing package repositories such as the Vaults of Parnassus [3]_, CPAN [4]_ and PAUSE [5]_ will be investigated as prior art in this field. Rationale ========= Python programmers have long needed a simple method of discovering existing modules and systems available for their use. It is arguable that the existence of these systems for other languages have been a significant contribution to their popularity. The existence of the catalog-sig, and the many discussions there indicate that there is a large population of users who recognise this need. The introduction of the distutils packaging system to Python simplified the process of distributing shareable code, and included mechanisms for capture of package metadata, but did little with the metadata save ship it with the package. The server should be hosted in the python.org domain, giving it an air of legitimacy that existing catalog efforts do not have. The interface for submitting information to the catalog should be as simple as possible - hopefully just one command-line command for more regular users. Issues of package dependency are not addressed due to the complexity of such a system. The original PEP which proposed such a system was dropped as the author realised that platform packaging system (RPM, apt, etc) already handle dependencies, installation and removal. Issues of package dissemination (storage on a central server) are not addressed because they require assumptions about availability of storage and bandwidth that I am not in a position to make. Specification ============= The specification takes three parts, the `web interface`_, the `distutils register command`_ and the `distutils trove categorisation`_. Web Interface ------------- A web interface is implemented over a simple store. The interface is available through the python.org domain, either directly or as packages.python.org. The store has columns for all metadata fields. The (name, version) double is used as a uniqueness key. Additional submissions for an existing (name, version) will result in an *update* operation. The web interface implements the following commands: **index** Lists known packages, optionally filtered. An additional HTML page, **search**, presents a form to the user which is used to customise the index view. The index will include a browsing interface like that presented in the Trove interface design section 4.3. The results will be paginated, sorted alphabetically and only showing the most recent version. Most recent version information will be determined using the distutils LooseVersion class. **display** Displays information about the package. All fields are displayed as plain text. The "url" (or "home_page") field is hyperlinked. **submit** Accepts a POST form submission of metadata about a package. The "name" and "version" fields are mandatory, as they uniquely identify an entry in the index. Submit will automatically determine whether to create a new entry or updating an existing entry. The metadata is checked for correctness where appropriate - specifically the Trove discriminators are compared with the allowed set. An update will update all information about the package based on the new submitted information. **user** Registers a new user with the index. Requires username, password and email address. Passwords will be stored on the server as SHA hashes. If the username exists: 1. If valid HTTP Basic auth is provided, the password and email address are updated with the submission information, or 2. If no valid auth is provided, the user is informed that the login is already taken. Registration will be a three-step process, involving: 1. User submission of details to the *register* command, 2. Web server sending email to the user's email address with a URL to visit to confirm registration with a random one-time key, and 3. User visits URL with the key and confirms registration. Several user Roles will exist: Admin Can assign Owner Role - they decide who may submit for a given package name. Owner Owns a package name, may assign Maintainer Role for that name Maintainer Can submit and update info for a particular package name **password_reminder** Sends a password reminder to the user's email address. The **submit** command will require HTTP Basic authentication, preferrably over an HTTPS connection. Distutils Register Command -------------------------- An additional distutils command, "register" is implemented which posts the package metadata to the central server. The register command automatically handles user registration; the user is presented with three options: 1. login and submit package information 2. register as a new packager 3. send password reminder email On UN*X systems, the user will be prompted at exit to save their username/password to a file in their home directory in the file ``.pythonpackagerc``. A similar system could be used on Windows, I suppose. Notification of changes to a package entry will be sent to all users who have submitted information about the package. That is, the original submitter and any subsequent updaters. The register command will include a --verify option which performs a test submission to the server without actually committing the data. The server will perform its submission verification checks as usual and report any errors it would have reported during a normal submission. This is useful for verifying correctness of Trove discriminators. Distutils Trove Categorisation ------------------------------ The Trove concept of *discriminators* will be added to the metadata set available to package authors. The list of discriminators will be available through the web, and added to the package like so:: setup( name = "roundup", version = __version__, discriminators = [ 'Development Status :: 4 - Beta', 'Environment :: Console (Text Based)', 'Environment :: Web Environment', 'Intended Audience :: End Users/Desktop', 'Intended Audience :: Developers', 'Intended Audience :: System Administrators', 'License :: OSI Approved :: Python License', 'Operating System :: MacOS X', 'Operating System :: Microsoft :: Windows', 'Operating System :: POSIX', 'Programming Language :: Python', 'Topic :: Communications :: Email', 'Topic :: Office/Business', 'Topic :: Software Development :: Bug Tracking', ], url = 'http://sourceforge.net/projects/roundup/', ... ) It was decided that strings would be used for the discriminator entries due to the deep nesting that would be involved in a more formal Python structure. The original Trove specification that discriminator namespaces be separated by slashes ("/") unfortunately collides with many of the names having slashes in them (eg. "OS/2"). The double-colon solution (" :: ") implemented by Sourceforge and Freshmeat gets around this limitation. The list of discriminator values on the module index has been merged from Freshmeat (with their permission) and Sourceforge (awaiting their approval). This list will be made available through the web interface as a text list which may then be copied to the ``setup.py`` file. Reference Implementation ======================== Reference code and demonstration server are available at http://mechanicalcat.net:8081/ ===== =================================================== Done Feature ===== =================================================== Y Submission Y Index Y Display Y Search N User rego N Trove N User verification N Password reminder ===== =================================================== In the three days 22nd and 23rd October after the first announcement to the catalog-sig (22nd) and distutils-sig (23rd), the prototype had 50 visitors (not including myself), three of whom used the register command to submit package information. References ========== .. [1] distutils packaging system (http://www.python.org/doc/current/lib/module-distutils.html) .. [2] Trove (http://tuxedo.org/~esr/trove/) .. [3] Vaults of Parnassus (http://www.vex.net/parnassus/) .. [4] CPAN (http://www.cpan.org/) .. [5] PAUSE (http://pause.cpan.org/) .. [6] PEP 262, A Database of Installed Python Packages (http://www.python.org/peps/pep-0262.html) Copyright ========= This document has been placed in the public domain. Acknowledgements ================ Anthony Baxter and Toby Sargeant for encouragement and feedback during initial drafting. The many participants of the distutils and catalog SIGs for their ideas over the years. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: From martin@v.loewis.de Sat Oct 26 09:23:20 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 26 Oct 2002 10:23:20 +0200 Subject: [Catalog-sig] RFC: pypan - a Python package manager In-Reply-To: <1035583471.1574.144.camel@hepcat> References: <1035582310.1575.126.camel@hepcat> <3DB9BD75.5020405@ActiveState.com> <1035583471.1574.144.camel@hepcat> Message-ID: Suchandra Thapa writes: > Couldn't we do the same thing? Have package distribution done using > http or ftp servers then add more sophisticated package handling done > later. I think what is lacking is the huge amount of energy and the lot of people actively working together. Regards, Martin From martin@v.loewis.de Sat Oct 26 09:37:43 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 26 Oct 2002 10:37:43 +0200 Subject: [Catalog-sig] New proposal, with PEP In-Reply-To: <200210260931.26702.rjones@ekit-inc.com> References: <200210260931.26702.rjones@ekit-inc.com> Message-ID: Richard Jones writes: > I've now posted three messages on the subject of my online catalog effort - > four if you count the message I sent to the distutils sig. None of those > posts have generated any feedback to me. Not even "piss off, you're wasting > our precious time". That's a bit damn disheartening. I think this is the problem. If you get disheartened easily, expect to give up sooner or later, like everybody else before you. Every time somebody comes along and offers some kind of catalog, I ask whether they actually want to operate it, or whether they only want to write the software - or perhaps even just define a spec. I'm not interested in specs, and I'm only mildly interested in software. What I want to see is the catalog in operation. > People have looked at and used the prototype though, so I suppose at > least the posts weren't ignored. I'm persisting regardless, because > I believe I've got a good idea, and I have friends who also believe > I'm on to a good thing too. I also know there's a history of little > support for these projects. I think you are misinterpreting history. People used to get quite involved in discussing specs; these days, they might be tired because of fear that the project won't progress beyond the spec phase. As for contributions to the software: Most software was good enough when released initially. As for actually using the infrastructure: That was typically impossible because of the lack of infrastructure to use. What other support would you expect? Regards, Martin From lac@strakt.com Sat Oct 26 09:35:54 2002 From: lac@strakt.com (Laura Creighton) Date: Sat, 26 Oct 2002 10:35:54 +0200 Subject: [Catalog-sig] RFC: pypan - a Python package manager In-Reply-To: Message from Chris Barker of "Fri, 25 Oct 2002 15:35:16 PDT." <3DB9C724.DDBA2A2C@noaa.gov> References: <1035582310.1575.126.camel@hepcat> <3DB9BD75.5020405@ActiveState.com> <3DB9C724.DDBA2A2C@noaa.gov> Message-ID: <200210260835.g9Q8Zsen027437@ratthing-b246.strakt.com> This is what the PBF is planning to do with Python-in-a-Tie. What packages do you think belong in this release? I am trying to get packages sent and tested on the Snake Farm, but it turns out that a fair number of people simply don't care if their packages run on Solaris, HP or other architectures right now. Laura Creighton Chris Barker wrote: > > When I go to set up Python on a new machine (or upgrade). I have to go > to a bunch of different web sites to get the packages I generally use, > but for the most part, getting them to install has been pretty painless. > Some I just copy, some I ./configure; make; make install, some I > ./setup.py build, etc. On Windows, most of them come with installers. > > If I could just get them from one web site I'd be a whole lot happier > right there. If they all installed the same way that would be great too. > > I'd like it even more if there was one installer that installed a > "complete" system. There is no such thing, of course, but there are a > limited number of packages that are in really general use (PIL, mxTools, > Numeric, ...). All I would be wasting is some disk space and download > bandwidth to download a pile of stuff in one fell swoop. If a particular > set of packages where semi-officially known as CompletePython version > 2.2 (or whatever), then when you wanted to distribute your code, you > could just require CompletePython 2.2, and your users would have what > they need. > > I guess what I'm getting at is an expanded "batteries included" > philosophy. It is great that the standard library has as much as it > does, but it's not enough. Trying to fold all that other stuff into the > standard library wouldn't work, of course, but if we managed to > establish a list of commonly used third party packages, each maintained > by different folks, but obtainable from one source, we'd have a great > start. > > I'm inspired by the old "Python on Linux" site (sorry, I can't remember > who put that together). All I had to do was go there, download > everything, and do an rpm -U *, and I had most of the stuff I needed. > > What it would take to get this going is someone to host the site, and a > small group of folks to decide on a package list and start populating > the site. As it grows, it would be useful to build the tools to automate > the whole thing more, but we could get it started without any fancy > technology. > > I proposed something like this on c.l.p a while back (1yr or so??). I > got surprisingly little support, and I just couldn't do it alone, so it > died. We really need a core group of a few folks (ideally one with > experience in building packages/installers on each of the major > platforms). > > What I envision is being able to go to www.completepython.org, click on > version 2.2, click on Windows|OS_X|Linux and get a list of maybe 25-50 > packages, any of which I could click on, or a "Download All" button. > That's it. Maybe if that package list grows to 100s (or thousands) it > would require more organization, but let's cross that bridge when we > come to it. > > Yow! that rant got a lot longer than I had planned. Thanks for bearing > with me, if you got this far. > > -Chris > > > -- > Christopher Barker, Ph.D. > Oceanographer > > NOAA/OR&R/HAZMAT (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker@noaa.gov > > _______________________________________________ > Catalog-sig mailing list > Catalog-sig@python.org > http://mail.python.org/mailman/listinfo/catalog-sig From rjones@ekit-inc.com Sat Oct 26 13:05:44 2002 From: rjones@ekit-inc.com (Richard Jones) Date: Sat, 26 Oct 2002 22:05:44 +1000 Subject: [Catalog-sig] New proposal, with PEP In-Reply-To: References: <200210260931.26702.rjones@ekit-inc.com> Message-ID: <200210262205.44480.rjones@ekit-inc.com> On Sat, 26 Oct 2002 6:37 pm, Martin v. Loewis wrote: > Richard Jones writes: > > I've now posted three messages on the subject of my online catalog effort > > - four if you count the message I sent to the distutils sig. None of > > those posts have generated any feedback to me. Not even "piss off, you're > > wasting our precious time". That's a bit damn disheartening. > > I think this is the problem. If you get disheartened easily, expect to > give up sooner or later, like everybody else before you. Note that I continued on to say that I have not given up. Just that some feedback (either positive, negative _or_ ambivalent) would be nice. > Every time somebody comes along and offers some kind of catalog, I ask > whether they actually want to operate it, or whether they only want to > write the software - or perhaps even just define a spec. > > I'm not interested in specs, and I'm only mildly interested in > software. What I want to see is the catalog in operation. It's in operation right now (minus the user-based features which are pretty trivial to implement) at the hosting I use. I've had several users querying the catalog and submitting information to it. It's not in operation at python.org though, which I see as being important to its acceptance. I don't have any control over that. Hence the PEP. Ongoing support would be minimal, and I'd be gladly putting my hand up to take on that job, assuming I can have access to the machine python.org is hosted on. > > People have looked at and used the prototype though, so I suppose at > > least the posts weren't ignored. I'm persisting regardless, because > > I believe I've got a good idea, and I have friends who also believe > > I'm on to a good thing too. I also know there's a history of little > > support for these projects. > > I think you are misinterpreting history. People used to get quite > involved in discussing specs; these days, they might be tired because > of fear that the project won't progress beyond the spec phase. > > As for contributions to the software: Most software was good enough > when released initially. > > As for actually using the infrastructure: That was typically > impossible because of the lack of infrastructure to use. > > What other support would you expect? For this to work, I need: 1. the web interface installed on python.org, and either someone there to look after it or access so I can look after it, 2. the "register" command included in the next non-patch python distribution (ie. 2.3), and 3. optionally have the distutils metadata expanded to include Trove discriminators I deliberately minimised the infrastructure requirement of the implementation so that it could run with little impact on python.org. I could just leave it running on my host, but that'd have nowhere near the legitemacy of it running on python.org. Richard From doug@hellfly.net Sat Oct 26 13:44:47 2002 From: doug@hellfly.net (Doug Hellmann) Date: Sat, 26 Oct 2002 08:44:47 -0400 Subject: [Catalog-sig] New proposal, with PEP In-Reply-To: <200210260931.26702.rjones@ekit-inc.com> References: <200210260931.26702.rjones@ekit-inc.com> Message-ID: <200210261244.IAA17993@branagan.hellfly.net> On Friday 25 October 2002 7:31 pm, Richard Jones wrote: > [ > I've now posted three messages on the subject of my online catalog effort - > four if you count the message I sent to the distutils sig. None of those > posts have generated any feedback to me. Not even "piss off, you're wasting > our precious time". That's a bit damn disheartening. People have looked at > and used the prototype though, so I suppose at least the posts weren't > ignored. I'm persisting regardless, because I believe I've got a good idea, > and I have friends who also believe I'm on to a good thing too. I also know > there's a history of little support for these projects. I missed your first post, but I like the PEP and I've tried out the reference implementation with some success by submitting HappyDoc. Definitely enough to make me want to see the rest of it implemented. You couldn't have made the interface any easier, that's for sure! Doug From martin@v.loewis.de Sat Oct 26 19:37:00 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 26 Oct 2002 20:37:00 +0200 Subject: [Catalog-sig] New proposal, with PEP In-Reply-To: <200210262205.44480.rjones@ekit-inc.com> References: <200210260931.26702.rjones@ekit-inc.com> <200210262205.44480.rjones@ekit-inc.com> Message-ID: Richard Jones writes: > Note that I continued on to say that I have not given up. Just that some > feedback (either positive, negative _or_ ambivalent) would be nice. Ok, so here's my negative feedback: Why would anybody want to use such a thing? As long as it contains only a few dozen packages, I really don't see a need. To make it larger, I think it must accommodate a wider range of packages, not just those that get installed through distutils. But if that is the way to go, how is this different from the Vaults, or Freshmeat? If I were to look for Python packages, I'd look at http://freshmeat.net/browse/178 Nicely organized by Trove category and all. > It's not in operation at python.org though, which I see as being > important to its acceptance. I don't have any control over > that. Hence the PEP. Ongoing support would be minimal, and I'd be > gladly putting my hand up to take on that job, assuming I can have > access to the machine python.org is hosted on. I'd be happy to install it on python.org, just give me instructions (in private email). > 1. the web interface installed on python.org, and either someone there to > look after it or access so I can look after it, I can certainly help here. > 2. the "register" command included in the next non-patch python > distribution (ie. 2.3), and That will be difficult; you need to propose it on python-dev. For the time being, you should find a means to do without. > 3. optionally have the distutils metadata expanded to include Trove > discriminators That will be really hard, I guess - people won't see the need to add this to their setup calls. It would also cause the packages to break on older distutils installations. Regards, Martin From rjones@ekit-inc.com Sat Oct 26 23:54:29 2002 From: rjones@ekit-inc.com (Richard Jones) Date: Sun, 27 Oct 2002 08:54:29 +1000 Subject: [Catalog-sig] New proposal, with PEP In-Reply-To: References: <200210260931.26702.rjones@ekit-inc.com> <200210262205.44480.rjones@ekit-inc.com> Message-ID: <200210270954.30130.rjones@ekit-inc.com> On Sun, 27 Oct 2002 5:37 am, Martin v. Loewis wrote: > Richard Jones writes: > > Note that I continued on to say that I have not given up. Just that some > > feedback (either positive, negative _or_ ambivalent) would be nice. > > Ok, so here's my negative feedback: > > Why would anybody want to use such a thing? As long as it contains > only a few dozen packages, I really don't see a need. > > To make it larger, I think it must accommodate a wider range of > packages, not just those that get installed through distutils. It's trivial to provide a form interface to manually submit package information. I will do so now. > But if > that is the way to go, how is this different from the Vaults, or > Freshmeat? If I were to look for Python packages, I'd look at Because: 1. there's no integration with distutils, and consequently no one-shot, trivial mechanism for submitting metadata, 2. neither of the above are hosted at python.org, and hence don't have any of the legitemacy that that hosting would bring, and 3. Freshmeat is a pain to use, and only supports open-source Linux projects (or at a minimum open-source projects that are available on Linux). > > It's not in operation at python.org though, which I see as being > > important to its acceptance. I don't have any control over > > that. Hence the PEP. Ongoing support would be minimal, and I'd be > > gladly putting my hand up to take on that job, assuming I can have > > access to the machine python.org is hosted on. > > I'd be happy to install it on python.org, just give me instructions > (in private email). Installation on python.org can wait until I've finished the code, but thanks for the offer! It's good to know that if this is accepted then the python.org step is possible :) > > 1. the web interface installed on python.org, and either someone there to > > look after it or access so I can look after it, > > I can certainly help here. > > > 2. the "register" command included in the next non-patch python > > distribution (ie. 2.3), and > > That will be difficult; you need to propose it on python-dev. Hence the PEP. I _think_ I'm following correct procedure here... > For the time being, you should find a means to do without. The "register" command could almost be considered the most important component of this proposal, making it trivial for people to submit information to the central index. > > 3. optionally have the distutils metadata expanded to include Trove > > discriminators > > That will be really hard, I guess - people won't see the need to add > this to their setup calls. People will add it when they want their package to be easier to find. One of the features I've built into the specification is the notion of updating the metadata - so if they want to add in a longer description at a later date, they can just edit their setup and re-invoke the register command. Same thing with adding or editing the trove discriminators. > It would also cause the packages to break > on older distutils installations. That's a problem, yes. I'm not sure what the solution would be. Older packages run with the new code wouldn't have a problem. Newer packages run with older code will break on the "trove" keyword argument to setup(). I'm not sure what the solution would be. Richard From cliechti@gmx.net Sun Oct 27 00:18:54 2002 From: cliechti@gmx.net (Chris Liechti) Date: Sun, 27 Oct 2002 01:18:54 +0200 Subject: [Catalog-sig] New proposal, with PEP In-Reply-To: <200210270954.30130.rjones@ekit-inc.com> Message-ID: <2U2WTPIEZTCB2ZHBVQOKLKB92ZSC76.3dbb22de@rock> Am 27.10.2002 00:54:29, schrieb Richard Jones : >On Sun, 27 Oct 2002 5:37 am, Martin v. Loewis wrote: >> It would also cause the packages to break >> on older distutils installations. > >That's a problem, yes. I'm not sure what the solution would be. Older packages >run with the new code wouldn't have a problem. Newer packages run with older >code will break on the "trove" keyword argument to setup(). I'm not sure what >the solution would be. how about a try... catch and a second setup method? from distutils.core import setup try: from distutils.core import extended_setup except ImportError: setup(...) else: extended_setup(...) or maybe it's possible to get the distultils version? whatever, i think that solvabe with a bit more code in the setup script. maybe not that nice 'cause the setup script get more complicated but it would be optional to use the extened way anyway. chris From martin@v.loewis.de Sun Oct 27 08:49:46 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 27 Oct 2002 09:49:46 +0100 Subject: [Catalog-sig] New proposal, with PEP In-Reply-To: <200210270954.30130.rjones@ekit-inc.com> References: <200210260931.26702.rjones@ekit-inc.com> <200210262205.44480.rjones@ekit-inc.com> <200210270954.30130.rjones@ekit-inc.com> Message-ID: Richard Jones writes: > 2. neither of the above are hosted at python.org, and hence don't > have any of the legitemacy that that hosting would bring, and In my eyes, Freshmeat has enough legitemacy on its own. > > That will be difficult; you need to propose it on python-dev. > > Hence the PEP. I _think_ I'm following correct procedure here... Sure. At some point, you'd need to submit it to the PEP editor, see PEP 1. > That's a problem, yes. I'm not sure what the solution would > be. Older packages run with the new code wouldn't have a > problem. Newer packages run with older code will break on the > "trove" keyword argument to setup(). I'm not sure what the solution > would be. Unless you can find a hack (like putting data into a separate file for the register command to find them there), I think there is no solution. Wait until Python 2.4 or 2.5 for users to start using the feature (i.e. when they are willing to give up compatibility with older Python releases, for other reasons). Regards, Martin From rjones@ekit-inc.com Sun Oct 27 11:40:07 2002 From: rjones@ekit-inc.com (Richard Jones) Date: Sun, 27 Oct 2002 21:40:07 +1000 Subject: [Catalog-sig] New proposal, with PEP In-Reply-To: References: <200210260931.26702.rjones@ekit-inc.com> <200210270954.30130.rjones@ekit-inc.com> Message-ID: <200210272240.07569.rjones@ekit-inc.com> On Sun, 27 Oct 2002 7:49 pm, Martin v. Loewis wrote: > Richard Jones writes: > > 2. neither of the above are hosted at python.org, and hence don't > > have any of the legitemacy that that hosting would bring, and > > In my eyes, Freshmeat has enough legitemacy on its own. Sure, but not in my eyes. And we're not alone in our respective opinions. > > > That will be difficult; you need to propose it on python-dev. > > > > Hence the PEP. I _think_ I'm following correct procedure here... > > Sure. At some point, you'd need to submit it to the PEP editor, see > PEP 1. Sure, I just wanted to get some feedback on the proposal before going through the formal process. > > That's a problem, yes. I'm not sure what the solution would > > be. Older packages run with the new code wouldn't have a > > problem. Newer packages run with older code will break on the > > "trove" keyword argument to setup(). I'm not sure what the solution > > would be. > > Unless you can find a hack (like putting data into a separate file for > the register command to find them there), I think there is no > solution. Wait until Python 2.4 or 2.5 for users to start using the > feature (i.e. when they are willing to give up compatibility with > older Python releases, for other reasons). So that would imply building the keyword in now and not advertising it? Richard From martin@v.loewis.de Sun Oct 27 21:23:55 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 27 Oct 2002 22:23:55 +0100 Subject: [Catalog-sig] New proposal, with PEP In-Reply-To: <200210272240.07569.rjones@ekit-inc.com> References: <200210260931.26702.rjones@ekit-inc.com> <200210270954.30130.rjones@ekit-inc.com> <200210272240.07569.rjones@ekit-inc.com> Message-ID: Richard Jones writes: > > Unless you can find a hack (like putting data into a separate file for > > the register command to find them there), I think there is no > > solution. Wait until Python 2.4 or 2.5 for users to start using the > > feature (i.e. when they are willing to give up compatibility with > > older Python releases, for other reasons). > > So that would imply building the keyword in now and not advertising it? No, documenting it as "new in Python 2.x" (for x=3 or x=4) would describe the problem precisely: it is then the user's choice whether or not to use it. Regards, Martin From Juergen Hermann" Message-ID: On Sun, 27 Oct 2002 08:54:29 +1000, Richard Jones wrote: >> It would also cause the packages to break >> on older distutils installations. > >That's a problem, yes. I'm not sure what the solution would be. Older p= ackages >run with the new code wouldn't have a problem. Newer packages run with = older >code will break on the "trove" keyword argument to setup(). I'm not sur= e what >the solution would be. We already had such breakage with older distutils versions (pre 1.0 I th= ink), and my way to solve it was: if hasattr(distutils.dist.DistributionMetadata, 'get_keywords'): setup_args['keywords'] =3D "wiki web" if hasattr(distutils.dist.DistributionMetadata, 'get_platforms'): setup_args['platforms'] =3D "win32 posix" apply(setup, (), setup_args) Some official distutils "capability" mechanism would be nicer, of course= . Don't check version numbers! Ciao, J=FCrgen -- J=FCrgen Hermann, Developer WEB.DE AG, http://webde-ag.de/ From lac@strakt.com Mon Oct 28 13:09:02 2002 From: lac@strakt.com (Laura Creighton) Date: Mon, 28 Oct 2002 14:09:02 +0100 Subject: [Catalog-sig] New proposal, with PEP In-Reply-To: Message from Richard Jones of "Sat, 26 Oct 2002 22:05:44 +1000." <200210262205.44480.rjones@ekit-inc.com> References: <200210260931.26702.rjones@ekit-inc.com> <200210262205.44480.rjones@ekit-inc.com> Message-ID: <200210281309.g9SD92en006449@ratthing-b246.strakt.com> I am interested in cataloging software for the ability to add metadata; indeed it is only the ability to add metadata that makes any extra work worthwhile. But I wonder if making the register command a library function that works with various versions of Python would not be a better idea. Then whenever somebody tried to download something from the web interface on python.org the first thing that would be checked is 'do you have this library' and if the answer is no, then it would first install this library for you and then whatever it is that you came here to get. The Python Business Forum is very interested in this software, by the way. Do you want to test it on the Snake Farm? http://www.lysator.liu.se/~sfarmer/ Laura Creighton From nobody@maui.dnsvault.com Wed Oct 30 00:27:03 2002 From: nobody@maui.dnsvault.com (Nobody) Date: Tue, 29 Oct 2002 19:27:03 -0500 Subject: [Catalog-sig] A larger gold balance! Message-ID: Untitled

Hello all E-Gold and EVOcash account holders!

   We here at Your Gold Chance are honest, reliable and willing to help you achieve what you want most..... A larger e-gold balance! We have developed a program to give you a smart return without the usual wait of hyip's. For speed and convenience we utilize online digital currencies. We accept E-Gold and EVOcash.

    All investments are based on a 33 week plan. We pay you 10% per week for a total of 33 weeks. You first payment starts 7 days after you invest and your capital is returned to you at the end of the 33th week.

   The minimum Investment is $25.00USD and the maximum Investment is $25,000 USD. Do not send us any amount outside these parameters otherwise your investment will be returned without profit.

If you are interested in this program visit site:

http://www.yourgoldchance.net/invest.html

All additional information you can get here:

http://www.yourgoldchance.net/

If you would like to contact us, please send us an email:

yourgoldchance@yourgoldchance.net

Our company will continue its best efforts to make good on every payments on our program without any delay.

Thanks for choosing Your Gold Chance!
Karen Andreozzi,
President
Yor Gold Chance, Inc.

From rjones@ekit-inc.com Wed Oct 30 00:34:50 2002 From: rjones@ekit-inc.com (Richard Jones) Date: Wed, 30 Oct 2002 11:34:50 +1100 Subject: [Catalog-sig] New proposal, with PEP In-Reply-To: <200210281309.g9SD92en006449@ratthing-b246.strakt.com> References: <200210260931.26702.rjones@ekit-inc.com> <200210262205.44480.rjones@ekit-inc.com> <200210281309.g9SD92en006449@ratthing-b246.strakt.com> Message-ID: <200210301134.50475.rjones@ekit-inc.com> On Tue, 29 Oct 2002 12:09 am, Laura Creighton wrote: > I am interested in cataloging software for the ability to add > metadata; indeed it is only the ability to add metadata that makes > any extra work worthwhile. What metadata would you be talking about adding? > But I wonder if making the register command > a library function that works with various versions of Python would > not be a better idea. Having the "register" command is central to the adoption of the proposal. It will be made available as a download for people running pre-adoption versions of Python (ie. currently 2.1 and 2.2 and hopefully not 2.3). > Then whenever somebody tried to download > something from the web interface on python.org the first thing that > would be checked is 'do you have this library' and if the answer is > no, then it would first install this library for you and then whatever > it is that you came here to get. I'm not sure how this works. Issues of package installation and dependency have killed so many proposals so far, including PEP 262, that I'm specifically not addressing them. They can be added later - we just need something now to start the ball rolling. > The Python Business Forum is very interested in this software, by the > way. Do you want to test it on the Snake Farm? > http://www.lysator.liu.se/~sfarmer/ Not sure what that buys me since there's nothing to actually build. Richard From amk@amk.ca Wed Oct 30 21:20:19 2002 From: amk@amk.ca (Andrew Kuchling) Date: Wed, 30 Oct 2002 16:20:19 -0500 Subject: [Catalog-sig] URLs for Python packages wanted Message-ID: I'm making yet another attempt at writing a Python catalog. This one will traverse a list of specified URLs and download and index any source packages that those URLs point to. (The idea being, if you have a directory that holds your code, you can simply drop new files into it and they'll be indexed To seed the initial database, I need the URLs of HTTP or FTP download directories. If you have a significant amount of Distutils-packaged software available, please mail me a URL for your collection. (For example, most of my code is available from http://www.amk.ca/files/python/ .) Thanks! --amk (www.amk.ca) OTHELLO: Put out the light, and then put out the light. -- _Othello_, V, ii From thomas.heller@ion-tof.com Thu Oct 31 09:16:45 2002 From: thomas.heller@ion-tof.com (Thomas Heller) Date: 31 Oct 2002 10:16:45 +0100 Subject: [Catalog-sig] Re: URLs for Python packages wanted References: Message-ID: Andrew Kuchling writes: > I'm making yet another attempt at writing a Python catalog. This one > will traverse a list of specified URLs and download and index any > source packages that those URLs point to. (The idea being, if you > have a directory that holds your code, you can simply drop new files > into it and they'll be indexed > If you need code for extracting metadata (not the complete metadata is included) and some other useful info from bdist_wininst created binary windows installer packages, you can look into the make_packagelist.py file in http://starship.python.net/crew/theller/pypan/packages/uploaded/pypan-0.2.zip. This is not a persitent URL. Thomas From thomas.heller@ion-tof.com Thu Oct 31 10:24:24 2002 From: thomas.heller@ion-tof.com (Thomas Heller) Date: 31 Oct 2002 11:24:24 +0100 Subject: [Catalog-sig] PEP 243 - Module Repository Upload Mechanism Message-ID: I've been playing with a PEP 243 implementation recently. To remind you, this is Sean Reifschneider's Module Repository Upload Mechanism PEP: http://www.python.org/peps/pep-0243.html . With some inspiration from his Swalow code, and some help from himself, I've played with a CGI script implenting this. Different from the PEP, I've not used his X-Swalow-XXX headers to return results to the uploader, instead the cgi-script returns a HTML page (if submitted from a web-browser), or plain text (if called programmatically). Additionally to the swalow code, this script accepts an OpenPGP compatible signature for the uploaded file, which I create with GnuPG. I've patched distutils to accept --submit and --sign options for the bdist_wininst and sdist commands, which will shell out to GnuPG to create a signature for the created distribution, and then upload the distribution to the server. The cgi script itself verifies the signature again a keyring on the server, and, if successful, moves the uploaded file to the public area. Here's the transcript of a sample session: C:\pypan>python setup.py sdist --submit --sign running sdist reading manifest file 'MANIFEST' creating pypan-0.2 creating pypan-0.2\PyPan copying files to pypan-0.2... copying README -> pypan-0.2 copying make_packagelist.py -> pypan-0.2 copying pypan.py -> pypan-0.2 copying setup.py -> pypan-0.2 copying PyPan\__init__.py -> pypan-0.2\PyPan copying PyPan\avail.py -> pypan-0.2\PyPan copying PyPan\install.py -> pypan-0.2\PyPan copying PyPan\installed.py -> pypan-0.2\PyPan copying PyPan\tarfile.py -> pypan-0.2\PyPan C:\Xilinx_ISE\bin\nt\zip.exe -rq dist\pypan-0.2.zip pypan-0.2 removing 'pypan-0.2' (and everything under it) You need a passphrase to unlock the secret key for user: "Thomas Heller " 1024-bit DSA key, ID B4985CBA, created 2002-10-24 File `dist\pypan-0.2.zip.asc' exists. Overwrite (y/N)? y created dist\pypan-0.2.zip.asc submitting dist\pypan-0.2.zip ######## uploader: saving as ./uploads/pypan-0.2.zip uploader: signature found uploader: Signature saved as ./uploads/pypan-0.2.zip.asc uploader: run 'gpg --batch --verify ./uploads/pypan-0.2.zip.asc ./uploads/pypan-0.2.zip 2>&1' gpg: Warning: using insecure memory! gpg: Signature made Thu Oct 31 04:48:23 2002 EST using DSA key ID B4985CBA gpg: Good signature from "Thomas Heller " gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. gpg: Fingerprint: DB49 621F C353 2ACF 4954 1DF3 5818 2B59 B498 5CBA uploader: seems to be a source archive file uploader: parsed ok uploader: final destination ./packages/uploaded/pypan-0.2.zip uploader: final destination ./packages/uploaded/pypan-0.2.zip.asc unknown filetype uploaded/pypan-0.2.zip.asc unknown filetype uploaded/pypan-0.2.win32.exe.asc unknown filetype packages.xml uploader: now 13 packages The lines following the '#######' is the text returned by the CGI script. There is a security concern which has to be addressed, I'm aware of that: when creating the signature you have to supply your passphrase to the running distutils sdist or bdist_wininst command. I don't think I like this. Comments? Thomas From martin@v.loewis.de Thu Oct 31 19:08:49 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 31 Oct 2002 20:08:49 +0100 Subject: [Catalog-sig] PEP 243 - Module Repository Upload Mechanism In-Reply-To: References: Message-ID: Thomas Heller writes: > There is a security concern which has to be addressed, I'm aware of > that: when creating the signature you have to supply your passphrase > to the running distutils sdist or bdist_wininst command. I don't think > I like this. Are you really supplying it to distutils? I'd expect that gpg could read it directly from the terminal... Regards, Martin From thomas.heller@ion-tof.com Thu Oct 31 19:34:33 2002 From: thomas.heller@ion-tof.com (Thomas Heller) Date: 31 Oct 2002 20:34:33 +0100 Subject: [Catalog-sig] PEP 243 - Module Repository Upload Mechanism In-Reply-To: References: Message-ID: martin@v.loewis.de (Martin v. Loewis) writes: > Thomas Heller writes: > > > There is a security concern which has to be addressed, I'm aware of > > that: when creating the signature you have to supply your passphrase > > to the running distutils sdist or bdist_wininst command. I don't think > > I like this. > > Are you really supplying it to distutils? I'd expect that gpg could > read it directly from the terminal... No, I'm not supplying it to distutils. Maybe I wasn't clear, here is the line of code: os.system("gpg --armor --output %s.asc --detach-sig %s" % (filename, filename)) and gpg reads it from the console. The user, however, runs python setup.py sdist --sign --submit and then is required to enter his passphrase. He might be concerned where it goes... Thomas From martin@v.loewis.de Thu Oct 31 20:28:11 2002 From: martin@v.loewis.de (Martin v. Loewis) Date: 31 Oct 2002 21:28:11 +0100 Subject: [Catalog-sig] PEP 243 - Module Repository Upload Mechanism In-Reply-To: References: Message-ID: Thomas Heller writes: > The user, however, runs > > python setup.py sdist --sign --submit > > and then is required to enter his passphrase. He might be concerned > where it goes... I think this is fine. The potential distutils users won't just type a password if some program asks for it. So the documentation needs to explain what the password is good for, and how precisely the processing works. Regards, Martin