From moshez@zadka.site.co.il Thu Nov 9 02:16:32 2000 From: moshez@zadka.site.co.il (Moshe Zadka) Date: Thu, 09 Nov 2000 04:16:32 +0200 Subject: [Catalog-sig] Strawman: Debian Message-ID: I don't know how many people here use Linux, and of those, I don't know how many people use Debian GNU/Linux. Debian, as a distribution has a couple of tools to manage distribution of several thousands packages with complex interdependancy. I suggest, as a strawman, that we outright steal these tools, and use them for managing Python modules. Let me summarize the procedures used in Debian to perform things as installations, upgrades and removals of software. The low-level is handled with dpkg -- Debian Packaging tool. It understands the format of a .deb file (the format in which Debian packages are distributed). It can check both the control information to see, e.g., that no higher version of that package is already installed, and that all dependancies are installed properly. Then it unpacks the distribution, and executes a configuration script. (For example, in the Python .deb, the configuration script byte-compiles all Python modules). As a way to facilitate installations, Debian also has a tool called apt (A Packaging Tool) which knows which packages exist in the Debian distribution, and what their connections are (conflicts, dependancies, recommendations, etc.) To impress everyone, I'll show an actual apt-get session: moshez@darjeeling:~/prog/my/pmh$ sudo apt-get install xterm Password: Reading Package Lists... Done Building Dependency Tree... Done The following extra packages will be installed: libxaw7 xfree86-common xlibs The following NEW packages will be installed: libxaw7 xfree86-common xlibs xterm 0 packages upgraded, 4 newly installed, 0 to remove and 0 not upgraded. Need to get 2233kB of archives. After unpacking 6984kB will be used. Do you want to continue? [Y/n] y Get:1 http://http.us.debian.org woody/main libxaw7 4.0.1-1 [202kB] ....gets other packages (my connection is slow, so I won't wait for it to finish)... ...installs them.... ...configures them.... Voila! I have "xterm" installed. The .debs are built from source packages with a little extra control information. If I want to change some .deb, I merely have to apt-get source This downloads the original package source, the Debian diff, opens the tarball and apply the diff. After I change the source a bit, I can issue simple Makefile commands and I have an alternative .deb. I'll usually change the version, so it won't get overridden by apt-get. Of course, this is only the beginning: apt-get is truly amazing. I think we can make this system (perhaps stealing only ideas, not code) work just as well for Python. -- Moshe Zadka This is a signature anti-virus. Please stop the spread of signature viruses! From moshez@zadka.site.co.il Thu Nov 9 04:53:30 2000 From: moshez@zadka.site.co.il (Moshe Zadka) Date: Thu, 09 Nov 2000 06:53:30 +0200 Subject: [Catalog-sig] Strawman: Debian Message-ID: I sent this before, when only AMK and I were subscribed. Here it is again: I don't know how many people here use Linux, and of those, I don't know how many people use Debian GNU/Linux. Debian, as a distribution has a couple of tools to manage distribution of several thousands packages with complex interdependancy. I suggest, as a strawman, that we outright steal these tools, and use them for managing Python modules. Let me summarize the procedures used in Debian to perform things as installations, upgrades and removals of software. The low-level is handled with dpkg -- Debian Packaging tool. It understands the format of a .deb file (the format in which Debian packages are distributed). It can check both the control information to see, e.g., that no higher version of that package is already installed, and that all dependancies are installed properly. Then it unpacks the distribution, and executes a configuration script. (For example, in the Python .deb, the configuration script byte-compiles all Python modules). As a way to facilitate installations, Debian also has a tool called apt (A Packaging Tool) which knows which packages exist in the Debian distribution, and what their connections are (conflicts, dependancies, recommendations, etc.) To impress everyone, I'll show an actual apt-get session: moshez@darjeeling:~/prog/my/pmh$ sudo apt-get install xterm Password: Reading Package Lists... Done Building Dependency Tree... Done The following extra packages will be installed: libxaw7 xfree86-common xlibs The following NEW packages will be installed: libxaw7 xfree86-common xlibs xterm 0 packages upgraded, 4 newly installed, 0 to remove and 0 not upgraded. Need to get 2233kB of archives. After unpacking 6984kB will be used. Do you want to continue? [Y/n] y Get:1 http://http.us.debian.org woody/main libxaw7 4.0.1-1 [202kB] ....gets other packages (my connection is slow, so I won't wait for it to finish)... ...installs them.... ...configures them.... Voila! I have "xterm" installed. The .debs are built from source packages with a little extra control information. If I want to change some .deb, I merely have to apt-get source This downloads the original package source, the Debian diff, opens the tarball and apply the diff. After I change the source a bit, I can issue simple Makefile commands and I have an alternative .deb. I'll usually change the version, so it won't get overridden by apt-get. Of course, this is only the beginning: apt-get is truly amazing. I think we can make this system (perhaps stealing only ideas, not code) work just as well for Python. -- Moshe Zadka This is a signature anti-virus. Please stop the spread of signature viruses! From akuchlin@mems-exchange.org Wed Nov 8 21:17:51 2000 From: akuchlin@mems-exchange.org (Andrew Kuchling) Date: Wed, 08 Nov 2000 16:17:51 -0500 Subject: [Catalog-sig] Sources of inspiration Message-ID: Other systems to look at: Debian: apt (see Moshe's post) FreeBSD: ports system RPMfind: www.rpmfind.net XEmacs: the 'packages' system Perl: CPAN.pm Perl: OSD/PPM OSD was pointed out to me by David Ascher, and stands for Open Software Distribution (OSD). See www.w3.org/tr/note-osd.html, www.microsoft.com/workshop/management/osd/osdfaq.asp. "OSD is an XML vocabulary for describing a software distribution package as a "bill of materials" and defining relationships and dependencies among its parts. OSD-based software distribution implementers include Marimba Inc.'s Castanet, Microsoft's Internet Explorer, Novell Inc.'s ZENworks and ActiveState Tool Corp.'s Win32 Perl." --amk From akuchlin@mems-exchange.org Wed Nov 8 21:18:25 2000 From: akuchlin@mems-exchange.org (Andrew Kuchling) Date: Wed, 08 Nov 2000 16:18:25 -0500 Subject: [Catalog-sig] Initial requirements Message-ID: [This is an edited version of a posting I made to the Distutils-SIG some months ago.] There are a few steps needed to find and install a package: 1) Discovery : which module does what I need? 2) Download : where can I get a copy? 3) Security : is this actually from the package author, and not a Trojan horse? 4) Installation : how do I set it up? Distutils focuses on the hardest and most complicated step, #4. Requirements from #1) 1.1) Users can browse through a list of available packages, browse through a hierarchy like Parnassus, or do keyword searches. 1.2) Packages are also indexed by author and date, so you can see all of Moshe's packages, or the 10 most recently changed packages. 1.3) For each author, their name, e-mail and home page URL are stored. 1.4) Information about a package is only extracted from the metadata included in the package's setup.py file. This ensures that the info is up-to-date, saves users the effort of entering data, and encourages people to use the Distutils. Requirements from #2) 2.1) Given a package name, software can retrieve one or more download URLs for the package. 2.2) The catalog should keep copies of the code, not just point to remote sites, in order to prevent disasters 2.3) It should be possible to mirror the catalog without too much trouble, using a conventional FTP or HTTP mirroring script. Requirements from #3) 3.1) Users can check a signature on the downloaded package, using an external tool such as GnuPG. 3.2) Checking the signature is optional, and can be skipped if the external tool isn't available. (We could implement our own signature scheme with Python code, but that's a bad idea; security is hard, and few people will bother to generate keys that are only useful only for distributing Python modules. Related requirement: the Distutils sdist and bdist_* commands should have a --sign switch to sign the generated .tgz, .rpm, or whatever file. Requirements from #4) None Not covered in these initial requirements: checking for updated versions, tracking dependencies. These seem to be difficult problems that require a database of Python packages installed on a system. --amk From robin@alldunn.com Wed Nov 8 21:38:11 2000 From: robin@alldunn.com (Robin Dunn) Date: Wed, 8 Nov 2000 13:38:11 -0800 Subject: [Catalog-sig] Initial requirements References: Message-ID: <015b01c049cc$31bdeb20$3225d2d1@ARES> > > Not covered in these initial requirements: checking for updated > versions, tracking dependencies. These seem to be difficult problems > that require a database of Python packages installed on a system. > As long as it's some client-side tool (command line or GUI) doing #1 and #2 and not just some web-based interface, and as long as all the package meta-data is easily fetchable, then the tool should be able to manage this without too much problem. It can keep track of what it has installed and compare with the meta-data for currently available packages. -- Robin Dunn Software Craftsman robin@AllDunn.com http://wxPython.org Java give you jitters? http://wxPROs.com Relax with wxPython! From akuchlin@mems-exchange.org Wed Nov 8 21:45:31 2000 From: akuchlin@mems-exchange.org (Andrew Kuchling) Date: Wed, 8 Nov 2000 16:45:31 -0500 Subject: [Catalog-sig] Initial requirements In-Reply-To: <015b01c049cc$31bdeb20$3225d2d1@ARES>; from robin@alldunn.com on Wed, Nov 08, 2000 at 01:38:11PM -0800 References: <015b01c049cc$31bdeb20$3225d2d1@ARES> Message-ID: <20001108164531.A27952@kronos.cnri.reston.va.us> On Wed, Nov 08, 2000 at 01:38:11PM -0800, Robin Dunn wrote: >without too much problem. It can keep track of what it has installed and >compare with the meta-data for currently available packages. What if the user installs packages w/o using any of the tools, by downloading it and running "python setup.py install"? You'd really need a Python-wide database of installed packages; at the very least this is a question for the Distutils. --amk From robin@alldunn.com Wed Nov 8 23:54:14 2000 From: robin@alldunn.com (Robin Dunn) Date: Wed, 8 Nov 2000 15:54:14 -0800 Subject: [Catalog-sig] Initial requirements References: <015b01c049cc$31bdeb20$3225d2d1@ARES> <20001108164531.A27952@kronos.cnri.reston.va.us> Message-ID: <01a001c049df$33418340$3225d2d1@ARES> > > What if the user installs packages w/o using any of the tools, by > downloading it and running "python setup.py install"? Then you slap the user's hand and tell 'em not to do that! > You'd really > need a Python-wide database of installed packages; at the very least > this is a question for the Distutils. I agree distutils should update the DB when packages are installed, but should it be the master of it? If so then I would think that it should also be the one that checks dependencies, fetches dependent packages, etc., which I think is overloading distutils responsibilities too much, and maybe limits the ability of other tools from interacting with the data and also being a package manager. Also, the current entry point into distutils is a package's setup.py file which wouldn't make much sense for a generic package manager. So there is going to have to be a separate utility to front-end distutils anyway so why not keep it all separate so that we can end up with different ones for different needs. For example, a command-line version**, a tkinter version, a wx version, one integrated with IDLE, Boa, PythonWin, or PythonWorks, etc... Here is what I was thinking of when I spent a few brain cycles on this a couple weeks ago: 1. There should be a module (or package)** that is part of the standard distribution that manages access to a local DB (could be as simple as a shelve of pickled objects, or maybe an XML file) that contains the meta-data about the installed packages. The module should handle reading the DB into a collection of objects that can be read by the various package manager tools, adding new meta-data objects to the DB, deleting ones that have been uninstalled, locking the DB file, etc. 2. The distutils install command should (by default) use that module to update the DB with the installed package's info, refuse to install if there are dependencies missing, etc. There should be flags to disable both of these in case the package tools want to update the DB themselves or whatever. 3. The various package tools also use the module to access/update the DB, but go the next step and know how to fetch the list of archive mirrors, fetch the master package list from an archive, fetch a package and dependencies from an archive, install and uninstall packages, etc. Much of this could be factored out into the standard library module/package to go along with the DB access, but it should all be designed to be extended by the package tool so they have the freedom to customize behaviour (especially the IDE integrations) and add whiz-bangs where appropriate. 4. While I agree that everybody should move to using distutils the simple fact is that it's either going to take a long time to get there, or that not everybody will do it. For this I think it would be nice to allow the package tools and the archives to deal with non-distutils based packages, perhaps simply by adding to the meta-data the name of a python file in the distribution to execute to do the install. This could default to "setup.py install" or something... 5. The package meta-data should also include information about pre-built binary distributions for various platforms. The package tools can then fetch the proper binary if available and install it (either un[g]zip it in the right place, or execute an .exe, run rpm or whatever) ** As long as they are not tightly integrated with distutils perhaps the command-line package tool and the library module could be distributed as part of distutils for convenience. Disclaimer: All this is just my rambling thoughts on the subject and I've already changed my opinions a couple times since starting to write it... ;-) There will probably be a better solution that everybody likes better, but maybe some of these ideas will help out too. -- Robin Dunn Software Craftsman robin@AllDunn.com http://wxPython.org Java give you jitters? http://wxPROs.com Relax with wxPython! From guido@python.org Thu Nov 9 02:19:00 2000 From: guido@python.org (Guido van Rossum) Date: Wed, 08 Nov 2000 21:19:00 -0500 Subject: [Catalog-sig] Initial requirements In-Reply-To: Your message of "Wed, 08 Nov 2000 15:54:14 PST." <01a001c049df$33418340$3225d2d1@ARES> References: <015b01c049cc$31bdeb20$3225d2d1@ARES> <20001108164531.A27952@kronos.cnri.reston.va.us> <01a001c049df$33418340$3225d2d1@ARES> Message-ID: <200011090219.VAA25330@cj20424-a.reston1.va.home.com> > 4. While I agree that everybody should move to using distutils the simple > fact is that it's either going to take a long time to get there, or that not > everybody will do it. For this I think it would be nice to allow the > package tools and the archives to deal with non-distutils based packages, > perhaps simply by adding to the meta-data the name of a python file in the > distribution to execute to do the install. This could default to "setup.py > install" or something... I think it's okay if a 3rd party creates a distutil'ed version of such packages. After all this is exactly what the Debian and Red Hat package managers do. --Guido van Rossum (home page: http://www.python.org/~guido/) From robin@alldunn.com Thu Nov 9 02:43:26 2000 From: robin@alldunn.com (Robin Dunn) Date: Wed, 8 Nov 2000 18:43:26 -0800 Subject: [Catalog-sig] Initial requirements References: <015b01c049cc$31bdeb20$3225d2d1@ARES> <20001108164531.A27952@kronos.cnri.reston.va.us> <01a001c049df$33418340$3225d2d1@ARES> <200011090219.VAA25330@cj20424-a.reston1.va.home.com> Message-ID: <003c01c049f6$d6779ec0$0b01a8c0@ARES> > > I think it's okay if a 3rd party creates a distutil'ed version of such > packages. After all this is exactly what the Debian and Red Hat > package managers do. Good point. -- Robin Dunn Software Craftsman robin@AllDunn.com http://wxPython.org Java give you jitters? http://wxPROs.com Relax with wxPython! From calvin@cs.uni-sb.de Thu Nov 9 21:46:37 2000 From: calvin@cs.uni-sb.de (Bastian Kleineidam) Date: Thu, 9 Nov 2000 22:46:37 +0100 (CET) Subject: [Catalog-sig] DB format and ChangeLog In-Reply-To: Message-ID: Hi all, I have two comments: - make a readable DB format, no binary pickling solutions. Debian uses a rfc822 DB with "name: value" entries. - when I am installing new or updated versions of a package I am always curious what has changed since the last version (bug fixes, new features etc.). If the changes are important enough, I want to download and install the new version. If not, then I wait for the next release. So there should be an option to download/view only the changelog. Bastian Kleineidam From akuchlin@mems-exchange.org Thu Nov 9 22:10:48 2000 From: akuchlin@mems-exchange.org (Andrew Kuchling) Date: Thu, 9 Nov 2000 17:10:48 -0500 Subject: [Catalog-sig] Re: OSD (was: Too much good stuff) In-Reply-To: ; from mwa@gate.net on Thu, Nov 09, 2000 at 02:26:47PM -0500 References: <20001109170158.404361D18F@dinsdale.python.org> Message-ID: <20001109171048.B16625@kronos.cnri.reston.va.us> On Thu, Nov 09, 2000 at 02:26:47PM -0500, Mark W. Alexander wrote: >I looked at that some time ago. Despite Microsoft's involvement, I >have to say I think it's an outstanding idea to have the python >catalogs be based on a w3c standard. OSD is not a W3C standard; it's a W3C Note dated 1997, which means it's purely informational and not any sort of standard. Anyone can write something up and submit it as a W3C Note. I finally read the OSD spec yesterday, and wasn't very impressed. Packages are nodes, connected by arcs representing the dependencies. It's dressed up with lots of XML fanciness, but it doesn't provide anything very sophisticated. About the only other users of OSD seem to be Marimba, and ActiveState's Perl Package Manager format. Compatibility with the latter might be a good reason to use OSD. --amk From akuchlin@mems-exchange.org Thu Nov 9 22:23:54 2000 From: akuchlin@mems-exchange.org (Andrew Kuchling) Date: Thu, 9 Nov 2000 17:23:54 -0500 Subject: [Catalog-sig] Too much good stuff In-Reply-To: ; from mwa@gate.net on Thu, Nov 09, 2000 at 02:26:47PM -0500 References: <20001109170158.404361D18F@dinsdale.python.org> Message-ID: <20001109172354.C16625@kronos.cnri.reston.va.us> On Thu, Nov 09, 2000 at 02:26:47PM -0500, Mark W. Alexander wrote: >What about external (non-python) dependencies that would not be in the >DB? This would include pretty much any shared library that a wrapper >module needs. Do we carry enough meta-data to cover all cascading >dependencies? If so, how do we verify them? Search the system? >Query the native package manager? (My head's starting to hurt....) This is why I'm strongly against worrying about tracking dependencies right now: the issue is complicated and messy to solve. A solution that only handles storing and finding packages is much easier to do, and already much better than what currently exists. Let's finish listing requirements for 1 through 3 and design something first, leaving dependencies for later. --amk From guido@python.org Thu Nov 9 22:46:50 2000 From: guido@python.org (Guido van Rossum) Date: Thu, 09 Nov 2000 17:46:50 -0500 Subject: [Catalog-sig] Too much good stuff In-Reply-To: Your message of "Thu, 09 Nov 2000 17:23:54 EST." <20001109172354.C16625@kronos.cnri.reston.va.us> References: <20001109170158.404361D18F@dinsdale.python.org> <20001109172354.C16625@kronos.cnri.reston.va.us> Message-ID: <200011092246.RAA32622@cj20424-a.reston1.va.home.com> > On Thu, Nov 09, 2000 at 02:26:47PM -0500, Mark W. Alexander wrote: > >What about external (non-python) dependencies that would not be in the > >DB? This would include pretty much any shared library that a wrapper > >module needs. Do we carry enough meta-data to cover all cascading > >dependencies? If so, how do we verify them? Search the system? > >Query the native package manager? (My head's starting to hurt....) > > This is why I'm strongly against worrying about tracking dependencies > right now: the issue is complicated and messy to solve. A solution > that only handles storing and finding packages is much easier to do, > and already much better than what currently exists. Let's finish > listing requirements for 1 through 3 and design something first, > leaving dependencies for later. Good point. Version and dependency tracking tend to be flamebait. The distutils sig almost went up in flames over a discussion on how to define version numbers. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From akuchlin@mems-exchange.org Thu Nov 9 23:03:59 2000 From: akuchlin@mems-exchange.org (Andrew Kuchling) Date: Thu, 09 Nov 2000 18:03:59 -0500 Subject: [Catalog-sig] New Web pages Message-ID: I've added the requirements document (slightly modified) and a list of links to other cataloging systems to the SIG's pages at www.python.org/sigs/catalog-sig/ . Suggestions, additional links, &c. are welcome... --amk From Juergen Hermann" Message-ID: On Thu, 09 Nov 2000 17:46:50 -0500, Guido van Rossum wrote: >Good point. Version and dependency tracking tend to be flamebait. >The distutils sig almost went up in flames over a discussion on how to >define version numbers. :-) Don't track or resolve it automatically, but enable us to yell. It would be nice though if a package could print an error msg after querying a central version database for a minimal needed version of a utility package (i.e. create a version query API). Bye, J=FCrgen From moshez@zadka.site.co.il Fri Nov 10 08:23:10 2000 From: moshez@zadka.site.co.il (Moshe Zadka) Date: Fri, 10 Nov 2000 10:23:10 +0200 Subject: [Catalog-sig] Too much good stuff In-Reply-To: Message from "Mark W. Alexander" of "Thu, 09 Nov 2000 14:26:47 EST." References: Message-ID: > > Message: 1 > > From: Moshe Zadka > > I don't know how many people here use Linux, and of those, I don't > > know how many people use Debian GNU/Linux. Debian, as a distribution > > has a couple of tools to manage distribution of several thousands > > packages with complex interdependancy. I suggest, as a strawman, > > that we outright steal these tools, and use them for managing Python > > modules. > > I like the idea, but I don't think the implementation is universal > enough. Huh? Why don't you think that we can port apt-get/dpkg into anything under the sun? (pun intended). It's just C code. > I've got Linux desktops (multiple distributions) and HP > and Solaris servers. So? Nobody said apt-get/dpkg need to work only on Debian -- you just need to change the place where they place the status file. -- Moshe Zadka This is a signature anti-virus. Please stop the spread of signature viruses! From akuchlin@mems-exchange.org Thu Nov 16 15:18:07 2000 From: akuchlin@mems-exchange.org (Andrew Kuchling) Date: Thu, 16 Nov 2000 10:18:07 -0500 Subject: [Catalog-sig] First cut at metadata Message-ID: Here's a proposed list of metadata to track in the first version of a catalog. All of these fields are fairly straightforward in semantics, and don't drag us into tangled issues. Anything simple that's missing from this list? --amk Information about an author =========================== Name Home page GnuPG/PGP key (both the actual key, and just the ID) E-mail address (used as ID?) E-mail address public: y/n Information about a package =========================== Name Version Supported Platforms Description Keywords Homepage URL Author IDs License Download link Date of release Left out for now: Dependency information Information about a document ============================ Name Author Description URL of HTML version URL of printable version (Either URL can be omitted if not applicable.) From akuchlin@mems-exchange.org Thu Nov 16 15:45:45 2000 From: akuchlin@mems-exchange.org (Andrew Kuchling) Date: Thu, 16 Nov 2000 10:45:45 -0500 Subject: [Catalog-sig] UI for managing catalog Message-ID: What types of interfaces do we need to support for managing the catalog. A Web-based version, through CGI, is clearly required. Do we want to require FTP? I'm inclined to say "no", since we want to make it easy to run mirrors, and it's easier to run just a Web server than a Web server + FTP server. On the other hand, it's harder to mirror Web pages; we'll probably need to provide an index page that points to all the files. The scripts underlying this system will run on a single host, generating static pages that will get copied by mirroring sites. Here's a proposal, that hews very closely to the CPAN model. Users register themselves as developers, and get an ID. For elegance, we'll try to encourage re-using developer IDs from other registration schemes: CPAN, Debian, SourceForge, etc. Developers then have two options: 1) Upload their distribution through a CGI form, in either .zip or .tgz format. A script will extract the setup.py file to get the metadata; users could also enter all this manually, but keeping an accurate setup.py is encouraged. If the developer has a public key, a signature for the distribution will be checked as a protective measure against hacked distributions. 2) Submit a URL from which the distribution can be downloaded. A script will then fetch it, extract setup.py, and everything follows as before. We can then run automated scripts which check for updated versions of software and automatically update the catalog. I don't understand how WebDAV would fit into this, and therefore it should be left out of the first version. --amk From guido@python.org Thu Nov 16 16:48:24 2000 From: guido@python.org (Guido van Rossum) Date: Thu, 16 Nov 2000 11:48:24 -0500 Subject: [Catalog-sig] First cut at metadata In-Reply-To: Your message of "Thu, 16 Nov 2000 10:18:07 EST." References: Message-ID: <200011161648.LAA10426@cj20424-a.reston1.va.home.com> Looking good. Some nits: > Information about an author > =========================== > Name > Home page > GnuPG/PGP key (both the actual key, and just the ID) > E-mail address (used as ID?) > E-mail address public: y/n On the one hand I like using the email address as ID, because it avoids having to remember my username for a site I don't visit very often. On the other hand it may become a pain to change -- users change their email address all the time and we must support this. > Information about a package > =========================== > Name > Version > Supported Platforms Should we give a fixed set of choices, or a free form field? The problem with a fixed set of choices is that there are lots of subtle variations, e.g. do you present Windows as a choice or Win95, Win98, WinNT, Win2000, WinME. Ditto Unix or Linux or Red Hat Linux or Red Hat Linux 6.1? The problem with free form is that it's hard to use in a search (e.g. for many people Unix and Linux are synomyms). > Description > Keywords > Homepage URL > Author IDs > License I recommend to use the categories offered by sourceforge here. > Download link > Date of release > > Left out for now: Dependency information Submittors could be urged to add this to the Description text if there are special requirements. > Information about a document > ============================ > Name > Author > Description > URL of HTML version > URL of printable version > (Either URL can be omitted if not applicable.) What about downloadable versions? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Thu Nov 16 16:51:53 2000 From: guido@python.org (Guido van Rossum) Date: Thu, 16 Nov 2000 11:51:53 -0500 Subject: [Catalog-sig] UI for managing catalog In-Reply-To: Your message of "Thu, 16 Nov 2000 10:45:45 EST." References: Message-ID: <200011161651.LAA10447@cj20424-a.reston1.va.home.com> > 2) Submit a URL from which the distribution can be downloaded. A > script will then fetch it, extract setup.py, and everything follows as > before. Is this added complexity (for us) worth having? --Guido van Rossum (home page: http://www.python.org/~guido/) From bkc@murkworks.com Thu Nov 16 17:39:38 2000 From: bkc@murkworks.com (Brad Clements) Date: Thu, 16 Nov 2000 12:39:38 -0500 Subject: [Catalog-sig] First cut at metadata In-Reply-To: <20001116170200.D6A1A1D26F@dinsdale.python.org> Message-ID: <3A13D1E6.21393.FFD1077@localhost> > > Supported Platforms > > Should we give a fixed set of choices, or a free form field? The > problem with a fixed set of choices is that there are lots of subtle > variations, e.g. do you present Windows as a choice or Win95, Win98, > WinNT, Win2000, WinME. Ditto Unix or Linux or Red Hat Linux or Red > Hat Linux 6.1? The problem with free form is that it's hard to use in > a search (e.g. for many people Unix and Linux are synomyms). I think we should have a fixed set of choices. When we get to the point of having a platform specific-websearch, it will make life a lot easier. Then there's also the issue of binary vs. source for those packages that have C code. Can we take any design guidance from RPM as well as CPAN? Although we don't need to get that sophisticated to start with, I think its worthwhile having the basic information that could support that in the future. So, for example on dependencies, couldn't a package just point to other packages in the system by "id". And have an option for "external package" with some kind of human readable text field that describes it. This way if we ever get automated download of packages with depends, most things will work automatically unless you download a package with an "external" dependency. At which point human intervention is required. Also, if you get a dependant package that doesn't support your platform, it won't download either. Brad Clements, bkc@murkworks.com (315)268-1000 http://www.murkworks.com (315)268-9812 Fax netmeeting: ils://ils.murkworks.com AOL-IM: BKClements From amos@digicool.com Thu Nov 16 18:36:32 2000 From: amos@digicool.com (Amos Latteier) Date: Thu, 16 Nov 2000 10:36:32 -0800 Subject: [Catalog-sig] UI for managing catalog References: <200011161651.LAA10447@cj20424-a.reston1.va.home.com> Message-ID: <3A142930.7078C00D@digicool.com> Guido van Rossum wrote: > > > 2) Submit a URL from which the distribution can be downloaded. A > > script will then fetch it, extract setup.py, and everything follows as > > before. > > Is this added complexity (for us) worth having? I think so. In fact I already have rough code to do exactly this. It downloads a distutils package, extracts the setup.py and parses it. I'd like to see the distutils meta-data in setup.py used as much as possible. If simply packing your stuff with distutils gets you into the catalog easily, then I think this will motivate folks to use it. In the long run, I'd like to distutils packing tools to be able to contact the catalog and upload the package. So for example you could use a distutils command like this python setup.py catalog_dist To build and submit a distribution to the catalog. -Amos -- Amos Latteier mailto:amos@digicool.com Digital Creations http://www.digicool.com From akuchlin@mems-exchange.org Thu Nov 16 19:28:37 2000 From: akuchlin@mems-exchange.org (Andrew Kuchling) Date: Thu, 16 Nov 2000 14:28:37 -0500 Subject: [Catalog-sig] UI for managing catalog In-Reply-To: <3A142930.7078C00D@digicool.com>; from amos@digicool.com on Thu, Nov 16, 2000 at 10:36:32AM -0800 References: <200011161651.LAA10447@cj20424-a.reston1.va.home.com> <3A142930.7078C00D@digicool.com> Message-ID: <20001116142837.B24422@kronos.cnri.reston.va.us> On Thu, Nov 16, 2000 at 10:36:32AM -0800, Amos Latteier wrote: >Guido van Rossum wrote: >> Is this added complexity (for us) worth having? >I think so. In fact I already have rough code to do exactly this. It >downloads a distutils package, extracts the setup.py and parses it. And the hard part of this is processing the setup.py safely, not the downloading, which is just a matter of using urllib.urlopen(). Amos, is this essentially the same code as discussed in the distutils-sig some months, or have you made lots of changes to it? > python setup.py catalog_dist Yes! And this is an example of the catalog producing a change in the Distutils and hence in the standard library... though if it misses the deadline for 2.1, no big deal. --amk From amos@digicool.com Thu Nov 16 19:26:43 2000 From: amos@digicool.com (Amos Latteier) Date: Thu, 16 Nov 2000 11:26:43 -0800 Subject: [Catalog-sig] UI for managing catalog References: <200011161651.LAA10447@cj20424-a.reston1.va.home.com> <3A142930.7078C00D@digicool.com> <20001116142837.B24422@kronos.cnri.reston.va.us> Message-ID: <3A1434F3.77D776D9@digicool.com> Andrew Kuchling wrote: > > On Thu, Nov 16, 2000 at 10:36:32AM -0800, Amos Latteier wrote: > >I think so. In fact I already have rough code to do exactly this. It > >downloads a distutils package, extracts the setup.py and parses it. > > And the hard part of this is processing the setup.py safely, not the > downloading, which is just a matter of using urllib.urlopen(). Yes, it's the same old code. It is not safe. I had problems getting Rexec work here, and punted on it. This is a problem, regardless of whether the catalog fetches the distribution, or someone uploads it. In either case the catalog must run potentially untrusted code to extract meta-data. -Amos -- Amos Latteier mailto:amos@digicool.com Digital Creations http://www.digicool.com From martin@loewis.home.cs.tu-berlin.de Thu Nov 16 19:43:03 2000 From: martin@loewis.home.cs.tu-berlin.de (Martin v. Loewis) Date: Thu, 16 Nov 2000 20:43:03 +0100 Subject: [Catalog-sig] UI for managing catalog In-Reply-To: <200011161651.LAA10447@cj20424-a.reston1.va.home.com> (message from Guido van Rossum on Thu, 16 Nov 2000 11:51:53 -0500) References: <200011161651.LAA10447@cj20424-a.reston1.va.home.com> Message-ID: <200011161943.UAA00873@loewis.home.cs.tu-berlin.de> > > 2) Submit a URL from which the distribution can be downloaded. A > > script will then fetch it, extract setup.py, and everything follows as > > before. > > Is this added complexity (for us) worth having? I'd think so, yes. Developers are often proud of their software, so they want it to be located on their home page. They will go through the effort of registering it with the catalog, but they are less inclined to place the distribution into multiple locations - especially when internet bandwidth gets in the way. I know that I was cursing internet bandwidth/SF/US business hours when I was trying to upload PyXML 0.6.2 this week. It took ages to get it onto SF, when I really wanted to go home. I had preferred to put it on a local Web server, and SF fetch it for me. Regards, Martin From martin@loewis.home.cs.tu-berlin.de Thu Nov 16 19:48:03 2000 From: martin@loewis.home.cs.tu-berlin.de (Martin v. Loewis) Date: Thu, 16 Nov 2000 20:48:03 +0100 Subject: [Catalog-sig] UI for managing catalog In-Reply-To: <3A142930.7078C00D@digicool.com> (message from Amos Latteier on Thu, 16 Nov 2000 10:36:32 -0800) References: <200011161651.LAA10447@cj20424-a.reston1.va.home.com> <3A142930.7078C00D@digicool.com> Message-ID: <200011161948.UAA00917@loewis.home.cs.tu-berlin.de> > In the long run, I'd like to distutils packing tools to be able to > contact the catalog and upload the package. That would be nice and useful, indeed. Regards, Martin From moshez@zadka.site.co.il Fri Nov 17 14:20:18 2000 From: moshez@zadka.site.co.il (Moshe Zadka) Date: Fri, 17 Nov 2000 16:20:18 +0200 Subject: [Catalog-sig] UI for managing catalog In-Reply-To: Message from Andrew Kuchling of "Thu, 16 Nov 2000 10:45:45 EST." References: Message-ID: [Andrew] > What types of interfaces do we need to support for managing the > catalog. A Web-based version, through CGI, is clearly required. Of course. By CGI, you probably mean "server side code" rather then really CGI. CGI is an evil protocol, and will die soon anyway. Why not use a real tool, like Zope? [Andrew] > Do we want to require FTP? No. FTP is an evil protocol, and will die soon anyway. [Andrew] > ...it's harder to > mirror Web pages; Not if you have WebDAV. Note that you're talking about two different things (we might decide to require them both on each CPyAN server, we might not): 1. Packages mirror. 2. Searching interface. These do not have to be necessarily related. An easy way to do it is to have two cookies: "Preferred-Packages:" and "Preferred-Searching:". The master searching server will redirect to the correct Preferred-Searching server, which will then use the "Preferred-Packages" cookie to construct URLs for the correct Packages mirror. We can talk a lot about ways to do it, but I am pretty much sure it's doable. (I'm a web developer by profession, so I do this kind of things for a living. You can trust me.) One advantage would be that there can be fewer searching interfaces (which is not as high a load on a server) then packages mirror. More, packages mirror can use either FTP or WebDAV and the searching interface won't care a bit. That's talking only about the user side of things -- not the developer. Note, BTW, that there can be other ways to access a web page then via a web-browser. Especially if we subscribe to a very rigid set of queries, it can be done by a Python module. [Andrew] > I don't understand how WebDAV would fit into this, and therefore it > should be left out of the first version. No! We don't have to use all of WebDAV, but one thing we could use is that in WebDAV, you *can* get a list of all files in a directory. That's critical for mirroring. -- Moshe Zadka This is a signature anti-virus. Please stop the spread of signature viruses! From moshez@zadka.site.co.il Fri Nov 17 14:23:21 2000 From: moshez@zadka.site.co.il (Moshe Zadka) Date: Fri, 17 Nov 2000 16:23:21 +0200 Subject: [Catalog-sig] UI for managing catalog In-Reply-To: Message from Guido van Rossum of "Thu, 16 Nov 2000 11:51:53 EST." <200011161651.LAA10447@cj20424-a.reston1.va.home.com> References: <200011161651.LAA10447@cj20424-a.reston1.va.home.com> Message-ID: > > 2) Submit a URL from which the distribution can be downloaded. A > > script will then fetch it, extract setup.py, and everything follows as > > before. > > Is this added complexity (for us) worth having? Well, it can be left out for the first version. I'm sure it's needed in the long run: it's much easier on people (like me) whose home computer is connected at low speed. I upload it anyway to my website, and I'd rather CPyAN would download it without me having to sacrifice my limited bandwidth for that. -- Moshe Zadka This is a signature anti-virus. Please stop the spread of signature viruses! From martin@loewis.home.cs.tu-berlin.de Thu Nov 16 19:56:55 2000 From: martin@loewis.home.cs.tu-berlin.de (Martin v. Loewis) Date: Thu, 16 Nov 2000 20:56:55 +0100 Subject: [Catalog-sig] UI for managing catalog In-Reply-To: <3A1434F3.77D776D9@digicool.com> (message from Amos Latteier on Thu, 16 Nov 2000 11:26:43 -0800) References: <200011161651.LAA10447@cj20424-a.reston1.va.home.com> <3A142930.7078C00D@digicool.com> <20001116142837.B24422@kronos.cnri.reston.va.us> <3A1434F3.77D776D9@digicool.com> Message-ID: <200011161956.UAA01036@loewis.home.cs.tu-berlin.de> > Yes, it's the same old code. It is not safe. I had problems getting > Rexec work here, and punted on it. Can you elaborate on the problems, please? Is there something that intrinsically can't work, or did you just not get through with it? Regards, Martin From martin@loewis.home.cs.tu-berlin.de Fri Nov 17 10:21:28 2000 From: martin@loewis.home.cs.tu-berlin.de (Martin v. Loewis) Date: Fri, 17 Nov 2000 11:21:28 +0100 Subject: [Catalog-sig] UI for managing catalog In-Reply-To: (message from Moshe Zadka on Fri, 17 Nov 2000 16:20:18 +0200) References:

Message-ID: <200011171021.LAA00881@loewis.home.cs.tu-berlin.de> > Of course. By CGI, you probably mean "server side code" rather then > really CGI. CGI is an evil protocol, and will die soon anyway. Why > not use a real tool, like Zope? [...] > No. FTP is an evil protocol, and will die soon anyway. You have quite strong opinions on harmless software specifications :-) I don't think the server-side implementation approach matters at the moment; I'd rather not make Zope a prerequisite. As for ftp, I personally consider it much more powerful than http for file transfer. For example, it has a standard index facility (DIR), which http doesn't - it is quite a pain to analyse an index.html automatically. > No! We don't have to use all of WebDAV, but one thing we could use is > that in WebDAV, you *can* get a list of all files in a directory. That's > critical for mirroring. Ah, mirroring - that's another reason not to mandate Zope, and another area where ftp does much better than anything else :-) Regards, Martin From moshez@zadka.site.co.il Fri Nov 17 18:49:30 2000 From: moshez@zadka.site.co.il (Moshe Zadka) Date: Fri, 17 Nov 2000 20:49:30 +0200 Subject: [Catalog-sig] UI for managing catalog In-Reply-To: Message from "Martin v. Loewis" of "Fri, 17 Nov 2000 11:21:28 +0100." <200011171021.LAA00881@loewis.home.cs.tu-berlin.de> References:

<200011171021.LAA00881@loewis.home.cs.tu-berlin.de> Message-ID: [Moshe] > No. FTP is an evil protocol, and will die soon anyway. [Martin v. Loewis] > You have quite strong opinions on harmless software specifications :-) Ummmm....harmless? FTP mixes OSI levels. Where I come from, it's a shooting offense. Frequently, the trial can be skipped. [Martin] > I don't think the server-side implementation approach matters at the > moment; I'd rather not make Zope a prerequisite. Did you look at the part where I dissociate searching and grabbing packages? [Martin] > As for ftp, I personally consider it much more powerful than http for > file transfer. For example, it has a standard index facility (DIR), > which http doesn't WebDAV has. [Martin] > - it is quite a pain to analyse an index.html > automatically. Luckily, we're not living in the '90s anymore. We have WebDAV. Why not use it, instead of a brain dead protocol which cannot be made secure, and will die an anguished death when IPv6 takes over the world? And anyway, my proposed scheme allows for FTP mirrors (as opposed to meta-data searchers) as well. -- Moshe Zadka This is a signature anti-virus. Please stop the spread of signature viruses! From akuchlin@mems-exchange.org Fri Nov 17 15:39:27 2000 From: akuchlin@mems-exchange.org (Andrew Kuchling) Date: Fri, 17 Nov 2000 10:39:27 -0500 Subject: [Catalog-sig] UI for managing catalog In-Reply-To: ; from moshez@zadka.site.co.il on Fri, Nov 17, 2000 at 04:20:18PM +0200 References:

Message-ID: <20001117103927.B25735@kronos.cnri.reston.va.us> On Fri, Nov 17, 2000 at 04:20:18PM +0200, Moshe Zadka wrote: >[Andrew] >Of course. By CGI, you probably mean "server side code" rather then >really CGI. CGI is an evil protocol, and will die soon anyway. Why >not use a real tool, like Zope? We do not want to require people to install the massive bulk of Zope. Ideally I'd like running a package-only mirror to simply require running wget or some similar tool; a searchable mirror should simply require the additional step of setting up a CGI script and mirroring the additional index file(s). --amk From jeremy@alum.mit.edu Fri Nov 17 04:29:11 2000 From: jeremy@alum.mit.edu (Jeremy Hylton) Date: Thu, 16 Nov 2000 23:29:11 -0500 (EST) Subject: [Catalog-sig] UI for managing catalog In-Reply-To: References:

<200011171021.LAA00881@loewis.home.cs.tu-berlin.de> Message-ID: <14868.46103.298676.649511@bitdiddle.concentric.net> >>>>> "MZ" == Moshe Zadka writes: MZ> [Moshe] >> No. FTP is an evil protocol, and will die soon anyway. MZ> [Martin v. Loewis] >> You have quite strong opinions on harmless software >> specifications :-) MZ> Ummmm....harmless? FTP mixes OSI levels. Where I come from, it's MZ> a shooting offense. Frequently, the trial can be skipped. This conversation would be a lot more productive if we could avoid useless discussions like this. (I realize it's hard, because I haven't been able to avoid saying: The OSI model is useless.) Calling a protocol "evil," does not help us make progress on the catalog. Jeremy From moshez@zadka.site.co.il Sat Nov 18 00:45:38 2000 From: moshez@zadka.site.co.il (Moshe Zadka) Date: Sat, 18 Nov 2000 02:45:38 +0200 Subject: [Catalog-sig] UI for managing catalog In-Reply-To: Message from Andrew Kuchling of "Fri, 17 Nov 2000 10:39:27 EST." <20001117103927.B25735@kronos.cnri.reston.va.us> References:

<20001117103927.B25735@kronos.cnri.reston.va.us> Message-ID: [Andrew] >Of course. By CGI, you probably mean "server side code" rather then >really CGI. CGI is an evil protocol, and will die soon anyway. Why >not use a real tool, like Zope? [Moshe] > We do not want to require people to install the massive bulk of Zope. > Ideally I'd like running a package-only mirror to simply require > running wget or some similar tool; a searchable mirror should simply > require the additional step of setting up a CGI script and mirroring > the additional index file(s). Let me just note, that AFAIK, CPAN requires mod_perl. But in any case, that is a mute issue, for the time being. Could you start a list of open questions that we need to answer before starting to implement something? Here are some points to think about: * Registration/package uploading interface * Mirroring interface * How to seperate the meta-data from the content -- in the Debian system, e.g., each node has a complete copy of all meta-data locally. An "update" operation updates it. Quite likely, the meta-data file will change a lot, but requiring mirrors to generate it themselves is unrealistic. * Completing meta-data: among other things, how to specify dependancies on non-Python packages? (e.g., zlib) * What format will the packages be in? I recommend specifying *one* *standard* *format*, and if at all possible, make sure Python 2.1 supports it natively. * How do we handle collisions between package names/module names? Unlike Perl, Python has the problem that if there is a module "foo" there cannot be a module "foo.bar" too. * Who do we think will host the master archive? (Alternatives: SF, DC...) -- Moshe Zadka This is a signature anti-virus. Please stop the spread of signature viruses! From amos@digicool.com Fri Nov 17 20:29:49 2000 From: amos@digicool.com (Amos Latteier) Date: Fri, 17 Nov 2000 15:29:49 -0500 Subject: [Catalog-sig] UI for managing catalog In-Reply-To: <200011161956.UAA01036@loewis.home.cs.tu-berlin.de> Message-ID: On Thu, 16 Nov 2000 20:56:55 +0100 "Martin v. Loewis" wrote: > > Yes, it's the same old code. It is not safe. I had > problems getting > > Rexec work here, and punted on it. > > Can you elaborate on the problems, please? Is there > something that > intrinsically can't work, or did you just not get through > with it? The problem is that you need to run python code (setup.py) to extract meta-data from a distutils distribution. This is part of the distutils design, and this has been discussed on the distutils list. In the case of the catalog, you may not trust the code that has either been uploaded, or that the catalog has fetched from a URL. My though was that you could use rexec to execute untrusted setup.py files. However, I don't know much about rexec and was never able to get it to work right with sample setup.py files. No doubt someone else could do better. If it turns out the for some reason rexec can't provide a solution to this problem, I see some alternatives. We could find someway to trust the setup.py file, so a restricted execution environment wouldn't be necessary. Or we could require that meta-data come from somewhere else (such as manually filling out a web form). Or we could enhance the distutils to write meta-data to a text file in a simple format when it builds a distribution. -Amos From bkc@murkworks.com Sun Nov 19 15:21:10 2000 From: bkc@murkworks.com (Brad Clements) Date: Sun, 19 Nov 2000 10:21:10 -0500 Subject: [Catalog-sig] UI for managing catalog In-Reply-To: References: <200011161956.UAA01036@loewis.home.cs.tu-berlin.de> Message-ID: <3A17A5EF.568.1EF154E8@localhost> On 17 Nov 2000, at 15:29, Amos Latteier wrote: > Or we could > enhance the > distutils to write meta-data to a text file in a simple > format when it > builds a distribution. > I think this is the best solution. Brad Clements, bkc@murkworks.com (315)268-1000 http://www.murkworks.com (315)268-9812 Fax netmeeting: ils://ils.murkworks.com AOL-IM: BKClements From faassen@vet.uu.nl Mon Nov 20 14:08:07 2000 From: faassen@vet.uu.nl (Martijn Faassen) Date: Mon, 20 Nov 2000 15:08:07 +0100 Subject: [Catalog-sig] Hi, and.. Message-ID: <20001120150807.A13900@vet.uu.nl> Hi everybody, Cool new SIG, I just found out about it. It seems that Tim Middleton, the enigmatic creator of the Vaults of Parnassus has not yet joined this party. It'd be appropriate to email him a notification in case he missed it. Has someone done this yet? If not, could they? His email address is x@vex.net Regards, Martijn P.S. yes, I know Tim (or whatever his name is!) is deep in the PSU hierarchy, but that's all the more reason to let him in on this. :) From smendes@systemsfusion.com Mon Nov 20 16:28:30 2000 From: smendes@systemsfusion.com (Sandro Mendes) Date: Mon, 20 Nov 2000 09:28:30 -0700 Subject: [Catalog-sig] Text Based Application Message-ID: <3A19512E.633C7A18@systemsfusion.com> Hi, Does anybody know of a text based version of Tk? I'm looking for something like ncurses for python? Sandro From fdrake@acm.org Mon Nov 20 16:34:49 2000 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Mon, 20 Nov 2000 11:34:49 -0500 (EST) Subject: [Catalog-sig] Text Based Application In-Reply-To: <3A19512E.633C7A18@systemsfusion.com> References: <3A19512E.633C7A18@systemsfusion.com> Message-ID: <14873.21161.513220.481463@cj42289-a.reston1.va.home.com> Sandro Mendes writes: > Does anybody know of a text based version of Tk? I'm looking for > something like ncurses for python? The curses module that comes Python uses ncurses; you may need to enable the module _curses if you built from sources. -Fred -- Fred L. Drake, Jr. PythonLabs at Digital Creations From akuchlin@mems-exchange.org Mon Nov 20 22:45:35 2000 From: akuchlin@mems-exchange.org (Andrew Kuchling) Date: Mon, 20 Nov 2000 17:45:35 -0500 Subject: [Catalog-sig] UI for managing catalog In-Reply-To: ; from moshez@zadka.site.co.il on Sat, Nov 18, 2000 at 02:45:38AM +0200 References:

<20001117103927.B25735@kronos.cnri.reston.va.us>

Message-ID: <20001120174535.A22219@kronos.cnri.reston.va.us> On Sat, Nov 18, 2000 at 02:45:38AM +0200, Moshe Zadka wrote: >Could you start a list of open questions that we need to answer before >starting to implement something? >* Registration/package uploading interface >* Mirroring interface >* How to seperate the meta-data from the content -- in the Debian system, > e.g., each node has a complete copy of all meta-data locally. An "update" > operation updates it. Quite likely, the meta-data file will change a lot, > but requiring mirrors to generate it themselves is unrealistic. >* Completing meta-data: among other things, how to specify dependancies > on non-Python packages? (e.g., zlib) I've given my opinion for most of these issues in previous postings: 1) file uploads and automated retrieval from your download site 2) mirroring through something simple like wget 3) not previously discussed -- I don't know. 4) avoiding dependencies for now because it's a rat's nest of issues >* What format will the packages be in? I recommend specifying *one* *standard* > *format*, and if at all possible, make sure Python 2.1 supports it natively. .tar.gz and .zip seem like the two natural choices -- .tar.gz for Unix, .zip for Windows people. Luckily code is available to read both of these formats. >* How do we handle collisions between package names/module names? Unlike Perl, > Python has the problem that if there is a module "foo" there cannot be a > module "foo.bar" too. Unique naming; it's called mxODBC, for example, even though its package name is ODBC or, soon, mx.ODBC. >* Who do we think will host the master archive? (Alternatives: SF, DC...) Beats me; at the moment I'd rather work on a strawman implementation and worry about long-term hosting later. --amk