From sf_kersteinroie at bezeqint.net Wed Dec 1 21:34:38 2004 From: sf_kersteinroie at bezeqint.net (Roie Kerstein) Date: Wed Dec 1 21:52:04 2004 Subject: [Catalog-sig] Galois field in python Message-ID: Hello! Does anybody know a package for computations on Galois fields in python? It is very important for secret sharing implementation. Thank you. -- Best regards Roie Kerstein From ianb at colorstudy.com Fri Dec 10 19:51:01 2004 From: ianb at colorstudy.com (Ian Bicking) Date: Fri Dec 10 19:54:54 2004 Subject: [Catalog-sig] Re: [chiPy] Meeting followup- CPAN for Python? In-Reply-To: <20041210174530.GB14635@chloe.inkdroid.org> References: <20041210174530.GB14635@chloe.inkdroid.org> Message-ID: <41B9F015.8090803@colorstudy.com> I'm copying the Catalog-SIG mailing list, since there's other interested people there, and if we want to do something it would be good to coordinate. Ed Summers wrote: > I also found the discussion of a python software repository to be one of > the higlights of the evening. I'm sure having brian d foy there, who has > been instrumental in building the cpan added to this. I for one am > interested in volunteering some time on a project to develop a software > archive for python, but I'd like to find out why other efforts failed or > fizzled first. > > Is there resistance to a centralized archive of python software? I know > that one advantage that python (ruby|java) has over perl is that many > common things such as XML parsing, HTTP, SMTP, come as part of the core, > whereas in Perl they are added on with the cpan utility. Perhaps this is > why having a central repository hasn't been that important? I don't think so. Really, I don't see much resistence to this, I'd chaulk it entirely up to inertia. People should look at what exists now: http://www.python.org/pypi http://sourceforge.net/projects/pypi/ Note that registration is automatic if you use distutils (just run "python setup.py register"), though distutils is not required to submit to PyPI. If you want to discuss PyPI and related topics, there's a mailing list: http://www.python.org/sigs/catalog-sig/ Richard Jones is the author, and he's very willing to include other people in PyPI's development. There's several PEPs noted on that page. There's also a distutils-sig, which relates: http://www.python.org/sigs/distutils-sig/ Distutils has much, but not all, of the metadata needed. setuptools is one extension meant to support more granular packages and dependencies: http://cvs.eby-sarna.com/PEAK/setuptools/ Frank Drake has also been doing some work with this, to distribute Zope 3 in a series of smaller packages; I'm not sure where that work lives. Maybe Twisted people have also worked on this, as they wanted to split the Twisted installation up, but I haven't heard anything about that for a while. Anyway, this might be a distraction, in that it's a much more complex problem than a CPAN-ish thing. Here's a recent thread on the topic of CPAN/Python: http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/85681185e372c9e1/a61ffe06820e3d88 I think there's an incremental way to get there: * Track package names into PyPI. For now these will serve as identifiers for installation and dependencies. Conflicting package names already cause a lot of problems anyway, so it would be nice if they'd be unique. * Add a field for package dependencies, based on those names. Forget about version requirements for now. Though in theory, since distutil setup.py files are just Python scripts, this should be extensible. * Add a field for source download; maybe make it a dictionary, so you can give several links for different sources (e.g., source tarball/zip, rpm, windows installer, debian package, etc). * Create a script that can query PyPI, get the link(s), then download it. PyPI already has an XML-RPC interface, I believe. Because of SF, the downloader has to be a little smart about the load balancing page in that case, but that's relatively easy. These all seem pretty doable to me, and not very controversial. Except the package name and dependency part, which isn't "controversial" per se, but has some subtle problems. Oh, if only we used unambiguous names like com.colorstudy.sqlobject! j/k From there, we can implement a package cacher that downloads packages based on that metadata. We don't have to copy CPAN, in that it's okay if the cached files aren't the canonical location for the package, just a backup. Perhaps we can add a field to the metadata that indicates if the author prefers for the cached version to be canonical (i.e., they don't have a good host). Or something to "setup.py register" that can upload to a known server. Or something to PyPI where mirrors can register that they have a file, and it can do URL rotation. We can even do all of those, and see which one works best. The point being: once we have the data we can start using it, but getting the data is the place to start. Well, these are some of my ideas. If people are interested in this in general, we could try to organize a little mini-sprint; a number of us would come together and try to bang it out. We could try to schedule and coordinate this with Richard or other interested people over IRC. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org From richardjones at optushome.com.au Fri Dec 10 23:20:17 2004 From: richardjones at optushome.com.au (Richard Jones) Date: Fri Dec 10 23:20:38 2004 Subject: [Catalog-sig] Re: Meeting followup- CPAN for Python? In-Reply-To: <41B9F015.8090803@colorstudy.com> References: <20041210174530.GB14635@chloe.inkdroid.org> <41B9F015.8090803@colorstudy.com> Message-ID: <200412110920.27461.richardjones@optushome.com.au> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Sat, 11 Dec 2004 05:51 am, Ian Bicking wrote: > * Track package names into PyPI. ?For now these will serve as > identifiers for installation and dependencies. ?Conflicting package > names already cause a lot of problems anyway, so it would be nice if > they'd be unique. This step is done. In PyPI, a name is owned by someone (or some people). > * Add a field for package dependencies, based on those names. ?Forget > about version requirements for now. ?Though in theory, since distutil > setup.py files are just Python scripts, this should be extensible. Also, handling versioning implies that people retain versioned source downloads - something we can't rely on. Possibly what we need is for the package to be able to indicate which versions it's been *tested* with, so it can warn that the download might not work? IIRC a problem with CPAN was version incompatibilities breaking things (I know I've hosed my Perl install a couple of times) Now, having dependencies list is all well and good - what do we do with them? If we have automated checking against those dependencies, then that implies that we remember what's installed. AMK has done some work on this: Database of Installed Python Packages http://python.org/peps/pep-0262.html Other relavent PEPs: Package Index and Metadata for Distutils http://python.org/peps/pep-0301.html (*still* marked "under consideration"... sigh) Metadata for Python Software Packages v1.1 http://python.org/peps/pep-0314.html Metadata version 1.1 has fields for "download-url", "requires", "provides", "obsoletes" and "conflicts". I guess we need to change "download-url" to "source-url". Mmm. Ambiguity. It also removes Platform, and specifies the formatting of the long description field. The only open issue in that PEP is the license field. I believe some people would like it to remain, but only be used when their license doesn't appear in the Classifiers list, and they need to include the full text of their license (or some other statement about it). See PyPI bug 693471. Also of interest: Module Repository Upload Mechanism http://python.org/peps/pep-0243.html > * Add a field for source download; maybe make it a dictionary, so you > can give several links for different sources (e.g., source tarball/zip, > rpm, windows installer, debian package, etc). I think just sticking with the source installer is fine. This is all invoked by setup.py, isn't it? > * Create a script that can query PyPI, get the link(s), then download > it. ?PyPI already has an XML-RPC interface, I believe. PyPI's web interface is a complete shambles. I really want to reimplement it in something easier to maintain, and more extensible. As an aside: the pypi sqlite database really needs to move to postgresql. > Because of SF, > the downloader has to be a little smart about the load balancing page in > that case, but that's relatively easy. We'll be special-casing for them, but there will probably be other download sites like this :( > Well, these are some of my ideas. If people are interested in this in > general, we could try to organize a little mini-sprint; a number of us > would come together and try to bang it out. We could try to schedule > and coordinate this with Richard or other interested people over IRC. Looks like I'll be attending PyCon, so a sprint there would be possible. Richard -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) iD8DBQFBuiErrGisBEHG6TARAn51AJwOT9hEBVKsumDM9/PBj6BbBATf/QCggmLG Q1RxvgXOjCr67pr5H08RYDs= =9LZy -----END PGP SIGNATURE----- From ianb at colorstudy.com Sat Dec 11 00:22:19 2004 From: ianb at colorstudy.com (Ian Bicking) Date: Sat Dec 11 00:26:24 2004 Subject: [Catalog-sig] Re: Meeting followup- CPAN for Python? In-Reply-To: <200412110920.27461.richardjones@optushome.com.au> References: <20041210174530.GB14635@chloe.inkdroid.org> <41B9F015.8090803@colorstudy.com> <200412110920.27461.richardjones@optushome.com.au> Message-ID: <41BA2FAB.5080702@colorstudy.com> Richard Jones wrote: > On Sat, 11 Dec 2004 05:51 am, Ian Bicking wrote: > >>* Track package names into PyPI. For now these will serve as >>identifiers for installation and dependencies. Conflicting package >>names already cause a lot of problems anyway, so it would be nice if >>they'd be unique. > > This step is done. In PyPI, a name is owned by someone (or some people). True. The downside is that it isn't the Python package name, it's just a name. The Python package name seems like a natural identifier; though some projects have no packages (e.g., many applications), and some have multiple packages. >>* Add a field for package dependencies, based on those names. Forget >>about version requirements for now. Though in theory, since distutil >>setup.py files are just Python scripts, this should be extensible. > > > Also, handling versioning implies that people retain versioned source > downloads - something we can't rely on. Well, we couldn't necessarily resolve problems, but distutils could at least complain obnoxiously. > Possibly what we need is for the package to be able to indicate which versions > it's been *tested* with, so it can warn that the download might not work? That seems fairly easy. > IIRC a problem with CPAN was version incompatibilities breaking things (I > know I've hosed my Perl install a couple of times) Doing multiple installs of different versions in Python isn't that hard, but it's not that easy either. That's a fairly expansive problem. > Now, having dependencies list is all well and good - what do we do with them? I don't think we should look that far ahead ;) If I was going to leave something out of this iteration, the first thing would be the dependencies list. You can always put this at the top of your setup.py: try: import some_package except ImportError: print "You have not installed a required package: some_package" sys.exit(1) > If we have automated checking against those dependencies, then that implies > that we remember what's installed. AMK has done some work on this: > > Database of Installed Python Packages > http://python.org/peps/pep-0262.html Certainly useful. > Other relavent PEPs: > > Package Index and Metadata for Distutils > http://python.org/peps/pep-0301.html > (*still* marked "under consideration"... sigh) > Metadata for Python Software Packages v1.1 > http://python.org/peps/pep-0314.html > > Metadata version 1.1 has fields for "download-url", "requires", "provides", > "obsoletes" and "conflicts". I guess we need to change "download-url" to > "source-url". Mmm. Ambiguity. Indeed, it seems like half of the packages don't have source on the other side of their download URL. Lots don't even have a download page, or even a package-specific page. In cases of ambiguity, authors choose to be lazy... It also needs to support multiple kinds of downloads. Freshmeat has this, though the kinds of downloads are fixed, which seems unnecessary for us. System-specific packages are usually prefered if available. Well, sometimes -- usually system-specific packages are harder to use for multi-version installs. Anyway, there's a place for several kinds of packages. > It also removes Platform, and specifies the > formatting of the long description field. The only open issue in that PEP is > the license field. I believe some people would like it to remain, but only be > used when their license doesn't appear in the Classifiers list, and they need > to include the full text of their license (or some other statement about it). > See PyPI bug 693471. > > Also of interest: > > Module Repository Upload Mechanism > http://python.org/peps/pep-0243.html > > > >>* Add a field for source download; maybe make it a dictionary, so you >>can give several links for different sources (e.g., source tarball/zip, >>rpm, windows installer, debian package, etc). > > > I think just sticking with the source installer is fine. This is all invoked > by setup.py, isn't it? Well, by phrasing this in terms of data, not functionality, we make this potentially frontend-neutral. We can already generate rpm's from distutils, and I don't know why debs haven't also been added to that. In many cases they would be preferred. >>* Create a script that can query PyPI, get the link(s), then download >>it. PyPI already has an XML-RPC interface, I believe. > > > PyPI's web interface is a complete shambles. I really want to reimplement it > in something easier to maintain, and more extensible. As an aside: the pypi > sqlite database really needs to move to postgresql. Would that be hard? Is it mostly a sysadmin issue, or a coding issue? What do you want to change in PyPI's interface? >>Because of SF, >>the downloader has to be a little smart about the load balancing page in >>that case, but that's relatively easy. > > We'll be special-casing for them, but there will probably be other download > sites like this :( Yeah... well, at least for SF it is worth doing the special case, simply because of the number of files hosted there. After that, who knows. >>Well, these are some of my ideas. If people are interested in this in >>general, we could try to organize a little mini-sprint; a number of us >>would come together and try to bang it out. We could try to schedule >>and coordinate this with Richard or other interested people over IRC. > > > Looks like I'll be attending PyCon, so a sprint there would be possible. Well, people here at ChiPy have been showing some interest, and most of them probably won't be going to PyCon. So people might want to try tackling some of these things on a different schedule. And the PyCon sprints won't be during the weekend, so that would also make it hard for non-PyCon people to participate. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org From richardjones at optushome.com.au Sat Dec 11 02:55:12 2004 From: richardjones at optushome.com.au (Richard Jones) Date: Sat Dec 11 02:55:54 2004 Subject: [Catalog-sig] Re: Meeting followup- CPAN for Python? In-Reply-To: <41BA2FAB.5080702@colorstudy.com> References: <200412110920.27461.richardjones@optushome.com.au> <41BA2FAB.5080702@colorstudy.com> Message-ID: <200412111255.12930.richardjones@optushome.com.au> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Sat, 11 Dec 2004 10:22 am, Ian Bicking wrote: > Richard Jones wrote: > > On Sat, 11 Dec 2004 05:51 am, Ian Bicking wrote: > >>* Track package names into PyPI. For now these will serve as > >>identifiers for installation and dependencies. Conflicting package > >>names already cause a lot of problems anyway, so it would be nice if > >>they'd be unique. > > > > This step is done. In PyPI, a name is owned by someone (or some people). > > True. The downside is that it isn't the Python package name, it's just > a name. The Python package name seems like a natural identifier; though > some projects have no packages (e.g., many applications), and some have > multiple packages. Ah yes - setup.py name != python package name. This has caused confusion -- some people accidentally submit packages to pypi with *their* name in the "name" field of setup.py... In CPAN (and PAUSE) there's no such disinction, IIRC. The "provides" field in metadata 1.1 solves this problem, I think. Ownership of names would flow through. > It also needs to support multiple kinds of downloads. Freshmeat has > this, though the kinds of downloads are fixed, which seems unnecessary > for us. System-specific packages are usually prefered if available. > Well, sometimes -- usually system-specific packages are harder to use > for multi-version installs. Anyway, there's a place for several kinds > of packages. How about we start with just source-url and go from there? We'll probably want some system somewhere that verifies that the source url works. Doing that at registration time might be a little too slow. > > It also removes Platform, and specifies the > > formatting of the long description field. The only open issue in that PEP > > is the license field. I believe some people would like it to remain, but > > only be used when their license doesn't appear in the Classifiers list, > > and they need to include the full text of their license (or some other > > statement about it). See PyPI bug 693471. > > > > Also of interest: > > > > Module Repository Upload Mechanism > > http://python.org/peps/pep-0243.html > > > >>* Add a field for source download; maybe make it a dictionary, so you > >>can give several links for different sources (e.g., source tarball/zip, > >>rpm, windows installer, debian package, etc). > > > > I think just sticking with the source installer is fine. This is all > > invoked by setup.py, isn't it? > > Well, by phrasing this in terms of data, not functionality, we make this > potentially frontend-neutral. We can already generate rpm's from > distutils, and I don't know why debs haven't also been added to that. > In many cases they would be preferred. Do the existing RPM / exe installers invoke setup.py to do the install? If they don't how will we fire the appropriate package database update (aka PEP 262)? > >>* Create a script that can query PyPI, get the link(s), then download > >>it. PyPI already has an XML-RPC interface, I believe. > > > > PyPI's web interface is a complete shambles. I really want to reimplement > > it in something easier to maintain, and more extensible. As an aside: the > > pypi sqlite database really needs to move to postgresql. > > Would that be hard? Is it mostly a sysadmin issue, or a coding issue? Not hard. Needs time, which I don't have. > What do you want to change in PyPI's interface? It's a hand-coded, unextensible mess. Mostly, it runs as a cgi-bin, which really needs to change to mod_python. > Well, people here at ChiPy have been showing some interest, and most of > them probably won't be going to PyCon. So people might want to try > tackling some of these things on a different schedule. And the PyCon > sprints won't be during the weekend, so that would also make it hard for > non-PyCon people to participate. OK. Mostly what's needed is a thorough design in a PEP - then just about anyone could do the small amount of coding by themselves. A PSF grant could help... Richard -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) iD8DBQFBulOArGisBEHG6TARAuSqAJ9SQFALpszc8wpReashWjD0kQazegCfZnLJ Mb0AT9+IVORHtvc4L3vjBYw= =p++l -----END PGP SIGNATURE----- From ianb at colorstudy.com Sat Dec 11 06:55:18 2004 From: ianb at colorstudy.com (Ian Bicking) Date: Sat Dec 11 06:55:15 2004 Subject: [Catalog-sig] Re: Meeting followup- CPAN for Python? In-Reply-To: <200412111255.12930.richardjones@optushome.com.au> References: <200412110920.27461.richardjones@optushome.com.au> <41BA2FAB.5080702@colorstudy.com> <200412111255.12930.richardjones@optushome.com.au> Message-ID: <41BA8BC6.4050503@colorstudy.com> Richard Jones wrote: > On Sat, 11 Dec 2004 10:22 am, Ian Bicking wrote: > >>Richard Jones wrote: >> >>>On Sat, 11 Dec 2004 05:51 am, Ian Bicking wrote: >>> >>>>* Track package names into PyPI. For now these will serve as >>>>identifiers for installation and dependencies. Conflicting package >>>>names already cause a lot of problems anyway, so it would be nice if >>>>they'd be unique. >>> >>>This step is done. In PyPI, a name is owned by someone (or some people). >> >>True. The downside is that it isn't the Python package name, it's just >>a name. The Python package name seems like a natural identifier; though >>some projects have no packages (e.g., many applications), and some have >>multiple packages. > > > Ah yes - setup.py name != python package name. This has caused confusion -- > some people accidentally submit packages to pypi with *their* name in the > "name" field of setup.py... > > In CPAN (and PAUSE) there's no such disinction, IIRC. > > The "provides" field in metadata 1.1 solves this problem, I think. Ownership > of names would flow through. OK, that sounds good. >>It also needs to support multiple kinds of downloads. Freshmeat has >>this, though the kinds of downloads are fixed, which seems unnecessary >>for us. System-specific packages are usually prefered if available. >>Well, sometimes -- usually system-specific packages are harder to use >>for multi-version installs. Anyway, there's a place for several kinds >>of packages. > > > How about we start with just source-url and go from there? A dictionary seems like such a simple extension, and multiple download links are useful for a good many packages already. The keys would just be strings, "windows-installer", "rpm", "source". > We'll probably want some system somewhere that verifies that the source url > works. Doing that at registration time might be a little too slow. OK, internally I guess there should be a flag "not-checked", "working", "broken". Later if we do backups we could use this to signal when we need to replace the link with the backup link. >>>It also removes Platform, and specifies the >>>formatting of the long description field. The only open issue in that PEP >>>is the license field. I believe some people would like it to remain, but >>>only be used when their license doesn't appear in the Classifiers list, >>>and they need to include the full text of their license (or some other >>>statement about it). See PyPI bug 693471. >>> >>>Also of interest: >>> >>>Module Repository Upload Mechanism >>> http://python.org/peps/pep-0243.html >>> >>> >>>>* Add a field for source download; maybe make it a dictionary, so you >>>>can give several links for different sources (e.g., source tarball/zip, >>>>rpm, windows installer, debian package, etc). >>> >>>I think just sticking with the source installer is fine. This is all >>>invoked by setup.py, isn't it? >> >>Well, by phrasing this in terms of data, not functionality, we make this >>potentially frontend-neutral. We can already generate rpm's from >>distutils, and I don't know why debs haven't also been added to that. >>In many cases they would be preferred. > > > Do the existing RPM / exe installers invoke setup.py to do the install? If > they don't how will we fire the appropriate package database update (aka PEP > 262)? No, I don't think any of them are based on setup.py, at least at the point of installation. Instead setup.py captures all the necessary build information when the package is created. What would have to happen is these systems would have to run the PEP 262-updating scripts after installation had finished. It's not really a very big difference, since it'll probably all be the same code any way it gets run, just slightly different orders. >>>>* Create a script that can query PyPI, get the link(s), then download >>>>it. PyPI already has an XML-RPC interface, I believe. >>> >>>PyPI's web interface is a complete shambles. I really want to reimplement >>>it in something easier to maintain, and more extensible. As an aside: the >>>pypi sqlite database really needs to move to postgresql. >> >>Would that be hard? Is it mostly a sysadmin issue, or a coding issue? > > > Not hard. Needs time, which I don't have. > > > >>What do you want to change in PyPI's interface? > > > It's a hand-coded, unextensible mess. Mostly, it runs as a cgi-bin, which > really needs to change to mod_python. Or to... WSGI! It'd probably be a smoother transition than mod_python, since WSGI looks a lot like cgi. That's something I might find interesting to try. Well... this is also a distraction from my own projects; it's always a difficulty with these things for me. I'm sure you know the feeling. So... one underlying motivation is getting some of the other people at ChiPy involved; there was a lot of discussion about this, and people following up on that discussion. It seems like a good project. Looking at the code, it looks like simply moving HTML out to templates would help quite a bit. Then moving to Postgres, which would probably be easy enough. Well, this is the bread and butter of my day work, so it seems easy enough to me... I'm not sure what other people feel. Has anyone else checked out pypi and given it a look? To checkout: cvs -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/pypi login cvs -z3 -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/pypi co pypi Then to get it started... I made a directory /tmp/pypi, and copied config.ini into it, editing it a bit. I made sure pypi.cgi was available through Apache. It may take some configuration to get .cgi files to be run for you. I had to make sure the directory the database was located in was writable by the web server. Lastly, it would be useful to have a full database available. Is there a place we can pick up a dump of the pypi database? It doesn't have to be very fresh. >>Well, people here at ChiPy have been showing some interest, and most of >>them probably won't be going to PyCon. So people might want to try >>tackling some of these things on a different schedule. And the PyCon >>sprints won't be during the weekend, so that would also make it hard for >>non-PyCon people to participate. > > > OK. > > Mostly what's needed is a thorough design in a PEP - then just about anyone > could do the small amount of coding by themselves. A PSF grant could help... What portions do you think need PEPs? The internal design of PyPI doesn't seem to be a PEP concern. The added metadata is a PEP concern, I guess. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org From ehs at pobox.com Sat Dec 11 16:46:47 2004 From: ehs at pobox.com (Ed Summers) Date: Sat Dec 11 16:46:51 2004 Subject: [Catalog-sig] Re: Meeting followup- CPAN for Python? In-Reply-To: <200412111255.12930.richardjones@optushome.com.au> References: <200412110920.27461.richardjones@optushome.com.au> <41BA2FAB.5080702@colorstudy.com> <200412111255.12930.richardjones@optushome.com.au> Message-ID: <20041211154647.GB5974@chloe.inkdroid.org> On Sat, Dec 11, 2004 at 12:55:12PM +1100, Richard Jones wrote: > Ah yes - setup.py name != python package name. This has caused confusion -- > some people accidentally submit packages to pypi with *their* name in the > "name" field of setup.py... > > In CPAN (and PAUSE) there's no such disinction, IIRC. FWIW there is a distinction, the setup.py name (aka distribution name) has dashes in it, whereas the actual perl package has colons. distribution name: XML-RSS package namespace: XML::RSS Not only that, but they need not be the same...especially distributions which install modules into multiple package namespaces such as in http://search.cpan.org/~gbarr/libnet-1.19/ distribution name: libnet package namespaces: Net::Cmd, Net::Config, Net::Domain, Net::FTP ... To be honest I don't think there is anything preventing someone from uploading a CPAN module that installs into a namespace that someone else (even the perl core distribution)...not that this would be advisable. //Ed From richardjones at optushome.com.au Sat Dec 11 23:33:54 2004 From: richardjones at optushome.com.au (Richard Jones) Date: Sat Dec 11 23:34:45 2004 Subject: [Catalog-sig] Re: Meeting followup- CPAN for Python? In-Reply-To: <41BA8BC6.4050503@colorstudy.com> References: <200412111255.12930.richardjones@optushome.com.au> <41BA8BC6.4050503@colorstudy.com> Message-ID: <200412120933.58532.richardjones@optushome.com.au> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Sat, 11 Dec 2004 04:55 pm, Ian Bicking wrote: > Richard Jones wrote: > > On Sat, 11 Dec 2004 10:22 am, Ian Bicking wrote: > >>It also needs to support multiple kinds of downloads. Freshmeat has > >>this, though the kinds of downloads are fixed, which seems unnecessary > >>for us. System-specific packages are usually prefered if available. > >>Well, sometimes -- usually system-specific packages are harder to use > >>for multi-version installs. Anyway, there's a place for several kinds > >>of packages. > > > > How about we start with just source-url and go from there? > > A dictionary seems like such a simple extension, and multiple download > links are useful for a good many packages already. The keys would just > be strings, "windows-installer", "rpm", "source". We should probably use the same names as the distutils dist command. > > We'll probably want some system somewhere that verifies that the source > > url works. Doing that at registration time might be a little too slow. > > OK, internally I guess there should be a flag "not-checked", "working", > "broken". Later if we do backups we could use this to signal when we > need to replace the link with the backup link. Yup. > Or to... WSGI! It'd probably be a smoother transition than mod_python, > since WSGI looks a lot like cgi. That's something I might find > interesting to try. Well... this is also a distraction from my own > projects; it's always a difficulty with these things for me. I'm sure > you know the feeling. :) I'll look into WSGI. > So... one underlying motivation is getting some of the other people at > ChiPy involved; there was a lot of discussion about this, and people > following up on that discussion. It seems like a good project. > > Looking at the code, it looks like simply moving HTML out to templates > would help quite a bit. Then moving to Postgres, which would probably > be easy enough. Well, this is the bread and butter of my day work, so > it seems easy enough to me... I'm not sure what other people feel. Has > anyone else checked out pypi and given it a look? To checkout: > > cvs -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/pypi login > cvs -z3 -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/pypi co pypi > > Then to get it started... > > I made a directory /tmp/pypi, and copied config.ini into it, editing it > a bit. I made sure pypi.cgi was available through Apache. It may take > some configuration to get .cgi files to be run for you. I had to make > sure the directory the database was located in was writable by the web > server. > > Lastly, it would be useful to have a full database available. Is there > a place we can pick up a dump of the pypi database? It doesn't have to > be very fresh. Snarfed just now from creosote: http://www.mechanicalcat.net/temp/pkgbase.db.bz2 (874 Kb) > > Mostly what's needed is a thorough design in a PEP - then just about > > anyone could do the small amount of coding by themselves. A PSF grant > > could help... > > What portions do you think need PEPs? The internal design of PyPI > doesn't seem to be a PEP concern. The added metadata is a PEP concern, > I guess. I guess there's a bunch of individual PEPs at the moment. They need review by both yourself and me (I've read them, but not with an eye to review / implementation). The method you're describing to download and install using a PyPI-supplied URL would be an extension to distutils in the Python core, and should be done as a PEP to allow community scrutiny of the approach. That PEP would also be a good place to capture feedback and address issues of security (eg. "we don't address security issues"). Also, there's no real sense of a greater strategy, which is what we're discussing in these emails. I don't know whether that's a new PEP, or just a document somewhere that describes how all these disparate parts form a cohesive whole. Richard -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) iD8DBQFBu3XWrGisBEHG6TARAvxWAJ4n54VcFhbo3/Q72jc+nNJ8HKcy+QCeKdWM t7+j2JxWbLSRE++MZvZDduw= =7XNZ -----END PGP SIGNATURE----- From amk at amk.ca Mon Dec 13 20:29:15 2004 From: amk at amk.ca (A.M. Kuchling) Date: Mon Dec 13 20:29:56 2004 Subject: [Catalog-sig] Re: [chiPy] Meeting followup- CPAN for Python? In-Reply-To: <41B9F015.8090803@colorstudy.com> References: <20041210174530.GB14635@chloe.inkdroid.org> <41B9F015.8090803@colorstudy.com> Message-ID: <20041213192915.GA29353@rogue.amk.ca> On Fri, Dec 10, 2004 at 12:51:01PM -0600, Ian Bicking wrote: > * Add a field for source download; maybe make it a dictionary, so you > can give several links for different sources (e.g., source > tarball/zip, rpm, windows installer, debian package, etc). An up-and-coming standard for info about packages is an RDF vocabulary called DOAP: http://usefulinc.com/doap . IMHO we should think about harmonizing the Distutils-supported metadata with DOAP. I've submitted a crude patch to PyPI that generates DOAP data for projects. Things in DOAP not in Distutils/PyPI: CVS/SVN/BK repository (+ anonymous root, module name) Mailing lists Wiki URL Bug database Screenshot URL Date project was created Releases Versions are a revision number (1.2.3, 1.0a1, etc.) plus name and date. Multiple language literals (e.g. give both an English and a French description). However, just adding keywords to setup.core() for all these things may not be a good idea -- if DOAP adds new features, we need to add support for them. Perhaps there should be some third-party way of constructing a DOAP file, after which you'd leave the metadata out of the setup.core() invocation. OTOH, requiring an RDF parser to deal with the metadata is also nasty; parsers are pretty large, and adding one to the core might be a struggle. --amk From ianb at colorstudy.com Wed Dec 15 21:04:31 2004 From: ianb at colorstudy.com (Ian Bicking) Date: Wed Dec 15 21:08:44 2004 Subject: [Catalog-sig] Re: [chiPy] Meeting followup- CPAN for Python? In-Reply-To: <20041213192915.GA29353@rogue.amk.ca> References: <20041210174530.GB14635@chloe.inkdroid.org> <41B9F015.8090803@colorstudy.com> <20041213192915.GA29353@rogue.amk.ca> Message-ID: <41C098CF.5040704@colorstudy.com> A.M. Kuchling wrote: > On Fri, Dec 10, 2004 at 12:51:01PM -0600, Ian Bicking wrote: > >>* Add a field for source download; maybe make it a dictionary, so you >>can give several links for different sources (e.g., source >>tarball/zip, rpm, windows installer, debian package, etc). > > > An up-and-coming standard for info about packages is an RDF vocabulary > called DOAP: http://usefulinc.com/doap . IMHO we should think about > harmonizing the Distutils-supported metadata with DOAP. I've > submitted a crude patch to PyPI that generates DOAP data for projects. > > Things in DOAP not in Distutils/PyPI: > > CVS/SVN/BK repository (+ anonymous root, module name) > Mailing lists > Wiki URL > Bug database > Screenshot URL > Date project was created > Releases Like multiple released branches, or historical releases? > Versions are a revision number (1.2.3, 1.0a1, etc.) plus > name and date. > Multiple language literals (e.g. give both an English and a > French description). > > However, just adding keywords to setup.core() for all these things may > not be a good idea -- if DOAP adds new features, we need to add > support for them. Perhaps there should be some third-party way of > constructing a DOAP file, after which you'd leave the metadata out of > the setup.core() invocation. I'd like this anyway -- it feels distracting that setup.py contains both metadata for PyPI, and control data for distutils. There's some intersection, but most of the data really belongs to one or the other. I'm heavily drawn to using source to represent data, but despite that I think a static file would be appropriate for most of these descriptions. Or maybe a feature to setup() that reads metadata from a static file (so you could generate the data in other ways if you really wanted, e.g., an analogous .ini file). > OTOH, requiring an RDF parser to deal with the metadata is also nasty; > parsers are pretty large, and adding one to the core might be a > struggle. Is an RDF parser very complicated? It's mostly just XML with namespaces isn't it? Do the standard library's XML parsers handle namespaces well? -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org From amk at amk.ca Wed Dec 15 21:26:51 2004 From: amk at amk.ca (A.M. Kuchling) Date: Wed Dec 15 21:27:40 2004 Subject: [Catalog-sig] Re: [chiPy] Meeting followup- CPAN for Python? In-Reply-To: <41C098CF.5040704@colorstudy.com> References: <20041210174530.GB14635@chloe.inkdroid.org> <41B9F015.8090803@colorstudy.com> <20041213192915.GA29353@rogue.amk.ca> <41C098CF.5040704@colorstudy.com> Message-ID: <20041215202651.GA10402@rogue.amk.ca> On Wed, Dec 15, 2004 at 02:04:31PM -0600, Ian Bicking wrote: > Like multiple released branches, or historical releases? Either/both, I think; releases have revision #/name/date, so you could have stable and unstable, or just a record of every version released. > Is an RDF parser very complicated? It's mostly just XML with namespaces > isn't it? Do the standard library's XML parsers handle namespaces well? Usually RDF parsers end up providing some sort of graph API as well. It may not matter much if we only generate DOAP in the stdlib and never need to read it; minidom or even 'print >>' can do the job well enough. --amk From ianb at colorstudy.com Wed Dec 15 21:35:40 2004 From: ianb at colorstudy.com (Ian Bicking) Date: Wed Dec 15 21:39:47 2004 Subject: [Catalog-sig] Re: [chiPy] Meeting followup- CPAN for Python? In-Reply-To: <20041215202651.GA10402@rogue.amk.ca> References: <20041210174530.GB14635@chloe.inkdroid.org> <41B9F015.8090803@colorstudy.com> <20041213192915.GA29353@rogue.amk.ca> <41C098CF.5040704@colorstudy.com> <20041215202651.GA10402@rogue.amk.ca> Message-ID: <41C0A01C.5060106@colorstudy.com> A.M. Kuchling wrote: >>Is an RDF parser very complicated? It's mostly just XML with namespaces >>isn't it? Do the standard library's XML parsers handle namespaces well? > > > Usually RDF parsers end up providing some sort of graph API as well. > It may not matter much if we only generate DOAP in the stdlib and > never need to read it; minidom or even 'print >>' can do the job well > enough. That would only matter if we wanted to provide, say, a dependency graph, right? For a single package there's no resolvable relations (until it is entered into PyPI, or if we had a package database). And even then, the graph API will be the same algorithms we'd need anyway. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org From amk at amk.ca Wed Dec 15 22:40:21 2004 From: amk at amk.ca (A.M. Kuchling) Date: Wed Dec 15 22:41:07 2004 Subject: [Catalog-sig] Re: [chiPy] Meeting followup- CPAN for Python? In-Reply-To: <41C0A01C.5060106@colorstudy.com> References: <20041210174530.GB14635@chloe.inkdroid.org> <41B9F015.8090803@colorstudy.com> <20041213192915.GA29353@rogue.amk.ca> <41C098CF.5040704@colorstudy.com> <20041215202651.GA10402@rogue.amk.ca> <41C0A01C.5060106@colorstudy.com> Message-ID: <20041215214021.GA10819@rogue.amk.ca> On Wed, Dec 15, 2004 at 02:35:40PM -0600, Ian Bicking wrote: > That would only matter if we wanted to provide, say, a dependency graph, > right? For a single package there's no resolvable relations (until it Not really; the RDF model is itself a graph. e.g. the metadata for a package would be something like: -> rdf:type -> doap:Project -> doap:CVSRepository cvs.sf.net -> doap:maintainer -> -> rdf:type -> foaf:Person -> foaf:mbox -> amk@amk.ca -> foaf:ircNickname -> amk_ See http://www.amk.ca/talks/2004-12-02/rdf-graph.png for another example graph. There are some small RDF parsers out there, but I don't know if they're fully standard-compliant. --amk From richardjones at optushome.com.au Thu Dec 30 05:35:30 2004 From: richardjones at optushome.com.au (Richard Jones) Date: Thu Dec 30 05:35:39 2004 Subject: [Catalog-sig] PyCon sprint for PEPs 314 and 243? In-Reply-To: <200412110920.27461.richardjones@optushome.com.au> References: <41B9F015.8090803@colorstudy.com> <200412110920.27461.richardjones@optushome.com.au> Message-ID: <200412301535.34522.richardjones@optushome.com.au> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 So, is anyone interested in getting together before PyCon for a sprint to finalise and/or implement PEPs 314, 243 (I guess we should ask for some PSF money, though we've obviously missed the most recent grant allocation round) and possibly 262 (I consider this optional and don't believe it should get in the way of implementing the other two)? Metadata for Python Software Packages v1.1 http://python.org/peps/pep-0314.html Module Repository Upload Mechanism http://python.org/peps/pep-0243.html Database of Installed Python Packages http://python.org/peps/pep-0262.html I'd want to revise PEP 243 at some stage. I want to *only* implement source distribution uploading for now. Also, there's some aspects of the protocol that need work (eg. we should use standard HTTP responses, not X-Swalow-Status). Richard -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) iD8DBQFB04WWrGisBEHG6TARAu7YAJwKj2BtJ272bOUHZsGi/sA8oaGcdgCfTCx5 iT7u2WHeuLmXdgdZTyyBwX0= =LjA2 -----END PGP SIGNATURE----- From richardjones at optushome.com.au Thu Dec 30 05:56:28 2004 From: richardjones at optushome.com.au (Richard Jones) Date: Thu Dec 30 05:56:34 2004 Subject: [Catalog-sig] PyCon sprint for PEPs 314 and 243? In-Reply-To: <200412301535.34522.richardjones@optushome.com.au> References: <200412110920.27461.richardjones@optushome.com.au> <200412301535.34522.richardjones@optushome.com.au> Message-ID: <200412301556.28923.richardjones@optushome.com.au> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thu, 30 Dec 2004 03:35 pm, Richard Jones wrote: > I'd want to revise PEP 243 at some stage. I want to *only* implement source > distribution uploading for now. Also, there's some aspects of the protocol > that need work (eg. we should use standard HTTP responses, not > X-Swalow-Status). Just thinking out loud - stuff we need to design / also implement: . organisation of repository on the server, . repository mirroring setup, and . a download mechanism including using mirrors. (and probably others) Richard -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) iD8DBQFB04p8rGisBEHG6TARAm9/AJ43A+0B6sJUOPoFJKJo/I0ZUeVF1QCghgS6 MU0bAGxCmBUVFZJKyiDWFGM= =Lg27 -----END PGP SIGNATURE----- From jafo at tummy.com Thu Dec 30 18:18:27 2004 From: jafo at tummy.com (Sean Reifschneider) Date: Thu Dec 30 18:19:15 2004 Subject: [Catalog-sig] PyCon sprint for PEPs 314 and 243? In-Reply-To: <200412301556.28923.richardjones@optushome.com.au> References: <200412110920.27461.richardjones@optushome.com.au> <200412301535.34522.richardjones@optushome.com.au> <200412301556.28923.richardjones@optushome.com.au> Message-ID: <20041230171827.GA11064@tummy.com> On Thu, Dec 30, 2004 at 03:56:28PM +1100, Richard Jones wrote: >. a download mechanism including using mirrors. Is there any reason not to see if "yum" can be modified to do this? They have a lot of things figured out as far as mirror lists, downloading, and package lists. Sean -- Electronic ways to get in touch with Canadians? Don't they generally involve a moose and an antenna? Sean Reifschneider, Member of Technical Staff tummy.com, ltd. - Linux Consulting since 1995. Qmail, Python, SysAdmin From bob at redivi.com Thu Dec 30 18:53:48 2004 From: bob at redivi.com (Bob Ippolito) Date: Thu Dec 30 19:09:08 2004 Subject: [Catalog-sig] PyCon sprint for PEPs 314 and 243? In-Reply-To: <20041230171827.GA11064@tummy.com> References: <200412110920.27461.richardjones@optushome.com.au> <200412301535.34522.richardjones@optushome.com.au> <200412301556.28923.richardjones@optushome.com.au> <20041230171827.GA11064@tummy.com> Message-ID: On Dec 30, 2004, at 12:18 PM, Sean Reifschneider wrote: > On Thu, Dec 30, 2004 at 03:56:28PM +1100, Richard Jones wrote: >> . a download mechanism including using mirrors. > > Is there any reason not to see if "yum" can be modified to do this? > They > have a lot of things figured out as far as mirror lists, downloading, > and > package lists. Yum is GPL. -bob From amk at amk.ca Thu Dec 30 19:27:57 2004 From: amk at amk.ca (A.M. Kuchling) Date: Thu Dec 30 19:28:19 2004 Subject: [Catalog-sig] PyCon sprint for PEPs 314 and 243? In-Reply-To: <200412301535.34522.richardjones@optushome.com.au> References: <41B9F015.8090803@colorstudy.com> <200412110920.27461.richardjones@optushome.com.au> <200412301535.34522.richardjones@optushome.com.au> Message-ID: <20041230182757.GA31617@rogue.amk.ca> On Thu, Dec 30, 2004 at 03:35:30PM +1100, Richard Jones wrote: > So, is anyone interested in getting together before PyCon for a sprint to > finalise and/or implement PEPs 314, 243 (I guess we should ask for some PSF > money, though we've obviously missed the most recent grant allocation round) > and possibly 262 (I consider this optional and don't believe it should get in > the way of implementing the other two)? I was thinking of organizing another Bug Day in January; perhaps it should be a Catalog-SIG sprint day instead. Traditionally they've been on Saturdays; Richard, are you free on any of the weekends in January? Anyone else want to participate, but will be unavailable on one or another weekend in January? Regarding PEP 314: the most important question I have right now is: is it worthwhile to continue with the PKG-INFO, or should we use DOAP instead? If we stick with PKG-INFO, then we can finish up 314 pretty quickly because the list of open issues is pretty short (unless more issues get raised). If the latter, then we need to figure out what shape DOAP support would take? I agree that PEP 262 is a side issue. --amk From richardjones at optushome.com.au Fri Dec 31 10:28:35 2004 From: richardjones at optushome.com.au (Richard Jones) Date: Fri Dec 31 10:28:39 2004 Subject: [Catalog-sig] PyCon sprint for PEPs 314 and 243? In-Reply-To: <20041230171827.GA11064@tummy.com> References: <200412301556.28923.richardjones@optushome.com.au> <20041230171827.GA11064@tummy.com> Message-ID: <200412312028.35688.richardjones@optushome.com.au> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Fri, 31 Dec 2004 04:18 am, Sean Reifschneider wrote: > On Thu, Dec 30, 2004 at 03:56:28PM +1100, Richard Jones wrote: > >. a download mechanism including using mirrors. > > Is there any reason not to see if "yum" can be modified to do this? They > have a lot of things figured out as far as mirror lists, downloading, and > package lists. Without looking into it too closely, it seems like yum is closely tied to RPMs and their associated package databases. Also, I'm not sure I want to try to solve dependencies. Richard -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) iD8DBQFB1RvDrGisBEHG6TARAku+AJ9WnmaSIVQbG5b00C8qiVtsBB1wZACdHPLK Ke/660bvYq24oyS201eKVwk= =bF09 -----END PGP SIGNATURE----- From richardjones at optushome.com.au Fri Dec 31 10:34:50 2004 From: richardjones at optushome.com.au (Richard Jones) Date: Fri Dec 31 10:34:55 2004 Subject: [Catalog-sig] PyCon sprint for PEPs 314 and 243? In-Reply-To: <20041230182757.GA31617@rogue.amk.ca> References: <200412301535.34522.richardjones@optushome.com.au> <20041230182757.GA31617@rogue.amk.ca> Message-ID: <200412312034.50767.richardjones@optushome.com.au> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Fri, 31 Dec 2004 05:27 am, A.M. Kuchling wrote: > On Thu, Dec 30, 2004 at 03:35:30PM +1100, Richard Jones wrote: > > So, is anyone interested in getting together before PyCon for a sprint to > > finalise and/or implement PEPs 314, 243 (I guess we should ask for some > > PSF money, though we've obviously missed the most recent grant allocation > > round) and possibly 262 (I consider this optional and don't believe it > > should get in the way of implementing the other two)? > > I was thinking of organizing another Bug Day in January; perhaps it > should be a Catalog-SIG sprint day instead. Traditionally they've > been on Saturdays; Richard, are you free on any of the weekends in > January? I may have something on the 15th and 16th, but otherwise I could be free. Note that I'm in UTC+11 at the moment. > Regarding PEP 314: the most important question I have right now is: is > it worthwhile to continue with the PKG-INFO, or should we use DOAP > instead? I admit to not having looked really closely at DOAP - on my initial glace my brain kicked into its "I need a lot of time to absorb all this" mode and kinda paged out on me. PKG-INFO isn't used in PyPI at the moment - it crafts its own HTTP request. At the moment I believe it's only used as a file in the source distributions. You might want to ask the broader comminuty whether anyone's actually using it for anything. Having said that, IIRC DOAP is a bit of a shift from the actual meta-data spec of the PKG-INFO - that has more ramifications. As I mentioned, I still haven't looked into it in detail. Richard -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) iD8DBQFB1R06rGisBEHG6TARAq3qAJwLcFCYFZbeMEI1qI5G+DgEW2MVaACcCryX 2vW+NY1jM7v49sXBeNCUKVM= =6hGu -----END PGP SIGNATURE----- From bob at redivi.com Fri Dec 31 10:48:10 2004 From: bob at redivi.com (Bob Ippolito) Date: Sat Jan 1 14:49:55 2005 Subject: [Catalog-sig] PyCon sprint for PEPs 314 and 243? In-Reply-To: <200412301535.34522.richardjones@optushome.com.au> References: <41B9F015.8090803@colorstudy.com> <200412110920.27461.richardjones@optushome.com.au> <200412301535.34522.richardjones@optushome.com.au> Message-ID: <14E239E4-5B11-11D9-95EA-000A9567635C@redivi.com> On Dec 29, 2004, at 11:35 PM, Richard Jones wrote: > So, is anyone interested in getting together before PyCon for a sprint > to > finalise and/or implement PEPs 314, 243 (I guess we should ask for > some PSF > money, though we've obviously missed the most recent grant allocation > round) > and possibly 262 (I consider this optional and don't believe it should > get in > the way of implementing the other two)? > > Metadata for Python Software Packages v1.1 > http://python.org/peps/pep-0314.html > Module Repository Upload Mechanism > http://python.org/peps/pep-0243.html > Database of Installed Python Packages > http://python.org/peps/pep-0262.html > > I'd want to revise PEP 243 at some stage. I want to *only* implement > source > distribution uploading for now. Also, there's some aspects of the > protocol > that need work (eg. we should use standard HTTP responses, not > X-Swalow-Status). I'm interested in helping this along. Similar to your revisions with PEP-243, here's what I think should happen to PEP 314: - It should be required to list every "top-level" package/module that is installed as Provides fields (else distutils will do it for you). "top-level" isn't absolute, as there will be cases like zope.interface. By top-level I mean the highest level that is to be installed by this setup. - Every Provides must provide their own version (for Requires), distutils will default these to the global version if not explicitly specified. - Requires should not be arbitrary. They should be exactly Python module names. It just gets too messy otherwise. Having such a meta-requirement is completely and utterly useless without also having PEP-262 implemented and accessible at runtime. If you say that a package requires *some* DBI API 2.0 driver, how would you find one? In order to support this sort of use case, it might be worthwhile to have a way to say Requires "foo (>1.0) OR bar (>1.2)" (say a package that's compatible with PEAK or Twisted, but needs one or the other). -bob