[Catalog-sig] Re: Meeting followup- CPAN for Python?

[Ian Bicking]

Richard Jones wrote:
> On Sat, 11 Dec 2004 10:22 am, Ian Bicking wrote:
> 
>>Richard Jones wrote:
>>
>>>On Sat, 11 Dec 2004 05:51 am, Ian Bicking wrote:
>>>
>>>>* Track package names into PyPI.  For now these will serve as
>>>>identifiers for installation and dependencies.  Conflicting package
>>>>names already cause a lot of problems anyway, so it would be nice if
>>>>they'd be unique.
>>>
>>>This step is done. In PyPI, a name is owned by someone (or some people).
>>
>>True.  The downside is that it isn't the Python package name, it's just
>>a name. The Python package name seems like a natural identifier; though 
>>some projects have no packages (e.g., many applications), and some have
>>multiple packages.
> 
> 
> Ah yes - setup.py name != python package name. This has caused confusion -- 
> some people accidentally submit packages to pypi with *their* name in the 
> "name" field of setup.py...
> 
> In CPAN (and PAUSE) there's no such disinction, IIRC.
> 
> The "provides" field in metadata 1.1 solves this problem, I think. Ownership 
> of names would flow through.

OK, that sounds good.

>>It also needs to support multiple kinds of downloads.  Freshmeat has
>>this, though the kinds of downloads are fixed, which seems unnecessary
>>for us.  System-specific packages are usually prefered if available.
>>Well, sometimes -- usually system-specific packages are harder to use
>>for multi-version installs.  Anyway, there's a place for several kinds
>>of packages.
> 
> 
> How about we start with just source-url and go from there?

A dictionary seems like such a simple extension, and multiple download 
links are useful for a good many packages already.  The keys would just 
be strings, "windows-installer", "rpm", "source".

> We'll probably want some system somewhere that verifies that the source url 
> works. Doing that at registration time might be a little too slow.

OK, internally I guess there should be a flag "not-checked", "working", 
"broken".  Later if we do backups we could use this to signal when we 
need to replace the link with the backup link.

>>>It also removes Platform, and specifies the
>>>formatting of the long description field. The only open issue in that PEP
>>>is the license field. I believe some people would like it to remain, but
>>>only be used when their license doesn't appear in the Classifiers list,
>>>and they need to include the full text of their license (or some other
>>>statement about it). See PyPI bug 693471.
>>>
>>>Also of interest:
>>>
>>>Module Repository Upload Mechanism
>>>  http://python.org/peps/pep-0243.html
>>>
>>>
>>>>* Add a field for source download; maybe make it a dictionary, so you
>>>>can give several links for different sources (e.g., source tarball/zip,
>>>>rpm, windows installer, debian package, etc).
>>>
>>>I think just sticking with the source installer is fine. This is all
>>>invoked by setup.py, isn't it?
>>
>>Well, by phrasing this in terms of data, not functionality, we make this
>>potentially frontend-neutral.  We can already generate rpm's from
>>distutils, and I don't know why debs haven't also been added to that.
>>In many cases they would be preferred.
> 
> 
> Do the existing RPM / exe installers invoke setup.py to do the install? If 
> they don't how will we fire the appropriate package database update (aka PEP 
> 262)?

No, I don't think any of them are based on setup.py, at least at the 
point of installation.  Instead setup.py captures all the necessary 
build information when the package is created.

What would have to happen is these systems would have to run the PEP 
262-updating scripts after installation had finished.  It's not really a 
very big difference, since it'll probably all be the same code any way 
it gets run, just slightly different orders.

>>>>* Create a script that can query PyPI, get the link(s), then download
>>>>it.  PyPI already has an XML-RPC interface, I believe.
>>>
>>>PyPI's web interface is a complete shambles. I really want to reimplement
>>>it in something easier to maintain, and more extensible. As an aside: the
>>>pypi sqlite database really needs to move to postgresql.
>>
>>Would that be hard?  Is it mostly a sysadmin issue, or a coding issue?
> 
> 
> Not hard. Needs time, which I don't have.
> 
> 
> 
>>What do you want to change in PyPI's interface?
> 
> 
> It's a hand-coded, unextensible mess. Mostly, it runs as a cgi-bin, which 
> really needs to change to mod_python.

Or to... WSGI!  It'd probably be a smoother transition than mod_python, 
since WSGI looks a lot like cgi.  That's something I might find 
interesting to try.  Well... this is also a distraction from my own 
projects; it's always a difficulty with these things for me.  I'm sure 
you know the feeling.

So... one underlying motivation is getting some of the other people at 
ChiPy involved; there was a lot of discussion about this, and people 
following up on that discussion.  It seems like a good project.

Looking at the code, it looks like simply moving HTML out to templates 
would help quite a bit.  Then moving to Postgres, which would probably 
be easy enough.  Well, this is the bread and butter of my day work, so 
it seems easy enough to me... I'm not sure what other people feel.  Has 
anyone else checked out pypi and given it a look?  To checkout:

cvs -d:pserver:anonymous at cvs.sourceforge.net:/cvsroot/pypi login
cvs -z3 -d:pserver:anonymous at cvs.sourceforge.net:/cvsroot/pypi co pypi

Then to get it started...

I made a directory /tmp/pypi, and copied config.ini into it, editing it 
a bit.  I made sure pypi.cgi was available through Apache.  It may take 
some configuration to get .cgi files to be run for you.  I had to make 
sure the directory the database was located in was writable by the web 
server.

Lastly, it would be useful to have a full database available.  Is there 
a place we can pick up a dump of the pypi database?  It doesn't have to 
be very fresh.

>>Well, people here at ChiPy have been showing some interest, and most of
>>them probably won't be going to PyCon.  So people might want to try
>>tackling some of these things on a different schedule.  And the PyCon
>>sprints won't be during the weekend, so that would also make it hard for
>>non-PyCon people to participate.
> 
> 
> OK.
> 
> Mostly what's needed is a thorough design in a PEP - then just about anyone 
> could do the small amount of coding by themselves. A PSF grant could help...

What portions do you think need PEPs?  The internal design of PyPI 
doesn't seem to be a PEP concern.  The added metadata is a PEP concern, 
I guess.

-- 
Ian Bicking  /  ianb at colorstudy.com  / http://blog.ianbicking.org