[Catalog-sig] Back to RPC

[Ian Bicking]

Richard Jones wrote:
> [Barry - SVN question for you buried down below]
> 
> On Mon, 23 May 2005 08:34 am, Ian Bicking wrote:
> 
>>I think all the information I really need is how to get an sdist
>>package, or a checkout -- right now it's important to be to be able to
>>install out of repositories, not just released versions.
> 
> 
> There's something that's not been on *my* radar :)
> 
> Do we really want this in PyPI's database? Related: will people really want to 
> *maintain* this stuff in PyPI's database?

Well, it's a pretty stable URL for most projects, more stable than the 
download URL.  Also, if we have tools that use the data, they *have* to 
be maintained, so they will be maintained ;) -- but it's also why I'm 
concerned that these things are correctable regardless of the presence 
or interest of the package "owner".  There will be workarounds -- I'd 
rather those happen in PyPI than to build those into external tools and 
external annotations of the index.

>>So, I think all the data is there now.  I might search package_urls for
>>the right release, packagetype 'sdist'.  Though there isn't any place
>>for a repository URL, and that gets a little vague -- in theory you
>>really could have a per-version repository in the form of a branch.  But
>>that's not very useful information.  But maybe that can be disambiguated
>>with a URL "svn-trunk" or "cvs-trunk", which doesn't imply that the
>>repository is version-specific, even if the database schema requires it
>>be put there.
> 
> 
> Well, as soon as you start talking SVN *and* CVS you talking about totally 
> different meta-data. For SVN, you've got just a URL (in my understanding of 
> SVN's branching/tagging) and for CVS you've got a ... well, it's not quite a 
> URL, and a tag name.

Yeah, it's too bad there aren't real URLs for CVS.  You could do 
pserver:hostname/path.../modulename, or something like that, but it's 
just making it up.

> And of course, this ignores the plethora of other revision-control 
> repositories that people use.

Yes, but it's a process.  Maybe darcs-trunk could be added, and so on. 
Right now I'm actually only concerned about svn, because all the 
projects I care about installing automatically are in svn.  I wouldn't 
be implementing anything else in my own client libraries.

It's a problem for me, because I'm starting to maintain or be involved 
with several very interrelated projects that are most useful when used 
in concert, but have valid independent existances.  So if I'm working 
with any one of them outside of a released version, there's a good 
chance all of them need to be installed from checkouts.  (Incidentally, 
I'm not expecting to install any of these globally; I've pretty much 
given up on globally installing any but a handful of non-stdlib packages)

> So *perhaps* you just want to mandate SVN and just deal with losing all the 
> potential Sourceforge contributions? I'm not sure what it is you're actually 
> trying to do, so I couldn't say :)

Sure.  It's a process; if there's demand (and code), then there's 
nothing stopping CVS repositories from having their access encoded into 
a single URI-like string.  There's a lot of projects still in CVS, but 
not a lot of the immature projects where installing non-released 
versions is particularly useful.

>>From there it's all client-side work to install the package, so it
>>doesn't involve PyPI.  The only issue is giving URLs to packages that
>>don't have URLs.  It would be a lot easier from my perspective if I
>>didn't have to go through the package author when URLs were missing,
>>maybe with notification going back to the owner so they can adjust their
>>setup.py file.  But lets ignore that for a little while, though it's
>>also very important to me, as I'd like to use this stuff sooner rather
>>than later.
> 
> 
> URLs/uploaded files for all registered packages is something I don't think 
> we'll ever achieve.

If we make it matter, that will help a lot.

>>Anyway, summarizing:
>>
>>* Agree and document release_urls.packagetype values.  I'd like:
>>   * sdist
>>   * bdist
>>   * bdist_dumb
>>   * bdist_rpm
>>   * bdist_wininst
>>   * bdist_egg
>>   * svn-tag
>>   * svn-trunk
>>   * cvs-trunk (is there any agreed-upon format for CVS URLs?  What about
>>     branches and tags?)
>>
>>Also, these really need to be documented somewhere on the pypi site,
>>that's the only way agreeing on it means anything ;)
> 
> 
> They should be listed in the PEP (or non-PEP) design document you're writing 
> (hint hint <wink>).

Oh, sure, I guess I can do that.  Does the PEP process really add much 
here, or would a document be sufficient?  The presence of PyPI 
documentation in the PEP index is actually a bit unhelpful, compared to 
documentation sitting right in PyPI (and maybe besides distutil-related 
documentation).

>>* Add some functions to rpc.py:
>>   * package_releases(package_name): list of release versions, as strings
>>   * package_stable_version(package_name): packages.stable_version
>>   * package_urls(package_name, version): A list of {'url': url,
>>     'packagetype': packagetype}
>>   * package_data(package_name, version): A dictionary that basically
>>     summarizes the releases table, plus release_classifiers.
>>   * providing_packages(specifier): A list of (name, version) from
>>     release_provides.
>>   * requiring_packages(specifier): A list of (name, version) from
>>     release_requires.
> 
> 
> Good list.
> 
> 
> 
>>I don't entirely understand "specifier" and "release_obsoletes". Are 
>>specifiers package names, or a slightly more abstract version of package
>>names?  Or something more structured?
> 
> 
> See the PEP: http://www.python.org/peps/pep-0314.html

OK.  I won't plan to do much with this information right now, but maybe 
if there's easy access to the information someone will take up the cause.

>>While it isn't important to me now, I think it would be good to change
>>rpc.search to take a single dictionary argument, which it would pass in
>>as the query spec.  Probably the return value should also be cleaned up,
>>as store.query_packages returns a kind of odd data structure.
> 
> 
> Sounds fine to me. As I understand it, the code there is more of a 
> proof-of-concept.

Certainly, there's not a useful list of functions currently.

>>The adding URL thing is a bit more of a policy decision.  Maybe we can
>>just add a form to request a role, which will email the owner and give
>>them a quick form (maybe just one link) to add the user.
> 
> 
> Er, so this is someone asking to have permission to add the URL on behalf of 
> the package owner?
> 
> Seems reasonable to me, I suppose.

Yes, something like that.  Maybe just building a contact form into the 
package display (or linked off that) is sufficient, that would mail all 
owners and maintainers of a package (with language in the form to 
indicate that only metadata errors should be reported through that form).

Later, maybe something could be added to record these issues, let the 
owners mark them "resolved", and put out whiney messages to owners if 
they don't do resolve them, and maybe alert "editors" eventually.  But 
just a contact form is a start.

>>If I can get CVS commit access I can start working on some of these; a
>>recent database dump would also be useful for testing.
> 
> 
> PyPI's just moved to svn.python.org, but I couldn't say more than that 'cos I 
> don't know how it works. I've cc'ed Barry Warsaw in case he's not on this 
> list.

OK.  Could I get CVS access too? (username: ianbicking)

>>Incidentally, I notice the PyPI page is quite slow.  Thoughts?  I think
>>I'm noticing that templates are loaded off disk for each request, which
>>be really slow.  Is PyPI running under mod_python now?
> 
> 
> It's being moved right now. AMK's in the final stages of apache config.

OK, then the speed doesn't really mean anything, and if it's CGI there's 
no useful caching of templates that can be done.  But I can look at that 
too.

> Something that concerns me about all this is the referencing of older versions 
> of software. I can't help this niggling feeling I have of people down the 
> track asking package developers to fix bugs in ancient branches of their 
> code. And people on both sides of the equation getting snarky about it.

Well, you can't predict or solve all future problems ;)  I assume most 
of the client tools will heavily prefer the "stable version" of the package.

-- 
Ian Bicking  /  ianb at colorstudy.com  / http://blog.ianbicking.org