[Catalog-sig] Please turn off ratings

[P.J. Eby]

At 11:33 AM 4/6/2011 -0400, Brian Jones wrote:
>I think if PyPI had a ratings system that more closely matched some 
>kind of (perhaps updated) consensus on what would be "useful", this 
>conversation could focus more on what that looks like and who might 
>implement it, which would be more constructive IMHO. Attempting to 
>completely eradicate a feature by trying to collect anecdotal input 
>doesn't seem like the way to go.
>
>Why don't we start a new thread about what a good ratings system 
>looks like and try to get actual consensus from it so real work can get done?

Better still - let's discuss the actual use cases for a rating 
system, before trying to design one.

So far, the people who've stepped up here say they want to warn 
people about "bad or broken" packages, or ones that are unmaintained, 
or where the author is unreachable.

This doesn't sound so much like a use case for a rating/comment 
system, as it does for a reporting mechanism for unreachable authors, 
or maybe something else.

Side note: the ratings system's way of summarizing things doesn't 
make much sense either; setuptools currently rates around 2.36, with 
7 people voting 0 or 1, and 7 voting 4 or 5 -- this means that the 
summary is equally objectionable to *both groups*...  too high for 
the people voting low, too low for the people voting high!  So, as 
Jacob's been saying, the number seems pretty meaningless.

(This is a general problem with ratings of controversial topics, but 
it's especially likely given that currently, only super-exposed *and* 
controversial packages seem to *get* reviews at all.  None of my 
other packages has received a single rating, AFAICT, despite many 
having download counts in the tens of thousands.)


>I'd be happy if someone came up with a system that was useful enough 
>to justify its existence. Maybe there are just particular features 
>of the existing system that make it hard or unintuitive for end 
>users to use in a meaningful way, which then makes any resulting 
>'rating' useless to anyone else who happens by.

The primary problem is that it's set up as an essentially 
moderator-free forum, so comment quality is low.  People ignore bug 
reporting links and report them in comments anyway, and there's 
nothing the package author can do about it.

One then ends up with a public bug-solving thread on the 
package...  which then communicates to visitors that this is in fact 
the way to get a problem solved, thereby encouraging more of the same.

In short, the social design of the feature is poor - a bad attempt at 
a technical solution to a fundamentally social problem.  Its only 
saving grace is that it is *also* difficult to use (by requiring a 
PyPI account), so it doesn't get used very much.  ;-)

Note also Laura Creighton's comments regarding people wishing for 
ponies... the people who want comments on PyPI are essentially two groups:

1. People who want to leave negative comments
2. People who want to read other people's comments

I haven't yet seen anybody say, "I want the comment feature because I 
want to post my in-depth evaluations and positive 
experiences"...  which should be a good indicator of what sort of 
comments we should expect to see on packages.

Essentially, this is because the people who actually care enough to 
post an informative review, will do so on their personal blogs.

My guess is that, if PyPI ratings or reviews *required* a link to a 
blog posting, and had to be a summary or excerpt of that posting, 
approved by a human moderator, and with the ability for anyone 
(including the package author) to report that the link has become 
broken (and to thus get the rating/review removed), then you'd 
actually have a useful review/rating system.

The persons who want negative reports could still write them, but 
they'd first have to do so on their own blog, where it's more 
directly associated with their online persona, and subject to other 
feedback mechanisms (such as comment on the blog).

And it would discourage bug reports, because, well, who puts a bug 
report on their blog and expects to get an answer there?

The big downside of course is that somebody would have to run 
this.  It couldn't just be an unattended, hands-free system.



>Regardless, I suspect some of the vitriol toward the existing system 
>is really about features of the system, and not about the existence 
>of the system. Catalogs have ratings systems. I'd argue that a lot 
>of sites that have ratings systems are, in fact, catalogs to one 
>degree or another. Including Amazon.

The difference is that those systems have moderation mechanisms, and 
moderators...  they have filters, appointed moderators, and 
crowd-sourced voting.

And on Amazon, there are *still* people who post eBay-style feedback, 
"Shipped fast, would buy again" and other such crap on a product 
review, because they've completely missed the point of Amazon reviews.

Doing a quality rating system means there needs to be somebody who:

1) actually cares about the ongoing quality of the reviews, and
2) has the ability to *do* something about it (like trimming discussions)

(Oh...  and on Amazon, manufacturers can post responses to reviews, 
but the reviewer doesn't get to respond to that, ad infinitum.  It's 
just review+response, with an additional off-page comment system for 
further discussions...  which usually aren't of much value as a 
"review" system, which is why they're off-page.)


>Jacob, what would make you *want* those emails you're getting as a 
>package owner? What would make users *want* to leave feedback that 
>would be useful to maintainers & other users?

I think this is mistaken -- reviews are not author feedback.  As a 
package author, I want people to report problems to wherever I've 
asked them to report problems: a bug tracker or mailing list.  If 
they want to say something nice, well, why not on their blog?