[Python-ideas] Anonymizing the PyCon review process

[Terry Jones]

Last night I got a couple of PyCon talks rejected, and someone else sent me
a rejection email they'd received. I wasn't surprised at the rejections,
but I was quite surprised that many of the review comments were at least in
part based on the presenter (sometimes the incorrectly assumed presenter)
instead of on the proposed talk.

Out of the 14 reviews, 5 of them have comments about the author of the
proposal. I'll just give 2 examples here, taken from 2 different reviewers
on 2 different proposals.

1. Imagine you're a relatively unknown Python programmer, you submit a talk
to PyCon, and get back a review whose first sentence reads:

  I don't know the reputation of this particular speaker, so I won't "+1"

That sends a pretty unfortunate message: Because you're not a recognized
Python person, I wont give you the thumbs up. Maybe I'm being naive or
simplistic, but I'd have hoped one route to becoming recognized would be by
giving a PyCon talk. From the POV of the review recipient the leading
justification for a non-recommendation has nothing to do with the talk!

2. What if you gave a PyCon talk in an earlier year that wasn't rated as
highly as other talks? You send in a PyCon proposal, and get this back:

    I like XXX but honestly his talk at pycon 09 went poorly.

That was the *entire* review in this case. What's the message here? Sounds
like: well, you gave a talk once and it wasn't so great, so part of my vote
against your proposal is because of that. It's like telling people to go
away and not bother ever submitting again. Again, it's that's not based on
the current talk proposal. People tend to get better at giving talks. If a
proposal's content is technically good enough to get in, let them give
another talk and help them to make it better. Just when is XXX supposed to
re-send another PyCon talk, if ever?  What makes this even worse is that
XXX was not even the primary author of the proposal, and was not to be the
speaker. So here we have a review that's negative *entirely* due to a talk
that someone else gave in a previous year. How discouraging.  Should the
person in the future "take one for the team" and decline to be listed as a
co-author on joint proposals - even though they're not going to speak -
fearing that a reviewer will reply with a -1 and a one-line dismissal?
That's the unfortunate dynamic that the above "review" has created.


I hope this doesn't sound like personal sour grapes. It's not at all. I've
had *tons* of rejection letters in my life (see http://bit.ly/1xytIr),
including from PyCon.  They're water off a duck's back at this point :-) I
do however care about Python and the Python community.

The most important point is the message that's sent back to aspiring
speakers.  Reviews that are based on the supposed character, or old talks,
or how recognized you are or aren't, or on a guess as to which of multiple
authors might be doing the presenting - all of those send a bad message.
They make PyCon look insular and cliquey. If the committee of people is (or
merely gives the impression of being) inwardly focused, the community and
in the longer term perhaps the language itself will suffer through reduced
diversity and through discouraging precisely the people who are animated
enough and have the initiative and ambition to submit talks. Those are
*exactly* the wrong folks to discourage.

The obvious suggestion is to anonymize the review process. That's standard
in mature conferences. It doesn't eliminate bias (in fact you *don't want*
to eliminate bias - you need it to survive, you need it to assess quality),
but it does reduce the opportunity for judgment based on the wrong things.
When I say "wrong" I mean: if you're going to judge based on stuff that's
not just the proposal content, then ask for a CV, or a speaking record, or
whatever you intend to consider in the review process.

Anonymizing conference reviewing has healthy effects. I've seen it up close
in academic circles. It's like a breath of fresh air and the results are
surprising.  When they did it in the genetic algorithms world, all of a
sudden really interesting talks were being accepted from all over the world
and many very experienced researchers were having multiple talks rejected.
That was unexpected, refreshing, and generally agreed to be a very healthy
and embracing/welcoming move.

If PyCon doesn't move to anonymizing reviews, then at least *try* not to
base acceptance decisions on who a speaker is (or, worse, who it's presumed
to be).  If for some reason you have to, it's *perhaps* better not to tell
the poor submitter that they're being rejected in part based on who they
are or aren't.  The CFP requests a talk proposal, and that's what should be
reacted to in the review response, even if there's more to the story. Some
of the comments above *might* be appropriate for a conference committee
meeting, but not for the first (or only!) line of a review.

OK, rant over :-) Regards to everyone & thanks for all the PyCon work. I
know how much work it is, and that it's not easy. I hope to be able to make
it to Atlanta.

Terry Jones