We're going through the PyCon '09 review process, and participating in the process has been pretty interesting. (I joined the Program Committee in large part because I was told to put up or shut up after I critiqued PyCon '08. Ahh, the open source world... where you're encouraged to go fix things when you complain :). In particular, this is the first review process I've seen where regular communication between the reviewers and authors is expected, and proposals are modified in response to reviewer comments.

There are a couple of drawbacks to this process. One is that there's no clear boundary between reviewer opinion and expectation. It's one thing to say "I don't understand X, Y, or Z in your proposal; could you clarify, please?" and another to say "I don't agree with X, Y, or Z, and I won't push your proposal unless you change your views." The former seems pretty legit, but the latter strikes me as being counter to the conference ethos of encouraging diversity in views. While I don't think anyone has been that explicit, there have been extensive conversations between reviewers and authors that have had much the same effect...

Another drawback of this system is that authors can express their frustration at reviewer comments pretty directly. Sometimes this frustration is legit, but other times it's hysterically off-base; you don't make any friends when you tell your reviewers that they're idiots (just as one purely hypothetical example...)

On the flip side, I think several proposals have been dramatically improved through reviewer feedback. I don't know how well this kind of insta-review process might work for academic journals -- I believe PLoS One is trying it out? -- and I'll be watching some of the early experiments with interest.

Paranthetically, let me add that we have a bunch of great proposals, and Ivan Krstic is doing a fantastic job of running things! So I expect PyCon '09 to be a very good conference.

--titus

Legacy Comments

Posted by Doug Napoleone on 2008-12-03 at 10:35.

I know you already know about all I am going to type here, but other
people might not be aware so... (also feel free to edit or redact any
part you wish)    The boundary between opinion and expectation is
defiantly blurred in some instances. To try to counter act for the 'I
want you to change your opinion' types of reviews and comments, we
have at least 3 reviewers per proposal, and the reviews are not what
determines if a talk gets accepted or not.     The (rather lengthy)
IRC meetings going over the proposals together is where we as a group
whittle down the proposals, and attempt to ensure that no one is
gaming the system. Ivan is doing a fantastic job keeping things
focused and getting the job done. The interesting work will be
starting shortly now...    As for your purely hypothetical example, if
an author behaves in a 'hysterically off-base' way, isn't that good
information to have as a reviewer?     PyCon is quite unlike other
pier-review based entities, in that we are about helping and promoting
the community. We want to help make things better as that is to the
benefit of all. In the past we have even selected talk proposals to be
submitted to other conferences like PyWorks and OSCon. These were
great proposals which we were forced to decline due to limited
space/time, or similar reasons. As such even the work put into helping
an author make a proposal better for even a declined proposal is not a
wasted effort by any stretch of the imagination.    Lastly, no system
is perfect. We are all just volunteers trying to do the best we can.
We tried to set up as many checks and balances as possible while
maintaining a community feel to the process. None of us gets paid to
spend the tens of hours we put into the program committee.     Thank
you very much for stepping forward. Part of the 'put up or shut up'
response from last year was because the PC was simply overwhelmed. We
asked for help from the community and that help (at the time) did not
arrive. We underestimated the growth of PyCon (400->1000 in 2
years) and people were busy and had other commitments. Late November
is a bad time to try to get volunteer help. No ones fault really, it
just happened. As such a battered PC responded to some criticisms with
'don't like it? step up and change it, it's your conference...',
usually the unfounded ones ;-)    We have a fantastic group in the
program committee this year, and I hope to get more iconic pythonistas
to take part next year as well. There were a number of people whom had
to decline due to a lack of availability this year which I hope will
not be the case next year.