We're going through the PyCon '09 review process, and participating in the process has been pretty interesting. (I joined the Program Committee in large part because I was told to put up or shut up after I critiqued PyCon '08. Ahh, the open source world... where you're encouraged to go fix things when you complain :). In particular, this is the first review process I've seen where regular communication between the reviewers and authors is expected, and proposals are modified in response to reviewer comments.
There are a couple of drawbacks to this process. One is that there's no clear boundary between reviewer opinion and expectation. It's one thing to say "I don't understand X, Y, or Z in your proposal; could you clarify, please?" and another to say "I don't agree with X, Y, or Z, and I won't push your proposal unless you change your views." The former seems pretty legit, but the latter strikes me as being counter to the conference ethos of encouraging diversity in views. While I don't think anyone has been that explicit, there have been extensive conversations between reviewers and authors that have had much the same effect...
Another drawback of this system is that authors can express their frustration at reviewer comments pretty directly. Sometimes this frustration is legit, but other times it's hysterically off-base; you don't make any friends when you tell your reviewers that they're idiots (just as one purely hypothetical example...)
On the flip side, I think several proposals have been dramatically improved through reviewer feedback. I don't know how well this kind of insta-review process might work for academic journals -- I believe PLoS One is trying it out? -- and I'll be watching some of the early experiments with interest.
Paranthetically, let me add that we have a bunch of great proposals, and Ivan Krstic is doing a fantastic job of running things! So I expect PyCon '09 to be a very good conference.
Posted by Doug Napoleone on 2008-12-03 at 10:35.
I know you already know about all I am going to type here, but other people might not be aware so... (also feel free to edit or redact any part you wish) The boundary between opinion and expectation is defiantly blurred in some instances. To try to counter act for the 'I want you to change your opinion' types of reviews and comments, we have at least 3 reviewers per proposal, and the reviews are not what determines if a talk gets accepted or not. The (rather lengthy) IRC meetings going over the proposals together is where we as a group whittle down the proposals, and attempt to ensure that no one is gaming the system. Ivan is doing a fantastic job keeping things focused and getting the job done. The interesting work will be starting shortly now... As for your purely hypothetical example, if an author behaves in a 'hysterically off-base' way, isn't that good information to have as a reviewer? PyCon is quite unlike other pier-review based entities, in that we are about helping and promoting the community. We want to help make things better as that is to the benefit of all. In the past we have even selected talk proposals to be submitted to other conferences like PyWorks and OSCon. These were great proposals which we were forced to decline due to limited space/time, or similar reasons. As such even the work put into helping an author make a proposal better for even a declined proposal is not a wasted effort by any stretch of the imagination. Lastly, no system is perfect. We are all just volunteers trying to do the best we can. We tried to set up as many checks and balances as possible while maintaining a community feel to the process. None of us gets paid to spend the tens of hours we put into the program committee. Thank you very much for stepping forward. Part of the 'put up or shut up' response from last year was because the PC was simply overwhelmed. We asked for help from the community and that help (at the time) did not arrive. We underestimated the growth of PyCon (400->1000 in 2 years) and people were busy and had other commitments. Late November is a bad time to try to get volunteer help. No ones fault really, it just happened. As such a battered PC responded to some criticisms with 'don't like it? step up and change it, it's your conference...', usually the unfounded ones ;-) We have a fantastic group in the program committee this year, and I hope to get more iconic pythonistas to take part next year as well. There were a number of people whom had to decline due to a lack of availability this year which I hope will not be the case next year.