My notes for a panel presentation on reviewing papers and grants

On January 27th, 2014, the MSU BEACON Center graduate students held a panel on how to review. The panel was organized by Emily Weigel (@choosy_female) and Jory Schossau, Bjorn Ostman (@CarnyEvolution), Kristin Parent, Rich Lenski (@RELenski), and Arend Hintze were panel members, together with me.

I put together the following notes, somewhat amended in response to Rich, Kristin, Bjorn, and Arend's comments, and the students' questions. I hope you find them useful!

Note that we talked about both grant and paper reviews, and Rich could provide additional perspective on paper reviews because he is an editor of PNAS.


Goals as reviewer

Your goals on review should be:

  • help editor/panel make good decision
  • help authors write a good paper

Titus's approach to reviewing

  1. Skim-read the grant or paper; highlight areas that don't make sense.

  2. Decide if you are pro-, anti-, or on the fence.

  3. If pro, try to write a review that objectively states why. Pay special attention to framing in a larger context, if you can.

  4. If anti, identify major flaws clearly and objectively. Behave as if the authors know who you are and where you live.

  5. If on the fence, look harder for support one way or another; if you can't find, explain that you're on the fence, and why.

  6. I structure my reviews like so:

    Broad overview: why this is important, or not.

    Big-picture problems ("they failed to convince me of X, Y or Z")

    Nits to pick (bad figures; spelling; grammar; etc.)

Note that generally editors want recommendations separate from your text, so that they don't get into arguments with authors about things like "but 2/3 reviewers said accept!"

It's easy to find BS reasons to reject papers; try to find good reasons instead.


Things that I hate to see in grants that will lead me to be grumpy (says Titus):

  • misspellings
  • bad grammar
  • bad science, defined as "stuff that is clearly wrong/badly done", not just stuff I disagree with

Do say when you don't know something/aren't an expert. You will be helping the editor!

Things you should try doing:


I have posted other peoples' reviews for one of my papers here: http://ivory.idyll.org/blog/khmer-counting-reviews.html


Grants

I post many of my grants on my Web site:

http://ged.msu.edu/research.html

I passed out reviews for three grants; I cannot make the reviews public, but here are the grants.

2011-NSF-CAREER: http://ged.msu.edu/downloads/2011-nsf-career.pdf
  • my takeaway: "Good proposal, bad PI, not enough preliminary data"
2012-NSF-CAREER: http://ged.msu.edu/downloads/2012-career-nsf-final.pdf
  • my takeaway: "OK proposal, OK PI, but too much preliminary data"
2012-NIH-BIGDATA: http://ged.msu.edu/downloads/2012-bigdata-nsf.pdf
  • my takeaway: "Please fund this person."

Note, 2012 grants were written at same time, but got very different reviews. Reviews depend heavily on reviewer!


Entertaining #SixWordPeerReview tweets from Rich:

http://storify.com/ctitusbrown/rich-lenski-and-sixwordpeerreview/

These are absolutely not how you should write reviews. But they are funny.


Tracy's links:

Env. Micro. referee comments:

http://onlinelibrary.wiley.com/doi/10.1111/j.1462-2920.2011.02660.x/full

Some examples:

"It is sad to see so much enthusiasm and effort go into analyzing a dataset that is just not big enough."

"You call the sample fresh water, this is confusing as it is saline water."

"The biggest problem with this manuscript, which has nearly sucked the will to live out of me, is the terrible writing style."

Again, try to avoid this kind of thing in real reviews :)


My blog post on why current peer review is broken: http://ivory.idyll.org/blog/angenmap-discussion-on-peer-review.html


Another recent blog post that I found interesting:

http://svpow.com/2014/01/20/how-to-become-good-at-peer-review-three-points-of-disagreement/


Questions? Comments? Ask!

--titus

Comments !

(Please check out the comments policy before commenting.)