I recently had the pleasure of meeting with Randy LeVeque, Bill Howe, and Steven Roberts at UW, along with Jory Schossau, after the UW bootcamp that Jory and I ran. I already knew Bill from before (see our conversation on VMs and reproducibility) and Steven had taken the workshop, but this was the first time I'd talked with Randy. Fernando Perez had introduced us virtually because we were all interested in scientific reproducibility, and I used the opportunity to arrange a coffee meeting with Randy.

Randy, Bill, Steven and I have all spent a lot of time thinking about reproducibility in various ways, and they (among others) at UW are trying to figure out how to encourage reproducible scientific practices. Our conversation was far-ranging and somewhat random -- like all good conversations ;) -- but I wanted to highlight two ideas that came up.

First, we talked about the idea of pre-publication peer review for reproducibility: group X submits a paper, and group Y reviews that paper and attempts to reproduce the results in it before it is sent out to other reviewers. Then, when it is sent out to other reviewers, it is sent along with a letter stating that group Y verified the reproducibility of the paper's results; this lets the other reviewers can focus on things other than reproducibility. If this could somehow be married to a shortened time to publication, I think it could be a winner.

Second, we could run reproducibility "hangouts" where we spend time going through how specific papers did it. Some background: the UW folk are trying to think about how to encourage reproducibility, and they've had some well-attended meetings on the topic. Not being at UW myself (and knowing that even if I were, there are interesting things going on in many locations), I suggested that perhaps we could do something online with Google Hangouts or whatnot. The idea would be to pick a paper for a given month, and then have several groups spend part of that month going through the paper and trying to reproduce it. We could then get together on Google Hangout openly to discuss where we ran into trouble. We could (and should) involve the original authors, if they are interested. In addition, we could alleviate the pain and agony of trying to reproduce irreproducible papers by focusing on papers that claim they are reproducible -- that way we'd learn something about how other people are doing it, and maybe also find that perhaps they are not as reproducible as the original authors claimed.

...always nice to chat with people who think this stuff is important :)

--titus