1. The wonderful ongoing discussion around significance and reproducibility.

In addition to Victoria Stodden, Brian Nosek and John Ioannidis have been leaders in banging various drums (and executing various research agenda) that are showing us that we're not thinking very clearly about issues fundamental to science.

For me, the blog post that really blew my mind was Dorothy Bishop's summary of the situation in psychology. To quote:

Nosek et al have demonstrated that much work in psychology is not reproducible in the everyday sense that if I try to repeat your experiment I can be confident of getting the same effect. Implicit in the critique by Gilbert et al is the notion that many studies are focused on effects that are both small and fragile, and so it is to be expected they will be hard to reproduce. They may well be right, but if so, the solution is not to deny we have a problem, but to recognize that under those circumstances there is an urgent need for our field to tackle the methodological issues of inadequate power and p-hacking, so we can distinguish genuine effects from false positives.

Read the whole thing. It's worth it.

Relevance to #futurepaper: detailed methods and exact repeatability is a prerequisite for conversations about what we really care about, which is: "is this effect real?"

2. Blogging as a way to explore issues without prior approval from Top People.

I've always hated authority ever since I noticed the propensity for authorities to protect their position over seeking truth. This manifests in many ways, one of which through control over peer review and scientific publishing processes.

With that in mind, it's worth reading up on Susan Fiske's "methodological terrorism" draft, in which Fiske, a professor at Princeton and an editor at PNAS, "publicly compares some of her colleagues to terrorists" (ref). Fiske is basically upset that people are daring to critique published papers via social media.

There are a bunch of great responses; I'll highlight just one, by Chris Chambers:

So what’s really going on here? The truth is that we are in the midst of a power struggle, and it’s not between Fiske’s “destructo-critics” and their victims, but between reformers who are trying desperately to improve science and a vanguard of traditionalists who, every so often, look down from their thrones to throw a log in the road. As the body count of failed replications continues to climb, a new generation want a different kind of science and they want it now. They're demanding greater transparency in research practices. They want to be rewarded for doing high quality research regardless of the results. They want researchers to be accountable for the quality of the work they produce and for the choices they make as public professionals. It's all very sensible, constructive, and in the public interest.

Yeah!

The long and short of it is that I'm really excited about how science and the scientific process are being discussed openly via blogs and Twitter. (You can also read my "Top 10 reasons why blog posts are better than scientific papers.)

Relevance to #futurepaper: there are many alternate "publishing" models that offer advantages over our current publishing and dissemination system. They also offer potential disadvantages, of course.

3. Open source as a model for open science.

Two or three times every year I come back to this wonderful chapter by K. Jarrod Millman and Fernando Perez entitled "Developing open source scientific practice." This wonderful chapter breaks down all the ways in which current computational science practice falls short of the mark and could learn from standard open source software development practices.

For one quote (although the chapter offers far more!),

Asking about reproducibility by the time a manuscript is ready for submission to a journal is too late: this problem must be tackled from the start, not as an afterthought tacked-on at publication time. We must therefore look for approaches that allow researchers to fluidly move back and forth between the above stages and that integrate naturally into their everyday practices of research, collaboration, and publishing, so that we can simultaneously address the technical and social aspects of this issue.

Please, go read it!

Relevance to #futurepaper: tools and processes prior to publication matter!

4. Computational narratives as the engine of collaborative data science.

That's the title of the most recent Project Jupyter grant application, authored by Fernando Perez and Brian Granger (and funded!).

It's hard to explain to people who haven't seen it, but the Jupyter Notebook is the single most impactful thing to happen to the science side of the data science ecosystem in the last decade. Not content with that, Fernando and Brian lay out a stunning vision for the future of Jupyter Notebook and the software ecosystem around it.

Quote:

the core problem we are trying to solve is the collaborative creation of reproducible computational narratives that can be used across a wide range of audiences and contexts.

The bit that grabs me the most in this grant proposal is the bit on collaboration, but your mileage may vary - the whole thing is a great read!

Relevance to #futurepaper: hopefully obvious.

5. mybinder: deploy running Jupyter Notebooks from GitHub repos in Docker containers

Another thing that I'm super excited about are the opportunities provided by lightweight composition of many different services. If you haven't seen binder (mybinder.org), you should go play with it!

What binder does is let you spin up running Jupyter Notebooks based on the contents of GitHub repositories. Even cooler, you can install and configure the execution environment however you want using Dockerfiles.

If this all sounds like gobbledygook to you, check out this link to a binder for exploring the LIGO data. Set up by Min Ragan-Kelly, this link spools up an executable environment (in a Jupyter Notebook) for exploring the LIGO data. Single click, no fuss, no muss.

I find this exciting because binder is one example (of several!) where people are building a new publication service by composing several well-supported software packages.

Relevance to #futurepaper: ever wanted to give people a chance to play with your publication's analysis pipeline as well as your data? Here you go.

6. Overlay journals.

As preprints grow, the question of "why do we have journals anyway?" looms. The trend of overlay journals provides a natural mixing point between preprints and more traditional (and expensive) publishing.

An overlay journal is a journal that sits on top of a preprint server. To quote,

“The only objection to just putting things on arXiv is that it’s not peer reviewed, so why not have a community-based effort that provides a peer-review service for the arXiv?" [Peter Coles] says — pointing out that academics already carry out peer review for scientific publishers, usually at no cost.

Relevance to #futurepaper: many publishers offer very little in the way of services beyond this, so why pay them for it when the preprint server already exists?

7. Bjorn Brembs.

Bjorn is one of these people that, if he were less nice, I'd find irritating in his awesomeness. He researches flies or something, and he consistently pushes the boundaries of process in his publications.

Two examples -- living figures that integrate data from outside scientists, and systematic openness - to quote from Lenny Teytelman,

The paper was posted as a preprint prior to submission and all previous versions of the article are available as biorxiv preprints. The published research paper is open access. The raw data are available at figshare. All authors were listed with their ORCID IDs and all materials referenced with RRIDs. All methods are detailed with DOIs on protocols.io. The blog post gives the history and context of the work. It's a fascinating and accessible read for non-fly scientists and non-scientists alike. Beautiful!

Bjorn also has a wonderful paper on just how bad the Impact Factor and journal status-seeking system is, and his blog post on what a modern scholarly infrastructure should look like is worth reading.

Relevance to #futurepaper: hopefully obvious.

8. Idea futures or prediction markets.

There are other ways of reaching consensus than peer review, and idea futures are one of the most fascinating. To quote,

Our policy-makers and media rely too much on the "expert" advice of a self-interested insider's club of pundits and big-shot academics. These pundits are rewarded too much for telling good stories, and for supporting each other, rather than for being "right". Instead, let us create betting markets on most controversial questions, and treat the current market odds as our best expert consensus. The real experts (maybe you), would then be rewarded for their contributions, while clueless pundits would learn to stay away. You should have a free-speech right to bet on political questions in policy markets, and we could even base a new form of government on idea futures.

Balaji Srinivasan points out that the bitcoin blockchain is another way of reaching consensus, and I think that's worth reading, too.

Relevance to #futurepaper: there are other ways of doing peer review and reaching consensus than blocking publication until you agree with the paper.

9. Open peer review by a selected papers network.

This proposal by Chris Lee, a friend and colleague at UCLA, outlines how to do peer review via (essentially) a blog chain. To quote,

A selected-papers (SP) network is a network in which researchers who read, write, and review articles subscribe to each other based on common interests. Instead of reviewing a manuscript in secret for the Editor of a journal, each reviewer simply publishes his review (typically of a paper he wishes to recommend) to his SP network subscribers. Once the SP network reviewers complete their review decisions, the authors can invite any journal editor they want to consider these reviews and initial audience size, and make a publication decision. Since all impact assessment, reviews, and revisions are complete, this decision process should be short. I show how the SP network can provide a new way of measuring impact, catalyze the emergence of new subfields, and accelerate discovery in existing fields, by providing each reader a fine-grained filter for high-impact.

I think this is a nice concrete example of an alternate way to do peer review that should actually work.

There's a lot of things that could tie into this, including trust metrics; cryptographic signing of papers, reviews, and decisions so that they are verifiable; verifiable computing a la worldmake; etc.

Relevance to #futurepaper? Whether or not you believe this could work, figuring out why you think what you think is a good way to explore what the publishing landscape could look like.

10. A call to arms: make outbreak research open access.

What would you call a publishing ecosystem that actively encourages withholding of information that could save lives, all in the name of reputation building and job security?

Inhumane? Unethical? Just plain wrong?

All of that.

Read Yowziak, Schaffner, and Sabeti's article, "Data sharing: make outbreak research open access."

There are horror stories galore about what bad data sharing does, but one of the most affecting is in this New Yorker article by Seth Mnookin, in which he quotes Daniel MacArthur;

The current academic publication system does patients an enormous disservice.

The larger context is that our failure to have and use good mechanisms of data publication is killing people. Maybe we should fix that?

Relevance to #futurepaper: open access to papers and data and software is critical to society.

Anyway, so that's what's on the tip of my brain this fine morning.

--titus