A while back, someone else's graduate student asked me (slightly edited
to protect the innocent :) --
I already have two independent sets of de novo transcriptome
assemblies and annotations of the NGS data [...] 1) from the
company who did the sequencing and analysis, and 2) from our
pipeline here. It would be great to have a third set from your lab
to compare. Do you think such a comparison of independent analyses
is worthwhile?
My response, somewhat edited:
Well, yes, and no -- first, just to make my hypocrisy clear, see:
https://peerj.com/preprints/505v1/
(which is about exactly this!)
I definitely think evaluating pipelines and assemblies is worthwhile, but at
the end of the day it's time consuming and I worry that it distracts from
the science. My experience is that it takes a lot more time and effort
than anyone realizes before they start the project -- either that, or
the researchers do a bad job of evaluation :). So my two guidelines (for
myself) are --
how reproducible is the analysis by you, and by others? If you're
simply evaluating what one person did against what some other
person did, without having carefully recorded what was actually
done, then it's all anecdatal. In this case, no one is going to
learn anything long term, and you shouldn't bother.
A good way to self-evaluate here: suppose one of your reviewers
said, "tweak these parameters because I believe that they make a
big difference." How long would it take you to reanalyze
everything? If the answer is going to be "an annoyingly long time"
then don't write the paper in the first place!
Is there anything to be learned about the underlying science? In
our case we pushed a bit to show that though Trinity seems to have
recovered more low abundance isoforms, we got essentially no new
biologically relevant information out of it - something that I
think will hold generally true for a wide range of data sets. To
me, that's reasonably important and useful information. Not sure I
would have liked the paper as much if we'd just said "hey, look, we
get basically the same results, with some small differences. Cool?"
A good self-evaluation question: is anyone going to find anything
you say surprising or useful for building intuition and
understanding? If not, why bother doing the work?
---
So that was my response. You'll see some of these sentiments echoed
in my blog post about the Critical Assessment of Metagenome
Interpretation; if
you're not automating it and it's not reproducible, why bother? The
software is going to change tomorrow and your benchmark will be
outdated.
This is also why I'm a huge fan of nucleotid.es. These principles are also guiding
our development of computational protocols and paper pipelines. And, fundamentally,
it's why I'm planning to rain swift and sudden death down on
benchmarking and comparison papers that aren't highly reproducible.
--titus
There are comments.