The Story begins

The story of my role in the Haemonchus paper starts about 3 years ago. My friend & colleague Erich Schwarz is a guy who specializes in nematode genomics, and he was sequencing, assembling, and analyzing the Haemonch genome with a team down in Melbourne (largely Robin Gasser's group, I think) as well as Paul Sternberg's lab at Caltech.

Well, I say he was assembling it, but he wasn't being very successful at it. You could have said "throwing assemblers at it desperately" instead. Every time I talked with him he regaled me with tales of woe, some of which have made it into my slides (see slide 13) -- to quote, "The power of next-gen sequencing: get 180x coverage... and then watch your assemblies never finish." The basic problem was that every assembler he tried would slurp in the reads, churn churn churn, and then at some point die. Or not die. But one thing no assembler would do was output anything even remotely resembling a decent assembly. Basic quality filtering helped a bit -- Erich could get the sequencing data from short-insert libraries to assemble, but as soon as he put in the bigger insert data, kablooey. It didn't seem to have anything to do with compute power; rather, something was fundamentally weird about the sequence.

While he was busy throwing himself at large volumes of sequencing data, I was working with Jason Pell and Adina Howe on metagenomics -- that is, we were throwing ourselves at even larger volumes of sequencing data. Most vexing. Eventually we figured out a technique (partitioning) that let us cut the problem down to size, but it clearly wouldn't work on genomes so was of no use to Erich.

As we worked, however, we found that there were annoying levels of connectivity in our metagenomic data. And our efforts at removing this connectivity turned out to be of great use (although it wasn't the final kicker).

When Erich and I discussed his assembly problems, we figured it had to have something to do with repeats and polymorphism. If you have a repeat-rich genome and a lot of polymorphism, you're gonna have a hard time assembling much; plus, efforts to traverse those regions and assemble them into contigs (or ignore them) were going to run into combinatorial complexity. So when I said we had a way to remove highly connected regions but that it was meant for metagenomes, Erich asked me to run it on his data.

I said no. There was no reason to believe that it would work.

Erich insisted.

I still said no.

He continued to insist. At some point, he must have bought me some good steak or something. And then I decided to run our delumping code just to get him off my back.

... Voila. Erich took the delumped data, fed it into Velvet, and went to sleep. He woke up the next morning to a crash, or at least what he assumed was a crash -- he'd never had an assembly complete in less than a week on this data set. But, of course, it turned out to be an actual real, useful assembly! And that was the start of the end of the basic assembly problem for him.

This delumping wasn't the final filtering stage actually used for the assembly. My lab was in an intense stage where we were throwing ideas around a lot, and about 6 months later we figured out diginorm which turned out to be a much better tactic for all sorts of things. (Although we're still very fond of partitioning. More on that in a few weeks.) But that initial result was the breakthrough in many ways, even though even today we have at best an imperfect understanding of why it worked.

What was wrong with the Haemonch data?

As we now know, as well as originating from many different individuals the sequence was filled with microbial contaminants; moreover the genomes had been subjected to whole-genome amplification, which results in dramatic coverage variation. So we had an almost perfect quadrifecta: a eukaryote, steeped in a virtual metagenome, with high polymorphism, and really nasty amplification bias. (At one point, I estimated that the polymorphism rate was 10% -- one in ten nucleotides was different between the two haplotypes of the diploid genome -- although I believe that number was probably inflated by a high sequencing error rate. But still.) Digital normalization turned out to solve the biggest of these problems, allowing an initial assembly; this led to discovering the error problem and the metagenome problem, which led to decontamination protocols; and then finally Erich built a whole pipeline to post-process the data into a nice assembly. It was a heck of a lot of work on his part, and I still feel marginally guilty about freeriding into coauthorship (but only marginally ;).

And that, boys and girls, is how I ended up in the genome assembly game, as opposed to staying in my neat little boxes of metagenome and transcriptome assembly.

Since then, we've applied khmer based techniques -- largely diginorm, but some others as well -- to a variety of genomes, and discovered that in many cases they work quite well in terms of producing an initial assembly. I believe that at least 2-3 more papers will be coming out soon from other groups, pointing out that diginorm performs reasonably well on genome assembly.

Suuuuuuuuuper awesome.

It's always nice to actually be useful.

More, I think it's a good story about the, ahem, serendipity that comes from casual scientific interactions. I was working on soil metagenomes, Erich was working on nematodes; whoda thunk our technology would help him?

What's next, bub?

We may build some standard protocols and assessment mechanisms for genome assembly into khmer. I know that some groups have made a habit out of diginorming their genomic data into oblivion, and it would be nice to codify these practices into something more generally usable.

We also think we can use diginorm-based techniques to do some clever things with combined Illumina and PacBio data, so hopefully more on that soon.

And, while I'm still planning to stick mostly in my particular corner (metagenomes for everyone! transcriptomes for everyone!) I may nip out every now and then and poke around with a genome.

--titus