Daily Life in an Ivory Basement

I spent part of Saturday and Sunday working on Cartwheel, and then I spent part of Monday morning nailing down a bug that I thought I'd introduced. The process of tracking down and fixing this bug led me to give serious thanks for version control and automated tests, and at last gave me a nice compact story to tell when urging scientists to use version control.

(I doubt anyone reading this needs encouragement to use VCS (although I'd bet that not everyone tests like they should; I certainly don't!) but it does serve as a good cautionary tale.)

Briefly, I had introduced a bunch of new logic to allow parsing and exporting of results from within the Cartwheel Web site itself. This led to increased complexity in the Web site, of course, but I'd written automated functional tests along with the new code, and everything was working.

...at least, everything was working until I ran all of the functional tests, not just the ones immediately relevant to my code. After a serious amount of effort (ref note 0), I discovered that a section C test failed if (and only if):

(any three of four section A tests were enabled) AND (a particular section B test was enabled)

Now, this was puzzling, because section A, section B and section C were almost entirely decoupled (ref note 1).

I gamely dove in and started disabling various parts of the various tests (ref note 2). By focusing on the simplest tests (ref note 3) I quickly discovered that any database reference led to the error, and this led me instantly to the database caching layer (ref note 4).

OK, so everyone has gone through this, right? Why was this special for me?

Note 0 -- because my tests were automated, I could replicate the fault on demand, which meant that I could eventually figure out exactly what tests had this bad interaction. Try that without automation!

Note 1 -- experienced developers will immediately recognize the presence of epistatic interactions between modules in test faults as a sign that you need to take a close look at whatever layer is in common between the modules. I, for some reason, did not. Note to self: remember this.

Note 2 -- try systematically disabling (and re-enabling) portions of your code without version control. You will quickly fall into a suicidal state and end up with large portions of your application disabled because you forgot what you did.

Note 3 -- focusing on deconstructing the simplest tests seems obvious in hindsight, but it wasn't at the time. Note to self: remember this.

Note 4 -- see note 1. Why did it take me so long to get here? Also note that because of the level of the problems, it's extremely unlikely that any unit tests would have caught this; huzzah for functional tests!

In hindsight, then, this problem was mainly special because I handled it so obtusely ;). I had the advantage of an automated test suite, full knowledge of all the source code and architecture, full version control, and almost two decades of experience in programming, and it still took me about 6 straight hours to track this bug down. I don't know how people without this infrastructure manage, and I certainly don't know if I would have even found it without the test suite; there's no indication that it was causing problems with the site yet, but it would have if website load increased by only a little bit.

So, use version control & write tests, everyone!

For those of who that have come this far, here's a little treat from the testing-in-python list: Kumar McMillan gives the best description of the difference between unit tests and functional tests that I've seen.

Peace out,

--titus

This morning, I described my recent work-related activities to Tracy as "Whack-a-Mole". For those of you who aren't familiar with the Hasbro Whac-a-Mole game, it's a fun game for four year olds where you, err, hit moles on the head with a hammer as they pop up out of the game. The goal is to get as many moles as you can - but of course there's an endless supply. You might note that it's less fun when you're older than four, and the "moles" you are whacking are problems that you need to deal with in the course of your daily job.

So anyway, I've been trying to stay on top of too many projects, and I've noticed that whenever I really focus on one of the projects (say, the testing books) the other projects all pop up a bunch of issues with "urgent!" notices on them. No sooner do I whack one of them than do another two or three pop up.

But it gets worse. I have a few projects that in themselves are whack-a-mole projects. One of them is Cartwheel, my moderately popular but slowly obsolescing bioinformatics framework. This sucker has been a problem all along, and it's why I got interested in testing: it's the project that motivated twill. It's big enough that keeping track of all the software dependencies, remote interfaces, and epistatic interactions between the modules requires a certain amount of brainpower. I do not have a big enough brain to devote a corner of it to this project indefinitely, so it's turned into a whack-a-mole project: one day, a problem will pop up in module A; another day, module B; and another day, module C. It doesn't matter how many problems I whack, up pops another!

Now, fortunately for me (and unfortunately for the moles' heads), I've discovered automated testing. Everytime I find a new probem, I outline it with automated tests. Then I fix the problem, and write some more tests. This means that not only am I whacking the mole's head, but I'm covering the mole hole with something -- concrete or canvas, depending on the day and the nature of the hole. Whatever the covering, that module is going to have fewer mole heads popping up in the future, and pursued to its logical end, the application's problem areas will eventually be covered with tests. No more uncovered mole holes!

(Cue amusing image of a bunch of frustrated moles banging their heads against soft but unyielding mole hole covers. Hmm, that seems like a good testing book cover... ;)

Just another bonus of automated testing: less and less whack-a-mole! (Try selling that to your corporate bosses...)

All right, back to writing tests. (Oh, look, another mole head! thud Poor moles...)

--titus

I've just finished the last rough draft of a short e-book, An Intro to Testing Web apps with twill and Selenium. The book has four sections:

1. Introduction
2. twill intro
3. Selenium intro
4. Testing an app with twill and Selenium

The "app" being tested in #4 is just the very simple poll app from the Django tutorial, and I'd like to get some comments from a Django expert (which I am not!) I'm particularly interested in thoughts on how to distribute the app for readers, but I also want a gut check from someone who knows this stuff cold. Any takers? Let me know.

(The e-book will be available sometime in the next few months from O'Reilly, for people who are generally interested in these topics. It should cost about $10.)

cheers, --titus

No, I'm not dead yet.

I just finished reading The Children of Hurin, which led me to re-read The Silmarillion. Brilliant.

I also just got my grubby little hands on a UK copy of Reaper's Gale, Steven Erikson's new book in the Malazan Empire series. It's big. Really big. I haven't started it yet.

I've been doing a lot of bio hacking. More on that soon, I hope.

I finally powered through the last big effort on our first O'Reilly book, An Introduction to Web Testing with twill and Selenium. Whew.

I'm going to be travelling a lot over the next few months: first to Lawrence Livermore (to give the Intermediate Software Carpentry course), then maybe to Portland (to visit friends, cousin, and Powell's Books), then off to Michigan (to visit Michigan State), and then to Massachusetts and Woods Hole. Whee.

Oh, and Tracy is defending Thursday. Assuming everything goes well, we will officially be the most overeducated couple in our immediate family: two Phuds.

--titus

http://lists.idyll.org/pipermail/testing-in-python/2007-March/000233.html (and associated thread)

http://www.taniquetil.com.ar/facundo/py_patchs.html

http://groups.google.com/group/turbogears/browse_thread/thread/5e20cd6515aa8332/c07131062e8d1596#c07131062e8d1596

http://www.nabble.com/Document-instrumentation---metrics-%28long%29-was-Re%3A-Re%3A-The-nature-of-executive-%22pushback%22-to-agile-technologies--p9401494.html

http://testanything.org/wiki/index.php/

http://lwn.net/Articles/224249/

http://www.advogato.org/person/ncm/diary.html?start=173

http://www.itworld.com/AppDev/nlsebiz070313/

http://blog.extracheese.org/2007/04/are-your-tests-lying-to-you.html

http://www.youtube.com/watch?v=ebsw3NM1M0Y&NR=1

http://www-static.cc.gatech.edu/~shivers/autoweapons.html

glen cook

http://www.voidspace.org.uk/python/weblog/arch_d7_2007_04_28.shtml#e695

http://www.itworld.com/Tech/2327/nlsebiz070501/index.html