Daily Life in an Ivory Basement

Hey cdfrey, ta0kira, I syndicate to advogato because I wanted to syndicate my posts to PlanetPython as well as advogato, at a time when advogato didn't have the tools to do that. Also, this way I can use my own damn editor, on my own damn laptop, to write the posts; I can spell check them, I can write them offline, and I can post them when I damn well please.

And yes, I do damn well read advogato even though I syndicate to it.

And I don't think too many of my posts are horribly off-topic, although I can probably fine-tune them a bit more by using an 'advogato' tag if need be.

Maybe what you should really be complaining about is the lack of tools that let you hide (or unhide) certain posters on a personal basis...

--titus

My last post, Good code coverage: Necessity vs Sufficiency, about how you should maintain high code coverage with your automated tests, seems to have really struck a nerve in a small group of people -- I got some fantastic comments, with some great pointers. Michael Foord's comment, 'Too often "testing is no silver bullet" is used as an excuse not to test' hit the nail on the head, and Andrew Dalke linked to his excellent & detailed discussion with Glyph about code coverage, as well as pointing out the SQLite testing policy which is, frankly, astounding.

But I'm not writing this just to point out those links. No, this is partly in response to Ned Batchelder's plaintive question about why I picked on him. I picked on him because of his article's title: "there are flaws in coverage measurement." Now, Ned is the author of coverage, the main code coverage tool available for Python -- and I know he thinks code coverage is really important. And it is entirely legitimate to write critical essays about code coverage, especially statement coverage. But I also get frustrated with people who write blog posts with even moderately sensational and misrepresentative blog post titles. And that's why I included him in the post complaining about such things.

Got that? I hate sensational and misrepresentative titles.

Good day.

--titus

I get really frustrated with posts that claim your unit tests lie to you or 100% code coverage is fallacious or there are flaws in coverage measurement. These are sensationalist headlines that encourage bad behavior, by confusing new or inexperienced or argumentative or lazy developers: "well, we all know test coverage isn't that important..."

There is a fundamental confusion here between necessary and sufficient. In plain English, things that are necessary are, well, necessary for a process; things that are sufficient are, collectively, sufficient.

One does not imply the other. Tires are necessary but not sufficient for a car.

Why do I bring this up? Because near-100% code coverage is basically necessary for a complete automated test suite, but not sufficient.

If you do not have near-100% code coverage, then some portion of your code is not being executed by your tests. That means that code is untested. It can't possibly be tested if you aren't executing it!

(As a corollary, no test that adds to code coverage is useless. It may not be as useful as it could be, but it is still running code that is otherwise untested. Room for improvement does not equal uselessness!)

OTOH, there are many reasons that 100% code coverage isn't sufficient to make your test suite good. For one, code coverage isn't branch coverage. For another, code coverage can't tell you if your feature set meets your client's needs or is in fact functional -- all it can do is tell you that your feature set meets your expectations at the moment.

So, let me suggest this simple pair of statements:

1. High (statement) code coverage in your tests is important.
2. High code coverage doesn't guarantee that your tests are complete.

I feel like high code coverage is actually a pretty low bar for a project. Whether or not you're using TDD, if you don't have high code coverage numbers, you're not really testing very thoroughly; it's generally pretty easy to amp up your code coverage numbers quickly, although writing the test fixtures can be a problem. I'll talk a bit about this in my PyCon talk.

Regardless of your tests suite, your project will probably fail in a boring and project-specific way.

--titus

P.S. Many people do get it, which is nice.

I'm going to stop using my Paypal account regularly as soon as I finish transferring some money around. They're too screwed up to be reliable, and their customer service is either incompetent or dishonest.

Luckily, many places are now accepting credit cards or Google Checkout, with which I've had no problems.

--titus

Over at OLPC News, Wayan Vota asks: "Do you get better FOSS code ... if the developers are paid or unpaid?"

Interesting question, especially as considered in light of the OLPC code base.

As of about a year ago, virtually everybody I talked to was shocked and stunned at the poor quality of the OLPC code base. My personal encounters with the code left me frustrated at the poor Python style, angry at the simultaneously over- and under-engineered build process, livid at the poor quality control, and blue about the long-term prospects of the OLPC software. I was not alone, although most people were polite enough not to be a jerk about it like me. (Looking back, perhaps if more people had been jerks earlier, things could have improved. Or not.)

At the time I felt that the OLPC code, which might have been a triumph for the OLPC project, a gem of Python code, and a win for the open source community, had turned into a cesspool of code and a potentially infinite time sink for people working on it.

I haven't revisited the project since then, and I know that community enthusiasm has waned for the OLPC. I don't think the poor quality of the code was a huge factor in this decline, but it certainly decreased the interest that serious hackers had in getting involved in the project -- at least, it did for everyone I talked with.

So, let's get back to the original question: in an OSS project, is it better to let the community write your code for you, or should you have a core development team?

Let me rephrase that another way, actually: if your business practices drive you to write crappy, hurried code, with no attention to any form of reasonable software development processes (agile or otherwise), no automated testing, and very little QA of any kind, are you better off turning software development over to the community?

The answer becomes less difficult then: yes, if it's an option, you should jettison your software development efforts. If you can't do things competently, you should remove yourself from the responsibility of doing it. Abdicating responsibility is the responsible thing to do in these circumstances, if you can make it work.

But... what went wrong, anyway? I don't know. I'd guess that the OLPC software development team was overmatched from the beginning: responsibility of developing new software for a novel mix of hardware, together with a novel GUI interface, in an environment of much enthusiasm and relatively little funding, is hard. Add to this the enthusiastic but not very disciplined-sounding roadmap, as well as the unrealistic deadlines set due to PR and political considerations, and it's a recipe for disaster. When you also throw in an overwhelmed OLPC management team, I don't think you can expect anything but disaster.

So, looking back, I think the OLPC was probably headed into a mess from day 1, and while the fubar'ed software development process didn't help, it was at least partly driven by circumstances outside the control of the software developers. The software developers were hamstrung from the get go. And in that environment, turning the majority of your software development over to someone else -- someone not under your direct control -- may in fact be the best option.

I think that's actually a much more interesting question: at what point should an organization realize that they are so incompetent at managing and doing "in-house" software development that they should try to outsource it? What metrics should they look at, and how should the decision be made?

--titus

p.s. The general question of whether to go for paid or unpaid development is silly, of course. The answer is, "it depends." Duh. The interesting question is what questions you need to ask to figure it out!

p.p.s. Disclaimer: the individual OLPC people I know, up to and including lower management, are smart, enthusiastic, and good people. I place the majority of the blame squarely at the top.