why use buildbots?
So, uhh, why use buildbots, anyway?
- They build your code and run your tests without your conscious involvement.
Obvious, yes -- that is, after all, ostensibly the point of buildbots.
But it has more benefits than you might immediately.
For this to work, you must have a systematized and automated build process.
You must also have some automated tests.
And your your build process and tests are being run on a regular
basis, whether or not any particular developer feels like it. And if
the build or tests fail, then more likely than not, something changed
to make them fail -- and now you'll know.
These are all good and necessary things.
- They can build your code and run your tests in multiple environments.
buildbots can build and run your project on whatever operating systems
you or your colleagues can access, and report the results to you, with
a minimum of setup.
This is the main reason I use buildbots myself: to run tests on other
versions of Python, and other operating systems. I'm a UNIX guy, and
I develop on Linux; therefore my software usually works on Linux. My
pure Python code generally works on Mac OS X, too, although I
sometimes run into trouble with compiled code. But I don't ever
run my software on Windows systems, because I don't have Windows handy;
so my code often doesn't work on Windows. This is where a Windows
buildbot comes in really handy, by catching the errors that I otherwise
wouldn't even notice.
There's a more subtle point here that many people miss, which is the
ability of buildbots to test dependence on a specific full stack of
hardware and software. Most developers work with at most one or two
build environments, including compiler or interpreter versions,
operating system patchlevels, etc. The more different versions you
have being tested, the more you can detect sensitivities to specific
operating system or compiler or language features; whether or not
cross-compiler or cross-version compatibility important to you is a
different question, of course, but it's nice to know.
The most entertaining aspect of this is when buildbots detect when
developers -- especially inexperienced ones -- introduce unintended or
unauthorized new dependencies. "Hey, Joe, since when does our software
depend on FizBuzz!?"
These latter points feed particularly into #3 and #4:
- They provide a de facto set of docs on your build & test environment.
buildbots require explicit build instructions, so if you've got one running
at least your project has some form of build documentation. Not a good
one, maybe not an explicit one, but something.
This is not a concern for most big open source projects, because they
usually have fairly straightforward and well-documented build
environments (although not all -- OLPC/Sugar was horrific!) Where I
think this really helps is for small private projects and especially
for for academic projects, where the level of software engineering
expertise can be, ahem, poor. Having explicit build instructions that
graduate student B can use to build & run the code now that graduate
student A has left the project is quite helpful.
4. They are evidence that it is possible to build your code and run
your tests on at least some platform.
You might be surprised how much some projects really need this kind of
evidence :). As with #3, small private projects and academic projects
benefit the most from this.
- They can run all the tests, even the slow ones, regularly.
This is the third reason that software professionals like continuous
integration and buildbots: many tests (in particular, integration and
acceptance tests) may take a loooong time to run, and developers may
end up simply not running all of them. With buildbots, you can run
them on a daily basis and detect problems, without distracting or
defocusing your developers.
Are buildbots overkill for your project?
buildbots require setup and maintenance effort, which (in our zero-sum
world) takes that effort away from developing new features, exploratory
testing, etc. When does the benefit outweigh the cost?
Almost always, I believe.
For small side projects that you may not be constantly focused on,
having the tests alert you when something breaks is really helpful.
But even if you're in a mature software engineering setting and you
have a good build process, a good set of documentation on how to build
your software, and a commitment to running the tests regularly, many
of the advantages above still apply. In particular, #1 (building w/o
conscious effort), #2 (building across multiple environments), and #5
(running all of the tests, especially the slow ones), are advantageous
for all projects.
I think buildbots aren't that useful for projects that are mostly UI
(which is hard to develop automated tests for) or that are at a very
early stage (where you're accumulating technical debt on a daily
basis) or that depend on lots of specialized hardware. What else?