Daily Life in an Ivory Basement

One of the advantages of this year's PyCon was that it was (again) held in Chicago, the home town of Leapfrog Online. Since they use twill quite a bit, and were bothered by some of the poor design decisions and bugginess, they were keen to get together with me to move twill forward. So we scheduled a sprint for the Monday after PyCon.

In preparation for the sprint, I did a bit of research into how widely twill was being used. Downloads only roughly correlate, but I was surprised to discover that in just the last year, there were over 6,000 downloads from my site; this doesn't count Debian users, who can install it from one of the Debian dists. I'd also been surprised by the number of people at PyCon who came up to me and told me that they were using twill internally in their companies -- at least two very expert groups had settled on it for some of their internal monitoring and testing. Very cool! What this told me is that twill is very nice, simple and usable for many people and we shouldn't get too adventuresome; good thing to know ;).

The sprint basically consisted of us talking through a few fundamental issues like bundling and future development, then fixing a few items, while I forwarded on all of the bug reports I've gotten over the last two years.

The source code has now moved to code.google.com/p/twill and you can see all of the issues in the usual place.

During the sprint we made a few decisions:

• 0.9.2 is Coming Real Soon, as a largely feature/API-stable release that fixes a number of simple bugs and integrates the latest mechanize.
• for 0.9.2 and 1.0 we will provide both bundled and unbundled versions of twill; the bundled versions will contain BeautifulSoup, mechanize, ClientForm, and pyparsing. The unbundled version will simply specify what versions of those packages it needs. This unbundling will help packagers out while letting individuals (like, say, Windows users) install twill easily.
• 1.0 is further down the road, but will only add a few features. The main goal of 1.0 is to be nice & stable.
• 2.0 and beyond is on the table but exactly what it will be is unclear. I have my own ideas but since I'm not doing much Web developing I may let others take over.

Since the sprint, Pam Z. finished putting the issues into the tracker and we've been slowly trying to work through them.

Props to Pam Z., Nat W., Kevin B., and Jesse for coming to the sprint, and to Terry Peppers and Leapfrog for pushing it! And thanks to Leapfrog for an excellent steak dinner afterwards ;)

--titus

I'm switching several projects from darcs to either git on github, or svn on Google Code.

twill, a simple Web testing tool/HTTP driver in Python, was switched over to Google Code several months ago: see http://code.google.com/p/twill. I'll post more on twill development soon, I hope.

I just put scotch in the twill svn as well. scotch is an HTTP recording and playback tool written in Python with WSGI interfaces.

I also moved figleaf to github, at http://github.com/ctb/figleaf/tree/master. figleaf is a flexible code coverage recording and analysis tool for Python.

This move will hopefully open up development a bit more; I've been inactive for too long, and I feel that one obstacle to participation in these projects has been my self-hosting of the DVCS archives in darcs.

Note, I will probably move pinocchio, pygr-draw, zounds, and blastparser over to github as well, whenever I get around to it. I'm planning to decomission my darcs repository as I swizzle machines around while moving my virtual home over to MSU; the URLs will still work for pulling, but I'm going to stop pushing things to them.

--titus

Noah and Grig have been CCing me on a conversation about JoelOnChecklists and Grig's post. Noah's writing a book chapter on this stuff, and asked for some tips.

Here are mine.

First, I have a bunch of individual twill scripts in a directory that are run every hour. These scripts are mostly of the form,

% cat neuro-is-alive
go http://neuro.caltech.edu/
code 200

find "Shimojo"

That is, they verify that the host is alive, successfully serving content, and serving content with the right keyword.

I run them from cron:

10 * * * * /usr/bin/twill-sh /u/t/.tests/* > /dev/null

and they have been invaluable for telling me when machines are broken. Obviously they don't replace "proper" monitoring software, but they do detect downtime, misconfigurations, etc. -- and they're really cheap to write/update/disable. (Remember, folk, KISS...)

Second, I use twill to test my DNS setup. I have a few scripts that are run hourly (see mechanism above) against both my master name server and the public caching name servers provided by my ISP:

extend_with dns_check
dns_a alife.org. 134.10.15.75 $dns_server

This gives me security that my entire DNS system is working, and also lets me do "test-driven name service", where I can write the test first ("I want the A record for alife.org to point to X.Y.Z"), then write the bind config & verify that it works.

I think I have managed to mildly annoy my ISP a few times by asking why their name servers were returning bad or outdated or inconsistent information ;)

Third, to test mailman installs and the queue runner (which has a habit of dying on my machine ;(:

I set up the following: one of my machines sends a message to a mailman list, which forwards to a single alias file, which in turn saves the message to a Web-accessible location. The saved message is wiped each time a new message is sent.

Another machine checks that Web-accessible location for correct content using the twill tests (above).

It will fail if the saved message is ever not wiped -- I didn't bother putting a time stamp in ;) -- but this gives me some security that I will detect system-wide mailman failures in the future.

This test setup also serves as a fairly simple test of e-mail configuration and delivery.

I think I can simplify this third test by adding some sendmail commands into twill that allow sending of an e-mail containing a unique identifier, followed by a check of the list archive mbox. I'd have to write new code for that, though, and the above fits my needs.

These simple tests really keep my machines on the straight-and-narrow. Since I run most of this stuff for fun and not for profit, this simple and easily maintainable system test infrastructure is all I really need.

--titus

Hi folks,

just a quick note -- Kumar McMillan has offered to take over wsgi_intercept. You can see the new project over at code.google.com, http://code.google.com/p/wsgi-intercept/.

While I will miss the income from the project, I think that Kumar will treat it well.

--titus

I don't feel like I need to "defend" twill -- it's successful beyond both my expectations and my cognizance (I have no idea who's actually using it, but it's apparently a lot of people!), but I may need to promote it better. I ran across this post earlier today. It shows how you can use mechanize to do some simple "screen scraping," and it spurred me to check the following scripts into a new "advocacy" section of the twill archive.

Here's how you can use a twill script to do what Greg did with mechanize:

add_extra_header User-Agent "Mozilla/5.0 (compatible; MyProgram/0.1)"
go http://www.python.org/
show

You can also use straight Python, if that's your poison:

from twill.commands import *
import twill

add_extra_header('User-Agent', 'Mozilla/5.0 (compatible; MyProgram/0.1)')
go("http://python.org/")
html = twill.get_browser().get_html()
print html

Note that twill is based on mechanize, and so promoting twill doesn't mean pushing mechanize down. mechanize is amazingly powerful -- but sometimes you want to just go grab some HTML, and twill (tries to) make that easy.

One unexpected result of all this -- I discovered that the function get_browser() wasn't actually exported from twill.commands by default, which is silly. It's the first function I wanted to call in order to do something minimally complex. So now the API is one call bigger ;). One more corner rubbed off...

--titus