Daily Life in an Ivory Basement

I am in need of some test automation tools for a GTK/GNOME GUI, and after scanning the Python Testing Tools Taxonomy and doing a bit of investigating, it looks like there are three possibilites.

dogtail and the Linux Desktop Testing Project both use accessibility libraries to drive GNOME applications. I found two comparisons here and here. It seems like dogtail is more me-friendly (better Python interface, less XML) and LDTP is more mature and up-to-date.

A third option is Xnee, which is a generic (and non-Python) X-Windows event recorder and playback system. There's a brief mention of it "in the wild" here.

I would be interested in any personal knowledge people have of these projects. None of them seem to be that widely used or mentioned, based solely on Google juice, so anything you can tell me would help. Right now I'm thinking that I will have to just try each of them out to see which one fits my needs best... <sigh>

--titus

I spent a fairly satisfying two days setting up unit and functional tests for figleaf, my code coverage package. I implemented the tests using a technique that I gather is called vertical slicing -- I just called it "starting a new phase of a project" before ;). The idea is that you implement a single test, with necessary fixtures, from end to end; then you widen it. This way you don't get overwhelmed with planning ahead for different features or other distractions: you just get one thing working, with the attendant psychological reward of satisfaction.

Why all the testing? Well, after twill, figleaf is the most popular of my packages, but it is also much uglier than it should be. A number of bug reports have been sent to me, and I need to deal with them. Since I am also hoping to shoehorn it into a large Python-based project, this is an apropos time.

Some of the fun new features I am addign for figleaf 0.7 include a much enhanced and more configurable reporting/annotating interface, and (something I'm particularly proud of) tracing of statements only within specific files.

Anyway, I ran into a few entertaining problems with the tests.

First, python2.4 and python2.5 differ in what they consider an executed statement. Most notably, python2.4 will call the trace function when 'pass' is executed, while python2.5 will not. This means my regression tests need to compare against version-specific "known good" data.

Second, easy_install was mucking around with my imported packages and I couldn't get the figleaf-under-test to be imported ahead of the installed version. It turns out that this code re-scans sys.path and thus imports the correct package:

import figleaf  # probably unnecessary
sys.path.insert(0, '/path/to/correct/figleaf')
reload(sys.modules['figleaf'])

Third, testing command-line executable programs is a huge pain in the butt. There's room for someone to come up with a framework for this purpose. Right now I think texttest is probably the best framework, but I have to say that it seems overcomplicated for what it does; it would be nice to have something simpler. I think that the main features that a test framework for command-line programs needs are a way of compactly comparing test output to known-good output, and a nice way of specifying configurations (directory locations, config files, etc) so that a variety of features can be tested.

Fourth and finally, achieving true test isolation is a necessary art. Because I was fiddling with sys.settrace and some module-global-singletons in the code under test, I had to be very careful to include appropriate tests in the setup code and appropriate teardown code to remove/reset everything. I kept on running into new areas where this was a problem, but it taught me a lot about where my own code kept state. Very useful.

cheers, --titus

Every few days, a new hysterical screed against formal education, undergrad studies, or grad studies comes across my screen. As a result I've been mulling over the hot-button issue of academic study, both undergrad and grad, for the last few months. Why are there so many loud people decrying the usefulness of a formal education? Which of the (many!) things that are shoved down your throat in academia are worthwhile, and which aren't? Why does this country (the USA) have such a focus on it -- and why do US institutions have such a good academic rep?

Unfortunately I think it's going to be difficult for me to appear neutral in this discussion, because I'm now firmly ensconsed as a member of the establishment (well, barring tenure ;). I'm the child of an academic (my father is a physics professor); I went to a snooty liberal arts undergrad college (Reed College, BA Math); I received my PhD at an institution that is a pillars of modern research (Caltech, PhD Biology); and I'm starting a faculty position (at Michigan State U.) in Computer Science and Microbiology & Molecular Genetics. This last position will no doubt require me to extort work out of dozens if not hundreds of graduate students, with the full intent of making their lives as miserable as mine was during much of my own graduate school experience.

On the flip side, I'm certainly not your traditional 1980s ivory tower academic. I was introduced to Open Source programming by Mark Galassi back in high school, and I've been a contributor to various projects ever since. I'm especially active (or I try to be active) in contributing to a bunch of projects in the Python community. During my graduate work I implemented actual! functioning! software! that is used by hundreds of people. I've consulted for real companies, I've run one or two (unsuccessful) startups, and I've written one fairly pragmatic book on Web testing. I got the faculty offer at MSU not just in spite of this, but because I expressed a certain (guarded) unhappiness with computer science in general -- I still don't know what they were thinking, but I'm happy they offered me a job ;).

So I think that while I'm not a card carrying genius (like Paul Graham) or an annoying corporate know-it-all (like Joel Spolsky) or even a terribly successful academic (like, umm... lots of people?) I have gotten my feet damp in all these areas. I know what real software looks like. I know what academic software looks like (oh boy, do I ever!) I've done scientific research, albeit not so much in computer science as in biology. In fact, my interest in software development testing comes from having seen how corporations are often at a loss to implement novel features and pragmatic test regimes; how research software presents different programming challenges while also being a source of unending (and often useless) novelty; and how open source programs and methodology provide solutions, however confusing and contradictory, to many corporate and academic challenges.

Now that I've presented my creds and tried to convince you that I have a chance of knowing what I'm talking about, let's set up some straw men.

Straw man #1: Computer Science undergrad education is useless.

I'm sensitive to this argument because I frankly have no formal CS training. I took a compiler class (C+), an OS course (B? A? not sure), and a graphics course (F) back in the dawn of time. Virtually everything I know about programming I learned by myself or from others, and virtually none of it is taught in any formal CS curriculum that I've seen.

Now that I've seen the elephant, however, I am sad that I didn't pay more attention when I had the chance. The compiler course would have been really useful, in particular, for reasons discussed @CTB here. The few things I remember from the OS course come in handy every time I need to think about network programming, memory management, and efficient OS interaction.

So I would say this: CS undergrad curricula has a much tougher job to do than most give it credit. CS undergrad first has to separate the wheat from the chaff: there are unmotivated people, there are dumb people, and there is even some evidence that people who lack a certain mindset cannot learn to program. Then people with at least minimal motivation and aptitude must be taught, from scratch, to program -- sometimes over the corpses of their earlier high school or open source/PHP/Perl training. Finally, these programmers, often with less than two years of actual training, are thrown into the cauldron of design, methodologies, and implementation details that characterizes even an approximation of real-world software development. While the poor students are struggling with that load, they're expected to master the theoretical aspects of algorithmic analysis, compiler design, language theory, image processing, and lord knows what else. In another 2 years.

Given all of that, is it any wonder that most CS undergrad curricula are a mess?

The real question to be asked, however, is: does this all serve a useful purpose? And there I think the answer is, unambiguously, yes. The purpose is not to churn out quality programmers, because that's impossible without actual exposure to real-world programming. The purpose is to educate people in the breadth of possibilities -- to give them a taste of what is out there, with recourse to experienced, knowledgeable, and hopefully inspiring professors whose job it is to answer questions.

Straw man #2: Graduate school is generally a waste of time.

If you use sys.settrace to set a tracing function, and that function prints to sys.stdout`, then don't ever trash ``sys.stdout, even briefly. You will raise an invisible exception and your trace function will be removed.

(I don't know precisely what is supposed to happen when a trace function raises an exception, but it seems like it is being removed. Fair enough. But frustrating.)

This cost me a few days of intermittent repair attempts on figleaf.

Perhaps I will track down this behavior and document it in the Python docs...

--titus

There's been lots of discussion recently about the global interpreter lock, and how evil it is. (I personally don't think it's evil. More on that some day when I write code again, if ever.)

Then I read this article, on parrot and Threading Building Blocks, and somethign clicked. I quote, from http://osstbb.intel.com/:

Threading Building Blocks is not just a threads-replacement library. It represents a higher-level, task-based parallelism that abstracts platform details and threading mechanism for performance and scalability.

Perhaps I'm naive -- no, wait, I definitely am -- but doesn't this sound a little bit like any threadsafe function in Python?

When I call a C function from CPython -- say,

>>> my_very_own_C_function()

or

>>> someone_elses_C_function()

I don't actually know if it's thread-aware or not. I don't need to know, unless I become concerned about performance -- in which case I can very quickly check to see if it's threadsafe by grepping the code, reading the docs, or just running a few benchmarks. I do know (or at least can presume) that it's threadsafe, though, because all CPython extension functions are by default run in a threadsafe manner. Now let's suppose I want to make it thread-aware: I go through the code, isolate sections that can be run without reference to Python objects, and wrap those sections in Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS macros. Voila, thread-aware code that will execute considerably faster on multihreaded platforms.

So, I have an interpreter that (by default) runs code in a threadsafe environment, but can be gently nudged into running thread-aware code in parallel and in a machine-independent way ("abstracting platform details"); the code is inherently task based (I do try to make functions do tasks, like most programmers); and, err, it's definitely higher-level.

Yes, this analogy is somewhat tongue in cheek. Nonetheless it's worth pointing out that the GiL discussions give the impression that people want something great (multithreaded code) for free (that is, without having to think about it or write it). From my current perspective, Python does a pretty good job of letting you get on with your (multithreaded) life while forcing you to think of threads only in code where you specifically care.

--titus