Daily Life in an Ivory Basement

Steve Yegge recently wrote a long article, "Code's Worst Enemy", about how "many lines of code" causes problems in projects.

That's obviously pretty silly. To see why, let's examine a little project I've recently started; conservatively, I estimate that it incorporates well over a million lines of code:

print 'hello, world'

Well, that's one line.

But what's needed to run it? The Python interpreter; the C compiler (to build the Python interpreter); the libraries necessary to run Python and actually make that statement appear on the screen; and the Linux (or Mac O$ X, or Window$) operating system and drivers needed to bind them all together.

There's easily a million lines of C code in there, if not ten million. So have I just coded on the most bloated, worst project of all time?

Nope. It's not the lines of code that matter. It's the lines of code you need to think about that matter.

When I write Python code, I rarely need to worry about anything other than the code I'm writing. I don't need to think about the std lib all that much, I certainly don't worry about the CPython core code, and I touch on the UNIX kernel very infrequently. Why?

Because all of that other code is nicely encapsulated, behaves in expected ways, and rarely breaks.

And this is why having a full language, good libraries, and a reliable OS are all ways to decrease the "brain load" of your software.

I also think it points to a deep truth of software engineering, which is that a good library API is one that you don't need to think about much. A good library should be compact, inclusive of core features, work reliably, and contain functionality orthogonal to your code. All of these things will help you worry about the core functionality of your own code and not about any other code.

By almost any measure (excepting that of life itself) our software is unimaginably (and unmanageably) complex already. We manage that complexity as best we can by encapsulating functionality in libraries, APIs, protocols, and "expectations" that are fulfilled, more often than not. And that's the real lesson you should take away from Yegge's post: that writing a 500k Ball of Mud is a bad idea indeed, but more because his design process failed well before he got to 500k LoJC.

--titus

P.S. JavaScript? Really? This reminds me of Phil Greenspun telling me how great Tcl was as a way to develop a large framework -- and that didn't end well...

Brett Cannon notes that GHOP is working out well, and muses about the future of the CPython test infrastructure (among other things). This is something I'm interested in as well (guess where all those testing tasks in GHOP came from? ;) and I've been confused, if not frustrated, by the apparent complexity of the CPython test suite.

One of the techniques I set up a year or so ago, after I first wrote figleaf, was to run code coverage analysis on the trunk tests. This was more trouble than it should have been: the test runner itself needed to be patched. This is because code coverage analysis relies on setting the trace function with sys.settrace, and a number of tests delete the system trace function, either with sys.settrace or with internall calls.

Tonight, inspired both by Brett's post and some cleanup work I'm doing on figleaf, I tracked down all of the modules responsible for deleting the trace function and excluded them from my test run. The result? A (fairly simple) command-line code coverage analysis run that I think could easily meet Brett's request for a nightly coverage analysis.

To run this yourself, you need to grab the latest version of figleaf, unpack it somewhere, and then run the following command in your build directory:

% ./python /path/to/bin/figleaf Lib/test/regrtest.py `cat traceless`
% ./python /path/to/bin/figleaf2html

(See the bottom of this post for the contents of the traceless file.)

This runs figleaf with the dev version of Python and excludes all of the modules that set the trace function, directly or indirectly.

The results can be seen here for a short while.

It would be easy to get this running on Windows as well, if only there were a nice way to tell regrtest to exclude those modules without having a really long command line...

--titus

Here's the 'traceless' file:

-x test_trace
-x test_scope
-x test_doctest
-x test_asynchat
-x test_asyncore
-x test_capi
-x test_decimal
-x test_docxmlrpc
-x test_ftplib
-x test_logging
-x test_poplib
-x test_queue
-x test_smtplib
-x test_sys
-x test_telnetlib
-x test_threaded_import
-x test_threadedtempfile
-x test_threading
-x test_threading_local
-x test_urllib2_localnet
-x test_xmlrpc
-x test_builtin
-x test_hotshot
-x test_exceptions
-x test_coercion
-x test_richcmp

Andrew Binstock hits the nail perfectly on the head with his post, Beautiful Code vs Readable Code.

--titus

Matt Harrison's post, Gnome devs too lazy for python?, and the linked post by Thomas Vander Stichele strongly typed, are both really entertaining and illuminating.

I preserve them here for my own reference.

--titus

So, the Google Highly Open Participation Contest is going quite well for Python, with about 25 additions/reworkings of core CPython documentation and tests and dozens of contributions to other projects.

One of the really unique opportunities that GHOP offers for the Python community is being underutilized, however, and that is the potential for doc, example, and tutorial translation. Students from all over the world can participate in GHOP, and we are happy to post tasks that have them translate things for Python projects.

For example, as a result of Andre Roberge's involvement in GHOP, we've had quite a few different translations made up for Crunchy -- as of today, these include German, Hungarian, Esperanto, Portuguese, Italian, Macedonian, Estonian, and Spanish.

I'm happy to write up the task descriptions for Python projects if you give me specific documents or example scripts that need translations. Bear in mind that these tasks don't need to be too big, and in fact if they're particularly small we can always request a screencast demo in a particular language.

I think translation tasks fit especially well with the GHOP project because they encourage involvement in new projects by younger, potentially less experienced students, and the translated docs facilitate wider project use (and hopefully participation) by others.

So, please, get in touch with us over at the Python GHOP project or just e-mail me personally.

--titus