I don't feel like I need to "defend" twill -- it's successful beyond both my expectations and my cognizance (I have no idea who's actually using it, but it's apparently a lot of people!), but I may need to promote it better. I ran across this post earlier today. It shows how you can use mechanize to do some simple "screen scraping," and it spurred me to check the following scripts into a new "advocacy" section of the twill archive.
Here's how you can use a twill script to do what Greg did with mechanize:
add_extra_header User-Agent "Mozilla/5.0 (compatible; MyProgram/0.1)" go http://www.python.org/ show
You can also use straight Python, if that's your poison:
from twill.commands import * import twill add_extra_header('User-Agent', 'Mozilla/5.0 (compatible; MyProgram/0.1)') go("http://python.org/") html = twill.get_browser().get_html() print html
Note that twill is based on mechanize, and so promoting twill doesn't mean pushing mechanize down. mechanize is amazingly powerful -- but sometimes you want to just go grab some HTML, and twill (tries to) make that easy.
One unexpected result of all this -- I discovered that the function get_browser() wasn't actually exported from twill.commands by default, which is silly. It's the first function I wanted to call in order to do something minimally complex. So now the API is one call bigger ;). One more corner rubbed off...
Posted by Diane on 2007-02-13 at 17:30.
$ easy_install twill ... $ ipython In: from twill.commands import * In: import twill In: go("http://google.com") In: html = twill.get_browser().get_html() In: html Out: '' In: twill.<em>_version_</em> Out: '0.8.5' Somehow I think there's supposed to be some data in that html variable? Also the version of twill obtained via easy_install doesn't have 'add_extra_header'. Finally why is there twill.commands.get_browser() and twill.get_browser()
Posted by Titus Brown on 2007-02-14 at 03:34.
Now that's just mean, pointing out that I haven't made a release in a while ;) twill.commands.get_browser implements twill.get_browser. The other functions are all available in the latest version of twill. --titus