E-Scribe News : a programmer’s blog

About Me

PBX I'm Paul Bissex. I build web applications using open source software, especially Django. Backstory: In the 1990s I did graphic design for newspapers and magazines. Then I wrote technology commentary and reviews for Wired, Salon.com, Chicago Tribune, and lots of little places you've never heard of. Then I taught photographers how to create good websites. I co-wrote a book (see below) along the way. Current story: I am helping turn a giant media corporation into a digital enterprise. Feel free to email me.

Book

I'm co-author of "Python Web Development with Django", an excellent guide to my favorite web framework. Published by Addison-Wesley, it is available from Amazon and your favorite technical bookstore as well.

Colophon

Built using Django, served by Apache and mod_wsgi. The database is SQLite. The operating system is FreeBSD, on a VPS hosted at Johncompanies.com. Comment-spam protection by Akismet. Vintage topo imagery from the Maptech archive. The markup engine is Markdown.

Pile o'Tags

Stuff I Use

bitbucket, Django, Emacs, FreeBSD, Git, jQuery, LaunchBar, Markdown, Mercurial, OS X, Python, Review Board, S3, SQLite, Sublime Text, Ubuntu Linux

Spam Report

At least 201066 pieces of comment spam killed since January 2008, mostly via Akismet.

Creativity and Constraint

O'Reilly Media recently asked in their "Programming Today" newsletter:

"For many old-school software engineers, developing code has always been as much an art as a science. But as the industry focuses more on practices such as Test-Driven Development, and Patterns become the lingua franca of programming, is there any room left for real creativity in coding? Or, has it become an exercise in cookie-cutter production, putting together components in new ways, but without any real room for individual style? Share your thoughts with us..."

My response:

The idea that TDD or Patterns are in opposition to creativity in coding is a false dichotomy.

Our work is inherently about working effectively within constraints.

Those constraints may be related to the business need -- time to delivery, performance requirements, budget limits that circumscribe personnel or technology choices.

Or those constraints may be inherent in the technologies our projects use -- we may have bigger or smaller standard libraries, larger or smaller variety of developer tools, greater or lesser impediments to integration. And we have varying levels of access to the source of software products/services/components that our projects are built on.

I'd say these facts are no more an indication of impinged creativity than are the material facts of other craftspeople. A carpenter must work with the limits of wood as a material -- but a good carpenter knows and leverages its strong suits as well.

If a software shop is managed such that TDD or Patterns becomes a doctrine that must be followed without regard for its effectiveness, that's a management problem with that shop -- not an indictment of the tools and techniques being abused.

Great art and craft and science work of all sorts always has constraints. Creativity manifests itself in response to those constraints.

Those are my thoughts.

Friday, April 26th, 2013

Booktools

A little-known bit of trivia about our book, Python Web Development with Django: we wrote the manuscript in Markdown.

I think it was my idea. One of the major motivations for using a text-based format -- versus the unfortunate de facto standard, Microsoft Word -- was integration with good developer tools and workflow.

Our manuscript and all our project code was in a Subversion repo, so each author always had the latest updates. HTML generated from the Markdown files was great for generating nice printed/printable output too.

We could have used any number of similar formats: Markdown, Textile, reStructuredText. If we did it again we'd probably use reST plus Sphinx. That would grant all the same advantages, plus give us a little more formatting flexibility and tool support.

This workflow enabled certain kinds of programmatic action on our text, notably two things: automated testing of the interactive examples within the text, and automated extraction of example snippets from source code files.

I wrote scripts for each of these tasks. I've cleaned them up a little, to try to make them a little more general-purpose, and published them (with minimal example files) in a bitbucket repo: http://bitbucket.org/pbx/booktools

There's a little documentation in the docstrings of the scripts. Here's the summary:

To test code snippets in the manuscript file: test_snippets.py example/text.txt

To extract code from source files into the manuscript file: try_excerpt.py example/text.txt

Authors, make use of this if you can -- or maybe even better, take inspiration from the idea and implement a system of your own.

Tuesday, September 14th, 2010
+ +
2 comments

A different kind of URL shortener

Today I'm launching my first Google App Engine site. While I built it largely to play with GAE, it is also useful in its own right (I like to think so anyway). It does two different things:

Link shortening without redirection. Put in a godawful long Amazon link and get back a shorter Amazon link. Works with eBay and a few others too. I welcome recipes for other sites. (For the programmers in the audience, which is most of you -- yes, the processing is via regular expressions.)

It does some basic checks to confirm that the shortened URL returns the same page as the original one.

Link expansion. Put in a link from a URL shortening/redirection service, e.g. bit.ly, and see where it redirects to. Works with a slew of popular link-shorteners, including the house brands goo.gl and nyti.ms.

Some of the shortening services do offer a way to see the link target before you visit it, but they're all different; this presents a simple unified interface to that feature.

There's a bookmarklet too. If you have someone in your online life who frequently bombards you with, say, mile-long eBay links, tell them about it.

http://urlworks.appspot.com/

Sunday, August 29th, 2010
+ +
4 comments

The syncbox

I move between a couple different computers regularly: my old 12" PowerBook and the 15" MacBook Pro my job provides me with. Like all multi-computer users I periodically bump up against the challenges of what files (and versions) are where, especially when there's work in progress.

To further complicate things, I also have an extra laptop running Ubuntu. And sometimes I just SSH to my web server from somebody else's machine.

I spent a while thinking about solutions. Some people keep a "master" home directory on a server, using rsync to pull new copies (or freshen old copies) on machines where they work. Being an rsync fan, I tried this approach. After my first accidental rsync --delete casualty, though, I started thinking about ways to preserve history.

That's when the ideal solution hit me (making a big resonant "DUH" sound): distributed version control. Perfect synchronization: check. Multi-platform clients: check. Full history: check.

I created a Mercurial repository on my web server, then cloned it out to the two laptops.

For stuff that needs to be secure, I decided that simple command-line encryption was the answer (hence this tweet from a while back with a Blowfish encrypt/decrypt one-liner). And I use SSH for transport, so even the plaintext stuff is safe from in-transit snooping.

I call the synced directory "syncbox". It contains a little script for keeping things in sync. It amounts to these steps:

hg addremove
hg commit -m "Update"
hg push
hg fetch

Ironically, after having set all this up, I got an invite to try Dropbox, a nifty-looking service that offers many of the same benefits and many other features besides (e.g. desktop OS integration, selective file sharing, browser-based acess option). About all I can tout for advantages of my approach are: 1) unlimited history (Dropbox gives you 30 days), 2) no additional fees if I exceed 2G of storage, and 3) I control it completely.

Sunday, June 13th, 2010
+ +
2 comments

Branching and merging in real life

At work I still mostly use Subversion for version control. Its main selling points: stable, performs as expected, integrates nicely with Trac, holds all our old stuff (legacy inertia).

Note that "pain-free branching and merging" is not on that list. (And don't give me the old "branching is cheap in svn!" line. It's not about the branching, it's about the merging.) A couple years ago I started also using Mercurial and plan to eventually replace svn with it entirely. The aspect of Mercurial that made my life better recently is its support for branching and merging.

The scenario: an important internal web app (in use all day every school day) needed some significant changes on a short timetable. Normally I'd work on the app thus: edit the staging copy, commit, update the live copy. I didn't want to take that approach here. I knew that during the development window there might arise unrelated urgent change requests; I wanted to keep the new code isolated during development, but also deploy and track those unrelated urgent changes. Branching seemed like the right approach.

I could have made a full clone of the app (hg clone mainrepo newrepo). However, handling environment dependencies (web server, PythonPath, database) would have added time and fussiness to the job, and time was in short supply. So, using Mercurial's named-branches feature, I made a new branch (hg branch newstuff) right inside the fully-functional staging copy of the app. That way I was able to develop and test as usual, secure that my unproven work-in-progress was not "polluting" the current app's revision history.

To handle "unrelated urgent changes" as mentioned above, I'd:

  1. Commit any current work on the "newstuff" branch
  2. Switch to the main branch (hg update -r default)
  3. Make the urgent change, test, commit, update the live copy
  4. Switch back to the new branch (hg update -r newstuff)

It took me a couple tries to understand how branch-switching worked, but it's simple: you really are updating your working directory to a new revision, it just happens to be a revision stored in a different branch from the current one.

It was fun looking at the graph (via HgWeb) and seeing my two parallel branches with their individual commits.

The moment of truth came at the end of the day Friday, when it was time to merge the tested and complete "newstuff" code with the current live codebase. It was dead simple, and effectively instantaneous. Condensed version: hg update -r default; hg merge -r newstuff; hg ci -m "merged new stuff". Followed by: update live copy and let out a big sigh.

Saturday, November 7th, 2009
+ +
8 comments