E-Scribe News : a programmer’s blog

About Me

PBX I'm Paul Bissex. I build web applications using open source software, especially Django. Backstory: In the 1990s I did graphic design for newspapers and magazines. Then I wrote technology commentary and reviews for Wired, Salon.com, Chicago Tribune, and lots of little places you've never heard of. Then I taught photographers how to create good websites. I co-wrote a book (see below) along the way. Current story: I am helping turn a giant media corporation into a digital enterprise. Feel free to email me.

Book

I'm co-author of "Python Web Development with Django", an excellent guide to my favorite web framework. Published by Addison-Wesley, it is available from Amazon and your favorite technical bookstore as well.

Colophon

Built using Django, served by Apache and mod_wsgi. The database is SQLite. The operating system is FreeBSD, on a VPS hosted at Johncompanies.com. Comment-spam protection by Akismet. Vintage topo imagery from the Maptech archive. The markup engine is Markdown.

Pile o'Tags

Stuff I Use

bitbucket, Django, Emacs, FreeBSD, Git, jQuery, LaunchBar, Markdown, Mercurial, OS X, Python, Review Board, S3, SQLite, Sublime Text, Ubuntu Linux

Spam Report

At least 222162 pieces of comment spam killed since January 2008, mostly via Akismet.

The story of dpaste.com 2.0

Eight years ago, I launched a simple pastebin site written in Django.

In those early Django days I spent a lot of time in the #django IRC channel. I thought we should have a pastebin that knew how to correctly colorize our code, and which was written in our framework to boot. So I wrote one. Eventually its URL ended up in the channel topic, then in the Django source code itself.

Over the years, changes have been minimal. I switched from a Javascript-powered colorizer to Pygments. I added an API. I fixed things that broke. Mostly I just kept it running and usable. (I'll also note that many excellent new pastebins were created in those years as well.)

Of course I kept a growing list of "someday" improvements and fixes. Despite having no free time (full-time job, twin baby boys, 100-year-old house), a year or so ago I started hacking away on the items on that list. In very tiny steps.

Eventually I had enough in place for what I call the 2.0 release of dpaste.com. I presented a preview to my co-workers (many staunch dpaste users among them), and went live about three weeks ago.

I call out the fun new stuff on the about page:

One obvious change is the switch from auto-incrementing integer IDs (1.7 million at last count) to 7-digit base-32 IDs like "1S2BP7E". This ID is generated by hashing some of the paste data fields, using MurmurHash. I wrote a simple library called basewhat to handle the base-32 conversions.

Some other improvements are less visible but still important:

Every good developer knows that version control plus a good test suite help you go faster with more confidence. If you also deal with the operations side (i.e. deployment), I'd add Fabric to that list. By reducing error and tedium in maintenance and deployment tasks, Fabric likewise increases your likelihood of doing them consistently and correctly. If pulling the latest code to the live server and restarting the app is a single command I can run from anywhere, I'm going to do it more.

Thanks to all the authors of the open source software that this project is built on. And thanks to all the users, who motivate me to keep it going just by using it.

As they say in customer service land, "We know you have a choice of pastebins, and we thank you for choosing dpaste.com."

Friday, May 16th, 2014
+ + + +
6 comments

Creativity and Constraint

O'Reilly Media recently asked in their "Programming Today" newsletter:

"For many old-school software engineers, developing code has always been as much an art as a science. But as the industry focuses more on practices such as Test-Driven Development, and Patterns become the lingua franca of programming, is there any room left for real creativity in coding? Or, has it become an exercise in cookie-cutter production, putting together components in new ways, but without any real room for individual style? Share your thoughts with us..."

My response:

The idea that TDD or Patterns are in opposition to creativity in coding is a false dichotomy.

Our work is inherently about working effectively within constraints.

Those constraints may be related to the business need -- time to delivery, performance requirements, budget limits that circumscribe personnel or technology choices.

Or those constraints may be inherent in the technologies our projects use -- we may have bigger or smaller standard libraries, larger or smaller variety of developer tools, greater or lesser impediments to integration. And we have varying levels of access to the source of software products/services/components that our projects are built on.

I'd say these facts are no more an indication of impinged creativity than are the material facts of other craftspeople. A carpenter must work with the limits of wood as a material -- but a good carpenter knows and leverages its strong suits as well.

If a software shop is managed such that TDD or Patterns becomes a doctrine that must be followed without regard for its effectiveness, that's a management problem with that shop -- not an indictment of the tools and techniques being abused.

Great art and craft and science work of all sorts always has constraints. Creativity manifests itself in response to those constraints.

Those are my thoughts.

Friday, April 26th, 2013

Booktools

A little-known bit of trivia about our book, Python Web Development with Django: we wrote the manuscript in Markdown.

I think it was my idea. One of the major motivations for using a text-based format -- versus the unfortunate de facto standard, Microsoft Word -- was integration with good developer tools and workflow.

Our manuscript and all our project code was in a Subversion repo, so each author always had the latest updates. HTML generated from the Markdown files was great for generating nice printed/printable output too.

We could have used any number of similar formats: Markdown, Textile, reStructuredText. If we did it again we'd probably use reST plus Sphinx. That would grant all the same advantages, plus give us a little more formatting flexibility and tool support.

This workflow enabled certain kinds of programmatic action on our text, notably two things: automated testing of the interactive examples within the text, and automated extraction of example snippets from source code files.

I wrote scripts for each of these tasks. I've cleaned them up a little, to try to make them a little more general-purpose, and published them (with minimal example files) in a bitbucket repo: http://bitbucket.org/pbx/booktools

There's a little documentation in the docstrings of the scripts. Here's the summary:

To test code snippets in the manuscript file: test_snippets.py example/text.txt

To extract code from source files into the manuscript file: try_excerpt.py example/text.txt

Authors, make use of this if you can -- or maybe even better, take inspiration from the idea and implement a system of your own.

Tuesday, September 14th, 2010
+ +
2 comments

A different kind of URL shortener

Today I'm launching my first Google App Engine site. While I built it largely to play with GAE, it is also useful in its own right (I like to think so anyway). It does two different things:

Link shortening without redirection. Put in a godawful long Amazon link and get back a shorter Amazon link. Works with eBay and a few others too. I welcome recipes for other sites. (For the programmers in the audience, which is most of you -- yes, the processing is via regular expressions.)

It does some basic checks to confirm that the shortened URL returns the same page as the original one.

Link expansion. Put in a link from a URL shortening/redirection service, e.g. bit.ly, and see where it redirects to. Works with a slew of popular link-shorteners, including the house brands goo.gl and nyti.ms.

Some of the shortening services do offer a way to see the link target before you visit it, but they're all different; this presents a simple unified interface to that feature.

There's a bookmarklet too. If you have someone in your online life who frequently bombards you with, say, mile-long eBay links, tell them about it.

http://urlworks.appspot.com/

Sunday, August 29th, 2010
+ +
4 comments

The syncbox

I move between a couple different computers regularly: my old 12" PowerBook and the 15" MacBook Pro my job provides me with. Like all multi-computer users I periodically bump up against the challenges of what files (and versions) are where, especially when there's work in progress.

To further complicate things, I also have an extra laptop running Ubuntu. And sometimes I just SSH to my web server from somebody else's machine.

I spent a while thinking about solutions. Some people keep a "master" home directory on a server, using rsync to pull new copies (or freshen old copies) on machines where they work. Being an rsync fan, I tried this approach. After my first accidental rsync --delete casualty, though, I started thinking about ways to preserve history.

That's when the ideal solution hit me (making a big resonant "DUH" sound): distributed version control. Perfect synchronization: check. Multi-platform clients: check. Full history: check.

I created a Mercurial repository on my web server, then cloned it out to the two laptops.

For stuff that needs to be secure, I decided that simple command-line encryption was the answer (hence this tweet from a while back with a Blowfish encrypt/decrypt one-liner). And I use SSH for transport, so even the plaintext stuff is safe from in-transit snooping.

I call the synced directory "syncbox". It contains a little script for keeping things in sync. It amounts to these steps:

hg addremove
hg commit -m "Update"
hg push
hg fetch

Ironically, after having set all this up, I got an invite to try Dropbox, a nifty-looking service that offers many of the same benefits and many other features besides (e.g. desktop OS integration, selective file sharing, browser-based acess option). About all I can tout for advantages of my approach are: 1) unlimited history (Dropbox gives you 30 days), 2) no additional fees if I exceed 2G of storage, and 3) I control it completely.

Sunday, June 13th, 2010
+ +
2 comments