JSXHint 0.3.0 released


A quick note: v0.3.0 of JSXHint is now published and available via NPM.

npm install -g jsxhint

Why the big deal? Well, there are finally tests! And travis integration to show that the tests work.

Why I Canceled Amazon Prime


Mr Bezos:

Today I had to cancel my Amazon Prime membership due to circumstances that are partially out of your control. You see, the problem with Amazon Prime is that you are shipping almost entirely via the USPS. Normally, I bet, in most of the country, this doesn't bode a single issue. It not only helps the beleaguered agency, but doesn't change life for many of your customers. Well, except for those that live in large metropolitan areas.

Being a resident of Brooklyn has it's ups and downs, but one of the major downs is that the postal system is pretty wildly broken here. Carriers don't usually finish delivering the mail until well after 8PM in my neighborhood and since they're on foot, this limits their capacity to carry packages with them. This means that when the carrier comes to my house, they don't ever have the package, but rather one of the dreaded package pick-up slips.

This means that I need to now go down to the post office in my neighborhood (luckily mine is only a few blocks away — my friend who lives down the street has to go nearly two miles with no easy public transportation) and wait in line to get my packages, but then I have to carry them home.

As my wife and I are expecting our second child, you'd image that the number of packages we've been ordering has increased dramatically: car seats, strollers, cribs, etc. For our first kid it was great — UPS or FedEx happily rang the door bell and gave us our stuff, but this experience has been entirely different. This morning however, was the final straw. I went to pick up more package with my wife's name on them. Despite the fact we have the same last name and despite the fact we have the same address the post office refused to hand them over to me without my wife's ID. The hard part though: my wife needs her ID during the day as she drives. When I asked why the carrier never has the packages, the supervisor brought out my carrier who started yelling at my telling me I'm never home. At this point four of the six people in line behind me started saying they have the same problem: they only get slips and have to come to the post office to pick them up.

An all too common sight at the post office

An all too common sight at the post office

This isn't the best way to build loyal customers.

The time before this, they actually gave me my packages, but then I was left carrying over 50lbs worth of cribs and strollers all the way home. Now imagine if my eight and a half month pregnant wife had to do that? Or someone who is mobility impaired and relying on shipping-to-the-door.

I understand that you're probably looking to save a little money, but, in the end, the customer service might not be worth it. Before we actually canceled our prime membership today my wife and I had received over $100 worth of account credits for having to deal with the post office. Given how quickly your agents offer this remedy, we can't be the only ones.

However, I would happily pay more than $79/year for a version of prime which guarantees that the USPS will not be involved in shipping: not via FedEx SmartPost, not direct, not ever. I would venture to say there are a lot more people like me out there.

Thank you for your time,

Todd Kennedy, former Amazon Prime member.

JSXHint, ReactJS and Pre-Mature Optimization


By now many of you have probably heard about the new project from Facebook called ReactJS. There was a lot of buzz gained about it around the time their Javascript Jabber interview was released.

We did some investigation into the library at Condé Nast after a coworker brought it up while we were reviewing schemes for server-side rendering of javascript app content for indexing by Google, Facebook, Twitter, etc, so this prompted a deeper inspection into the toolkit. The first thing to get my head wrapped around was the custom DOM creation and manipulation syntax they call JSX.

Lets start with the elephant in the room: JSX?

Is this some sort of template language? Specifically no. This might have been the first big stumbling block. What looks like to be a templating language is actually an in-line DSL that gets transpiled directly into JavaScript by the JSX transpiler.

We write this:

/** @jsx React.DOM */
<div class="alert">
    <button type="button" class="close" onClink={this.closeHandler}>&times;</button>
    Status for {this.props.data.get('brandLead')} was {this.props.state === 'failure' ? 'not' : ''} saved

(Be honest. First reaction: Gah. That looks likes E4X.) Yeah, yeah, I know, but like ignore that for a minute, since after we run the JSX transpiler on this, we get:

React.DOM.div( {className:"alert"},
              React.DOM.button( {type:"button", className:"close", onClick:this.closeHandler}, "×"),
" Status for ", this.props.data.get('brandLead'), " was ", this.props.state === 'failure' ? 'not' : '', " saved "            )

(Whitespacing is intentional - this is a cut and paste from the output) Much better? Right? Eh. Sort of. At least we're not relying on a partially implemented ECMA standard this time.

So now for the next question, what is React.DOM and where does it come from? See that /* @jsx React.DOM */ comment? That's a transpiler directive that tells the JSX transpiler to load the React.DOM object and look for the tags you've defined in your HTML snippet as functions in that object.

This is where the "it's not a template language" rings totally true. We aren't actually writing HTML in our source, we're short-cutting using the React.DOM object and it's methods directly.1.

OK. So what's the point of a DOM DSL?

First we need to start with a statement: DOM manipulation is slow.

Running this extremely reactive case:

var ul = document.createElement('ul');
for(var i = 0; i > 100000; i++){

Is slower than:

var ul = document.createElement('ul');
for(var j = 0; j > 100000; j++){

results in Firefox Aurora 27.0a2 (2013-11-10)


Creating elements in memory is quick -- copying those elements into the DOM is where the slowness occurs. This is due to a variety of issues, most namely reflow/paint. Changing the items in the DOM causes the browser to re-paint the display, apply styles, etc. We want to keep those operations to an absolute minimum, especially if we're dealing with something that needs to update the DOM frequently. (Like Facebook's constant stream of crap.) What JSX attempts to fix is DOM manipulation specifically. When you work with JSX or the React.DOM object, you're working on a sort of "shadow DOM" that handles all the state changes and gets the new version of the DOM, compares it to the current version in the browser, and then only makes the necessary changes to the current DOM. The downside to this? Directly manipulating the DOM is a) difficult and b) not recommended since you lose most of what you're using React for in the first place.

Which leads me to my summary of JSX:

  1. In the third line of the source example we've got a this.props.state === ternary operator. All the bally-hoo about logic-less and less-logic templates aside, in most2 templating languages you can't just throw in any random JavaScript statement, but rather must use the template's specific DSL to effect logic. JSX, however, since it's "just JavaScript" allows for inline execution of arbitrary JavaScript.
  2. JSX files are un-lintable without extending the tools used for linting.3
  3. The developers are required to update the React.DOM class with any new HTML tags that might be implemented, if you require custom DOM elements for some other purpose, you'll need to write your own mixin to allow rendering of these components in your site.
  4. It becomes extremely difficult to use jQuery/mootools/etc and their ecosystems of plugins without doing some custom work with them. (More on this in another post).

1This is why they say JSX is not required for using React. You can write your HTML out directly using the React.DOM syntax. Kind of a Fasutian bargain if you ask me...

2I understand that underscore's templates allow for direct JavaScript interpolation, but the majority do not.

3Probably befitting it's own post, I created JSXHint to be able to lint files containing JSX syntax directly. (It transpiles them on the fly and then checks the output).

We'll dive deeper into React (leaving JSX's utility as an argument for the reader since it's really opinion based more than anything) in the next post.

Sharing is Caring: The Talk


Here are the slides I used for talk I gave at last night's HTML5 Application Developers Meetup. You can see them directly on github.

There might be video forth coming, but the sound was pretty miserable sadly.

Update And here's the video!

Sharing is Caring with Todd Kennedy from John K. Paul on Vimeo.

Putting Your Money Where Your Mouth Is


In my previous1 posts2 I've been talking about a method for sharing Backbone models between the front and back end so you can unify that part of your code base.

After a few minutes of thinking about this I said this would probably make a great module that you can import rather than having to cut and paste between a bunch of files on some asshole's blog.

It's published both into NPM and Bower depending on how you like to organize your code, so fire up your command lines and do a little npm install mongosync or bower install mongosync. The NPM version works with Browserify, either standalone or bundled into a package as part of a larger project.

The most beguiling part by far was managing to have the code export itself correctly depending on how you want to package your stuff, being three main choices: RequireJS, CommonJS or, for the brave, foolish or bold, it'll even attach itself directly to the context in which it is run (so, for many people using a plain old <script> tag, the window object.)

Thanks to some help from @thlorenz, this wasn't super hard to do:

if(typeof exports !== 'undefined'){
    module.exports = SharedModel;
} else if (typeof define === 'function' && define.amd ){
        return SharedModel;
} else {
    global.SharedModel = SharedModel;

We just have to do some detection to figure out if exports (commonJS) is available, and if so, great, lets attach this to module.exports. If that's not there, we should look to see if we're in a AMD/define condition (RequireJS) and then actually make a define() call, and then finally attach ourselves to the global object (which is passed in as the outer closure's this by the IIFE wrapper.)

The other thing that I changed is that this actually returns a Backbone.Model instance with correct version of Backbone.Sync attached as well as the idAttribute set to _id. So instead of having to handle the logic behind swapping out sync on your own, it handles that and then returns the model itself for you to .extend() away from.

More accurately though, it actually retuns a function which then returns the model instance. This allows you to pass in the server and collection name to MongoJS.

SharedModel = function(db, coll){
    // this part can only run on the backend, `process.browser` is from browserify
    // and allows browserify to determine it's on the server
    if(typeof process !== 'undefined' && !process.browser && typeof exports !== 'undefined'){
      server_name = db;
      collection_name = coll;
      Backbone.sync = sync;

    var model = Backbone.Model.extend({idAttribute: '_id'});
    return model;

In here however, we have to be careful to get our context right. If we're using Browserify we need to ensure that we're not in the browser, which we can test by the non-existance of process.browser (in plain node browser is undefined, so that works for that use-case as well) and the existance of exports. Then it's simply setting the passed in db and collection names to the closure's placeholder variables, that the replaced sync method also references and then passing back the model.

Wrapping this all up with the sync method (pretty much wholesale from the earlier post), and we're all set! The biggest caveat is that this doesn't work for collections correctly yet, but that work is already underway.

Source on Github

1 Don't Repeat Yourself - Share With Yourself

2 Context Aware Backbone Models

Context Aware Backbone Models


Previous post: Don't Repeat Yourself - Share With Yourself!

In my previous post I showed a drop-in Backbone.Sync replacement for saving models to MongoDB. Obviously this sync method needs to be available when you've instantiated your model on the back end, but shouldn't exist on the front end version of the model.

So how do we selectively override attaching our custom sync method? Honestly that depends on what module system you're using for your code on the front end. I'll show RequireJS/AMD and Browserify versions below. If you're using Stitch or something else, you're on your own, but you should have a good start here. (But you should look into Browserify!)


With RequireJS we're going to need to make it so that the define method that RequireJS relies upon is available on the back end (and can pull back in your defined module from the return statement), so we'll use the amdefine package to handle this. Using the typeof define switch also provides a decent hook to determine that we're on the backend.

'use strict';

var define;
var sync;

if (typeof define !== 'function') {
    define = require('amdefine')(module);
    sync = require('mongo_sync');

    var Backbone = require('backbone');

    if(typeof sync === 'function'){
        Backbone.Sync = sync;

    return Backbone.Model.extend({});

We're not using the existence of the window or process objects here to determine context — if you're using mocha via phantomJS you'll run into issues here -- both window and process are available in that context, hence why're using hooking into the type definition of define.

Of course you'll need to shim Backbone, Underscore and jQuery in for this to work properly on the front end.

  paths: {
    'jquery': 'path/to/jquery',
    'underscore': 'path/to/underscore',
    'backbone': 'path/to/backbone',
  shim: {
    'backbone': {
      deps: ['underscore', 'jquery'],
      exports: 'Backbone'
    'underscore': {
      exports: '_'


I am a recent convert to the world of browserify for certain contexts (mainly when your entire code base is done in JavaScript), and this makes it even easier:

'use strict';

var Backbone = require('backbone');

    Backbone.Sync = require('mongo_sync');

module.exports = Backbone.Model.extend({});

Since Browserify attaches browser to the process object when it's running in the browser, it's a simple hook to override the sync method when we're just not in the browser.

Of course this does require setting up Browserify specifically to be able to provide jQuery to Backbone. We'll leave that for another post though. (Tip: check out Browserify-Shim)

So there you have it -- a method for sharing the same Backbone models, validation methods on those models, etc between the front and back end, so you can stop repeating yourself and concentrate on writing good code!

Previous post: Don't Repeat Yourself - Share With Yourself!

Don't Repeat Yourself - Share With Yourself!


Next post: Context Aware Backbone Models

There's a common adage/philosophy in software development in general: don't repeat yourself. The idea is that if you're copying and pasting code between sections or re-writing code that looks incredibly similar to other code you've got, you should be refactoring those sections of code so that if you need to change the logic for one of them, you'll change it once and not have to bother finding all instances.

Root of all evil? Eh...

The first instance I had of this (and this will probably date me) will learn this is when my CS teacher in High School showed me how to declare a constant in Pascal. (Yes. I said this would probably date me.)

    Pi = 3.14;
    CircleDegrees = 360;

For you whippersnappers in C you get:

#define PI 3.14;


float const PI = 3.14;

And you youngin's that use Java might even do:

public static final float PI = 3.14;

But then you'd have to get off my lawn and quiet down. (Also, seriously, public, static AND final?!)

I'm not this old...

"OK Grandpa, why are you telling us about punchcards, the Hollerith Tabulating Machine, the Jaquard Loom and your battles with the luddites?"

"That reminds of the time that I killed the Kaiser and prevented World War I...wait...no...back to programming."

So the above examples are pretty reductive, no? Defining constants so you can make sure that PI is the same wherever you use it (and yes, I know about #include <math.h> and import java.lang.Math — like I said, it was reductive.) So where I am going with this?

To co-opt Atwood's Law: all developer blogs eventually lead to Javascript.

Having just finished a lovely two days at Backbone Conf I figured this was an appropriate post to make. Recently we've been working on some projects that are powered by JS on both the backend (thanks Node.js) and the front-end, and we want to share code between them, since, well, have you been reading this so far?

But first a brief digression

When I first started using Backbone I was doing all of backend work in Python. Since the backend had the connection to the database, I'd define my model in the ORM I was using, usually SQLObject.But then I'd have to write a similar model for the front end and then endeavor to keep them in sync as the project would go under envitable redesigns, especially during prototyping and early development.

Mostly because this:

class Post(SQLObject):
    body = UnicodeCol()
    author = ForeignKey('users')
    posted_on = DateTimeCol(default=datetime.datetime.now)
    created_on = DateTimeCol(default=datetime.datetime.now)
    tags = RelatedJoin('tag')

is a pain in the ass to maintain along with this:

var Post = Backbone.Model.extend({
    defaults: {
        body: "",
        author: 0,
        posted_on: (new Date).toString(),
        created_on: (new Date).toString(),
        tags: []

And then you get the point of creating validators, and trying to maintain validation logic between both domains, and it's enough to make you want to stop programming.

Thankfully, having been a JavaScript developer for the past too many years (and ergo, not being scared of JavaScript), you say "oh I'll just write my backend in JavaScript and share code between the two!" And since Backbone is such a small library, we'll use the model system to demonstrate how this works since there will be much less to worry about.

Backbone's model system is really based around the concept of documents, so we're going to use mongoDB for our data persistance layer. The easiest way to make this all work is to just override Backbone.Sync (which is specifically designed for this.)

The method signature for Backbone.Sync is simple:

Backbone.Sync = function(method, model, options)

Where method maps to HTTP verbs:

model is the model you're operating on, and options is a hash of options for the sync method (which we'll ignore for now since we don't need it on the backend for this simple implementation).

So we'll start by writing a quick case statement inside our new Sync:

module.exports = function(method, model, options){
        case 'create':
        case 'update':
        case 'patch':
        case 'delete':
        case 'read':

We're going to make update and patch the same since mongoDB will handle "upserting" just changed data for us if that's the case with the model.

Now that we've got a basic skeleton for this, we need to also provide some interface for talking to mongo as well. We're going to use the very lightweight wrapper mongojs which will abstract some of the nastiness of the native mongoDB wrapper from us, so lets add that

var mongojs = require('mongojs');
module.exports = function(method, model, options){
    // nothing has changed in here yet

The native Backbone.Sync method nomially returns a jQuery $.Deferred() (unless you're using Zepto or something else as your $.ajax interface) object, and the since mongoDB works (like a lot of JavaScript) asynchronously, we're going to mimic that interface and return a then-able object. So lets add a default callback for the database operations which will resolve the promise and return that promise at the end of the sync method. (Also we call in our express app object here since we're using express in order to get our db config.)

var q = require('q');
var mongojs = require('mongojs');

module.exports = function(method, model, options){
    var d = q.defer();
    var app = require('../app');
    var db = mongojs(app.get('db_server'));
    var collection = db.collection(app.get('db_collection'));

    var  db_callback = function(err, response){
        } else {

    // nothing else has changed

    return d.promise;

MongoDB does some obnoxious stuff though that we're going to have to account for in that callback. If you've performed an "upsert" (which is an update to an existing model in place), the driver will actually just return 1 to let you know it was successful. Every other normal operation returns a copy of the data you've just sent. So we need to modify our callback for that operation. In this case the model being passed into the Sync method already has all the data we need attached to it, so we're just going to return back it's JSON representation once we know the datastore has been updated. And since mongoDB returns an array everytime, we'll resolve with just the first since we're only ever dealing with on model here (this is not collection-aware!)

    var  db_callback = function(err, response){
        } else {
            if(res === 1){
            } else {

Now lets just wire up our switch statement:

    case 'create':
        collection.insert(model.toJSON(), {safe: true}, db_callback);
    case 'update':
    case 'patch':
        collection.save(model.toJSON(), db_callback);
    case 'delete':
        d.reject(new Error('This is left as an exercise to the reader'));
    case 'read':
        var _id = model.get('_id');
        var query_params = {};
            query_params._id = mongojs.ObjectID(_id);
        } else {
            d.reject(new Error('You must provide an _id to lookup'));
        collection.find(query_params, db_callback);
        d.reject(new Error('Unimplemented'));

So all together you've now got:

MongoDB variant of Backbone.Sync

But then you've still got to figure out how to make the model do different things depending on the context: when you are on the client you want your save method to send data via an ajax call to the backend, but on the backend you want the save method to be able to actually write the data to your data persistance layer. Since this will depend on what method you're using for modules on the front end (and you are using modules on the front end right?), we'll leave this for the next post.

Next post: Context Aware Backbone Models

Talk about working for the man!


If I may (and like who is going to stop me? Superman?), I'm going to paraphrase one of my favorite scenes from Aliens where I am Bishop, and my wife is Ripley.

This is not the scene I wanted, damn you google images!

Bishop: I'm sorry if I scared you. That platform start-up was just becoming too unstable. I had to circle and hope that things didn't get too rough to take you off.

Ripley: Bishop, you did okay.

Bishop: I did?

Ripley: Oh, yeah.

And like that poof! another start-up I'm done with. There are a lot of reasons behind this decision, but suffice to say when it came down to it, the economics didn't make sense. The salary was too low coupled with a meager share of stock that after a few years working a higher paying job would have been impossible to overcome the shortfall.

And that's the dirty secret: unless you're a founder or a co-founder, your equity sucks*. The long hours, the lack of decent benefits and the low pay all should be overridden by the awesome options you've just been offered. But, and that's a large but, how many of these start-ups are going to be successful? How many of them are going to actually pay you out? The answer is frighteningly small. Sure, the numbers look good on paper, but you're going to go through a few rounds of valuation, your equity will decrease and you'll work super hard for it all...to enrich someone else.

* Obviously there are exceptions to every rule, but the exceptions are far outweighed by the rule...

Yoda knows
401(k)? Life insurance? Pre-tax fucking subway fare? A start-up employee recieves not these things.

Lets put it this way. You're working at a start-up. You've got some decent stock, say, like, 200,000 options. You vest over four years and your strike is $1/share (which is WAY too high for an early employee, but you feel "good" about this one.) You're making 50k under market value.

That means in four years, your stock is going to have to NET (taxes are a healthy 20%, plus you've got fees, so it's gotta be pretty high...) you a $1 a share in order to be worth it, assuming you never recieve a raise or a bonus at your new job, and you don't count your 401(k) match, health benefits, less-stressful working conditions and shorter hours.

Now, sure, your stike price is hopefully lower (pennies!) and your option pool is hopefully larger if you're taking that sort of pay cut, but the reality is it frequently is not.

So, that being said, I've moved back to Times Square to work for Condé Nast Traveler. The people are nice, the caféteria is excellent, we use Github, Trello and Pivotal Tracker and we're building a platform on top of node.js and express from scratch.

The best part though? We're hiring! So if you want to get the chance to write, completely from scratch, a new site on top of a content platform (also being written from the ground-up) in node.js, you should let me know.

Running Node.js as an Upstart service


Other posts in this series:

  1. Switching to Node.js
  2. Running Node.js as an Upstart service

I'm going to start with the final step first, being that it was the most vexing to solve, thanks to the vagaries of upstart configuration files. I know it seems backwards, but really, this was such a pain in the ass, it felt it demanded documentation as quickly as possible.

The node ecosystem, being a relatively young ecosystem, is missing some of the deployment tools that most other development enviroments have had for some time; this is most strikingly obvious when you're looking to deploy your node.js backend out on your live servers. My first question was "what's the equivilent of WSGI (or Rack or Servlets) for node?" (And yes I know that WSGI is blocking and Node is not, but the details of implementation aren't as important to this question as the existance of such a gateway.)

As far as I can find, the answer is none! You start your service using the command line and your script is responsible for creating a listener on a port. So long as you get traffic to the port on which it's listening — your service is going. You don't to worry about setting up an WSGI server, you don't have to handle any complex deployment scenarios — just get your code out on the server and run the command.

Obviously, then we need to consider the following:

  1. How to get node to run like a daemon
  2. How to start/stop the service along with all the other services
  3. How to restart the service automatically when it crashes
  4. How to handle running along side other web services on the same machine

Daemonizing your processes

*Sort of like press your luck...*

There are several ways to handle running a program as a non-interactive service. We could write it as a daemon process using fork/spawn and spawning the necessary processes for our script to daemonize itself or we could use a wrapper program like start-stop-daemon.*

Using child_process.spawn we could write our server specifically to spawn child processes. This, for some reason, feels messy — I'm trying to write a web-application, not a daemon, and would prefer to keep my code as clean as possible towards that singular purpose. I also don't really like writing daemons. I'd rather do:

import time, sys

    while True:
except KeyboardInterrupt:

Then be forced to handle daemonizing a process — I mean seriously, what has that cute little devil done to you! Ergo, I decided on start-stop-daemon which has a unix-appropriate-length man page for it, but more importantly, is designed to handle this exact scenario.

I figured the next two would be easy to tackle: since we're using Ubuntu linux on our AWS machines we will use Upstart, the system process management system that replaces the SYS V style init system with something a little more managed (similar to launchctl on NeXTMacOS).

Upstart has a vexing problem however: it's documentation, while volumnous for an open-source project kind of blows. The cookbook has no real recipies in it, and it's biggest problem: there's no way to validate the syntax of you upstart script. Either it runs or it doesn't. If it doesn't — good luck on figuring it out.

Some furious Googling later left me with the following upstart script:

# updater_node

description     "Updater node.js front end"
author          "Todd Kennedy"

start on runlevel [2345]

stop on runlevel [06]


# What environment are we in
env NODE_ENV=<%= node.chef_environment %>
env NODE_HOME=/var/opt/node

chdir $NODE_HOME/app

# Start the process
exec start-stop-daemon --start --chuid ubuntu --make-pidfile --pidfile $NODE_HOME/run/node-upstart.pid --exec /usr/local/bin/node -- $NODE_HOME/app/server.js >> $NODE_HOME/log/node-upstart.log 2>&1

(As you can see we are using chef to manage our deployments -- this just prints out the name of the appropriate environment: development, testing, production, so that our app can configure itself appropriately).

This gets placed into /etc/init/node.conf and can be run with the standard stop, start, restart, status, etc commands. But how do you know if you've gotten the syntax right? It will tab-complete when you try to interact with it: start no➥ should print out 'node'. If it doesn't you've, got some digging to do!

* I understand there are million and one ways to do this on unix, these are the two options I considered for this purpose however.

On Github and Workflows


It's no secret I am a fan of git, but even moreso of Github.

Recently though I've been using hub as my command-line interface to Github (and therefore, git as well). The ability to generate pull-requests from the command line saves a lot of clicking and interfacing through the website.

However, it's syntax leaves much to be desired. But why are pull-requests so important you ask? Code review, obviously.

Back in the mid-fall we hired a former Hooters-wing-eating-contest-champion/dev-ops/front-end/Vermonter as the result of the outfall of a sadly demised fellow New York start-up. One of the first things he implemented (along with setting us up with a lovely chef automation system) was our new workflow. Until JT came along, the other developer and I were mere git neanderthals — working primarily in the master branch, with only occasionally making new branches for radical changes. His suggestion was to set up the following:

We would hence forth ONLY work on branches — each ticket in our tracking system (JIRA if you must know, and thank you MTV for training me on how to use it over four years) would become a new branch modeled after [ticket type]/[ticket ID]. So, bug number 123 in the WEB project in Jira would become bug/WEB-123. If multiple developers are working together to write a new feature, we first branch that feature off of development as feature/awesome_new_project, and then we all make our branches off of that for each ticket. Each time the ticket is complete, we make a pull-request back to the "parent" branch (either development or the feature/awesome_new_project branch).

branching diagram
The complexity of this graph shows we're very productive

These pull-requests serve two primary purposes:

Whenever a pull-request is made, the creator of the pull request is not allowed to merge that pull-request into the branch until one of three conditions have been met: they either need to receive "cake" (:cake: in the Github comment field), have resolved/answered all of the questions posted to the pull-request, or 24 hours have passed since the pull-request was generated. This allows all of us to get better insight into what other people are working on, and have a good chance to point out things that we would have done differently, or might have solved in alternate methods in other places in the code. Remember: above all else, consistancy.

Alas, making pull-requests, even with hub, is time consuming! There is just so much typing!

But, good sirs, this is also where my love of bash enters the picture. With a quick function defined in my .bash_profile, making pull-requests is even easier.

function pr (){
    git_branch=$(git branch 2>/dev/null | sed -n '/^\*/s/^\* //p')
    git pull-request "$@" -b [github owner]:development -h [github owner]:$git_branch

This will cause a pull-request to be generated for the currently checked-out branch back into the development branch of the repo (obviously replace [github owner] with your github code owner's username), so instead of typing out all that malarky I can just type pr "adding foobaz to the frutz".

Of course with all this branching, you need to make sure you're pulling from the correct place, which leads to this function:

function branch (){
    if [ -n "${args[1]}" ]
        git stash

    git checkout development && git pull && git checkout -b ${args[0]}

    if [ -n "${args[1]}" ]
        git stash pop

(I'm postive there's a more bash-y way to do those tests to see if you should stash, but this works for all intents and purposes).

This allows you to create a new branch off development, and if you've done any work that needs to be on that branch, stash it, update from development, create your branch and then apply your stash on top of the new branch. Nifty.

Switching to Node.js


Other posts in this series:

  1. Switching to Node.js
  2. Running Node.js as an Upstart service

In the past I've alluded to troubles/concerns we've had with using the Lift framework to server as our delivery mechanism for our web-browser client application — nothing from a technical nature (other than it's soft-of Byzantine desire to control the HTML experience from the backend), but it just hasn't lead us to the sorts of rapid development experience we need for what we're doing.

After we re-launched the site as a Backbone application in the Fall, our attentions swung to the backend to see what we could do to better seperate our client application from our business API, but more importantly, prevent as much Lift and Scala from dealing with HTML as possible.

I sat down and came up with a list of things that our new system would require:

  1. Ability to parse our client-side templates so we can assemble the HTML for the page and reduce our post-load DOM manipulation.
  2. Capacity to make REST requests against the API back-end to slipstream data into the HTML response before its sent to the client.
  3. Have an active ecosystem of available packages to reduce the amount of time we spent writing code.
  4. Written in a language known by at least two of our staff.
  5. Not have a horribly complicated setup for serving behind Nginx
  6. Not be overkill. We have no need for an ORM or database connectivity, so that should be, at the most, an optional component, and we already have a templating language (see #1), so that shouldn't be required either.
  7. Allow for the rapid devleopment and deployment of semi-static content (where the content of the page is text, but it's served from with in the app container to maintain header functionality).

With this set of core requirements, we started looking at what systems and languages would meet these needs. The REST API and active ecosystem requirements were pretty much met by any nameable-off-the-top-of-our-heads system: Node (Express, Gedddy), Python (Flask, web.py), Ruby (Sinatra, Rails), Java/JVM (Groovy/Grails, Spring)... really the list never ends so if I missed your favorite combination, my apologies.

Ruby was the first go due to lack of in-house expertise. (Yes. We are a start-up. And no, we have no actual Ruby developers in house. How odd!)

I quickly pulled Java out of it since we already were running a servlet via Tomcat on the machines and didn't want to mess with deploying two of them simultaniously (and Java is, well, just so boring!)

This left Python (a personal favorite of mine) and JavaScript. Being that dust.js is only usable from within JavaScript made it a forced decision to go with Node (using Express since it was so much lighter weight than anything else).

I started writing the core of the server the day after Christmas while hiding from my in-laws (all of whom are lovely, but numerous) and we launched it six weeks later assuming the serving of all of our HTML (and static content). In that time we:

  1. Built a multi-site configuration system that allows the server to load configuration files (and therefore switch the look, feel & operation) of our site depending on the URL. (This is for partner who white label our client)
  2. Built a REST client (on top of restify) to grab data from the API to slipstream into the page before returning it to the user
  3. Designed a very-light weight URL routing system allowing for the deployment of new static pages without altering routing logic
  4. Setup our server infrastructure to have NGINX proxy just API requests to the Tomcat servlet, and all other requests to the node server
  5. Wrote a script to run Node as an Upstart service (our AWS machines run Ubuntu)
  6. Re-wrote all of our code in Coffeescript
  7. Setup grunt.js to manage our build environment
  8. Attained a 75% coverage rate for our new code

It's been a busy past six weeks, but it's been totally worth it to get this launched.

I'm going to start dissecting the choices we made in depth in a series of posts here over the next couple of weeks starting with the end first (as all stories should be) — getting node.js setup to run as a reliable daemon process under unix.

From a desktop GUI to a Javascript-powered webapp


As a homebrewer and a programmer I am consistantly trying to find the right brewing software to use to create my recipies — unfortunately either a lot of them are PC only (which isn't too much of an issue, but more of a pain), or their interfaces are not, well, good.

So what's a programmer to do? Well, try to write his own! It started about three years ago as a Python-based GUI application using wxWidgets to create the interface. This way it would be easily cross-platform, look like the underlying operating system, and not require a ton of oddly created interfaces.

I even got as far as writing a bunch of code to support the math behind brewing (equations that let you predict the amount of sugar you'll extract from grains over a certain temperature and time, how bitter your beer will be, and how much volume you'll evaporate on a given boil), wrote a system for converting between measurement systems (and time conversions), created the data structures for storage, and even my own crazy layout engine for wxWidgets.




But that bit of code golf aside, the biggest part was the generation of all the objects that stored the data about recipies. This was all defined using SQLObject a Python ORM I tend to favor due to it's simple declarative syntax, but admittedly it has a very small following.

To 64 bits and beyond!

Then Apple upgraded the OS to 64 bits and deprecated the Carbon library, on top of which the 32-bit wxWidgets library for OS X was based. The 64-bit version based on Cocoa was no where near ready for primetime (the app would simple just segfault everytime I tried to run it) so a new tactic was needed.

Given the fact that 99% of the programming I've ever done involves web servers and browsers, and the fact that HTMl suffers from much fewer issues when it's renderer is upgraded, it seemed like a web-based solution would be in order, even if it was just run locally on the command line.

With some extensive Django experience this was my first stop. However, I didn't want to re-declare my objects in the Django ORM, nor did I feel like dealing with using a non-Django ORM with their views. Some quick searching resulted in finding Flask, which is excellent in it's minimalism providing a nice interface to WSGI (via Werkzeug), URL routing facilities and a templating library (which you don't even have to use.)

My favorite way to learn a new framework however is to build a blog with it first; joking aside it provides enough level of complexity (search, authentication, listing, admin interfaces, etc) that you can get a handle of the basic concept quickly and then move onto grander things.

Henceforth instead of calling the project BeerMaker it changed to b(eer)log; and would provide people the ability to blog about the recipes they created using the software. The blog part went fast (and is live, but totally unused now), and then came the recipe generation software.

The problem with webpages

After few attempts at creating a web interface to the project I realized that creating a recipe is an interactive type thing — as you enter values into the form, fields need to be constantly recalculated to show what effects adding 1.5lbs of Maris Otter will have on your color, available sugars and potential amount of alcohol by volume.

an early attempt
I am not a designer

So out went all my Python beer math stuff, and in comes JavaScript and my third attempt at trying to finish up this piece of software. The backend API is nearly complete (and still built on top of those original SQLObject models from three years ago (albeit heavily modified)), complete with view annotations (so you can explain what class of user is able to read/write which attribute on the given object) and a system for generating JSON-able Python dict objects directly from the SQLObject represtation, including both the generated and static attributes for the model.

But that's a post for next time.

$.Deferred is for lovers


Two of the biggest challenges I had when I was learning JavaScript (and, from what I can tell, a lot of people have these similar challenges) was dealing with running code asynchronously and the concept of a closure. And to be fair — I'm certainly no expert on these concepts myself. I'd blame it on the fact that my college degree is in Journalism, but I don't remember the computer science department offering classes in JavaScript development. (Of course this was back in the stone-ages of the late 90s. And yes, the dinosaurs were really cool.)

We're finding this in our interviews as well. (Have I mentioned that we're hiring?) The number of candidates we've had that are unable to answer basic questions like "what is a closure" and "why does jQuery return deferred objects for asynchronous methods and why is it a good thing?" That last one I realize might not be a basic question, but if you're giving yourself a 7 out of 10 on JavaScript and have jQuery listed as something you know, you should at least have a concept of how to answer that question. (And this isn't even getting into having people write code in an interview, this is just the light-weight q&a first!)

But still, no amount of fancy book learnin' will really help you when it comes to actually writing code that has to deal with real-world scenarios. Today I'm going to talk about how I learned how to stop worrying and love asynchronous programming. Closures? I'll handle those at a later date.

Asynchronous for the win

Most people's first interaction with asynchronous code is when they're trying to use XMLHttpRequest to retrieve data from a remote server to make their page "dynamic." (Yes, I remember when AJAX really did involve parsing XML documents, and yes, I am very glad that JSON exists.)

If you're anything like me, the first thing you did was something like this: (excepting all IE-specific stuff that made you want to drive over to Redmond and stab Bill Gates in the eye-sockets):

var xhr = new XMLHttpRequest();
xhr.open('GET', 'http://myawesomeapp.com/services/', true);
if(xhr.status === 200) console.log(xhr.responseText);

And, again, if you're anything like me, you sat around that coffee house in Seattle scratching your head thinking that maybe the WiFi was overloaded, but no you can get to Google still, or maybe that document doesn't exist on the server, but no you just copy and pasted the URL into your browser and that's working just fine, so like, what gives!

So you do a little Googling (and holy christ what I would've given to have had MDN back then!) and you come across the fact that the third argument to XMLHttpRequest.open sets whether or not the call should be made asynchronously. Praise hosanna! I'll just set that bastard to false and I'll be on my merry way.

Only wait. What is this spinning cursor? Why the heck is nothing working! DAMN YOUSE! DAMN YOUSE asynchronous codes!

So you Google some more and you find this "jQuery" thing (although it didn't exist at the time I'm going to retcon it into this story, and certainly jQuery 1.5 didn't exist, but whatever) and see the getJSON method. So you try:

var awesome_json_doc = $.getJSON('http://awesomewebservices.net/isntalinktwitbook', function(data){ console.log(data); });
> Object {done: function, fail: function, progress: function, state: function, isResolved: function…}


What the heck is process and state?

Congratulations: you have just discovered the jQuery $.Deferred object, which after jQuery 1.5, is what jQuery returns whenever you call a function that runs asynchronously.

Make good on your promises

$.Deferred is a massively powerful tool based on the Promises/A spec (and if you're not using jQuery, there are some alternate implementations: node-promise, when.js). Where jQuery's implementation really shines is that all asynchronous methods return a deferred object. This makes it very easy to create scenarios where certain functions will fire only when various asynchronous operations have completed. And, even if the state of the deferred object has been "decided" (resolved or rejected), adding additional callbacks to object will cause them to execute for that given state.

This might sound a little complex, so lets dive into some real-world usage examples.

On Updater I've mentioned that we use DustJS for our template rendering. Dust is a great templating language (I happen to like it), but unlike mustache or hogan.js, it's render method runs asynchronously.

var compiled_template = dust.compile("I run asynchronously!", "async_example");
dust.loadSource(compiled_template); //register our template with dust
var template = dust.render("async_example", {}, function(err, out){ console.log(out); });
typeof template
"I run asynchronously!"

As you can see the result of the call to dust.render is undefined (in fact you wouldn't actually do that var template = thing in the real world), but later on, when the template is fully compiled and rendered, dust invokes the callback provided as the last parameter which prints the template out to the console.

Given that we have no control over when the template is rendered (although if you're interested, the JavaScript execution context is pretty fascinating), we have no control when our callback will occur. So, if we need something in the DOM to exist, we're in a bit of a pickle. For example, lets say we wanted to make an input field into a date input using some jQuery date plug-in:

var compiled_template = dust.compile('<input type="text" value="" id="test_field">','my_form');
dust.loadSource(compiled_template); //register our template with dust
dust.render("my_form", {}, function(err, out){ $('form').append(out); });

Since the DOM might or might not have the rendered template available in it when we try to access it using a jQuery selector, we can't guarantee that this code will work; sometimes you'll get a date input and sometimes you won't. (And don't tell me to use type='date' until the number of visitors using IE8 & IE9 drops below 1%.) So how do we do this?

Promises to the rescue!

var template_deferred = $.Deferred();
template_deferred.done(function(){ $('#test_field').dateInput(); });
var compiled_template = dust.compile('<input type="text" value="" id="test_field">','my_form');
dust.loadSource(compiled_template); //register our template with dust
dust.render("my_form", {}, function(err, out){ $('form').append(out); template_deferred.resolve(); });

We've done three new things here:

  1. Created a new $.Deferred object and assigned it to template_deferred
  2. Attached a callback to the done method of the newly created deferred object
  3. Resolved the deferred object when we rendered the template, errors be damned!

When we did template_deferred.resolve() we told jQuery that anything attached to the done and always methods of the deferred object were ready to be called, so it calls all the available callbacks in the order to which they were attached. But we still have no error handling here. We could easily add an if statement in the dust.render callback and call template_deferred.reject. This would fire all the callbacks attached to both the fail and always method.

(There's one more event method available on the $.Deferred object: progress. progress callbacks are fired when the deferred object has it's notify method called. There's also a version of notify, resolve and reject called notifyWith, resolveWith and rejectWith that allow you to pass a context for the callback, but I'll cover those at a later date.)

$.when and .then

So now we've got all these awesome deferred objects. But what if we need to wait until more than one promise? jQuery has us covered on that matter too: $.when and .then. This allows us to treat deferred objects like booleans:

$.when(dfd_one, dfd_two).then(function(){
    console.log('both deferreds were resolved!');

Or a personal favorite of mine (which we use when we're rendering a collection of Backbone models since we don't want to attach them to the DOM until they're all rendered for speed purposes):

$.when.apply($, _.pluck(array_of_views, 'dfd')).then(function(){
    $('body').append(_.pluck(array_of_views, 'el'));

And yes, there's some advanced usage on there — _.pluck and apply. They're like chocolate and peanut butter here. We store the deferred object for the template rendering as dfd on each view, and Backbone's convention is to put the rendered element in el. Add apply and you can pass in an array of arguments to a method rather then enumerating across all of them.

A region manager for single-paged apps


In building out the new version of our site, it became apparently early on that I needed some good way to manage adding views to and removing views from the DOM. Backbone doesn't provide a lot of functionality in the perspective out of the box; Backbone.View provides both a remove and an undelegateEvents method, but neither of those gets automatically called when the view object is destroyed (although that has to do more with javascript's object model and the fact that delete doesn't work like you think it should).

The remove method just calls the remove method of the DOM manipulation library you're using (jQuery in our case), and undelegateEvents just deletes the events delegated to the object via the events hash provided in the view declaration (and doesn't necessarily need tobe called but in special circumstances).

The problem here though is that if you're adding and removing views from the DOM and you're not cleaning up after yourself correctly you're going to end up with dreaded zombie attacks — where due to the nature of how javascript closures work (if there's a reference left open to them they will remain in memory; a reference can be as simple as a bound DOM event), views you think have been destroyed are still in memory (causing memory leaks) AND responding to events as if they were still in the page.

Luckily there are a couple of patterns and libraries that help you with regards to these issues. For a short time while building out the prototype for the site I was using Backbone.Marionette's regions system for handling this, but after switching to Dust for templating and having to start using the asynchronous version of it, I realized it was overkill. The only part of Backbone.Marionette I was using was the region manager, and having to include all that code in my page was killing me. (Plus I kept having a bug where the view wasn't defined when I was trying to close it and I didn't feel like spending didn't have the time to troubleshoot it.)

So I whipped up this example inspired by the region manager provided by Backbone.Marionette, but designed from the ground up to be asynchronous — this means not only do you need jQuery, but you also need your render method from your view to return an instance of a $.Deferred object that is resolved when the view is attached to the DOM element. (If you're wondering what $.Deferred is, or how promises work, or really, just about asynchronous programming in Javascript in general, I HIGHLY recommend Async Javascript by Trevor Burnham.)


Usage example

var MyView = Backbone.View.extend({
    render: function(){
        var dfd = $.Deferred();
        this.$el.html('<h1>I am a title!</h1>')
        return dfd;

var MyRegion = new Region('div#region');

MyRegion.show(new MyView);

The key point being that the render function resolves the deferred object it creates when the HTML generated by the view is attached to the DOM. And because it too returns a deferred object, you can use that to manipulate the view after the view has been attached.

You'll notice that we call MyView.close() in the clear method on the region. This is because we not only bind DOM events to our view methods, but our application uses an event pipeline or dispatch system to allow independent blocks of code (closures) communicate with each other. Since we're doing some extra binding in the initialize method, we have to take great care to unbind those listeners on the dispatch queue, or we'll still end up with zombie event listeners performing actions from beyond the grave. But I'll leve that pattern alone for a follow up post.

Job Interviews are Hard, Lets Keep Doing Them


Let me describe a scenario for you: you're sitting in a foreign room with some bad fluorescent lighting overhead and a hastily erased white-board in a suit that you probably haven't worn since your college graduation/best-friends wedding/grandfather's funeral. A random person you've never met rushes into the room and introduces himself, minorly out of breath and says sorry he's running late. He pulls a piece of paper out of a folder, gives it a once over and says, "Well, why don't you tell me a little about yourself to start."

Interviewing sucks What level of hell are you? You're in the job interview?

Let's face it: interviewing for a new job sucks. I know very few people who enjoy the entire process, and even fewer people who enjoy giving interviews. The problem is that they're inevitable awkward: someone has something you want and you've got to show them how awesome you are so that they'll give it to you. Even worse I find are interviews at tech companies, not only are you sitting there jumping through random hoops explaining what it means that functions are first-order, but inevitably someone is going to make you write code. Or worse, give you code and make you find the error in it.

This isn't even just for programming jobs though; the entire tech industry is frought with seemingly (for the interviewee) left-field questions that aren't about the domain of experience you're trying to sell them, and for the interviewer, awash with people who say they've led teams to build applications, where by "team" they mean their cat sat on their lap while they wrote the app at home on the weekends.

As a company though, how you sift out the talkers from the doers? You ask these left-field questions and watch how the candidates react to them. But, frequently, I feel that we go to far.

In November of 2010, I was anxiously looking to move to Seattle. My wife and I had just had our first kid, she's from Bellingham, and I love the area, and with grandma a two hour drive north there's not a lot that two newly minted parents aren't going to like about that. Going for a tech interview at a large company (you know, the kinds that are willing to pay relocation from Brooklyn) in the Seattle area usually means you're going to one of two places: Amazon or Microsoft.

So I'm interviewing at Amazon. The job title is Product Developer, Mobile. (I was a product developer at MTV Networks at the time, in the mobile group, so this isn't a stretch.) First part of the interview was with H.R. setting expectations: I'm going to see four different people from various parts of the group who will quiz me on my knowledge around products, mobile, software programming concepts and project management experience.

First interviewer comes into the room and after the small talk starts the interview goes like this:

"So, how does your computer make the Amazon homepage appear when you type amazon.com into the location bar of your browser"

"Well, you type in amazon.com and it asks the DNS server where amazon.com is located and the DNS server says 'amazon.com is, so when you ask for that I'm going to take you there'."

"What is DNS?"

"It's the domain name resolution service."

"What does that mean?"

"Computers need some sort of addressing mechanism that allows them to locate themselves over a network, since computers deal with numbers better than words, these addresses are typically conveyed in numbers. Humans are bad with long series of numbers, so we came up with a system to look up the numbers based on some words."

"Yeah, but how does it work?"

"What do you mean?"

"What does 'your computer asks the DNS server what amazon.com is' mean? How does the DNS server know that?"

"Well, when you register a domain name, you create a bunch of records for it in a DNS server; most typically you create a least one A record which says this IP address answers for this domain name, and you might create some MX records which explain where the mail goes, and some CNAME records to say 'Oh you want that? Look at this instead.'

So you create these records. And in that record is also something saying 'this DNS server is responsible for the authoritative answer for this domain name'. And DNS servers are created in a hiearchy -- as in there are thirteen servers out there called the root DNS servers. Each of these has a bunch of children servers they answer to, and those children have children, etc -- like a tree. So my computer asks the closest DNS server it knows. If it's never head of amazon.com, it asks it's parent. And if the parent hasn't heard of it, it asks its parent, and so forth until the root server gets asked. The root will know the answer (or will know where to get it) so it'll eventually filter back down that chain of command."

"Excellent. OK, so know it knows where amazon.com is. What happens next? What do you mean 'it goes there'? What goes there? How does it know what to get?"

This line of questioning went on for two hours.

A two hour explaination of how the DNS system works, how TCP/IP routing works, how the HTTP protocol works, what an HTTP verb is, what HTTP verbs there are, how a load-balancer works, what round-robin DNS is, and so forth.

Mind you this line of questioning was for a job where I would responsible for the consumer experience of Amazon's (then two, maybe three) mobile phone applications would be. A job that distinctly doesn't touch this level of networking knowledge.

Now for some reason I didn't get this job. (And by that I mean the next two hours were spent explaining how you'd program an elevator control system to fairly distribute services across multiple floors, but allow someone paying more rent to have priority over the elevator, but without interrupting a current delivery.) I thought the line of questioning was a little absurd and it probably started to show a little.

In retrospect, however, this line of questioning makes a little more sense to me. Interviewing someone for a product development position, when their experience is with in-house tools that can't be shown (I was responsible for the systems that allows MTV Networks to deliver video to it's online partners; the Daily Show going to Hulu and Comcast for On-Demand within two hours of being on air is an excellent example), you have to get creative with your line of questioning; it's really about the thought processes I was demonstrating when given a left-field question and the reaction and composition under stress that they were watching.

Serving a Static Site via S3


The inspiration to finally launch this site came about in a funny way. While reading Hacker News I came across a post about a new static website generator written in Python. Having played around with Jekyll a few years ago for an unrelated project I was intrigued; one of the "downsides" (and I use that term lightly) of Jekyll is that it's written in Ruby, which, honestly, I'm not that interested in exploring much deeper. (When I decided to learn Python I initially chose it over Ruby because the syntax made more sense to me.)

Cactus, on the other hand, is written in Python, so I can happily roll up my sleeves and muck around with (and most importantly, contribute back to) it, and it's templating system is based on Django's templating language, which was my application framework of choice until I learned about Flask last year. (Which, really, warrents it's own discussion around all-inclusive vs minimum viable frameworks.)

The other thing that caught me was the fact that it's designed to use S3 as it's serving/storage mechanism out of the box (thanks to the wonderful boto project). I have a VPS host already which runs a friend's WordPress blog, but I'm really not interested in beefing up that hardware to run TWO wordpress blogs (and dealing with the subsequent traffic), nor did I feel like SCP'ing a bunch of files out to a specific bath and updating indexes, and so on and so on. Already paying for a service though, I was wary of paying MORE to host something entirely not myself.

However taking a look at some data 48 hours after I launched this new blog leads to an interesting conclusion: running a purely static site on Amazon S3 is really, really cheap.

Like, pennies cheap.

It will cost around $0.49 to serve all 88K of this site to 5k visitors a month.

That's cheaper than the amount I spend on on shaving supplies for my neckbeard.

How to Keep your Marketing Team Happy


In one of my previous posts I mentioned that we are using Dust to render our HTML templates for the site. That's only a half-truth. One of the biggest issues that people encounter with JavaScript webapps is that it's very easy to write an HTML page that comes back from the server that looks like this:

<!DOCTYPE html>
    <meta charset="utf-8">
    <title>I'm a webapp!</title>
        <script src="/js/my_awesome_app.js"></script>

(And yes Virgina, that is a valid HTML5 document. Don't believe me?)

The problem with this, as your marketing/seo/social networking teams (you have those, right?) will tell you is that there's nothing there. How is Google going to index that if there's no content? How can people share that page on Facebook without the <og:title> or <og:description> tags?

The first thing that pops into mind is "oh, I'll just do this in Javascript".

    // get info about this page somehow
    // this is, by the way, a dumb way to do this
             .html('This page kicks some major butt'));

That's great! You've loaded your page, whatever method you needed to get the data about the page ran (ideally it was bootstrapped into the page on load), and then you added it to the generic document that came down from the server.

Problem solved? Right?

Only not.

The problem here (and by problem I mean it's a good thing) is that the Facebook open-graph page scraper does not render JavaScript. All it sees here is your HTML skeleton as above.

Anything you want a web robot, crawler or scraper to be able to index you'll need to give to that user-agent when the document is returned from the server — you cannot rely on any client-side rendering technique to suppliment that data.

Right now, as a stop-gap solution, we're still using Lift to render all the pages that you see when you load our site. This was useful since we didn't need to worry about re-architecting our URL-routing solution, and we could rely on the backend to bootstrap data into the page when it loads and you're logged in.

Extra credit problems

So we've gone through the trouble of bootstrapping all of our user data into the page so we can stop incurring that silly extra HTTP overhead of re-requesting data that we already know we're going to need, but we're still relying on JavaScript to render the entire UI, which we can't do until at least the DOM ready event has been fired.

If we're going to be worried about the performance of our pages building and rendering (especially on older machines and browsers cough I.E. 8 I'm looking at you cough), you might as well be, you know, Worried with a capital-W. What does this mean?

This means that you should be rendering ALL of your HTML before you send it down the pipe. This way a user can interact with your page almost immediately — before the DOM ready event even fires the user can start entering data into forms.

This means we shouldn't just be sending this:

<div id="header"></div>
<div class="background vert_spacer">
    <div id="main">
        <div id="nav"></div>
        <div id="workflow">
            <lift:bind name="page_title"><h1 id="page_title"></h1></lift:bind>
            <div id="flash_message"></div>
            <div id="verify_message"></div>
            <div id="user_data"></div>
            <div id="section"></div>
            <div id="verification_section"></div>
            <div id="authentication"></div>
    <br class="clear" />
<div id="footer"></div>

With Great Power Comes Hard Work

So if this so great, why don't we do it yet? Because, well, frankly, it's hard. This means that you need to write your application with the mindset that it might not have to render the pages it's responsible for. You have to make it so your client-side templates can also be rendered by your server's page language (which, Dust.JS cannot be, at least by Scala). You also need to take into account that the user might enter data and try to interact with the page before the appropriate listeners/handlers have been attached.

We're currently prototyping out some solutions to use to solve this problem, mostly around Node since that would enable us to keep our current templating language, pre-render the content and still bootstrap the user data before we send it down.

Fixing Adwords Tracking


Preface: I'm not an SEO/SEM/conversions/adwords/etc expert.

As a small start-up (I think I'm supposed to use some lofty adjective like agile or product-focused or something of the ilk) that isn't just doing it for pure technology sake, we have several key metrics that we have to track, all around converting people who come to our site into users. (This should sound familar for anyone with a site that allows sign up.)

One of the tools we use to drive traffic to our site is Google AdWords. I'd like to tell you more about how and why we use this tool, but I'll be honest — after several years trying to squeeze advertising money out of a stone (and yes, that is me in the cheese hat) at an early internet company and writing more than one script to try to defeat early pop-up blockers, I have sworn off internet advertising). Given this history I'm probably the least-well-versed member of the office in how AdWords works.

So when we re-launched our new site on top of Backbone we noticed one thing right off the bat -- we were no longer able to track whether or not people had arrived at our site via AdWords or via other methods.

After some quick inquiries, it looks like Google Analytics is able to track users from AdWords based on URL parameters appended to the URL when the click on the ad, most importantly the gclid parameter. Normally this isn't an issue, but since we're building a single-paged app, this presents a bit of difficulty; when a user arrives at the application, we are using the Backbone router/history to control the URLs so we don't break the back button. Whenever you call .navigate though this strips out the URL parameters (and any hash fragements) and rewrites the URL to what you've specified.

After confirming with the Google AdWords/Analytics folks that there's no way to set the gclid manually in the tracking calls, the only option we had was to not call .navigate on the first load of the application.

var first_run = true;
} else {
    first_run = false;

This allows the gclid parameter to stay in the URL long enough to record the conversion event.

From Lift to Backbone


I want everyone reading this to raise their hand if they're a Scala developer. If you're also a Lift developer, raise your other hand. If you've got both hands up, I am impressed — it's a rare combination.

When I started at Updater (only a few short months ago!) the site was built on top of the Lift framework for Scala. Lift is an interesting tool; it's primary motivation seems to be to get your backend developers to be able to operate without frontend developers (to a point). It does a lot of obfuscation of the DOM upon building the page (ostensibly for security purposes, but I'd argue obfuscation is not security), which makes it tremendously difficult to manipulate the page after it's been rendered in pure javascript — the point being to write all your code in Scala/Lift and let the framework worry about the front end interaction.

Due to the aforementioned lack of easily available resources who are versed in Scala and Lift (and want to work for jellybeans, stock options and Budweiser at a start-up), it was decided that we would migrate the front-end off Lift and implement it as a single-paged app on top of a RESTful API. This would enable us to hire more tech resources without breaking our backs trying to find people with Lift experience, or spending time teaching them the framework.

This is where I come into the story.

As anyone interested in building software for the web at this point has heard, javascript web applications (or single-paged apps) are the new hotness. Because of this sudden celebrity there are a plethora of frameworks to use as the base for your application: Angular, Ember, Backbone and Spine are just a few out of many.

For our new site launch I decided to go with Backbone for a number of reasons.

  1. Maturity. Backbone is already used on a whole host of sites people use throughout the day including USA Today, Hulu and Trello
  2. Small footprint. The code base is small enough (1400 lines of code including comments) that you can easy read it and understand eveything it does.
  3. Minimal dependencies. It requires a DOM manipulation library (we are already using jQuery) and Underscore. Writing javascript without some sort of shim for older browsers to handle things like .forEach and .filter is hard enough, so Underscore would already be in our stack even if we weren't using Backbone.
  4. Lack of rigidity. Rewriting an entire site from scratch means you're going to be re-inventing the wheel several times over, and trying to replicate, at times, existing use patterns. Backbone prescribes no usage pattern for how you assemble it bits and pieces together. It's not an MVC framework. It's not an MVP framework. In fact, I'd hesitate to call it a framework; it's really just a code library that forces you into no behavior. When you're rapidly changing core assumptions about how your application works, this is a really good thing.
  5. Familiarity. I've been using Backbone for over a year now, so this was a definite ally in the "plus" column.

The downside to using a library vs using a framework is that you will end up writing a lot more boilerplate code than you would had you decided to go with another codebase. The default render method for Backbone does nothing, leaving the templating, DOM manipulation, etc up to you to define. So instead of inheriting some sort of default rendering method you have to deal with, you can write your own. This allows us more flexibility in templating engines — if the render method cannot be run asynchronously, you'll need to jump through a lot more hoops to use something like Dust. This can be easily recitified however by defining your own base view class and extending from that.


(I'll go more into depth about this chunk of code later.)

A simple library can be helpful when debugging as well; the more code you've written the more code you'll innately understand and the easier it will be figure out whats going on when the system breaks. (See: The Law of Leaky Abstractions). If you've ever gotten some odd unreported

The final layer of the stack is our templating library, Dust. Underscore has a built in micro-templating library (which I've used and loved), but it doesn't handle pre-compiled templates. We considered Mustache, but I actually believe templates should be able to handle a little bit of logic. Finally, at the suggestion of a friend at a former job, we settled on using Dust.js. The templates can be pre-compliled on the development end so the user just recieves highly optimized functions that return strings, and the rendering occurs asynchronously so you can still get other things done while waiting for your template. (This, of course, is a double-edged sword!)

Over the course of the next few posts I plan on taking you through some of the patterns we've developed on top of Backbone and how we handle interaction between the backend and frontend.