NOTE: There is now a relevant answer on StackOverflow. Unfortunately, this answer wasn’t there when I needed it.

The problem

Recently, while adding iCloud support to the Day One iOS app, I ran into an issue copying files to and from the application’s Documents directory and its corresponding iCloud directory. Day One can run in one of three modes:

1. Sync entries via iCloud
2. Sync entries via Dropbox
3. Don’t sync entries

If you’ve developed for iCloud at all, you know that iCloud documents live within specific, “ubiquitous” directories on the user’s device (discovered via NSFileManager’s URLForUbiquityContainerIdentifier: method). So in order to enable iCloud sync, we needed to copy all files into the appropriate ubiquitous container. Conversely, if the user elects to disable iCloud then we need to copy all files from the ubiquitous container into the application’s Documents directory.

The assumption

We wrote a simple method to copy files from one directory to another, either going into or coming out of iCloud. It uses a basic NSDirectoryEnumerator to find all the files in the current source directory, check if they are present in the destination directory, and copy them there if they aren’t. Here’s a simplified example:

NSURL *sourceDir = [self currentWorkingDir]; // current directory, ie ubiquitous iCloud directory
NSURL *destDir = [self destDirForMovingFromDir:sourceDir]; // where should we copy to?
NSFileManager *fm = [NSFileManager defaultManager];
NSDirectoryEnumerator *dirEnum = [fm enumeratorAtPath:[sourceDir path]];
 
NSError *mergeError;
while ((sourceFile = [dirEnum nextObject])) {
    NSURL *sourceFileURL = [sourceDir URLByAppendingPathComponent:sourceFile];
    NSURL *destFileURL = [destDir URLByAppendingPathComponent:[sourceFile lastPathComponent]];
    if (![fm fileExistsAtPath:[destFileURL path]]) {
        if (![fm copyItemAtURL:sourceFileURL toURL:destFileURL error:&mergeError]) {
            NSLog(@"ERROR (copy error): %@ -> %@ - %@", sourceFileURL, destFileURL, mergeError);
        }
    }
}

I frequently found “bad file descriptor” errors being logged out. We were assuming that the NSDirectoryEnumerator would only list files that were actually present on the device. As it turns out, that is not the case when enumerating a ubiquitous directory.

The solution

Once you realize that the NSDirectoryEnumerator will list all ubiquitous files that exist on any device, regardless of whether or not they have been downloaded to the current device, the solution becomes pretty obvious:

You can’t copy a file that hasn’t been downloaded.

Adding that check is fairly straight forward. Since our method doesn’t know if we are going into or out of iCloud, we implemented a pretty general solution. Modifying the above example, it looks something like this:

...
while ((sourceFile = [dirEnum nextObject])) {
    ...
    // By default, assume all files are able to be copied.
    NSNumber *ableToBeCopied = [NSNumber numberWithBool:YES];
 
    // However, ubiquitous items that are not downloaded CANNOT be copied
    if ([fm isUbiquitousItemAtURL:sourceFileURL]) {
        [sourceFileURL getResourceValue:&ableToBeCopied forKey:NSURLUbiquitousItemIsDownloadedKey error:nil];
    }
 
    // Now copy the file if it doesn't exist and is able to be copied
    if (![fm fileExistsAtPath:[destFileURL path]] && [ableToBeCopied boolValue]) {
        if (![fm copyItemAtURL:sourceFileURL toURL:destFileURL error:&mergeError]) {
            NSLog(@"ERROR (copy error): %@ -> %@ - %@", sourceFileURL, destFileURL, mergeError);
        }
    }
}

Conclusion

See what I mean? Pretty obvious, and I can’t believe I lost so much time tracking this down. If the source file is ubiquitous, it might not be downloaded, and thus you might not be able to copy it. So check it first.

Like many, the suggestion that JavaScript could be used in a fully architected system initially left me doubting the sanity of the person with whom I spoke. Over the years I have done several different types of Web Development; I am best known for my work with browser plugins in general and the FireBreath framework in particular. What most do not know is that I actually started out doing a lot of PHP and JavaScript development and that it was actually my work with JavaScript that eventually led me to learning about NPAPI and ActiveX plugin technologies.

Throughout all this I experimented with many different types of javascript code; I wrote AJAX applications long before I’d heard the term coined using a system called JSRS — which is surprisingly still around, or at least the page I got it from. I worked for Move Networks on their JavaScript SDK and I wrote several AJAX heavy websites, such as HamStudy.org. Many reading this will scoff when I admit that while I considered myself fairly well versed in JavaScript I still wrote it as if it were a simple scripting language. Others may wonder what is wrong with that? JavaScript more than perhaps any other language I have used gives you just enough rope to shoot yourself in the foot with. I have learned over the last few months just how little I knew — and didn’t know that I didn’t know.

My upward climb started off simply enough — I wanted to learn about some of the new technologies that I had heard about; Backbone, MVC frameworks, client-side template libraries… I had written my own “frameworks” on which to base my websites, all of which were side projects, but I found myself spending much more time than I liked updating and maintaining those frameworks. I thought perhaps I could use someone else’s system and save myself some trouble.

I purchased a book called JavaScript Web Applications which claimed to teach how to write an actual application with JavaScript. The table of contents looked interesting, and besides teaching about the principles used it also had instructions on several different MVC frameworks that I wanted to know about, including Backbone.

This book was the beginning — as a result of this purchase I soon came to grips with just how comparitively incomptetent my old code had been, and started to actually understand that JavaScript can indeed be a real language. Over the course of a few posts I would like to share with you the stack that has grown out of this system: What the components are, how they help, and why they are integral to the overall solution. I will start out in this post with the basics and my next post will detail the specific tweaks that we have made to them that have made our system so successful.

Dependency Management

Having been so long-winded with my earlier explanation, I will now spare you the details of all of the things I tried, researched, or experimented with during my search. One thing quickly became clear to me, though: Perhaps the #1 reason that my previous code had ended up so ugly was the need to avoid creating lots of JavaScript files. These can be difficult to deploy cleanly as it is not a good idea to have too many script includes in your production site and combining them can be an annoying process. The end result of this problem is that I had about 10 JavaScript files — and felt guilty about it — and all of my functionality was stuffed in those files. I had page-specific files, site-specific files, and general reuse library files, and all of them tried to register themselves cleanly in the global scope without clobbering each other. In short, it was a mess.

Enter require.js. Require.js is a system in which all javascript files are wrapped in a function. This is actually a really common pattern, but with require.js that function is passed into the “define” function call along with a list of dependencies. Require will first resolve these dependencies and then when they are all resolved it will call the function you provided, passing in the exports of each dependency as arguments in the order that the dependencies are listed in the define call.

The question left unanswered, then, is what are the exports of the other dependencies? Simply put, each “module” exports some value by simply returning that value from the function passed into the define call (See file1.js and file2.js).

define(["backbone", "./reqfile2"], function(Backbone, File2) {
    // In this file we do whatever it is we do; we know file2 was loaded first and we can use what it exports


    return File2.text + " from two different files";
});

define([], function() {
    var File2 = {};
    File2.text = "This is some text";
    return File2;
});

The natural result of so much flexibility is a lot of small files; this is really good for code organization because it makes it easy to seperate things into logical pieces and even to include whole modules consisting of multiple files (because one can depend on multiple others and combine them in its return value). The downside is that with lots of small files your page load time starts to get a bit out of control. Fortunately, require.js has an answer for this problem as well!

Require.js has an optimize phase that will examine the dependencies of your main file and combine all of them into a single file so that they are all loaded at once. The require.js system still makes sure that the functions are called in the correct order, but makes it very easy for your production site to be seamlessly updated to use a combined and minified single javascript file instead of the potentially dozens of small other files.

To me, this was a nearly life-changing WIN.

JavaScript Templates

If dependency management became the #1 most useful change in my architecture, a close #2 would have the be JavaScript Templates. The more I did things on the client the more client-side HTML I created in JavaScript; I experimented with several methods but still ended up with really hard to read html mixed in with my JavaScript as strings. When the time came to update one of those sites it was painful; not all of the code was in the same place, it wasn’t easy to read, it wasn’t easy therefore to change.

I considered different types of templates but had previously rejected the idea. The problem with templates, you see, is that you still have to include the template code somehow. The most “traditional” method would be to pretend you’re on a backend and just store them one template per file. Of course, then if you have any real number of these files you find yourself making a multitude of AJAX requests to download all of these templates; it’s clean from a file structure perspective and fairly easy to use, but to me the added complexity of all those requests is not acceptible. Other options include storing the template in javascript strings (same problem as I had before without templates) and storing them in a special script tag on the html page, which had the (to me) serious disadvantage of being nearly impossible to reasonably use on multiple pages. It was a problem.

Enter require.js and the text plugin. The text plugin of require.js is an idea both simple and brilliant; it allows you to specify files that should be loaded and the text of those files will be passed into your page function as the value of that dependency. With the text plugin you can save your templates each in their own file and have require load the text when the page loads. Of course, this doesn’t solve the problem we mentioned earlier! You still have potentially lots of AJAX requests being made to load those files. Problem? Not at all! Remember the optimize step in require.js; just like the normal dependencies are loaded and combined into one file the text plugin provides a mechanism to serialize those templates into javascript as strings and combine them into that same file. You get the best of both worlds! When you’re developing you have the flexibility of having each template in an easy-to-read file, but when you deploy those templates are packaged with your javascript.

define(["backbone", "underscore", "text!./mytemplate.jst"], function(Backbone, _, tplCode) {
    
    var template = _.template(tplCode);


    function doSomething(val) {
        alert("Here is the rendered template: ", tplCode({key: val}));
    }
    return doSomething;
});

Top all of this off with the fact that for several of the most popular template libraries there are already require.js plugins, such as requirejs-tpl. These plugins work like the text plugin, but they compile the template string into an actual template function before passing it back to resolve the dependency. Then, when you optimize, they actually compile your template code into javascript functions and minify them along with the rest of your JS code!

To me, this was a nearly life-changing WIN.

Deferred Objects

This may seem like it doesn’t fit with the other items on this page, but if templates and dependency management have revolutionized the organization of my code then deferred objects have revolutionized the way that I write JavaScript code. One of my coworkers at GradeCam, Robin, has already begun a small introductory series on how deferred objects work, so I’m not going to go into those details, but I do want to explain the conceptual shift in how pieces of the system interact that this led me to. It is both subtle and extremely powerful.

Most JavaScript operations anymore are asynchronous; in other words, they return immediately even though the operation in question hasn’t been completed. This can make it difficult to handle result cases; the traditional method is to pass a callback into the function to be called when the operation completes. You then perform some other operation once that function is called back and so forth.

The challenge with that is that it is a significant deviation from the patterns that we are all accustomed to. Granted, it’s important to understand what is actually happening, but often you may want to call a function in a place where you don’t actually want to handle the results; a good example is when you are requesting data from a model in your controller code, but you will actually be rendering it withing your view and your controller doesn’t really care what the results are. The view, of course, should not need to know or care where the data came from. In fact, it shouldn’t really need to know if the data is immediately available or will be available later on — it should display it when it is available and not worry about the other details.

Code speaks louder than prose, so here is an example of the difference between these two approaches; in both async1.js and async2.js three async operations take place. In async1.js they are handled in a traditional way, and in async2.js they are handled using jQuery.Deferred.

function asyncOper1(value, callback) {
    setTimeout(function() {
        callback("One second has passed since " + value);
    }, 1000);
}


function asyncOper2(value, callback) {
    setTimeout(function() {
        callback("Two seconds were added, then " + value);
    }, 2000);
}


function asyncOper3(value, callback) {
    setTimeout(function() {
        callback("A long time (3 seconds) went by before " + value);
    }, 3000);
}


function alertResult(value) {
    alert(value);
}


// Call them all; alert the final result
asyncOper1("we started", function(strVal1) {
    asyncOper2(strVal1, function(strVal2) {
        asyncOper3(strVal2, function(strVal3) {
            alertResult(strVal3);
        });
    });
});

function asyncOper1(valueDfd) {
    return $.when(valueDfd).pipe(function(value) {
        var dfd = new $.Deferred()
        setTimeout(function() {
            dfd.resolve("One second has passed since " + value);
        }, 1000);
        return dfd.promise();
    });
}


function asyncOper2(valueDfd) {
    return $.when(valueDfd).pipe(function(value) {
        var dfd = new $.Deferred()
        setTimeout(function() {
            dfd.resolve("Two seconds were added, then " + value);
        }, 2000);
        return dfd.promise();
    });
}


function asyncOper3(valueDfd) {
    return $.when(valueDfd).pipe(function(value) {
        var dfd = new $.Deferred()
        setTimeout(function() {
            dfd.resolve("A long time (3 seconds) went by before " + value);
        }, 3000);
        return dfd.promise();
    });
}


function alertResult(valueDfd) {
    $.when(valueDfd).done(function(value) {
        alert(value);
    });
}


var res1, res2, res3;


res1 = asyncOper1("we started");
res2 = asyncOper2(res1);
res3 = asyncOper3(res2);


alertResult(res3);

As you can see, the Deferred object method is slightly more complex at first; however, the complexity is where it belongs. Each method need know nothing about the others — they all assume that the input value *might* be Deferred. They return a Deferred object, and so they use the Deferred.pipe method, which makes it easy to chain deferred objects. Because each of the methods returns a Deferred object and is aware that it may receive a Deferred object, you can call one after the other as though they were happening synchronously — because indeed they will end up doing so, in all practical purposes. However, you shouldn’t think of this as hiding the fact that it is asynchronous, but rather as returning asynchronous return values.

Because we used $.when, you can pass a regular parameter in to any of those methods as well and chain them as much as you want. It will handle them all perfectly.

To me, this was a completely life-changing WIN.

This is just the beginning

I have a lot more to talk about here; this is just the foundation. If you know nothing more about our stack, you should understand these things. Next time, though, I’ll talk about how we tie this together using Backbone.js Routers, Views, and Models, how those tie into our templates, and how Deferred objects allowed us to add data cacheing to our data models completely transparently — without changing a line of code outside of the model classes.

As you recall from the first part jQuery Deferred Objects – Part I, deferred objects give you a way to organize asynchronous blocks of code in to a self-managed callback queue, while allowing you to attach 1..* callbacks. Also, you’ll recall that the basics of using jQuery deferred objects include essentially the following “simplified” steps:
1. Created a deferred object
2. Attached .done and .fail callbacks
3. Called resolve or reject, thereby triggering the appropriate respective callbacks

Promises

This was fine as a first step in understanding what’s going on, but in all likelyhood you’ll use a slightly different approach. What’s wrong with returning a deferred object itself? It gives the caller access to resolve or reject directly, and that means other callers could be affected as a result. What we really want to do is return a “promise”. From the jQuery Promise Documentation we learn:
“This object provides a subset of the methods of the Deferred object (then, done, fail, always, pipe. isResolved, and isRejected) to prevent users from changing the state of the Deferred.”

$.when(
    $.ajax( "/push_to_twitter" ),
    $.ajax( "/push_to_facebook" ),
    $.ajax( "/push_to_linkedin" ),
    $.ajax( "/generate_report" )
).then( it_succeeded, it_failed);
 
function it_succeeded(report_data){
    // Now do something with the report data e.g. show
    // an awesome summary with pretty bars and graphs!
}
function it_failed(){
    // Treat user's depression, OCD and other ailments
    // Prescribe prozac, etc., etc...
}

Possible Uses

Possible use cases are endless but here are some examples:
1. You need to make multiple asynchronous Ajax requests – with Deferred objects you can handle the results together without concern for which order they finish in.
2. You want to be able to make the call in one region of code, then pass the result as an argument to a different part, but the call may be asynchronous.
3. Easily return data from a modal dialog
4. Caching – an API properly supporting Deferred objects can implement caching without any changes to the consumer of the API.

An Example: Caching

For caching, jQuery’s when .. then provides us the tools we need to return both fresh and cached data from one interface. Take the following code:

var cacheA = {};
function fetch_some_objectA(key){
    if (cacheA[key]) { return cacheA[key]; }
    return cacheA[key] = $.ajax('/fooEndpoint', {
        // ajax options here
    }).done(function(res){
        cacheA[key] = res;
    });
}
var cacheB = {};
function fetch_some_objectB(key){
    if (cacheB[key]) { return cacheB[key]; }
    return cacheB[key] = $.ajax('/barEndpoint', {
        // ajax options here
    }).done(function(res){
        cacheB[key] = res;
    });
}
 
$.when(fetch_some_objectA('foo'), fetch_some_objectB('bar').done(function(dataA, dataB){
    // Do whatever you want with said data
}).fail(function(errA, errB) {
    // Deal with error handling
});

The basics of caching with Deferred objects

There are a couple of cases to consider here; the first case is that you are requesting object(‘foo’) for the first time. In this case, cacheA["foo"] will not exist and so an ajax call is created. $.ajax returns a deferred object, which is saved in the cache. The reason this is important is that thereafter, future requests will get that same deferred object, thus preventing two requests from being made to the same endpoint. Once the ajax call completes the done handler we set updates the cache with the actual object. Any calls that ran fetchsomeobjectA(‘foo’) while the request was “in progress” will be notified with the data once the deferred is resolved. Calls made after the request has resolved will get the actual data straight away. So, how do you tell if you have a deferred or the real data? The answer is simple: you don’t need to.

This is where jQuery.when neatly ties everything together. jQuery.when ($.when) accepts any number of arguments and returns a jquery.Deferred object. That object will be resolved once all of the arguments are resolved; however, if those objects are not jQuery.Deferred objects then they will be considered resolved immediately. From the documentation, jQuery.when takes either a Deferred or object: “If a single argument is passed to jQuery.when and it is not a Deferred, it will be treated as a resolved Deferred and any doneCallbacks attached will be executed immediately.”

After passing through a jQuery.when, a Deferred object that resolves to a string “A” and the string “A” become functionally equivalent — no matter which you passed in, you’ll still get the same thing:

$.when("fdsafdsa").done(function(var) { alert(var); });

This will immediately alert “fdsafdsa”. Now essentially the same using the deferred resolve idiom:

var dfd = new $.Deferred();
$.when(dfd).done(function(var) { alert(var); });
dfd.resolve("fdsafdsa");

This will also alert the same thing; it doesn’t matter if it’s a real object or a deferred, once you make it into the when it’ll be a real object. This makes caching really slick because you don’t need to care if you have the object yet or not, you just let jQuery.when figure that out and tell it what should happen once it does.

One small problem…

If you actually copy in this code and play with it long enough, you’ll notice something that doesn’t quite work how you expect. When the jQuery.ajax call returns you’ll actually get a slightly different object in the jQuery.when callback; this actually makes sense if you look at it closer. From the jQuery codebase, when the ajax call completes it is either resolved or rejected with the following lines:

// Success/Error
if ( isSuccess ) {
    deferred.resolveWith( callbackContext, [ success, statusText, jqXHR ] );
} else {
    deferred.rejectWith( callbackContext, [ jqXHR, statusText, error ] );
}

Notice that there are actually three arguments in either case; this may surprise you, since most examples you’ll see of the callbacks accept just one, like the above code. Most don’t need the other two, but jQuery.when isn’t going to discard those extra values and yet it still needs to keep one argument per input in the callback function. Therefore, in the $.when block above you’ll actually have both dataA and dataB resolve to an array with three values (success, statusText, jqXHR); however, we normally would only cache one of those values (success) so when reading from the cache we don’t get that array. The solution is to make our functions a little bit more complicated by using the jquery pipe method to restructure the result a little bit:

var cacheA = {};
function fetch_some_objectA(key){
    if (cacheA[key]) { return cacheA[key]; }
    return cacheA[key] = $.ajax('/fooEndpoint', {
        // ajax options here
    }).pipe(function(res){
        return cacheA[key] = res;
    });
}
var cacheB = {};
function fetch_some_objectB(key){
    if (cacheB[key]) { return cacheB[key]; }
    return cacheB[key] = $.ajax('/barEndpoint', {
        // ajax options here
    }).pipe(function(res){
        return cacheB[key] = res;
    });
}
 
$.when(fetch_some_objectA('foo'), fetch_some_objectB('bar').done(function(dataA, dataB){
    // Do whatever you want with said data
}).fail(function(errA, errB) {
    // Deal with error handling
});

You may notice the assignment of the result to the cache happens twice. The reason is that you first want to assign the deferred object to the cache when you start the request — otherwise if you call the function again before the ajax request finishes it’ll start a new request rather than reusing the old one.

With this new object (the result of the .pipe) the resolved value will be as we expect it. This relatively simple pattern can get a little more complex if you’re doing a lot with pipe, but for the consumer the interface is really simple.

Admittedly, there is some complexity to using this pattern but the “win” is that your interface now has a “Single Point of Contact” (SPOC)
Let’s look at a convenient analogy…

An Example: Placing a Special Order

Let’s imagine that, as an active supporter of your local community businesses, you make it a point to go to your favorite little “mom and pop” bookstore to place orders for books. The lady at the counter asks for your phone number or email, and says she’ll contact you as soon as your books come in. Now obviously, there’s the small chance that, after the fact, the wholesaler, manufacturer, etc., will have ran out of the ordered book(s). Should this happen, you’d probably feel it was acceptable for the bookstore to contact you and give you the option to wait or cancel your order. However, you would not want to here directly from the wholesaler or manufacturer directly. This would probably confuse you and you may even complain to the bookstore that they should manage their purchases better should this happen! The expectation you have as a customer in this example is that you only have to deal with a single point of contact. In code, we can have a similar convenience using deferred objects:

// Contrived ajax and random failure stubs so we can
// pretend to have "mostly successful" requests
function randomTrueFalse() {
    Math.random();
    var r = Math.round(Math.random()*10);
    return (r%9 !== 0);
};
 
function ajaxStub(endpoint) {
    var dfdResult = new $.Deferred();
    var bool = randomTrueFalse();
 
    setTimeout(function() {
        if(bool) {
            dfdResult.resolve('success: '+endpoint);
        }
        else {
            dfdResult.reject('fail: '+endpoint);
        }
    }, 100);
    return dfdResult.promise();
};
 
var manufacturer = (function() {
    return {
 
        orderBook: function(bookName) {
            return $.when(ajaxStub('/book/'+bookName)).pipe(function(respObj) {
                console.log("Manufacturer-DONE: ", respObj);
                return respObj;
            }, function(errObj) { console.log("Manufacturer-FAIL: ", errObj); return errObj; });
        }
    };
}());
 
var wholesaler = (function() {
    var m = manufacturer;
    return {
        orderBook: function(bookName) {
            return $.when(m.orderBook(bookName)).pipe(function(respObj) {
                console.log("Wholesaler-DONE: ", respObj);
                return respObj;
            }, function(errObj) { console.log("Wholesaler-FAIL: ", errObj); return errObj; });
        }
    };
}());
 
var retailer = (function() {
    var w = wholesaler;
    return {
        orderBook: function(bookName) {
            return $.when(w.orderBook(bookName)).pipe(function(respObj) {
                console.log("Retailer-DONE: ", respObj);
                dfdResult.resolve(respObj);
            }, function(errObj) { console.log("Retailer-FAIL: ", errObj); return errObj; });
        }
    };
}());
 
// Notice that, although we have multiple potential points of failure above (e.g.
// Manufacturer, Wholesaler, Retailer, the customer only needs to deal with the
// retailer directly.)
function makePurchase() {
 
    // Place an order for two books
    for(var i=1; i<3; i++) {
        $.when(retailer.orderBook("Great Book - Part " + i)).done(function(respObj) {
            console.log("Purchaser-DONE... respObj: ", respObj);
        }).fail(function(err) {
            console.log("Purchaser-FAIL... err: ", err);
        });
    }
};
makePurchase();
 
/*
Random output for placing order of two books might look like:
Manufacturer-FAIL: fail: /book/Great Book - Part 1
Wholesaler-FAIL: fail: /book/Great Book - Part 1
Retailer-FAIL: fail: /book/Great Book - Part 1
Purchaser-FAIL... err: fail: /book/Great Book - Part 1
Manufacturer-DONE: success: /book/Great Book - Part 2
Wholesaler-DONE: success: /book/Great Book - Part 2
Retailer-DONE: success: /book/Great Book - Part 2
Purchaser-DONE... respObj: success: /book/Great Book - Part 2
*/

Start at the bottom at the makePurchase() call and you should see that we’ve essentially modeled our story of a customer (purchaser) placing an order for two books. Should the purchase succeed or fail, the purchaser will only ever have to deal with the retailer directly (as expected), because the fact that deferred promises can sort of “bubble back up” the call chain. This keeps the interface quite clean, and the caller’s required knowledge of inner working to a minimum.

Conclusion

I’ve purposely kept the examples contrived so you can focus on the principles of deferred objects. If you’d like to see a variety of other example implementations, Julian Aubourg and Addy Osmani have written a wonderful article you should definitey have a look at. jQuery deferred objects take a bit to get used to, but provide some very compelling advantages for web applications that require heavy ajax or graphics.

Given the asynchronous nature of the web, and specifically JavaScript, a general asynchronous callback pattern has proliferated. So you have something you need to defer a bit so you use a timeout or an interval and you’re good to go…what’s the issue you may be asking? Well…web applications have become more and more complex, and what was once a nested callback or two, is turning into a hairy beast of nested callbacks.

The Deferred pattern addresses this nested nightmare by using using an idiom much like publish/subscribe (Observer) or Notifications (if you come from the land of iOS development). However, notification is a bit different because the caller has access to the Deferred object itself and can determine whether to call success or fail callbacks. This is sort of a forward reference, but will hopefully be clearer further down in this article. That said, let’s dig in…

If you’re familiar with stacks and queues then the underlying concept should be pretty straight forward. Essentially, the Deferred object implementation keeps a stack of callback function objects (similar to subscribers) for success or failure. Conventionally, these are called done callbacks or fail callbacks. Specifically, these function objects get pushed on the Deferred object when you call .then(myfunc), .done(myfunc), or .fail(myfunc):

var deferred = new Deferred();
deferred.done(function(message) {
  alert(message);
});
setTimeout(3000, function() {
  deferred.resolve("Hey buddy!");
});

First we create a deferred object. Immediately, the deferred object goes in to the “Pending” state.

Next, and here’s the important part to “grok”, by calling .done(… on the Deferred object, we are essentially asking it to add our function object to its array of callback functions.

Ok, so now we have 1..* functions queued (or stacked depending on implementation details) on the deferred object. Now one of two things will happen (at some point):
1. We call deferred.resolve (or one of it’s variants)
2. We call deferred.reject

If we do 1., this will “trigger” the deferred object to call all of the queued resolve category functions. These are the .done, .then, variant callbacks.

If we do 2., this will “trigger” the deferred object call all of the queued fail variant functions.

There are some semantic details to all of this, but that drives to the essence of the pattern. If you’re still not clear, I’ve drawn this extremely ugly diagram to help to visualize the essence of the deferred pattern (as described in both the jQuery API docs and CommonJS Promises page). Open your browser wide and start on the left and work to the right:

Cool! So how is this Deferred object doing it’s magic internally? Here’s a simplified example of how it may be implemented:

 
<h1>JavaScript Deferred Objects Example</h1>
 
function Deferred() {
    var callbacks = [];
    this.done =  function(cb) {
        callbacks.push(cb);
    }
 
    this.resolve = function(arg) {
        for(var i = 0; i < callbacks.length; i++) {
            callbacks[i](arg);
        }
    }
}
 
// Now we use the deferred object
var deferred = new Deferred();
deferred.done(function(message) {
  alert(message);
});
setTimeout(function() {
  deferred.resolve(&quot;Hey buddy!&quot;);
}, 3000);

What’s interesting about this pattern is that the owner of the deferred object determines whether we go in to the resolved or rejected state not the deferred object itself. This is slightly different from how a traditional Pub/Sub pattern would work, where the Publishing object itself would likely determine who/when to callback it’s subscribers. Because of this, it’s common to see usage where an intermediary function that is supplying some service (e.g. fetchResource, hitDatabase, etc.), will negotiate the initialization of the deferred object, determine whether to call resolve or reject, etc., and simply return the deferred object. So it looks like:

The above diagram only shows part of the picture. Once the Broker object calls resolve or reject, the original caller’s respective .done, .fail, etc., will get called.

All of this is possible because of JavaScript’s ability to treat functions as first class citizens (a fancy way of saying you can pass functions around and call them later and treat them like good ol’ objects!).

I hope this posting was somewhat helpful and revealed the magic behind Deferred Objects.

EDIT

My good friend Richard Bateman offers a point of clarification. There is actually a difference between .then and .done; .then allows you to provide two callbacks, the first for done and the second for fail.

Next part

For more on jquery deferred objects, see part 2 of this series.

However, after a lot of discussion we ended up scrapping this plan for a slightly different one.  I had already written a patch to make backbone.js routers support multiple route handlers, though, and I didn’t think it made sense to just throw it out, so I’ll provide it here.  Keep in mind that this may not work with future versions of backbone.js!

So, without further ado, here it is:

define(["require", "./underscore", "./backbone"], function(require, _, Backbone) {
    // Backbone.js doesn't quite do what we want it to, but it almost does;
    // we can modify it so that it does do what we want pretty easily, thanks
    // to the eminent hackability of javascript.
    //
    // We may need to update this when backbone.js updates! :-/


    var patched_methods = {
        // Save and rename the old route function, modified to work with the new
        override_route : function(route, callback) {
            this.handlers.unshift({route : route, callback : [callback]});
        },


        // Add a route to be tested when the fragment changes.
        // RB 9/15/2011: We don't want routes added later to
        //               override previous routes; we want
        //               routes added later to get called as well.
        //
        //               With this change, callback becomes an array
        //               of callbacks instead of just one.
        route : function(route, callback) {
            var appended = _.any(this.handlers, function(handler) {
                if (handler.route.toString() == route.toString()) {
                    handler.callback.unshift(callback);
                    return true;
                }
            });
            if (!appended) {
                this.handlers.unshift({route : route, callback : [callback]});
            }
        },


        // Add a feature to remove a route, since we no longer override them
        unroute : function(route, callback) {
            var found_and_removed = _.any(this.handlers, function(handler) {
                if (handler.route == route) {
                    var idx = handler.callback.indexOf(callback);
                    if (idx > -1) {
                        handler.callback.splice(idx, 1);
                        return true;
                    } else {
                        return false;
                    }
                }
            });
        },


        // Attempt to load the current URL fragment. If a route succeeds with a
        // match, returns `true`. If no defined routes matches the fragment,
        // returns `false`.
        // RB 9/15/2011: overriding this function
        loadUrl : function(fragmentOverride) {
            var fragment = this.fragment = this.getFragment(fragmentOverride);
            var matched = _.any(this.handlers, function(handler) {
                if (handler.route.test(fragment)) {
                    _.each(handler.callback, function(callback) {
                        callback(fragment);
                    });
                    return true;
                }
            });
            return matched;
        }
    };


    _.extend(Backbone.History.prototype, patched_methods);
    return Backbone;
});

• TotalTerminal has replaced Visor
• Mac OS X Lion has been released

TotalTerminal changes up the way it launches Terminal — instead of being automatically injected via SIMBL, it manually injects the plugin into Terminal.app. Well, I wasn’t going to let this deter me.

Getting back to Nirvana after installing TotalTerminal is not all that bad. Most of the steps haven’t changed from my original post so I will only go into detail about the new steps here. Briefly, here are the steps I followed.

1. Remove TerminalColours.bundle SIMBL plugin — per the TotalTerminal FAQ, it injects its own version and having TerminalColours.bundle installed creates a conflict. (~/Library/Application\ Support/SIMBL/Plugins)
2. Fix the Home / End keys (see original post)
3. Copy Terminal.app to VisorTerminal.app (see original post)
4. Install TotalTerminal
5. Modify TotalTerminal to use VisorTerminal.app instead of Terminal.app

That’s right, there’s really only one thing to change, and it’s very simple

TotalTerminal.app has a simple, compiled AppleScript which launches Terminal and injects itself. The compiled file is located at /Applications/TotalTerminal.app/Contents/Resources/Scripts/main.scpt. But, we don’t want the compiled binary version, we want the original so we can make it use VisorTerminal.app.

Fortunately the uncompiled source file is available in the TotalTerminal repo on Github.

Only one change is necessary.

--- TotalTerminal.app/Contents/Resources/Scripts/main.applescript
+++ TotalTerminal.app/Contents/Resources/Scripts/main.applescript
@@ -1,4 +1,4 @@
-tell application "Terminal"
+tell application "VisorTerminal"
        delay 1 -- this delay is important to prevent random "Connection is Invalid -609" AppleScript errors
        try
                event BATTinit

Once that is done, you just need to compile the script and replace the one that shipped with TotalTerminal.

$ osacompile -o main.scpt main.applescript
$ mv main.scpt /Applications/TotalTerminal.app/Contents/Resources/Scripts

That’s it.

You should be good to go. Put TotalTerminal.app into your user startup applications if you wish (I do), or launch it manually by launching TotalTerminal.app.

Enjoy the restoration of terminal nirvana.

For the lazy…

My modified and compiled main.applescript is available for download.

I tried VMWare server for awhile, but found that every time I updated my system everything broke and I had to recompile drivers.

I tried VirtualBox in headless mode, which worked much better; still, it was hard to keep things running well since something always seemed to go wrong if the server got restarted for any reason.

Finally, I decided to invest in a machine to dedicate to the task. I did some research and decided on VMWare ESXi as the host platform — the free version is plenty for what I need.

Finding “Supported” Hardware

The list of supported hardware that I found was filled with things outside of my price range; okay, I admit it, I’m a bit cheap. I’ll save you the sob story of the time it took me to piece together a system that I thought might work and just give you the list of what I settled on. This is not a “supported” setup, but it does work (with a little tinkering with the setup file):

• $179.99 – G.SKILL 16GB of ram – I wanted a little more performance, so I got the faster set.
• 2 x $139.99 – Seagate Barracuda XT 2TB 7200 RPM SATA 6.0GB/s hard drive – My goal was to mirror these; that hasn’t turned out to be directly possible, but I have ideas for workarounds.
• $29.99 (+$9.99 shipping) Rosewill R101-P-BK Mid Tower case – Just about anything would work, but this case seems adequate so far
• $59.99 Rosewill Green Series 630W Power supply – Again, I wasn’t really picky, but this one looked decent and specifically says it works for i7 cpus
• $299.99 Intel Core i7-2600 3.4Ghz quad core CPU – Note that this is the 2600, not the 2600k. There are virtualization features missing on the 2600k.
• $139.99 Asus P8Q67-M LGA 1155 Intel Q67 Motherboard – This one is where it really got scary and dicy. The new CPUs (like the 2600, not the 2600k) support something called VT-D. Now, I don’t have this fully working yet, but the Q67 boards are supposed to support it as well; mine says it does, and I have enabled it, but I haven’t seen it work in VMWare yet, so I’m not 100% sure if this is working or not.  However, since most of the hardware needing support is on this board, which motherboard you get matters a lot.

Total cost of the system after shipping: $1,007.25

Installing VMWare ESXi

The first thing I tried, of course, was to install from the ISO I downloaded from VMWare.  Unfortunately, that gave me an ugly python error message; I forget exactly what it said. A little constructive googling turned up that it meant that ESXi couldn’t find a valid network card.  DRAT! I knew I was right to be worried about the motherboard choice; it has an Intel chipset, but the NIC is not supported by VMWare ESXi. I was quite put out and even started looking to see if any local computer stores had a NIC that was supported so I could get things going today. However, I found something better!

Customizing the install disk

As it turns out, there is a website (vm-help.com) for people who are running ESXi on hardware other than the “official supported” hardware. It took me awhile to find what I needed, but it turns out someone else has already had the issue I did (probably on a different motherboard) and created a driver customization archive (oem.tgz) with the drivers I need!

Here are the steps:

1. Download the latest VMWare ESXi install iso; the one I got was called “VMware-VMvisor-Installer-4.1.0.update1-348481.x86_64.iso”
2. Found and downloaded the oem.tgz file (2 rar files that I extracted) that supports the Intel 82579 Gigabit ethernet card
3. Followed these instructions to use the mkesxiaio shell script (on a linux machine) to customize my ISO
4. Burned the new ISO and installed

That was it! It works now! I did of course have to turn on all the virtualization features in the motherboard BIOS settings; I also set the SATA controller to AHCI mode. Really, though, everything seems to work perfectly and I’m now in the process of adding virtual machines.

Life is good =]

Yes, I wrote FireBreath. A lot of others have contributed.

A different approach to learning NPAPI

Now that I have a good example already up and running, it seems pointless to keep trying to describe things with words and code snippets when I can simply walk you through what I have done — what is in use and works well. To that end, and in hopes that it will foster more capable NPAPI developers to help with FireBreath development (we still don’t support printing, for example), I have created a series of 5 minute screencasts providing a brief overview of FireBreath’s NPAPI framework.

I don’t expect that this is as comprehensive as it should be; I’m hoping that comments will help me see what future segments I should create.  I have posted these screencasts on youtube and will link to them from here. You can also find a more complete series that includes this one on the FireBreath website.

I just finished setting up a build server on Mac OS X using Jenkins (formerly Hudson). The company I’m working for (GradeCam) uses git and gitolite for our source control and so I expected no trouble using Jenkins to build our tools using the git plugin.

However, I quickly ran into a snag: the source control server is on a public address and so our source code is not available except via ssh, and gitolite ssh access uses private key authentication.  Well, I’m an experience unix sysadmin, so that didn’t sound like a big issue — after all, setting up public key authentication is childs play, right?

Default install

The default installation of Jenkins on Mac OS X (at the time of this writing) installs a Launch Agent plist to /Library/LaunchAgents/org.jenkins-ci.plist. This plist file causes Jenkins to load as user “daemon”, which sounds fine — except that the home directory for the “daemon” user is /var/root, same as for user root.  This means that the .ssh dir in there will never have the right permissions for a private key to be used.

Creating a new hidden user

My solution was to create a new “hidden” user for Jenkins to run under. Following instructions I found on a blog post, I created a user “jenkins” with a home directory “/Users/Shared/Jenkins/Home”:

sudo dscl . create /Users/jenkins
sudo dscl . create /Users/jenkins PrimaryGroupID 1
sudo dscl . create /Users/jenkins UniqueID 300   
sudo dscl . create /Users/jenkins UserShell /bin/bash
sudo dscl . passwd /Users/jenkins $PASSWORD
sudo dscl . create /Users/jenkins home /Users/Shared/Jenkins/Home/

I then stopped Jenkins: “sudo launchctl unload -w /Library/LaunchAgents/org.jenkins-ci.plist” and edited the plist file to set the username to jenkins instead of daemon.  “chown -R jenkins: /Users/Shared/Jenkins/Home” sets the permissions how they need to be, and then “sudo launchctl load -w /Library/LaunchAgents/org.jenkins-ci.plist” should get you up and running!

To get git over ssh running, “sudo su – jenkins” to get a console as the jenkins user and set up the ssh keys and such. Make sure you can ssh to where you want to go (or even do a test git clone) because you need to save the keys so it doesn’t ask for them when jenkins tries to do the clone.

That should do you! Hope it helps someone.

We needed something easier. We did not want to go to a workflow where there is one git user on the system, every employee authenticates with their own public key to that user, and everyone with a key has access to everything. We wanted more granular control, but also an easier management workflow. While gitolite does use a single system user under the covers, it provides per-user access control (governed by public keys), easy repo management, and a couple other nice features like wildcard repositories and branch-level access restrictions.

However, one of our developers is in the habit of accessing a random server, cloning a repo (using his password), and pushing from environments other than his development environment. Obviously, his private key is not going to be on each of the machines he does this from. Gitolite does not allow password-based access for obvious reasons — it needs to know who you are to determine permissions — so gitolite was cramping his style.

There are a couple ways around this:

1. Add a public key for each system he pushes changes from. Since he often logs in as “administrator” or “root” on individual systems, this obviously is not a good solution.
2. Have the developer enable SSH Agent Forwarding from his machine. This works well, as long as the user does not chain sessions (connect to server A, from there to server B, and clone on B) or is willing / able / allowed to enable Agent Forwarding on each system in the chain. (NOTE: on Mac OS X Snow Leopard, enabling SSH Agent Forwarding is as simple as running ssh-add and adding two lines to your ~/.ssh/config file:

   # add key to agent (I do this in ~/.profile)
   ssh-add $HOME/.ssh/id_rsa
    
   # ~/.ssh/config
   Host *
       ForwardAgent yes

3. Come up with something else.

What we really wanted was a way to allow password-based access to the repositories. Since gitolite does not support this itself, we had to come up with another way. As it turns out, doing so is not too bad, though it does involve a little bash shell black magic.

What we came up with is creating a system user on the git server for each developer who wants password access to git repositories, using a custom script as their defined shell (instead of /bin/sh or /bin/bash). Don’t forget to make your new script executable.

This script needs to look at what the user is trying to do (git action, gitolite ADC, some random shell command, nothing…) and act accordingly. Here’s the script we came up with, which will be dissected below to explain what each part does in more detail than the inline doc (gist).

#!/bin/bash
shift # get rid of -c
 
# if no commands, just open a shell
if [[ $# -eq 0 ]]; then
        /bin/bash -l
 
# if the first arg is a git- command, that means it is something like git-push, etc... so forward it
elif [[ $1 == git-* ]]; then
        ssh -q -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no git@localhost $*
 
# if the first arg is SET_ENV_ONLY, we sourced this file in order to set up the gitolite function
elif [[ $1 == "SET_ENV_ONLY" ]]; then
        gitolite () {
                ssh -q -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no git@localhost $*
        }
 
# if there is at least one non-git command, source this file in a new shell and create the 'gitolite' function
else
        /bin/bash -c "source $0 shiftme SET_ENV_ONLY; $*"
fi

If the user is not trying to do anything (just wants to open a shell), then there are no arguments and we should just open a bash prompt:

if [[ $# -eq 0 ]]; then
        /bin/bash -l

Okay, the simple bit is out of the way. Now on to the black magic.

If the user is trying to perform a git command (via git push, etc.) then we create a new SSH session to localhost (remember, this is running on the same server as gitolite) and pass along the commands ($*). Since the user has already logged in, this ssh command will run from their account using their private / public key. The git commands are things like git-receive-pack, git-upload-pack, etc, so we match the first script argument against “git-”.

Finally, we don’t care about validating the host’s fingerprint (we are already on the host) and there can be no man in the middle (since we are going to localhost), so we can turn off StrictHostKeyChecking. We also don’t want to print out the various fingerprint messages that SSH will generate when you ignore host keys, so we make the new SSH connection quiet. That’s it for this block:

elif [[ $1 == git-* ]]; then
        ssh -q -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no git@localhost $*

I am going to go over the last two pieces in reverse order, because it is easier to follow.

If the user sent some SSH commands that were not native git commands, we need to process those commands. So, we will execute the remaining commands in a new bash process.

BUT, what if the user is trying to perform a gitolite command (such as expand or an ADC), what should we do? We can’t just trust that all commands the user sends will either be a git command or a gitolite command, so we need to set up some more black magic to handle that for us. So the first thing we will do when we open a new bash process to handle the remaining commands is source this custom script ($0) with a custom value we made up (SET_ENV_ONLY). That way, if this script executes again we will know it was done by ourselves and we can set up some more magic.

else
        /bin/bash -c "source $0 shiftme SET_ENV_ONLY; $*"

Finally for the last block.

If our new script is being executed, and the first argument is SET_ENV_ONLY (after shifting), then it is because we sourced it from our new bash process from the previous step. What we do now is set up a gitolite command which does something very similar to what happens if the user is performing a native git command: it opens an SSH connection to git@host and performs the gitolite commands.

After this has been done, “gitolite expand” is the same as “ssh git@git.company.com expand”. This makes it so the developer can still run gitolite commands and ADCs, he just has to prefix each command with gitolite. Something like this:

$ ssh [DEVELOPER]@git.company.com "gitolite expand; echo $PATH; gitolite info"

is equivalent to this:

$ ssh git@git.company.com expand
$ ssh [DEVELOPER]@git.company.com echo $PATH
$ ssh git@git.company.com info

Now you just need to create accounts that use this script as their shell. Each user can add whatever public keys they want to their authorized_keys file, or they can authenticate solely with their password. The only requirement is that their server account has a private / public key generated, and that the public key is added to gitolite.

$ ssh root@git.company.com
$ useradd -s /usr/local/bin/custom_gitbash.sh -m [DEVELOPER]
$ su [DEVELOPER]
$ ssh-keygen -t rsa
$ cp ~/.ssh/id_rsa.pub ~/[DEVELOPER]@git.company.com.pub
# add the new public key to the gitolite keydir