Like many, the suggestion that JavaScript could be used in a fully architected system initially left me doubting the sanity of the person with whom I spoke. Over the years I have done several different types of Web Development; I am best known for my work with browser plugins in general and the FireBreath framework in particular. What most do not know is that I actually started out doing a lot of PHP and JavaScript development and that it was actually my work with JavaScript that eventually led me to learning about NPAPI and ActiveX plugin technologies.

Throughout all this I experimented with many different types of javascript code; I wrote AJAX applications long before I’d heard the term coined using a system called JSRS — which is surprisingly still around, or at least the page I got it from. I worked for Move Networks on their JavaScript SDK and I wrote several AJAX heavy websites, such as HamStudy.org. Many reading this will scoff when I admit that while I considered myself fairly well versed in JavaScript I still wrote it as if it were a simple scripting language. Others may wonder what is wrong with that? JavaScript more than perhaps any other language I have used gives you just enough rope to shoot yourself in the foot with. I have learned over the last few months just how little I knew — and didn’t know that I didn’t know.

My upward climb started off simply enough — I wanted to learn about some of the new technologies that I had heard about; Backbone, MVC frameworks, client-side template libraries… I had written my own “frameworks” on which to base my websites, all of which were side projects, but I found myself spending much more time than I liked updating and maintaining those frameworks. I thought perhaps I could use someone else’s system and save myself some trouble.

I purchased a book called JavaScript Web Applications which claimed to teach how to write an actual application with JavaScript. The table of contents looked interesting, and besides teaching about the principles used it also had instructions on several different MVC frameworks that I wanted to know about, including Backbone.

This book was the beginning — as a result of this purchase I soon came to grips with just how comparitively incomptetent my old code had been, and started to actually understand that JavaScript can indeed be a real language. Over the course of a few posts I would like to share with you the stack that has grown out of this system: What the components are, how they help, and why they are integral to the overall solution. I will start out in this post with the basics and my next post will detail the specific tweaks that we have made to them that have made our system so successful.

Dependency Management

Having been so long-winded with my earlier explanation, I will now spare you the details of all of the things I tried, researched, or experimented with during my search. One thing quickly became clear to me, though: Perhaps the #1 reason that my previous code had ended up so ugly was the need to avoid creating lots of JavaScript files. These can be difficult to deploy cleanly as it is not a good idea to have too many script includes in your production site and combining them can be an annoying process. The end result of this problem is that I had about 10 JavaScript files — and felt guilty about it — and all of my functionality was stuffed in those files. I had page-specific files, site-specific files, and general reuse library files, and all of them tried to register themselves cleanly in the global scope without clobbering each other. In short, it was a mess.

Enter require.js. Require.js is a system in which all javascript files are wrapped in a function. This is actually a really common pattern, but with require.js that function is passed into the “define” function call along with a list of dependencies. Require will first resolve these dependencies and then when they are all resolved it will call the function you provided, passing in the exports of each dependency as arguments in the order that the dependencies are listed in the define call.

The question left unanswered, then, is what are the exports of the other dependencies? Simply put, each “module” exports some value by simply returning that value from the function passed into the define call (See file1.js and file2.js).

define(["backbone", "./reqfile2"], function(Backbone, File2) {
    // In this file we do whatever it is we do; we know file2 was loaded first and we can use what it exports


    return File2.text + " from two different files";
});

define([], function() {
    var File2 = {};
    File2.text = "This is some text";
    return File2;
});

The natural result of so much flexibility is a lot of small files; this is really good for code organization because it makes it easy to seperate things into logical pieces and even to include whole modules consisting of multiple files (because one can depend on multiple others and combine them in its return value). The downside is that with lots of small files your page load time starts to get a bit out of control. Fortunately, require.js has an answer for this problem as well!

Require.js has an optimize phase that will examine the dependencies of your main file and combine all of them into a single file so that they are all loaded at once. The require.js system still makes sure that the functions are called in the correct order, but makes it very easy for your production site to be seamlessly updated to use a combined and minified single javascript file instead of the potentially dozens of small other files.

To me, this was a nearly life-changing WIN.

JavaScript Templates

If dependency management became the #1 most useful change in my architecture, a close #2 would have the be JavaScript Templates. The more I did things on the client the more client-side HTML I created in JavaScript; I experimented with several methods but still ended up with really hard to read html mixed in with my JavaScript as strings. When the time came to update one of those sites it was painful; not all of the code was in the same place, it wasn’t easy to read, it wasn’t easy therefore to change.

I considered different types of templates but had previously rejected the idea. The problem with templates, you see, is that you still have to include the template code somehow. The most “traditional” method would be to pretend you’re on a backend and just store them one template per file. Of course, then if you have any real number of these files you find yourself making a multitude of AJAX requests to download all of these templates; it’s clean from a file structure perspective and fairly easy to use, but to me the added complexity of all those requests is not acceptible. Other options include storing the template in javascript strings (same problem as I had before without templates) and storing them in a special script tag on the html page, which had the (to me) serious disadvantage of being nearly impossible to reasonably use on multiple pages. It was a problem.

Enter require.js and the text plugin. The text plugin of require.js is an idea both simple and brilliant; it allows you to specify files that should be loaded and the text of those files will be passed into your page function as the value of that dependency. With the text plugin you can save your templates each in their own file and have require load the text when the page loads. Of course, this doesn’t solve the problem we mentioned earlier! You still have potentially lots of AJAX requests being made to load those files. Problem? Not at all! Remember the optimize step in require.js; just like the normal dependencies are loaded and combined into one file the text plugin provides a mechanism to serialize those templates into javascript as strings and combine them into that same file. You get the best of both worlds! When you’re developing you have the flexibility of having each template in an easy-to-read file, but when you deploy those templates are packaged with your javascript.

define(["backbone", "underscore", "text!./mytemplate.jst"], function(Backbone, _, tplCode) {
    
    var template = _.template(tplCode);


    function doSomething(val) {
        alert("Here is the rendered template: ", tplCode({key: val}));
    }
    return doSomething;
});

Top all of this off with the fact that for several of the most popular template libraries there are already require.js plugins, such as requirejs-tpl. These plugins work like the text plugin, but they compile the template string into an actual template function before passing it back to resolve the dependency. Then, when you optimize, they actually compile your template code into javascript functions and minify them along with the rest of your JS code!

To me, this was a nearly life-changing WIN.

Deferred Objects

This may seem like it doesn’t fit with the other items on this page, but if templates and dependency management have revolutionized the organization of my code then deferred objects have revolutionized the way that I write JavaScript code. One of my coworkers at GradeCam, Robin, has already begun a small introductory series on how deferred objects work, so I’m not going to go into those details, but I do want to explain the conceptual shift in how pieces of the system interact that this led me to. It is both subtle and extremely powerful.

Most JavaScript operations anymore are asynchronous; in other words, they return immediately even though the operation in question hasn’t been completed. This can make it difficult to handle result cases; the traditional method is to pass a callback into the function to be called when the operation completes. You then perform some other operation once that function is called back and so forth.

The challenge with that is that it is a significant deviation from the patterns that we are all accustomed to. Granted, it’s important to understand what is actually happening, but often you may want to call a function in a place where you don’t actually want to handle the results; a good example is when you are requesting data from a model in your controller code, but you will actually be rendering it withing your view and your controller doesn’t really care what the results are. The view, of course, should not need to know or care where the data came from. In fact, it shouldn’t really need to know if the data is immediately available or will be available later on — it should display it when it is available and not worry about the other details.

Code speaks louder than prose, so here is an example of the difference between these two approaches; in both async1.js and async2.js three async operations take place. In async1.js they are handled in a traditional way, and in async2.js they are handled using jQuery.Deferred.

function asyncOper1(value, callback) {
    setTimeout(function() {
        callback("One second has passed since " + value);
    }, 1000);
}


function asyncOper2(value, callback) {
    setTimeout(function() {
        callback("Two seconds were added, then " + value);
    }, 2000);
}


function asyncOper3(value, callback) {
    setTimeout(function() {
        callback("A long time (3 seconds) went by before " + value);
    }, 3000);
}


function alertResult(value) {
    alert(value);
}


// Call them all; alert the final result
asyncOper1("we started", function(strVal1) {
    asyncOper2(strVal1, function(strVal2) {
        asyncOper3(strVal2, function(strVal3) {
            alertResult(strVal3);
        });
    });
});

function asyncOper1(valueDfd) {
    return $.when(valueDfd).pipe(function(value) {
        var dfd = new $.Deferred()
        setTimeout(function() {
            dfd.resolve("One second has passed since " + value);
        }, 1000);
        return dfd.promise();
    });
}


function asyncOper2(valueDfd) {
    return $.when(valueDfd).pipe(function(value) {
        var dfd = new $.Deferred()
        setTimeout(function() {
            dfd.resolve("Two seconds were added, then " + value);
        }, 2000);
        return dfd.promise();
    });
}


function asyncOper3(valueDfd) {
    return $.when(valueDfd).pipe(function(value) {
        var dfd = new $.Deferred()
        setTimeout(function() {
            dfd.resolve("A long time (3 seconds) went by before " + value);
        }, 3000);
        return dfd.promise();
    });
}


function alertResult(valueDfd) {
    $.when(valueDfd).done(function(value) {
        alert(value);
    });
}


var res1, res2, res3;


res1 = asyncOper1("we started");
res2 = asyncOper2(res1);
res3 = asyncOper3(res2);


alertResult(res3);

As you can see, the Deferred object method is slightly more complex at first; however, the complexity is where it belongs. Each method need know nothing about the others — they all assume that the input value *might* be Deferred. They return a Deferred object, and so they use the Deferred.pipe method, which makes it easy to chain deferred objects. Because each of the methods returns a Deferred object and is aware that it may receive a Deferred object, you can call one after the other as though they were happening synchronously — because indeed they will end up doing so, in all practical purposes. However, you shouldn’t think of this as hiding the fact that it is asynchronous, but rather as returning asynchronous return values.

Because we used $.when, you can pass a regular parameter in to any of those methods as well and chain them as much as you want. It will handle them all perfectly.

To me, this was a completely life-changing WIN.

This is just the beginning

I have a lot more to talk about here; this is just the foundation. If you know nothing more about our stack, you should understand these things. Next time, though, I’ll talk about how we tie this together using Backbone.js Routers, Views, and Models, how those tie into our templates, and how Deferred objects allowed us to add data cacheing to our data models completely transparently — without changing a line of code outside of the model classes.

As you recall from the first part jQuery Deferred Objects – Part I, deferred objects give you a way to organize asynchronous blocks of code in to a self-managed callback queue, while allowing you to attach 1..* callbacks. Also, you’ll recall that the basics of using jQuery deferred objects include essentially the following “simplified” steps:
1. Created a deferred object
2. Attached .done and .fail callbacks
3. Called resolve or reject, thereby triggering the appropriate respective callbacks

Promises

This was fine as a first step in understanding what’s going on, but in all likelyhood you’ll use a slightly different approach. What’s wrong with returning a deferred object itself? It gives the caller access to resolve or reject directly, and that means other callers could be affected as a result. What we really want to do is return a “promise”. From the jQuery Promise Documentation we learn:
“This object provides a subset of the methods of the Deferred object (then, done, fail, always, pipe. isResolved, and isRejected) to prevent users from changing the state of the Deferred.”

$.when(
    $.ajax( "/push_to_twitter" ),
    $.ajax( "/push_to_facebook" ),
    $.ajax( "/push_to_linkedin" ),
    $.ajax( "/generate_report" )
).then( it_succeeded, it_failed);
 
function it_succeeded(report_data){
    // Now do something with the report data e.g. show
    // an awesome summary with pretty bars and graphs!
}
function it_failed(){
    // Treat user's depression, OCD and other ailments
    // Prescribe prozac, etc., etc...
}

Possible Uses

Possible use cases are endless but here are some examples:
1. You need to make multiple asynchronous Ajax requests – with Deferred objects you can handle the results together without concern for which order they finish in.
2. You want to be able to make the call in one region of code, then pass the result as an argument to a different part, but the call may be asynchronous.
3. Easily return data from a modal dialog
4. Caching – an API properly supporting Deferred objects can implement caching without any changes to the consumer of the API.

An Example: Caching

For caching, jQuery’s when .. then provides us the tools we need to return both fresh and cached data from one interface. Take the following code:

var cacheA = {};
function fetch_some_objectA(key){
    if (cacheA[key]) { return cacheA[key]; }
    return cacheA[key] = $.ajax('/fooEndpoint', {
        // ajax options here
    }).done(function(res){
        cacheA[key] = res;
    });
}
var cacheB = {};
function fetch_some_objectB(key){
    if (cacheB[key]) { return cacheB[key]; }
    return cacheB[key] = $.ajax('/barEndpoint', {
        // ajax options here
    }).done(function(res){
        cacheB[key] = res;
    });
}
 
$.when(fetch_some_objectA('foo'), fetch_some_objectB('bar').done(function(dataA, dataB){
    // Do whatever you want with said data
}).fail(function(errA, errB) {
    // Deal with error handling
});

The basics of caching with Deferred objects

There are a couple of cases to consider here; the first case is that you are requesting object(‘foo’) for the first time. In this case, cacheA["foo"] will not exist and so an ajax call is created. $.ajax returns a deferred object, which is saved in the cache. The reason this is important is that thereafter, future requests will get that same deferred object, thus preventing two requests from being made to the same endpoint. Once the ajax call completes the done handler we set updates the cache with the actual object. Any calls that ran fetchsomeobjectA(‘foo’) while the request was “in progress” will be notified with the data once the deferred is resolved. Calls made after the request has resolved will get the actual data straight away. So, how do you tell if you have a deferred or the real data? The answer is simple: you don’t need to.

This is where jQuery.when neatly ties everything together. jQuery.when ($.when) accepts any number of arguments and returns a jquery.Deferred object. That object will be resolved once all of the arguments are resolved; however, if those objects are not jQuery.Deferred objects then they will be considered resolved immediately. From the documentation, jQuery.when takes either a Deferred or object: “If a single argument is passed to jQuery.when and it is not a Deferred, it will be treated as a resolved Deferred and any doneCallbacks attached will be executed immediately.”

After passing through a jQuery.when, a Deferred object that resolves to a string “A” and the string “A” become functionally equivalent — no matter which you passed in, you’ll still get the same thing:

$.when("fdsafdsa").done(function(var) { alert(var); });

This will immediately alert “fdsafdsa”. Now essentially the same using the deferred resolve idiom:

var dfd = new $.Deferred();
$.when(dfd).done(function(var) { alert(var); });
dfd.resolve("fdsafdsa");

This will also alert the same thing; it doesn’t matter if it’s a real object or a deferred, once you make it into the when it’ll be a real object. This makes caching really slick because you don’t need to care if you have the object yet or not, you just let jQuery.when figure that out and tell it what should happen once it does.

One small problem…

If you actually copy in this code and play with it long enough, you’ll notice something that doesn’t quite work how you expect. When the jQuery.ajax call returns you’ll actually get a slightly different object in the jQuery.when callback; this actually makes sense if you look at it closer. From the jQuery codebase, when the ajax call completes it is either resolved or rejected with the following lines:

// Success/Error
if ( isSuccess ) {
    deferred.resolveWith( callbackContext, [ success, statusText, jqXHR ] );
} else {
    deferred.rejectWith( callbackContext, [ jqXHR, statusText, error ] );
}

Notice that there are actually three arguments in either case; this may surprise you, since most examples you’ll see of the callbacks accept just one, like the above code. Most don’t need the other two, but jQuery.when isn’t going to discard those extra values and yet it still needs to keep one argument per input in the callback function. Therefore, in the $.when block above you’ll actually have both dataA and dataB resolve to an array with three values (success, statusText, jqXHR); however, we normally would only cache one of those values (success) so when reading from the cache we don’t get that array. The solution is to make our functions a little bit more complicated by using the jquery pipe method to restructure the result a little bit:

var cacheA = {};
function fetch_some_objectA(key){
    if (cacheA[key]) { return cacheA[key]; }
    return cacheA[key] = $.ajax('/fooEndpoint', {
        // ajax options here
    }).pipe(function(res){
        return cacheA[key] = res;
    });
}
var cacheB = {};
function fetch_some_objectB(key){
    if (cacheB[key]) { return cacheB[key]; }
    return cacheB[key] = $.ajax('/barEndpoint', {
        // ajax options here
    }).pipe(function(res){
        return cacheB[key] = res;
    });
}
 
$.when(fetch_some_objectA('foo'), fetch_some_objectB('bar').done(function(dataA, dataB){
    // Do whatever you want with said data
}).fail(function(errA, errB) {
    // Deal with error handling
});

You may notice the assignment of the result to the cache happens twice. The reason is that you first want to assign the deferred object to the cache when you start the request — otherwise if you call the function again before the ajax request finishes it’ll start a new request rather than reusing the old one.

With this new object (the result of the .pipe) the resolved value will be as we expect it. This relatively simple pattern can get a little more complex if you’re doing a lot with pipe, but for the consumer the interface is really simple.

Admittedly, there is some complexity to using this pattern but the “win” is that your interface now has a “Single Point of Contact” (SPOC)
Let’s look at a convenient analogy…

An Example: Placing a Special Order

Let’s imagine that, as an active supporter of your local community businesses, you make it a point to go to your favorite little “mom and pop” bookstore to place orders for books. The lady at the counter asks for your phone number or email, and says she’ll contact you as soon as your books come in. Now obviously, there’s the small chance that, after the fact, the wholesaler, manufacturer, etc., will have ran out of the ordered book(s). Should this happen, you’d probably feel it was acceptable for the bookstore to contact you and give you the option to wait or cancel your order. However, you would not want to here directly from the wholesaler or manufacturer directly. This would probably confuse you and you may even complain to the bookstore that they should manage their purchases better should this happen! The expectation you have as a customer in this example is that you only have to deal with a single point of contact. In code, we can have a similar convenience using deferred objects:

// Contrived ajax and random failure stubs so we can
// pretend to have "mostly successful" requests
function randomTrueFalse() {
    Math.random();
    var r = Math.round(Math.random()*10);
    return (r%9 !== 0);
};
 
function ajaxStub(endpoint) {
    var dfdResult = new $.Deferred();
    var bool = randomTrueFalse();
 
    setTimeout(function() {
        if(bool) {
            dfdResult.resolve('success: '+endpoint);
        }
        else {
            dfdResult.reject('fail: '+endpoint);
        }
    }, 100);
    return dfdResult.promise();
};
 
var manufacturer = (function() {
    return {
 
        orderBook: function(bookName) {
            return $.when(ajaxStub('/book/'+bookName)).pipe(function(respObj) {
                console.log("Manufacturer-DONE: ", respObj);
                return respObj;
            }, function(errObj) { console.log("Manufacturer-FAIL: ", errObj); return errObj; });
        }
    };
}());
 
var wholesaler = (function() {
    var m = manufacturer;
    return {
        orderBook: function(bookName) {
            return $.when(m.orderBook(bookName)).pipe(function(respObj) {
                console.log("Wholesaler-DONE: ", respObj);
                return respObj;
            }, function(errObj) { console.log("Wholesaler-FAIL: ", errObj); return errObj; });
        }
    };
}());
 
var retailer = (function() {
    var w = wholesaler;
    return {
        orderBook: function(bookName) {
            return $.when(w.orderBook(bookName)).pipe(function(respObj) {
                console.log("Retailer-DONE: ", respObj);
                dfdResult.resolve(respObj);
            }, function(errObj) { console.log("Retailer-FAIL: ", errObj); return errObj; });
        }
    };
}());
 
// Notice that, although we have multiple potential points of failure above (e.g.
// Manufacturer, Wholesaler, Retailer, the customer only needs to deal with the
// retailer directly.)
function makePurchase() {
 
    // Place an order for two books
    for(var i=1; i<3; i++) {
        $.when(retailer.orderBook("Great Book - Part " + i)).done(function(respObj) {
            console.log("Purchaser-DONE... respObj: ", respObj);
        }).fail(function(err) {
            console.log("Purchaser-FAIL... err: ", err);
        });
    }
};
makePurchase();
 
/*
Random output for placing order of two books might look like:
Manufacturer-FAIL: fail: /book/Great Book - Part 1
Wholesaler-FAIL: fail: /book/Great Book - Part 1
Retailer-FAIL: fail: /book/Great Book - Part 1
Purchaser-FAIL... err: fail: /book/Great Book - Part 1
Manufacturer-DONE: success: /book/Great Book - Part 2
Wholesaler-DONE: success: /book/Great Book - Part 2
Retailer-DONE: success: /book/Great Book - Part 2
Purchaser-DONE... respObj: success: /book/Great Book - Part 2
*/

Start at the bottom at the makePurchase() call and you should see that we’ve essentially modeled our story of a customer (purchaser) placing an order for two books. Should the purchase succeed or fail, the purchaser will only ever have to deal with the retailer directly (as expected), because the fact that deferred promises can sort of “bubble back up” the call chain. This keeps the interface quite clean, and the caller’s required knowledge of inner working to a minimum.

Conclusion

I’ve purposely kept the examples contrived so you can focus on the principles of deferred objects. If you’d like to see a variety of other example implementations, Julian Aubourg and Addy Osmani have written a wonderful article you should definitey have a look at. jQuery deferred objects take a bit to get used to, but provide some very compelling advantages for web applications that require heavy ajax or graphics.

However, after a lot of discussion we ended up scrapping this plan for a slightly different one.  I had already written a patch to make backbone.js routers support multiple route handlers, though, and I didn’t think it made sense to just throw it out, so I’ll provide it here.  Keep in mind that this may not work with future versions of backbone.js!

So, without further ado, here it is:

define(["require", "./underscore", "./backbone"], function(require, _, Backbone) {
    // Backbone.js doesn't quite do what we want it to, but it almost does;
    // we can modify it so that it does do what we want pretty easily, thanks
    // to the eminent hackability of javascript.
    //
    // We may need to update this when backbone.js updates! :-/


    var patched_methods = {
        // Save and rename the old route function, modified to work with the new
        override_route : function(route, callback) {
            this.handlers.unshift({route : route, callback : [callback]});
        },


        // Add a route to be tested when the fragment changes.
        // RB 9/15/2011: We don't want routes added later to
        //               override previous routes; we want
        //               routes added later to get called as well.
        //
        //               With this change, callback becomes an array
        //               of callbacks instead of just one.
        route : function(route, callback) {
            var appended = _.any(this.handlers, function(handler) {
                if (handler.route.toString() == route.toString()) {
                    handler.callback.unshift(callback);
                    return true;
                }
            });
            if (!appended) {
                this.handlers.unshift({route : route, callback : [callback]});
            }
        },


        // Add a feature to remove a route, since we no longer override them
        unroute : function(route, callback) {
            var found_and_removed = _.any(this.handlers, function(handler) {
                if (handler.route == route) {
                    var idx = handler.callback.indexOf(callback);
                    if (idx > -1) {
                        handler.callback.splice(idx, 1);
                        return true;
                    } else {
                        return false;
                    }
                }
            });
        },


        // Attempt to load the current URL fragment. If a route succeeds with a
        // match, returns `true`. If no defined routes matches the fragment,
        // returns `false`.
        // RB 9/15/2011: overriding this function
        loadUrl : function(fragmentOverride) {
            var fragment = this.fragment = this.getFragment(fragmentOverride);
            var matched = _.any(this.handlers, function(handler) {
                if (handler.route.test(fragment)) {
                    _.each(handler.callback, function(callback) {
                        callback(fragment);
                    });
                    return true;
                }
            });
            return matched;
        }
    };


    _.extend(Backbone.History.prototype, patched_methods);
    return Backbone;
});

I tried VMWare server for awhile, but found that every time I updated my system everything broke and I had to recompile drivers.

I tried VirtualBox in headless mode, which worked much better; still, it was hard to keep things running well since something always seemed to go wrong if the server got restarted for any reason.

Finally, I decided to invest in a machine to dedicate to the task. I did some research and decided on VMWare ESXi as the host platform — the free version is plenty for what I need.

Finding “Supported” Hardware

The list of supported hardware that I found was filled with things outside of my price range; okay, I admit it, I’m a bit cheap. I’ll save you the sob story of the time it took me to piece together a system that I thought might work and just give you the list of what I settled on. This is not a “supported” setup, but it does work (with a little tinkering with the setup file):

• $179.99 – G.SKILL 16GB of ram – I wanted a little more performance, so I got the faster set.
• 2 x $139.99 – Seagate Barracuda XT 2TB 7200 RPM SATA 6.0GB/s hard drive – My goal was to mirror these; that hasn’t turned out to be directly possible, but I have ideas for workarounds.
• $29.99 (+$9.99 shipping) Rosewill R101-P-BK Mid Tower case – Just about anything would work, but this case seems adequate so far
• $59.99 Rosewill Green Series 630W Power supply – Again, I wasn’t really picky, but this one looked decent and specifically says it works for i7 cpus
• $299.99 Intel Core i7-2600 3.4Ghz quad core CPU – Note that this is the 2600, not the 2600k. There are virtualization features missing on the 2600k.
• $139.99 Asus P8Q67-M LGA 1155 Intel Q67 Motherboard – This one is where it really got scary and dicy. The new CPUs (like the 2600, not the 2600k) support something called VT-D. Now, I don’t have this fully working yet, but the Q67 boards are supposed to support it as well; mine says it does, and I have enabled it, but I haven’t seen it work in VMWare yet, so I’m not 100% sure if this is working or not.  However, since most of the hardware needing support is on this board, which motherboard you get matters a lot.

Total cost of the system after shipping: $1,007.25

Installing VMWare ESXi

The first thing I tried, of course, was to install from the ISO I downloaded from VMWare.  Unfortunately, that gave me an ugly python error message; I forget exactly what it said. A little constructive googling turned up that it meant that ESXi couldn’t find a valid network card.  DRAT! I knew I was right to be worried about the motherboard choice; it has an Intel chipset, but the NIC is not supported by VMWare ESXi. I was quite put out and even started looking to see if any local computer stores had a NIC that was supported so I could get things going today. However, I found something better!

Customizing the install disk

As it turns out, there is a website (vm-help.com) for people who are running ESXi on hardware other than the “official supported” hardware. It took me awhile to find what I needed, but it turns out someone else has already had the issue I did (probably on a different motherboard) and created a driver customization archive (oem.tgz) with the drivers I need!

Here are the steps:

1. Download the latest VMWare ESXi install iso; the one I got was called “VMware-VMvisor-Installer-4.1.0.update1-348481.x86_64.iso”
2. Found and downloaded the oem.tgz file (2 rar files that I extracted) that supports the Intel 82579 Gigabit ethernet card
3. Followed these instructions to use the mkesxiaio shell script (on a linux machine) to customize my ISO
4. Burned the new ISO and installed

That was it! It works now! I did of course have to turn on all the virtualization features in the motherboard BIOS settings; I also set the SATA controller to AHCI mode. Really, though, everything seems to work perfectly and I’m now in the process of adding virtual machines.

Life is good =]

Yes, I wrote FireBreath. A lot of others have contributed.

A different approach to learning NPAPI

Now that I have a good example already up and running, it seems pointless to keep trying to describe things with words and code snippets when I can simply walk you through what I have done — what is in use and works well. To that end, and in hopes that it will foster more capable NPAPI developers to help with FireBreath development (we still don’t support printing, for example), I have created a series of 5 minute screencasts providing a brief overview of FireBreath’s NPAPI framework.

I don’t expect that this is as comprehensive as it should be; I’m hoping that comments will help me see what future segments I should create.  I have posted these screencasts on youtube and will link to them from here. You can also find a more complete series that includes this one on the FireBreath website.

I just finished setting up a build server on Mac OS X using Jenkins (formerly Hudson). The company I’m working for (GradeCam) uses git and gitolite for our source control and so I expected no trouble using Jenkins to build our tools using the git plugin.

However, I quickly ran into a snag: the source control server is on a public address and so our source code is not available except via ssh, and gitolite ssh access uses private key authentication.  Well, I’m an experience unix sysadmin, so that didn’t sound like a big issue — after all, setting up public key authentication is childs play, right?

Default install

The default installation of Jenkins on Mac OS X (at the time of this writing) installs a Launch Agent plist to /Library/LaunchAgents/org.jenkins-ci.plist. This plist file causes Jenkins to load as user “daemon”, which sounds fine — except that the home directory for the “daemon” user is /var/root, same as for user root.  This means that the .ssh dir in there will never have the right permissions for a private key to be used.

Creating a new hidden user

My solution was to create a new “hidden” user for Jenkins to run under. Following instructions I found on a blog post, I created a user “jenkins” with a home directory “/Users/Shared/Jenkins/Home”:

sudo dscl . create /Users/jenkins
sudo dscl . create /Users/jenkins PrimaryGroupID 1
sudo dscl . create /Users/jenkins UniqueID 300   
sudo dscl . create /Users/jenkins UserShell /bin/bash
sudo dscl . passwd /Users/jenkins $PASSWORD
sudo dscl . create /Users/jenkins home /Users/Shared/Jenkins/Home/

I then stopped Jenkins: “sudo launchctl unload -w /Library/LaunchAgents/org.jenkins-ci.plist” and edited the plist file to set the username to jenkins instead of daemon.  “chown -R jenkins: /Users/Shared/Jenkins/Home” sets the permissions how they need to be, and then “sudo launchctl load -w /Library/LaunchAgents/org.jenkins-ci.plist” should get you up and running!

To get git over ssh running, “sudo su – jenkins” to get a console as the jenkins user and set up the ssh keys and such. Make sure you can ssh to where you want to go (or even do a test git clone) because you need to save the keys so it doesn’t ask for them when jenkins tries to do the clone.

That should do you! Hope it helps someone.

Enter gitolite

Gitolite has a really cool feature that I make heavy use of called “wildcard repositories”. The way I use it, this lets me set up a group of users (@wildcardusers) that can create whatever arbitrary repositories they want in a subdirectory of the same name as their username.

In the gitolite.conf file:

@wildcard_users = richard jarom ben
 
repo    CREATOR/.*
    C     = @wildcard_users
    CRW+D = CREATOR
    RW    = WRITERS
    R     = READERS

Voila! I can now push to any repo looking like git@my_server.net:richard/my_repo.git

Setting the default remote HEAD

Ben and I happily used this for the better part of a day before we ran into our first disappointing snag. Ben is a huge fan (and I as well to a lesser extent) of git-flow, but with the layout we normally use with git-flow the default repository should be named “develop” instead of “master”.  Well, no problem, right?  A quick git checkout -b develop and we have a develop branch, then git branch -D master and the master branch is gone.  git push origin :master removes it from the gitolite server, and we’re ready to roll!

Oops! When we clone the repository, it clones okay but gives us an error at the end that it can’t find the HEAD ref. This is aparently a normal git configuration thing that has to be done on the server, but users who can create wildcard repositories shouldn’t need shell access on the box!

Fortunately, gitolite supports Admin Defined Commands (ADCs). They are suprisingly easy to use if you know a bit of shell script programming (or python, as I’ll explain later). As of a commit last night, there is now a set-head ADC in the contrib directory in the main repo.

ssh git@my_server.net set-head richard/my_repo.git refs/heads/develop

Simple as that, the HEAD is changed!  Note that ADCs have to be enabled and configured before you can use them.

Giving other users access to your repo

As you probably gathered earlier, this server is used by a small group of people who often work on projects together. Sometimes we need to give someone else access to our repository.  We might also want to just give access to anyone through the git:// protocol (which gitolite supports when configured correctly using the daemon user).

You can do this using the setperms and getperms built-in ADC.  However, these are a little inflexible for my tastes, and I had a few minutes, so I wrote a quick python helper class for writing ADCs and threw together the perms ADC, which is also in the main repo now.

You can run the command and see the help for yourself, but here is a quick example of what you can do:

richard@machine:~ $ ssh git@my_server.net perms
Usage: perms <command> <repository> <args>
             list <repository> [TYPE]
             clear <repository>
             add <repository> <TYPE> [user and group list]
             set <repository> <TYPE> [user and group list]
             remove <repository> <TYPE> [user and group list]
 
richard@machine:~ $ ssh git@my_server.net perms list richard/test.git
WRITERS:
    fred
READERS:
    ben
    jarom
richard@machine:~ $ ssh git@my_server.net perms add richard/test.git WRITERS billy
WRITERS:
    fred
    billy
richard@machine:~ $ ssh git@my_server.net perms add richard/test.git READERS joe
READERS:
    ben
    jarom
    joe
richard@machine:~ $ ssh git@my_server.net perms remove richard/test.git READERS jarom
READERS:
    ben
    joe
richard@machine:~ $ ssh git@my_server.net perms set richard/test.git READERS ben jarom
READERS:
    ben
    jarom
richard@machine:~ $ ssh git@my_server.net perms list richard/test.git
WRITERS:
    fred
    billy
READERS:
    ben
    jarom

I needed to find the ATL headers that should be used with the currently selected generator in my CMake project (FireBreath). The reason for this is that if the user is running Visual Studio Express edition, the ATL headers won’t be there so we need to look for the Windows DDK (Driver Development Kit) and use the headers from that.  If that isn’t there, we should throw an error.

Turns out that finding the path to the VC directory isn’t really that hard, but you have to know how =]  I found the registry paths (as well as how to use them in CMake) in the CMake source code.  Here is the snippet in hopes it will help someone:

if (MSVC10)
    GET_FILENAME_COMPONENT(VS_DIR [HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\VisualStudio\\10.0\\Setup\\VS;ProductDir] REALPATH CACHE)
elseif (MSVC90)
    GET_FILENAME_COMPONENT(VS_DIR [HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\VisualStudio\\9.0\\Setup\\VS;ProductDir] REALPATH CACHE)
elseif (MSVC80)
    GET_FILENAME_COMPONENT(VS_DIR [HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\VisualStudio\\8.0\\Setup\\VS;ProductDir] REALPATH CACHE)
endif()

This should work on Visual Studio 2005, 2008, and 2010. Other versions could be added if you care. FireBreath only supports 2005 on, so I didn’t bother. (Might actually work on older versions, but I haven’t tried). The path to the Visual Studio directory should end up in VS_DIR.

One cardinal rule of browser plugins is that you should never block the thread when processing a method or property call from Javascript.  In FireBreath, that means that any method or property on your JSAPI must never block, but should return in a timely manner.

The reason for this is that Javascript is effectively not a multi-threaded language. The way that Javascript handles setTimeout and setInterval varies, but from the perspective of your plugin, all JSAPI calls come in on the same thread — the UI thread. If you block on a JSAPI call the whole web browser will lock up until you finish.

In the programming circles that I hang out in, this is considered a Bad Thing.

However, there are often cases when you may need to perform some operation that is inherently blocking — such as searching the filesystem, processing a large file, or making a network request. There are doubtless several ways you could deal with this, but perhaps the easiest is to run the blocking operation on a new thread, and then call back into Javascript when you’re done.

If you think about it, this is why AJAX calls are all asynchronous.

FB::JSObjectPtr to the rescue!

One of the types supported by FireBreath is FB::JSObject. This represents any type of Javascript object that is passed in — so basically anything other than a primitive datatype. This includes arrays, raw objects (key : value stores), and functions as well as other things like DOM objects.

FB:JSObject has a method InvokeAsync that works the same way as Invoke would on a normal JSAPI object except that the call is asynchronous, so it doesn’t have a return value and thus doesn’t wait for the call to complete before returning. If we want, we can even pass back a reference to the current API object in the callback. To call a function (instead of a method on an object), we pass an empty string in for the method name when we call Invoke or InvokeAsync.

// Say we have a FB::JSObjectPtr callback
callback->InvokeAsync("", FB::variant_list_of(shared_from_this()));

Creating a thread

FireBreath relies on the boost thread library; this means that it is quite simple to create a new thread, particularly to run something minor. Keep in mind that my purpose isn’t to talk about proper techniques for multithreaded programming, so you’ll have to worry about things like making sure that your threads have all ended before shutting down and so forth on your own.

With boost thread, basically you just create a function that will be run on the other thread and use boost::bind to provide that method object (with a shared_from_this to the object instance to keep it from going away while the thread is running) to the thread constructor. Edit 1/19/2010: Someone rightly pointed out that using shared_from_this may not be (in fact usually is not) optimal. If you want to stop the thread when the JSAPI object goes away, you should pass “this” instead of shared_from_this. The code below has been updated to reflect that.

    boost::thread t(boost::bind(&MyPluginAPI::doSomethingTimeConsuming_thread,
         this, num, callback));

Putting it together

bool MyPluginAPI::doSomethingTimeConsuming( int num, FB::JSObjectPtr &callback )
{
    boost::thread t(boost::bind(&MyPluginAPI::doSomethingTimeConsuming_thread,
         this, num, callback));
    return true; // the thread is started
}
 
void MyPluginAPI::doSomethingTimeConsuming_thread( int num, FB::JSObjectPtr &callback )
{
    // Do something that takes a long time here
    int result = num * 10;
    callback->InvokeAsync("", FB::variant_list_of(shared_from_this())(result));
}

As you can see, when doSomethingTimeConsuming is called it accepts a callback. It creates a new thread and gives it the callback function.

Synchronous calls

It should also be noted that calls to Invoke on a FB::JSObject will also be threadsafe — they can be called from any thread. Keep in mind that a mutex and signal are being used behind to scenes to block until after the call is made to Javascript on the main thread, so there will be a performance hit for this.

It is an interesting thing to me that so many people seem to have a hard time understanding how the FireBreath windowing abstraction works, since to me it seems fairly clear. Of course, I wrote it, so that’s probably the reason =] There are a few things you should understand before you start trying to do your drawing on Windows:

PluginWindow

Every platform has a PluginWindow.  In fact, some platforms (most notably Mac) have several different types of PluginWindow depending on which drawing model and event model you use.  The PluginWindow is the source of all system events for the plugin.

In the default plugin skeleton generated by fbgen there are several Events mapped:

BEGIN_PLUGIN_EVENT_MAP()
    EVENTTYPE_CASE(FB::AttachedEvent, onWindowAttached, FB::PluginWindow)
    EVENTTYPE_CASE(FB::DetachedEvent, onWindowDetached, FB::PluginWindow)
END_PLUGIN_EVENT_MAP()

Then the matching handlers are:

virtual bool onWindowAttached(FB::AttachedEvent *evt, FB::PluginWindow *wnd);
virtual bool onWindowDetached(FB::DetachedEvent *evt, FB::PluginWindow *wnd);

FB::AttachedEvent will be fired each time a window is attached to the plugin.  FB::DetachedEvent will be fired each time a window is detached from the plugin.  NOTE: Though in practice each of these usually get fired only once per plugin instance, there is no guarantee in the browser contract that it won’t take away the window it gave you and give you a different one.

PluginWindowWin

The main plugin object is, by default, a platform agnostic object.  There are several ways that you can do your windows specific drawing:

1. Make your plugin object windows specific
   

   This is certainly the easiest way, though the least friendly to cross-platform development; you could just change the type of the event source specified in EVENTTYPE_CASE to FB::PluginWindowWin (as well as the type in the handlers) and you will have your object cast in the way you need it.

   Note that FB::PluginWindowWin is in

   #include "Win/PluginWindowWin.h"

2. Create a platform specific subclass of your plugin object
   

   This is actually fairly easy. Simply create an object (Such as MyPluginWin) in the Win/ directory that extends your main (MyPlugin) object and move Factory.cpp into your Win/ directory.  If you support other platforms, you’ll need to create a subclass and factory for each platform you want to support.  Then modify each Factory.cpp so that createPlugin returns the platform-specific plugin object.  Then you can make platform-specific event code to get your PluginWindowWin object.

3. Use #ifdef FB_WIN to have platform specific code in your plugin object
   

   Don’t do this.  It’s messy and ugly.  Still, you can if you want. To get a PluginWindowWin* from your PluginWindow* object, you can do:

   FB::PluginWindowWin *pwnd = wnd->get_as<FB::PluginWindowWin*>();

4. Create a platform specific object and pass the window and other information to it to draw
   

   Use the same method mentioned previously to convert the PluginWindow to a PluginWindowWin. This is the method that the BasicMediaPlayer example (found in examples/) uses.

Once you have your PluginWindowWin object, you can get the information you need to do the actual drawing with a few functions:

1. getHWND() – returns the HWND assigned to the browser plugin
2. getBrowserHWND() – returns the HWND of the top-level browser window
3. InvalidateWindow() – instructs the browser to invalidate the window and give you a RefreshEvent (fired when a WM_PAINT is received)

When to draw

Once you have the HWND, the next thing you need to know is when you can draw.  There are a few guidelines:

• Whenever a RefreshEvent is received, you must redraw. If you are using a secondary thread to draw, make sure you have some way of passing the message to that thread or you will get flickering.
• You can create your own rendering / drawing loop and draw whenever you want. In order to do this, you must create thread to handle the drawing; remember that in the instance of a DetachedEvent you should be able to tell this to stop drawing! If you choose to use something like OpenGL or D3D, you should (sometimes must) do all initialization and drawing on the same secondary thread.
• Remember that these are windowed plugins, which means that the plugin will “float” above everything else on the page.

Handling custom windows messages (your own WINPROC)

If you need to handle the windows messages directly rather than using the FireBreath abstractions, there are two ways to do this.

1. Extend PluginWindowWin and override the virtual function CustomWinProc
   

   This is the most direct way; CustomWinProc will get called for all unhandled messages.  If you don’t do anything but return false in HandleEvent, all messages will be unhandled. To override which object is created for the PluginWindow, implement the createPluginWindowWin method on your Factory to return your custom window class (which must extend PluginWindowWin)

2. Handle WindowsEvent
   

   This is the way that we recommend. This can be done by simply adding an EVENTTYPE_CASE line. All of the arguments given in an event loop are available as members of the WindowsEvent object. This makes it possible to pass the object through to the correct place without having a platform-specialized class, if desired.

FireBreath is flexible

One of the main design goals of FireBreath is to allow a user to extend it however they want to make it work how they are most accustomed; that is why there are so many options. Some restrictions are imposed externally by the nature of browser plugins. In all things, we recommend doing the simplest thing and only override existing classes if you need to — those are the solutions most likely to break with future versions.

As always, if you have questions or need help, feel free to drop into the IRC room!