Like many, the suggestion that JavaScript could be used in a fully architected system initially left me doubting the sanity of the person with whom I spoke. Over the years I have done several different types of Web Development; I am best known for my work with browser plugins in general and the FireBreath framework in particular. What most do not know is that I actually started out doing a lot of PHP and JavaScript development and that it was actually my work with JavaScript that eventually led me to learning about NPAPI and ActiveX plugin technologies.

Throughout all this I experimented with many different types of javascript code; I wrote AJAX applications long before I’d heard the term coined using a system called JSRS — which is surprisingly still around, or at least the page I got it from. I worked for Move Networks on their JavaScript SDK and I wrote several AJAX heavy websites, such as HamStudy.org. Many reading this will scoff when I admit that while I considered myself fairly well versed in JavaScript I still wrote it as if it were a simple scripting language. Others may wonder what is wrong with that? JavaScript more than perhaps any other language I have used gives you just enough rope to shoot yourself in the foot with. I have learned over the last few months just how little I knew — and didn’t know that I didn’t know.

My upward climb started off simply enough — I wanted to learn about some of the new technologies that I had heard about; Backbone, MVC frameworks, client-side template libraries… I had written my own “frameworks” on which to base my websites, all of which were side projects, but I found myself spending much more time than I liked updating and maintaining those frameworks. I thought perhaps I could use someone else’s system and save myself some trouble.

I purchased a book called JavaScript Web Applications which claimed to teach how to write an actual application with JavaScript. The table of contents looked interesting, and besides teaching about the principles used it also had instructions on several different MVC frameworks that I wanted to know about, including Backbone.

This book was the beginning — as a result of this purchase I soon came to grips with just how comparitively incomptetent my old code had been, and started to actually understand that JavaScript can indeed be a real language. Over the course of a few posts I would like to share with you the stack that has grown out of this system: What the components are, how they help, and why they are integral to the overall solution. I will start out in this post with the basics and my next post will detail the specific tweaks that we have made to them that have made our system so successful.

Dependency Management

Having been so long-winded with my earlier explanation, I will now spare you the details of all of the things I tried, researched, or experimented with during my search. One thing quickly became clear to me, though: Perhaps the #1 reason that my previous code had ended up so ugly was the need to avoid creating lots of JavaScript files. These can be difficult to deploy cleanly as it is not a good idea to have too many script includes in your production site and combining them can be an annoying process. The end result of this problem is that I had about 10 JavaScript files — and felt guilty about it — and all of my functionality was stuffed in those files. I had page-specific files, site-specific files, and general reuse library files, and all of them tried to register themselves cleanly in the global scope without clobbering each other. In short, it was a mess.

Enter require.js. Require.js is a system in which all javascript files are wrapped in a function. This is actually a really common pattern, but with require.js that function is passed into the “define” function call along with a list of dependencies. Require will first resolve these dependencies and then when they are all resolved it will call the function you provided, passing in the exports of each dependency as arguments in the order that the dependencies are listed in the define call.

The question left unanswered, then, is what are the exports of the other dependencies? Simply put, each “module” exports some value by simply returning that value from the function passed into the define call (See file1.js and file2.js).

define(["backbone", "./reqfile2"], function(Backbone, File2) {
    // In this file we do whatever it is we do; we know file2 was loaded first and we can use what it exports


    return File2.text + " from two different files";
});

define([], function() {
    var File2 = {};
    File2.text = "This is some text";
    return File2;
});

The natural result of so much flexibility is a lot of small files; this is really good for code organization because it makes it easy to seperate things into logical pieces and even to include whole modules consisting of multiple files (because one can depend on multiple others and combine them in its return value). The downside is that with lots of small files your page load time starts to get a bit out of control. Fortunately, require.js has an answer for this problem as well!

Require.js has an optimize phase that will examine the dependencies of your main file and combine all of them into a single file so that they are all loaded at once. The require.js system still makes sure that the functions are called in the correct order, but makes it very easy for your production site to be seamlessly updated to use a combined and minified single javascript file instead of the potentially dozens of small other files.

To me, this was a nearly life-changing WIN.

JavaScript Templates

If dependency management became the #1 most useful change in my architecture, a close #2 would have the be JavaScript Templates. The more I did things on the client the more client-side HTML I created in JavaScript; I experimented with several methods but still ended up with really hard to read html mixed in with my JavaScript as strings. When the time came to update one of those sites it was painful; not all of the code was in the same place, it wasn’t easy to read, it wasn’t easy therefore to change.

I considered different types of templates but had previously rejected the idea. The problem with templates, you see, is that you still have to include the template code somehow. The most “traditional” method would be to pretend you’re on a backend and just store them one template per file. Of course, then if you have any real number of these files you find yourself making a multitude of AJAX requests to download all of these templates; it’s clean from a file structure perspective and fairly easy to use, but to me the added complexity of all those requests is not acceptible. Other options include storing the template in javascript strings (same problem as I had before without templates) and storing them in a special script tag on the html page, which had the (to me) serious disadvantage of being nearly impossible to reasonably use on multiple pages. It was a problem.

Enter require.js and the text plugin. The text plugin of require.js is an idea both simple and brilliant; it allows you to specify files that should be loaded and the text of those files will be passed into your page function as the value of that dependency. With the text plugin you can save your templates each in their own file and have require load the text when the page loads. Of course, this doesn’t solve the problem we mentioned earlier! You still have potentially lots of AJAX requests being made to load those files. Problem? Not at all! Remember the optimize step in require.js; just like the normal dependencies are loaded and combined into one file the text plugin provides a mechanism to serialize those templates into javascript as strings and combine them into that same file. You get the best of both worlds! When you’re developing you have the flexibility of having each template in an easy-to-read file, but when you deploy those templates are packaged with your javascript.

define(["backbone", "underscore", "text!./mytemplate.jst"], function(Backbone, _, tplCode) {
    
    var template = _.template(tplCode);


    function doSomething(val) {
        alert("Here is the rendered template: ", tplCode({key: val}));
    }
    return doSomething;
});

Top all of this off with the fact that for several of the most popular template libraries there are already require.js plugins, such as requirejs-tpl. These plugins work like the text plugin, but they compile the template string into an actual template function before passing it back to resolve the dependency. Then, when you optimize, they actually compile your template code into javascript functions and minify them along with the rest of your JS code!

To me, this was a nearly life-changing WIN.

Deferred Objects

This may seem like it doesn’t fit with the other items on this page, but if templates and dependency management have revolutionized the organization of my code then deferred objects have revolutionized the way that I write JavaScript code. One of my coworkers at GradeCam, Robin, has already begun a small introductory series on how deferred objects work, so I’m not going to go into those details, but I do want to explain the conceptual shift in how pieces of the system interact that this led me to. It is both subtle and extremely powerful.

Most JavaScript operations anymore are asynchronous; in other words, they return immediately even though the operation in question hasn’t been completed. This can make it difficult to handle result cases; the traditional method is to pass a callback into the function to be called when the operation completes. You then perform some other operation once that function is called back and so forth.

The challenge with that is that it is a significant deviation from the patterns that we are all accustomed to. Granted, it’s important to understand what is actually happening, but often you may want to call a function in a place where you don’t actually want to handle the results; a good example is when you are requesting data from a model in your controller code, but you will actually be rendering it withing your view and your controller doesn’t really care what the results are. The view, of course, should not need to know or care where the data came from. In fact, it shouldn’t really need to know if the data is immediately available or will be available later on — it should display it when it is available and not worry about the other details.

Code speaks louder than prose, so here is an example of the difference between these two approaches; in both async1.js and async2.js three async operations take place. In async1.js they are handled in a traditional way, and in async2.js they are handled using jQuery.Deferred.

function asyncOper1(value, callback) {
    setTimeout(function() {
        callback("One second has passed since " + value);
    }, 1000);
}


function asyncOper2(value, callback) {
    setTimeout(function() {
        callback("Two seconds were added, then " + value);
    }, 2000);
}


function asyncOper3(value, callback) {
    setTimeout(function() {
        callback("A long time (3 seconds) went by before " + value);
    }, 3000);
}


function alertResult(value) {
    alert(value);
}


// Call them all; alert the final result
asyncOper1("we started", function(strVal1) {
    asyncOper2(strVal1, function(strVal2) {
        asyncOper3(strVal2, function(strVal3) {
            alertResult(strVal3);
        });
    });
});

function asyncOper1(valueDfd) {
    return $.when(valueDfd).pipe(function(value) {
        var dfd = new $.Deferred()
        setTimeout(function() {
            dfd.resolve("One second has passed since " + value);
        }, 1000);
        return dfd.promise();
    });
}


function asyncOper2(valueDfd) {
    return $.when(valueDfd).pipe(function(value) {
        var dfd = new $.Deferred()
        setTimeout(function() {
            dfd.resolve("Two seconds were added, then " + value);
        }, 2000);
        return dfd.promise();
    });
}


function asyncOper3(valueDfd) {
    return $.when(valueDfd).pipe(function(value) {
        var dfd = new $.Deferred()
        setTimeout(function() {
            dfd.resolve("A long time (3 seconds) went by before " + value);
        }, 3000);
        return dfd.promise();
    });
}


function alertResult(valueDfd) {
    $.when(valueDfd).done(function(value) {
        alert(value);
    });
}


var res1, res2, res3;


res1 = asyncOper1("we started");
res2 = asyncOper2(res1);
res3 = asyncOper3(res2);


alertResult(res3);

As you can see, the Deferred object method is slightly more complex at first; however, the complexity is where it belongs. Each method need know nothing about the others — they all assume that the input value *might* be Deferred. They return a Deferred object, and so they use the Deferred.pipe method, which makes it easy to chain deferred objects. Because each of the methods returns a Deferred object and is aware that it may receive a Deferred object, you can call one after the other as though they were happening synchronously — because indeed they will end up doing so, in all practical purposes. However, you shouldn’t think of this as hiding the fact that it is asynchronous, but rather as returning asynchronous return values.

Because we used $.when, you can pass a regular parameter in to any of those methods as well and chain them as much as you want. It will handle them all perfectly.

To me, this was a completely life-changing WIN.

This is just the beginning

I have a lot more to talk about here; this is just the foundation. If you know nothing more about our stack, you should understand these things. Next time, though, I’ll talk about how we tie this together using Backbone.js Routers, Views, and Models, how those tie into our templates, and how Deferred objects allowed us to add data cacheing to our data models completely transparently — without changing a line of code outside of the model classes.

However, after a lot of discussion we ended up scrapping this plan for a slightly different one.  I had already written a patch to make backbone.js routers support multiple route handlers, though, and I didn’t think it made sense to just throw it out, so I’ll provide it here.  Keep in mind that this may not work with future versions of backbone.js!

So, without further ado, here it is:

define(["require", "./underscore", "./backbone"], function(require, _, Backbone) {
    // Backbone.js doesn't quite do what we want it to, but it almost does;
    // we can modify it so that it does do what we want pretty easily, thanks
    // to the eminent hackability of javascript.
    //
    // We may need to update this when backbone.js updates! :-/


    var patched_methods = {
        // Save and rename the old route function, modified to work with the new
        override_route : function(route, callback) {
            this.handlers.unshift({route : route, callback : [callback]});
        },


        // Add a route to be tested when the fragment changes.
        // RB 9/15/2011: We don't want routes added later to
        //               override previous routes; we want
        //               routes added later to get called as well.
        //
        //               With this change, callback becomes an array
        //               of callbacks instead of just one.
        route : function(route, callback) {
            var appended = _.any(this.handlers, function(handler) {
                if (handler.route.toString() == route.toString()) {
                    handler.callback.unshift(callback);
                    return true;
                }
            });
            if (!appended) {
                this.handlers.unshift({route : route, callback : [callback]});
            }
        },


        // Add a feature to remove a route, since we no longer override them
        unroute : function(route, callback) {
            var found_and_removed = _.any(this.handlers, function(handler) {
                if (handler.route == route) {
                    var idx = handler.callback.indexOf(callback);
                    if (idx > -1) {
                        handler.callback.splice(idx, 1);
                        return true;
                    } else {
                        return false;
                    }
                }
            });
        },


        // Attempt to load the current URL fragment. If a route succeeds with a
        // match, returns `true`. If no defined routes matches the fragment,
        // returns `false`.
        // RB 9/15/2011: overriding this function
        loadUrl : function(fragmentOverride) {
            var fragment = this.fragment = this.getFragment(fragmentOverride);
            var matched = _.any(this.handlers, function(handler) {
                if (handler.route.test(fragment)) {
                    _.each(handler.callback, function(callback) {
                        callback(fragment);
                    });
                    return true;
                }
            });
            return matched;
        }
    };


    _.extend(Backbone.History.prototype, patched_methods);
    return Backbone;
});

I tried VMWare server for awhile, but found that every time I updated my system everything broke and I had to recompile drivers.

I tried VirtualBox in headless mode, which worked much better; still, it was hard to keep things running well since something always seemed to go wrong if the server got restarted for any reason.

Finally, I decided to invest in a machine to dedicate to the task. I did some research and decided on VMWare ESXi as the host platform — the free version is plenty for what I need.

Finding “Supported” Hardware

The list of supported hardware that I found was filled with things outside of my price range; okay, I admit it, I’m a bit cheap. I’ll save you the sob story of the time it took me to piece together a system that I thought might work and just give you the list of what I settled on. This is not a “supported” setup, but it does work (with a little tinkering with the setup file):

• $179.99 – G.SKILL 16GB of ram – I wanted a little more performance, so I got the faster set.
• 2 x $139.99 – Seagate Barracuda XT 2TB 7200 RPM SATA 6.0GB/s hard drive – My goal was to mirror these; that hasn’t turned out to be directly possible, but I have ideas for workarounds.
• $29.99 (+$9.99 shipping) Rosewill R101-P-BK Mid Tower case – Just about anything would work, but this case seems adequate so far
• $59.99 Rosewill Green Series 630W Power supply – Again, I wasn’t really picky, but this one looked decent and specifically says it works for i7 cpus
• $299.99 Intel Core i7-2600 3.4Ghz quad core CPU – Note that this is the 2600, not the 2600k. There are virtualization features missing on the 2600k.
• $139.99 Asus P8Q67-M LGA 1155 Intel Q67 Motherboard – This one is where it really got scary and dicy. The new CPUs (like the 2600, not the 2600k) support something called VT-D. Now, I don’t have this fully working yet, but the Q67 boards are supposed to support it as well; mine says it does, and I have enabled it, but I haven’t seen it work in VMWare yet, so I’m not 100% sure if this is working or not.  However, since most of the hardware needing support is on this board, which motherboard you get matters a lot.

Total cost of the system after shipping: $1,007.25

Installing VMWare ESXi

The first thing I tried, of course, was to install from the ISO I downloaded from VMWare.  Unfortunately, that gave me an ugly python error message; I forget exactly what it said. A little constructive googling turned up that it meant that ESXi couldn’t find a valid network card.  DRAT! I knew I was right to be worried about the motherboard choice; it has an Intel chipset, but the NIC is not supported by VMWare ESXi. I was quite put out and even started looking to see if any local computer stores had a NIC that was supported so I could get things going today. However, I found something better!

Customizing the install disk

As it turns out, there is a website (vm-help.com) for people who are running ESXi on hardware other than the “official supported” hardware. It took me awhile to find what I needed, but it turns out someone else has already had the issue I did (probably on a different motherboard) and created a driver customization archive (oem.tgz) with the drivers I need!

Here are the steps:

1. Download the latest VMWare ESXi install iso; the one I got was called “VMware-VMvisor-Installer-4.1.0.update1-348481.x86_64.iso”
2. Found and downloaded the oem.tgz file (2 rar files that I extracted) that supports the Intel 82579 Gigabit ethernet card
3. Followed these instructions to use the mkesxiaio shell script (on a linux machine) to customize my ISO
4. Burned the new ISO and installed

That was it! It works now! I did of course have to turn on all the virtualization features in the motherboard BIOS settings; I also set the SATA controller to AHCI mode. Really, though, everything seems to work perfectly and I’m now in the process of adding virtual machines.

Life is good =]

I just finished setting up a build server on Mac OS X using Jenkins (formerly Hudson). The company I’m working for (GradeCam) uses git and gitolite for our source control and so I expected no trouble using Jenkins to build our tools using the git plugin.

However, I quickly ran into a snag: the source control server is on a public address and so our source code is not available except via ssh, and gitolite ssh access uses private key authentication.  Well, I’m an experience unix sysadmin, so that didn’t sound like a big issue — after all, setting up public key authentication is childs play, right?

Default install

The default installation of Jenkins on Mac OS X (at the time of this writing) installs a Launch Agent plist to /Library/LaunchAgents/org.jenkins-ci.plist. This plist file causes Jenkins to load as user “daemon”, which sounds fine — except that the home directory for the “daemon” user is /var/root, same as for user root.  This means that the .ssh dir in there will never have the right permissions for a private key to be used.

Creating a new hidden user

My solution was to create a new “hidden” user for Jenkins to run under. Following instructions I found on a blog post, I created a user “jenkins” with a home directory “/Users/Shared/Jenkins/Home”:

sudo dscl . create /Users/jenkins
sudo dscl . create /Users/jenkins PrimaryGroupID 1
sudo dscl . create /Users/jenkins UniqueID 300   
sudo dscl . create /Users/jenkins UserShell /bin/bash
sudo dscl . passwd /Users/jenkins $PASSWORD
sudo dscl . create /Users/jenkins home /Users/Shared/Jenkins/Home/

I then stopped Jenkins: “sudo launchctl unload -w /Library/LaunchAgents/org.jenkins-ci.plist” and edited the plist file to set the username to jenkins instead of daemon.  “chown -R jenkins: /Users/Shared/Jenkins/Home” sets the permissions how they need to be, and then “sudo launchctl load -w /Library/LaunchAgents/org.jenkins-ci.plist” should get you up and running!

To get git over ssh running, “sudo su – jenkins” to get a console as the jenkins user and set up the ssh keys and such. Make sure you can ssh to where you want to go (or even do a test git clone) because you need to save the keys so it doesn’t ask for them when jenkins tries to do the clone.

That should do you! Hope it helps someone.

Enter gitolite

Gitolite has a really cool feature that I make heavy use of called “wildcard repositories”. The way I use it, this lets me set up a group of users (@wildcardusers) that can create whatever arbitrary repositories they want in a subdirectory of the same name as their username.

In the gitolite.conf file:

@wildcard_users = richard jarom ben
 
repo    CREATOR/.*
    C     = @wildcard_users
    CRW+D = CREATOR
    RW    = WRITERS
    R     = READERS

Voila! I can now push to any repo looking like git@my_server.net:richard/my_repo.git

Setting the default remote HEAD

Ben and I happily used this for the better part of a day before we ran into our first disappointing snag. Ben is a huge fan (and I as well to a lesser extent) of git-flow, but with the layout we normally use with git-flow the default repository should be named “develop” instead of “master”.  Well, no problem, right?  A quick git checkout -b develop and we have a develop branch, then git branch -D master and the master branch is gone.  git push origin :master removes it from the gitolite server, and we’re ready to roll!

Oops! When we clone the repository, it clones okay but gives us an error at the end that it can’t find the HEAD ref. This is aparently a normal git configuration thing that has to be done on the server, but users who can create wildcard repositories shouldn’t need shell access on the box!

Fortunately, gitolite supports Admin Defined Commands (ADCs). They are suprisingly easy to use if you know a bit of shell script programming (or python, as I’ll explain later). As of a commit last night, there is now a set-head ADC in the contrib directory in the main repo.

ssh git@my_server.net set-head richard/my_repo.git refs/heads/develop

Simple as that, the HEAD is changed!  Note that ADCs have to be enabled and configured before you can use them.

Giving other users access to your repo

As you probably gathered earlier, this server is used by a small group of people who often work on projects together. Sometimes we need to give someone else access to our repository.  We might also want to just give access to anyone through the git:// protocol (which gitolite supports when configured correctly using the daemon user).

You can do this using the setperms and getperms built-in ADC.  However, these are a little inflexible for my tastes, and I had a few minutes, so I wrote a quick python helper class for writing ADCs and threw together the perms ADC, which is also in the main repo now.

You can run the command and see the help for yourself, but here is a quick example of what you can do:

richard@machine:~ $ ssh git@my_server.net perms
Usage: perms <command> <repository> <args>
             list <repository> [TYPE]
             clear <repository>
             add <repository> <TYPE> [user and group list]
             set <repository> <TYPE> [user and group list]
             remove <repository> <TYPE> [user and group list]
 
richard@machine:~ $ ssh git@my_server.net perms list richard/test.git
WRITERS:
    fred
READERS:
    ben
    jarom
richard@machine:~ $ ssh git@my_server.net perms add richard/test.git WRITERS billy
WRITERS:
    fred
    billy
richard@machine:~ $ ssh git@my_server.net perms add richard/test.git READERS joe
READERS:
    ben
    jarom
    joe
richard@machine:~ $ ssh git@my_server.net perms remove richard/test.git READERS jarom
READERS:
    ben
    joe
richard@machine:~ $ ssh git@my_server.net perms set richard/test.git READERS ben jarom
READERS:
    ben
    jarom
richard@machine:~ $ ssh git@my_server.net perms list richard/test.git
WRITERS:
    fred
    billy
READERS:
    ben
    jarom

I needed to find the ATL headers that should be used with the currently selected generator in my CMake project (FireBreath). The reason for this is that if the user is running Visual Studio Express edition, the ATL headers won’t be there so we need to look for the Windows DDK (Driver Development Kit) and use the headers from that.  If that isn’t there, we should throw an error.

Turns out that finding the path to the VC directory isn’t really that hard, but you have to know how =]  I found the registry paths (as well as how to use them in CMake) in the CMake source code.  Here is the snippet in hopes it will help someone:

if (MSVC10)
    GET_FILENAME_COMPONENT(VS_DIR [HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\VisualStudio\\10.0\\Setup\\VS;ProductDir] REALPATH CACHE)
elseif (MSVC90)
    GET_FILENAME_COMPONENT(VS_DIR [HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\VisualStudio\\9.0\\Setup\\VS;ProductDir] REALPATH CACHE)
elseif (MSVC80)
    GET_FILENAME_COMPONENT(VS_DIR [HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\VisualStudio\\8.0\\Setup\\VS;ProductDir] REALPATH CACHE)
endif()

This should work on Visual Studio 2005, 2008, and 2010. Other versions could be added if you care. FireBreath only supports 2005 on, so I didn’t bother. (Might actually work on older versions, but I haven’t tried). The path to the Visual Studio directory should end up in VS_DIR.