NOTE: There is now a relevant answer on StackOverflow. Unfortunately, this answer wasn’t there when I needed it.

The problem

Recently, while adding iCloud support to the Day One iOS app, I ran into an issue copying files to and from the application’s Documents directory and its corresponding iCloud directory. Day One can run in one of three modes:

1. Sync entries via iCloud
2. Sync entries via Dropbox
3. Don’t sync entries

If you’ve developed for iCloud at all, you know that iCloud documents live within specific, “ubiquitous” directories on the user’s device (discovered via NSFileManager’s URLForUbiquityContainerIdentifier: method). So in order to enable iCloud sync, we needed to copy all files into the appropriate ubiquitous container. Conversely, if the user elects to disable iCloud then we need to copy all files from the ubiquitous container into the application’s Documents directory.

The assumption

We wrote a simple method to copy files from one directory to another, either going into or coming out of iCloud. It uses a basic NSDirectoryEnumerator to find all the files in the current source directory, check if they are present in the destination directory, and copy them there if they aren’t. Here’s a simplified example:

NSURL *sourceDir = [self currentWorkingDir]; // current directory, ie ubiquitous iCloud directory
NSURL *destDir = [self destDirForMovingFromDir:sourceDir]; // where should we copy to?
NSFileManager *fm = [NSFileManager defaultManager];
NSDirectoryEnumerator *dirEnum = [fm enumeratorAtPath:[sourceDir path]];
 
NSError *mergeError;
while ((sourceFile = [dirEnum nextObject])) {
    NSURL *sourceFileURL = [sourceDir URLByAppendingPathComponent:sourceFile];
    NSURL *destFileURL = [destDir URLByAppendingPathComponent:[sourceFile lastPathComponent]];
    if (![fm fileExistsAtPath:[destFileURL path]]) {
        if (![fm copyItemAtURL:sourceFileURL toURL:destFileURL error:&mergeError]) {
            NSLog(@"ERROR (copy error): %@ -> %@ - %@", sourceFileURL, destFileURL, mergeError);
        }
    }
}

I frequently found “bad file descriptor” errors being logged out. We were assuming that the NSDirectoryEnumerator would only list files that were actually present on the device. As it turns out, that is not the case when enumerating a ubiquitous directory.

The solution

Once you realize that the NSDirectoryEnumerator will list all ubiquitous files that exist on any device, regardless of whether or not they have been downloaded to the current device, the solution becomes pretty obvious:

You can’t copy a file that hasn’t been downloaded.

Adding that check is fairly straight forward. Since our method doesn’t know if we are going into or out of iCloud, we implemented a pretty general solution. Modifying the above example, it looks something like this:

...
while ((sourceFile = [dirEnum nextObject])) {
    ...
    // By default, assume all files are able to be copied.
    NSNumber *ableToBeCopied = [NSNumber numberWithBool:YES];
 
    // However, ubiquitous items that are not downloaded CANNOT be copied
    if ([fm isUbiquitousItemAtURL:sourceFileURL]) {
        [sourceFileURL getResourceValue:&ableToBeCopied forKey:NSURLUbiquitousItemIsDownloadedKey error:nil];
    }
 
    // Now copy the file if it doesn't exist and is able to be copied
    if (![fm fileExistsAtPath:[destFileURL path]] && [ableToBeCopied boolValue]) {
        if (![fm copyItemAtURL:sourceFileURL toURL:destFileURL error:&mergeError]) {
            NSLog(@"ERROR (copy error): %@ -> %@ - %@", sourceFileURL, destFileURL, mergeError);
        }
    }
}

Conclusion

See what I mean? Pretty obvious, and I can’t believe I lost so much time tracking this down. If the source file is ubiquitous, it might not be downloaded, and thus you might not be able to copy it. So check it first.

As you recall from the first part jQuery Deferred Objects – Part I, deferred objects give you a way to organize asynchronous blocks of code in to a self-managed callback queue, while allowing you to attach 1..* callbacks. Also, you’ll recall that the basics of using jQuery deferred objects include essentially the following “simplified” steps:
1. Created a deferred object
2. Attached .done and .fail callbacks
3. Called resolve or reject, thereby triggering the appropriate respective callbacks

Promises

This was fine as a first step in understanding what’s going on, but in all likelyhood you’ll use a slightly different approach. What’s wrong with returning a deferred object itself? It gives the caller access to resolve or reject directly, and that means other callers could be affected as a result. What we really want to do is return a “promise”. From the jQuery Promise Documentation we learn:
“This object provides a subset of the methods of the Deferred object (then, done, fail, always, pipe. isResolved, and isRejected) to prevent users from changing the state of the Deferred.”

$.when(
    $.ajax( "/push_to_twitter" ),
    $.ajax( "/push_to_facebook" ),
    $.ajax( "/push_to_linkedin" ),
    $.ajax( "/generate_report" )
).then( it_succeeded, it_failed);
 
function it_succeeded(report_data){
    // Now do something with the report data e.g. show
    // an awesome summary with pretty bars and graphs!
}
function it_failed(){
    // Treat user's depression, OCD and other ailments
    // Prescribe prozac, etc., etc...
}

Possible Uses

Possible use cases are endless but here are some examples:
1. You need to make multiple asynchronous Ajax requests – with Deferred objects you can handle the results together without concern for which order they finish in.
2. You want to be able to make the call in one region of code, then pass the result as an argument to a different part, but the call may be asynchronous.
3. Easily return data from a modal dialog
4. Caching – an API properly supporting Deferred objects can implement caching without any changes to the consumer of the API.

An Example: Caching

For caching, jQuery’s when .. then provides us the tools we need to return both fresh and cached data from one interface. Take the following code:

var cacheA = {};
function fetch_some_objectA(key){
    if (cacheA[key]) { return cacheA[key]; }
    return cacheA[key] = $.ajax('/fooEndpoint', {
        // ajax options here
    }).done(function(res){
        cacheA[key] = res;
    });
}
var cacheB = {};
function fetch_some_objectB(key){
    if (cacheB[key]) { return cacheB[key]; }
    return cacheB[key] = $.ajax('/barEndpoint', {
        // ajax options here
    }).done(function(res){
        cacheB[key] = res;
    });
}
 
$.when(fetch_some_objectA('foo'), fetch_some_objectB('bar').done(function(dataA, dataB){
    // Do whatever you want with said data
}).fail(function(errA, errB) {
    // Deal with error handling
});

The basics of caching with Deferred objects

There are a couple of cases to consider here; the first case is that you are requesting object(‘foo’) for the first time. In this case, cacheA["foo"] will not exist and so an ajax call is created. $.ajax returns a deferred object, which is saved in the cache. The reason this is important is that thereafter, future requests will get that same deferred object, thus preventing two requests from being made to the same endpoint. Once the ajax call completes the done handler we set updates the cache with the actual object. Any calls that ran fetchsomeobjectA(‘foo’) while the request was “in progress” will be notified with the data once the deferred is resolved. Calls made after the request has resolved will get the actual data straight away. So, how do you tell if you have a deferred or the real data? The answer is simple: you don’t need to.

This is where jQuery.when neatly ties everything together. jQuery.when ($.when) accepts any number of arguments and returns a jquery.Deferred object. That object will be resolved once all of the arguments are resolved; however, if those objects are not jQuery.Deferred objects then they will be considered resolved immediately. From the documentation, jQuery.when takes either a Deferred or object: “If a single argument is passed to jQuery.when and it is not a Deferred, it will be treated as a resolved Deferred and any doneCallbacks attached will be executed immediately.”

After passing through a jQuery.when, a Deferred object that resolves to a string “A” and the string “A” become functionally equivalent — no matter which you passed in, you’ll still get the same thing:

$.when("fdsafdsa").done(function(var) { alert(var); });

This will immediately alert “fdsafdsa”. Now essentially the same using the deferred resolve idiom:

var dfd = new $.Deferred();
$.when(dfd).done(function(var) { alert(var); });
dfd.resolve("fdsafdsa");

This will also alert the same thing; it doesn’t matter if it’s a real object or a deferred, once you make it into the when it’ll be a real object. This makes caching really slick because you don’t need to care if you have the object yet or not, you just let jQuery.when figure that out and tell it what should happen once it does.

One small problem…

If you actually copy in this code and play with it long enough, you’ll notice something that doesn’t quite work how you expect. When the jQuery.ajax call returns you’ll actually get a slightly different object in the jQuery.when callback; this actually makes sense if you look at it closer. From the jQuery codebase, when the ajax call completes it is either resolved or rejected with the following lines:

// Success/Error
if ( isSuccess ) {
    deferred.resolveWith( callbackContext, [ success, statusText, jqXHR ] );
} else {
    deferred.rejectWith( callbackContext, [ jqXHR, statusText, error ] );
}

Notice that there are actually three arguments in either case; this may surprise you, since most examples you’ll see of the callbacks accept just one, like the above code. Most don’t need the other two, but jQuery.when isn’t going to discard those extra values and yet it still needs to keep one argument per input in the callback function. Therefore, in the $.when block above you’ll actually have both dataA and dataB resolve to an array with three values (success, statusText, jqXHR); however, we normally would only cache one of those values (success) so when reading from the cache we don’t get that array. The solution is to make our functions a little bit more complicated by using the jquery pipe method to restructure the result a little bit:

var cacheA = {};
function fetch_some_objectA(key){
    if (cacheA[key]) { return cacheA[key]; }
    return cacheA[key] = $.ajax('/fooEndpoint', {
        // ajax options here
    }).pipe(function(res){
        return cacheA[key] = res;
    });
}
var cacheB = {};
function fetch_some_objectB(key){
    if (cacheB[key]) { return cacheB[key]; }
    return cacheB[key] = $.ajax('/barEndpoint', {
        // ajax options here
    }).pipe(function(res){
        return cacheB[key] = res;
    });
}
 
$.when(fetch_some_objectA('foo'), fetch_some_objectB('bar').done(function(dataA, dataB){
    // Do whatever you want with said data
}).fail(function(errA, errB) {
    // Deal with error handling
});

You may notice the assignment of the result to the cache happens twice. The reason is that you first want to assign the deferred object to the cache when you start the request — otherwise if you call the function again before the ajax request finishes it’ll start a new request rather than reusing the old one.

With this new object (the result of the .pipe) the resolved value will be as we expect it. This relatively simple pattern can get a little more complex if you’re doing a lot with pipe, but for the consumer the interface is really simple.

Admittedly, there is some complexity to using this pattern but the “win” is that your interface now has a “Single Point of Contact” (SPOC)
Let’s look at a convenient analogy…

An Example: Placing a Special Order

Let’s imagine that, as an active supporter of your local community businesses, you make it a point to go to your favorite little “mom and pop” bookstore to place orders for books. The lady at the counter asks for your phone number or email, and says she’ll contact you as soon as your books come in. Now obviously, there’s the small chance that, after the fact, the wholesaler, manufacturer, etc., will have ran out of the ordered book(s). Should this happen, you’d probably feel it was acceptable for the bookstore to contact you and give you the option to wait or cancel your order. However, you would not want to here directly from the wholesaler or manufacturer directly. This would probably confuse you and you may even complain to the bookstore that they should manage their purchases better should this happen! The expectation you have as a customer in this example is that you only have to deal with a single point of contact. In code, we can have a similar convenience using deferred objects:

// Contrived ajax and random failure stubs so we can
// pretend to have "mostly successful" requests
function randomTrueFalse() {
    Math.random();
    var r = Math.round(Math.random()*10);
    return (r%9 !== 0);
};
 
function ajaxStub(endpoint) {
    var dfdResult = new $.Deferred();
    var bool = randomTrueFalse();
 
    setTimeout(function() {
        if(bool) {
            dfdResult.resolve('success: '+endpoint);
        }
        else {
            dfdResult.reject('fail: '+endpoint);
        }
    }, 100);
    return dfdResult.promise();
};
 
var manufacturer = (function() {
    return {
 
        orderBook: function(bookName) {
            return $.when(ajaxStub('/book/'+bookName)).pipe(function(respObj) {
                console.log("Manufacturer-DONE: ", respObj);
                return respObj;
            }, function(errObj) { console.log("Manufacturer-FAIL: ", errObj); return errObj; });
        }
    };
}());
 
var wholesaler = (function() {
    var m = manufacturer;
    return {
        orderBook: function(bookName) {
            return $.when(m.orderBook(bookName)).pipe(function(respObj) {
                console.log("Wholesaler-DONE: ", respObj);
                return respObj;
            }, function(errObj) { console.log("Wholesaler-FAIL: ", errObj); return errObj; });
        }
    };
}());
 
var retailer = (function() {
    var w = wholesaler;
    return {
        orderBook: function(bookName) {
            return $.when(w.orderBook(bookName)).pipe(function(respObj) {
                console.log("Retailer-DONE: ", respObj);
                dfdResult.resolve(respObj);
            }, function(errObj) { console.log("Retailer-FAIL: ", errObj); return errObj; });
        }
    };
}());
 
// Notice that, although we have multiple potential points of failure above (e.g.
// Manufacturer, Wholesaler, Retailer, the customer only needs to deal with the
// retailer directly.)
function makePurchase() {
 
    // Place an order for two books
    for(var i=1; i<3; i++) {
        $.when(retailer.orderBook("Great Book - Part " + i)).done(function(respObj) {
            console.log("Purchaser-DONE... respObj: ", respObj);
        }).fail(function(err) {
            console.log("Purchaser-FAIL... err: ", err);
        });
    }
};
makePurchase();
 
/*
Random output for placing order of two books might look like:
Manufacturer-FAIL: fail: /book/Great Book - Part 1
Wholesaler-FAIL: fail: /book/Great Book - Part 1
Retailer-FAIL: fail: /book/Great Book - Part 1
Purchaser-FAIL... err: fail: /book/Great Book - Part 1
Manufacturer-DONE: success: /book/Great Book - Part 2
Wholesaler-DONE: success: /book/Great Book - Part 2
Retailer-DONE: success: /book/Great Book - Part 2
Purchaser-DONE... respObj: success: /book/Great Book - Part 2
*/

Start at the bottom at the makePurchase() call and you should see that we’ve essentially modeled our story of a customer (purchaser) placing an order for two books. Should the purchase succeed or fail, the purchaser will only ever have to deal with the retailer directly (as expected), because the fact that deferred promises can sort of “bubble back up” the call chain. This keeps the interface quite clean, and the caller’s required knowledge of inner working to a minimum.

Conclusion

I’ve purposely kept the examples contrived so you can focus on the principles of deferred objects. If you’d like to see a variety of other example implementations, Julian Aubourg and Addy Osmani have written a wonderful article you should definitey have a look at. jQuery deferred objects take a bit to get used to, but provide some very compelling advantages for web applications that require heavy ajax or graphics.

Given the asynchronous nature of the web, and specifically JavaScript, a general asynchronous callback pattern has proliferated. So you have something you need to defer a bit so you use a timeout or an interval and you’re good to go…what’s the issue you may be asking? Well…web applications have become more and more complex, and what was once a nested callback or two, is turning into a hairy beast of nested callbacks.

The Deferred pattern addresses this nested nightmare by using using an idiom much like publish/subscribe (Observer) or Notifications (if you come from the land of iOS development). However, notification is a bit different because the caller has access to the Deferred object itself and can determine whether to call success or fail callbacks. This is sort of a forward reference, but will hopefully be clearer further down in this article. That said, let’s dig in…

If you’re familiar with stacks and queues then the underlying concept should be pretty straight forward. Essentially, the Deferred object implementation keeps a stack of callback function objects (similar to subscribers) for success or failure. Conventionally, these are called done callbacks or fail callbacks. Specifically, these function objects get pushed on the Deferred object when you call .then(myfunc), .done(myfunc), or .fail(myfunc):

var deferred = new Deferred();
deferred.done(function(message) {
  alert(message);
});
setTimeout(3000, function() {
  deferred.resolve("Hey buddy!");
});

First we create a deferred object. Immediately, the deferred object goes in to the “Pending” state.

Next, and here’s the important part to “grok”, by calling .done(… on the Deferred object, we are essentially asking it to add our function object to its array of callback functions.

Ok, so now we have 1..* functions queued (or stacked depending on implementation details) on the deferred object. Now one of two things will happen (at some point):
1. We call deferred.resolve (or one of it’s variants)
2. We call deferred.reject

If we do 1., this will “trigger” the deferred object to call all of the queued resolve category functions. These are the .done, .then, variant callbacks.

If we do 2., this will “trigger” the deferred object call all of the queued fail variant functions.

There are some semantic details to all of this, but that drives to the essence of the pattern. If you’re still not clear, I’ve drawn this extremely ugly diagram to help to visualize the essence of the deferred pattern (as described in both the jQuery API docs and CommonJS Promises page). Open your browser wide and start on the left and work to the right:

Cool! So how is this Deferred object doing it’s magic internally? Here’s a simplified example of how it may be implemented:

 
<h1>JavaScript Deferred Objects Example</h1>
 
function Deferred() {
    var callbacks = [];
    this.done =  function(cb) {
        callbacks.push(cb);
    }
 
    this.resolve = function(arg) {
        for(var i = 0; i < callbacks.length; i++) {
            callbacks[i](arg);
        }
    }
}
 
// Now we use the deferred object
var deferred = new Deferred();
deferred.done(function(message) {
  alert(message);
});
setTimeout(function() {
  deferred.resolve(&quot;Hey buddy!&quot;);
}, 3000);

What’s interesting about this pattern is that the owner of the deferred object determines whether we go in to the resolved or rejected state not the deferred object itself. This is slightly different from how a traditional Pub/Sub pattern would work, where the Publishing object itself would likely determine who/when to callback it’s subscribers. Because of this, it’s common to see usage where an intermediary function that is supplying some service (e.g. fetchResource, hitDatabase, etc.), will negotiate the initialization of the deferred object, determine whether to call resolve or reject, etc., and simply return the deferred object. So it looks like:

The above diagram only shows part of the picture. Once the Broker object calls resolve or reject, the original caller’s respective .done, .fail, etc., will get called.

All of this is possible because of JavaScript’s ability to treat functions as first class citizens (a fancy way of saying you can pass functions around and call them later and treat them like good ol’ objects!).

I hope this posting was somewhat helpful and revealed the magic behind Deferred Objects.

EDIT

My good friend Richard Bateman offers a point of clarification. There is actually a difference between .then and .done; .then allows you to provide two callbacks, the first for done and the second for fail.

Next part

For more on jquery deferred objects, see part 2 of this series.

• TotalTerminal has replaced Visor
• Mac OS X Lion has been released

TotalTerminal changes up the way it launches Terminal — instead of being automatically injected via SIMBL, it manually injects the plugin into Terminal.app. Well, I wasn’t going to let this deter me.

Getting back to Nirvana after installing TotalTerminal is not all that bad. Most of the steps haven’t changed from my original post so I will only go into detail about the new steps here. Briefly, here are the steps I followed.

1. Remove TerminalColours.bundle SIMBL plugin — per the TotalTerminal FAQ, it injects its own version and having TerminalColours.bundle installed creates a conflict. (~/Library/Application\ Support/SIMBL/Plugins)
2. Fix the Home / End keys (see original post)
3. Copy Terminal.app to VisorTerminal.app (see original post)
4. Install TotalTerminal
5. Modify TotalTerminal to use VisorTerminal.app instead of Terminal.app

That’s right, there’s really only one thing to change, and it’s very simple

TotalTerminal.app has a simple, compiled AppleScript which launches Terminal and injects itself. The compiled file is located at /Applications/TotalTerminal.app/Contents/Resources/Scripts/main.scpt. But, we don’t want the compiled binary version, we want the original so we can make it use VisorTerminal.app.

Fortunately the uncompiled source file is available in the TotalTerminal repo on Github.

Only one change is necessary.

--- TotalTerminal.app/Contents/Resources/Scripts/main.applescript
+++ TotalTerminal.app/Contents/Resources/Scripts/main.applescript
@@ -1,4 +1,4 @@
-tell application "Terminal"
+tell application "VisorTerminal"
        delay 1 -- this delay is important to prevent random "Connection is Invalid -609" AppleScript errors
        try
                event BATTinit

Once that is done, you just need to compile the script and replace the one that shipped with TotalTerminal.

$ osacompile -o main.scpt main.applescript
$ mv main.scpt /Applications/TotalTerminal.app/Contents/Resources/Scripts

That’s it.

You should be good to go. Put TotalTerminal.app into your user startup applications if you wish (I do), or launch it manually by launching TotalTerminal.app.

Enjoy the restoration of terminal nirvana.

For the lazy…

My modified and compiled main.applescript is available for download.

We needed something easier. We did not want to go to a workflow where there is one git user on the system, every employee authenticates with their own public key to that user, and everyone with a key has access to everything. We wanted more granular control, but also an easier management workflow. While gitolite does use a single system user under the covers, it provides per-user access control (governed by public keys), easy repo management, and a couple other nice features like wildcard repositories and branch-level access restrictions.

However, one of our developers is in the habit of accessing a random server, cloning a repo (using his password), and pushing from environments other than his development environment. Obviously, his private key is not going to be on each of the machines he does this from. Gitolite does not allow password-based access for obvious reasons — it needs to know who you are to determine permissions — so gitolite was cramping his style.

There are a couple ways around this:

1. Add a public key for each system he pushes changes from. Since he often logs in as “administrator” or “root” on individual systems, this obviously is not a good solution.
2. Have the developer enable SSH Agent Forwarding from his machine. This works well, as long as the user does not chain sessions (connect to server A, from there to server B, and clone on B) or is willing / able / allowed to enable Agent Forwarding on each system in the chain. (NOTE: on Mac OS X Snow Leopard, enabling SSH Agent Forwarding is as simple as running ssh-add and adding two lines to your ~/.ssh/config file:

   # add key to agent (I do this in ~/.profile)
   ssh-add $HOME/.ssh/id_rsa
    
   # ~/.ssh/config
   Host *
       ForwardAgent yes

3. Come up with something else.

What we really wanted was a way to allow password-based access to the repositories. Since gitolite does not support this itself, we had to come up with another way. As it turns out, doing so is not too bad, though it does involve a little bash shell black magic.

What we came up with is creating a system user on the git server for each developer who wants password access to git repositories, using a custom script as their defined shell (instead of /bin/sh or /bin/bash). Don’t forget to make your new script executable.

This script needs to look at what the user is trying to do (git action, gitolite ADC, some random shell command, nothing…) and act accordingly. Here’s the script we came up with, which will be dissected below to explain what each part does in more detail than the inline doc (gist).

#!/bin/bash
shift # get rid of -c
 
# if no commands, just open a shell
if [[ $# -eq 0 ]]; then
        /bin/bash -l
 
# if the first arg is a git- command, that means it is something like git-push, etc... so forward it
elif [[ $1 == git-* ]]; then
        ssh -q -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no git@localhost $*
 
# if the first arg is SET_ENV_ONLY, we sourced this file in order to set up the gitolite function
elif [[ $1 == "SET_ENV_ONLY" ]]; then
        gitolite () {
                ssh -q -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no git@localhost $*
        }
 
# if there is at least one non-git command, source this file in a new shell and create the 'gitolite' function
else
        /bin/bash -c "source $0 shiftme SET_ENV_ONLY; $*"
fi

If the user is not trying to do anything (just wants to open a shell), then there are no arguments and we should just open a bash prompt:

if [[ $# -eq 0 ]]; then
        /bin/bash -l

Okay, the simple bit is out of the way. Now on to the black magic.

If the user is trying to perform a git command (via git push, etc.) then we create a new SSH session to localhost (remember, this is running on the same server as gitolite) and pass along the commands ($*). Since the user has already logged in, this ssh command will run from their account using their private / public key. The git commands are things like git-receive-pack, git-upload-pack, etc, so we match the first script argument against “git-”.

Finally, we don’t care about validating the host’s fingerprint (we are already on the host) and there can be no man in the middle (since we are going to localhost), so we can turn off StrictHostKeyChecking. We also don’t want to print out the various fingerprint messages that SSH will generate when you ignore host keys, so we make the new SSH connection quiet. That’s it for this block:

elif [[ $1 == git-* ]]; then
        ssh -q -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no git@localhost $*

I am going to go over the last two pieces in reverse order, because it is easier to follow.

If the user sent some SSH commands that were not native git commands, we need to process those commands. So, we will execute the remaining commands in a new bash process.

BUT, what if the user is trying to perform a gitolite command (such as expand or an ADC), what should we do? We can’t just trust that all commands the user sends will either be a git command or a gitolite command, so we need to set up some more black magic to handle that for us. So the first thing we will do when we open a new bash process to handle the remaining commands is source this custom script ($0) with a custom value we made up (SET_ENV_ONLY). That way, if this script executes again we will know it was done by ourselves and we can set up some more magic.

else
        /bin/bash -c "source $0 shiftme SET_ENV_ONLY; $*"

Finally for the last block.

If our new script is being executed, and the first argument is SET_ENV_ONLY (after shifting), then it is because we sourced it from our new bash process from the previous step. What we do now is set up a gitolite command which does something very similar to what happens if the user is performing a native git command: it opens an SSH connection to git@host and performs the gitolite commands.

After this has been done, “gitolite expand” is the same as “ssh git@git.company.com expand”. This makes it so the developer can still run gitolite commands and ADCs, he just has to prefix each command with gitolite. Something like this:

$ ssh [DEVELOPER]@git.company.com "gitolite expand; echo $PATH; gitolite info"

is equivalent to this:

$ ssh git@git.company.com expand
$ ssh [DEVELOPER]@git.company.com echo $PATH
$ ssh git@git.company.com info

Now you just need to create accounts that use this script as their shell. Each user can add whatever public keys they want to their authorized_keys file, or they can authenticate solely with their password. The only requirement is that their server account has a private / public key generated, and that the public key is added to gitolite.

$ ssh root@git.company.com
$ useradd -s /usr/local/bin/custom_gitbash.sh -m [DEVELOPER]
$ su [DEVELOPER]
$ ssh-keygen -t rsa
$ cp ~/.ssh/id_rsa.pub ~/[DEVELOPER]@git.company.com.pub
# add the new public key to the gitolite keydir

Resizing images is a fairly common need; there are lots of libraries around that do this.  However, I just needed to create thumbnails on-the-fly, and I certainly didn’t want to triple the size of my binary by linking against ImageMagick just to support resizing common image types.

GDI+ has the capability to do this, but surprisingly I was unable to find a clear example of how to do this just using GDI+ in C++ — lots of examples in C#, but I needed to do it in C++.

In the hopes that the next person to attempt this will simply find this blog post, here is how to do it in 6 easy(ish) steps:

Step 1: Initialize GDI+

If you haven’t already done so, you need to initailize GDI+.  This is easy, and can be done with the following commands (also note the header file you need to include):

// We'll use these headers:
#include <Gdiplus.h>
#include <string>
// In your header file, class, etc:
ULONG_PTR(m_gdiplusToken);
// Somewhere where it will run once before you need to use GDI:
GdiplusStartupInput gdiplusstartupinput;
GdiplusStartup(&m_gdiplusToken, &gdiplusstartupinput, NULL);

Step 2: Load the image:

std::wstring filename(L"someImage.jpg");
Gdiplus::Bitmap bmp(filename.c_str());

That was hard, wasn’t it?

Step 3: Resize the image:

I wrote a function from this. I adapted it from one I found somewhere, but now I can’t find the link. Note that this will preserve the aspect ratio of the image so that it fits in the specified boundaries but does not stretch. This means that if you specify 128×128, the resulting image might actually be something like 128×96, 102×128, etc.

Gdiplus::Bitmap* GDIPlusImageProcessor::ResizeClone(Bitmap *bmp, INT width, INT height)
{
    UINT o_height = bmp->GetHeight();
    UINT o_width = bmp->GetWidth();
    INT n_width = width;
    INT n_height = height;
    double ratio = ((double)o_width) / ((double)o_height);
    if (o_width > o_height) {
        // Resize down by width
        n_height = static_cast<UINT>(((double)n_width) / ratio);
    } else {
        n_width = static_cast<UINT>(n_height * ratio);
    }
    Gdiplus::Bitmap* newBitmap = new Gdiplus::Bitmap(n_width, n_height, bmp->GetPixelFormat());
    Gdiplus::Graphics graphics(newBitmap);
    graphics.DrawImage(bmp, 0, 0, n_width, n_height);
    return newBitmap;
}

And then later after you load the image into bmp:

INT width = height = 128;
Gdiplus::Bitmap *resizedBmp = ResizeClone(&bmp, width, height);

Step 4: Get the encoder:

The Gdiplus::Bitmap object stores the data in raw format so that it can be manipulated. At this point you get to decide what format to save it in. I just wanted to resize it, so I keep it in the original format. I created a simple map to map the supported image filenames to the mimetype, which we can use to find the correct encoder.

Here is what I store in my map:

    m_mtMap[".jpeg"] = "image/jpeg";
    m_mtMap[".jpe"] = "image/jpeg";
    m_mtMap[".jpg"] = "image/jpeg";
    m_mtMap[".png"] = "image/png";
    m_mtMap[".gif"] = "image/gif";
    m_mtMap[".tiff"] = "image/tiff";
    m_mtMap[".tif"] = "image/tiff";
    m_mtMap[".bmp"] = "image/bmp";

I found a utility function for looking up the encoder from the mimetype:

int GDIPlusImageProcessor::GetEncoderClsid(const WCHAR* form, CLSID* pClsid)
{
    UINT num;
    UINT size;
    ImageCodecInfo* pImageCodecInfo=NULL;
    GetImageEncodersSize(&num,&size);
    if(size==0)
        return -1;
    pImageCodecInfo=(ImageCodecInfo*)(malloc(size));
    if(pImageCodecInfo==NULL)
        return -1;
    GetImageEncoders(num,size,pImageCodecInfo);
    for(UINT j=0;j<num;j++)
    {
        if(wcscmp(pImageCodecInfo[j].MimeType,form)==0)
        {
            *pClsid = pImageCodecInfo[j].Clsid;
            free(pImageCodecInfo);
            return j;
        }
    }
    free(pImageCodecInfo);
    return -1;
}

Step 5: Save the image:

Now that we have a function to get the encoder CLSID, we can save the image. If we want to save it as a file, we can do it like this:

CLSID encId;
std::wstring extension = getExtension(filename);
std::wstring mimetype = getMimeTypeFrom(extension);
if (GetEncoderClsid(mimetype, &encId) > -1) {
    resizedBmp->Save(std::wstring(L"Output") + extension, &encId);
}

Now, in my case this wasn’t actually the goal; I needed the byte data for that file. I guess I could have saved it and then opened it and read it back, but since I am using this in a FireBreath plugin for Facebook that can be problematic. You can get the bytes directly like this:

        // If we were able to determine the filetype, save it to a stream
        IStream *buffer;
        CreateStreamOnHGlobal(NULL, true, &buffer);
        resizedBmp->Save(buffer, &encId);
 
        // Find the size of the resulting buffer
        STATSTG statstg;
        buffer->Stat(&statstg, STATFLAG_DEFAULT);
        ULONG bmpBufferSize = (ULONG)statstg.cbSize.LowPart;
        int length = bmpBufferSize;
 
        // Seek to the beginning of the stream
        LARGE_INTEGER li = {0};
        buffer->Seek(li, STREAM_SEEK_SET, NULL);
 
        // Copy the resulting data from the stream into our result object
        char *data = new char[bmpBufferSize];
        res.content_type = content_type;
        ULONG bytesRead;
        buffer->Read(data, bmpBufferSize, &bytesRead);
        buffer->Release();

Step 6: Deinitialize GDI+ properly

You need the token you created in step 1:

    GdiplusShutdown(m_gdiplusToken);

That’s it!

Hope it helps someone!

This last week FireBreath has taken a big step — we have a new website at http://firebreath.org. The reasons for the move are varied, but what it comes down to is that Google Code’s wiki, while pretty decent for small projects, simply isn’t powerful enough to handle documentation for a project like this well.  Granted, it can do it, but not as well as other options.

Creating an integrated documentation system

My primary goal with this website is to provide a completely integrated documentation system for users of FireBreath that is easy for both me and others to update.  I am fairly happy with my solution so far which relies on Atlassian Confluence and Doxygen.  To find out why and how I chose this combination, as well as the steps I took to get here, read on.  Also, if you’re in a hurry and just want to see the end result, skip to the last couple sections of this post under “Doxygen/Confluence Nirvana” =]

Choosing Confluence

I should add a quick plug here for Atlassian Confluence. No, I don’t work for them, and previous to this I had only used their more well known solution JIRA, which is also not bad.  The main reason I had never really looked into them is that, well, I’m cheap, and those are commercial products.  However, as it turns out, Atlassian provides free licenses for open source projects!  Awesome!  It should also be mentioned that the commercial license for small organizations is pretty reasonable as well.

I installed it first just to test, and I did have to learn a bit of new wiki syntax; however, the syntax is easily understandable and powerful.  With the help of 4 or 5 vim search/replace regex strings, it probably took me all of an hour to convert 30+ wiki pages from google code format to confluence, and I can customize the theme and fonts!  When I found Gliffy (also free for open source projects) I was hooked and immediately applied for the open source license.  Less than 48 hours later I had it.

Creating doxygen docs

In a recent survey that we ask FireBreath users to fill out, many users mentioned frustration with the lack of good class documentation in FireBreath, and a few suggested something generated from Source Code.  After doing a little research, I chose Doxygen as the tool to use; in fact, it’s the only real C++ documentation generation tool I found.  It took a little learning, but soon I had class documentation generated from FireBreath source code and started adding documentation comments.

Writing good documentation for use with doxygen is difficult, but if you work at it you can come up with some pretty good stuff.  However, the index in doxygen isn’t quite what I wanted.

Integrating the doxygen docs in confluence

After I tweaked the stylesheet to make it more readable, I found that there was still something I didn’t like about the docs; we have lots of documentation written in our wiki, tutorials and explanations that don’t make sense to put in the code.  However, it’s in a completely different system from the class documentation generated by doxygen, so even if our docs link to each other, the experience is far from ideal!

We definitely need the docs integrated into the same system.

The iframe approach

The first thing I tried was embedding an iframe in a confluence wiki page.  This isn’t too hard; you just have to set up a user macro or enable the {iframe} macro.  I set a minimum height so that it was readable, and it wasn’t too bad.

The problem is, it doesn’t use the full available height, it has weird scroll bars, and different sized screens really throw off the look.

The {html-include} macro

The next thing I tried was enabling the html-include macro, adding classdocs.firebreath.org into the whitelist, and then using the following code:

{html-include:url=http://classdocs.firebreath.org/patched/classes.html}

This actually looked pretty good!  Everything was integrated, my class list was there!  I was pretty excited!  Well, at least until I tried clicking on a link.  All the links are relative, and html-include doesn’t rewrite the links.  So, naturally, none of them worked!  Also, since the <head> section of the page was missing, we had no stylesheet.

The missing stylesheet issue wasn’t that hard to fix; I simply copied the majority of the stylesheet from doxygen (which I had tweaked anyway to match the color scheme) and put it into a user macro.

<style>
/* contents of doxygen.css here */
</style>

I also added style tags to hide the tab navigation, since it looked just a bit out of place.

sed to the rescue!

So I now had class docs that I could put into a confluence page, but none of my links worked!  After a little thought, I came up with the following bash script:

#!/bin/bash
 
for fl in *.html; do
 mv $fl $fl.old
 sed 's/href="\([^#]\)/href="http:\/\/classdocs.firebreath.org\/\1/g' $fl.old > $fl.stage2
 sed 's/http:\/\/classdocs.firebreath.org\/javascript:/javascript:/g' $fl.stage2 > $fl.stage3
 sed 's/src="/src="http:\/\/classdocs.firebreath.org\//g' $fl.stage3 > $fl
 rm -f $fl.old
 rm -f $fl.stage2
 rm -f $fl.stage3
done

This script processes all of the links in the page and adds an absolute path to each!  Fantastic, now my links work!

Except…. well, I still wasn’t happy with it.  There had to be a better way!  Sure, the links work, but when you click on it you leave the page!

Doxygen-Confluence nirvana

I looked at a lot of options.  I downloaded the Atlassian plugin framework and looked at writing a plugin to help me.  I emailed Atlassian support.  I looked at a lot of things.  Finally, I gave up and broke out the python.  I have attached the resulting script.  Here is what it does:

Uses the following config options:

• Input path to doxygen docs
• Output path
• Parent pageId for:
  • Classes
  • Structs
  • Namespaces
  • Files
• Confluence URL
• Username/password passed on the command line

Performs the following general steps:

1. Expects that doxygen has been run and has generated docs/html and docs/xml directories
2. Loads and parses docs/xml/index.xml
3. Loop through all entries in the XML file
   1. Get the refid from the xml to figure out which HTML file exists for that entry
   2. Generate a “readable name” for each entry (replace “::” with a space, since confluence doesn’t allow “:” in page names)
   3. Add an entry to the search/replace list to map the html filename to the new page name
4. Process all .html files found; search for .html paths and replace then with the new page name path
5. Loop through all found pages
   1. use XML-RPC to get the page from confluence
   2. If none was found, create a new page
   3. Set the parent of the page (see config options above)
   4. Set the content of the page with:
      • {doxygen_init} – injects the CSS
      • {html-include} – with a web path to the output .html files
   5. Store the page in Confluence
6. Remove all pages that are descendants of one of the root pages that weren’t added this session

And voila!  Confluence now has all of the doxygen pages in, and since we rewrote all of the links they all point to each other.

Clean-up

If you, like me, think that including every class in the project could be a bit overkill and only include the ones that have documentation, you may want to remove the .html files that doxygen generates for the “non-visible” classes.  These aren’t linked to, so you don’t normally see them, but they do show up in the .xml file. Since part of the check the script does is to see if the expected .html file exists, you can fix this by simply deleting all .html files in the directory that nothing links to.  The best way I have found to do this is with the following script:

#!/bin/bash
for fl in *.html; do
 grep $fl * > /dev/null
 if  (( $? )); then
  rm -v $fl
 fi
done

The code

Attachment: doc2confluence.py.gz

Enjoy!

I also ran into a problem with words being tokenized with their surrounding punctuation, rather than simply the word — searching for “something” was definitely NOT the same thing as searching for “something.” — note the included punctuation in the second search.

The bottom line:

• make sure stopwords is configured the same way for the fields you are searching (or at least be sure you understand the ramifications)
• don’t be afraid to play with the “minimum-match” (mm) default configuration
• when it makes sense, strip surrounding punctuation using PatternReplaceFilterFactory using an index tokenizer

For the full, detailed write-up, head over to OpenSky’s engineering blog and read the original blog post.

The Goal

Create HTML elements dynamically which transition appropriately when the view controller appears.

The Problem

The shipping version of TuneKit.js doesn’t give you a chance to append elements to the DOM before viewDidAppear is invoked (which is after transitionToController).

Initial Attempts

I first tried appending the child elements (using TKUtils.buildElement in a loop over appData.songs) inside the viewDidAppear callback. Obviously, the problem here is that the elements don’t exist the first time the view controller appears, so can’t nicely fade in via your defined CSS animations.

Next, I tried moving the logic that create the HTML elements into viewWillAppear. The problem here was that this method was called before the desired view controller is added to the DOM — so the element I need to append the children to doesn’t exist.

The Solution

The solution was actually pretty simple, and works like a charm; you just need to be willing to edit TuneKit.js.

Before:

// pre-transition methods
controller._viewWillAppear();
controller.viewWillAppear();
 
// add it to the tree
this.view.appendChild(controller.view);
 
// transition
if (previous_controller !== null) {
  this.transitionToController(previous_controller, controller);
}

After:

// add it to the tree
this.view.appendChild(controller.view);
 
// pre-transition methods
controller._viewWillAppear();
controller.viewWillAppear();
 
// transition
if (previous_controller !== null) {
  this.transitionToController(previous_controller, controller);
}

Conclusion

Now that TuneKit appends the view controller to the DOM before calling the viewWillAppear callback, your dynamic elements can be created before the view appears and transitionToController is invoked.

Done.

I’ve recently been using PHP a lot.  I’ve also been using VIM a lot.  So, it makes sense that I would start using the PHP remote debugger, right?  Well, the problem is (and don’t take this the wrong way), the php remote debugger plugin for vim was written in 2004, and there are a few things missing from it.  So, being the overly-easily-amused developer I am, I embarked on a quest to fix it.

DBGp python library

It didn’t take me long to realize that the python classes (vimscript doesn’t do sockets, so you have to use something like python) provided with the old plugin was not extensible enough for what I needed.  I started re-factoring it, but fortunately before I got too far I found a DBGp python library written by ActiveState.  I had to do some hacking to get it all working together — vim doesn’t deal with import statements in python well — but it turned out to be perfect for my needs.

VimDebugger README

Here is the README from the github repo:

map <F11> :DbgStepInto<CR>
map <F10> :DbgStepOver<CR>
map <S-F11> :DbgStepOut<CR>
map <F5> :DbgRun<CR>
map <S-F5> :DbgDetach<CR>
map <F8> :DbgToggleBreakpoint<CR>