Examples: (an HTML5 compatible browser is required to view these)

Tracking Mouse Events in Canvas

The Canvas API provides very little support for tracking user interactions within a canvas drawing.  You get the standard mouse events you get with any other DOM element (onclick, onmousemove, onmouseup, etc.), and you are more or less on your own from there.  To determine if the user has interacted with a particular part of your canvas visualization, you need to take the clientX and clientY from the mouse event and subtract the absolute top and left of the canvas element from them to get an x and y relative to the top left of the canvas.  The canvas’s position can be determined by something like jQuery’s offset() function, or you can calculate it directly by recursing through the canvas’s offsetParents and looking at their offsetLeft, offsetTop, scrollLeft and scrollTop properties (this can be simplified if your HTML has less structure).

Once you have an x and y in the canvas’s coordinates, you can check if the mouse event is in some rectangle by checking that x and y are between the rectangle’s edges.  If you need a more precise target, you can use the canvas isPointInPath() method by first drawing the target shape using the canvas drawing API and then checking context.isPointInPath(x,y).

The Problem With Interactivity and Transforms

This works fine until you start using transforms to draw your controls.  In the interest map project, I used the bounding box method to detect clicks on the arrow buttons on the right and left of the x axis.  I wanted to use the same code to check the buttons on the top and bottom of the y axis, but soon realized this wouldn’t work. Because the buttons are drawn in a rotated coordinate system, the coordinates I used to draw the buttons were no longer relative to the top left of the canvas, so they didn’t match the coordinates of the mouse event.  In this case the actual coordinates were relatively easy to calculate, but there are other interactive parts of the visualization whose canvas coordinates would be much more difficult to calculate.

To a certain extent, the isPointInPath() method works around this problem.  If you draw a shape in a coordinate system defined by any number of transforms and then call isPointInPath() using canvas coordinates, you’ll still get the desired result.  The downside of this approach is that you need to re-create the sequence of transforms used to draw the control and then draw the correct shape at the time you handle the mouse event. So you need to maintain a copy of at least some of your drawing logic in order to create your mouse targets in your event handlers.  I wasn’t very happy with this solution so I developed another one.

Make2DContextQueryable

The problem with the canvas transform API and interactivity is that it’s a one way street. You provide the directions, and canvas draws the right picture.  Once you need to figure out where all this transforming has gotten you, you’ve got no support from the API.  Make2DContextQueryable fills in this gap.

It’s no mystery how all the canvas transform methods combine to create a transform matrix, and once you have the the matrix, you only need to invert it (think 1/A) to create a transform that converts points in the original coordinates to the current transformed coordinates. Make2DContextQueryable wraps all of the canvas transform methods with functions that first apply the transform to our own matrix before calling the original method on the canvas context. The new matrix is stored in a stack so that save() and restore() work as expected.  While you’re drawing a control, you can call t = context.currrentTransform() to get and store its coordinate transform. Then in the event handler, you can either call context.canvasToContext({x:canvasX,y:canvasY}, t) to convert the mouse location to a point in the control’s coordinate system and then test it yourself, or call context.setTransform(t.a,t.b,t.c,t.d,t.e,t.f), draw a path around the click area (in the control’s coordinate system) and call isPointInPath(canvasX, canvasY).

Here’s the Make2DContextQueryable function definition:

(function() {
window.TransformArithmetic = {};
var ta = TransformArithmetic;
ta.project=  function(p,t) { return {x:t.a*p.x+t.c*p.y+t.e, y:t.b*p.x+t.d*p.y+t.f}; };
ta.multiply=function(t2,t1){ return {a:t1.a*t2.a+t1.b*t2.c, c:t1.c*t2.a+t1.d*t2.c,
b:t1.a*t2.b+t1.b*t2.d, d:t1.c*t2.b+t1.d*t2.d,
e:t1.e*t2.a+t1.f*t2.c+t2.e, f:t1.e*t2.b+t1.f*t2.d+t2.f}; };
ta.translate=function(x,y) { return {a:1,b:0,c:0,d:1,e:x,f:y}; };
ta.rotate= function(theta) { return {a:Math.cos(theta), b:Math.sin(theta),
c:-Math.sin(theta),d:Math.cos(theta),e:0,f:0}; }
ta.scale=    function(x,y) { return {a:x,b:0,c:0,d:y,e:0,f:0}; }
ta.ident=       function() { return {a:1,b:0,c:0,d:1,e:0,f:0}; }
ta.clone=      function(t) { return {a:t.a,b:t.b,c:t.c,d:t.d,e:t.e,f:t.f}; }
ta.inverse=    function(t) { var det=t.a*t.d-t.c*t.b; return {a:t.d/det, b:-t.b/det, c:-t.c/det,
d:t.a/det, e:(t.c*t.f-t.e*t.d)/det, f:(t.e*t.b-t.a*t.f)/det}; }
ta.interp=function(t1,t2,i) { var t={}, a=['a','b','c','d','e','f'];
for (var j=0,k;k=a[j];j++) t[k]=t1[k]+(t2[k]-t1[k])*i; return t; }
window.Make2DContextQueryable = function(ctx) {
if ( ctx.contextToCanvas ) return;
var transforms = [ta.ident()];
function current() { return transforms[transforms.length-1]; }
function set(t) { transforms[transforms.length-1] = t; }
function apply_to(t) { transforms[transforms.length-1] = ta.multiply(current(),t); }
function proxy_before(o,fn,f) { var oldf=o[fn]; o[fn]=function(){
f.apply(o,arguments); oldf.apply(o,arguments); };}
proxy_before(ctx,'save',function() { transforms.push(ta.clone(current())); });
proxy_before(ctx,'restore',function() { if (transforms.length>1) transforms.pop();
else transforms[0]=ident(); });
proxy_before(ctx,'scale',function(x,y) { apply_to(ta.scale(x,y)); });
proxy_before(ctx,'rotate',function(theta) { apply_to(ta.rotate(theta)); });
proxy_before(ctx,'translate',function(x,y) { apply_to(ta.translate(x,y)); });
proxy_before(ctx,'transform',function(a,b,c,d,e,f) { apply_to({a:a,b:b,c:c,d:d,e:e,f:f}); });
proxy_before(ctx,'setTransform',function(a,b,c,d,e,f) { set({a:a,b:b,c:c,d:d,e:e,f:f}); });
ctx.contextToCanvas = function(p) { return ta.project(p,current()); }
ctx.canvasToContext = function(p) { return ta.project(p,inverse(current())); }
ctx.currentTransform = function() { return current(); }
return ctx;
}
})();

And here’s how it’s used a revised version of the tree application (full source):

function init() {
var canvas = document.getElementById('display');
// call Make2DContextQueryable to add our transform recorder
// functions.  Although the function returns the context, the
// the modifications are made to the element itself, so we don't
// need to call the function again.
var ctx = Make2DContextQueryable(canvas.getContext('2d'));
ctx.font = FONT; ctx.strokeStyle = OVER_STROKE;
ctx.lineWidth = OVER_LINE_WIDTH;
tree.branches = gen_tree(1);
draw();
canvas.onmousemove = function(evt) {
evt = evt || window.event;
var canvas = document.getElementById('display');
var canvasX = evt.clientX - canvas.offsetLeft
+ document.body.scrollLeft,
canvasY = evt.clientY - canvas.offsetTop
+ document.body.scrollTop;
if ( over(canvas.getContext('2d'), tree, {x:canvasX, y:canvasY}) )
draw();
}
}
function over( ctx, branch, canvasCoords ) {
// tranform the mouse point into the branch's coordinates
// (branch tranform was saved in the middle of the draw function)
var ctxCoords = ctx.canvasToContext(canvasCoords,branch.transform),
changed = false,
text_width = ctx.measureText(PARTS[branch.type]).width;
// perform simple rectangle test to see if mouse is over branch
var is_over = (ctxCoords.x >= 0
&& ctxCoords.x <= text_width + LEFT_OFFSET
&& ctxCoords.y <= TOP_OFFSET && ctxCoords.y >= -30 );
if ( is_over != branch.over ) {
branch.over = is_over;
changed = true;
}
if ( ! branch.branches ) return changed;
return changed || over(ctx,branch.branches[1],canvasCoords)
|| over(ctx,branch.branches[0],canvasCoords);
}

The functions under the TransformArithmetic namespace are a by-product of the logic needed to replicate the context transforms, but they can be quite useful themselves. The second interactive tree demo (source) uses these functions to calculate a transform that puts the mouse-overed branch of the tree in the center of the screen. When you mouse over a branch, the application records the active transform, AT. If we wanted this branch to be drawn in the canvas’s coordinates, we would simply need to invert AT and apply it using the ctx.transform() method. We actually want to translate it to the center of the canvas, however.  Let AT be the product of that translation, TL, and all the branch transforms used to get the branch, TT. So what we really want is TL/TT (here ‘/’ means multiply by the inverse). Since AT=TL*TT, TT=(1/TL)*AT (remember order is important), so TL/((1/TL)*AT) is the transform we need. Using the fact that the inversion of a translation matrix is simply the negation of the translation factors e and f, we can create the transform with the following code:

var ta=TransformArithmetic;
// translate a little left of center
ctx.translate( 100, 300 );
// subtract out the translation from the active transform and invert it
var at = ta.inverse(ta.multiply({a:1,b:0,c:0,d:1,e:-100, f:-300},active_transform));
// apply the calculated matrix
ctx.transform( at.a, at.b, at.c, at.d, at.e, at.f );

The computations needed to determine whether the mouse is over a branch in this new coordinate system are a little more complicated, but can be calculated using a similar algebraic manipulation. The TransformArithmetic also contains an interp() function that will interpolate between two transforms, thus making animation possible.

Wrapping the transform functions may seem a little extreme, but it goes a long way towards simplifying interactive visualizations.  It’s also handy to have the transform math lying around.  I’ve been using it to create bezier curve functions that permit the second two control points to be drawn in a different coordinate system.

This is the second post in a series on creating custom interactive visualizations in canvas.  The first post is here.

The canvas API contains five methods (rotate, scale, translate, transform, and setTransform) used to transform the drawing context. We typically use the transform API when we want to rotate or scale some element of the visualization (especially text). In canvas, we don’t actually move the elements. Instead we transform into a new coordinate system where the element is no longer scaled, rotated or translated and then draw normally. The difference is subtle, and you can get by with ignoring it so long as you remember to execute the transforms before you draw.   But, once you embrace this way of thinking, transformed coordinates can be a powerful way to break down complicated drawing tasks into simple steps. In the remainder of the post I’ll describe how to use transforms effectively and how to overcome the difficulties involved in making transformed objects interactive.

Examples: (an HTML5 browser is required to view these)

The Basic Idea

Choosing the right coordinate system simplifies calculations.

In the interest map project I described in a previous post, I want to use bezier curves to draw arrows using my own formula for the ratios of the arrow length, arrow head length, and arrow head height. The x and y coordinates of the head and tail of the arrow are known, but using these to figure out the location of the sides of the arrow head and its control points require a lot of rather messy trigonometry. If we create a coordinate system where the tail of the arrow is at the origin and the head lies on the x axis (this still requires a little bit of trig), the coordinates simplify to {0,0}, {l-C,f(l)}, {l,0}, and {l-C,-f(l)} where C is the length of the arrow head and f(l) is a function that gives its height. Also notice that, after a transform, we can use the same code to draw the horizontal and vertical axes (so long as we can handle the difference in lengths).  Complex visualizations can usually be broken up into components like this that each have a natural coordinate system.  The trick then becomes calculating an appropriate transform.

A Bit About The Math

I’m not going to get deep into linear algebra or affine geometry, but I think a little knowledge of how transforms are implemented is required to use them effectively. Any change of coordinates can be represented by a single transformation matrix.  You don’t need to know what a matrix is so long as you understand that it’s kind of like a number in the sense that you can multiply it with points or other transformation matrices.  When you multiply a point from your transformed coordinates with a transformation matrix, you get another point that is transformed back into the original coordinate system.  The canvas context always has a transformation matrix, and whenever you call a drawing API method (e.g. fillRect, bezierCurveTo, arcTo, etc.) it multiplies each of the points by this matrix before it renders any pixels from the shape to the canvas.

When you multiply a transformation matrix by a another transformation matrix, you get a new transformation matrix that is a combination of the two.  For example, if you have a transformation, A, that represents a coordinate system rotated 45 degrees and another transformation, B, that represents a translation of the origin 100 pixels along the x axis, then C=A*B is a transformation where the origin is 100 pixels out on the 45 degree line.  A lot of the power of Canvas’s transformation API derives from the ability to combine transforms like this.  With the exception of setTransform(), all of Canvas’s transform methods (rotate, scale, translate, and transform) are applied by creating a matrix representing the operation and then multiplying it with its current transform to create its new transform.  One important different between matrix multiplication and normal multiplication in that matrix multiplication does not commute (A*B != B*A).  So you’ll get a different result if call context.rotate() and then context.translate() than you would if you called context.translate() then context.rotate().

Save and Restore

The save() and restore() methods in canvas are a convenient way to avoid specifying stroke and fill styles over and over again in a drawing, but they become critical when you start using transforms.  The save() method takes all the current state of the canvas context including the current transform and pushes it on a stack.  You can then make as many modifications or transforms as you like.  When you’re done drawing, you call context.restore() and everything is back to the way it was before you called save().  You can always reset the fillStyle or lineWidth, but there is no trivial way to reset the transform (you can use setTransform() if you really know what you’re doing).  Using save() and restore() is habit you want to acquire.

Some Concrete Examples

Alright, enough theory: Here’s some code. The first example is the obligatory fractal demonstration. 2D transforms make quick work of fractals, because, by definition, they are the same thing over and over at different scales and in different places. At any rate, the application first creates a binary tree (see the full source). Then, beginning with the trunk it draws the text at the origin (in its transformed coordinates). Then, for each branch, it translates the origin to the end of the word, rotates in the direction of the next branch, and scales to the size of the branch. It then repeats the process until it gets to a leaf.

function draw() {
    var ctx = document.getElementById('display').getContext('2d');
    ctx.font = 'bold 30pt Georgia,Times New Roman';
    ctx.translate( ctx.canvas.width / 2, ctx.canvas.height );
    ctx.rotate( -Math.PI/2 );
    draw_branch( ctx, tree );
}
function draw_branch(ctx, branch) {
  ctx.fillStyle = branch.branches ? WOOD_COLOR : LEAF_COLOR;
  // draw text in default font
  ctx.fillText(PARTS[branch.type], LEFT_OFFSET, TOP_OFFSET);
  if ( ! branch.branches ) return;
  for ( var i=0,b; b=branch.branches[i]; i++ ) {
    ctx.save();   // IMPORTANT!!
    // move origin to end of text
    var text_width = ctx.measureText(PARTS[branch.type]).width;
    ctx.translate(text_width + LEFT_OFFSET, 0);
    // scale coordinates to predetermined value for branch
    ctx.scale( b.scale, b.scale );
    ctx.rotate( b.theta );   // rotate axes to be in line with branch
    draw_branch(ctx, b);
    ctx.restore();   // IMPORTANT!!
  }
}

The name tags in the interest map (source) presented a more realistic problem. We wanted to simulate the way people attached their name tags on the skill cards. The placement of each name tag was essentially random except that the tags were always near the edge of the card and they were always nearly horizontal. This requires 4 transforms: A translation to the center of the skill card (A), A rotation to give it a unique position on the card (B), A translation to the edge of the card with a random messiness factor (C) and a rotation back that’s roughly equal to B so the text is roughly horizontal (D).  The following is the (somewhat simplified) function that transforms the coordinates then draws the box and the text around the origin.  In this function, the angle of the name tag on the card, r, is given.

function nametag( ctx, cardx, cardy, person, r ) {
  var text_width = ctx.measureText(person.name).width,
        width = text_width+10;
  ctx.save();
  // make center of card the origin
  ctx.translate( cardx, cardy );
  // rotate in direction of tag placement on card
  ctx.rotate( r );
  // translate to edge of card (with a little permutation)
  ctx.translate( -TAG_RADIUS -  + 4*Math.random(), 0 );
  // rotate back with permutation so that text is nearly horizontal
  ctx.rotate( -r + .4*(Math.random()-.5) );
  /* ... set box style parameters ...  */
  // draw box centered around (transformed) origin
  ctx.beginPath(); ctx.rect( -width/2, -10, width, 20 ); ctx.closePath();
  ctx.fill();
  ctx.stroke();
  /* ... set font style ... */
  // draw text centered around (transformed) origin
  ctx.beginPath(); ctx.fillText( person.name, text_width/2, 0 ); ctx.closePath();
  draw_nametag( ctx, person, interest );
  ctx.restore();  // DON'T FORGET!!!
}

Most of the time, thinking in terms of coordinate transforms will make your life a lot easier. The exceptions come when you need to relate shapes drawn in one coordinate system with shapes or points specified in another. This is especially true when you need make transformed elements interactive. I’ll talk more about this problem and some solutions in the next post.

So I’ve spent the last few weeks building Chrono.js, an application metrics server. Chrono is still under development so it’s really not appropriate for general usage but it makes for a decent example app.

Sample Request

Here’s a basic request which fetches data from MongoDB and renders it via jqtpl:

app.get('/', function(req, res) {
  db.open(function(err, ignored) {
    if (err) console.log(err);
    db.collectionNames(function(err, names) {
      if (err) console.log(err);
      db.close();
      pro = _(names).chain()
        .pluck('name')
        .map(function(x) { return x.substr(x.indexOf('.') + 1); })
        .reject(function(x) { return (x.indexOf('system.') == 0); })
        .value();
      res.render('index.html.jqtpl', {
        locals: {
          metrics: pro
        }
      });
    });
  });
});

We are scanning MongoDB for the set of metric collections and displaying them in the selector so the user can switch between them. See the main source file for more examples about parameter handling, headers, JSON, etc.

Installation

You will need MongoDB installed and running locally. You should already have Node.js and npm installed from Part I. Let’s install the javascript libraries required and Chrono itself:

npm install express expresso jqtpl mongodb underscore tobi
git clone git://github.com/mperham/chrono.js.git

Express is a lightweight application server, similar to Sinatra. jqtpl is jQuery Templates, which allows us to dynamically render HTML. mongodb is the MongoDB client driver for Node.js. expresso is a TDD framework and tobi is an HTTP functional testing package. Finally, underscore is a great little utility library which provides JavaScript with a much more sane functional programming API.

Running

You can run the tests:

expresso

or run the Chrono.js server:

node chrono.js

Click http://localhost:3000/load/registrations to load in some fake data and visit http://localhost:3000 (select ‘registrations’) to see the results, graphed in living color for you!

Testing

Expresso gives us a simple DSL for testing our endpoints but I found it to be lacking basic setup/teardown functionality which I had to roll myself in order to insert some test data. I would have preferred to use vows.js but it appears to be incompatible with tobi. Check out the Chrono.js test suite for what I came up with, here’s a small sample:

  exports['POST /metrics'] = function() {
    assert.response(app,
      { url: '/metrics', method: 'POST', headers: { 'content-type': 'application/json' }, data: JSON.stringify({ k: 'registrations', v: 4, at: parseInt(Number(new Date())/1000) }) },
      { status: 201, headers: { 'Content-Type': 'text/html; charset=utf8' }, body: '' }
    );
  }

With expresso, we export the set of functions to run from our test module. Expresso runs them all in parallel and collects the results. The parallelism means that you must be careful with any test data you create. Since MongoDB doesn’t support transactions, we can’t use transactions to isolate each of our tests (e.g. see Rails’s transactional fixtures) so you need to be careful about the data created or deleted by each test and how you assert the current state.

Conclusion

While I’ve gotten much better in the last week, I’ll admit I’m still uncomfortable with Node.js. Its asynchronous semantics mean that your code runs as part of a chain of callbacks; knowing when, where and how that chain works is still difficult for me to understand. Frequently you’ll just get a black screen and a process that won’t exit or an error message that may or may not be related to the actual bug.

That said, I still remember being very frustrated with Ruby, its frequent use of “magic” and poor documentation (e.g. navigating this still befuddles me). I overcame those issues to be very comfortable with Ruby in general; I hope more time with Node.js will give me the same relief.

Google’s V8 JavaScript engine provides the underlying VM for executing JavaScript code. Since V8 is typically embedded in a browser, it does not expose any real notion of FileSystems, Processes and I/O streams that are fundamental to a server-side application. Node.js provides core APIs that wrap the standard POSIX functions available on all modern Unixy systems, along with higher-level APIs for HTTP/S support.

Node.js is designed to be an asynchronous system: your application code executes only on a single thread and no blocking I/O is allowed. Instead, Node.js uses an event loop to process I/O events via the libev C library.

Let’s dive into the Node code to see how it initializes itself. This writeup is current as of 0.4.2 or 3/14/2011 but obviously things will change over time.

Process Initialization

Node initializes itself in the node::Start method in node.cc. node::Start performs the housekeeping necessary to use V8 and libev in a well-behaved process. A brief walkthru:

• Line 2240-2275: Parse the cmd line arguments, dividing them up between node and v8.
• Line 2280: Setup signal handlers so Ctrl-C will actually terminate the Node.js process.
• Line 2290-2330: Configure various libev watchers, include the Idle detector which uses a simple hueristic to determine when it would be best for V8 to run garbage collection.
• Line 2332: Initialize V8!
• Line 2338-2364: Debugging support
• Line 2367-2374: Call node::Load to bootstrap the Node.js APIs, see below
• Line 2385: The key line: pass control to libev event loop. This function will not return until the application code decides to exit the process based on some event or SIGINT or SIGTERM are received.
• Line 2389-2405: Process is exiting. Execute any “exit” event callbacks in the application code and return.

Bootstrap

node::Load performs the actual Node.js API bootstrap in three stages:

• Stage 1: Using C++, it sets up a process object and fills it with various constants and methods, so your JavaScript code can access “process.argv”, “process.env”, etc. As far as I can tell, the most fundamental part is the Process.binding function which allows JS code to lazily bootstrap the C++ bindings. For instance, this line to fetch the ‘fs’ binding actually causes the code within node_file.cc to expose various POSIX file functions to the JavaScript module.
• Stage 2: It executes the JavaScript in src/node.js. This code sets up the various I/O streams (stdin, stdout and stderr) and creates the JavaScript module loading system which knows how to find JavaScript files in lib/ when require(module) is called.
• Stage 3: Node.js executes the user’s code, via debugger, repl or user-provided script file. Remember this script doesn’t block so it will load quickly and return back to node::Load.

Once the user’s code has been loaded, we should have set up any listening sockets, timers, etc. The entire node::Load function unwinds and we enter the event loop, waiting for I/O or Timer events to execute code.

Thanks for staying with me this far – I hope you learned something new. Next up, I’ll show you how to configure and run a full stack application on node.js using MongoDB and Express!

For this post, I’m just going to cover the basics; I’ll follow up soon with deeper posts.

Overview

JavaScript has an interesting history – it hasn’t developed like most other languages; until recently, executing JavaScript meant embedding it in a web page for a browser to execute. A few things happened which radically hastened the rise in JavaScript as an reasonable server-side language:

• AJAX and the Browser Wars have resulted in dramatic improvements in Javascript runtime performance and high-quality developer tools.
• Node.js built Process, File and Network I/O APIs on top of Google’s V8 JavaScript engine, allowing command line programs and daemons to be built in JavaScript.

Node.js adds a friendly command line face to V8 and APIs that are conceptually similar to Ruby’s EventMachine library: all I/O is asynchronous and threads are unavailable to user code. Additionally JavaScript is a prototype-based language, not object-oriented. This makes for a programming model that is radically different from what Ruby or Java developers are used to.

Installation

I’m going to assume OSX and I like to install things with Homebrew. We’ll install node and npm, node’s package manager, with these commands:

    brew update # update Homebrew's formulas to the latest
    brew install node # install node
    curl http://npmjs.org/install.sh | sudo sh # install npm

Once installed, you should be able to run node --help and npm --help.

A minimal web server using Node.js:

var http = require('http');
http.createServer(function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('Hello Worldn');
}).listen(8124, "127.0.0.1");

Copy that code into hello.js and run it:

node hello.js

Now let’s slam it with some requests:

ab -n 10000 -c 50 http://127.0.0.1:8124/

Results:

Server Hostname:        127.0.0.1
Server Port:            8124
Document Path:          /
Document Length:        12 bytes
Concurrency Level:      50
Time taken for tests:   1.479 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      760000 bytes
HTML transferred:       120000 bytes
Requests per second:    6760.79 [#/sec] (mean)
Time per request:       7.396 [ms] (mean)
Time per request:       0.148 [ms] (mean, across all concurrent requests)
Transfer rate:          501.78 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       3
Processing:     1    7   3.6      7      20
Waiting:        1    7   3.6      7      20
Total:          1    7   3.6      7      22

Percentage of the requests served within a certain time (ms)
  50%      7
  66%      9
  75%     10
  80%     11
  90%     12
  95%     13
  98%     15
  99%     16
 100%     22 (longest request)

Not bad. Of course, this is using localhost and a trivial app but at least we know it’s up and running well. In my next post, we’ll explore the Node.js source code itself.

Our goal was to create a visualization that would focus in on individual and collective patterns in the skills and interests chosen while still inviting people to explore and draw new conclusions.  It was important to leave some aspects of the interpretation to the group that created it, so we knew we needed an interactive visualization (the opportunity to explore new HTML5 influenced me here).

The result of this effort (this page requires an HTML5 compatible browser) is a single page canvas application that plots Carbon Fives skills and interests along various axes.  In the rest of the article I’ll explain how we converted the data to something more meaningful.

Identifying Meaning

Staring at the wall filled with skill cards and covered with name tags, it was tempting to try to make sense of it all by asking why a certain color seemed to be bunched up in a corner or why others seemed more scattered.  The cards were arbitrarily distributed, however, so any abnormalities were the product of statistical fluctuation.  If we really wanted to interpret the data as a two dimensional graph, we would have to rearrange the cards in such a way that their position on the on the wall had meaning.

So we began identifying opposing qualities that were applicable to all the skills in varying degrees.  In the end, we came up with “‘server side / client side’, ‘technical / creative’, ‘c5 core / c5 expansion’, ‘public facing / in house’, ‘trendy / old school’.  We then sat down together and tried to figure out where to place each skill on a scale where, say, ‘public facing’ would be zero and ‘in house’ would be a one.   The key to this exercise was to talk about where each skill falls relative to the other skills we’ve already discussed and to keep it quick.  I suspect we would get different orderings if we tried again, but the things that are obviously one thing or the other would still migrate towards the ends, and difference in the rest gets averaged out in the calculations anyway.

Once we had multiple orderings for the skills, we had to digitize the location of all the name tags.  We did this by assigning a number to each of the skill cards, and then writing down that number in either a skill column or an interest column next to the name of each person who put a name tag on that card. This is close to how the data is represented in the application.  It took a couple of passes to get the data right and we had to fill in a few gaps for employees that missed out on the exercise, but eventually I had all the data I needed to create the visualization.

Designing the Visualization

The arrows start at the average location of a person's skills and end at the average location of his or her interests.

The visualization itself is designed to look as much like the the wall as possible, while taking into account the limitations of legibility and  vector graphics.  When the application starts, it shows all the cards — each with the appropriate name tags — ordered horizontally according to whether the skill is more of a client-side or server-side skill, and vertically by whether the skill is more technical or creative.  Clicking the arrows next to the axis labels switches that axis to a new dichotomy and reorders the cards accordingly.  This view of the data is mainly to allow you to view the actual data.  If you click on the ‘show arrows’ button in the top right, you get a view more conducive to story telling.  Each of the arrows represents a Carbon Five employee.  The arrow starts at the average location of all of the person’s skills, and points to the average location of all of the person’s interests.  In a sense, it’s visualizing the direction of the person’s professional development.

You can also change the axes in the arrows view, and doing so brings up some interesting contrasts.  In some combinations, almost all the arrows are pointing in the same direction, and in others they all criss-cross.  In most combinations, all the arrows are in a pack, but in some there are a few arrows clustered together but away from the pack.  The interpretation of these phenomena depend on the axis in question.   The data has inaccuracies and biases built in at every point, but it has still been interesting to spot some feature in the visualization, interpret what it means and then check the names of the people behind the arrows.  In most cases the characterization highlights something that is unique about one of the Carbon Five team.

Building Your Own Skill Map

Aside from the choice of skill cards, there is nothing about  the exercise or the application that is peculiar to Carbon Five.  If you can get your company to generate their own skill map, you can use the source to create your own visualization by changing the data section in the top of the application.  In fact, I would be willing to do this for you if you can send me the data in some sort of reasonable digital format (limit the first 5 takers within the next year).

In the next post I will be discussing some of the programming challenges involved in writing interactive canvas applications.

update:

The second post on using transforms effectively in canvas applications is up.

I immediately added nesting of rules, similar to how SASS lets you dry-up your stylesheets. Then, with a little help from Alex, I added basic color conversions and functions like “darken” and “saturate”(demo here). I then added support for macros, so you could mix-in behaviors like rounded corners, drop shadows, and of course the ubiquitous (and awkwardly named) “clearfix”. My last pass was to make sure there are reasonable extension points for other developers. Now you can write concise and expressive CSS right along with your Javascript.

Since it’s POJS (Plain Old Javascript), it is a compact language. There’s was no need for me to write math or looping or logic functions, since Javascript’s already got them (unlike SASS). DSLs are the hottest things these days, but even the not-so-forgiving syntax of Javascript works pretty well in this case.

Why would you want to use this? Well, I thought that was the big question, but everyone I show it to has an answer. For me, I am writing pages that generate lots of their HTML markup anyway, so it is awkward to keep a separate CSS file in sync (and drag it down from the server). It works well to write self-contained jQuery plugins, as I sketched out here. I started using on a new initiative (that I can’t share yet) of my latest project, and it made prototyping and initial build-out much faster. It also makes using sprites painless– all our subtitles are in one sprite and I simply add a has: subtitle("Emotion") as one of the attributes and it works. (Previously I’d written some Ruby code to generate a CSS. Ughh.)

Please, give it a spin, and file bugs or send me comments on github.

The Applications

• Snowballs

  Snowballs is a video game that was mostly designed by my 7 year old daughter, Ada. She started with a drawing that contained a maze filled with dots, a player and some monsters, and she explained how you throw snowballs at the monsters and build walls to block them. I convinced her we could get roughly the same effect by freezing the monsters instead of blocking them, and then I began developing. We started iterating, and she continued to contribute ideas which, somewhat surprisingly, always made the game more fun. The arrow keys move the player, and the spacebar launches a snowball. It’s implemented as a single Canvas tag using a JavaScript interval as a run loop. The maze, players, monsters, and snowballs are all drawn with JS functions using the Canvas drawing API. Since this application uses the Canvas tag, it won’t run in Internet Explorer or older versions of other browsers.

• Flashcards

  I wrote Flashcards one evening when Ada’s teacher suggested using flash cards to help her memorize the words she shouldn’t need to sound out. The idea is that the parent clicks the spacebar when the child reads the word and they work together to reduce the time needed to read all the words. The application doesn’t need any graphics (other than some external images at the end), so it doesn’t use any HTML 5 features (browser compatibility is unknown).

• Timetest

  Like Flashcards, I wrote Timetest to help Ada with her schoolwork. In this case, she was getting frustrated with a lesson on analog clocks, so I wrote this application to make it more of a game. The goal is to answer as many questions as possible in the given time. The test turned out to be more difficult than it really needed to be, so I created an easier version (accessed by adding #easy to the URL). Since this application uses the Canvas tag, it won’t run in Internet Explorer or older versions of other browsers.

• Arithmetic

  Arithmetic is another educational application designed to drill addition that uses carrying. It provides spaces at the top, which you can click to mark a carry from the previous column. It doesn’t have any fancy HTML 5 features, but it does have a lot of stats.

• Surround

  My wife and I had designed a fireplace surround and mantel for our living room, but when it came to actually buying the materials and building the thing, we realized we didn’t know exactly how to scale it. Since the design was entirely composed of rectangles, I realized it would be possible to build a simple one-off CAD program to help us agree on some dimensions. The application makes use of the Canvas tag and the new HTML 5 slider inputs (which aren’t currently supported in Firefox). It translates between pixels and feet/inches constrained to an 8th of an inch, in order to make the measurements easier to use in the shop.

The goal of these exercises was to produce something simple and quick, but not entirely without polish. All the applications except Snowballs took only an evening or two. Each application (also excluding Snowballs) ended up under 300 lines of HTML, CSS, and JS combined.

Developing without a JavaScript framework

Limiting the applications to a single page means no importing jQuery or Prototype. This might seem like a big constraint in an application written entirely in JavaScript. But these frameworks really only help out with DOM traversal/manipulation, event handling, animation, Ajax, form validation. All the applications make do with a small amount of HTML, so it’s easy enough to grab elements by ID, and a few visits to Quirksmode gave me some reasonably cross-browser event-handling code. Animation isn’t really that hard in JS, and I needed it to be more custom than frameworks allow anyway. The rest I didn’t need. The truth is, I never miss having a framework (though I’m not about to stop using jQuery when working on client’s sites).

Advantages and disadvantages

• They’re quick to start. Since it’s a single page, there’s no setup and no integration.
• They’re easy to deploy. It’s only one relatively small file, so you can e-mail it to friends, and you can host it anywhere. If you want to spend a little time making it touch-screen compatible, you can get it on all the smarter smart phones without bothering with app stores. On the other hand, there is no way prevent people from stealing the application or the code if you were developing it commercially, since the source is always just a right-click away.
• HTML+CSS is a powerful display framework. Although it lacks built-in support for some of the fancier UI widgets, there is no GUI framework that I’m aware of (and I’ve developed with quite a few) that provides the same level of design control with the same ease of development as these technologies. Of course, I’ve been swimming in HTML/CSS for years, so this opinion is a bit biased.
• Canvas has a powerful vector graphics API. I had done a fair amount of Cocoa programming prior to these projects, so this didn’t come as a surprise. The real power of Canvas lies in its transformation functions. I’d like to find an application for some more advanced spline generation. The downside is that Canvas and HTML don’t really mix, so you can have great text layout in HTML and sophisticated text animations in Canvas, but not both at the same time.
• JavaScript is a capable if not optimal language. No one is going to make the case that JavaScript is the greatest language ever designed, but what it lacks in performance and language features, it partially makes up in powerful fundamentals and simplicity. JavaScript is neither a great functional language nor a great object-oriented one, but it is sufficiently powerful to develop in either of these paradigms. Snowballs, for example, acts a lot like a simulation and benefited from stateful objects. The other applications required less state and I found several uses for higher-order functions, though I wouldn’t consider any of them properly functional.
• Single-page applications are fun. These exercises have reminded me that, once the responsibilities of software engineering are stripped away, there is a joy to be found in coding for coding’s sake.

All of these applications are available on GitHub, but it’s probably easier to right-click on the page and choose view source.

There are a couple of different options for JS testing available at this point but Blue Ridge’s plugin is the first I have seen which made writing test driven code as easy as I have come to expect while still providing all the functionality I need. In particular;

• Running my tests in a browser is fast. Refreshing fixture pages to rerun tests is dramatically more efficient than waiting for selenium tests to cycle.
• I can debug my tests as they run against their fixtures pages and inspect or manipulate the DOM at any point.
• I have mock and fake objects available to test behavior and stub out dependencies.
• My JS tests run in continuous integration, just like all the other tests.

By writing fixtures pages and tests first I am immediately using all of my code in two locations and running it against slightly different DOMs. That has pushed me to encapsulate most of my code as jQuery plugins and think carefully about how much I depend on the page’s structure. I end up with terse selectors instead of using whatever attributes are readily available and a clear configuration point for my page’s behavior as I chain together plugin calls in $(document).ready().

We’d like to see what other developers have done and share what we have learned so far so Carbon Five is hosting an open discussion of JavaScript testing in our office this week.

Topic: Javascript Testing
When: Tuesday Nov 17
6pm: arrive/network
6:30 – 8 presentation and discussion
Where: Carbon Five office: 171 2nd Street, 4th Floor

Please RSVP if you would like to attend.