Genetic Symphony – A Genentech, IDEO, and Carbon Five collaboration for TED2012. Participants provide a genetic sample via a cheek swab, which is processed overnight and used to generate a unique song and visualization. Tech includes: Backbone, CoffeeScript, Audia, PhoneGap, SoX, RMagick, Rails, Jasmine and Sinon.

Presenters: David Hendee and Sean Durham, Carbon Five

ChatOS – A collaborative UX/programming experiment that allows participants to build a shared real-time web experience. It provides the basic plumbing that makes collaborative programming possible and leaves it up to the participants to build the tools and design the interactions. The talk will provide a live, interactive demonstration, and will briefly cover the technology behind the application including Node.js, Mocha.js, HTML 5, Web Sockets, and Mongo DB.

Presenter: Alex Cruikshank, Carbon Five

Escape from the Pyramid of Doom – If you set out to build real, reliable apps using Node.js you will need to tame the callback-based asynchronous jungle you find yourself in. For the last several months at Good Eggs we have explored solutions that include a combination of third party libraries, homegrown libraries, and convention. For this talk, we use our journey to provide a survey of different techniques for writing asynchronous code in Node.js including fibrous, the library solution that we currently use to support our established conventions.

Presenters: Alon Salant and Randy Puro, Good Eggs

Q & A – We want to continue discussing anything already mentioned, or go over other topics that might be interesting (e.g. js testing, tech stacks, hosting, project management, etc).

Wednesday April 25th. Doors at 6 and talks start around 6:30. Food and drinks will be provided. Huge thanks to CBSi for the venue and sponsoring food/drinks!

Please RSVP on the SF JS meetup event page!

Learn more about Dart at http://dartlang.org.

Thanks Seth! Looking forward to the next one.

Happy JS hacking!

Examples: (an HTML5 compatible browser is required to view these)

Tracking Mouse Events in Canvas

The Canvas API provides very little support for tracking user interactions within a canvas drawing.  You get the standard mouse events you get with any other DOM element (onclick, onmousemove, onmouseup, etc.), and you are more or less on your own from there.  To determine if the user has interacted with a particular part of your canvas visualization, you need to take the clientX and clientY from the mouse event and subtract the absolute top and left of the canvas element from them to get an x and y relative to the top left of the canvas.  The canvas’s position can be determined by something like jQuery’s offset() function, or you can calculate it directly by recursing through the canvas’s offsetParents and looking at their offsetLeft, offsetTop, scrollLeft and scrollTop properties (this can be simplified if your HTML has less structure).

Once you have an x and y in the canvas’s coordinates, you can check if the mouse event is in some rectangle by checking that x and y are between the rectangle’s edges.  If you need a more precise target, you can use the canvas isPointInPath() method by first drawing the target shape using the canvas drawing API and then checking context.isPointInPath(x,y).

The Problem With Interactivity and Transforms

This works fine until you start using transforms to draw your controls.  In the interest map project, I used the bounding box method to detect clicks on the arrow buttons on the right and left of the x axis.  I wanted to use the same code to check the buttons on the top and bottom of the y axis, but soon realized this wouldn’t work. Because the buttons are drawn in a rotated coordinate system, the coordinates I used to draw the buttons were no longer relative to the top left of the canvas, so they didn’t match the coordinates of the mouse event.  In this case the actual coordinates were relatively easy to calculate, but there are other interactive parts of the visualization whose canvas coordinates would be much more difficult to calculate.

To a certain extent, the isPointInPath() method works around this problem.  If you draw a shape in a coordinate system defined by any number of transforms and then call isPointInPath() using canvas coordinates, you’ll still get the desired result.  The downside of this approach is that you need to re-create the sequence of transforms used to draw the control and then draw the correct shape at the time you handle the mouse event. So you need to maintain a copy of at least some of your drawing logic in order to create your mouse targets in your event handlers.  I wasn’t very happy with this solution so I developed another one.

Make2DContextQueryable

The problem with the canvas transform API and interactivity is that it’s a one way street. You provide the directions, and canvas draws the right picture.  Once you need to figure out where all this transforming has gotten you, you’ve got no support from the API.  Make2DContextQueryable fills in this gap.

It’s no mystery how all the canvas transform methods combine to create a transform matrix, and once you have the the matrix, you only need to invert it (think 1/A) to create a transform that converts points in the original coordinates to the current transformed coordinates. Make2DContextQueryable wraps all of the canvas transform methods with functions that first apply the transform to our own matrix before calling the original method on the canvas context. The new matrix is stored in a stack so that save() and restore() work as expected.  While you’re drawing a control, you can call t = context.currrentTransform() to get and store its coordinate transform. Then in the event handler, you can either call context.canvasToContext({x:canvasX,y:canvasY}, t) to convert the mouse location to a point in the control’s coordinate system and then test it yourself, or call context.setTransform(t.a,t.b,t.c,t.d,t.e,t.f), draw a path around the click area (in the control’s coordinate system) and call isPointInPath(canvasX, canvasY).

Here’s the Make2DContextQueryable function definition:

(function() {
window.TransformArithmetic = {};
var ta = TransformArithmetic;
ta.project=  function(p,t) { return {x:t.a*p.x+t.c*p.y+t.e, y:t.b*p.x+t.d*p.y+t.f}; };
ta.multiply=function(t2,t1){ return {a:t1.a*t2.a+t1.b*t2.c, c:t1.c*t2.a+t1.d*t2.c,
b:t1.a*t2.b+t1.b*t2.d, d:t1.c*t2.b+t1.d*t2.d,
e:t1.e*t2.a+t1.f*t2.c+t2.e, f:t1.e*t2.b+t1.f*t2.d+t2.f}; };
ta.translate=function(x,y) { return {a:1,b:0,c:0,d:1,e:x,f:y}; };
ta.rotate= function(theta) { return {a:Math.cos(theta), b:Math.sin(theta),
c:-Math.sin(theta),d:Math.cos(theta),e:0,f:0}; }
ta.scale=    function(x,y) { return {a:x,b:0,c:0,d:y,e:0,f:0}; }
ta.ident=       function() { return {a:1,b:0,c:0,d:1,e:0,f:0}; }
ta.clone=      function(t) { return {a:t.a,b:t.b,c:t.c,d:t.d,e:t.e,f:t.f}; }
ta.inverse=    function(t) { var det=t.a*t.d-t.c*t.b; return {a:t.d/det, b:-t.b/det, c:-t.c/det,
d:t.a/det, e:(t.c*t.f-t.e*t.d)/det, f:(t.e*t.b-t.a*t.f)/det}; }
ta.interp=function(t1,t2,i) { var t={}, a=['a','b','c','d','e','f'];
for (var j=0,k;k=a[j];j++) t[k]=t1[k]+(t2[k]-t1[k])*i; return t; }
window.Make2DContextQueryable = function(ctx) {
if ( ctx.contextToCanvas ) return;
var transforms = [ta.ident()];
function current() { return transforms[transforms.length-1]; }
function set(t) { transforms[transforms.length-1] = t; }
function apply_to(t) { transforms[transforms.length-1] = ta.multiply(current(),t); }
function proxy_before(o,fn,f) { var oldf=o[fn]; o[fn]=function(){
f.apply(o,arguments); oldf.apply(o,arguments); };}
proxy_before(ctx,'save',function() { transforms.push(ta.clone(current())); });
proxy_before(ctx,'restore',function() { if (transforms.length>1) transforms.pop();
else transforms[0]=ident(); });
proxy_before(ctx,'scale',function(x,y) { apply_to(ta.scale(x,y)); });
proxy_before(ctx,'rotate',function(theta) { apply_to(ta.rotate(theta)); });
proxy_before(ctx,'translate',function(x,y) { apply_to(ta.translate(x,y)); });
proxy_before(ctx,'transform',function(a,b,c,d,e,f) { apply_to({a:a,b:b,c:c,d:d,e:e,f:f}); });
proxy_before(ctx,'setTransform',function(a,b,c,d,e,f) { set({a:a,b:b,c:c,d:d,e:e,f:f}); });
ctx.contextToCanvas = function(p) { return ta.project(p,current()); }
ctx.canvasToContext = function(p) { return ta.project(p,inverse(current())); }
ctx.currentTransform = function() { return current(); }
return ctx;
}
})();

And here’s how it’s used a revised version of the tree application (full source):

function init() {
var canvas = document.getElementById('display');
// call Make2DContextQueryable to add our transform recorder
// functions.  Although the function returns the context, the
// the modifications are made to the element itself, so we don't
// need to call the function again.
var ctx = Make2DContextQueryable(canvas.getContext('2d'));
ctx.font = FONT; ctx.strokeStyle = OVER_STROKE;
ctx.lineWidth = OVER_LINE_WIDTH;
tree.branches = gen_tree(1);
draw();
canvas.onmousemove = function(evt) {
evt = evt || window.event;
var canvas = document.getElementById('display');
var canvasX = evt.clientX - canvas.offsetLeft
+ document.body.scrollLeft,
canvasY = evt.clientY - canvas.offsetTop
+ document.body.scrollTop;
if ( over(canvas.getContext('2d'), tree, {x:canvasX, y:canvasY}) )
draw();
}
}
function over( ctx, branch, canvasCoords ) {
// tranform the mouse point into the branch's coordinates
// (branch tranform was saved in the middle of the draw function)
var ctxCoords = ctx.canvasToContext(canvasCoords,branch.transform),
changed = false,
text_width = ctx.measureText(PARTS[branch.type]).width;
// perform simple rectangle test to see if mouse is over branch
var is_over = (ctxCoords.x >= 0
&& ctxCoords.x <= text_width + LEFT_OFFSET
&& ctxCoords.y <= TOP_OFFSET && ctxCoords.y >= -30 );
if ( is_over != branch.over ) {
branch.over = is_over;
changed = true;
}
if ( ! branch.branches ) return changed;
return changed || over(ctx,branch.branches[1],canvasCoords)
|| over(ctx,branch.branches[0],canvasCoords);
}

The functions under the TransformArithmetic namespace are a by-product of the logic needed to replicate the context transforms, but they can be quite useful themselves. The second interactive tree demo (source) uses these functions to calculate a transform that puts the mouse-overed branch of the tree in the center of the screen. When you mouse over a branch, the application records the active transform, AT. If we wanted this branch to be drawn in the canvas’s coordinates, we would simply need to invert AT and apply it using the ctx.transform() method. We actually want to translate it to the center of the canvas, however.  Let AT be the product of that translation, TL, and all the branch transforms used to get the branch, TT. So what we really want is TL/TT (here ‘/’ means multiply by the inverse). Since AT=TL*TT, TT=(1/TL)*AT (remember order is important), so TL/((1/TL)*AT) is the transform we need. Using the fact that the inversion of a translation matrix is simply the negation of the translation factors e and f, we can create the transform with the following code:

var ta=TransformArithmetic;
// translate a little left of center
ctx.translate( 100, 300 );
// subtract out the translation from the active transform and invert it
var at = ta.inverse(ta.multiply({a:1,b:0,c:0,d:1,e:-100, f:-300},active_transform));
// apply the calculated matrix
ctx.transform( at.a, at.b, at.c, at.d, at.e, at.f );

The computations needed to determine whether the mouse is over a branch in this new coordinate system are a little more complicated, but can be calculated using a similar algebraic manipulation. The TransformArithmetic also contains an interp() function that will interpolate between two transforms, thus making animation possible.

Wrapping the transform functions may seem a little extreme, but it goes a long way towards simplifying interactive visualizations.  It’s also handy to have the transform math lying around.  I’ve been using it to create bezier curve functions that permit the second two control points to be drawn in a different coordinate system.

This is the second post in a series on creating custom interactive visualizations in canvas.  The first post is here.

The canvas API contains five methods (rotate, scale, translate, transform, and setTransform) used to transform the drawing context. We typically use the transform API when we want to rotate or scale some element of the visualization (especially text). In canvas, we don’t actually move the elements. Instead we transform into a new coordinate system where the element is no longer scaled, rotated or translated and then draw normally. The difference is subtle, and you can get by with ignoring it so long as you remember to execute the transforms before you draw.   But, once you embrace this way of thinking, transformed coordinates can be a powerful way to break down complicated drawing tasks into simple steps. In the remainder of the post I’ll describe how to use transforms effectively and how to overcome the difficulties involved in making transformed objects interactive.

Examples: (an HTML5 browser is required to view these)

The Basic Idea

Choosing the right coordinate system simplifies calculations.

In the interest map project I described in a previous post, I want to use bezier curves to draw arrows using my own formula for the ratios of the arrow length, arrow head length, and arrow head height. The x and y coordinates of the head and tail of the arrow are known, but using these to figure out the location of the sides of the arrow head and its control points require a lot of rather messy trigonometry. If we create a coordinate system where the tail of the arrow is at the origin and the head lies on the x axis (this still requires a little bit of trig), the coordinates simplify to {0,0}, {l-C,f(l)}, {l,0}, and {l-C,-f(l)} where C is the length of the arrow head and f(l) is a function that gives its height. Also notice that, after a transform, we can use the same code to draw the horizontal and vertical axes (so long as we can handle the difference in lengths).  Complex visualizations can usually be broken up into components like this that each have a natural coordinate system.  The trick then becomes calculating an appropriate transform.

A Bit About The Math

I’m not going to get deep into linear algebra or affine geometry, but I think a little knowledge of how transforms are implemented is required to use them effectively. Any change of coordinates can be represented by a single transformation matrix.  You don’t need to know what a matrix is so long as you understand that it’s kind of like a number in the sense that you can multiply it with points or other transformation matrices.  When you multiply a point from your transformed coordinates with a transformation matrix, you get another point that is transformed back into the original coordinate system.  The canvas context always has a transformation matrix, and whenever you call a drawing API method (e.g. fillRect, bezierCurveTo, arcTo, etc.) it multiplies each of the points by this matrix before it renders any pixels from the shape to the canvas.

When you multiply a transformation matrix by a another transformation matrix, you get a new transformation matrix that is a combination of the two.  For example, if you have a transformation, A, that represents a coordinate system rotated 45 degrees and another transformation, B, that represents a translation of the origin 100 pixels along the x axis, then C=A*B is a transformation where the origin is 100 pixels out on the 45 degree line.  A lot of the power of Canvas’s transformation API derives from the ability to combine transforms like this.  With the exception of setTransform(), all of Canvas’s transform methods (rotate, scale, translate, and transform) are applied by creating a matrix representing the operation and then multiplying it with its current transform to create its new transform.  One important different between matrix multiplication and normal multiplication in that matrix multiplication does not commute (A*B != B*A).  So you’ll get a different result if call context.rotate() and then context.translate() than you would if you called context.translate() then context.rotate().

Save and Restore

The save() and restore() methods in canvas are a convenient way to avoid specifying stroke and fill styles over and over again in a drawing, but they become critical when you start using transforms.  The save() method takes all the current state of the canvas context including the current transform and pushes it on a stack.  You can then make as many modifications or transforms as you like.  When you’re done drawing, you call context.restore() and everything is back to the way it was before you called save().  You can always reset the fillStyle or lineWidth, but there is no trivial way to reset the transform (you can use setTransform() if you really know what you’re doing).  Using save() and restore() is habit you want to acquire.

Some Concrete Examples

Alright, enough theory: Here’s some code. The first example is the obligatory fractal demonstration. 2D transforms make quick work of fractals, because, by definition, they are the same thing over and over at different scales and in different places. At any rate, the application first creates a binary tree (see the full source). Then, beginning with the trunk it draws the text at the origin (in its transformed coordinates). Then, for each branch, it translates the origin to the end of the word, rotates in the direction of the next branch, and scales to the size of the branch. It then repeats the process until it gets to a leaf.

function draw() {
    var ctx = document.getElementById('display').getContext('2d');
    ctx.font = 'bold 30pt Georgia,Times New Roman';
    ctx.translate( ctx.canvas.width / 2, ctx.canvas.height );
    ctx.rotate( -Math.PI/2 );
    draw_branch( ctx, tree );
}
function draw_branch(ctx, branch) {
  ctx.fillStyle = branch.branches ? WOOD_COLOR : LEAF_COLOR;
  // draw text in default font
  ctx.fillText(PARTS[branch.type], LEFT_OFFSET, TOP_OFFSET);
  if ( ! branch.branches ) return;
  for ( var i=0,b; b=branch.branches[i]; i++ ) {
    ctx.save();   // IMPORTANT!!
    // move origin to end of text
    var text_width = ctx.measureText(PARTS[branch.type]).width;
    ctx.translate(text_width + LEFT_OFFSET, 0);
    // scale coordinates to predetermined value for branch
    ctx.scale( b.scale, b.scale );
    ctx.rotate( b.theta );   // rotate axes to be in line with branch
    draw_branch(ctx, b);
    ctx.restore();   // IMPORTANT!!
  }
}

The name tags in the interest map (source) presented a more realistic problem. We wanted to simulate the way people attached their name tags on the skill cards. The placement of each name tag was essentially random except that the tags were always near the edge of the card and they were always nearly horizontal. This requires 4 transforms: A translation to the center of the skill card (A), A rotation to give it a unique position on the card (B), A translation to the edge of the card with a random messiness factor (C) and a rotation back that’s roughly equal to B so the text is roughly horizontal (D).  The following is the (somewhat simplified) function that transforms the coordinates then draws the box and the text around the origin.  In this function, the angle of the name tag on the card, r, is given.

function nametag( ctx, cardx, cardy, person, r ) {
  var text_width = ctx.measureText(person.name).width,
        width = text_width+10;
  ctx.save();
  // make center of card the origin
  ctx.translate( cardx, cardy );
  // rotate in direction of tag placement on card
  ctx.rotate( r );
  // translate to edge of card (with a little permutation)
  ctx.translate( -TAG_RADIUS -  + 4*Math.random(), 0 );
  // rotate back with permutation so that text is nearly horizontal
  ctx.rotate( -r + .4*(Math.random()-.5) );
  /* ... set box style parameters ...  */
  // draw box centered around (transformed) origin
  ctx.beginPath(); ctx.rect( -width/2, -10, width, 20 ); ctx.closePath();
  ctx.fill();
  ctx.stroke();
  /* ... set font style ... */
  // draw text centered around (transformed) origin
  ctx.beginPath(); ctx.fillText( person.name, text_width/2, 0 ); ctx.closePath();
  draw_nametag( ctx, person, interest );
  ctx.restore();  // DON'T FORGET!!!
}

Most of the time, thinking in terms of coordinate transforms will make your life a lot easier. The exceptions come when you need to relate shapes drawn in one coordinate system with shapes or points specified in another. This is especially true when you need make transformed elements interactive. I’ll talk more about this problem and some solutions in the next post.

So I’ve spent the last few weeks building Chrono.js, an application metrics server. Chrono is still under development so it’s really not appropriate for general usage but it makes for a decent example app.

Sample Request

Here’s a basic request which fetches data from MongoDB and renders it via jqtpl:

app.get('/', function(req, res) {
  db.open(function(err, ignored) {
    if (err) console.log(err);
    db.collectionNames(function(err, names) {
      if (err) console.log(err);
      db.close();
      pro = _(names).chain()
        .pluck('name')
        .map(function(x) { return x.substr(x.indexOf('.') + 1); })
        .reject(function(x) { return (x.indexOf('system.') == 0); })
        .value();
      res.render('index.html.jqtpl', {
        locals: {
          metrics: pro
        }
      });
    });
  });
});

We are scanning MongoDB for the set of metric collections and displaying them in the selector so the user can switch between them. See the main source file for more examples about parameter handling, headers, JSON, etc.

Installation

You will need MongoDB installed and running locally. You should already have Node.js and npm installed from Part I. Let’s install the javascript libraries required and Chrono itself:

npm install express expresso jqtpl mongodb underscore tobi
git clone git://github.com/mperham/chrono.js.git

Express is a lightweight application server, similar to Sinatra. jqtpl is jQuery Templates, which allows us to dynamically render HTML. mongodb is the MongoDB client driver for Node.js. expresso is a TDD framework and tobi is an HTTP functional testing package. Finally, underscore is a great little utility library which provides JavaScript with a much more sane functional programming API.

Running

You can run the tests:

expresso

or run the Chrono.js server:

node chrono.js

Click http://localhost:3000/load/registrations to load in some fake data and visit http://localhost:3000 (select ‘registrations’) to see the results, graphed in living color for you!

Testing

Expresso gives us a simple DSL for testing our endpoints but I found it to be lacking basic setup/teardown functionality which I had to roll myself in order to insert some test data. I would have preferred to use vows.js but it appears to be incompatible with tobi. Check out the Chrono.js test suite for what I came up with, here’s a small sample:

  exports['POST /metrics'] = function() {
    assert.response(app,
      { url: '/metrics', method: 'POST', headers: { 'content-type': 'application/json' }, data: JSON.stringify({ k: 'registrations', v: 4, at: parseInt(Number(new Date())/1000) }) },
      { status: 201, headers: { 'Content-Type': 'text/html; charset=utf8' }, body: '' }
    );
  }

With expresso, we export the set of functions to run from our test module. Expresso runs them all in parallel and collects the results. The parallelism means that you must be careful with any test data you create. Since MongoDB doesn’t support transactions, we can’t use transactions to isolate each of our tests (e.g. see Rails’s transactional fixtures) so you need to be careful about the data created or deleted by each test and how you assert the current state.

Conclusion

While I’ve gotten much better in the last week, I’ll admit I’m still uncomfortable with Node.js. Its asynchronous semantics mean that your code runs as part of a chain of callbacks; knowing when, where and how that chain works is still difficult for me to understand. Frequently you’ll just get a black screen and a process that won’t exit or an error message that may or may not be related to the actual bug.

That said, I still remember being very frustrated with Ruby, its frequent use of “magic” and poor documentation (e.g. navigating this still befuddles me). I overcame those issues to be very comfortable with Ruby in general; I hope more time with Node.js will give me the same relief.

Google’s V8 JavaScript engine provides the underlying VM for executing JavaScript code. Since V8 is typically embedded in a browser, it does not expose any real notion of FileSystems, Processes and I/O streams that are fundamental to a server-side application. Node.js provides core APIs that wrap the standard POSIX functions available on all modern Unixy systems, along with higher-level APIs for HTTP/S support.

Node.js is designed to be an asynchronous system: your application code executes only on a single thread and no blocking I/O is allowed. Instead, Node.js uses an event loop to process I/O events via the libev C library.

Let’s dive into the Node code to see how it initializes itself. This writeup is current as of 0.4.2 or 3/14/2011 but obviously things will change over time.

Process Initialization

Node initializes itself in the node::Start method in node.cc. node::Start performs the housekeeping necessary to use V8 and libev in a well-behaved process. A brief walkthru:

• Line 2240-2275: Parse the cmd line arguments, dividing them up between node and v8.
• Line 2280: Setup signal handlers so Ctrl-C will actually terminate the Node.js process.
• Line 2290-2330: Configure various libev watchers, include the Idle detector which uses a simple hueristic to determine when it would be best for V8 to run garbage collection.
• Line 2332: Initialize V8!
• Line 2338-2364: Debugging support
• Line 2367-2374: Call node::Load to bootstrap the Node.js APIs, see below
• Line 2385: The key line: pass control to libev event loop. This function will not return until the application code decides to exit the process based on some event or SIGINT or SIGTERM are received.
• Line 2389-2405: Process is exiting. Execute any “exit” event callbacks in the application code and return.

Bootstrap

node::Load performs the actual Node.js API bootstrap in three stages:

• Stage 1: Using C++, it sets up a process object and fills it with various constants and methods, so your JavaScript code can access “process.argv”, “process.env”, etc. As far as I can tell, the most fundamental part is the Process.binding function which allows JS code to lazily bootstrap the C++ bindings. For instance, this line to fetch the ‘fs’ binding actually causes the code within node_file.cc to expose various POSIX file functions to the JavaScript module.
• Stage 2: It executes the JavaScript in src/node.js. This code sets up the various I/O streams (stdin, stdout and stderr) and creates the JavaScript module loading system which knows how to find JavaScript files in lib/ when require(module) is called.
• Stage 3: Node.js executes the user’s code, via debugger, repl or user-provided script file. Remember this script doesn’t block so it will load quickly and return back to node::Load.

Once the user’s code has been loaded, we should have set up any listening sockets, timers, etc. The entire node::Load function unwinds and we enter the event loop, waiting for I/O or Timer events to execute code.

Thanks for staying with me this far – I hope you learned something new. Next up, I’ll show you how to configure and run a full stack application on node.js using MongoDB and Express!

For this post, I’m just going to cover the basics; I’ll follow up soon with deeper posts.

Overview

JavaScript has an interesting history – it hasn’t developed like most other languages; until recently, executing JavaScript meant embedding it in a web page for a browser to execute. A few things happened which radically hastened the rise in JavaScript as an reasonable server-side language:

• AJAX and the Browser Wars have resulted in dramatic improvements in Javascript runtime performance and high-quality developer tools.
• Node.js built Process, File and Network I/O APIs on top of Google’s V8 JavaScript engine, allowing command line programs and daemons to be built in JavaScript.

Node.js adds a friendly command line face to V8 and APIs that are conceptually similar to Ruby’s EventMachine library: all I/O is asynchronous and threads are unavailable to user code. Additionally JavaScript is a prototype-based language, not object-oriented. This makes for a programming model that is radically different from what Ruby or Java developers are used to.

Installation

I’m going to assume OSX and I like to install things with Homebrew. We’ll install node and npm, node’s package manager, with these commands:

    brew update # update Homebrew's formulas to the latest
    brew install node # install node
    curl http://npmjs.org/install.sh | sudo sh # install npm

Once installed, you should be able to run node --help and npm --help.

A minimal web server using Node.js:

var http = require('http');
http.createServer(function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('Hello Worldn');
}).listen(8124, "127.0.0.1");

Copy that code into hello.js and run it:

node hello.js

Now let’s slam it with some requests:

ab -n 10000 -c 50 http://127.0.0.1:8124/

Results:

Server Hostname:        127.0.0.1
Server Port:            8124
Document Path:          /
Document Length:        12 bytes
Concurrency Level:      50
Time taken for tests:   1.479 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      760000 bytes
HTML transferred:       120000 bytes
Requests per second:    6760.79 [#/sec] (mean)
Time per request:       7.396 [ms] (mean)
Time per request:       0.148 [ms] (mean, across all concurrent requests)
Transfer rate:          501.78 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       3
Processing:     1    7   3.6      7      20
Waiting:        1    7   3.6      7      20
Total:          1    7   3.6      7      22

Percentage of the requests served within a certain time (ms)
  50%      7
  66%      9
  75%     10
  80%     11
  90%     12
  95%     13
  98%     15
  99%     16
 100%     22 (longest request)

Not bad. Of course, this is using localhost and a trivial app but at least we know it’s up and running well. In my next post, we’ll explore the Node.js source code itself.

Our goal was to create a visualization that would focus in on individual and collective patterns in the skills and interests chosen while still inviting people to explore and draw new conclusions.  It was important to leave some aspects of the interpretation to the group that created it, so we knew we needed an interactive visualization (the opportunity to explore new HTML5 influenced me here).

The result of this effort (this page requires an HTML5 compatible browser) is a single page canvas application that plots Carbon Fives skills and interests along various axes.  In the rest of the article I’ll explain how we converted the data to something more meaningful.

Identifying Meaning

Staring at the wall filled with skill cards and covered with name tags, it was tempting to try to make sense of it all by asking why a certain color seemed to be bunched up in a corner or why others seemed more scattered.  The cards were arbitrarily distributed, however, so any abnormalities were the product of statistical fluctuation.  If we really wanted to interpret the data as a two dimensional graph, we would have to rearrange the cards in such a way that their position on the on the wall had meaning.

So we began identifying opposing qualities that were applicable to all the skills in varying degrees.  In the end, we came up with “‘server side / client side’, ‘technical / creative’, ‘c5 core / c5 expansion’, ‘public facing / in house’, ‘trendy / old school’.  We then sat down together and tried to figure out where to place each skill on a scale where, say, ‘public facing’ would be zero and ‘in house’ would be a one.   The key to this exercise was to talk about where each skill falls relative to the other skills we’ve already discussed and to keep it quick.  I suspect we would get different orderings if we tried again, but the things that are obviously one thing or the other would still migrate towards the ends, and difference in the rest gets averaged out in the calculations anyway.

Once we had multiple orderings for the skills, we had to digitize the location of all the name tags.  We did this by assigning a number to each of the skill cards, and then writing down that number in either a skill column or an interest column next to the name of each person who put a name tag on that card. This is close to how the data is represented in the application.  It took a couple of passes to get the data right and we had to fill in a few gaps for employees that missed out on the exercise, but eventually I had all the data I needed to create the visualization.

Designing the Visualization

The arrows start at the average location of a person's skills and end at the average location of his or her interests.

The visualization itself is designed to look as much like the the wall as possible, while taking into account the limitations of legibility and  vector graphics.  When the application starts, it shows all the cards — each with the appropriate name tags — ordered horizontally according to whether the skill is more of a client-side or server-side skill, and vertically by whether the skill is more technical or creative.  Clicking the arrows next to the axis labels switches that axis to a new dichotomy and reorders the cards accordingly.  This view of the data is mainly to allow you to view the actual data.  If you click on the ‘show arrows’ button in the top right, you get a view more conducive to story telling.  Each of the arrows represents a Carbon Five employee.  The arrow starts at the average location of all of the person’s skills, and points to the average location of all of the person’s interests.  In a sense, it’s visualizing the direction of the person’s professional development.

You can also change the axes in the arrows view, and doing so brings up some interesting contrasts.  In some combinations, almost all the arrows are pointing in the same direction, and in others they all criss-cross.  In most combinations, all the arrows are in a pack, but in some there are a few arrows clustered together but away from the pack.  The interpretation of these phenomena depend on the axis in question.   The data has inaccuracies and biases built in at every point, but it has still been interesting to spot some feature in the visualization, interpret what it means and then check the names of the people behind the arrows.  In most cases the characterization highlights something that is unique about one of the Carbon Five team.

Building Your Own Skill Map

Aside from the choice of skill cards, there is nothing about  the exercise or the application that is peculiar to Carbon Five.  If you can get your company to generate their own skill map, you can use the source to create your own visualization by changing the data section in the top of the application.  In fact, I would be willing to do this for you if you can send me the data in some sort of reasonable digital format (limit the first 5 takers within the next year).

In the next post I will be discussing some of the programming challenges involved in writing interactive canvas applications.

update:

The second post on using transforms effectively in canvas applications is up.

I immediately added nesting of rules, similar to how SASS lets you dry-up your stylesheets. Then, with a little help from Alex, I added basic color conversions and functions like “darken” and “saturate”(demo here). I then added support for macros, so you could mix-in behaviors like rounded corners, drop shadows, and of course the ubiquitous (and awkwardly named) “clearfix”. My last pass was to make sure there are reasonable extension points for other developers. Now you can write concise and expressive CSS right along with your Javascript.

Since it’s POJS (Plain Old Javascript), it is a compact language. There’s was no need for me to write math or looping or logic functions, since Javascript’s already got them (unlike SASS). DSLs are the hottest things these days, but even the not-so-forgiving syntax of Javascript works pretty well in this case.

Why would you want to use this? Well, I thought that was the big question, but everyone I show it to has an answer. For me, I am writing pages that generate lots of their HTML markup anyway, so it is awkward to keep a separate CSS file in sync (and drag it down from the server). It works well to write self-contained jQuery plugins, as I sketched out here. I started using on a new initiative (that I can’t share yet) of my latest project, and it made prototyping and initial build-out much faster. It also makes using sprites painless– all our subtitles are in one sprite and I simply add a has: subtitle("Emotion") as one of the attributes and it works. (Previously I’d written some Ruby code to generate a CSS. Ughh.)

Please, give it a spin, and file bugs or send me comments on github.