Interactivity in HTML5 Canvas Visualizations

In the last canvas visualization post I discussed the canvas API’s transform functionality and how it greatly simplifies drawing complex visualizations.  In this post, I’ll talk a little about making canvas visualizations interactive and about problems you might encounter when mixing transforms and interactivity in a canvas application.

Examples: (an HTML5 compatible browser is required to view these)
Interest Map ApplicationTree ApplicationInteractive Tree ApplicationAnimated Interactive Tree

Tracking Mouse Events in Canvas

The Canvas API provides very little support for tracking user interactions within a canvas drawing.  You get the standard mouse events you get with any other DOM element (onclick, onmousemove, onmouseup, etc.), and you are more or less on your own from there.  To determine if the user has interacted with a particular part of your canvas visualization, you need to take the clientX and clientY from the mouse event and subtract the absolute top and left of the canvas element from them to get an x and y relative to the top left of the canvas.  The canvas’s position can be determined by something like jQuery’s offset() function, or you can calculate it directly by recursing through the canvas’s offsetParents and looking at their offsetLeft, offsetTop, scrollLeft and scrollTop properties (this can be simplified if your HTML has less structure).

Once you have an x and y in the canvas’s coordinates, you can check if the mouse event is in some rectangle by checking that x and y are between the rectangle’s edges.  If you need a more precise target, you can use the canvas isPointInPath() method by first drawing the target shape using the canvas drawing API and then checking context.isPointInPath(x,y).

The Problem With Interactivity and Transforms

This works fine until you start using transforms to draw your controls.  In the interest map project, I used the bounding box method to detect clicks on the arrow buttons on the right and left of the x axis.  I wanted to use the same code to check the buttons on the top and bottom of the y axis, but soon realized this wouldn’t work. Because the buttons are drawn in a rotated coordinate system, the coordinates I used to draw the buttons were no longer relative to the top left of the canvas, so they didn’t match the coordinates of the mouse event.  In this case the actual coordinates were relatively easy to calculate, but there are other interactive parts of the visualization whose canvas coordinates would be much more difficult to calculate.

To a certain extent, the isPointInPath() method works around this problem.  If you draw a shape in a coordinate system defined by any number of transforms and then call isPointInPath() using canvas coordinates, you’ll still get the desired result.  The downside of this approach is that you need to re-create the sequence of transforms used to draw the control and then draw the correct shape at the time you handle the mouse event. So you need to maintain a copy of at least some of your drawing logic in order to create your mouse targets in your event handlers.  I wasn’t very happy with this solution so I developed another one.

Make2DContextQueryable

The problem with the canvas transform API and interactivity is that it’s a one way street. You provide the directions, and canvas draws the right picture.  Once you need to figure out where all this transforming has gotten you, you’ve got no support from the API.  Make2DContextQueryable fills in this gap.

It’s no mystery how all the canvas transform methods combine to create a transform matrix, and once you have the the matrix, you only need to invert it (think 1/A) to create a transform that converts points in the original coordinates to the current transformed coordinates. Make2DContextQueryable wraps all of the canvas transform methods with functions that first apply the transform to our own matrix before calling the original method on the canvas context. The new matrix is stored in a stack so that save() and restore() work as expected.  While you’re drawing a control, you can call t = context.currrentTransform() to get and store its coordinate transform. Then in the event handler, you can either call context.canvasToContext({x:canvasX,y:canvasY}, t) to convert the mouse location to a point in the control’s coordinate system and then test it yourself, or call context.setTransform(t.a,t.b,t.c,t.d,t.e,t.f), draw a path around the click area (in the control’s coordinate system) and call isPointInPath(canvasX, canvasY).

Here’s the Make2DContextQueryable function definition:

(function () {
    window.TransformArithmetic = {};
    var ta = TransformArithmetic;
    ta.project = function (p, t) {
        return {
            x: t.a * p.x + t.c * p.y + t.e,
            y: t.b * p.x + t.d * p.y + t.f
        };
    };
    ta.multiply = function (t2, t1) {
        return {
            a: t1.a * t2.a + t1.b * t2.c,
            c: t1.c * t2.a + t1.d * t2.c,
            b: t1.a * t2.b + t1.b * t2.d,
            d: t1.c * t2.b + t1.d * t2.d,
            e: t1.e * t2.a + t1.f * t2.c + t2.e,
            f: t1.e * t2.b + t1.f * t2.d + t2.f
        };
    };
    ta.translate = function (x, y) {
        return {
            a: 1,
            b: 0,
            c: 0,
            d: 1,
            e: x,
            f: y
        };
    };
    ta.rotate = function (theta) {
        return {
            a: Math.cos(theta),
            b: Math.sin(theta),
            c: -Math.sin(theta),
            d: Math.cos(theta),
            e: 0,
            f: 0
        };
    }
    ta.scale = function (x, y) {
        return {
            a: x,
            b: 0,
            c: 0,
            d: y,
            e: 0,
            f: 0
        };
    }
    ta.ident = function () {
        return {
            a: 1,
            b: 0,
            c: 0,
            d: 1,
            e: 0,
            f: 0
        };
    }
    ta.clone = function (t) {
        return {
            a: t.a,
            b: t.b,
            c: t.c,
            d: t.d,
            e: t.e,
            f: t.f
        };
    }
    ta.inverse = function (t) {
        var det = t.a * t.d - t.c * t.b;
        return {
            a: t.d / det,
            b: -t.b / det,
            c: -t.c / det,
            d: t.a / det,
            e: (t.c * t.f - t.e * t.d) / det,
            f: (t.e * t.b - t.a * t.f) / det
        };
    }
    ta.interp = function (t1, t2, i) {
        var t = {}, a = ['a', 'b', 'c', 'd', 'e', 'f'];
        for (var j = 0, k; k = a[j]; j++) t[k] = t1[k] + (t2[k] - t1[k]) * i;
        return t;
    }
    window.Make2DContextQueryable = function (ctx) {
        if (ctx.contextToCanvas) return;
        var transforms = [ta.ident()];

        function current() {
            return transforms[transforms.length - 1];
        }

        function set(t) {
            transforms[transforms.length - 1] = t;
        }

        function apply_to(t) {
            transforms[transforms.length - 1] = ta.multiply(current(), t);
        }

        function proxy_before(o, fn, f) {
            var oldf = o[fn];
            o[fn] = function () {
                f.apply(o, arguments);
                oldf.apply(o, arguments);
            };
        }
        proxy_before(ctx, 'save', function () {
            transforms.push(ta.clone(current()));
        });
        proxy_before(ctx, 'restore', function () {
            if (transforms.length & gt; 1) transforms.pop();
            else transforms[0] = ident();
        });
        proxy_before(ctx, 'scale', function (x, y) {
            apply_to(ta.scale(x, y));
        });
        proxy_before(ctx, 'rotate', function (theta) {
            apply_to(ta.rotate(theta));
        });
        proxy_before(ctx, 'translate', function (x, y) {
            apply_to(ta.translate(x, y));
        });
        proxy_before(ctx, 'transform', function (a, b, c, d, e, f) {
            apply_to({
                a: a,
                b: b,
                c: c,
                d: d,
                e: e,
                f: f
            });
        });
        proxy_before(ctx, 'setTransform', function (a, b, c, d, e, f) {
            set({
                a: a,
                b: b,
                c: c,
                d: d,
                e: e,
                f: f
            });
        });
        ctx.contextToCanvas = function (p) {
            return ta.project(p, current());
        }
        ctx.canvasToContext = function (p) {
            return ta.project(p, inverse(current()));
        }
        ctx.currentTransform = function () {
            return current();
        }
        return ctx;
    }
})();

And here’s how it’s used a revised version of the tree application (full source):

function init() {
    var canvas = document.getElementById('display');
    // call Make2DContextQueryable to add our transform recorder
    // functions.  Although the function returns the context, the
    // the modifications are made to the element itself, so we do not
    // need to call the function again.
    var ctx = Make2DContextQueryable(canvas.getContext('2d'));
    ctx.font = FONT;
    ctx.strokeStyle = OVER_STROKE;
    ctx.lineWidth = OVER_LINE_WIDTH;
    tree.branches = gen_tree(1);
    draw();
    canvas.onmousemove = function (evt) {
        evt = evt || window.event;
        var canvas = document.getElementById('display');
        var canvasX = evt.clientX - canvas.offsetLeft + document.body.scrollLeft,
            canvasY = evt.clientY - canvas.offsetTop + document.body.scrollTop;
        if (over(canvas.getContext('2d'), tree, {
            x: canvasX,
            y: canvasY
        })) draw();
    }
}

function over(ctx, branch, canvasCoords) {
    // tranform the mouse point into the branch's coordinates
    // (branch tranform was saved in the middle of the draw function)
    var ctxCoords = ctx.canvasToContext(canvasCoords, branch.transform),
        changed = false,
        text_width = ctx.measureText(PARTS[branch.type]).width;
    // perform simple rectangle test to see if mouse is over branch
    var is_over = (ctxCoords.x & gt; = 0 & amp; & amp; ctxCoords.x & lt; = text_width + LEFT_OFFSET & amp; & amp; ctxCoords.y & lt; = TOP_OFFSET & amp; & amp; ctxCoords.y & gt; = -30);
    if (is_over != branch.over) {
        branch.over = is_over;
        changed = true;
    }
    if (!branch.branches) return changed;
    return changed || over(ctx, branch.branches[1], canvasCoords) || over(ctx, branch.branches[0], canvasCoords);
}

The functions under the TransformArithmetic namespace are a by-product of the logic needed to replicate the context transforms, but they can be quite useful themselves. The second interactive tree demo (source) uses these functions to calculate a transform that puts the mouse-overed branch of the tree in the center of the screen. When you mouse over a branch, the application records the active transform, AT. If we wanted this branch to be drawn in the canvas’s coordinates, we would simply need to invert AT and apply it using the ctx.transform() method. We actually want to translate it to the center of the canvas, however.  Let AT be the product of that translation, TL, and all the branch transforms used to get the branch, TT. So what we really want is TL/TT (here ‘/’ means multiply by the inverse). Since AT=TL*TT, TT=(1/TL)*AT (remember order is important), so TL/((1/TL)*AT) is the transform we need. Using the fact that the inversion of a translation matrix is simply the negation of the translation factors e and f, we can create the transform with the following code:

var ta = TransformArithmetic;
// translate a little left of center
ctx.translate(100, 300);
// subtract out the translation from the active transform and invert it
var at = ta.inverse(ta.multiply({
    a: 1,
    b: 0,
    c: 0,
    d: 1,
    e: -100,
    f: -300
}, active_transform));
// apply the calculated matrix
ctx.transform(at.a, at.b, at.c, at.d, at.e, at.f);

The computations needed to determine whether the mouse is over a branch in this new coordinate system are a little more complicated, but can be calculated using a similar algebraic manipulation. The TransformArithmetic also contains an interp() function that will interpolate between two transforms, thus making animation possible.

Wrapping the transform functions may seem a little extreme, but it goes a long way towards simplifying interactive visualizations.  It’s also handy to have the transform math lying around.  I’ve been using it to create bezier curve functions that permit the second two control points to be drawn in a different coordinate system.

About Alex Cruikshank

Alex is Carbon Five's resident mad genius. He is recently very involved in functional languages, and has been anchoring teams at C5 for the past decade.
This entry was posted in Design, Web and tagged , , , , . Bookmark the permalink.
  • Pingback: Interactivity in HTML5 Canvas Visualizations | Carbon Five Community | Web Scripts Maniacs

  • Pebanfield

    Nice visualizations! I’ve just started experimenting with canvas API. I’m surprised how little support there is for interactivity. Is there not another approach involving multiple canvas instances and div containers?

    Re – making 2d context queryable – I’m not yet understanding your approach. Are you describing something like a Ray Casting algorithm? I’m sorry I don’t find your code sample very readable.

    Perhaps it will make more sense once I’ve had more time to work through it myself. Cheers.

    Peter

  • http://twitter.com/03eltond Dave Elton

    Thanks, this was

  • Dave

    Thanks, this was extremely helpful in getting my mouse coordinates mapped to the context coordinates.