Partial image manipulation with canvas and webworkers

In my previous article I showed how to use box2d-web and deviceorientation to rol around a set of circles using standard HTML5 APIs. My initial goal of that article was to load in an image, translate it to a series of seperate circles, and let you play around with that. But that’s for later. For now I’ll show you how you can use canvas together with web workers to offload heavy computing functions to a background thread.

For this example we’ll take an input image and manipulate the image in blocks of 10x10 pixels. For each block we’ll calculate the dominating color and render a rectangle on our target in that color. Since an example says more then a thousand words, we’re going to create this:

For those wondering, that’s my daugther thinking very hard about something. You can see this demo in action here.

Getting started

To implement this example we don’t need to do that much. We just have to take the following steps:

1. Wait until the image is loaded.
2. Split the image into seperate parts ready for processing.
3. Configure a web worker to start processing when it receives a message
4. Calculate the dominating color from our image sample.
5. Render a rectangle on our target canvas

Let’s begin simple, and look at the image loading code.

Wait for image to be loaded

Before we start processing the image we have to make sure it is completely loaded. For this we use the following piece of code:

    // start processing when the document is loaded.
    $(document).ready(function () {

        // handles rendering the elements
        setupWorker();

        // wait for the image to be loaded, before we start processing it.
        $("#source").load(function () {

            // determine size of image
            var imgwidth = $(this).width();
            var imgheight = $(this).height();

            // create a canvas and make context available
            var targetCanvas = createTargetCanvas(imgwidth, imgheight);
            targetContext = targetCanvas.getContext("2d");

            // render elements
            renderElements(imgwidth, imgheight, $(this).get()[0]);
        });
    });

As you can see here, we register the jquery ready function first. This will trigger when the complete document is loaded. This however doesn’t have to mean that the images have also already been loaded. To make sure the image is ready to be processed, we add the jquery load function to our source image (has id of #source). When the image is loaded we determine the required size of our target canvas, on which we render the result, and fire of the rendering using the renderElements function. The renderElements function splits the image and fires of the webworkers.

Split the image into seperate parts ready for processing

The goal of this example is to create a kind of low pixel effect on our source image. We do this by selecting part of the image, calculate the dominating color, and render a square on the target canvas. The following code shows how you can use a temporary canvas to select part of the image.

    // process the image by splitting it in parts and sending it to the worker
    function renderElements(imgwidth, imgheight, image) {
        // determine image grid size
        var nrX = Math.round(imgwidth / bulletSize);
        var nrY = Math.round(imgheight / bulletSize);

        // iterate through all the parts of the image
        for (var x = 0; x < nrX; x++) {
            for (var y = 0; y < nrX; y++) {
                // create a canvas element we use for temporary rendering
                var canvas2 = document.createElement('canvas');
                canvas2.width = bulletSize;
                canvas2.height = bulletSize;
                var context2 = canvas2.getContext('2d');
                // render part of the image for which we want to determine the dominant color
                context2.drawImage(image, x * bulletSize, y * bulletSize, bulletSize, bulletSize, 0, 0, bulletSize, bulletSize);

                // get the data from the image
                var data = context2.getImageData(0, 0, bulletSize, bulletSize).data
                // convert data, which is a canvas pixel array, to a normal array
                // since we can't send the canvas array to a webworker
                var dataAsArray = [];
                for (var i = 0; i < data.length; i++) {
                    dataAsArray.push(data[i]);
                }

                // create a workpackage
                var wp = new workPackage();
                wp.colors = 5;
                wp.data = dataAsArray;
                wp.pixelCount = bulletSize * bulletSize;
                wp.x = x;
                wp.y = y;

                // send to our worker.
                worker.postMessage(wp);
            }
        }
    }

In this function we first determine in how many rows and columns we’re going to split up the image. We iterate over each of these elements and render that specific part of the image on a temporary canvas. From that canvas we get the data using the getImageData function. At this point we’ve got all the information we need for our worker to calculate the dominating color (this is an expensive operation). We store the info in a ‘workpackage’:

    function workPackage() {
        this.data = [];
        this.pixelCount = 0;
        this.colors = 0;
        this.x = 0;
        this.y = 0;

        this.result = [0, 0, 0];
    }

This is a convience class that serves as the message to and from our webworker. Note that we need to convert the result from the getImageData call to a normal array. Information to a webworker is copied, and chrome at least isn’t able to copy the resulting array from the getImageData operation. So far so good. We now have nice workpackages for each part of our screen, which we pass to a webworker using the worker.postMessage operation. But what does this worker look like, and how do we configure it?

Configure a web worker to start processing when it receives a message

We create the worker in the setupWorker operation that is called when our document is loaded.

    function setupWorker() {
        worker = new Worker('extractMainColor.js');
        worker.addEventListener('message', function (event) {

            // the workpackage contains the results
            var wp = event.data;

            // get the colors
            var colors = wp.result;

            drawRectangle(targetContext, wp.x, wp.y, bulletSize, colors[0]);
            //drawCircle(targetContext, wp.x, wp.y, bulletSize, colors[0]);

        }, false);
    }

Creating a worker, as you can see, is very simple. Just point the worker to the javascript he needs to execute. Note that there are all kind of restrictions with regards to the resources and objects a worker has access to. A good introduction to what can and what can’t be accessed can be found in this article. Once we defined the worker, we add an eventListener. This listener is called when the worker uses the postMessage operation. In our example this is used to pass the result back in the same workpackage. Based on this result we draw a rectangle (or some other figure) on our target canvas. The worker itself is very basic:

importScripts('quantize.js' , 'color-thief.js');

self.onmessage = function(event) {

    var wp = event.data;
    var foundColor = createPaletteFromCanvas(wp.data,wp.pixelCount, wp.colors);
    wp.result = foundColor;
    self.postMessage(wp);

};

This worker uses two external scripts to calculate and return the dominating color. It does this by getting the required information from the workpackage, calculate the dominating color, and return the result in the workpackage using the postMessage. Calculating the dominating color itself isn’t that easy. I ran across a great library named color-thief, that does this for you. Apparently you need to take more into account than just the RGB values, if you do that then you just get a set of brown colors.

Calculate the dominating color from our image sample.

I mentioned that I used the color-thief library to calculate the dominating color. I do this using this code:

createPaletteFromCanvas(wp.data,wp.pixelCount, wp.colors);

This, however, isn’t directly provided by color-thief. Color-thief assumes you want to use it directly on an image element on your page. I had to extend the color-thief library with the following simple operation so that it can work directly with binary data.

function createPaletteFromCanvas(pixels, pixelCount, colorCount) {

    // Store the RGB values in an array format suitable for quantize function
    var pixelArray = [];
    for (var i = 0, offset, r, g, b, a; i < pixelCount; i++) {
        offset = i * 4;
        r = pixels[offset + 0];
        g = pixels[offset + 1];
        b = pixels[offset + 2];
        a = pixels[offset + 3];
        // If pixel is mostly opaque and not white
        if (a >= 125) {
            if (!(r > 250 && g > 250 && b > 250)) {
                pixelArray.push([r, g, b]);
            }
        }
    }

    // Send array to quantize function which clusters values
    // using median cut algorithm

    var cmap = MMCQ.quantize(pixelArray, colorCount);
    var palette = cmap.palette();

    return palette;
}

This returns an array of most dominating colors (just as the normal color-thief functions do) but can work directly on the data from our worker.

Render a rectangle on our target canvas

And that’s pretty much it. At this point we’ve split our image into an array of subimages. Each part is sent to a webworker for processing. The webworker processes the image and passes the result back to our eventhandler. In the eventhandler we take the most dominating color and we can use that to draw on the canvas. In the figure at the beginning of this article I used rectangles:

Using this javascript (and with a bulletsize of 15):

    // draw a rectangle on the supplied context
    function drawRectangle(targetContext, x, y, bulletSize, colors) {
        targetContext.beginPath();
        targetContext.rect(x * bulletSize, y * bulletSize, bulletSize, bulletSize);
        targetContext.fillStyle = "rgba(" + colors + ",1)";
        targetContext.fill();
    }

But we could just as easily render circles:

Using this:

    // draw a circle on the supplied context
    function drawCircle(targetContext, x, y, bulletSize, colors) {
        var centerX = x * bulletSize + bulletSize / 2;
        var centerY = y * bulletSize + bulletSize / 2;
        var radius = bulletSize / 2;

        targetContext.beginPath();
        targetContext.arc(centerX, centerY, radius, 0, 2 * Math.PI, false);
        targetContext.fillStyle = "rgba(" + colors + ",1)";
        targetContext.fill();
    }

As you can see web workers are really easy to use, and canvas allows us much options to work with imagedata. In this example I only used a single web worker, more interesting would be to add a queue on which multiple workers would listen to really process elements in parallel. The demo and complete code for this article can be found here.