Using Kinect for real-time mapping – Part II: Create a real-time mask from depth image: Processing

[This article describes a topic using processing.  There is an article on doing the same with open frameworks too.]

The kinect can do a lot of magic. It can not only provide a depthImage with distances for each point in the room it can see, it also does stuff like blob detection, decides on where people are, which parts of their bodies are visible, what gestures they make, and so on. This magic is usually done in Libraries. We only write one or two instructions in our code and »somehow« it is done then.

In this example we use the kinects depthImage (which is provided by a library, yes). The rest we do ourselves. For sometimes it is not necessary to use the complex stuff.

The idea: We want a real-time mask (Let’s say we want to map a projection on people or objects that move). Masks can e.g. be black/white images, so what we do is very easy: We define a maximum and a minimum distance and use the distance information, the kinect delivers, to create a simple black/white Image, that is updated in real-time. In the next chapters we will apply this image as our mask in different projects.

import SimpleOpenNI.*; 
SimpleOpenNI  kinect;

int distance = 1500;
int distance2 = 3000;

PImage liveMap;

void setup()
{  
  size(640, 480);
  kinect = new SimpleOpenNI(this);
  kinect.setMirror(false);
  kinect.enableDepth();
  liveMap = createImage(640, 480, RGB);
}

void draw()
{
  background(color(0,0,0));  
  kinect.update();
  int[] depthValues = kinect.depthMap();
  liveMap.width = 640;
  liveMap.height = 480;
  liveMap.loadPixels();
  for (int y=0; y<480; y++) {
    for (int x=0; x<640; x++) { int i= x+(y*640); int currentDepthValue = depthValues[i]; if (currentDepthValue>distance&&currentDepthValue<distance2) {
        liveMap.pixels[i] = color(255,255,255);  
      } else {
        liveMap.pixels[i] = color(0,0,0);            
      }
    }  
  }
  liveMap.updatePixels();
  image(liveMap,0,0);
}

Let’s go and check the code, step by step.
First we import the needed library. Second line inits the kinect object.

import SimpleOpenNI.*;
SimpleOpenNI  kinect;

Next we set 2 variables for minimum and maximum distance to adapt to the setting and room we are in. Then we define the PImage will we use as a map.

int distance = 1500;
int distance2 = 3000;
PImage liveMap;

Inside the setup function we set the size of our canvas. Then we initialize our kinect object and do some settings on it: Third line defines if the image of kinect is mirrored. If you need this or not, depends on your setting / projector settings. Fourth line enables the depth image:

void setup()
{
size(640, 480);
kinect = new SimpleOpenNI(this);
kinect.setMirror(false);
kinect.enableDepth();

Next we create an empty image. It will become our mask, so it is not important what’s inside: we will overwrite the content. Only the size should be correct. [Thanks Martin from stxle.com, for improving my code!]

liveMap = createImage(640, 480, RGB);

Now, inside draw function, we set background color to black, update the kinect object and then get the depthMap image. As you can see, what we get is an Array. It contains pixels, but their values do not represent colors (as in a jpg e.g.) but distances (from the kinect, to be precise). One distance value for each point the kinect can see. We’ll get back to this.

void draw()
{
background(color(0,0,0));
kinect.update();
int[] depthValues = kinect.depthMap();

Next we just prepare our mask image to be able to overwrite its pixels.

liveMap.width = 640;
liveMap.height = 480;
liveMap.loadPixels();

And now we start the interesting point. We have an array containing 307200 distance values. One after the other. But as we know our depht image consists of 640 columns and 480 lines, we also can figure out that for example the 641st distance value represents the first pixel in the second line. So we do just some Maths to run 480 times all 640 pixels in the appropriate row. Like this we run through each pixel:

for (int y=0; y<480; y++) {
for (int x=0; x<640; x++) {

We then calculate the number of the pixel in the array and get it’s distance value:

int i= x+(y*640);
int currentDepthValue = depthValues[i];

Now we chekck if the distance of this pixel lies in our limits (as set in the variables distance and distance2 on top of our code). If yes, we will set the appropriate pixel in our mask image (which is an Array, just like our depthImage) to white. If no, we will set it to black:

if (currentDepthValue>distance&&currentDepthValue<distance2) {
liveMap.pixels[i] = color(255,255,255);
} else {
liveMap.pixels[i] = color(0,0,0);
}

This is repeated for every pixel of the depth image. So in the end we have set every pixle of our mask image to either white or black.
[Advanced: Note that people who think Math may ask why we do not iterate through the array “depthValues” in the first place instead of doing the 480 time 640 pixels loop. They are completely right. It is strange and unneccessarily complex to do it like we do. But: Our reason is a) it teaches us better how pixels and images work in processing. b) If we should choose at one point in our project to do stuff like treating every second line (or comlumn) differently (compare deinterlacing in video processing) we still have all the possibilities.]
Then we only update our mask image (“liveMap”) and are ready to use it. In this first example we just draw it on the screen for you to see it:

liveMap.updatePixels();
image(liveMap,0,0);

And that’s already it.