Using Kinect for real-time mapping – Part III: Create a real-time mask from depth image: Processing

In the previous chapter we created a real-time mask image by using a kinect sensor and proessing. Now we use this image to map a video.
When placing your kinect and video projector so that they are in sync we will then be able to project only on people and objects and spare the background – even when everything is moving.
Below you find the complete code. Better even: download the sektch (with resources and some annotations in code).

import SimpleOpenNI.*; 
import processing.video.*;

SimpleOpenNI kinect;

Movie myMovie;
PImage liveMap;
PImage clonedImage;

int distance = 1500; 
int distance2 = 3000;

void setup()
{ 
  size(640, 480, P2D);
  kinect = new SimpleOpenNI(this);
  if (kinect.isInit() == false)
  {
    println("Can't init SimpleOpenNI, maybe the camera is not connected!"); 
    exit();
    return;
  }
  kinect.setMirror(false);
  kinect.enableDepth();
  myMovie = new Movie(this, "lava3.mp4");
  myMovie.loop();
  liveMap = createImage(640, 480, RGB);
}

void draw()
{
  kinect.update();
  int[] depthValues = kinect.depthMap();
  liveMap.width = 640;
  liveMap.height = 480;
  liveMap.loadPixels();

  for (int y=0; y<480; y++) {
    for (int x=0; x<640; x++) { 
      int i= x+(y*640); int currentDepthValue = depthValues[i]; 
      if (currentDepthValue>distance&&currentDepthValue<distance2) {
        liveMap.pixels[i] = color(255,255,255); 
      } else {
        liveMap.pixels[i] = color(0,0,0); 
      }
    } 
  }
  liveMap.updatePixels();
  liveMap.filter(BLUR, 2);
  image(myMovie,0,0);
  clonedImage = get();
  background(color(0,0,0));
  image(clonedImage,0,0);
  clonedImage.mask(liveMap);
}

void movieEvent(Movie m) {
  m.read();
} 

Again let’s check the code step by step (I’ll mark the parts that change from previous chapter as bold).
First we import the needed libraries. New library is for video. Third line inits the kinect object.

import SimpleOpenNI.*;
import processing.video.*;

SimpleOpenNI  kinect;  

next we define – this is new – a video and 2 images. Then we set our distances just like in the previous chapter.

Movie myMovie;
PImage liveMap;
PImage clonedImage;

int distance = 1500; 
int distance2 = 3000;

next, inside setup function, we do the same as in the previous chapter. There are only 3 differences. First, we set the renderer to P2D (this is needed for the masking to work. This may be different for different processing versions. Just try it out. For Information on renderer/rendering modes please check the processing web page.). As second change we insert an if condition for error reporting in case kinect is not found. This is not needed, just to show you. And third we load the movie we want to play and start it by setting it to loop.

void setup()
{
  size(640, 480, P2D);
  kinect = new SimpleOpenNI(this);
  if (kinect.isInit() == false)
  {
    println("Can't init SimpleOpenNI, maybe the camera is not connected!"); 
    exit();
    return;
  }
  kinect.setMirror(false);
  kinect.enableDepth();
  myMovie = new Movie(this, "lava3.mp4");
  myMovie.loop();
  liveMap = liveMap = createImage(640, 480, RGB);

Next is the block of creating the live mask image – exactly as we did in the previous chapter:

void draw()
{
  kinect.update();
  int[] depthValues = kinect.depthMap();
  liveMap.width = 640;
  liveMap.height = 480;
  liveMap.loadPixels();

  for (int y=0; y<480; y++) {
    for (int x=0; x<640; x++) { int i= x+(y*640); int currentDepthValue = depthValues[i]; if (currentDepthValue>distance&&currentDepthValue<distance2) {
        liveMap.pixels[i] = color(255,255,255); 
      } else {
        liveMap.pixels[i] = color(0,0,0); 
      }
    } 
  }
  liveMap.updatePixels();

to get a more smooth result we blur the mask image a little bit:

liveMap.filter(BLUR, 2);

In Theory processing should be able to mask the video. But many things related to video changed over versions, so I instead use a small hack: First we draw the video. Then we copy the screen (so the drawn video) in an empty PImage and clear the screen again. Now we draw the copied image, which we will be able to mask, hopefully without having to think about processing versions:

  image(myMovie,0,0);
  clonedImage = get();
  background(color(0,0,0));
  image(clonedImage,0,0);
  clonedImage.mask(liveMap);

In the end we need the following little function to make the video work:

void movieEvent(Movie m) {
  m.read();
}