Putting Point Cloud into AR with Kinect and Processing

Setting up • Kinect for Windows SDK v1.8 https://www.microsoft.com/en-us/download/details.aspx?id=40278 • Processing 2.2.1 https://processing.org/download/ Unzip and put following folders into libraries of Processing (C:¥Users¥YOU¥Documents¥Processing¥ libraries) • SimpleOpenNI 1.96 https://code.google.com/archive/p/simple-openni/downloads • NyARToolKit (nyar4psg.zip) https://github.com/nyatla/NyARToolkit-for-Processing/releases Download and unzip following file into any folder. https://drive.google.com/file/d/1ahzAzp_s5gqg5WbfJkF5JSv5v87qm KZv/view?usp=sharing

Goal https://youtu.be/WeorvV9vZh8

Initialization of Kinect import SimpleOpenNI.*; //Load SimpleOpenNI SimpleOpenNI kinect; //Variable to access Kinect void setup(){ //Insatiate and initialize Kinect. kinect = new SimpleOpenNI(this); if(kinect.isInit() == false){ exit(); return; } }  Run processing.  Write following code.  Ctrl + s to save this code.

Run Application IR projection will started

Preparation of using NyARToolKit Move camera_para.dat & patt.sample1 from Sample to the same directory of your code. 【NyARToolKit】 【Sample3】

Preparation of using NyARToolKit import SimpleOpenNI.*; import jp.nyatla.nyar4psg.*; //Import NyARToolKit SimpleOpenNI kinect; MultiMarker ar; //Variable to access ARToolKit void setup(){ size(640,480,P3D); /*Initialization of Kinect (Abb)*/ ar = new MultiMarker( this, width, height, "camera_para.dat" ,NyAR4PsgConfig.CONFIG_PSG_PV221); } Window size Camera parameter

Show Color Image with NyARToolKit void draw(){ //Update Kinect kinect.update(); //Grab current color image. PImage rgbImage = kinect.rgbImage(); //Draw color image as background. ar.drawBackground(rgbImage); } void setup(){ /*Initialization of Kinect(Abb)*/ ar = new MultiMarker(this, width, /*Abb*/); kinect.enableRGB(); //Enable acquiring color image }

Marker Recognition with ARToolKit (1/2) void setup(){ /*Initialization of Kinect(Abb)*/ ar = new MultiMarker(this, width, /*Abb*/); //Load marker file (width is 80mm) ar.addARMarker("patt.sample1", 80); kinect.enableRGB(); } 【ARToolKit】  Using square frame and inner figure.  Do not use rotation symmetric figure.  You can use pre-defined markers if original ARToolkit is downloaded.

Marker Recognition with ARToolKit (2/2) void draw(){ kinect.update(); PImage rgbImage = kinect.rgbImage(); ar.drawBackground(rgbImage); } ar.detect(rgbImage); //Marker detection. if(!ar.isExistMarker(0)){ return; //Return if no marker is detected } ar.beginTransform(0); //Put on following object on marker fill(0,0,255); box(40); //Draw box as a test. ar.endTransform();

Run application It’s not on marker

Modification of CG’s Position ar.beginTransform(0); translate(0,0,20); fill(0,0,255); box(40); ar.endTransform(); Translate box to upper. x Y Z (0,0,0) (0,0,20) 40mm

Acquireing Depth Data void setup(){ /*Initialization of Kinect (Abb)*/ kinect.enableRGB(); kinect.enableDepth(); //Enable Depth Image } void draw(){ kinect.update(); PImage rgbImage = kinect.rgbImage(); ar.drawBackground(rgbImage); /* following code is abbreviated in this slide */ }

Acquiring Point Cloud Data void draw(){ kinect.update(); PImage rgbImage = kinect.rgbImage(); //Acqureing point cloud. //Point clouds is in 1D array. PVector[] realWorld = kinect.depthMapRealWorld(); /*以下省略*/ pixelと濃淡 3D 実空間

Drawing Point Cloud (1/2) void draw(){ /*Acquiring image from kinect (省略)*/ /*Marker detection (Abb.)*/ ar.beginTransform(0); // translate(0,0,20); fill(0,0,255); box(40); ar.endTransform(); } beginShape(POINTS); //Draw points of each position (Shown in the next page) endShape(); translate(0,0,20); fill(0,0,255); box(40); Remove Box Add

Drawing Point Cloud (2/2) beginShape(POINTS); endShape(); //Ａｄｄ ｆｏｌｌｏｗｎｉｎｇ ｃｏｄｅ for(int v=0; v0) { vertex(point.x, point.y, point.z); } } } X Y Z

Run Application Point are shown but not easy to see. Yellow points

Modification of Drawing Setting of Point Cloud scale(0.3); //shrink whole size of point cloud translate(0,0,-600); //Translate point cloud beginShape(POINTS); for(int v=0; v0) { vertex(point.x, point.y, point.z); } } } X Y Z

Modification of Each Point Size strokeWeight(2); //Draw as 2 pix point. scale(0.3); translate(0,0,-600); beginShape(POINTS); for(int v=0; v

Rotate Point Clouds strokeWeight(2); scale(0.3); //Rotate 90 degree(=π/2) to X axis. rotate(PI/2, 1, 0, 0); translate(0,0,-600); beginShape(POINTS); 回転

Remove Far Point Cloud beginShape(POINTS); for(int v=0; v0) { vertex(point.x, point.y, point.z); } } } endShape(); if(point.z>0 && point.z<1500) { vertex(point.x, point.y, point.z); } 除去 If distance<1500mm

Modify Position of Point Cloud Again 移動 strokeWeight(2); scale(0.3); rotate(PI/2, 1, 0, 0); translate(0,0,-600); //600mm後ろに戻す (場合による) beginShape(POINTS); for(int v=0; v

Assign Image Color to Point Cloud beginShape(POINTS); for(int v=0; v0) { vertex(point.x, point.y, point.z); } } } endShape(); stroke(pixelColor);

Run Application Not much the position between point and color

 Position of color camera and depth camera are different → Pixel coordinate is not match each other. Reason of Mismatch Color Depth

Match Color and Depth Images void setup(){ size(640, 480, P3D); kinect = new SimpleOpenNI(this); if(kinect.isInit() == false){ exit(); return; } ar = new MultiMarker( this, width, height, "camera_para.dat"); ar.addARMarker("patt.sample1", 80); kinect.enableRGB(); kinect.enableDepth(); kinect.alternativeViewPointDepthToImage(); } Add this code

Finished

No content