Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Putting Point Cloud into AR with Kinect and Processing

Putting Point Cloud into AR with Kinect and Processing

TakashiYoshinaga

October 21, 2018
Tweet

More Decks by TakashiYoshinaga

Other Decks in Technology

Transcript

  1. Setting up • Kinect for Windows SDK v1.8 https://www.microsoft.com/en-us/download/details.aspx?id=40278 •

    Processing 2.2.1 https://processing.org/download/ Unzip and put following folders into libraries of Processing (C:¥Users¥YOU¥Documents¥Processing¥ libraries) • SimpleOpenNI 1.96 https://code.google.com/archive/p/simple-openni/downloads • NyARToolKit (nyar4psg.zip) https://github.com/nyatla/NyARToolkit-for-Processing/releases Download and unzip following file into any folder. https://drive.google.com/file/d/1ahzAzp_s5gqg5WbfJkF5JSv5v87qm KZv/view?usp=sharing
  2. Initialization of Kinect import SimpleOpenNI.*; //Load SimpleOpenNI SimpleOpenNI kinect; //Variable

    to access Kinect void setup(){ //Insatiate and initialize Kinect. kinect = new SimpleOpenNI(this); if(kinect.isInit() == false){ exit(); return; } }  Run processing.  Write following code.  Ctrl + s to save this code.
  3. Preparation of using NyARToolKit Move camera_para.dat & patt.sample1 from Sample

    to the same directory of your code. 【NyARToolKit】 【Sample3】
  4. Preparation of using NyARToolKit import SimpleOpenNI.*; import jp.nyatla.nyar4psg.*; //Import NyARToolKit

    SimpleOpenNI kinect; MultiMarker ar; //Variable to access ARToolKit void setup(){ size(640,480,P3D); /*Initialization of Kinect (Abb)*/ ar = new MultiMarker( this, width, height, "camera_para.dat" ,NyAR4PsgConfig.CONFIG_PSG_PV221); } Window size Camera parameter
  5. Show Color Image with NyARToolKit void draw(){ //Update Kinect kinect.update();

    //Grab current color image. PImage rgbImage = kinect.rgbImage(); //Draw color image as background. ar.drawBackground(rgbImage); } void setup(){ /*Initialization of Kinect(Abb)*/ ar = new MultiMarker(this, width, /*Abb*/); kinect.enableRGB(); //Enable acquiring color image }
  6. Marker Recognition with ARToolKit (1/2) void setup(){ /*Initialization of Kinect(Abb)*/

    ar = new MultiMarker(this, width, /*Abb*/); //Load marker file (width is 80mm) ar.addARMarker("patt.sample1", 80); kinect.enableRGB(); } 【ARToolKit】  Using square frame and inner figure.  Do not use rotation symmetric figure.  You can use pre-defined markers if original ARToolkit is downloaded.
  7. Marker Recognition with ARToolKit (2/2) void draw(){ kinect.update(); PImage rgbImage

    = kinect.rgbImage(); ar.drawBackground(rgbImage); } ar.detect(rgbImage); //Marker detection. if(!ar.isExistMarker(0)){ return; //Return if no marker is detected } ar.beginTransform(0); //Put on following object on marker fill(0,0,255); box(40); //Draw box as a test. ar.endTransform();
  8. Acquireing Depth Data void setup(){ /*Initialization of Kinect (Abb)*/ kinect.enableRGB();

    kinect.enableDepth(); //Enable Depth Image } void draw(){ kinect.update(); PImage rgbImage = kinect.rgbImage(); ar.drawBackground(rgbImage); /* following code is abbreviated in this slide */ }
  9. Acquiring Point Cloud Data void draw(){ kinect.update(); PImage rgbImage =

    kinect.rgbImage(); //Acqureing point cloud. //Point clouds is in 1D array. PVector[] realWorld = kinect.depthMapRealWorld(); /*以下省略*/ pixelと濃淡 3D 実空間
  10. Drawing Point Cloud (1/2) void draw(){ /*Acquiring image from kinect

    (省略)*/ /*Marker detection (Abb.)*/ ar.beginTransform(0); // translate(0,0,20); fill(0,0,255); box(40); ar.endTransform(); } beginShape(POINTS); //Draw points of each position (Shown in the next page) endShape(); translate(0,0,20); fill(0,0,255); box(40); Remove Box Add
  11. Drawing Point Cloud (2/2) beginShape(POINTS); endShape(); //Add followning code for(int

    v=0; v<kinect.depthHeight(); v+=4){ for(int u=0; u<kinect.depthWidth(); u+=4){ int index = u + v * kinect.depthWidth(); PVector point = realWorld[index]; stroke(255,255,0); //Draw as yellow point if(point.z>0) { vertex(point.x, point.y, point.z); } } } X Y Z
  12. Modification of Drawing Setting of Point Cloud scale(0.3); //shrink whole

    size of point cloud translate(0,0,-600); //Translate point cloud beginShape(POINTS); for(int v=0; v<kinect.depthHeight(); v+=4){ for(int u=0; u<kinect.depthWidth(); u+=4){ int index = u + v * kinect.depthWidth(); PVector point = realWorld[index]; // stroke(255,255,0); //描画色 if(point.z>0) { vertex(point.x, point.y, point.z); } } } X Y Z
  13. Modification of Each Point Size strokeWeight(2); //Draw as 2 pix

    point. scale(0.3); translate(0,0,-600); beginShape(POINTS); for(int v=0; v<kinect.depthHeight(); v+=4){ /*Drawing code (Abb.)*/
  14. Rotate Point Clouds strokeWeight(2); scale(0.3); //Rotate 90 degree(=π/2) to X

    axis. rotate(PI/2, 1, 0, 0); translate(0,0,-600); beginShape(POINTS); 回転
  15. Remove Far Point Cloud beginShape(POINTS); for(int v=0; v<kinect.depthHeight(); v+=4){ for(int

    u=0; u<kinect.depthWidth(); u+=4){ int index = u + v * kinect.depthWidth(); PVector point = realWorld[index]; // stroke(255,255,0); if(point.z>0) { vertex(point.x, point.y, point.z); } } } endShape(); if(point.z>0 && point.z<1500) { vertex(point.x, point.y, point.z); } 除去 If distance<1500mm
  16. Modify Position of Point Cloud Again 移動 strokeWeight(2); scale(0.3); rotate(PI/2,

    1, 0, 0); translate(0,0,-600); //600mm後ろに戻す (場合による) beginShape(POINTS); for(int v=0; v<kinect.depthHeight(); v+=4){ /*Drawing point cloud (Abb.)*/ translate(0,300,-600); //Move upper direction (prox. 300mm)
  17. Assign Image Color to Point Cloud beginShape(POINTS); for(int v=0; v<kinect.depthHeight();

    v+=4){ for(int u=0; u<kinect.depthWidth(); u+=4){ int index = u + v * kinect.depthWidth(); PVector point = realWorld[index]; // color pixelColor = rgbImage.pixels[index]; stroke(255,255,0); if(point.z>0) { vertex(point.x, point.y, point.z); } } } endShape(); stroke(pixelColor);
  18.  Position of color camera and depth camera are different

    → Pixel coordinate is not match each other. Reason of Mismatch Color Depth
  19. Match Color and Depth Images void setup(){ size(640, 480, P3D);

    kinect = new SimpleOpenNI(this); if(kinect.isInit() == false){ exit(); return; } ar = new MultiMarker( this, width, height, "camera_para.dat"); ar.addARMarker("patt.sample1", 80); kinect.enableRGB(); kinect.enableDepth(); kinect.alternativeViewPointDepthToImage(); } Add this code