Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Putting Point Cloud into AR with Kinect and Processing

Putting Point Cloud into AR with Kinect and Processing

357c9e58ecce2865f9eb748192e5143f?s=128

TakashiYoshinaga

October 21, 2018
Tweet

More Decks by TakashiYoshinaga

Other Decks in Technology

Transcript

  1. Putting Point Cloud into AR with Kinect and Processing

  2. Setting up • Kinect for Windows SDK v1.8 https://www.microsoft.com/en-us/download/details.aspx?id=40278 •

    Processing 2.2.1 https://processing.org/download/ Unzip and put following folders into libraries of Processing (C:¥Users¥YOU¥Documents¥Processing¥ libraries) • SimpleOpenNI 1.96 https://code.google.com/archive/p/simple-openni/downloads • NyARToolKit (nyar4psg.zip) https://github.com/nyatla/NyARToolkit-for-Processing/releases Download and unzip following file into any folder. https://drive.google.com/file/d/1ahzAzp_s5gqg5WbfJkF5JSv5v87qm KZv/view?usp=sharing
  3. Goal https://youtu.be/WeorvV9vZh8

  4. Initialization of Kinect import SimpleOpenNI.*; //Load SimpleOpenNI SimpleOpenNI kinect; //Variable

    to access Kinect void setup(){ //Insatiate and initialize Kinect. kinect = new SimpleOpenNI(this); if(kinect.isInit() == false){ exit(); return; } }  Run processing.  Write following code.  Ctrl + s to save this code.
  5. Run Application IR projection will started

  6. Preparation of using NyARToolKit Move camera_para.dat & patt.sample1 from Sample

    to the same directory of your code. 【NyARToolKit】 【Sample3】
  7. Preparation of using NyARToolKit import SimpleOpenNI.*; import jp.nyatla.nyar4psg.*; //Import NyARToolKit

    SimpleOpenNI kinect; MultiMarker ar; //Variable to access ARToolKit void setup(){ size(640,480,P3D); /*Initialization of Kinect (Abb)*/ ar = new MultiMarker( this, width, height, "camera_para.dat" ,NyAR4PsgConfig.CONFIG_PSG_PV221); } Window size Camera parameter
  8. Show Color Image with NyARToolKit void draw(){ //Update Kinect kinect.update();

    //Grab current color image. PImage rgbImage = kinect.rgbImage(); //Draw color image as background. ar.drawBackground(rgbImage); } void setup(){ /*Initialization of Kinect(Abb)*/ ar = new MultiMarker(this, width, /*Abb*/); kinect.enableRGB(); //Enable acquiring color image }
  9. Marker Recognition with ARToolKit (1/2) void setup(){ /*Initialization of Kinect(Abb)*/

    ar = new MultiMarker(this, width, /*Abb*/); //Load marker file (width is 80mm) ar.addARMarker("patt.sample1", 80); kinect.enableRGB(); } 【ARToolKit】  Using square frame and inner figure.  Do not use rotation symmetric figure.  You can use pre-defined markers if original ARToolkit is downloaded.
  10. Marker Recognition with ARToolKit (2/2) void draw(){ kinect.update(); PImage rgbImage

    = kinect.rgbImage(); ar.drawBackground(rgbImage); } ar.detect(rgbImage); //Marker detection. if(!ar.isExistMarker(0)){ return; //Return if no marker is detected } ar.beginTransform(0); //Put on following object on marker fill(0,0,255); box(40); //Draw box as a test. ar.endTransform();
  11. Run application It’s not on marker

  12. Modification of CG’s Position ar.beginTransform(0); translate(0,0,20); fill(0,0,255); box(40); ar.endTransform(); Translate

    box to upper. x Y Z (0,0,0) (0,0,20) 40mm
  13. Acquireing Depth Data void setup(){ /*Initialization of Kinect (Abb)*/ kinect.enableRGB();

    kinect.enableDepth(); //Enable Depth Image } void draw(){ kinect.update(); PImage rgbImage = kinect.rgbImage(); ar.drawBackground(rgbImage); /* following code is abbreviated in this slide */ }
  14. Acquiring Point Cloud Data void draw(){ kinect.update(); PImage rgbImage =

    kinect.rgbImage(); //Acqureing point cloud. //Point clouds is in 1D array. PVector[] realWorld = kinect.depthMapRealWorld(); /*以下省略*/ pixelと濃淡 3D 実空間
  15. Drawing Point Cloud (1/2) void draw(){ /*Acquiring image from kinect

    (省略)*/ /*Marker detection (Abb.)*/ ar.beginTransform(0); // translate(0,0,20); fill(0,0,255); box(40); ar.endTransform(); } beginShape(POINTS); //Draw points of each position (Shown in the next page) endShape(); translate(0,0,20); fill(0,0,255); box(40); Remove Box Add
  16. Drawing Point Cloud (2/2) beginShape(POINTS); endShape(); //Add followning code for(int

    v=0; v<kinect.depthHeight(); v+=4){ for(int u=0; u<kinect.depthWidth(); u+=4){ int index = u + v * kinect.depthWidth(); PVector point = realWorld[index]; stroke(255,255,0); //Draw as yellow point if(point.z>0) { vertex(point.x, point.y, point.z); } } } X Y Z
  17. Run Application Point are shown but not easy to see.

    Yellow points
  18. Modification of Drawing Setting of Point Cloud scale(0.3); //shrink whole

    size of point cloud translate(0,0,-600); //Translate point cloud beginShape(POINTS); for(int v=0; v<kinect.depthHeight(); v+=4){ for(int u=0; u<kinect.depthWidth(); u+=4){ int index = u + v * kinect.depthWidth(); PVector point = realWorld[index]; // stroke(255,255,0); //描画色 if(point.z>0) { vertex(point.x, point.y, point.z); } } } X Y Z
  19. Modification of Each Point Size strokeWeight(2); //Draw as 2 pix

    point. scale(0.3); translate(0,0,-600); beginShape(POINTS); for(int v=0; v<kinect.depthHeight(); v+=4){ /*Drawing code (Abb.)*/
  20. Rotate Point Clouds strokeWeight(2); scale(0.3); //Rotate 90 degree(=π/2) to X

    axis. rotate(PI/2, 1, 0, 0); translate(0,0,-600); beginShape(POINTS); 回転
  21. Remove Far Point Cloud beginShape(POINTS); for(int v=0; v<kinect.depthHeight(); v+=4){ for(int

    u=0; u<kinect.depthWidth(); u+=4){ int index = u + v * kinect.depthWidth(); PVector point = realWorld[index]; // stroke(255,255,0); if(point.z>0) { vertex(point.x, point.y, point.z); } } } endShape(); if(point.z>0 && point.z<1500) { vertex(point.x, point.y, point.z); } 除去 If distance<1500mm
  22. Modify Position of Point Cloud Again 移動 strokeWeight(2); scale(0.3); rotate(PI/2,

    1, 0, 0); translate(0,0,-600); //600mm後ろに戻す (場合による) beginShape(POINTS); for(int v=0; v<kinect.depthHeight(); v+=4){ /*Drawing point cloud (Abb.)*/ translate(0,300,-600); //Move upper direction (prox. 300mm)
  23. Assign Image Color to Point Cloud beginShape(POINTS); for(int v=0; v<kinect.depthHeight();

    v+=4){ for(int u=0; u<kinect.depthWidth(); u+=4){ int index = u + v * kinect.depthWidth(); PVector point = realWorld[index]; // color pixelColor = rgbImage.pixels[index]; stroke(255,255,0); if(point.z>0) { vertex(point.x, point.y, point.z); } } } endShape(); stroke(pixelColor);
  24. Run Application Not much the position between point and color

  25.  Position of color camera and depth camera are different

    → Pixel coordinate is not match each other. Reason of Mismatch Color Depth
  26. Match Color and Depth Images void setup(){ size(640, 480, P3D);

    kinect = new SimpleOpenNI(this); if(kinect.isInit() == false){ exit(); return; } ar = new MultiMarker( this, width, height, "camera_para.dat"); ar.addARMarker("patt.sample1", 80); kinect.enableRGB(); kinect.enableDepth(); kinect.alternativeViewPointDepthToImage(); } Add this code
  27. Finished

  28. None