Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Getting Started with Azure Kinect DK

357c9e58ecce2865f9eb748192e5143f?s=47 TakashiYoshinaga
September 08, 2019

Getting Started with Azure Kinect DK

Slice deck of tutorial of contents creation with Azure Kinect and Unity.

357c9e58ecce2865f9eb748192e5143f?s=128

TakashiYoshinaga

September 08, 2019
Tweet

More Decks by TakashiYoshinaga

Other Decks in Technology

Transcript

  1. Getting Started with Azure Kinect

  2. Download Pre-Build SDK and Asset http://arfukuoka.lolipop.jp/Azure Kinect/Sample.zip

  3. Goal of the Tutorial Visualization of Colored Point Cloud on

    AR marker
  4. Creating Project New

  5. Creating Project Create Project Project Name Directory

  6. Switching .NET Environment ①File ②Build Setting

  7. Switching .NET Environment Player Settings

  8. Switching .NET Environment Other Settings

  9. Switching .NET Environment Scripting Runtime Versionを .NET 4.X Equivalent Api

    Compatibility Levelを .NET 4.X
  10. Installation of Azure Kinect SDK Right ClickAssets

  11. Installation of Azure Kinect SDK ①Create ②Folder

  12. Installation of Azure Kinect SDK Rename it to Plugins

  13. Installation of Azure Kinect SDK Click Plugins

  14. Installation of Azure Kinect SDK ①Open following directory by explorer

    Sample¥Plugins ②Drag & Drop all Files into Plugins of this project
  15. Installation of Azure Kinect SDK ①Open the directory of which

    Azure Kinect SDK is installed C:¥Program Files¥Azure Kinect SDK v1.2.0¥sdk¥windows-desktop¥amd64¥release¥bin ②Drag and drop depthengine_2_0.dll into Plugins of your project
  16. Adding Object to Visualize Point Cloud Right Click

  17. Adding Object to Visualize Point Cloud Create Empty

  18. Adding Object to Visualize Point Cloud ①GameObject ②Rename into PointCloud

  19. Adding Object to Visualize Point Cloud Add Component

  20. Adding Object to Visualize Point Cloud ①seach ”mesh” ②Mesh Filter

  21. Adding Object to Visualize Point Cloud Add Component

  22. Adding Object to Visualize Point Cloud Mesh Renderer

  23. Adding Object to Visualize Point Cloud Open Materials

  24. Adding Object to Visualize Point Cloud Next step, material for

    drawing point cloud will be set to Element0
  25. Adding a Material to Draw Point Cloud ①Sample Folder ②Double

    Click PointCloud ③Import
  26. Adding a Material to Draw Point Cloud ①Assets ②Check ColoredVertex

    & PointCloud are added
  27. Adding a Material to Draw Point Cloud ①PointCloud ②Drag &

    Drop PointCloud into Element0
  28. Adding a Material to Draw Point Cloud Custom/ColoredVertex

  29. Adding a Script to Draw Point Cloud ①Point Cloud ②Add

    Component
  30. Adding a Script to Draw Point Cloud ①Remove search word

    ②New Script ③KinectScript ④Create and Add
  31. Adding a Script to Draw Point Cloud Check Script is

    added
  32. Ctrl +S

  33. Adding a Script to Draw Point Cloud Double Click KinectScript

    to open this script
  34. Adding a Script to Draw Point Cloud

  35. Start and Stop Azure Kinect using UnityEngine; using Microsoft.Azure.Kinect.Sensor; public

    class KinectScript : MonoBehaviour { Device device; //Variable to access to Kinect void Start() { InitKinect(); //Initialization of Kinect } void InitKinect() { //See Next Page. } /*Omitted */ }
  36. Adding a Script to Draw Point Cloud void InitKinect() {

    device = Device.Open(0); //OpenKinect device.StartCameras(new DeviceConfiguration { ColorFormat = ImageFormat.ColorBGRA32, ColorResolution = ColorResolution.R720p, DepthMode = DepthMode.NFOV_Unbinned, SynchronizedImagesOnly = true, CameraFPS = FPS.FPS30, }); } void OnDestroy() //Stop Kinect { device.StopCameras(); }
  37. Run Play

  38. Run LED is turned on

  39. Run Stop Play again

  40. Preparation of Drawing Point Cloud Device device; int width, height,

    num; //width, height and total Pixels void Start(){ /*Omitted*/ } void InitKinect() { device = Device.Open(0); device.StartCameras ( /*Omitted*/ ); //Calculation of total number of pixel. width = device.GetCalibration(). depth_camera_calibration.resolution_width; height = device.GetCalibration(). depth_camera_calibration.resolution_height; num = height * width; }
  41. Preparation of Drawing Point Cloud int width, height, num; Mesh

    mesh; //Used for Drawing Shape Vector3[] vertices; //Position of each point. Color32[] colors; //Color of each point. int[] indices; //Index list of drawing point void Start() { InitKinect(); InitMesh(); //Initialization of mesh } void InitMesh() { //See the next page. }
  42. Preparation of Drawing Point Cloud void InitMesh() { mesh =

    new Mesh(); //Enable drawing more than 65535 points mesh.indexFormat = UnityEngine.Rendering.IndexFormat.UInt32; //allocation of memory for treating information to //draw point cloud. vertices = new Vector3[num]; colors = new Color32[num]; indices = new int[num]; //See next page }
  43. Next Step Draw points as many number as pixel of

    depth image.
  44. Drawing Points /*Continued from previous step (in InitMesh function)*/ int

    index = 0; //index of vertex array. for (int y = 0; y < height; y++) { for (int x = 0; x < width; x++) { vertices[index].x = x ; vertices[index].y = y ; vertices[index].z = 2; colors[index].r = 0; colors [index].g = 0; colors [index].b = 255; colors [index].a = 255; indices[index] = index; index++; } } Temporally setting the coordinates of each vertex Temporally setting the color of each vertex Setting drawing list.
  45. Drawing Points /*Continued from the previous page.*/ int index =

    0; for (int y = 0; y < height; y++) { for (int x = 0; x < width; x++) { /*Omitted*/ } } //Assignment vertices and colors to mesh. mesh.vertices = vertices; mesh.colors32 = colors; //Setting drawing list and mesh topology. mesh.SetIndices(indices, MeshTopology.Points, 0); //Assingnment of mesh to MeshFilter of the gameObject. gameObject.GetComponent<MeshFilter>().mesh = mesh;
  46. Run Camera is too close to points

  47. Run ①Scene Tab ②Double Click Point Cloud

  48. Getting DataStream from Kinect Int[] indices; Transformation trans; //Coordinate tranlation

    void Start() { /*omitted*/ } void InitKinect() { /*Omitted*/ width = device.GetCalibration(). depth_camera_calibration.resolution_width; height = device.GetCalibration(). depth_camera_calibration.resolution_height; num = height * width; /*It’ll be used for transform coordinate system*/ trans = device.GetCalibration().CreateTransformation(); }
  49. Getting DataStream from Kinect using UnityEngine; using Microsoft.Azure.Kinect.Sensor; //Enable to

    use async function. using System.Threading.Tasks; public class KinectScript : MonoBehaviour { Device device; int width, height, num; Mesh mesh; Vector3[] vertices; Color32[] colors; int[] indices; /*Omitted*/
  50. Getting DataStream from Kinect void Start() { InitKinect(); InitMesh(); Task

    t = KinectLoop( ); } private async Task KinectLoop( ) { while (true) { //Continue to get DataStream from Kinect. } }
  51. Getting DataStream from Kinect While (true) { //Getting synchlonyzed frame

    of Kinect using (Capture capture = await Task.Run(() => device.GetCapture()).ConfigureAwait(true)) { //Getting Depth image with capture.Depth //And transform it into xyz by DeptuImageToPointCloud Image pImage = trans.DepthImageToPointCloud(capture.Depth); //Getting vertices data. Short3[] pointCloud = pImage.GetPixels<Short3>().ToArray(); /*See the next page*/ } }
  52. Getting DataStream from Kinect using (Capture capture = await Task.Run(()

    => device.GetCapture()).ConfigureAwait(true)) { Image pImage = trans.DepthImageToPointCloud(capture.Depth); Short3[] pointCloud = pImage.GetPixels<Short3>().ToArray(); //Assigning coordinate of each vertex to vertices. for (int i = 0; i < num; i++) { vertices[i].x = pointCloud[i].X * 0.001f; vertices[i].y = pointCloud[i].Y * 0.001f; vertices[i].z = pointCloud[i].Z * 0.001f; } //set vertices to mesh. mesh.vertices = vertices; mesh.RecalculateBounds(); }
  53. Modification of a Position of Camera ①MainCameraを選択 ②Positionを0 0 0

  54. Run Click Scene Tab if you want to see point

    cloud from any view point Upside Down
  55. Modification of Y Coordinate using (Capture capture = await Task.Run(()

    => device.GetCapture()).ConfigureAwait(true)) { Image pImage = trans.DepthImageToPointCloud(capture.Depth); Short3[] pointCloud = pImage.GetPixels<Short3>().ToArray(); // Assigning coordinate of each vertex to vertices. for (int i = 0; i < num; i++) { vertices[i].x = pointCloud[i].X * 0.001f; vertices[i].y = -pointCloud[i].Y * 0.001f; vertices[i].z = pointCloud[i].Z * 0.001f; } // set vertices to mesh. mesh.vertices = vertices; mesh.RecalculateBounds(); }
  56. Run Click Scene Tab if you want to see point

    cloud from any view point
  57. Assigning Color to Point Cloud while (true) { using (Capture

    capture = await Task.Run(() => device.GetCapture()).ConfigureAwait(true)) { //Getting color image which is matched to Depth resolution Image modifiedColor = trans.ColorImageToDepthCamera(capture); //Getting pixel array. BGRA[] colorArray = modifiedColor.GetPixels<BGRA>().ToArray(); Image pImage = trans.DepthImageToPointCloud(capture.Depth); Short3[] pointCloud = pImage.GetPixels<Short3>().ToArray(); /*See the next page*/ KinectLoop Function
  58. Assigning Color to Point Cloud /*Continued from the previous page*/

    for (int i = 0; i < num; i++) { vertices[i].x = pointCloud[i].X * 0.001f; vertices[i].y = -pointCloud[i].Y * 0.001f; vertices[i].z = pointCloud[i].Z * 0.001f; colors[i].a = 255; colors [i].b = colorArray[i].B; colors [i].g = colorArray[i].G; colors [i].r = colorArray[i].R; } mesh.vertices = vertices; //Setting color to mesh. mesh.colors32 = colors; mesh.RecalculateBounds();
  59. Run

  60. AR Visualization with Vuforia File

  61. AR Visualization with Vuforia Build Settings

  62. AR Visualization with Vuforia Player Settings

  63. AR Visualization with Vuforia ①XR Settings ②Vuforia Augmented Reality

  64. AR Visualization with Vuforia Accept

  65. AR Visualization with Vuforia チェックされてればOK

  66. AR Visualization with Vuforia Delete MainCamera

  67. AR Visualization with Vuforia Right click

  68. AR Visualization with Vuforia ①Vuforia Engine ②ARCamera

  69. AR Visualization with Vuforia ①Rigght click ②Vuforia Engine ③Image

  70. AR Visualization with Vuforia ①Double ImageTarget

  71. AR Visualization with Vuforia Drag & Drop PointCloud Into ImageTarget

  72. AR Visualization with Vuforia Confirm PointCloud being child of ImageTarget

  73. Modification of Whole Size of Point Cloud ①PointCloud ②Set Position

    and Scale as below Position 0 0.5 -1 Scale 1 1 1
  74. Modification of Whole Size of Point Cloud ②Open Vuforia Engine

    Configuration ①ARCamera
  75. Selecting Camera Device Select your Camera Device

  76. Marker

  77. Completion