Azure Kinect DK C/C++開発概要(仮)

2019/08/20 Azure Kinect DK Meetup Vol.00
https://tmcn.connpass.com/event/143242/

Transcript

1. Azure Kinect DK C/C++開発概要(仮)
2. Self Introduction 杉浦 司 (Tsukasa Sugiura) - Freelance Programmer - Microsoft MVP for Windows Development - Point Cloud Library Maintainer - @UnaNancyOwen
3. Azure Kinect DK Sensor
4. Azure Kinect SDK Introduction Azure Kinect Sensor SDK (k4a / k4arecord) Azure Kinect Body Tracking SDK (k4abt) - Color, Depth, Infrared, IMU, Point Cloud - Open Source Library hosted on GitHub (Depth Engine is Closed Source) - Cross Platform (Windows, Linux) - C API, C++ and C# Wrapper - Body Index Map, Skeleton (26 Joints/Person) - Deep Learning based Pose Estimation - Closed Source Library - Cross Platform (Windows, Linux) - C API, (C++ Wrapper)
5. How to Install Azure Kinect SDK? Azure Kinect Sensor SDK - Install Pre-Built SDK using Installer - Build and Install SDK from Source Code Azure Kinect Body Tracking SDK - Install SDK using Installer - Install NVIDIA GPU Driver and Visual C++ 2015 Runtime for ONNX Runtime (CUDA Backend) Download Azure Kinect Body Tracking SDK | Microsoft Docs https://docs.microsoft.com/en-us/azure/Kinect-dk/body-sdk-download About Azure Kinect Sensor SDK | Microsoft Docs https://docs.microsoft.com/en-us/azure/Kinect-dk/about-sensor-sdk
6. How to Generate Project with Azure Kinect SDK? cmake_minimum_required( VERSION 3.6 ) project( Solution ) add_executable( Project main.cpp ) # Azure Kinect Sensor SDK (Official Support) find_package( k4a REQUIRED ) find_package( k4arecord REQUIRED ) # Azure Kinect Body Tracking SDK (Un-Official Support) set( CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}" ) find_package( k4abt REQUIRED ) if( k4a_FOUND AND k4arecord_FOUND AND k4abt_FOUND ) target_link_libraries( Project k4a::k4a ) target_link_libraries( Project k4a::k4arecord ) target_link_libraries( Project k4a::k4abt ) endif() CMake Module for Azure Kinect Sensor SDK | GitHub Gists https://gist.github.com/UnaNancyOwen/5ce9115ba8b71d12982e5aa9f99788f1#file-findk4abt-cmake Include CMake and MS Build files in the MSI #370 | GitHub/Azure-Kinect-Sensor-SDK https://github.com/microsoft/Azure-Kinect-Sensor-SDK/issues/370
7. Azure Kinect Sensor SDK Overview Capture Image Playback Device
8. Azure Kinect Sensor SDK Programming // Azure Kinect Sensor SDK #include <k4a/k4a.h> #include <k4a/k4a.hpp> /* C++ Wrapper */ // Get Connected Devices const int32_t device_count = k4a::device::get_installed_count(); if( device_count == 0 ){ throw k4a::error( "Failed to found device!" ); } // Open Default Device k4a::device device = k4a::device::open( K4A_DEVICE_DEFAULT ); // Start Cameras with Configuration k4a_device_configuration_t configuration = K4A_DEVICE_CONFIG_INIT_DISABLE_ALL; configuration.color_format = K4A_IMAGE_FORMAT_COLOR_BGRA32; configuration.color_resolution = K4A_COLOR_RESOLUTION_1080P; configuration.depth_mode = K4A_DEPTH_MODE_NFOV_UNBINNED; configuration.wired_sync_mode = K4A_WIRED_SYNC_MODE_STANDALONE; configuration.synchronized_images_only = true; device.start_cameras( &configuration );
9. Azure Kinect Sensor SDK Programming // Get Capture k4a::capture capture; const std::chrono::milliseconds timeout( K4A_WAIT_INFINITE ); const bool result = device.get_capture( &capture, timeout ); if( !result ){ break; } // Get Color Image k4a::image color_image = capture.get_color_image(); cv::Mat color = k4a::get_mat( color_image ); // Get Depth Image k4a::image depth_image = capture.get_depth_image(); cv::Mat depth = k4a::get_mat( depth_image ); - k4a::get_mat() is utility function that defined in util.h Utility for Azure Kinect Sensor SDK | GitHub Gists https://gist.github.com/UnaNancyOwen/9f16ce7ea4c2673fe08b4ce4804fc209#file-util-h
10. Azure Kinect Sensor SDK Programming // Show Image depth.convertTo( depth, CV_8U, -255.0 / 5000.0, 255.0 ); cv::imshow( "color", color ); cv::imshow( "depth", depth ); // Wait Key const int32_t key = cv::waitKey( 30 ); if( key == 'q' ){ break; } // Close Device device.close(); // Clear Handle color_image.reset(); depth_image.reset(); capture.reset();
11. Azure Kinect Sensor SDK Programming // Initialize Transformation k4a::calibration calibration = device.get_calibration( configuration.depth_mode, configuration.color_resolution ); k4a::transformation transformation = k4a::transformation( calibration ); // Transform Color Image to Depth Camera k4a::image transformed_color_image = transformation.color_image_to_depth_camera( depth_image, color_image ); cv::Mat transformed_color = k4a::get_mat( transformed_color_image ); // Transform Depth Image to Color Camera k4a::image transformed_depth_image = transformation.depth_image_to_color_camera( depth_image ); cv::Mat transformed_depth = k4a::get_mat( transformed_depth_image ); // Transform Depth Image to Point Cloud k4a::image xyz_image = transformation.depth_image_to_point_cloud( depth_image, K4A_CALIBRATION_TYPE_DEPTH ); //k4a::image xyz_image = transformation.depth_image_to_point_cloud( transformed_depth_image, K4A_CALIBRATION_TYPE_COLOR ); cv::Mat xyz = k4a::get_mat( xyz_image ); - Transform Image to Other Coordinate System Color Image → Depth Camera Depth Image → Color Camera Depth Image → Point Cloud Add overload functions that returns k4a::image in k4a::transformation #596 | GitHub/Azure-Kinect-Sensor-SDK https://github.com/microsoft/Azure-Kinect-Sensor-SDK/pull/596 // Close Transformation transformation.destroy();
12. Azure Kinect Sensor SDK Programming // Azure Kinect Sensor SDK #include <k4arecord/playback.h> #include <k4arecord/playback.hpp> /* C++ Wrapper from Azure Kinect Sensor SDK v1.2.0 */ // Open Playback File k4a::playback playback = k4a::playback::open( "./path/to/file.mkv" ); // Get Capture k4a::capture capture; bool result = playback.get_next_capture( &capture ); if( !result ){ break; } // Close Playback playback.close(); - NOTE: k4arecorder (Azure Kinect Sensor SDK v1.1.x) has a bug that first 2-frames are empty. Move C++ wrapper for playback #493 | GitHub/Azure-Kinect-Sensor-SDK https://github.com/microsoft/Azure-Kinect-Sensor-SDK/pull/493
13. Azure Kinect Sensor SDK Programming Azure Kinect Office Sample Recordings | Microsoft Download Center https://www.microsoft.com/en-us/download/details.aspx?id=58385
14. Azure Kinect Body Tracking SDK Overview Capture Queue Playback Device Frame Body
15. Azure Kinect Body Tracking SDK Programming // Azure Kinect Body Tracking SDK #include <k4abt.h> #include "k4abt.hpp" /* Un-Official C++ Wrapper */ //#include <k4abt.hpp> /* Official C++ Wrapper Provide from Next Release */ // Create Tracker k4abt::tracker tracker = k4abt::tracker::create( calibration ); if( !tracker ){ throw k4a::error( "Failed to create tracker!" ); } // Enqueue Capture tracker.enqueue_capture( capture ); // Pop Tracker Result k4abt::frame body_frame = tracker.pop_result(); - Official C++ wrapper is coming soon … C++Wrapper for Azure Kinect Body Tracking SDK | GitHub Gists https://gist.github.com/UnaNancyOwen/41b578f5d272aa6f22cf8ff6565fb71b#file-k4abt-hpp
16. Azure Kinect Body Tracking SDK Programming // Get Body Index Map k4a::image body_index_map_image = body_frame.get_body_index_map(); cv::Mat body_index_map = k4a::get_mat( body_index_map_image ); // Draw Body Index Map cv::Mat body_index = cv::Mat::zeros( body_index_map.size(), CV_8UC3 ); body_index.forEach<cv::Vec3b>( [&]( cv::Vec3b& pixel, const int32_t* position ){ const int32_t x = position[1], y = position[0]; const uint32_t index = body_index_map.at<uint8_t>( y, x ); if( index != K4ABT_BODY_INDEX_MAP_BACKGROUND ){ pixel = colors[index % colors.size()]; } } ); - Background Index is 255 (K4ABT_BODY_INDEX_MAP_BACKGROUND)
17. Azure Kinect Body Tracking SDK Programming // Get Body Skeleton std::vector<k4abt_body_t> bodies = body_frame.get_bodies(); // Get Image that used for Inference k4a::capture body_capture = body_frame.get_capture(); k4a::image skeleton_image = body_capture.get_color_image(); cv::Mat skeleton = k4a::get_mat( skeleton_image ); // Draw Body Skeleton for( const k4abt_body_t& body : bodies ){ for( const k4abt_joint_t& joint : body.skeleton.joints ){ k4a_float2_t position; const bool result = calibration.convert_3d_to_2d( joint.position, K4A_CALIBRATION_TYPE_DEPTH, K4A_CALIBRATION_TYPE_COLOR, &position ); if( !result ){ continue; } const int32_t id = body.id; const cv::Point point( static_cast<int32_t>( position.xy.x ), static_cast<int32_t>( position.xy.y ) ); cv::circle( skeleton, point, 5, colors[id % colors.size()], -1 ); } } - Get Image that used for Inference - Convert Joint Position (3D) to 2D for Draw Circle on Image
18. Azure Kinect Body Tracking SDK Programming // Show Image cv::imshow( "body index", body_index ); cv::imshow( "skeleton", skeleton ); // Wait Key const int32_t key = cv::waitKey( 30 ); if( key == 'q' ){ break; } // Close Tracker tracker.destroy(); // Clear Handle body_index_map_image.reset(); skeleton_image.reset(); body_capture.reset(); body_frame.reset();