Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Tutorial of Basic Hand Tracking with Kinect & OpenCV

Tutorial of Basic Hand Tracking with Kinect & OpenCV

Slide for AR Fukuoka held 2nd March 2019

357c9e58ecce2865f9eb748192e5143f?s=128

TakashiYoshinaga

March 02, 2019
Tweet

More Decks by TakashiYoshinaga

Other Decks in Technology

Transcript

  1. Tutorial of Basic Hand Tracking with Kinect & OpenCV #AR_Fukuoka

  2. Download the Sample Beforehand http://arfukuoka.lolipop.jp/kinect2019mar/sample.zip

  3. Goal of the tutorial https://youtu.be/HX6mvVICaDk

  4. Related Work

  5. Hand Tracking And Finger Detection 【Procedure】

  6. Problem  Some skin color areas can cause a misrecognition.

    Ex) Face, Background image of environment  Simple solutions are.. method 1) Define the area for hand detection. method 2) Detect and exclude face area.
  7. Utilization of RGB-D Sensor Real-time RGB Image+Depth

  8. Utilization of RGB-D Sensor Available Cut off Threshold

  9. Utilization of RGB-D Sensor Available Cut off Threshold

  10. Procedure of Tutorial Input Depth Color Binarization Depth Value Skin

    Color AND Detection
  11. Development widh Visual Studio

  12. Creating New Project (1/2) ①File ②New ③Project...

  13. Creating New Project (2/2) ①VisualC# ②Windows Forms App ③OK

  14. Confirmation Start

  15. Confirmation Close

  16. Disabling AutoScaleMode ②Properties ③AutoScaleMode ④None ①Click

  17. Setting Up Visualization Area (1/11) ①ToolBox ②PictureBox ③Click on Form

  18. Setting Up Visualization Area (2/11) ①Select PictureBox ②Size=640,480

  19. Setting Up Visualization Area (3/11) SizeMode=Zoom *This operation means enable

    adjustment of image size to picturebox automatically.
  20. Setting Up Visualization Area (4/11) Adjust Form size wider

  21. Setting Up Visualization Area (5/11) ①Select PictureBox ②[Ctrl+C]→[Ctrl+V]

  22. Setting Up Visualization Area (6/11) ①Select 2nd PictureBox ②Size=200,150

  23. Setting Up Visualization Area (7/11) Place it on bottom left

    of the window
  24. Setting Up Visualization Area (8/11) Select and [Ctrl+C]→[Ctrl+V]

  25. Setting Up Visualization Area (9/11) Place it on bottom center

  26. Setting Up Visualization Area (10/11) Select and [Ctrl+C]→[Ctrl+V]

  27. Setting Up Visualization Area (11/11) Place it on bottom right

  28. Roll of Each PictuerBox ← Final Result ← Result of

    each Binarization
  29. Installation of OpenCV (1/4) ①Tools ②NuGet Pckage Manager ③Manage NuGet

    Package for Solution
  30. Installation of OpenCV (2/4) ①Brows ②nuget.org

  31. Installation of OpenCV (3/4) ①Search for “OpenCVsharp” ②OpenCvSharp3 AnyCPU ③Check

    ④Install
  32. Installation of OpenCV (4/4) Close NuGet

  33. Please wait for a moment till installation of OpenCV is

    finished
  34. Installation of KinectSDK (1/3) ①Right Click References ②Add Referenence

  35. Installation of KinectSDK (2/3) Search for Kinect

  36. Installation of KinectSDK (3/3) ①Check MicrosoftKinect 1.8.0.0 ②OK

  37. Today, we use color and depth image for this tutorial.

    Please use KinectGrabber provided for this tutorial since raw data of SDK is not easy to use for beginner.
  38. Installation of KinectGrabber (1/4) Move KinectGrabber.csinto the same folder as

    Form1.cs sample folder Your project
  39. Installation of KinectGrabber (2/4) Right click your project

  40. Installation of KinectGrabber (3/4) ①Add ②Existing Item

  41. Installation of KinectGrabber (4/4) ①KinectGrabber.cs ②Add

  42. Setting up Timer & Processing Interval Image processing and visualization

    will run at a certain interval by using timer. ①ToolBox ②Timer ③Form上をクリック
  43. Setting up Timer & Processing Interval ①Timer is added ➂Interval=30[ms]

  44. Adding Function for Periodical Processing Double Click timer1

  45. Adding Function for Periodical Processing timer1_Tick is generated Timer1_Tick function

    will run at a certain interval.(Not yet..)
  46. Let Timer to Start Automatically ①Form1[Design] ②Select Form1

  47. ①Click ②Double Click “Shown” Let Timer to Start Automatically

  48. Let Timer to Start Automatically namespace ARFukuokaCV { public partial

    class Form1 : Form { public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { } private void Form1_Shown(object sender, EventArgs e) { timer1.Start(); } } } ①Form1_Shown is generated ②Write this to start timer
  49. Procedure Input Depth Color Binarization Depth Value Skin Color AND

    Detection
  50. Procedure Binarization Depth Value Skin Color AND Detection Input Depth

    Color
  51. Waking up Kinect Sensor using KinectSample; //Importing KinectGrabber namespace Kinect_ARFukuoka

    { public partial class Form1 : Form { KinectGrabber kinect = new KinectGrabber(); public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { } /*****omitted below*****/
  52. Let Kinect to Stop Automatically ①Click ②Double Click FormClosing

  53. Let Kinect to Stop Automatically KinectGrabber kinect = new KinectGrabber();

    public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { } private void Form1_Shown(object sender, EventArgs e) { timer1.Start(); } private void Form1_FormClosing(object sender, FormClosingEventArgs e) { timer1.Stop(); kinect.Close(); }
  54. Run & Confirm Infrared exposure will be started

  55. Display Color/Depth Images KinectGrabber kinect = new KinectGrabber(); public Form1()

    { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { Bitmap color = kinect.GetColorImage(); //Acquiring color image Bitmap depth = kinect.GetDepthImage(); // Acquiring depth image pictureBox1.Image = color; //Displaying color image pictureBox2.Image = depth; //Displaying depthimage } private void Form1_Shown(object sender, EventArgs e) { timer1.Start(); }
  56. Run & Confirm

  57. Changing Depth Range to Visualize KinectGrabber kinect = new KinectGrabber();

    public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { float maxDepth = 8; Bitmap color = kinect.GetColorImage(); Bitmap depth = kinect.GetDepthImage(); //Depth画像取得 pictureBox1.Image = color; pictureBox2.Image = depth; } private void Form1_Shown(object sender, EventArgs e) { timer1.Start(); } Bitmap depth = kinect.GetDepthImage(maxDepth); This is carried out to let attendees familiarize Kinect
  58. Trial 1: Let’s Change Max Depth 【Information】  By using

    depth threshold, we can remove background image.  Surly you can do the same thing by applying depth filtering to raw depth data from Kinect SDK. maxDepth=8 maxDepth=4 maxDepth=2
  59. Trial 2: Let’s Change Max Depth Let's find the value

    of maxDepth where almost no face appears ex) maxDepth=1 ex) maxDepth=0.7f NG Good
  60. Next Step Separate each pixel into 2 information which should

    be processed or not
  61. Let’s start to use OpenCV!

  62. Importing OpenCV Library using KinectSample; using OpenCvSharp; using OpenCvSharp.Extensions; namespace

    Kinect_ARFukuoka { public partial class Form1 : Form { KinectGrabber kinect = new KinectGrabber(); public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { /*****omitted*****/ } /*****omitted below*****/
  63. Procedure Binarization Depth Value Skin Color AND Detection Input Depth

    Color
  64. Procedure AND Detection Input Depth Color Binarization Depth Value Skin

    Color
  65. Binarization of Gray Scale Image 0 255 0 255 Separate

    brightness described as 0~255 into less or above the threshold. Binarization is often used for definition of pixel which should be processed.
  66. Conversion of Bitmap image to Mat //Variable for holding depth

    image Mat depthMat; public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { float maxDepth = 0.75f; Bitmap color = kinect.GetColorImage(); Bitmap depth = kinect.GetDepthImage(maxDepth); //Conversion of Bitmap into Mat depthMat = BitmapConverter.ToMat(depth); pictureBox1.Image = color; pictureBox2.Image = depth; //Releasing memory of depthMat depthMat.Release(); } pictureBox2.Image = depth; pictureBox2.Image = depthMat.ToBitmap();
  67. Binarization of Depth Image //Function for binarization private void Binarization()

    { //BGR→GrayScale depthMat = depthMat.CvtColor(ColorConversionCodes.BGR2GRAY); //Binarize image with threshold=20 depthMat = depthMat.Threshold(20, 255, ThresholdTypes.Binary); } private void timer1_Tick(object sender, EventArgs e) { float maxDepth = 0.75f; Bitmap color = kinect.GetColorImage(); Bitmap depth = kinect.GetDepthImage(maxDepth); depthMat = BitmapConverter.ToMat(depth); //Calling Binarization Binarization(); pictureBox1.Image = color; pictureBox2.Image = depthMat.ToBitmap(); depthMat.Release(); }
  68. Run & Confirm Depth Binarization can remove background but still

    including things that isn’t human skin!
  69. Procedure AND Detection Input Depth Color Binarization Depth Value Skin

    Color
  70. Procedure AND Detection Input Depth Color Binarization Depth Value Skin

    Color
  71. Binarization of Color Image 【Want to do】  Making specific

    color into white and others int black. 【Problem】  Thresholding for RGB color is difficult 【Solution】  Using HSV color space  H(Hue) is described as 0~360° ※0~180 for OpenCV  S(Saturation) is 0~100% ※0~255 for OpenCV  V(Value) is 0~100% This means brightness of color ※0~255 for OpenCV RGB HSV
  72. Preparation of Mat for Color & Result //Add Mat for

    color(bgr) and result of binarization(bin) Mat depthMat; public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { float maxDepth = 0.75f; Bitmap color = kinect.GetColorImage(); Bitmap depth = kinect.GetDepthImage(maxDepth); depthMat = BitmapConverter.ToMat(depth); bgrMat = BitmapConverter.ToMat(color); Binarization(); pictureBox1.Image = color; pictureBox2.Image = depthMat.ToBitmap(); depthMat.Release(); bgrMat.Release(); binMat.Release(); } Mat depthMat, bgrMat, binMat;
  73. Binarization of HSV Image //Lower/Upper threshold of HSV Scalar lower

    = new Scalar(0, 0, 0); Scalar upper = new Scalar(180, 255, 255); public Form1() { InitializeComponent(); } private void Binarization() { depthMat = depthMat.CvtColor(ColorConversionCodes.BGR2GRAY); depthMat = depthMat.Threshold(20, 255, ThresholdTypes.Binary); //Convert BGR into HSV color space. Mat hsv = bgrMat.CvtColor(ColorConversionCodes.BGR2HSV); //Binarization of HSV image binMat = hsv.InRange(lower, upper); //Visualization of the result on pictureBox3 pictureBox3.Image = binMat.ToBitmap(); hsv.Release(); } lower upper
  74. Run & Confirm Shown as white image

  75. Trial:Changing Threshold of Hue Scalar lower = new OpenCvSharp.Scalar(50, 0,

    0); Scalar upper = new OpenCvSharp.Scalar(80, 255, 255); Green area is extracted
  76. Set Hue’s Threshold as All Range Scalar lower = new

    OpenCvSharp.Scalar(0, 0, 0); Scalar upper = new OpenCvSharp.Scalar(180, 255, 255);
  77. Making H-Threshold Modifiable via UI ①ToolBox ②GroupBox ③Click Form1

  78. Making H-Threshold Modifiable via UI ①Put it on the right

    side ②Click
  79. Making H-Threshold Modifiable via UI ①Size=450,190 ②Text=H

  80. Making H-Threshold Modifiable via UI ①ToolBox ②Label ③Click GloupBox

  81. Making H-Threshold Modifiable via UI ①Put label upper left of

    groupbox ②Text=Min
  82. Making H-Threshold Modifiable via UI ①ToolBox ②TrackBar ③Click GloupBox

  83. Making H-Threshold Modifiable via UI ①Put it on right of

    label ②Name=minH
  84. Making H-Threshold Modifiable via UI Maximum:180 Minimum: 0 Value: 0

  85. Making H-Threshold Modifiable via UI ①Select TrackBar ②Click ②Double Click

    Scroll
  86. Making H-Threshold Modifiable via UI //Called when value of trackbar

    minH is changed private void minH_Scroll(object sender, EventArgs e) { //lower value of H-threshold is changed lower.Val0 = minH.Value; } Start
  87. Run & Confirm Move Color which is lower than threshold

    is shown as black
  88. Making H-Threshold Modifiable via UI Select Label & TrackBar together

  89. Making H-Threshold Modifiable via UI [Ctrl+C]→[Ctrl+V]

  90. Making H-Threshold Modifiable via UI ①Select Label ③Text=Max ②Click

  91. Making H-Threshold Modifiable via UI Select TrackBar Name=maxH

  92. Making H-Threshold Modifiable via UI Value=180

  93. Making H-Threshold Modifiable via UI ①Select 2nd Trackbar ②Click ③Double

    Click Scroll
  94. Making H-Threshold Modifiable via UI //Called when value of trackbar

    maxH is changed private void maxH_Scroll(object sender, EventArgs e) { //upper value of H-threshold is changed upper.Val0 = maxH.Value; } Start
  95. Making H-Threshold Modifiable via UI Move Color which is between

    lower and upper threshold is shown as white area
  96. Trial: Let’s Extract Skin Area Noise should be removed

  97. Making S-Threshold Modifiable via UI Select GroupBox & [Ctrl+C]→[Ctrl+V]

  98. Making S-Threshold Modifiable via UI ①put groupBox2 under 1st groupbox

    ②Text= S ②Click
  99. Making S-Threshold Modifiable via UI ①Trackbar next to Min ②Name=minS

  100. Making S-Threshold Modifiable via UI Maximum=255

  101. Making S-Threshold Modifiable via UI ①Trackbar next to Max ②Name=maxS

  102. Making S-Threshold Modifiable via UI ②Maximum=255 ③Value=255

  103. Making S-Threshold Modifiable via UI ①Trackbar next to Min ③Double

    Click Scroll ②Click
  104. Making S-Threshold Modifiable via UI ①Trackbar next to Max ③Double

    Click Scroll ②Click
  105. Making S-Threshold Modifiable via UI //Called when value of trackbar

    minS is changed private void minS_Scroll(object sender, EventArgs e) { //lower value of S-threshold is changed lower.Val1 = minS.Value; } //Called when value of trackbar maxS is changed private void maxS_Scroll(object sender, EventArgs e) { //upper value of S-threshold is changed upper.Val1 = maxS.Value; }
  106. Run & Confirm Large S is shown as White Change

    minS Large S is shown as Black Change maxS min max min max
  107. Making V-Threshold Modifiable via UI Select 2nd GroupBox & [Ctrl+C]→[Ctrl+V]

  108. Making V-Threshold Modifiable via UI ①Put it under 2nd GroupBox

  109. Making V-Threshold Modifiable via UI ②Text= V ①Click

  110. Making V-Threshold Modifiable via UI ①Trackbar next to Min ②Name=minV

  111. Making V-Threshold Modifiable via UI ②Double Click Scroll ①Click

  112. Making V-Threshold Modifiable via UI ①Trackbar next to Max ③Name=maxV

    ①Click
  113. Making V-Threshold Modifiable via UI ②Double Click Scroll ①Click

  114. Making V-Threshold Modifiable via UI //Called when value of trackbar

    minV is changed private void minV_Scroll(object sender, EventArgs e) { //lower value of V-threshold is changed lower.Val2 = minV.Value; } //Called when value of trackbar maxV is changed private void maxV_Scroll(object sender, EventArgs e) { //upper value of V-threshold is changed upper.Val2 = maxV.Value; }
  115. Run & Confirm Small V is shown as black Change

    minV Small V is shown as white Change maxV min max min max
  116. Trial: Let’s Extract Hand Not only hand but also face

    is extracted by color binarization
  117. Procedure AND Detection Input Depth Color Binarization Depth Value Skin

    Color
  118. Procedure Detection Input Depth Color Binarization Depth Value Skin Color

    AND
  119. Extraction Common Area with AND private void Binarization() { //Conversion

    bgr to gray depthMat = depthMat.CvtColor(ColorConversionCodes.BGR2GRAY); //Binarization with threshold=20 depthMat = depthMat.Threshold(20, 255, ThresholdTypes.Binary); //Conversion bgr to hsv Mat hsv = bgrMat.CvtColor(ColorConversionCodes.BGR2HSV); //Binarization of HSV with lower/upper threshold binMat = hsv.InRange(lower, upper); //Visualization of the result pictureBox3.Image = binMat.ToBitmap(); //Extract common white area between binMat & depthMat Cv2.BitwiseAnd(binMat, depthMat, binMat); //Visualization of the result pictureBox4.Image = binMat.ToBitmap(); hsv.Release(); }
  120. Filling Lacked Area & Denoising depth HSV AND 【Filling Lacked

    Area & Denoising】 (1) Smoothing the image (2) Binarization of the image (1) (2) 欠損 ノイズ
  121. Smoothing & Binarization private void Binarization() { //Binarization of depth

    depthMat = depthMat.CvtColor(ColorConversionCodes.BGR2GRAY); depthMat = depthMat.Threshold(20, 255, ThresholdTypes.Binary); //Conversion of BGR to HSV Mat hsv = bgrMat.CvtColor(ColorConversionCodes.BGR2HSV); //Binarization of HSV binMat = hsv.InRange(lower, upper); //Visualization of the result pictureBox3.Image = binMat.ToBitmap(); Cv2.BitwiseAnd(binMat, depthMat, binMat); //Smoothing with 9*9 neighbor pixels binMat = binMat.Blur(new OpenCvSharp.Size(9, 9)); //Binarization with threshold=60 binMat = binMat.Threshold(60, 256, ThresholdTypes.Binary); pictureBox4.Image = binMat.ToBitmap(); hsv.Release(); }
  122. Procedure Detection Input Depth Color Binarization Depth Value Skin Color

    AND
  123. Procedure Input Depth Color Binarization Depth Value Skin Color AND

    Detection Next
  124. Preparation of Contour Extraction //Procedure of Contour Extraction will be

    written here private Point[] ContourExtraction() { Point[] contour = null; return contour; } private void timer1_Tick(object sender, EventArgs e) { float maxDepth = 0.75f; Bitmap color = kinect.GetColorImage(); Bitmap depth = kinect.GetDepthImage(maxDepth); depthMat = BitmapConverter.ToMat(depth); bgrMat = BitmapConverter.ToMat(color); Binarization(); //Points consisting contour are returned. Point[] contour = ContourExtraction(); pictureBox1.Image = color; /*****omitted below*****/ }
  125. Preparation of Contour Extraction

  126. Contours (Please edit ContorExtraction) Point[] contour = null; Point[][] lines;

    HierarchyIndex[] h; //Find contours from binMat binMat.FindContours(out lines, out h, RetrievalModes.External, ContourApproximationModes.ApproxSimple); double maxArea = -1; for (int i = 0; i < lines.Length; i++) { double area = Cv2.ContourArea(lines[i]); //Contour of maximum area is treated as hand if (area > maxArea) { maxArea = area; contour = lines[i]; } } return contour;
  127. Visualize Contourline private void DrawPolyLine(Point[] points, Scalar color) { /*described

    in next slide!*/ } private void timer1_Tick(object sender, EventArgs e) { /*省略*/ bgrMat = BitmapConverter.ToMat(color); Binarization(); Point[] contour = ContourExtraction(); if (contour != null) { DrawPolyLine( contour, Scalar.Green); } pictureBox1.Image = color; pictureBox1.Image = bgrMat.ToBitmap(); /*****omitted below*****/ } pictureBox1.Image = color;
  128. 輪郭の表示 private void DrawPolyLine(Point[] points, Scalar color) { Point p

    = points.Last(); for (int i = 0; i < points.Length; i++) { Cv2.Line(bgrMat, p, points[i], color, 2); p = points[i]; } } 0 1 2 3 4 points[0] 1 2 3 p p points[1] 2 3 4 0 1 2 p points[4] ・・・ i = 0 i = 1 i = 4
  129. Run & Confirm

  130. Procedure Input Depth Color Binarization Depth Value Skin Color AND

    Detection Next
  131. Convex Hull private int[] Polygonization(Point[] contour) { //Approximate contour line

    as convex hull. int[] hullIdx = Cv2.ConvexHullIndices(contour); //Approximate contour line as convex hull. Point[] hullPt = Cv2.ConvexHull(contour); DrawPolyLine(hullPt, Scalar.Blue); return hullIdx; } private void timer1_Tick(object sender, EventArgs e) { /***omitted***/ }
  132. Convex Hull private void timer1_Tick(object sender, EventArgs e) { float

    maxDepth = 0.75f; Bitmap color = kinect.GetColorImage(); Bitmap depth = kinect.GetDepthImage(maxDepth); depthMat = BitmapConverter.ToMat(depth); bgrMat = BitmapConverter.ToMat(color); Binarization(); Point[] contour = ContourExtraction(); if (contour != null) { DrawPolyLine( contour, Scalar.Green); int[] hull = Polygonization(contour); } pictureBox1.Image = bgrMat.ToBitmap(); pictureBox2.Image = depthMat.ToBitmap(); /*****omitted below*****/ }
  133. Run & Confirm

  134. Visualization of Vertices of Convex Hull private int[] Polygonization(Point[] contour)

    { int[] hullIdx = Cv2.ConvexHullIndices(contour); Point[] hullPt = Cv2.ConvexHull(contour); for(int i=0;i<hullIdx.Length;i++) { int index = hullIdx[i]; bgrMat.Circle(contour[index], 10, Scalar.Red, 2); } DrawPolyLine(bgr, hullPt, Scalar.Blue); return hullIdx; }  Many points are exists at the tip of finger  They should integrated into the one point.
  135. Clustering Integrate near points into one point 0 1 2

    3 4 5
  136. Clustering private int[] Polygonization(Point[] contour) { int[] hullIdx = Cv2.ConvexHullIndices(contour);

    Point[] hullPt = Cv2.ConvexHull(contour); //Terating label numbers of each vertices of hull int[] labels; //clustering based on the role written in Predicate Cv2.Partition(hullPt, out labels, Predicate); for (int i = 0; i < hullIdx.Length; i++) { int index = hullIdx[i]; bgrMat.Circle(contour[index], 10, Scalar.Red, 2); //Visualization of label number bgrMat.PutText(labels[i].ToString(), contour[index], HersheyFonts.HersheyDuplex, 1, Scalar.White); } DrawPolyLine(hullPt, Scalar.Blue); return hullIdx; }
  137. Clustering private int[] Polygonization(Point[] contour) { /*****omitted above*****/ Cv2.Partition(hullPt, out

    labels, Predicate); /*****omitted below*****/ } private bool Predicate(Point t1, Point t2) { //2 points are treated as the same point if //distance is less than 20pixels if (t1.DistanceTo(t2) < 20) { return true; } else { return false; } }
  138. Run & Confirm

  139. Integration of Points Center Point 【Procedure】 Step1: Calculate min-area rectangle

    Step2: Make center of rectangle Step3: Compare distance to center Step4: Make farthest point as finger tip Min Area Rectangle
  140. Calculation of Center Point private int[] Polygonization(Point[] contour) { int[]

    hullIdx = Cv2.ConvexHullIndices(contour); Point[] hullPt = Cv2.ConvexHull(contour); int[] labels = new int[hullIdx.Length]; Cv2.Partition(hullPt, out labels, Predicate); RotatedRect rect = Cv2.MinAreaRect(hullPt); bgrMat.Circle((int)rect.Center.X, (int)rect.Center.Y, 10, Scalar.Cyan, 3); for (int i=0;i<hullIdx.Length;i++) { int index = hullIdx[i]; bgrMat.Circle(contour[index], 10, Scalar.Red, 2); bgrMat.PutText(labels[i].ToString(), contour[index], HersheyFonts.HersheyDuplex, 1, Scalar.White); } DrawPolyLine(bgr, hullPt, Scalar.Blue); return hullIdx; }
  141. Run & Confirm

  142. Integration of Points Center Point Min Area Rectangle 【Procedure】 Step1:

    Calculate min-area rectangle Step2: Make center of rectangle Step3: Compare distance to center Step4: Make farthest point as finger tip
  143. Integration of Points private int[] Polygonization(Point[] contour) { /*****omitted above*****/

    RotatedRect rect = Cv2.MinAreaRect(hullPt); bgrMat.Circle((int)rect.Center.X, (int)rect.Center.Y, 10, Scalar.Cyan, 3); hullIdx = HullSimplification(contour, hullIdx, labels, rect.Center); for (int i=0;i<hullIdx.Length;i++) { int index = hullIdx[i]; bgrMat.Circle(contour[index], 10, Scalar.Red, 2); } /*****omitted below*****/ } Paset the contents of copy_and_paste.text here.
  144. Run & Confirm

  145. Procedure Input Depth Color Binarization Depth Value Skin Color AND

    Detection
  146. Procedure Input Depth Color Binarization Depth Value Skin Color AND

    Detection
  147. Detection of Finger Tip private int FingerDetection(Point[] contour, int[] hull)

    { int num = 0;//Number of Finger return num; } private void timer1_Tick(object sender, EventArgs e) { int fingerNum = 0; /*****omitted*****/ Binarization(); Point[] contour = ContourExtraction(); if (contour != null) { DrawPolyLine( contour, Scalar.Green); int[] hull = Polygonization(contour); fingerNum = FingerDetection(contour, hull); } /*****omitted below*****/ }
  148. Detection of Finger Tip private int FingerDetection(Point[] contour, int[] hull)

    { int num = 0; //Calculation of defect points Vec4i[] defects = Cv2.ConvexityDefects(contour, hull); //Visualization of defect points. foreach (Vec4i v in defects) { bgrMat.Circle(contour[v.Item2], 5, Scalar.White, 2); } return num; } 【Info.】 ConvexityDefets calculates degree of lack versus convex hull Detail of return value is below. Item0: Index of start point Item1: Index of end point Item2: Index of defect point Item3: Distance to defect point Item0 Item2 Item1
  149. Conditions Considered as Finger Tip p1 tip p2 θ l1

    l2 θ<70[deg] && l 1 +l 2 >60[pixel]
  150. Detection of Finger Tip private int FingerDetection(Point[] contour, int[] hull)

    { Vec4i[] defects = Cv2.ConvexityDefects(contour, hull); foreach (Vec4i v in defects) { bgr.Circle(contour[v.Item2], 5, Scalar.White, 2); } //Judge each point is finger tip or not. for (int i = 0; i < defects.Length; i++) { Vec4i d1 = defects[i]; Vec4i d2 = defects[(i + 1) % defects.Length]; //Defect points and tip. Point p1 = contour[d1.Item2]; Point p2 = contour[d2.Item2]; Point tip = contour[d1.Item1]; //calculate vector p1 = p1 - tip; p2 = p2 - tip; } } p1 tip p2
  151. Detection of Finger Tip for (int i = 0; i

    < defects.Length; i++) { Vec4i d1 = defects[i]; Vec4i d2 = defects[(i + 1) % defects.Length]; Point p1 = contour[d1.Item2]; Point p2 = contour[d2.Item2]; Point tip = contour[d1.Item1]; p1 = p1 - tip; p2 = p2 - tip; //p1,p2の長さを計算 double l1 = Math.Sqrt(p1.X * p1.X + p1.Y * p1.Y); double l2 = Math.Sqrt(p2.X * p2.X + p2.Y * p2.Y); //内積の計算と角度の計算 double dot = Point.DotProduct(p1, p2) / (l1 * l2); double angle = Math.Acos(dot) * 180.0 / Math.PI; if (angle < 70 && (l1+l2) > 60) { bgrMat.Circle(tip, 5, Scalar.Orange, 3); num++; } }
  152. Display Number of Finger ①ToolBox ②Label ③Click Form1

  153. Display Number of Finger ①Put label on upper left of

    PictureBox ②Font Size=20
  154. Display Number of Finger private void timer1_Tick(object sender, EventArgs e)

    { float maxDepth = 0.75f; Bitmap color = kinect.GetColorImage(); Bitmap depth = kinect.GetDepthImage(maxDepth); depthMat = BitmapConverter.ToMat(depth); bgrMat = BitmapConverter.ToMat(color); Binarization(); Point[] contour = ContourExtraction(); int fingerNum = 0; if (contour != null) { DrawPolyLine( contour, Scalar.Green); int[] hull = Polygonization(contour); fingerNum = FingerDetection(contour, hull); } label7.Text = fingerNum.ToString(); pictureBox1.Image = bgrMat.ToBitmap(); pictureBox2.Image = depthMat.ToBitmap(); /*省略*/ }
  155. Complete!