Tutorial of Basic Hand Tracking with Kinect & OpenCV

Slide for AR Fukuoka held 2nd March 2019

1. Tutorial of Basic Hand Tracking with Kinect & OpenCV #AR_Fukuoka
2. Download the Sample Beforehand http://arfukuoka.lolipop.jp/kinect2019mar/sample.zip
3. Goal of the tutorial https://youtu.be/HX6mvVICaDk
4. Related Work
5. Hand Tracking And Finger Detection 【Procedure】
6. Problem  Some skin color areas can cause a misrecognition. Ex) Face, Background image of environment  Simple solutions are.. method 1) Define the area for hand detection. method 2) Detect and exclude face area.
7. Utilization of RGB-D Sensor Real-time RGB Image+Depth
8. Utilization of RGB-D Sensor Available Cut off Threshold
9. Utilization of RGB-D Sensor Available Cut off Threshold
10. Procedure of Tutorial Input Depth Color Binarization Depth Value Skin Color AND Detection
11. Development widh Visual Studio
12. Creating New Project (1/2) ①File ②New ③Project...
13. Creating New Project (2/2) ①VisualC# ②Windows Forms App ③OK
14. Confirmation Start
15. Confirmation Close
16. Disabling AutoScaleMode ②Properties ③AutoScaleMode ④None ①Click
17. Setting Up Visualization Area (1/11) ①ToolBox ②PictureBox ③Click on Form
18. Setting Up Visualization Area (2/11) ①Select PictureBox ②Size=640,480
19. Setting Up Visualization Area (3/11) SizeMode=Zoom *This operation means enable adjustment of image size to picturebox automatically.
20. Setting Up Visualization Area (4/11) Adjust Form size wider
21. Setting Up Visualization Area (5/11) ①Select PictureBox ②[Ctrl+C]→[Ctrl+V]
22. Setting Up Visualization Area (6/11) ①Select 2nd PictureBox ②Size=200,150
23. Setting Up Visualization Area (7/11) Place it on bottom left of the window
24. Setting Up Visualization Area (8/11) Select and [Ctrl+C]→[Ctrl+V]
25. Setting Up Visualization Area (9/11) Place it on bottom center
26. Setting Up Visualization Area (10/11) Select and [Ctrl+C]→[Ctrl+V]
27. Setting Up Visualization Area (11/11) Place it on bottom right
28. Roll of Each PictuerBox ← Final Result ← Result of each Binarization
29. Installation of OpenCV (1/4) ①Tools ②NuGet Pckage Manager ③Manage NuGet Package for Solution
30. Installation of OpenCV (2/4) ①Brows ②nuget.org
31. Installation of OpenCV (3/4) ①Search for “OpenCVsharp” ②OpenCvSharp3 AnyCPU ③Check ④Install
32. Installation of OpenCV (4/4) Close NuGet
33. Please wait for a moment till installation of OpenCV is finished
34. Installation of KinectSDK (1/3) ①Right Click References ②Add Referenence
35. Installation of KinectSDK (2/3) Search for Kinect
36. Installation of KinectSDK (3/3) ①Check MicrosoftKinect 1.8.0.0 ②OK
37. Today, we use color and depth image for this tutorial. Please use KinectGrabber provided for this tutorial since raw data of SDK is not easy to use for beginner.
38. Installation of KinectGrabber (1/4) Move KinectGrabber.csinto the same folder as Form1.cs sample folder Your project
39. Installation of KinectGrabber (2/4) Right click your project
40. Installation of KinectGrabber (3/4) ①Add ②Existing Item
41. Installation of KinectGrabber (4/4) ①KinectGrabber.cs ②Add
42. Setting up Timer & Processing Interval Image processing and visualization will run at a certain interval by using timer. ①ToolBox ②Timer ③Form上をクリック
43. Setting up Timer & Processing Interval ①Timer is added ➂Interval=30[ms]
44. Adding Function for Periodical Processing Double Click timer1
45. Adding Function for Periodical Processing timer1_Tick is generated Timer1_Tick function will run at a certain interval.(Not yet..)
46. Let Timer to Start Automatically ①Form1[Design] ②Select Form1
47. ①Click ②Double Click “Shown” Let Timer to Start Automatically
48. Let Timer to Start Automatically namespace ARFukuokaCV { public partial class Form1 : Form { public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { } private void Form1_Shown(object sender, EventArgs e) { timer1.Start(); } } } ①Form1_Shown is generated ②Write this to start timer
49. Procedure Input Depth Color Binarization Depth Value Skin Color AND Detection
50. Procedure Binarization Depth Value Skin Color AND Detection Input Depth Color
51. Waking up Kinect Sensor using KinectSample; //Importing KinectGrabber namespace Kinect_ARFukuoka { public partial class Form1 : Form { KinectGrabber kinect = new KinectGrabber(); public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { } /*****omitted below*****/
52. Let Kinect to Stop Automatically ①Click ②Double Click FormClosing
53. Let Kinect to Stop Automatically KinectGrabber kinect = new KinectGrabber(); public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { } private void Form1_Shown(object sender, EventArgs e) { timer1.Start(); } private void Form1_FormClosing(object sender, FormClosingEventArgs e) { timer1.Stop(); kinect.Close(); }
54. Run & Confirm Infrared exposure will be started
55. Display Color/Depth Images KinectGrabber kinect = new KinectGrabber(); public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { Bitmap color = kinect.GetColorImage(); //Acquiring color image Bitmap depth = kinect.GetDepthImage(); // Acquiring depth image pictureBox1.Image = color; //Displaying color image pictureBox2.Image = depth; //Displaying depthimage } private void Form1_Shown(object sender, EventArgs e) { timer1.Start(); }
56. Run & Confirm
57. Changing Depth Range to Visualize KinectGrabber kinect = new KinectGrabber(); public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { float maxDepth = 8; Bitmap color = kinect.GetColorImage(); Bitmap depth = kinect.GetDepthImage(); //Depth画像取得 pictureBox1.Image = color; pictureBox2.Image = depth; } private void Form1_Shown(object sender, EventArgs e) { timer1.Start(); } Bitmap depth = kinect.GetDepthImage(maxDepth); This is carried out to let attendees familiarize Kinect
58. Trial 1: Let’s Change Max Depth 【Information】  By using depth threshold, we can remove background image.  Surly you can do the same thing by applying depth filtering to raw depth data from Kinect SDK. maxDepth=8 maxDepth=4 maxDepth=2
59. Trial 2: Let’s Change Max Depth Let's find the value of maxDepth where almost no face appears ex) maxDepth=1 ex) maxDepth=0.7f NG Good
60. Next Step Separate each pixel into 2 information which should be processed or not
61. Let’s start to use OpenCV!
62. Importing OpenCV Library using KinectSample; using OpenCvSharp; using OpenCvSharp.Extensions; namespace Kinect_ARFukuoka { public partial class Form1 : Form { KinectGrabber kinect = new KinectGrabber(); public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { /*****omitted*****/ } /*****omitted below*****/
63. Procedure Binarization Depth Value Skin Color AND Detection Input Depth Color
64. Procedure AND Detection Input Depth Color Binarization Depth Value Skin Color
65. Binarization of Gray Scale Image 0 255 0 255 Separate brightness described as 0~255 into less or above the threshold. Binarization is often used for definition of pixel which should be processed.
66. Conversion of Bitmap image to Mat //Variable for holding depth image Mat depthMat; public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { float maxDepth = 0.75f; Bitmap color = kinect.GetColorImage(); Bitmap depth = kinect.GetDepthImage(maxDepth); //Conversion of Bitmap into Mat depthMat = BitmapConverter.ToMat(depth); pictureBox1.Image = color; pictureBox2.Image = depth; //Releasing memory of depthMat depthMat.Release(); } pictureBox2.Image = depth; pictureBox2.Image = depthMat.ToBitmap();
67. Binarization of Depth Image //Function for binarization private void Binarization() { //BGR→GrayScale depthMat = depthMat.CvtColor(ColorConversionCodes.BGR2GRAY); //Binarize image with threshold=20 depthMat = depthMat.Threshold(20, 255, ThresholdTypes.Binary); } private void timer1_Tick(object sender, EventArgs e) { float maxDepth = 0.75f; Bitmap color = kinect.GetColorImage(); Bitmap depth = kinect.GetDepthImage(maxDepth); depthMat = BitmapConverter.ToMat(depth); //Calling Binarization Binarization(); pictureBox1.Image = color; pictureBox2.Image = depthMat.ToBitmap(); depthMat.Release(); }
68. Run & Confirm Depth Binarization can remove background but still including things that isn’t human skin!
69. Procedure AND Detection Input Depth Color Binarization Depth Value Skin Color
70. Procedure AND Detection Input Depth Color Binarization Depth Value Skin Color
71. Binarization of Color Image 【Want to do】  Making specific color into white and others int black. 【Problem】  Thresholding for RGB color is difficult 【Solution】  Using HSV color space  H(Hue) is described as 0~360° ※0~180 for OpenCV  S(Saturation) is 0~100％ ※0~255 for OpenCV  V(Value) is 0~100％ This means brightness of color ※0~255 for OpenCV RGB HSV
72. Preparation of Mat for Color & Result //Add Mat for color(bgr) and result of binarization(bin) Mat depthMat; public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { float maxDepth = 0.75f; Bitmap color = kinect.GetColorImage(); Bitmap depth = kinect.GetDepthImage(maxDepth); depthMat = BitmapConverter.ToMat(depth); bgrMat = BitmapConverter.ToMat(color); Binarization(); pictureBox1.Image = color; pictureBox2.Image = depthMat.ToBitmap(); depthMat.Release(); bgrMat.Release(); binMat.Release(); } Mat depthMat, bgrMat, binMat;
73. Binarization of HSV Image //Lower/Upper threshold of HSV Scalar lower = new Scalar(0, 0, 0); Scalar upper = new Scalar(180, 255, 255); public Form1() { InitializeComponent(); } private void Binarization() { depthMat = depthMat.CvtColor(ColorConversionCodes.BGR2GRAY); depthMat = depthMat.Threshold(20, 255, ThresholdTypes.Binary); //Convert BGR into HSV color space. Mat hsv = bgrMat.CvtColor(ColorConversionCodes.BGR2HSV); //Binarization of HSV image binMat = hsv.InRange(lower, upper); //Visualization of the result on pictureBox3 pictureBox3.Image = binMat.ToBitmap(); hsv.Release(); } lower upper
74. Run & Confirm Shown as white image
75. Trial：Changing Threshold of Hue Scalar lower = new OpenCvSharp.Scalar(50, 0, 0); Scalar upper = new OpenCvSharp.Scalar(80, 255, 255); Green area is extracted
76. Set Hue’s Threshold as All Range Scalar lower = new OpenCvSharp.Scalar(0, 0, 0); Scalar upper = new OpenCvSharp.Scalar(180, 255, 255);
77. Making H-Threshold Modifiable via UI ①ToolBox ②GroupBox ③Click Form1
78. Making H-Threshold Modifiable via UI ①Put it on the right side ②Click
79. Making H-Threshold Modifiable via UI ①Size=450,190 ②Text=H
80. Making H-Threshold Modifiable via UI ①ToolBox ②Label ③Click GloupBox
81. Making H-Threshold Modifiable via UI ①Put label upper left of groupbox ②Text=Min
82. Making H-Threshold Modifiable via UI ①ToolBox ②TrackBar ③Click GloupBox
83. Making H-Threshold Modifiable via UI ①Put it on right of label ②Name=minH
84. Making H-Threshold Modifiable via UI Maximum:180 Minimum: 0 Value: 0
85. Making H-Threshold Modifiable via UI ①Select TrackBar ②Click ②Double Click Scroll
86. Making H-Threshold Modifiable via UI //Called when value of trackbar minH is changed private void minH_Scroll(object sender, EventArgs e) { //lower value of H-threshold is changed lower.Val0 = minH.Value; } Start
87. Run & Confirm Move Color which is lower than threshold is shown as black
88. Making H-Threshold Modifiable via UI Select Label & TrackBar together
89. Making H-Threshold Modifiable via UI [Ctrl+C]→[Ctrl+V]
90. Making H-Threshold Modifiable via UI ①Select Label ③Text=Max ②Click
91. Making H-Threshold Modifiable via UI Select TrackBar Name=maxH
92. Making H-Threshold Modifiable via UI Value=180
93. Making H-Threshold Modifiable via UI ①Select 2nd Trackbar ②Click ③Double Click Scroll
94. Making H-Threshold Modifiable via UI //Called when value of trackbar maxH is changed private void maxH_Scroll(object sender, EventArgs e) { //upper value of H-threshold is changed upper.Val0 = maxH.Value; } Start
95. Making H-Threshold Modifiable via UI Move Color which is between lower and upper threshold is shown as white area
96. Trial: Let’s Extract Skin Area Noise should be removed
97. Making S-Threshold Modifiable via UI Select GroupBox & [Ctrl+C]→[Ctrl+V]
98. Making S-Threshold Modifiable via UI ①put groupBox2 under 1st groupbox ②Text= S ②Click
99. Making S-Threshold Modifiable via UI ①Trackbar next to Min ②Name=minS
100. Making S-Threshold Modifiable via UI Maximum=255
101. Making S-Threshold Modifiable via UI ①Trackbar next to Max ②Name=maxS
102. Making S-Threshold Modifiable via UI ②Maximum=255 ③Value=255
103. Making S-Threshold Modifiable via UI ①Trackbar next to Min ③Double Click Scroll ②Click
104. Making S-Threshold Modifiable via UI ①Trackbar next to Max ③Double Click Scroll ②Click
105. Making S-Threshold Modifiable via UI //Called when value of trackbar minS is changed private void minS_Scroll(object sender, EventArgs e) { //lower value of S-threshold is changed lower.Val1 = minS.Value; } //Called when value of trackbar maxS is changed private void maxS_Scroll(object sender, EventArgs e) { //upper value of S-threshold is changed upper.Val1 = maxS.Value; }
106. Run & Confirm Large S is shown as White Change minS Large S is shown as Black Change maxS min max min max
107. Making V-Threshold Modifiable via UI Select 2nd GroupBox & [Ctrl+C]→[Ctrl+V]
108. Making V-Threshold Modifiable via UI ①Put it under 2nd GroupBox
109. Making V-Threshold Modifiable via UI ②Text= V ①Click
110. Making V-Threshold Modifiable via UI ①Trackbar next to Min ②Name=minV
111. Making V-Threshold Modifiable via UI ②Double Click Scroll ①Click
112. Making V-Threshold Modifiable via UI ①Trackbar next to Max ③Name=maxV ①Click
113. Making V-Threshold Modifiable via UI ②Double Click Scroll ①Click
114. Making V-Threshold Modifiable via UI //Called when value of trackbar minV is changed private void minV_Scroll(object sender, EventArgs e) { //lower value of V-threshold is changed lower.Val2 = minV.Value; } //Called when value of trackbar maxV is changed private void maxV_Scroll(object sender, EventArgs e) { //upper value of V-threshold is changed upper.Val2 = maxV.Value; }
115. Run & Confirm Small V is shown as black Change minV Small V is shown as white Change maxV min max min max
116. Trial: Let’s Extract Hand Not only hand but also face is extracted by color binarization
117. Procedure AND Detection Input Depth Color Binarization Depth Value Skin Color
118. Procedure Detection Input Depth Color Binarization Depth Value Skin Color AND
119. Extraction Common Area with AND private void Binarization() { //Conversion bgr to gray depthMat = depthMat.CvtColor(ColorConversionCodes.BGR2GRAY); //Binarization with threshold=20 depthMat = depthMat.Threshold(20, 255, ThresholdTypes.Binary); //Conversion bgr to hsv Mat hsv = bgrMat.CvtColor(ColorConversionCodes.BGR2HSV); //Binarization of HSV with lower/upper threshold binMat = hsv.InRange(lower, upper); //Visualization of the result pictureBox3.Image = binMat.ToBitmap(); //Extract common white area between binMat & depthMat Cv2.BitwiseAnd(binMat, depthMat, binMat); //Visualization of the result pictureBox4.Image = binMat.ToBitmap(); hsv.Release(); }
120. Filling Lacked Area & Denoising depth HSV AND 【Filling Lacked Area & Denoising】 (1) Smoothing the image (2) Binarization of the image (1) (2) 欠損 ノイズ
121. Smoothing & Binarization private void Binarization() { //Binarization of depth depthMat = depthMat.CvtColor(ColorConversionCodes.BGR2GRAY); depthMat = depthMat.Threshold(20, 255, ThresholdTypes.Binary); //Conversion of BGR to HSV Mat hsv = bgrMat.CvtColor(ColorConversionCodes.BGR2HSV); //Binarization of HSV binMat = hsv.InRange(lower, upper); //Visualization of the result pictureBox3.Image = binMat.ToBitmap(); Cv2.BitwiseAnd(binMat, depthMat, binMat); //Smoothing with 9*9 neighbor pixels binMat = binMat.Blur(new OpenCvSharp.Size(9, 9)); //Binarization with threshold=60 binMat = binMat.Threshold(60, 256, ThresholdTypes.Binary); pictureBox4.Image = binMat.ToBitmap(); hsv.Release(); }
122. Procedure Detection Input Depth Color Binarization Depth Value Skin Color AND
123. Procedure Input Depth Color Binarization Depth Value Skin Color AND Detection Next
124. Preparation of Contour Extraction //Procedure of Contour Extraction will be written here private Point[] ContourExtraction() { Point[] contour = null; return contour; } private void timer1_Tick(object sender, EventArgs e) { float maxDepth = 0.75f; Bitmap color = kinect.GetColorImage(); Bitmap depth = kinect.GetDepthImage(maxDepth); depthMat = BitmapConverter.ToMat(depth); bgrMat = BitmapConverter.ToMat(color); Binarization(); //Points consisting contour are returned. Point[] contour = ContourExtraction(); pictureBox1.Image = color; /*****omitted below*****/ }
125. Preparation of Contour Extraction
126. Contours (Please edit ContorExtraction) Point[] contour = null; Point[][] lines; HierarchyIndex[] h; //Find contours from binMat binMat.FindContours(out lines, out h, RetrievalModes.External, ContourApproximationModes.ApproxSimple); double maxArea = -1; for (int i = 0; i < lines.Length; i++) { double area = Cv2.ContourArea(lines[i]); //Contour of maximum area is treated as hand if (area > maxArea) { maxArea = area; contour = lines[i]; } } return contour;
127. Visualize Contourline private void DrawPolyLine(Point[] points, Scalar color) { /*described in next slide!*/ } private void timer1_Tick(object sender, EventArgs e) { /*省略*/ bgrMat = BitmapConverter.ToMat(color); Binarization(); Point[] contour = ContourExtraction(); if (contour != null) { DrawPolyLine( contour, Scalar.Green); } pictureBox1.Image = color; pictureBox1.Image = bgrMat.ToBitmap(); /*****omitted below*****/ } pictureBox1.Image = color;
128. 輪郭の表示 private void DrawPolyLine(Point[] points, Scalar color) { Point p = points.Last(); for (int i = 0; i < points.Length; i++) { Cv2.Line(bgrMat, p, points[i], color, 2); p = points[i]; } } 0 1 2 3 4 points[0] 1 2 3 p p points[1] 2 3 4 0 1 2 p points[4] ・・・ i = 0 i = 1 i = 4
129. Run & Confirm
130. Procedure Input Depth Color Binarization Depth Value Skin Color AND Detection Next
131. Convex Hull private int[] Polygonization(Point[] contour) { //Approximate contour line as convex hull. int[] hullIdx = Cv2.ConvexHullIndices(contour); //Approximate contour line as convex hull. Point[] hullPt = Cv2.ConvexHull(contour); DrawPolyLine(hullPt, Scalar.Blue); return hullIdx; } private void timer1_Tick(object sender, EventArgs e) { /***omitted***/ }
132. Convex Hull private void timer1_Tick(object sender, EventArgs e) { float maxDepth = 0.75f; Bitmap color = kinect.GetColorImage(); Bitmap depth = kinect.GetDepthImage(maxDepth); depthMat = BitmapConverter.ToMat(depth); bgrMat = BitmapConverter.ToMat(color); Binarization(); Point[] contour = ContourExtraction(); if (contour != null) { DrawPolyLine( contour, Scalar.Green); int[] hull = Polygonization(contour); } pictureBox1.Image = bgrMat.ToBitmap(); pictureBox2.Image = depthMat.ToBitmap(); /*****omitted below*****/ }
133. Run & Confirm
134. Visualization of Vertices of Convex Hull private int[] Polygonization(Point[] contour) { int[] hullIdx = Cv2.ConvexHullIndices(contour); Point[] hullPt = Cv2.ConvexHull(contour); for(int i=0;i<hullIdx.Length;i++) { int index = hullIdx[i]; bgrMat.Circle(contour[index], 10, Scalar.Red, 2); } DrawPolyLine(bgr, hullPt, Scalar.Blue); return hullIdx; }  Many points are exists at the tip of finger  They should integrated into the one point.
135. Clustering Integrate near points into one point ０ 1 2 3 4 5
136. Clustering private int[] Polygonization(Point[] contour) { int[] hullIdx = Cv2.ConvexHullIndices(contour); Point[] hullPt = Cv2.ConvexHull(contour); //Terating label numbers of each vertices of hull int[] labels; //clustering based on the role written in Predicate Cv2.Partition(hullPt, out labels, Predicate); for (int i = 0; i < hullIdx.Length; i++) { int index = hullIdx[i]; bgrMat.Circle(contour[index], 10, Scalar.Red, 2); //Visualization of label number bgrMat.PutText(labels[i].ToString(), contour[index], HersheyFonts.HersheyDuplex, 1, Scalar.White); } DrawPolyLine(hullPt, Scalar.Blue); return hullIdx; }
137. Clustering private int[] Polygonization(Point[] contour) { /*****omitted above*****/ Cv2.Partition(hullPt, out labels, Predicate); /*****omitted below*****/ } private bool Predicate(Point t1, Point t2) { //2 points are treated as the same point if //distance is less than 20pixels if (t1.DistanceTo(t2) < 20) { return true; } else { return false; } }
138. Run & Confirm
139. Integration of Points Center Point 【Procedure】 Step1: Calculate min-area rectangle Step2: Make center of rectangle Step3: Compare distance to center Step4: Make farthest point as finger tip Min Area Rectangle
140. Calculation of Center Point private int[] Polygonization(Point[] contour) { int[] hullIdx = Cv2.ConvexHullIndices(contour); Point[] hullPt = Cv2.ConvexHull(contour); int[] labels = new int[hullIdx.Length]; Cv2.Partition(hullPt, out labels, Predicate); RotatedRect rect = Cv2.MinAreaRect(hullPt); bgrMat.Circle((int)rect.Center.X, (int)rect.Center.Y, 10, Scalar.Cyan, 3); for (int i=0;i<hullIdx.Length;i++) { int index = hullIdx[i]; bgrMat.Circle(contour[index], 10, Scalar.Red, 2); bgrMat.PutText(labels[i].ToString(), contour[index], HersheyFonts.HersheyDuplex, 1, Scalar.White); } DrawPolyLine(bgr, hullPt, Scalar.Blue); return hullIdx; }
141. Run & Confirm
142. Integration of Points Center Point Min Area Rectangle 【Procedure】 Step1: Calculate min-area rectangle Step2: Make center of rectangle Step3: Compare distance to center Step4: Make farthest point as finger tip
143. Integration of Points private int[] Polygonization(Point[] contour) { /*****omitted above*****/ RotatedRect rect = Cv2.MinAreaRect(hullPt); bgrMat.Circle((int)rect.Center.X, (int)rect.Center.Y, 10, Scalar.Cyan, 3); hullIdx = HullSimplification(contour, hullIdx, labels, rect.Center); for (int i=0;i<hullIdx.Length;i++) { int index = hullIdx[i]; bgrMat.Circle(contour[index], 10, Scalar.Red, 2); } /*****omitted below*****/ } Paset the contents of copy_and_paste.text here.
144. Run & Confirm
145. Procedure Input Depth Color Binarization Depth Value Skin Color AND Detection
146. Procedure Input Depth Color Binarization Depth Value Skin Color AND Detection
147. Detection of Finger Tip private int FingerDetection(Point[] contour, int[] hull) { int num = 0;//Number of Finger return num; } private void timer1_Tick(object sender, EventArgs e) { int fingerNum = 0; /*****omitted*****/ Binarization(); Point[] contour = ContourExtraction(); if (contour != null) { DrawPolyLine( contour, Scalar.Green); int[] hull = Polygonization(contour); fingerNum = FingerDetection(contour, hull); } /*****omitted below*****/ }
148. Detection of Finger Tip private int FingerDetection(Point[] contour, int[] hull) { int num = 0; //Calculation of defect points Vec4i[] defects = Cv2.ConvexityDefects(contour, hull); //Visualization of defect points. foreach (Vec4i v in defects) { bgrMat.Circle(contour[v.Item2], 5, Scalar.White, 2); } return num; } 【Info.】 ConvexityDefets calculates degree of lack versus convex hull Detail of return value is below. Item0: Index of start point Item1: Index of end point Item2: Index of defect point Item3: Distance to defect point Item0 Item2 Item1
149. Conditions Considered as Finger Tip p1 tip p2 θ l1 l2 θ<70[deg] && l 1 +l 2 >60[pixel]
150. Detection of Finger Tip private int FingerDetection(Point[] contour, int[] hull) { Vec4i[] defects = Cv2.ConvexityDefects(contour, hull); foreach (Vec4i v in defects) { bgr.Circle(contour[v.Item2], 5, Scalar.White, 2); } //Judge each point is finger tip or not. for (int i = 0; i < defects.Length; i++) { Vec4i d1 = defects[i]; Vec4i d2 = defects[(i + 1) % defects.Length]; //Defect points and tip. Point p1 = contour[d1.Item2]; Point p2 = contour[d2.Item2]; Point tip = contour[d1.Item1]; //calculate vector p1 = p1 - tip; p2 = p2 - tip; } } p1 tip p2
151. Detection of Finger Tip for (int i = 0; i < defects.Length; i++) { Vec4i d1 = defects[i]; Vec4i d2 = defects[(i + 1) % defects.Length]; Point p1 = contour[d1.Item2]; Point p2 = contour[d2.Item2]; Point tip = contour[d1.Item1]; p1 = p1 - tip; p2 = p2 - tip; //p1,p2の長さを計算 double l1 = Math.Sqrt(p1.X * p1.X + p1.Y * p1.Y); double l2 = Math.Sqrt(p2.X * p2.X + p2.Y * p2.Y); //内積の計算と角度の計算 double dot = Point.DotProduct(p1, p2) / (l1 * l2); double angle = Math.Acos(dot) * 180.0 / Math.PI; if (angle < 70 && (l1+l2) > 60) { bgrMat.Circle(tip, 5, Scalar.Orange, 3); num++; } }
152. Display Number of Finger ①ToolBox ②Label ③Click Form1
153. Display Number of Finger ①Put label on upper left of PictureBox ②Font Size＝20
154. Display Number of Finger private void timer1_Tick(object sender, EventArgs e) { float maxDepth = 0.75f; Bitmap color = kinect.GetColorImage(); Bitmap depth = kinect.GetDepthImage(maxDepth); depthMat = BitmapConverter.ToMat(depth); bgrMat = BitmapConverter.ToMat(color); Binarization(); Point[] contour = ContourExtraction(); int fingerNum = 0; if (contour != null) { DrawPolyLine( contour, Scalar.Green); int[] hull = Polygonization(contour); fingerNum = FingerDetection(contour, hull); } label7.Text = fingerNum.ToString(); pictureBox1.Image = bgrMat.ToBitmap(); pictureBox2.Image = depthMat.ToBitmap(); /*省略*/ }
155. Complete!