Tutorial of Basic Hand Tracking with Kinect & OpenCV

Slide 1

Slide 1 text

Tutorial of Basic Hand Tracking with Kinect & OpenCV #AR_Fukuoka

Slide 2

Slide 2 text

Download the Sample Beforehand http://arfukuoka.lolipop.jp/kinect2019mar/sample.zip

Slide 3

Slide 3 text

Goal of the tutorial https://youtu.be/HX6mvVICaDk

Slide 4

Slide 4 text

Related Work

Slide 5

Slide 5 text

Hand Tracking And Finger Detection 【Procedure】

Slide 6

Slide 6 text

Problem  Some skin color areas can cause a misrecognition. Ex) Face, Background image of environment  Simple solutions are.. method 1) Define the area for hand detection. method 2) Detect and exclude face area.

Slide 7

Slide 7 text

Utilization of RGB-D Sensor Real-time RGB Image+Depth

Slide 8

Slide 8 text

Utilization of RGB-D Sensor Available Cut off Threshold

Slide 9

Slide 9 text

Utilization of RGB-D Sensor Available Cut off Threshold

Slide 10

Slide 10 text

Procedure of Tutorial Input Depth Color Binarization Depth Value Skin Color AND Detection

Slide 11

Slide 11 text

Development widh Visual Studio

Slide 12

Slide 12 text

Creating New Project (1/2) ①File ②New ③Project...

Slide 13

Slide 13 text

Creating New Project (2/2) ①VisualC# ②Windows Forms App ③OK

Slide 14

Slide 14 text

Confirmation Start

Slide 15

Slide 15 text

Confirmation Close

Slide 16

Slide 16 text

Disabling AutoScaleMode ②Properties ③AutoScaleMode ④None ①Click

Slide 17

Slide 17 text

Setting Up Visualization Area (1/11) ①ToolBox ②PictureBox ③Click on Form

Slide 18

Slide 18 text

Setting Up Visualization Area (2/11) ①Select PictureBox ②Size=640,480

Slide 19

Slide 19 text

Setting Up Visualization Area (3/11) SizeMode=Zoom *This operation means enable adjustment of image size to picturebox automatically.

Slide 20

Slide 20 text

Setting Up Visualization Area (4/11) Adjust Form size wider

Slide 21

Slide 21 text

Setting Up Visualization Area (5/11) ①Select PictureBox ②[Ctrl+C]→[Ctrl+V]

Slide 22

Slide 22 text

Setting Up Visualization Area (6/11) ①Select 2nd PictureBox ②Size=200,150

Slide 23

Slide 23 text

Setting Up Visualization Area (7/11) Place it on bottom left of the window

Slide 24

Slide 24 text

Setting Up Visualization Area (8/11) Select and [Ctrl+C]→[Ctrl+V]

Slide 25

Slide 25 text

Setting Up Visualization Area (9/11) Place it on bottom center

Slide 26

Slide 26 text

Setting Up Visualization Area (10/11) Select and [Ctrl+C]→[Ctrl+V]

Slide 27

Slide 27 text

Setting Up Visualization Area (11/11) Place it on bottom right

Slide 28

Slide 28 text

Roll of Each PictuerBox ← Final Result ← Result of each Binarization

Slide 29

Slide 29 text

Installation of OpenCV (1/4) ①Tools ②NuGet Pckage Manager ③Manage NuGet Package for Solution

Slide 30

Slide 30 text

Installation of OpenCV (2/4) ①Brows ②nuget.org

Slide 31

Slide 31 text

Installation of OpenCV (3/4) ①Search for “OpenCVsharp” ②OpenCvSharp3 AnyCPU ③Check ④Install

Slide 32

Slide 32 text

Installation of OpenCV (4/4) Close NuGet

Slide 33

Slide 33 text

Please wait for a moment till installation of OpenCV is finished

Slide 34

Slide 34 text

Installation of KinectSDK (1/3) ①Right Click References ②Add Referenence

Slide 35

Slide 35 text

Installation of KinectSDK (2/3) Search for Kinect

Slide 36

Slide 36 text

Installation of KinectSDK (3/3) ①Check MicrosoftKinect 1.8.0.0 ②OK

Slide 37

Slide 37 text

Today, we use color and depth image for this tutorial. Please use KinectGrabber provided for this tutorial since raw data of SDK is not easy to use for beginner.

Slide 38

Slide 38 text

Installation of KinectGrabber (1/4) Move KinectGrabber.csinto the same folder as Form1.cs sample folder Your project

Slide 39

Slide 39 text

Installation of KinectGrabber (2/4) Right click your project

Slide 40

Slide 40 text

Installation of KinectGrabber (3/4) ①Add ②Existing Item

Slide 41

Slide 41 text

Installation of KinectGrabber (4/4) ①KinectGrabber.cs ②Add

Slide 42

Slide 42 text

Setting up Timer & Processing Interval Image processing and visualization will run at a certain interval by using timer. ①ToolBox ②Timer ③Form上をクリック

Slide 43

Slide 43 text

Setting up Timer & Processing Interval ①Timer is added ➂Interval=30[ms]

Slide 44

Slide 44 text

Adding Function for Periodical Processing Double Click timer1

Slide 45

Slide 45 text

Adding Function for Periodical Processing timer1_Tick is generated Timer1_Tick function will run at a certain interval.(Not yet..)

Slide 46

Slide 46 text

Let Timer to Start Automatically ①Form1[Design] ②Select Form1

Slide 47

Slide 47 text

①Click ②Double Click “Shown” Let Timer to Start Automatically

Slide 48

Slide 48 text

Let Timer to Start Automatically namespace ARFukuokaCV { public partial class Form1 : Form { public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { } private void Form1_Shown(object sender, EventArgs e) { timer1.Start(); } } } ①Form1_Shown is generated ②Write this to start timer

Slide 49

Slide 49 text

Procedure Input Depth Color Binarization Depth Value Skin Color AND Detection

Slide 50

Slide 50 text

Procedure Binarization Depth Value Skin Color AND Detection Input Depth Color

Slide 51

Slide 51 text

Waking up Kinect Sensor using KinectSample; //Importing KinectGrabber namespace Kinect_ARFukuoka { public partial class Form1 : Form { KinectGrabber kinect = new KinectGrabber(); public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { } /*****omitted below*****/

Slide 52

Slide 52 text

Let Kinect to Stop Automatically ①Click ②Double Click FormClosing

Slide 53

Slide 53 text

Let Kinect to Stop Automatically KinectGrabber kinect = new KinectGrabber(); public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { } private void Form1_Shown(object sender, EventArgs e) { timer1.Start(); } private void Form1_FormClosing(object sender, FormClosingEventArgs e) { timer1.Stop(); kinect.Close(); }

Slide 54

Slide 54 text

Run & Confirm Infrared exposure will be started

Slide 55

Slide 55 text

Display Color/Depth Images KinectGrabber kinect = new KinectGrabber(); public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { Bitmap color = kinect.GetColorImage(); //Acquiring color image Bitmap depth = kinect.GetDepthImage(); // Acquiring depth image pictureBox1.Image = color; //Displaying color image pictureBox2.Image = depth; //Displaying depthimage } private void Form1_Shown(object sender, EventArgs e) { timer1.Start(); }

Slide 56

Slide 56 text

Run & Confirm

Slide 57

Slide 57 text

Changing Depth Range to Visualize KinectGrabber kinect = new KinectGrabber(); public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { float maxDepth = 8; Bitmap color = kinect.GetColorImage(); Bitmap depth = kinect.GetDepthImage(); //Depth画像取得 pictureBox1.Image = color; pictureBox2.Image = depth; } private void Form1_Shown(object sender, EventArgs e) { timer1.Start(); } Bitmap depth = kinect.GetDepthImage(maxDepth); This is carried out to let attendees familiarize Kinect

Slide 58

Slide 58 text

Trial 1: Let’s Change Max Depth 【Information】  By using depth threshold, we can remove background image.  Surly you can do the same thing by applying depth filtering to raw depth data from Kinect SDK. maxDepth=8 maxDepth=4 maxDepth=2

Slide 59

Slide 59 text

Trial 2: Let’s Change Max Depth Let's find the value of maxDepth where almost no face appears ex) maxDepth=1 ex) maxDepth=0.7f NG Good

Slide 60

Slide 60 text

Next Step Separate each pixel into 2 information which should be processed or not

Slide 61

Slide 61 text

Let’s start to use OpenCV!

Slide 62

Slide 62 text

Importing OpenCV Library using KinectSample; using OpenCvSharp; using OpenCvSharp.Extensions; namespace Kinect_ARFukuoka { public partial class Form1 : Form { KinectGrabber kinect = new KinectGrabber(); public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { /*****omitted*****/ } /*****omitted below*****/

Slide 63

Slide 63 text

Procedure Binarization Depth Value Skin Color AND Detection Input Depth Color

Slide 64

Slide 64 text

Procedure AND Detection Input Depth Color Binarization Depth Value Skin Color

Slide 65

Slide 65 text

Binarization of Gray Scale Image 0 255 0 255 Separate brightness described as 0~255 into less or above the threshold. Binarization is often used for definition of pixel which should be processed.

Slide 66

Slide 66 text

Conversion of Bitmap image to Mat //Variable for holding depth image Mat depthMat; public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { float maxDepth = 0.75f; Bitmap color = kinect.GetColorImage(); Bitmap depth = kinect.GetDepthImage(maxDepth); //Conversion of Bitmap into Mat depthMat = BitmapConverter.ToMat(depth); pictureBox1.Image = color; pictureBox2.Image = depth; //Releasing memory of depthMat depthMat.Release(); } pictureBox2.Image = depth; pictureBox2.Image = depthMat.ToBitmap();

Slide 67

Slide 67 text

Binarization of Depth Image //Function for binarization private void Binarization() { //BGR→GrayScale depthMat = depthMat.CvtColor(ColorConversionCodes.BGR2GRAY); //Binarize image with threshold=20 depthMat = depthMat.Threshold(20, 255, ThresholdTypes.Binary); } private void timer1_Tick(object sender, EventArgs e) { float maxDepth = 0.75f; Bitmap color = kinect.GetColorImage(); Bitmap depth = kinect.GetDepthImage(maxDepth); depthMat = BitmapConverter.ToMat(depth); //Calling Binarization Binarization(); pictureBox1.Image = color; pictureBox2.Image = depthMat.ToBitmap(); depthMat.Release(); }

Slide 68

Slide 68 text

Run & Confirm Depth Binarization can remove background but still including things that isn’t human skin!

Slide 69

Slide 69 text

Procedure AND Detection Input Depth Color Binarization Depth Value Skin Color

Slide 70

Slide 70 text

Procedure AND Detection Input Depth Color Binarization Depth Value Skin Color

Slide 71

Slide 71 text

Binarization of Color Image 【Want to do】  Making specific color into white and others int black. 【Problem】  Thresholding for RGB color is difficult 【Solution】  Using HSV color space  H(Hue) is described as 0~360° ※0~180 for OpenCV  S(Saturation) is 0~100％ ※0~255 for OpenCV  V(Value) is 0~100％ This means brightness of color ※0~255 for OpenCV RGB HSV

Slide 72

Slide 72 text

Preparation of Mat for Color & Result //Add Mat for color(bgr) and result of binarization(bin) Mat depthMat; public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { float maxDepth = 0.75f; Bitmap color = kinect.GetColorImage(); Bitmap depth = kinect.GetDepthImage(maxDepth); depthMat = BitmapConverter.ToMat(depth); bgrMat = BitmapConverter.ToMat(color); Binarization(); pictureBox1.Image = color; pictureBox2.Image = depthMat.ToBitmap(); depthMat.Release(); bgrMat.Release(); binMat.Release(); } Mat depthMat, bgrMat, binMat;

Slide 73

Slide 73 text

Binarization of HSV Image //Lower/Upper threshold of HSV Scalar lower = new Scalar(0, 0, 0); Scalar upper = new Scalar(180, 255, 255); public Form1() { InitializeComponent(); } private void Binarization() { depthMat = depthMat.CvtColor(ColorConversionCodes.BGR2GRAY); depthMat = depthMat.Threshold(20, 255, ThresholdTypes.Binary); //Convert BGR into HSV color space. Mat hsv = bgrMat.CvtColor(ColorConversionCodes.BGR2HSV); //Binarization of HSV image binMat = hsv.InRange(lower, upper); //Visualization of the result on pictureBox3 pictureBox3.Image = binMat.ToBitmap(); hsv.Release(); } lower upper

Slide 74

Slide 74 text

Run & Confirm Shown as white image

Slide 75

Slide 75 text

Trial：Changing Threshold of Hue Scalar lower = new OpenCvSharp.Scalar(50, 0, 0); Scalar upper = new OpenCvSharp.Scalar(80, 255, 255); Green area is extracted

Slide 76

Slide 76 text

Set Hue’s Threshold as All Range Scalar lower = new OpenCvSharp.Scalar(0, 0, 0); Scalar upper = new OpenCvSharp.Scalar(180, 255, 255);

Slide 77

Slide 77 text

Making H-Threshold Modifiable via UI ①ToolBox ②GroupBox ③Click Form1

Slide 78

Slide 78 text

Making H-Threshold Modifiable via UI ①Put it on the right side ②Click

Slide 79

Slide 79 text

Making H-Threshold Modifiable via UI ①Size=450,190 ②Text=H

Slide 80

Slide 80 text

Making H-Threshold Modifiable via UI ①ToolBox ②Label ③Click GloupBox

Slide 81

Slide 81 text

Making H-Threshold Modifiable via UI ①Put label upper left of groupbox ②Text=Min

Slide 82

Slide 82 text

Making H-Threshold Modifiable via UI ①ToolBox ②TrackBar ③Click GloupBox

Slide 83

Slide 83 text

Making H-Threshold Modifiable via UI ①Put it on right of label ②Name=minH

Slide 84

Slide 84 text

Making H-Threshold Modifiable via UI Maximum:180 Minimum: 0 Value: 0

Slide 85

Slide 85 text

Making H-Threshold Modifiable via UI ①Select TrackBar ②Click ②Double Click Scroll

Slide 86

Slide 86 text

Making H-Threshold Modifiable via UI //Called when value of trackbar minH is changed private void minH_Scroll(object sender, EventArgs e) { //lower value of H-threshold is changed lower.Val0 = minH.Value; } Start

Slide 87

Slide 87 text

Run & Confirm Move Color which is lower than threshold is shown as black

Slide 88

Slide 88 text

Making H-Threshold Modifiable via UI Select Label & TrackBar together

Slide 89

Slide 89 text

Making H-Threshold Modifiable via UI [Ctrl+C]→[Ctrl+V]

Slide 90

Slide 90 text

Making H-Threshold Modifiable via UI ①Select Label ③Text=Max ②Click

Slide 91

Slide 91 text

Making H-Threshold Modifiable via UI Select TrackBar Name=maxH

Slide 92

Slide 92 text

Making H-Threshold Modifiable via UI Value=180

Slide 93

Slide 93 text

Making H-Threshold Modifiable via UI ①Select 2nd Trackbar ②Click ③Double Click Scroll

Slide 94

Slide 94 text

Making H-Threshold Modifiable via UI //Called when value of trackbar maxH is changed private void maxH_Scroll(object sender, EventArgs e) { //upper value of H-threshold is changed upper.Val0 = maxH.Value; } Start

Slide 95

Slide 95 text

Making H-Threshold Modifiable via UI Move Color which is between lower and upper threshold is shown as white area

Slide 96

Slide 96 text

Trial: Let’s Extract Skin Area Noise should be removed

Slide 97

Slide 97 text

Making S-Threshold Modifiable via UI Select GroupBox & [Ctrl+C]→[Ctrl+V]

Slide 98

Slide 98 text

Making S-Threshold Modifiable via UI ①put groupBox2 under 1st groupbox ②Text= S ②Click

Slide 99

Slide 99 text

Making S-Threshold Modifiable via UI ①Trackbar next to Min ②Name=minS

Slide 100

Slide 100 text

Making S-Threshold Modifiable via UI Maximum=255

Slide 101

Slide 101 text

Making S-Threshold Modifiable via UI ①Trackbar next to Max ②Name=maxS

Slide 102

Slide 102 text

Making S-Threshold Modifiable via UI ②Maximum=255 ③Value=255

Slide 103

Slide 103 text

Making S-Threshold Modifiable via UI ①Trackbar next to Min ③Double Click Scroll ②Click

Slide 104

Slide 104 text

Making S-Threshold Modifiable via UI ①Trackbar next to Max ③Double Click Scroll ②Click

Slide 105

Slide 105 text

Making S-Threshold Modifiable via UI //Called when value of trackbar minS is changed private void minS_Scroll(object sender, EventArgs e) { //lower value of S-threshold is changed lower.Val1 = minS.Value; } //Called when value of trackbar maxS is changed private void maxS_Scroll(object sender, EventArgs e) { //upper value of S-threshold is changed upper.Val1 = maxS.Value; }

Slide 106

Slide 106 text

Run & Confirm Large S is shown as White Change minS Large S is shown as Black Change maxS min max min max

Slide 107

Slide 107 text

Making V-Threshold Modifiable via UI Select 2nd GroupBox & [Ctrl+C]→[Ctrl+V]

Slide 108

Slide 108 text

Making V-Threshold Modifiable via UI ①Put it under 2nd GroupBox

Slide 109

Slide 109 text

Making V-Threshold Modifiable via UI ②Text= V ①Click

Slide 110

Slide 110 text

Making V-Threshold Modifiable via UI ①Trackbar next to Min ②Name=minV

Slide 111

Slide 111 text

Making V-Threshold Modifiable via UI ②Double Click Scroll ①Click

Slide 112

Slide 112 text

Making V-Threshold Modifiable via UI ①Trackbar next to Max ③Name=maxV ①Click

Slide 113

Slide 113 text

Making V-Threshold Modifiable via UI ②Double Click Scroll ①Click

Slide 114

Slide 114 text

Making V-Threshold Modifiable via UI //Called when value of trackbar minV is changed private void minV_Scroll(object sender, EventArgs e) { //lower value of V-threshold is changed lower.Val2 = minV.Value; } //Called when value of trackbar maxV is changed private void maxV_Scroll(object sender, EventArgs e) { //upper value of V-threshold is changed upper.Val2 = maxV.Value; }

Slide 115

Slide 115 text

Run & Confirm Small V is shown as black Change minV Small V is shown as white Change maxV min max min max

Slide 116

Slide 116 text

Trial: Let’s Extract Hand Not only hand but also face is extracted by color binarization

Slide 117

Slide 117 text

Procedure AND Detection Input Depth Color Binarization Depth Value Skin Color

Slide 118

Slide 118 text

Procedure Detection Input Depth Color Binarization Depth Value Skin Color AND

Slide 119

Slide 119 text

Extraction Common Area with AND private void Binarization() { //Conversion bgr to gray depthMat = depthMat.CvtColor(ColorConversionCodes.BGR2GRAY); //Binarization with threshold=20 depthMat = depthMat.Threshold(20, 255, ThresholdTypes.Binary); //Conversion bgr to hsv Mat hsv = bgrMat.CvtColor(ColorConversionCodes.BGR2HSV); //Binarization of HSV with lower/upper threshold binMat = hsv.InRange(lower, upper); //Visualization of the result pictureBox3.Image = binMat.ToBitmap(); //Extract common white area between binMat & depthMat Cv2.BitwiseAnd(binMat, depthMat, binMat); //Visualization of the result pictureBox4.Image = binMat.ToBitmap(); hsv.Release(); }

Slide 120

Slide 120 text

Filling Lacked Area & Denoising depth HSV AND 【Filling Lacked Area & Denoising】 (1) Smoothing the image (2) Binarization of the image (1) (2) 欠損ノイズ

Slide 121

Slide 121 text

Smoothing & Binarization private void Binarization() { //Binarization of depth depthMat = depthMat.CvtColor(ColorConversionCodes.BGR2GRAY); depthMat = depthMat.Threshold(20, 255, ThresholdTypes.Binary); //Conversion of BGR to HSV Mat hsv = bgrMat.CvtColor(ColorConversionCodes.BGR2HSV); //Binarization of HSV binMat = hsv.InRange(lower, upper); //Visualization of the result pictureBox3.Image = binMat.ToBitmap(); Cv2.BitwiseAnd(binMat, depthMat, binMat); //Smoothing with 9*9 neighbor pixels binMat = binMat.Blur(new OpenCvSharp.Size(9, 9)); //Binarization with threshold=60 binMat = binMat.Threshold(60, 256, ThresholdTypes.Binary); pictureBox4.Image = binMat.ToBitmap(); hsv.Release(); }

Slide 122

Slide 122 text

Procedure Detection Input Depth Color Binarization Depth Value Skin Color AND

Slide 123

Slide 123 text

Procedure Input Depth Color Binarization Depth Value Skin Color AND Detection Next

Slide 124

Slide 124 text

Preparation of Contour Extraction //Procedure of Contour Extraction will be written here private Point[] ContourExtraction() { Point[] contour = null; return contour; } private void timer1_Tick(object sender, EventArgs e) { float maxDepth = 0.75f; Bitmap color = kinect.GetColorImage(); Bitmap depth = kinect.GetDepthImage(maxDepth); depthMat = BitmapConverter.ToMat(depth); bgrMat = BitmapConverter.ToMat(color); Binarization(); //Points consisting contour are returned. Point[] contour = ContourExtraction(); pictureBox1.Image = color; /*****omitted below*****/ }

Slide 125

Slide 125 text

Preparation of Contour Extraction

Slide 126

Slide 126 text

Contours (Please edit ContorExtraction) Point[] contour = null; Point[][] lines; HierarchyIndex[] h; //Find contours from binMat binMat.FindContours(out lines, out h, RetrievalModes.External, ContourApproximationModes.ApproxSimple); double maxArea = -1; for (int i = 0; i < lines.Length; i++) { double area = Cv2.ContourArea(lines[i]); //Contour of maximum area is treated as hand if (area > maxArea) { maxArea = area; contour = lines[i]; } } return contour;

Slide 127

Slide 127 text

Visualize Contourline private void DrawPolyLine(Point[] points, Scalar color) { /*described in next slide!*/ } private void timer1_Tick(object sender, EventArgs e) { /*省略*/ bgrMat = BitmapConverter.ToMat(color); Binarization(); Point[] contour = ContourExtraction(); if (contour != null) { DrawPolyLine( contour, Scalar.Green); } pictureBox1.Image = color; pictureBox1.Image = bgrMat.ToBitmap(); /*****omitted below*****/ } pictureBox1.Image = color;

Slide 128

Slide 128 text

輪郭の表示 private void DrawPolyLine(Point[] points, Scalar color) { Point p = points.Last(); for (int i = 0; i < points.Length; i++) { Cv2.Line(bgrMat, p, points[i], color, 2); p = points[i]; } } 0 1 2 3 4 points[0] 1 2 3 p p points[1] 2 3 4 0 1 2 p points[4] ・・・ i = 0 i = 1 i = 4

Slide 129

Slide 129 text

Run & Confirm

Slide 130

Slide 130 text

Procedure Input Depth Color Binarization Depth Value Skin Color AND Detection Next

Slide 131

Slide 131 text

Convex Hull private int[] Polygonization(Point[] contour) { //Approximate contour line as convex hull. int[] hullIdx = Cv2.ConvexHullIndices(contour); //Approximate contour line as convex hull. Point[] hullPt = Cv2.ConvexHull(contour); DrawPolyLine(hullPt, Scalar.Blue); return hullIdx; } private void timer1_Tick(object sender, EventArgs e) { /***omitted***/ }

Slide 132

Slide 132 text

Convex Hull private void timer1_Tick(object sender, EventArgs e) { float maxDepth = 0.75f; Bitmap color = kinect.GetColorImage(); Bitmap depth = kinect.GetDepthImage(maxDepth); depthMat = BitmapConverter.ToMat(depth); bgrMat = BitmapConverter.ToMat(color); Binarization(); Point[] contour = ContourExtraction(); if (contour != null) { DrawPolyLine( contour, Scalar.Green); int[] hull = Polygonization(contour); } pictureBox1.Image = bgrMat.ToBitmap(); pictureBox2.Image = depthMat.ToBitmap(); /*****omitted below*****/ }

Slide 133

Slide 133 text

Run & Confirm

Slide 134

Slide 134 text

Visualization of Vertices of Convex Hull private int[] Polygonization(Point[] contour) { int[] hullIdx = Cv2.ConvexHullIndices(contour); Point[] hullPt = Cv2.ConvexHull(contour); for(int i=0;i

Slide 135

Slide 135 text

Clustering Integrate near points into one point ０ 1 2 3 4 5

Slide 136

Slide 136 text

Clustering private int[] Polygonization(Point[] contour) { int[] hullIdx = Cv2.ConvexHullIndices(contour); Point[] hullPt = Cv2.ConvexHull(contour); //Terating label numbers of each vertices of hull int[] labels; //clustering based on the role written in Predicate Cv2.Partition(hullPt, out labels, Predicate); for (int i = 0; i < hullIdx.Length; i++) { int index = hullIdx[i]; bgrMat.Circle(contour[index], 10, Scalar.Red, 2); //Visualization of label number bgrMat.PutText(labels[i].ToString(), contour[index], HersheyFonts.HersheyDuplex, 1, Scalar.White); } DrawPolyLine(hullPt, Scalar.Blue); return hullIdx; }

Slide 137

Slide 137 text

Clustering private int[] Polygonization(Point[] contour) { /*****omitted above*****/ Cv2.Partition(hullPt, out labels, Predicate); /*****omitted below*****/ } private bool Predicate(Point t1, Point t2) { //2 points are treated as the same point if //distance is less than 20pixels if (t1.DistanceTo(t2) < 20) { return true; } else { return false; } }

Slide 138

Slide 138 text

Run & Confirm

Slide 139

Slide 139 text

Integration of Points Center Point 【Procedure】 Step1: Calculate min-area rectangle Step2: Make center of rectangle Step3: Compare distance to center Step4: Make farthest point as finger tip Min Area Rectangle

Slide 140

Slide 140 text

Calculation of Center Point private int[] Polygonization(Point[] contour) { int[] hullIdx = Cv2.ConvexHullIndices(contour); Point[] hullPt = Cv2.ConvexHull(contour); int[] labels = new int[hullIdx.Length]; Cv2.Partition(hullPt, out labels, Predicate); RotatedRect rect = Cv2.MinAreaRect(hullPt); bgrMat.Circle((int)rect.Center.X, (int)rect.Center.Y, 10, Scalar.Cyan, 3); for (int i=0;i

Slide 141

Slide 141 text

Run & Confirm

Slide 142

Slide 142 text

Integration of Points Center Point Min Area Rectangle 【Procedure】 Step1: Calculate min-area rectangle Step2: Make center of rectangle Step3: Compare distance to center Step4: Make farthest point as finger tip

Slide 143

Slide 143 text

Integration of Points private int[] Polygonization(Point[] contour) { /*****omitted above*****/ RotatedRect rect = Cv2.MinAreaRect(hullPt); bgrMat.Circle((int)rect.Center.X, (int)rect.Center.Y, 10, Scalar.Cyan, 3); hullIdx = HullSimplification(contour, hullIdx, labels, rect.Center); for (int i=0;i

Slide 144

Slide 144 text

Run & Confirm

Slide 145

Slide 145 text

Procedure Input Depth Color Binarization Depth Value Skin Color AND Detection

Slide 146

Slide 146 text

Procedure Input Depth Color Binarization Depth Value Skin Color AND Detection

Slide 147

Slide 147 text

Detection of Finger Tip private int FingerDetection(Point[] contour, int[] hull) { int num = 0;//Number of Finger return num; } private void timer1_Tick(object sender, EventArgs e) { int fingerNum = 0; /*****omitted*****/ Binarization(); Point[] contour = ContourExtraction(); if (contour != null) { DrawPolyLine( contour, Scalar.Green); int[] hull = Polygonization(contour); fingerNum = FingerDetection(contour, hull); } /*****omitted below*****/ }

Slide 148

Slide 148 text

Detection of Finger Tip private int FingerDetection(Point[] contour, int[] hull) { int num = 0; //Calculation of defect points Vec4i[] defects = Cv2.ConvexityDefects(contour, hull); //Visualization of defect points. foreach (Vec4i v in defects) { bgrMat.Circle(contour[v.Item2], 5, Scalar.White, 2); } return num; } 【Info.】 ConvexityDefets calculates degree of lack versus convex hull Detail of return value is below. Item0: Index of start point Item1: Index of end point Item2: Index of defect point Item3: Distance to defect point Item0 Item2 Item1

Slide 149

Slide 149 text

Conditions Considered as Finger Tip p1 tip p2 θ l1 l2 θ<70[deg] && l 1 +l 2 >60[pixel]

Slide 150

Slide 150 text

Detection of Finger Tip private int FingerDetection(Point[] contour, int[] hull) { Vec4i[] defects = Cv2.ConvexityDefects(contour, hull); foreach (Vec4i v in defects) { bgr.Circle(contour[v.Item2], 5, Scalar.White, 2); } //Judge each point is finger tip or not. for (int i = 0; i < defects.Length; i++) { Vec4i d1 = defects[i]; Vec4i d2 = defects[(i + 1) % defects.Length]; //Defect points and tip. Point p1 = contour[d1.Item2]; Point p2 = contour[d2.Item2]; Point tip = contour[d1.Item1]; //calculate vector p1 = p1 - tip; p2 = p2 - tip; } } p1 tip p2

Slide 151

Slide 151 text

Detection of Finger Tip for (int i = 0; i < defects.Length; i++) { Vec4i d1 = defects[i]; Vec4i d2 = defects[(i + 1) % defects.Length]; Point p1 = contour[d1.Item2]; Point p2 = contour[d2.Item2]; Point tip = contour[d1.Item1]; p1 = p1 - tip; p2 = p2 - tip; //p1,p2の長さを計算 double l1 = Math.Sqrt(p1.X * p1.X + p1.Y * p1.Y); double l2 = Math.Sqrt(p2.X * p2.X + p2.Y * p2.Y); //内積の計算と角度の計算 double dot = Point.DotProduct(p1, p2) / (l1 * l2); double angle = Math.Acos(dot) * 180.0 / Math.PI; if (angle < 70 && (l1+l2) > 60) { bgrMat.Circle(tip, 5, Scalar.Orange, 3); num++; } }

Slide 152

Slide 152 text

Display Number of Finger ①ToolBox ②Label ③Click Form1

Slide 153

Slide 153 text

Display Number of Finger ①Put label on upper left of PictureBox ②Font Size＝20

Slide 154

Slide 154 text

Display Number of Finger private void timer1_Tick(object sender, EventArgs e) { float maxDepth = 0.75f; Bitmap color = kinect.GetColorImage(); Bitmap depth = kinect.GetDepthImage(maxDepth); depthMat = BitmapConverter.ToMat(depth); bgrMat = BitmapConverter.ToMat(color); Binarization(); Point[] contour = ContourExtraction(); int fingerNum = 0; if (contour != null) { DrawPolyLine( contour, Scalar.Green); int[] hull = Polygonization(contour); fingerNum = FingerDetection(contour, hull); } label7.Text = fingerNum.ToString(); pictureBox1.Image = bgrMat.ToBitmap(); pictureBox2.Image = depthMat.ToBitmap(); /*省略*/ }

Slide 155

Slide 155 text

Complete!