Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

Tutorial of Basic Hand Tracking with Kinect & ...

Tutorial of Basic Hand Tracking with Kinect & OpenCV

Slide for AR Fukuoka held 2nd March 2019

TakashiYoshinaga

March 02, 2019
Tweet

More Decks by TakashiYoshinaga

Other Decks in Technology

Transcript

  1. Problem  Some skin color areas can cause a misrecognition.

    Ex) Face, Background image of environment  Simple solutions are.. method 1) Define the area for hand detection. method 2) Detect and exclude face area.
  2. Setting Up Visualization Area (3/11) SizeMode=Zoom *This operation means enable

    adjustment of image size to picturebox automatically.
  3. Today, we use color and depth image for this tutorial.

    Please use KinectGrabber provided for this tutorial since raw data of SDK is not easy to use for beginner.
  4. Setting up Timer & Processing Interval Image processing and visualization

    will run at a certain interval by using timer. ①ToolBox ②Timer ③Form上をクリック
  5. Let Timer to Start Automatically namespace ARFukuokaCV { public partial

    class Form1 : Form { public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { } private void Form1_Shown(object sender, EventArgs e) { timer1.Start(); } } } ①Form1_Shown is generated ②Write this to start timer
  6. Waking up Kinect Sensor using KinectSample; //Importing KinectGrabber namespace Kinect_ARFukuoka

    { public partial class Form1 : Form { KinectGrabber kinect = new KinectGrabber(); public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { } /*****omitted below*****/
  7. Let Kinect to Stop Automatically KinectGrabber kinect = new KinectGrabber();

    public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { } private void Form1_Shown(object sender, EventArgs e) { timer1.Start(); } private void Form1_FormClosing(object sender, FormClosingEventArgs e) { timer1.Stop(); kinect.Close(); }
  8. Display Color/Depth Images KinectGrabber kinect = new KinectGrabber(); public Form1()

    { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { Bitmap color = kinect.GetColorImage(); //Acquiring color image Bitmap depth = kinect.GetDepthImage(); // Acquiring depth image pictureBox1.Image = color; //Displaying color image pictureBox2.Image = depth; //Displaying depthimage } private void Form1_Shown(object sender, EventArgs e) { timer1.Start(); }
  9. Changing Depth Range to Visualize KinectGrabber kinect = new KinectGrabber();

    public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { float maxDepth = 8; Bitmap color = kinect.GetColorImage(); Bitmap depth = kinect.GetDepthImage(); //Depth画像取得 pictureBox1.Image = color; pictureBox2.Image = depth; } private void Form1_Shown(object sender, EventArgs e) { timer1.Start(); } Bitmap depth = kinect.GetDepthImage(maxDepth); This is carried out to let attendees familiarize Kinect
  10. Trial 1: Let’s Change Max Depth 【Information】  By using

    depth threshold, we can remove background image.  Surly you can do the same thing by applying depth filtering to raw depth data from Kinect SDK. maxDepth=8 maxDepth=4 maxDepth=2
  11. Trial 2: Let’s Change Max Depth Let's find the value

    of maxDepth where almost no face appears ex) maxDepth=1 ex) maxDepth=0.7f NG Good
  12. Importing OpenCV Library using KinectSample; using OpenCvSharp; using OpenCvSharp.Extensions; namespace

    Kinect_ARFukuoka { public partial class Form1 : Form { KinectGrabber kinect = new KinectGrabber(); public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { /*****omitted*****/ } /*****omitted below*****/
  13. Binarization of Gray Scale Image 0 255 0 255 Separate

    brightness described as 0~255 into less or above the threshold. Binarization is often used for definition of pixel which should be processed.
  14. Conversion of Bitmap image to Mat //Variable for holding depth

    image Mat depthMat; public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { float maxDepth = 0.75f; Bitmap color = kinect.GetColorImage(); Bitmap depth = kinect.GetDepthImage(maxDepth); //Conversion of Bitmap into Mat depthMat = BitmapConverter.ToMat(depth); pictureBox1.Image = color; pictureBox2.Image = depth; //Releasing memory of depthMat depthMat.Release(); } pictureBox2.Image = depth; pictureBox2.Image = depthMat.ToBitmap();
  15. Binarization of Depth Image //Function for binarization private void Binarization()

    { //BGR→GrayScale depthMat = depthMat.CvtColor(ColorConversionCodes.BGR2GRAY); //Binarize image with threshold=20 depthMat = depthMat.Threshold(20, 255, ThresholdTypes.Binary); } private void timer1_Tick(object sender, EventArgs e) { float maxDepth = 0.75f; Bitmap color = kinect.GetColorImage(); Bitmap depth = kinect.GetDepthImage(maxDepth); depthMat = BitmapConverter.ToMat(depth); //Calling Binarization Binarization(); pictureBox1.Image = color; pictureBox2.Image = depthMat.ToBitmap(); depthMat.Release(); }
  16. Run & Confirm Depth Binarization can remove background but still

    including things that isn’t human skin!
  17. Binarization of Color Image 【Want to do】  Making specific

    color into white and others int black. 【Problem】  Thresholding for RGB color is difficult 【Solution】  Using HSV color space  H(Hue) is described as 0~360° ※0~180 for OpenCV  S(Saturation) is 0~100% ※0~255 for OpenCV  V(Value) is 0~100% This means brightness of color ※0~255 for OpenCV RGB HSV
  18. Preparation of Mat for Color & Result //Add Mat for

    color(bgr) and result of binarization(bin) Mat depthMat; public Form1() { InitializeComponent(); } private void timer1_Tick(object sender, EventArgs e) { float maxDepth = 0.75f; Bitmap color = kinect.GetColorImage(); Bitmap depth = kinect.GetDepthImage(maxDepth); depthMat = BitmapConverter.ToMat(depth); bgrMat = BitmapConverter.ToMat(color); Binarization(); pictureBox1.Image = color; pictureBox2.Image = depthMat.ToBitmap(); depthMat.Release(); bgrMat.Release(); binMat.Release(); } Mat depthMat, bgrMat, binMat;
  19. Binarization of HSV Image //Lower/Upper threshold of HSV Scalar lower

    = new Scalar(0, 0, 0); Scalar upper = new Scalar(180, 255, 255); public Form1() { InitializeComponent(); } private void Binarization() { depthMat = depthMat.CvtColor(ColorConversionCodes.BGR2GRAY); depthMat = depthMat.Threshold(20, 255, ThresholdTypes.Binary); //Convert BGR into HSV color space. Mat hsv = bgrMat.CvtColor(ColorConversionCodes.BGR2HSV); //Binarization of HSV image binMat = hsv.InRange(lower, upper); //Visualization of the result on pictureBox3 pictureBox3.Image = binMat.ToBitmap(); hsv.Release(); } lower upper
  20. Trial:Changing Threshold of Hue Scalar lower = new OpenCvSharp.Scalar(50, 0,

    0); Scalar upper = new OpenCvSharp.Scalar(80, 255, 255); Green area is extracted
  21. Set Hue’s Threshold as All Range Scalar lower = new

    OpenCvSharp.Scalar(0, 0, 0); Scalar upper = new OpenCvSharp.Scalar(180, 255, 255);
  22. Making H-Threshold Modifiable via UI //Called when value of trackbar

    minH is changed private void minH_Scroll(object sender, EventArgs e) { //lower value of H-threshold is changed lower.Val0 = minH.Value; } Start
  23. Making H-Threshold Modifiable via UI //Called when value of trackbar

    maxH is changed private void maxH_Scroll(object sender, EventArgs e) { //upper value of H-threshold is changed upper.Val0 = maxH.Value; } Start
  24. Making H-Threshold Modifiable via UI Move Color which is between

    lower and upper threshold is shown as white area
  25. Making S-Threshold Modifiable via UI //Called when value of trackbar

    minS is changed private void minS_Scroll(object sender, EventArgs e) { //lower value of S-threshold is changed lower.Val1 = minS.Value; } //Called when value of trackbar maxS is changed private void maxS_Scroll(object sender, EventArgs e) { //upper value of S-threshold is changed upper.Val1 = maxS.Value; }
  26. Run & Confirm Large S is shown as White Change

    minS Large S is shown as Black Change maxS min max min max
  27. Making V-Threshold Modifiable via UI //Called when value of trackbar

    minV is changed private void minV_Scroll(object sender, EventArgs e) { //lower value of V-threshold is changed lower.Val2 = minV.Value; } //Called when value of trackbar maxV is changed private void maxV_Scroll(object sender, EventArgs e) { //upper value of V-threshold is changed upper.Val2 = maxV.Value; }
  28. Run & Confirm Small V is shown as black Change

    minV Small V is shown as white Change maxV min max min max
  29. Trial: Let’s Extract Hand Not only hand but also face

    is extracted by color binarization
  30. Extraction Common Area with AND private void Binarization() { //Conversion

    bgr to gray depthMat = depthMat.CvtColor(ColorConversionCodes.BGR2GRAY); //Binarization with threshold=20 depthMat = depthMat.Threshold(20, 255, ThresholdTypes.Binary); //Conversion bgr to hsv Mat hsv = bgrMat.CvtColor(ColorConversionCodes.BGR2HSV); //Binarization of HSV with lower/upper threshold binMat = hsv.InRange(lower, upper); //Visualization of the result pictureBox3.Image = binMat.ToBitmap(); //Extract common white area between binMat & depthMat Cv2.BitwiseAnd(binMat, depthMat, binMat); //Visualization of the result pictureBox4.Image = binMat.ToBitmap(); hsv.Release(); }
  31. Filling Lacked Area & Denoising depth HSV AND 【Filling Lacked

    Area & Denoising】 (1) Smoothing the image (2) Binarization of the image (1) (2) 欠損 ノイズ
  32. Smoothing & Binarization private void Binarization() { //Binarization of depth

    depthMat = depthMat.CvtColor(ColorConversionCodes.BGR2GRAY); depthMat = depthMat.Threshold(20, 255, ThresholdTypes.Binary); //Conversion of BGR to HSV Mat hsv = bgrMat.CvtColor(ColorConversionCodes.BGR2HSV); //Binarization of HSV binMat = hsv.InRange(lower, upper); //Visualization of the result pictureBox3.Image = binMat.ToBitmap(); Cv2.BitwiseAnd(binMat, depthMat, binMat); //Smoothing with 9*9 neighbor pixels binMat = binMat.Blur(new OpenCvSharp.Size(9, 9)); //Binarization with threshold=60 binMat = binMat.Threshold(60, 256, ThresholdTypes.Binary); pictureBox4.Image = binMat.ToBitmap(); hsv.Release(); }
  33. Preparation of Contour Extraction //Procedure of Contour Extraction will be

    written here private Point[] ContourExtraction() { Point[] contour = null; return contour; } private void timer1_Tick(object sender, EventArgs e) { float maxDepth = 0.75f; Bitmap color = kinect.GetColorImage(); Bitmap depth = kinect.GetDepthImage(maxDepth); depthMat = BitmapConverter.ToMat(depth); bgrMat = BitmapConverter.ToMat(color); Binarization(); //Points consisting contour are returned. Point[] contour = ContourExtraction(); pictureBox1.Image = color; /*****omitted below*****/ }
  34. Contours (Please edit ContorExtraction) Point[] contour = null; Point[][] lines;

    HierarchyIndex[] h; //Find contours from binMat binMat.FindContours(out lines, out h, RetrievalModes.External, ContourApproximationModes.ApproxSimple); double maxArea = -1; for (int i = 0; i < lines.Length; i++) { double area = Cv2.ContourArea(lines[i]); //Contour of maximum area is treated as hand if (area > maxArea) { maxArea = area; contour = lines[i]; } } return contour;
  35. Visualize Contourline private void DrawPolyLine(Point[] points, Scalar color) { /*described

    in next slide!*/ } private void timer1_Tick(object sender, EventArgs e) { /*省略*/ bgrMat = BitmapConverter.ToMat(color); Binarization(); Point[] contour = ContourExtraction(); if (contour != null) { DrawPolyLine( contour, Scalar.Green); } pictureBox1.Image = color; pictureBox1.Image = bgrMat.ToBitmap(); /*****omitted below*****/ } pictureBox1.Image = color;
  36. 輪郭の表示 private void DrawPolyLine(Point[] points, Scalar color) { Point p

    = points.Last(); for (int i = 0; i < points.Length; i++) { Cv2.Line(bgrMat, p, points[i], color, 2); p = points[i]; } } 0 1 2 3 4 points[0] 1 2 3 p p points[1] 2 3 4 0 1 2 p points[4] ・・・ i = 0 i = 1 i = 4
  37. Convex Hull private int[] Polygonization(Point[] contour) { //Approximate contour line

    as convex hull. int[] hullIdx = Cv2.ConvexHullIndices(contour); //Approximate contour line as convex hull. Point[] hullPt = Cv2.ConvexHull(contour); DrawPolyLine(hullPt, Scalar.Blue); return hullIdx; } private void timer1_Tick(object sender, EventArgs e) { /***omitted***/ }
  38. Convex Hull private void timer1_Tick(object sender, EventArgs e) { float

    maxDepth = 0.75f; Bitmap color = kinect.GetColorImage(); Bitmap depth = kinect.GetDepthImage(maxDepth); depthMat = BitmapConverter.ToMat(depth); bgrMat = BitmapConverter.ToMat(color); Binarization(); Point[] contour = ContourExtraction(); if (contour != null) { DrawPolyLine( contour, Scalar.Green); int[] hull = Polygonization(contour); } pictureBox1.Image = bgrMat.ToBitmap(); pictureBox2.Image = depthMat.ToBitmap(); /*****omitted below*****/ }
  39. Visualization of Vertices of Convex Hull private int[] Polygonization(Point[] contour)

    { int[] hullIdx = Cv2.ConvexHullIndices(contour); Point[] hullPt = Cv2.ConvexHull(contour); for(int i=0;i<hullIdx.Length;i++) { int index = hullIdx[i]; bgrMat.Circle(contour[index], 10, Scalar.Red, 2); } DrawPolyLine(bgr, hullPt, Scalar.Blue); return hullIdx; }  Many points are exists at the tip of finger  They should integrated into the one point.
  40. Clustering private int[] Polygonization(Point[] contour) { int[] hullIdx = Cv2.ConvexHullIndices(contour);

    Point[] hullPt = Cv2.ConvexHull(contour); //Terating label numbers of each vertices of hull int[] labels; //clustering based on the role written in Predicate Cv2.Partition(hullPt, out labels, Predicate); for (int i = 0; i < hullIdx.Length; i++) { int index = hullIdx[i]; bgrMat.Circle(contour[index], 10, Scalar.Red, 2); //Visualization of label number bgrMat.PutText(labels[i].ToString(), contour[index], HersheyFonts.HersheyDuplex, 1, Scalar.White); } DrawPolyLine(hullPt, Scalar.Blue); return hullIdx; }
  41. Clustering private int[] Polygonization(Point[] contour) { /*****omitted above*****/ Cv2.Partition(hullPt, out

    labels, Predicate); /*****omitted below*****/ } private bool Predicate(Point t1, Point t2) { //2 points are treated as the same point if //distance is less than 20pixels if (t1.DistanceTo(t2) < 20) { return true; } else { return false; } }
  42. Integration of Points Center Point 【Procedure】 Step1: Calculate min-area rectangle

    Step2: Make center of rectangle Step3: Compare distance to center Step4: Make farthest point as finger tip Min Area Rectangle
  43. Calculation of Center Point private int[] Polygonization(Point[] contour) { int[]

    hullIdx = Cv2.ConvexHullIndices(contour); Point[] hullPt = Cv2.ConvexHull(contour); int[] labels = new int[hullIdx.Length]; Cv2.Partition(hullPt, out labels, Predicate); RotatedRect rect = Cv2.MinAreaRect(hullPt); bgrMat.Circle((int)rect.Center.X, (int)rect.Center.Y, 10, Scalar.Cyan, 3); for (int i=0;i<hullIdx.Length;i++) { int index = hullIdx[i]; bgrMat.Circle(contour[index], 10, Scalar.Red, 2); bgrMat.PutText(labels[i].ToString(), contour[index], HersheyFonts.HersheyDuplex, 1, Scalar.White); } DrawPolyLine(bgr, hullPt, Scalar.Blue); return hullIdx; }
  44. Integration of Points Center Point Min Area Rectangle 【Procedure】 Step1:

    Calculate min-area rectangle Step2: Make center of rectangle Step3: Compare distance to center Step4: Make farthest point as finger tip
  45. Integration of Points private int[] Polygonization(Point[] contour) { /*****omitted above*****/

    RotatedRect rect = Cv2.MinAreaRect(hullPt); bgrMat.Circle((int)rect.Center.X, (int)rect.Center.Y, 10, Scalar.Cyan, 3); hullIdx = HullSimplification(contour, hullIdx, labels, rect.Center); for (int i=0;i<hullIdx.Length;i++) { int index = hullIdx[i]; bgrMat.Circle(contour[index], 10, Scalar.Red, 2); } /*****omitted below*****/ } Paset the contents of copy_and_paste.text here.
  46. Detection of Finger Tip private int FingerDetection(Point[] contour, int[] hull)

    { int num = 0;//Number of Finger return num; } private void timer1_Tick(object sender, EventArgs e) { int fingerNum = 0; /*****omitted*****/ Binarization(); Point[] contour = ContourExtraction(); if (contour != null) { DrawPolyLine( contour, Scalar.Green); int[] hull = Polygonization(contour); fingerNum = FingerDetection(contour, hull); } /*****omitted below*****/ }
  47. Detection of Finger Tip private int FingerDetection(Point[] contour, int[] hull)

    { int num = 0; //Calculation of defect points Vec4i[] defects = Cv2.ConvexityDefects(contour, hull); //Visualization of defect points. foreach (Vec4i v in defects) { bgrMat.Circle(contour[v.Item2], 5, Scalar.White, 2); } return num; } 【Info.】 ConvexityDefets calculates degree of lack versus convex hull Detail of return value is below. Item0: Index of start point Item1: Index of end point Item2: Index of defect point Item3: Distance to defect point Item0 Item2 Item1
  48. Conditions Considered as Finger Tip p1 tip p2 θ l1

    l2 θ<70[deg] && l 1 +l 2 >60[pixel]
  49. Detection of Finger Tip private int FingerDetection(Point[] contour, int[] hull)

    { Vec4i[] defects = Cv2.ConvexityDefects(contour, hull); foreach (Vec4i v in defects) { bgr.Circle(contour[v.Item2], 5, Scalar.White, 2); } //Judge each point is finger tip or not. for (int i = 0; i < defects.Length; i++) { Vec4i d1 = defects[i]; Vec4i d2 = defects[(i + 1) % defects.Length]; //Defect points and tip. Point p1 = contour[d1.Item2]; Point p2 = contour[d2.Item2]; Point tip = contour[d1.Item1]; //calculate vector p1 = p1 - tip; p2 = p2 - tip; } } p1 tip p2
  50. Detection of Finger Tip for (int i = 0; i

    < defects.Length; i++) { Vec4i d1 = defects[i]; Vec4i d2 = defects[(i + 1) % defects.Length]; Point p1 = contour[d1.Item2]; Point p2 = contour[d2.Item2]; Point tip = contour[d1.Item1]; p1 = p1 - tip; p2 = p2 - tip; //p1,p2の長さを計算 double l1 = Math.Sqrt(p1.X * p1.X + p1.Y * p1.Y); double l2 = Math.Sqrt(p2.X * p2.X + p2.Y * p2.Y); //内積の計算と角度の計算 double dot = Point.DotProduct(p1, p2) / (l1 * l2); double angle = Math.Acos(dot) * 180.0 / Math.PI; if (angle < 70 && (l1+l2) > 60) { bgrMat.Circle(tip, 5, Scalar.Orange, 3); num++; } }
  51. Display Number of Finger private void timer1_Tick(object sender, EventArgs e)

    { float maxDepth = 0.75f; Bitmap color = kinect.GetColorImage(); Bitmap depth = kinect.GetDepthImage(maxDepth); depthMat = BitmapConverter.ToMat(depth); bgrMat = BitmapConverter.ToMat(color); Binarization(); Point[] contour = ContourExtraction(); int fingerNum = 0; if (contour != null) { DrawPolyLine( contour, Scalar.Green); int[] hull = Polygonization(contour); fingerNum = FingerDetection(contour, hull); } label7.Text = fingerNum.ToString(); pictureBox1.Image = bgrMat.ToBitmap(); pictureBox2.Image = depthMat.ToBitmap(); /*省略*/ }