Tutorial: Creation of AR Coloring with OpenCV and ARCore

Description how to create AR Coloring app by using OpenCV, ARCore and Unity.

1. with OpenCV and ARCore Tutorial: Creation of AR Coloring
2. Download and Installation ①Sample Data for the Tutorial http://arfukuoka.lolipop.jp/nurie/sample.zip ②ARCoreSDK(v1.8.0) https://github.com/google-ar/arcore-unity-sdk/releases/tag/v1.8.0 ③Unity2017.4.15f1 or later https://unity3d.com/jp/unity/qa/lts-releases?version=2017.4 ④Android SDK https://developer.android.com/studio ※Please finish setting up Android build on Unity before hand.
3. You can see also… Source Code https://github.com/TakashiYoshinaga/AR-Coloring
4. Goal of The Tutorial https://youtu.be/E5Jy4iunpyQ https://youtu.be/MviBGZtiv5A
5. ARCore New marker-less AR platform which can available for Android devices. 【Features】 (1) Motion Tracking based on SLAM (2) Environmental Understanding (3) Light Estimation (4) Augmented Image (5) Cloud Anchor (6) Augmented Faces
6. OpenCV Plus  Unity asset of image processing based on OpenCV3.  OpenCVSharp was adapted to Unity environment.  Available on Windows/Mac/Android/iOS.
7. Procedure of Image Processing Clipping ROI Binarization Square Detection Transformation input output Attachment
8. Getting Started with Unity
9. Creation of a Unity Project New
10. Creation of a Unity Project Create Project Name of the Project
11. Adding a 3D Object Right Click 3DObject → Cube
12. Applying a Texture to Material Change the appearance by applying texture image
13. Creation of a Material File ①Assets ②Right Click
14. Creation of a Material File ①Create ②Material
15. Creation of a Material File ①NewMaterial ②Click here
16. Creation of a Material File ①Legacy Shaders ②Diffuse
17. Creation of a Material File Texture will be assigned here later
18. Importing a Texture Image ①Assets ②ImportNewAsset...
19. Importing a Texture Image ①Sample Folder ②Select ilovear & opencv ③Import
20. Applying a Texture Image NewMaterial
21. Applying a Texture Image ①See ilovear ②Drag & Drop
22. Applying a Texture Image Image was assigned
23. Applying a Material to 3D Object ①Click Cube ②Open Materials ※Click ▼
24. Applying a Material to 3D Object ①See NewMaterial ②Drag & Drop it into Element0
25. Applying a Material to 3D Object Appearance was changed
26. Changing a Material to 3D Object ①Cube ②Open the Detail Information of NewMaterial
27. Changing a Material to 3D Object ①See opencv ②Drag & Drop it Texture Area
28. Changing a Material to 3D Object Appearance was changed
29. Check Point & Next Step 【Check Point】 Appearance can be changed just by replacing texture file if a material to use texture was applied to3D model. 【Next Step】  Create texture from inner area of square frame.  Writing C# script to replace texture automaticaly.
30. Save the Scene ①File ②Save Scene as...
31. Save the Scene ①Write the name of scene ②Save
32. Installation of ARCore SDK ①Assets ②Import Package ③Open SDK from Custom Package arcore-unity-sdk-v1.8.0.unitypackage
33. Installation of ARCore SDK Import
34. Installation of ARCore SDK GoogleARCore was added
35. Setting Up Camera for AR Delete Main Camera
36. Setting Up Camera for AR Assets→GoogleARCore→Prefabs
37. Setting Up Camera for AR ①ARCore Device ②Drag & Drop into Hierarchy
38. Setting Up Camera for AR ①ARCore Device ②Double Click DefaultSessionConfig
39. Setting Up Camera for AR Switch Camera Focus Mode to Auto
40. Modification of Position & Scale of a CG Click Cube
41. Modification of Position & Scale of a CG Position: 0, -0.2, 0.5 Scale: All 0.1
42. 視点 ViewPoint 0.5 Y X -0.2 Z
43. Saving the Scene Ctrl/Command + S
44. Build Setting ①File ②Build Settings...
45. Build Setting ①Android ②Switch Platform
46. Build Setting ①Internal ②Player Settings
47. Build Setting ①Resolution and Presentation ②Select Portrait as Default Orientation
48. Build Setting ①Other Settings ②Turn Off Multithreaded Rendering
49. Build Setting ①Set original PackageName 例)com.fukuoka.test ②Set Minimum API Level as Andrid 7.0
50. Build Setting (for Unity2018) Set Allow unsafe Code Checked
51. Build Setting ①XR Settings ②Setting ARCore Supported Checked
52. Saving the Scene Ctrl/Command+S
53. Installation App Into Smartphone ①File ②Build & Run
54. Installation App Into Smartphone ①Set a name of installer(.apk) ②Save
55. Run
56. From Next Slide.. Preparation for Getting Started with Image Processing
57. Installation of OpenCV Plus Unity ①Window ②Asset Store
58. Installation of OpenCV Plus Unity Search “OpenCV Plus”
59. Installation of OpenCV Plus Unity Open OpenCV Plus
60. Installation of OpenCV Plus Unity Download
61. Installation of OpenCV Plus Unity Import
62. Installation of OpenCV Plus Unity Import
63. Installation of OpenCV Plus Unity OpenCV + Unity will appear
64. Setting Up the View of Unity Editor Game This operation means making your modification of UI layout of application comfortable in Unity Editor.
65. Setting Up the View of Unity Editor ①Click Free Aspect ②Click＋ This operation means making your modification of UI layout of application comfortable in Unity Editor.
66. Setting Up the View of Unity Editor ①Put Name ②800 × 1280
67. Importing Sample UI ②Import Package ①Assets ③Custom Package...
68. Importing Sample UI ②Open ①NurieUI.unitypackage
69. Importing Sample UI Import
70. Importing Sample UI Scene
71. Using Sample UI ①Canvas ②Drag & Drop into Hierarchy
72. Using Sample UI UI was added to the Scene UI might not be visible the view point which you using know. But it’s not problem! Please see next page.
73. Using Sample UI ①Double Click Canvas ②You can see whole UI
74. Using Sample UI ①Click x or z ②Modify view point to make UI to be facing to you
75. Role of UI Results of Image Processing Showing ROI Switch visibility of UI Starting capture
76. Preparation of Writing C# Script Write click on the blank of Hierarchy
77. Preparation of Writing C# Script Create Empty
78. Preparation of Writing C# Script ①Game Object ②Add Component
79. Preparation of Writing C# Script ①New Script ②Name of Script (ex. ColoringScript) ③Create and Add
80. Preparation of Writing C# Script ①Game Object ②ColoringScript was added
81. Preparation of Writing C# Script Double Click
82. Importing OpenCV and UnityEngine.UI using UnityEngine; using UnityEngine.UI; using OpenCvSharp; using OpenCvSharp.Demo; public class ColoringScript : MonoBehaviour { // Start関数は初期化のために一度だけ実行される void Start () { cg = GameObject.Find ("Robot Kyle"); } // Update関数は毎フレーム実行される void Update () { } }
83. Declaration of Variables public MeshRenderer target; //Rendering Setting of Cube public GameObject canvas; //Canvas which involves UI public RawImage viewL, viewR; //Result viewer UnityEngine.Rect capRect;//Region of screen shot Texture2D capTexture; //Texture of screenshot image Texture2D colTexture; //Result of image processing(color) Texture2D binTexture; //Result of image processing(gray) void Start () { } void Update () { } canvas viewL viewR colTexture binTexture
84. Linking Objects and Variables ①Click GameObject ②See Target
85. Linking Objects and Variables See Cube Drag & Drop into Target of ColoringScript
86. Linking Objects and Variables See Canvas Drag & Drop into Canvas of ColoringScript
87. Linking Objects and Variables See RawImage Drag & Drop into ViewL of ColoringScript
88. Linking Objects and Variables View RawImage(1) Drag & Drop into ViewR of ColoringScript
89. Procedure of Image Processing Clipping ROI Binarization Square Detection Transformation input output Attachment
90. Procedure of Image Processing Clipping Binarization Square Detection Transformation output Attachment input
91. Preparation of Screen Capture public MeshRenderer target; public GameObject canvas; public RawImage viewL, viewR; UnityEngine.Rect capRect; Texture2D capTexture; Texture2D colTexture; Texture2D binTexture; void Start () { int w = Screen.width; int h = Screen.height; //Definition of capture region as (0,0) to (w,h) capRect = new UnityEngine.Rect(0, 0, w, h); //Creating texture image of the size of capRect capTexture = new Texture2D(w, h, TextureFormat.RGB24, false); } width height (0,0)
92. Making Function of Image Processing void Start () { /*Code was omitted in the slide.*/ } IEnumerator ImageProcessing() { canvas.SetActive(false);//Making UIs invisible yield return new WaitForEndOfFrame(); capTexture.ReadPixels(capRect, 0, 0);//Starting capturing capTexture.Apply();//Apply captured image. /*Setting texture on the coloring target object (cube)*/ target.material.mainTexture = capTexture; canvas.SetActive(true);//Making UIs visible. } public void StartCV() { StartCoroutine(ImageProcessing());//Calling coroutine. } Write!
93. Calling StartCV() From UI ①Click Button ②Find Button ③Click + laying on bottom of OnClick()
94. Calling StartCV() From UI Find GameObject Drag & Drop into the area written None, of OnClick()
95. Calling StartCV() From UI ①Click No Function ②ColoringScript
96. Calling StartCV() From UI StartCV()
97. Ctrl/Command + S
98. Build & Run!
99. Run Capture
100. Procedure of Image Processing Clipping Binarization Square Detection Transformation output Attachment input
101. Procedure of Image Processing Binarization Square Detection Transformation output Attachment Clipping ROI input
102. Clipping Image of Inside the ROI void Start () { int w = Screen.width; int h = Screen.height; //Setting up position/size of ROI int sx = (int)(w * 0.2); //Start of X int sy = (int)(h * 0.3); //Start of Y w = (int)(w * 0.6); //Width of ROI h = (int)(h * 0.4); //Height of ROi //キャプチャする領域を保持したRectを作成 capRect = new UnityEngine.Rect(0, 0, w, h); capTexture = new Texture2D(w, h, TextureFormat.RGB24, false); } capRect = new UnityEngine.Rect(sx, sy, w, h); Replace!
103. Run Capture You can find coloring sheet (pdf) in Markers folder.
104. Refactoring (1/2) IEnumerator ImageProcessing() { canvas.SetActive(false); yield return new WaitForEndOfFrame(); //描画終了を待つ capTexture.ReadPixels(capRect, 0, 0);//キャプチャ開始 capTexture.Apply();//各画素の色をテクスチャに反映 /*下記の一行でオブジェクトにテクスチャを張り付け*/ target.material.mainTexture = capTexture; canvas.SetActive(true); } void CreateImage() { /*Cut & Paste Code of Image Creation*/ } void ShowImage() { /*Cut & Paste Code of Image Visualization*/ } Image Creation Visualization
105. Refactoring (2/2) IEnumerator ImageProcessing() { canvas.SetActive(false); yield return new WaitForEndOfFrame(); CreateImage(); //Image Creation ShowImage(); //Image Visualization canvas.SetActive(true); } void CreateImage() { capTexture.ReadPixels(capRect, 0, 0); capTexture.Apply(); } void ShowImage() { target.material.mainTexture = capTexture; }
106. Procedure of Image Processing Binarization Square Detection Transformation output Attachment Clipping ROI input
107. Clipping ROI input Procedure of Image Processing Square Detection Transformation output Attachment Binarization
108. From now on, we use OpenCV!
109. Binarization of Gray Scale Image 0 255 0 255 Binarization means splitting grayscale(0~255) into 0 or 255 by threshold. It’s very important technique to define pixels which should be processed.
110. Preparation of Using Image with OpenCV Texture2D colTexture; Texture2D binTexture; //Mat:Format of image for OpenCV //bgr is for color image、bin is for binarized image Mat bgr, bin; void Start () { int w = Screen.width; int h = Screen.height; int sx = (int)(w * 0.2); int sy = (int)(h * 0.3); w = (int)(w * 0.6); h = (int)(h * 0.4); capRect = new UnityEngine.Rect(sx, sy, w, h); capTexture = new Texture2D(w, h, TextureFormat.RGB24, false); }
111. Binarization void CreateImage() { capTexture.ReadPixels(capRect, 0, 0); capTexture.Apply(); //Conversion Texure2D to Mat bgr = OpenCvSharp.Unity.TextureToMat(capTexture); //Conversion Color Image to Gray Scale Image bin = bgr.CvtColor(ColorConversionCodes.BGR2GRAY); //Binarization of image with Otsu’s method. bin = bin.Threshold(100, 255, ThresholdTypes.Otsu); Cv2.BitwiseNot(bin, bin); } Color Image Gray Scale Binary Inverse
112. Visualization of Results void ShowImage() { //Releasing memories of textures. if (colTexture != null) { DestroyImmediate(colTexture); } if (binTexture != null) { DestroyImmediate(binTexture); } //Conversion of Mat to Texture2D colTexture = OpenCvSharp.Unity.MatToTexture(bgr); binTexture = OpenCvSharp.Unity.MatToTexture(bin); //Attaching texture to RawImage for visualization. viewL.texture = colTexture; viewR.texture = binTexture; //スクショ画像をモデルに適用 target.material.mainTexture = colTexture; }
113. Releasing Memories Allocated for Mat IEnumerator ImageProcessing() { canvas.SetActive(false); yield return new WaitForEndOfFrame(); CreateImage(); ShowImage(); //Releasing Memories allocated for two Mats bgr.Release(); bin.Release(); canvas.SetActive(true); }
114. Run ①Put marker into ROI ②Capture ③Clipped Image and Binarized Image
115. Clipping ROI input Procedure of Image Processing Square Detection Transformation output Attachment Binarization
116. Binarization Clipping ROI input Procedure of Image Processing Transformation output Attachment Square Detection
117. Preparation of Square Frame Detection IEnumerator ImageProcessing() { canvas.SetActive(false); yield return new WaitForEndOfFrame(); CreateImage(); Point[] corners; //4 corners of square will be memorized FindRect(out corners); //Square Frame Detection ShowImage(); //画像の表示 bgr.Release(); bin.Release(); canvas.SetActive(true); } void FindRect(out Point[] corners) { /*Code will be described from next page.*/ }
118. Contour Detection //Initialization of corners corners = null; //contour points and hierarchy Point[][] contours; HierarchyIndex[] h; //Contour detection bin.FindContours(out contours, out h, RetrievalModes.External, ContourApproximationModes.ApproxSimple); //Finding the contour of which has the most wide area. double maxArea = 0; for(int i = 0; i < contours.Length; i++) { double area = Cv2.ContourArea(contours[i]); if (area > maxArea) { maxArea = area; corners = contours[i]; } }
119. Visualization of the Result void FindRect(out Point[] corners) { /*Code is omitted in this slide.*/ double maxArea = 0; for (int i = 0; i < contours.Length; i++) { double area = Cv2.ContourArea(contours[i]); if (area > maxArea) { maxArea = area; corners = contours[i]; } } //Write a contour line of max area in bgr. if (corners != null) { bgr.DrawContours( new Point[][] { corners }, 0, Scalar.Red, 5); } }
120. Run
121. Polygon Approximation void FindRect(out Point[] corners) { /*Code is omitted in this slide*/ double maxArea = 0; for(int i = 0; i < contours.Length; i++) { //Calculate the length of contour line. double length = Cv2.ArcLength(contours[i], true); //Polygon Approximation. Point[] tmp = Cv2.ApproxPolyDP( contours[i], length * 0.01f, true); double area = Cv2.ContourArea(contours[i]); //If number of corner is 4. if (area > maxArea) { maxArea = area; corners = contours[i]; } } /*次のページに続く*/ if (tmp.Length == 4 && area > maxArea) corners = tmp;
122. Visualization of Corners void FindRect(out Point[] corners) { /*Code is omitted in this slide.*/ if (corners != null) { bgr.DrawContours( new Point[][] { corners }, 0, Scalar.Red, 5); //Draw circle on the position of corner point. for(int i = 0; i < corners.Length; i++) { bgr.Circle(corners[i], 20, Scalar.Blue, 5); } } }
123. Run 四角形以外は 認識されない
124. Before seeing next step… void FindRect(out Point[] corners) { /*Code is omitted in this slide.*/ //Comment out the visualization code. /*if (corners != null) { bgr.DrawContours( new Point[][] { corners }, 0, Scalar.Red, 5); //各頂点の位置に円を描画 for(int i = 0; i < corners.Length; i++) { bgr.Circle(corners[i], 20, Scalar.Blue, 5); } }*/ }
125. Binarization Clipping ROI input Procedure of Image Processing Transformation output Attachment Square Detection
126. Square Detection Binarization Clipping ROI input Procedure of Image Processing output Attachment Transformation
127. Perspective Transformation (0, 0) (255, 0) (0, 255) (255 255) [0] [1] [2] [3]  Deform distorted square to front view based on the result of calculation of perspective transformation matrix.
128. Perspective Transformation IEnumerator ImageProcessing() { canvas.SetActive(false); yield return new WaitForEndOfFrame(); CreateImage(); Point[] corners; FindRect(out corners); TransformImage(corners); //Deform distorted square. ShowImage(); //画像の表示 bgr.Release(); bin.Release(); canvas.SetActive(true); } void TransformImage(Point[] corners) { /*Code will be described from the next page.*/ }
129. Perspective Transformation void TransformImage(Point[] corners) { //Do nothing if square wasn’t found. if (corners == null) return; //Input detect corners. Point2f[] input = { corners[0], corners[1], corners[2], corners[3] }; //Define corners of square image. Point2f[] square = { new Point2f(0, 0), new Point2f(0, 255), new Point2f(255, 255), new Point2f(255, 0) }; //Calculation of transformation matrix. Mat transform = Cv2.GetPerspectiveTransform(input, square); //Deform image as front view square. Cv2.WarpPerspective(bgr,bgr,transform, new Size(256, 256)); } (0, 0) (255, 0) (0, 255) (255 255) [0] [1] [2] [3]
130. Run Result of deformation is different by the rotation of square frame in the ROI. Rotated
131.  Local position of corner at [0] depends on rotation of square.  Sorting is necessary to obtain image of front standing view. [0] [1] [2] [3] [0] [1] [2] [3] Succeeded Failed
132. Sorting Corner Points void TransformImage(Point[] corners) { if (corners == null) return; //Sorting SortCorners(corners); Point2f[] input = { corners[0], corners[1], corners[2], corners[3] }; Point2f[] square = { new Point2f(0, 0), new Point2f(0, 255), new Point2f(255, 255), new Point2f(255, 0) }; Mat transform = Cv2.GetPerspectiveTransform(input, square); Cv2.WarpPerspective(bgr,bgr,transform, new Size(256, 256)); } void SortCorners(Point[] corners) { /*Code of sorting is described in the next page.*/ }
133. Sorting Corner Points void SortCorners(Point[] corners) { System.Array.Sort(corners, (a, b) => a.X.CompareTo(b.X)); if (corners[0].Y > corners[1].Y) { corners.Swap(0, 1); } if (corners[3].Y > corners[2].Y) { corners.Swap(2, 3); } } [0] [1] [2] [3] [2] [0] [1] [3] [3] [0] [1] [2] Sort by X axis Sort [2][3] by Y axis
134. Clip a Image Inside the Square Frame void TransformImage(Point[] corners) { if (corners == null) return; SortCorners(corners); Point2f[] input = { corners[0], corners[1], corners[2], corners[3] }; Point2f[] square = { new Point2f(0, 0), new Point2f(0, 255), new Point2f(255, 255), new Point2f(255, 0) }; Mat transform = Cv2.GetPerspectiveTransform(input, square); Cv2.WarpPerspective(bgr,bgr,transform, new Size(256, 256)); int s = (int)(256*0.05);//Line width of frame is 5% of square int w = (int)(256*0.9);//Width of clipping area is 90% of square OpenCvSharp.Rect innerRect = new OpenCvSharp.Rect(s, s, w, w); bgr = bgr[innerRect]; }
135. Square Detection Binarization Clipping ROI input Procedure of Image Processing output Attachment Transformation
136. Transformation Square Detection Binarization Clipping ROI input Procedure of Image Processing output Attachment
137. Applying Texture to 3D Object void ShowImage() { //すでにcolTextureが存在するならいったん削除 if (colTexture != null) { DestroyImmediate(colTexture); } if (binTexture != null) { DestroyImmediate(binTexture); } //Matをテクスチャに変換 colTexture = OpenCvSharp.Unity.MatToTexture(bgr); binTexture = OpenCvSharp.Unity.MatToTexture(gray); //RawImageに切り抜き画像を表示 viewL.texture = colTexture; viewR.texture = binTexture; //Applying texture to target 3D object target.material.mainTexture = capTexture; //Show Canvas gain. canvas.SetActive(true); } target.material.mainTexture = colTexture; Replace
138. Run
139. Switching Visiblity of ROI ①Toggle ②Click + laying on the bottom of OnValueChanged
140. Switching Visiblity of ROI ①See Panel ②Drag&Drop it int None of OnValue Changed
141. Switching Visiblity of ROI Click No Function
142. Switching Visiblity of ROI ①GameObject ②Click SetActive of Dynamic bool
143. Run Turn off
144. Repalace 3D Model
145. Poly Download 3D model from Google Poly https://poly.google.com/
146. Getting 3D Model Input search word here Search「Duck」
147. Getting 3D Model Rubber Duck
148. Getting 3D Model Download → OBJ File
149. Getting 3D Model Unzip archive.zip
150. Importing 3D Model ①Assets ②Drag & Drop archive folder into Assets
151. Replacing 3D Model Delete Cube
152. Replacing 3D Model Assets → archive
153. Replacing 3D Model ①RubberDuck ②Drag&Drop into Hierarchy
154. Modification of Position/Size of 3D Model ①RubberDuck ②Position 0 -0.25 0.5 Rotation 0 150 0 Scale 0.03 0.03 0.03
155. Replacing 3D Model RubberDuck →RubberDuck_mesh
156. Replacing 3D Model GameObject
157. Replacing 3D Model Drag&Drop RubberDuck_mesh into Target
158. Complete You can find caroling sheet in the Markers folder.