Tutorial: Creation of StampAR with OpenCV and ARCore

This slide deck explains how to develop Stamp-AR by using OpenCV and ARCore on Unity.

Transcript

1. Tutorial: Creating Stamp-AR with OpenCV
2. Download and Installation ①Sample Data for the Tutorial http://arfukuoka.lolipop.jp/stampar/Sample.zip ②ARCoreSDK(v1.8.0) https://github.com/google-ar/arcore-unity-sdk/releases/tag/v1.8.0 ③Unity2017.4.15f1 or later https://unity3d.com/jp/unity/qa/lts-releases?version=2017.4 ④Android SDK https://developer.android.com/studio ※Please finish setting up Android build on Unity before hand.
3. Goal of The Tutorial https://youtu.be/a8o1ieL01_w
4. ARCore New marker-less AR platform which can available for Android devices. 【Features】 (1) Motion Tracking based on SLAM (2) Environmental Understanding (3) Light Estimation (4) Augmented Image (5) Cloud Anchor (6) Augmented Faces
5. OpenCV Plus Unity  Unity asset of image processing based on OpenCV3.  OpenCVSharp was adapted to Unity environment.  Available on Windows/Mac/Android/iOS.
6. Procedure Binarization capture Add Alpha Channel Coloring 2D→3D GOAL!
7. Getting Started with Unity
8. Creation of a Unity Project New
9. Creation of a Unity Project Create Project Name of a Project Save Directory
10. Addin a 3D Object ①Right Click ②3D Object ③Quad
11. How to Play Click Play Button Click Play Button to Stop View of the Camera
12. Modification of View Point of Scene [←] [→] Move to Right/Left [↑][↓] Zoom In/Out [Alt]+Drag Rotation +Drag Move to ↑↓←→
13. Modification of Game Object Click to Select Translation Rotation Scaling
14. Numerical Modification of Game Object  Detailed information is shown in the Inspector Tab  For instance, you can translate object by changing Position ②Change Position parameter to 0 0 0 ① Quad
15. Applying a Texture to Material Change the appearance by applying texture image to 3D model
16. Creation of a Material File ①Assets ②空白を右クリック
17. Creation of a Material File ①Create ②Material
18. Creation of a Material File ①Click Quad ②Open Materials ※Click▼button
19. Applying a Texture Image ①Find NewMaterial ※Do NOT click! ②Drag & Drop into Element0 of Materials
20. Applying a Texture Image ①Click Quad ②Open NewMaterial ※▼をクリック
21. Applying a Texture Image Open Shader List
22. Applying a Texture Image ①Unlit ②Transparent
23. Applying a Texture Image Texture image will be assigned from here
24. Importing Texture Image ①Assets ②ImportNewAsset...
25. Importing Texture Image ①Sample Folder ②Select logo.png ③Import
26. Assigning Texture to Material ①Click Quad
27. Assigning Texture to Material ①Find logo file ②Drag&Drop into texture area
28. Assigning Texture to Material
29. Check Point & Idea for StampAR 【Check Point】 Appearance can be set just by assigning texture file if a material to use texture was applied to3D model. 【Idea】  Make a image which has alpha channel.  Change color of opaque pixels by using script.  Generate plane which is attached this image.
30. Save the Scene ①File ②Save Scene as...
31. Save the Secne ①Write the name of this scene ②Save Please save the scene frequently with ctrl+s after this procedure
32. Installation of ARCore SDK ①Assets ②Import Package ③Open SDK from Custom Package arcore-unity-sdk-v1.8.0.unitypackage
33. Installation of ARCore SDK Import
34. Installation of ARCore SDK GoogleARCore will appear
35. Setting Up Camera for AR Delete Main Camera
36. Setting Up Camera for AR Assets→GoogleARCore→Prefabs
37. Setting Up Camera for AR ①ARCore Device ②Drag & Drop into Hierarchy
38. Setting Up Camera for AR ①ARCore Device ②Double Click DefaultSessionConfig
39. Setting Up Camera for AR Switch Camera Focus Mode to Auto
40. Modification of Position & Scale of a CG Click Quad
41. Modification of Position & Scale of a CG Position: 0, 0, 0.5 Scale: All 0.1
42. Modification of Position & Scale of a CG Double Click Quad Since Quad object is very smaller than default size, make the view point close to the object.
43. Saving the Scene Ctrl/Command + S
44. Build Setting ①File ②Build Settings...
45. Build Setting ①Android ②Switch Platform
46. Build Setting ①Internal ②Player Settings
47. Build Setting ①Resolution and Presentation ②Select Portrait as Default Orientation
48. Build Setting ①Other Settings ②Turn Off Multithreaded Rendering
49. Build Setting ①Set original PackageName 例)com.fukuoka.test ②Set Minimum API Level as Andrid 7.0
50. Build Setting (for Unity2018) Set Allow unsafe Code Checked
51. Build Setting ①XR Settings ②Setting ARCore Supported Checked
52. Saving the Scene Ctrl/Command+S
53. Installation App Into Smartphone ①File ②Build & Run
54. Installation App Into Smartphone ②Save ①Set a name of installer(.apk)
55. Run 3D object will be fixed the real space.
56. From Next Slide… Preparation for Getting Started with Image Processing
57. Installation of OpenCV Plus Unity ①Window ②Asset Store
58. Installation of OpenCV Plus Unity Search “OpenCV Plus”
59. Installation of OpenCV Plus Unity Click OpenCV Plus Unity
60. Installation of OpenCV Plus Unity Download
61. Installation of OpenCV Plus Unity Import
62. Installation of OpenCV Plus Unity Import
63. Installation of OpenCV Plus Unity OpenCV + Unity will appear
64. Setting Up the View of Unity Editor Game This operation means making your modification of UI layout of application comfortable in Unity Editor.
65. Setting Up the View of Unity Editor ①Click Free Aspect ②Click＋ This operation means making your modification of UI layout of application comfortable in Unity Editor.
66. Setting Up the View of Unity Editor ①Put Name ②800 × 1280 It’s not necessary to match actual resolution of your smartphone in this tutorial.
67. Importing Sample UI ②Import Package ①Assets ③Custom Package...
68. Importing Sample UI ②Open ①StampUI.unitypackage
69. Importing Sample UI Import
70. Importing Sample UI Scene
71. Importing Sample UI ①Assets ②Canvas will be added
72. Using Sample UI ①Canvas ②Drag&Drop into Hierarchy
73. Using Sample UI UI will appear in the Scene UI might not be visible the view point which you using know. But it’s not problem! Please see next page.
74. Using Sample UI ①Double Click Canvas ②You can see whole UI
75. Using Sample UI ②Modify view point to make UI to be facing to you ①Click x or z
76. Role of UI Start capture Change color Put stamp into the space Vewer of captured or processed image
77. Applying Alpha Texture to Viewer ①Open Canvas ②Click Assets
78. Applying Alpha Texture to Viewer ①RawImage ②Find Material
79. Applying Alpha Texture to Viewer ①Find NewMaterial ※Do NOT click ②Drag&Drop into Material
80. Applying Alpha Texture to Viewer Viewer became to be able to show alpha channel image
81. Preparation of Writing C# Script Write click on the blank of Hierarchy
82. Preparation of Writing C# Script Create Empty
83. Preparation of Writing C# Script ①Game Object ②Add Component
84. Preparation of Writing C# Script ①New Script ②Name of Script (ex. StampScript) ③Create and Add
85. Preparation of Writing C# Script ①Game Object ②ColoringScript will be added
86. Preparation of Writing C# Script Double Click
87. Importing OpenCV and UnityEngine.UI using UnityEngine; using UnityEngine.UI; using OpenCvSharp; using OpenCvSharp.Demo; public class StampScript : MonoBehaviour { // Start関数は初期化のために一度だけ実行される void Start () { cg = GameObject.Find ("Robot Kyle"); } // Update関数は毎フレーム実行される void Update () { } }
88. canvas Declaration of Variables //Canvas which involves UI parts public GameObject canvas; //Viewer of image. public RawImage preview; //Region of screen shot. UnityEngine.Rect capRect; //Texture of screen shot image. Texture2D capTexture; void Start () { } void Update () { } preview
89. Linking Objects and Variables ①Click GameObject ②Canvas & Preview will be added
90. Linking Objects and Variables Find Canvas Drag & Drop into Canvas of StampScript
91. Linking Objects and Variables See RawImage Drag& Drop into Previewof StampScript
92. Procedure Binarization capture Add Alpha Channel Coloring 2D→3D GOAL!
93. Procedure Binarization Add Alpha Channel Coloring 2D→3D GOAL! capture
94. Preaparation of Screen Capture //Canvas which involves UI parts public GameObject canvas; //Viewer of image. public RawImage preview; //Region of screen shot. UnityEngine.Rect capRect; //Texture of screen shot image. Texture2D capTexture; void Start () { int w = Screen.width; int h = Screen.height; //Definition of capture region as (0,0) to (w,h). capRect = new UnityEngine.Rect(0, 0, w, h); //Creating texture image of the size of capRect. capTexture = new Texture2D(w, h, TextureFormat.RGBA32, false); //Applying capTexture as texture of preview area. preview.material.mainTexture = capTexture; } width height (0,0)
95. Making Function of Image Processing void Start () { /*Code was omitted in the slide.*/ } IEnumerator ImageProcessing() { canvas.SetActive(false);//Making UIs invisible yield return new WaitForEndOfFrame(); capTexture.ReadPixels(capRect, 0, 0);//Starting capture capTexture.Apply();//Apply captured image. canvas.SetActive(true);//Making UIs visible } public void StartCV() { StartCoroutine(ImageProcessing());//Calling coroutine } Write!
96. Calling StartCV() From UI ①Click CaptureBtn ②Find Button ③Click + laying on bottom of OnClick()
97. Calling StartCV() From UI Find GameObject Drag & Drop into the area written None, of OnClick()
98. Calling StartCV() From UI ①Click No Function ②StampScript
99. Calling StartCV() From UI StartCV()
100. Ctrl/Command + S
101. Build & Run!
102. Run Captured image appears Capture
103. Another Way to Test Play
104. Another Way to Test Click Play again to STOP
105. Refactoring(1/2) IEnumerator ImageProcessing() { canvas.SetActive(false); yield return new WaitForEndOfFrame(); capTexture.ReadPixels(capRect, 0, 0); capTexture.Apply(); canvas.SetActive(true); } void CreateImages() { /* Cut & Paste Code of Image Creation */ } Image Creation Make CreateImages function since amount of source code for image creation will be increased later.
106. Refactoring(2/2) IEnumerator ImageProcessing() { canvas.SetActive(false); yield return new WaitForEndOfFrame(); CreateImages(); //画像の生成 canvas.SetActive(true); } void CreateImages() { capTexture.ReadPixels(capRect, 0, 0); capTexture.Apply(); } Do not forget to call CreateImages() from ImageProcessing()
107. Procedure Binarization Add Alpha Channel Coloring 2D→3D GOAL! capture
108. Procedure Add Alpha Channel Coloring 2D→3D GOAL! capture Binarization
109. From now on, we use OpenCV!
110. Binarization of Gray Scale Image 0 255 0 255 Binarization means splitting grayscale(0~255) into 0 or 255 by threshold. It’s very important technique to define pixels which should be processed.
111. Preparation of Using Image with OpenCV UnityEngine.Rect capRect; //Texture to store captured image. Texture2D capTexture; //Mat:Format of image for OpenCV //bgraMat is for color image with alpha channel. //binMat is for binarized image. Mat bgraMat, binMat; void Start () { int w = Screen.width; int h = Screen.height; //Definition of capture region as (0,0) to (w,h). capRect = new UnityEngine.Rect(0, 0, w, h); //Creating texture image of the size of capRect. capTexture = new Texture2D(w, h, TextureFormat.RGBA32, false); //Applying capTexture as texture of preview area. preview.material.mainTexture = capTexture; }
112. Binarization void CreateImage() { capTexture.ReadPixels(capRect, 0, 0); capTexture.Apply(); //Conversion Texure2D to Mat bgraMat = OpenCvSharp.Unity.TextureToMat(capTexture); //Conversion Color Image to Gray Scale Image binMat = bgraMat.CvtColor(ColorConversionCodes.BGRA2GRAY); //Binarization of image with Otsu’s method. binMat = binMat.Threshold(100, 255, ThresholdTypes.Otsu); //Conversion Gray Scale to BGRA to change its color later. bgraMat = binMat.CvtColor(ColorConversionCodes.GRAY2BGRA); } bgraMat binMat(GrayScale) binMat (Binarized) bgraMat (B=G=R)
113. Visualization of a Result of Binarization IEnumerator ImageProcessing() { canvas.SetActive(false); yield return new WaitForEndOfFrame(); CreateImages();//Creating Image SetColor(capTexture);//Setting color to capTexture canvas.SetActive(true); } //Setting color information of bgraMat to texture. void SetColor(Texture2D texture) { OpenCvSharp.Unity.MatToTexture(bgraMat, texture); }
114. Run ①Capture ②Binalized Image
115. Releasing Memories Allocated for Mat IEnumerator ImageProcessing() { canvas.SetActive(false); //Releasing Memories allocated for two Mats if (bgraMat != null) { bgraMat.Release(); } if (binMat != null) { binMat.Release(); } yield return new WaitForEndOfFrame(); CreateImages(); SetColor(capTexture); canvas.SetActive(true); }
116. Procedure Add Alpha Channel Coloring 2D→3D GOAL! capture Binarization
117. capture Binarization Procedure Coloring 2D→3D GOAL! Add Alpha Channel
118. Make White Pixels to Transparent (1/2) void SetColor(Texture2D texture) { //Do nothing if Mats are null if (bgraMat == null || binMat == null) { return; } unsafe { //Get pointer of pixel array of 2 Mats. byte* bgraPtr = bgraMat.DataPointer; byte* binPtr = binMat.DataPointer; //Calculate number of pixels of the image. int pixelCount = binMat.Width * binMat.Height; //Make white pixels to transparent for (int i = 0; i < pixelCount; i++) { } } OpenCvSharp.Unity.MatToTexture(bgraMat, texture); } 後ほど処理を記述
119. Make White Pixels to Transparent (2/2) byte* bgraPtr = bgraMat.DataPointer; byte* binPtr = binMat.DataPointer; int pixelCount = binMat.Width * binMat.Height; for (int i = 0; i < pixelCount; i++) { //Address of blue color of i-th pixel int bgraPos = i * 4; //If i-th pixel of binPtr is 255(white). if (binPtr[i] == 255) { bgraPtr[bgraPos + 3] = 0; } //If i-th pixel of binPtr is 0(black). else { bgraPtr[bgraPos + 3] = 255; } }
120. Data Structure of Pixel Array  Pixel data of each types, gray or bgra, are stored in 1-dimensional array of byte.  binPtr(gray) :  bgraPtr(color+alpha): binPtr bgraPtr Length of pixel array is n (=width*height) Length of pixel array is n*4 [0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] ・・・ ・・・ 0th pixel 1st 2nd
121. Run ①Capture ②Binarized image with alpha channel
122. capture Binarization Procedure Coloring 2D→3D GOAL! Add Alpha Channel
123. Add Alpha Channel capture Binarization Procedure 2D→3D GOAL! Coloring
124. Making Color Array //14colors. Please cut & paste from color.txt byte[,] colors = { { 255, 255, 255 },{ 18, 0, 230 }, { 0, 152, 243 }, { 0, 241, 255 }, { 31, 195, 143 }, { 68, 153, 0 }, { 150, 158, 0 }, { 233, 160, 0 }, { 183, 104, 0 }, { 136, 32, 29 }, { 131, 7, 146 }, { 127, 0, 228 }, { 79, 0, 229 }, { 0, 0, 0 } }; //index of color(colNo=0~13) int colorNo = 0; void Start() { int w = Screen.width; int h = Screen.height; int sx = (int)(h * 0.1); h = (int)(h * 0.8); /*Following code are omitted.*/ } ①copy&paste from colors.txt ②Write by yourself
125. Change Color of Black Pixels (1/2) void SetColor(Texture2D texture) { //Do nothing if Mats are null if (bgraMat == null || binMat == null) { return; } unsafe { //Get pointer of pixel array of 2 Mats. byte* bgraPtr = bgraMat.DataPointer; byte* binPtr = binMat.DataPointer; //Calculate number of pixels of the image. int pixelCount = binMat.Width * binMat.Height; //Setting transparency and changing color. for (int i = 0; i < pixelCount; i++) { } } OpenCvSharp.Unity.MatToTexture(bgraMat, texture); } Changing color of each pixel. (See next slide.)
126. Change Color of Black Pixels(2/2) for (int i = 0; i < pixelCount; i++) { //Address of blue color of i-th pixel. int bgraPos = i * 4; //If i-th pixel of binPtr is 255(white). if (binPtr[i] == 255) { bgraPtr[bgraPos + 3] = 0; } //If i-th pixel of binPtr is 0(black). else { bgraPtr[bgraPos] = colors[colorNo, 0]; //B bgraPtr[bgraPos + 1] = colors[colorNo, 1]; //G bgraPtr[bgraPos + 2] = colors[colorNo, 2]; //R bgraPtr[bgraPos + 3] = 255; } }
127. Run Please substitute 0~12 to colorNo before you test. colorNo=3
128. Change Color by Using Button public void ChangeColor() { colorNo++; colorNo %= colors.Length / 3; SetColor(capTexture); } //Setting color/alpha information to texture. void SetColor(Texture2D texture) { //Do nothing if each Mat are null. if (bgraMat == null || binMat == null) { return; } unsafe { //Getting pointer of each Mat. byte* bgraPtr = bgraMat.DataPointer; byte* binPtr = binMat.DataPointer; /*Following code are omitted.*/
129. Calling ChangeColor() From UI ①Click ColorBtn ②Find Button ③Click + laying on bottom of OnClick()
130. Calling ChangeColor() From UI Find GameObject Drag&Drop into None of OnClick()
131. Calling ChangeColor() From UI ①Click No Function ②StampScript
132. Calling ChangeColor() From UI ChangeColor()
133. Run Capture Color Color
134. Maximization Preview Area ①RawImage Click
135. Maximization Preview Area Select Full Screen
136. Maximization Preview Area Set Left,Top,Right, Bottom to 0
137. Maximization Preview Area Un check RawImage
138. Showing Preview After Capturing IEnumerator ImageProcessing() { canvas.SetActive(false); //Release memory if (bgraMat != null) { bgraMat.Release(); } if (binMat != null) { binMat.Release(); } yield return new WaitForEndOfFrame(); CreateImages(); SetColor(capTexture); canvas.SetActive(true); //Show preview area preview.enabled = true; }
139. Run Capture
140. Add Alpha Channel capture Binarization Procedure 2D→3D GOAL! Coloring
141. Coloring Add Alpha Channel capture Binarization Procedure 2D→3D GOAL!
142. Making Prefab of Textured Quad Click Quad ②Drag&Drop into QuadをAssets
143. Making Prefab of Textured Quad Delete Quad
144. Stamping Image in Real World. //Template object of textured quad. public GameObject original; void Start() { int w = Screen.width; int h = Screen.height; capRect = new UnityEngine.Rect(0, 0, w, h); capTexture = new Texture2D(w, h, TextureFormat.RGBA32, false); preview.material.mainTexture = capTexture; } //Function to putting stamp object. public void PutObject() { } Source is shown in the next page.
145. Calculation of Physical Size of a Stamp //Getteing camera. Camera cam = Camera.main; //Convert left-bottom of screen into 3D space(z=0.6m) Vector3 v1 = cam.ViewportToWorldPoint(new Vector3(0, 0, 0.6f)); //Convert right-upper of screen into 3D space(z=0.6m) Vector3 v2 = cam.ViewportToWorldPoint(new Vector3(1, 1, 0.6f)); //Convert left-upper of screen into 3D space(z=0.6m) Vector3 v3 = cam.ViewportToWorldPoint(new Vector3(0, 1, 0.6f)); //Calculate physical size of stamp. float w = Vector3.Distance(v2, v3); float h = Vector3.Distance(v1, v3); /*次のページに続く*/ (0,0) (1,1) (0,1) v1 v2 v3
146. Instantiation of Stamp in Real World /*Continued form source code of previous slide*/ GameObject stamp = GameObject.Instantiate(original); //Set position/rotation/size of stamp relative to camera. stamp.transform.parent = cam.transform; stamp.transform.localPosition = new Vector3(0, 0, 0.6f); stamp.transform.localRotation = Quaternion.identity; stamp.transform.localScale = new Vector3(w, h, 1); //Creating texture to apply the object instantiated above. Texture2D stampTexture = new Texture2D(capTexture.width, capTexture.height); //Setting color and applying texture. SetColor(stampTexture); stamp.GetComponent<Renderer>().material.mainTexture = stampTexture; //Detach stamp object from cameara. stamp.transform.parent = null; preview.enabled = false;
147. Linking Object and Variable GameObject
148. Linking Object and Variable Find Quad
149. Linking Object and Variable Drag&Drop into Original
150. Calling PutObject() from UI ①Click StampBtn ②Find Button ③Click + laying on bottom of OnClick()
151. Calling PutObject() from UI Find GameObject Drag&Drop into None of OnClick()
152. Calling PutObject() from UI ①Click No Function ②StampScript
153. Calling PutObject() from UI PutObject()
154. Run
155. Setting Max Number of Stamp //Template object of textured quad public GameObject original; //List for holding generated stamps List<GameObject> stampList = new List<GameObject>(); void Start() { int w = Screen.width; int h = Screen.height; capRect = new UnityEngine.Rect(0, 0, w, h); capTexture = new Texture2D(w, h, TextureFormat.RGBA32, false); preview.material.mainTexture = capTexture; }
156. Setting Max Number of Stamp public void PutObject() { /*Ommited above source code.*/ stamp.transform.parent = null; //Stamp is memorized and deleted by following code. stampList.Add(stamp); if (stampList.Count == 10) { DestroyImmediate(stampList[0]. GetComponent<Renderer>().material.mainTexture); DestroyImmediate(stampList[0]); stampList.RemoveAt(0); } preview.enabled = false; }