Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Tutorial: Creation of StampAR with OpenCV and ARCore

Tutorial: Creation of StampAR with OpenCV and ARCore

This slide deck explains how to develop Stamp-AR by using OpenCV and ARCore on Unity.

357c9e58ecce2865f9eb748192e5143f?s=128

TakashiYoshinaga

June 02, 2019
Tweet

More Decks by TakashiYoshinaga

Other Decks in Technology

Transcript

  1. Tutorial: Creating Stamp-AR with OpenCV

  2. Download and Installation ①Sample Data for the Tutorial http://arfukuoka.lolipop.jp/stampar/Sample.zip ②ARCoreSDK(v1.8.0)

    https://github.com/google-ar/arcore-unity-sdk/releases/tag/v1.8.0 ③Unity2017.4.15f1 or later https://unity3d.com/jp/unity/qa/lts-releases?version=2017.4 ④Android SDK https://developer.android.com/studio ※Please finish setting up Android build on Unity before hand.
  3. Goal of The Tutorial https://youtu.be/a8o1ieL01_w

  4. ARCore New marker-less AR platform which can available for Android

    devices. 【Features】 (1) Motion Tracking based on SLAM (2) Environmental Understanding (3) Light Estimation (4) Augmented Image (5) Cloud Anchor (6) Augmented Faces
  5. OpenCV Plus Unity  Unity asset of image processing based

    on OpenCV3.  OpenCVSharp was adapted to Unity environment.  Available on Windows/Mac/Android/iOS.
  6. Procedure Binarization capture Add Alpha Channel Coloring 2D→3D GOAL!

  7. Getting Started with Unity

  8. Creation of a Unity Project New

  9. Creation of a Unity Project Create Project Name of a

    Project Save Directory
  10. Addin a 3D Object ①Right Click ②3D Object ③Quad

  11. How to Play Click Play Button Click Play Button to

    Stop View of the Camera
  12. Modification of View Point of Scene [←] [→] Move to

    Right/Left [↑][↓] Zoom In/Out [Alt]+Drag Rotation +Drag Move to ↑↓←→
  13. Modification of Game Object Click to Select Translation Rotation Scaling

  14. Numerical Modification of Game Object  Detailed information is shown

    in the Inspector Tab  For instance, you can translate object by changing Position ②Change Position parameter to 0 0 0 ① Quad
  15. Applying a Texture to Material Change the appearance by applying

    texture image to 3D model
  16. Creation of a Material File ①Assets ②空白を右クリック

  17. Creation of a Material File ①Create ②Material

  18. Creation of a Material File ①Click Quad ②Open Materials ※Click▼button

  19. Applying a Texture Image ①Find NewMaterial ※Do NOT click! ②Drag

    & Drop into Element0 of Materials
  20. Applying a Texture Image ①Click Quad ②Open NewMaterial ※▼をクリック

  21. Applying a Texture Image Open Shader List

  22. Applying a Texture Image ①Unlit ②Transparent

  23. Applying a Texture Image Texture image will be assigned from

    here
  24. Importing Texture Image ①Assets ②ImportNewAsset...

  25. Importing Texture Image ①Sample Folder ②Select logo.png ③Import

  26. Assigning Texture to Material ①Click Quad

  27. Assigning Texture to Material ①Find logo file ②Drag&Drop into texture

    area
  28. Assigning Texture to Material

  29. Check Point & Idea for StampAR 【Check Point】 Appearance can

    be set just by assigning texture file if a material to use texture was applied to3D model. 【Idea】  Make a image which has alpha channel.  Change color of opaque pixels by using script.  Generate plane which is attached this image.
  30. Save the Scene ①File ②Save Scene as...

  31. Save the Secne ①Write the name of this scene ②Save

    Please save the scene frequently with ctrl+s after this procedure
  32. Installation of ARCore SDK ①Assets ②Import Package ③Open SDK from

    Custom Package arcore-unity-sdk-v1.8.0.unitypackage
  33. Installation of ARCore SDK Import

  34. Installation of ARCore SDK GoogleARCore will appear

  35. Setting Up Camera for AR Delete Main Camera

  36. Setting Up Camera for AR Assets→GoogleARCore→Prefabs

  37. Setting Up Camera for AR ①ARCore Device ②Drag & Drop

    into Hierarchy
  38. Setting Up Camera for AR ①ARCore Device ②Double Click DefaultSessionConfig

  39. Setting Up Camera for AR Switch Camera Focus Mode to

    Auto
  40. Modification of Position & Scale of a CG Click Quad

  41. Modification of Position & Scale of a CG Position: 0,

    0, 0.5 Scale: All 0.1
  42. Modification of Position & Scale of a CG Double Click

    Quad Since Quad object is very smaller than default size, make the view point close to the object.
  43. Saving the Scene Ctrl/Command + S

  44. Build Setting ①File ②Build Settings...

  45. Build Setting ①Android ②Switch Platform

  46. Build Setting ①Internal ②Player Settings

  47. Build Setting ①Resolution and Presentation ②Select Portrait as Default Orientation

  48. Build Setting ①Other Settings ②Turn Off Multithreaded Rendering

  49. Build Setting ①Set original PackageName 例)com.fukuoka.test ②Set Minimum API Level

    as Andrid 7.0
  50. Build Setting (for Unity2018) Set Allow unsafe Code Checked

  51. Build Setting ①XR Settings ②Setting ARCore Supported Checked

  52. Saving the Scene Ctrl/Command+S

  53. Installation App Into Smartphone ①File ②Build & Run

  54. Installation App Into Smartphone ②Save ①Set a name of installer(.apk)

  55. Run 3D object will be fixed the real space.

  56. From Next Slide… Preparation for Getting Started with Image Processing

  57. Installation of OpenCV Plus Unity ①Window ②Asset Store

  58. Installation of OpenCV Plus Unity Search “OpenCV Plus”

  59. Installation of OpenCV Plus Unity Click OpenCV Plus Unity

  60. Installation of OpenCV Plus Unity Download

  61. Installation of OpenCV Plus Unity Import

  62. Installation of OpenCV Plus Unity Import

  63. Installation of OpenCV Plus Unity OpenCV + Unity will appear

  64. Setting Up the View of Unity Editor Game This operation

    means making your modification of UI layout of application comfortable in Unity Editor.
  65. Setting Up the View of Unity Editor ①Click Free Aspect

    ②Click+ This operation means making your modification of UI layout of application comfortable in Unity Editor.
  66. Setting Up the View of Unity Editor ①Put Name ②800

    × 1280 It’s not necessary to match actual resolution of your smartphone in this tutorial.
  67. Importing Sample UI ②Import Package ①Assets ③Custom Package...

  68. Importing Sample UI ②Open ①StampUI.unitypackage

  69. Importing Sample UI Import

  70. Importing Sample UI Scene

  71. Importing Sample UI ①Assets ②Canvas will be added

  72. Using Sample UI ①Canvas ②Drag&Drop into Hierarchy

  73. Using Sample UI UI will appear in the Scene UI

    might not be visible the view point which you using know. But it’s not problem! Please see next page.
  74. Using Sample UI ①Double Click Canvas ②You can see whole

    UI
  75. Using Sample UI ②Modify view point to make UI to

    be facing to you ①Click x or z
  76. Role of UI Start capture Change color Put stamp into

    the space Vewer of captured or processed image
  77. Applying Alpha Texture to Viewer ①Open Canvas ②Click Assets

  78. Applying Alpha Texture to Viewer ①RawImage ②Find Material

  79. Applying Alpha Texture to Viewer ①Find NewMaterial ※Do NOT click

    ②Drag&Drop into Material
  80. Applying Alpha Texture to Viewer Viewer became to be able

    to show alpha channel image
  81. Preparation of Writing C# Script Write click on the blank

    of Hierarchy
  82. Preparation of Writing C# Script Create Empty

  83. Preparation of Writing C# Script ①Game Object ②Add Component

  84. Preparation of Writing C# Script ①New Script ②Name of Script

    (ex. StampScript) ③Create and Add
  85. Preparation of Writing C# Script ①Game Object ②ColoringScript will be

    added
  86. Preparation of Writing C# Script Double Click

  87. Importing OpenCV and UnityEngine.UI using UnityEngine; using UnityEngine.UI; using OpenCvSharp;

    using OpenCvSharp.Demo; public class StampScript : MonoBehaviour { // Start関数は初期化のために一度だけ実行される void Start () { cg = GameObject.Find ("Robot Kyle"); } // Update関数は毎フレーム実行される void Update () { } }
  88. canvas Declaration of Variables //Canvas which involves UI parts public

    GameObject canvas; //Viewer of image. public RawImage preview; //Region of screen shot. UnityEngine.Rect capRect; //Texture of screen shot image. Texture2D capTexture; void Start () { } void Update () { } preview
  89. Linking Objects and Variables ①Click GameObject ②Canvas & Preview will

    be added
  90. Linking Objects and Variables Find Canvas Drag & Drop into

    Canvas of StampScript
  91. Linking Objects and Variables See RawImage Drag& Drop into Previewof

    StampScript
  92. Procedure Binarization capture Add Alpha Channel Coloring 2D→3D GOAL!

  93. Procedure Binarization Add Alpha Channel Coloring 2D→3D GOAL! capture

  94. Preaparation of Screen Capture //Canvas which involves UI parts public

    GameObject canvas; //Viewer of image. public RawImage preview; //Region of screen shot. UnityEngine.Rect capRect; //Texture of screen shot image. Texture2D capTexture; void Start () { int w = Screen.width; int h = Screen.height; //Definition of capture region as (0,0) to (w,h). capRect = new UnityEngine.Rect(0, 0, w, h); //Creating texture image of the size of capRect. capTexture = new Texture2D(w, h, TextureFormat.RGBA32, false); //Applying capTexture as texture of preview area. preview.material.mainTexture = capTexture; } width height (0,0)
  95. Making Function of Image Processing void Start () { /*Code

    was omitted in the slide.*/ } IEnumerator ImageProcessing() { canvas.SetActive(false);//Making UIs invisible yield return new WaitForEndOfFrame(); capTexture.ReadPixels(capRect, 0, 0);//Starting capture capTexture.Apply();//Apply captured image. canvas.SetActive(true);//Making UIs visible } public void StartCV() { StartCoroutine(ImageProcessing());//Calling coroutine } Write!
  96. Calling StartCV() From UI ①Click CaptureBtn ②Find Button ③Click +

    laying on bottom of OnClick()
  97. Calling StartCV() From UI Find GameObject Drag & Drop into

    the area written None, of OnClick()
  98. Calling StartCV() From UI ①Click No Function ②StampScript

  99. Calling StartCV() From UI StartCV()

  100. Ctrl/Command + S

  101. Build & Run!

  102. Run Captured image appears Capture

  103. Another Way to Test Play

  104. Another Way to Test Click Play again to STOP

  105. Refactoring(1/2) IEnumerator ImageProcessing() { canvas.SetActive(false); yield return new WaitForEndOfFrame(); capTexture.ReadPixels(capRect,

    0, 0); capTexture.Apply(); canvas.SetActive(true); } void CreateImages() { /* Cut & Paste Code of Image Creation */ } Image Creation Make CreateImages function since amount of source code for image creation will be increased later.
  106. Refactoring(2/2) IEnumerator ImageProcessing() { canvas.SetActive(false); yield return new WaitForEndOfFrame(); CreateImages();

    //画像の生成 canvas.SetActive(true); } void CreateImages() { capTexture.ReadPixels(capRect, 0, 0); capTexture.Apply(); } Do not forget to call CreateImages() from ImageProcessing()
  107. Procedure Binarization Add Alpha Channel Coloring 2D→3D GOAL! capture

  108. Procedure Add Alpha Channel Coloring 2D→3D GOAL! capture Binarization

  109. From now on, we use OpenCV!

  110. Binarization of Gray Scale Image 0 255 0 255 Binarization

    means splitting grayscale(0~255) into 0 or 255 by threshold. It’s very important technique to define pixels which should be processed.
  111. Preparation of Using Image with OpenCV UnityEngine.Rect capRect; //Texture to

    store captured image. Texture2D capTexture; //Mat:Format of image for OpenCV //bgraMat is for color image with alpha channel. //binMat is for binarized image. Mat bgraMat, binMat; void Start () { int w = Screen.width; int h = Screen.height; //Definition of capture region as (0,0) to (w,h). capRect = new UnityEngine.Rect(0, 0, w, h); //Creating texture image of the size of capRect. capTexture = new Texture2D(w, h, TextureFormat.RGBA32, false); //Applying capTexture as texture of preview area. preview.material.mainTexture = capTexture; }
  112. Binarization void CreateImage() { capTexture.ReadPixels(capRect, 0, 0); capTexture.Apply(); //Conversion Texure2D

    to Mat bgraMat = OpenCvSharp.Unity.TextureToMat(capTexture); //Conversion Color Image to Gray Scale Image binMat = bgraMat.CvtColor(ColorConversionCodes.BGRA2GRAY); //Binarization of image with Otsu’s method. binMat = binMat.Threshold(100, 255, ThresholdTypes.Otsu); //Conversion Gray Scale to BGRA to change its color later. bgraMat = binMat.CvtColor(ColorConversionCodes.GRAY2BGRA); } bgraMat binMat(GrayScale) binMat (Binarized) bgraMat (B=G=R)
  113. Visualization of a Result of Binarization IEnumerator ImageProcessing() { canvas.SetActive(false);

    yield return new WaitForEndOfFrame(); CreateImages();//Creating Image SetColor(capTexture);//Setting color to capTexture canvas.SetActive(true); } //Setting color information of bgraMat to texture. void SetColor(Texture2D texture) { OpenCvSharp.Unity.MatToTexture(bgraMat, texture); }
  114. Run ①Capture ②Binalized Image

  115. Releasing Memories Allocated for Mat IEnumerator ImageProcessing() { canvas.SetActive(false); //Releasing

    Memories allocated for two Mats if (bgraMat != null) { bgraMat.Release(); } if (binMat != null) { binMat.Release(); } yield return new WaitForEndOfFrame(); CreateImages(); SetColor(capTexture); canvas.SetActive(true); }
  116. Procedure Add Alpha Channel Coloring 2D→3D GOAL! capture Binarization

  117. capture Binarization Procedure Coloring 2D→3D GOAL! Add Alpha Channel

  118. Make White Pixels to Transparent (1/2) void SetColor(Texture2D texture) {

    //Do nothing if Mats are null if (bgraMat == null || binMat == null) { return; } unsafe { //Get pointer of pixel array of 2 Mats. byte* bgraPtr = bgraMat.DataPointer; byte* binPtr = binMat.DataPointer; //Calculate number of pixels of the image. int pixelCount = binMat.Width * binMat.Height; //Make white pixels to transparent for (int i = 0; i < pixelCount; i++) { } } OpenCvSharp.Unity.MatToTexture(bgraMat, texture); } 後ほど処理を記述
  119. Make White Pixels to Transparent (2/2) byte* bgraPtr = bgraMat.DataPointer;

    byte* binPtr = binMat.DataPointer; int pixelCount = binMat.Width * binMat.Height; for (int i = 0; i < pixelCount; i++) { //Address of blue color of i-th pixel int bgraPos = i * 4; //If i-th pixel of binPtr is 255(white). if (binPtr[i] == 255) { bgraPtr[bgraPos + 3] = 0; } //If i-th pixel of binPtr is 0(black). else { bgraPtr[bgraPos + 3] = 255; } }
  120. Data Structure of Pixel Array  Pixel data of each

    types, gray or bgra, are stored in 1-dimensional array of byte.  binPtr(gray) :  bgraPtr(color+alpha): binPtr bgraPtr Length of pixel array is n (=width*height) Length of pixel array is n*4 [0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] ・・・ ・・・ 0th pixel 1st 2nd
  121. Run ①Capture ②Binarized image with alpha channel

  122. capture Binarization Procedure Coloring 2D→3D GOAL! Add Alpha Channel

  123. Add Alpha Channel capture Binarization Procedure 2D→3D GOAL! Coloring

  124. Making Color Array //14colors. Please cut & paste from color.txt

    byte[,] colors = { { 255, 255, 255 },{ 18, 0, 230 }, { 0, 152, 243 }, { 0, 241, 255 }, { 31, 195, 143 }, { 68, 153, 0 }, { 150, 158, 0 }, { 233, 160, 0 }, { 183, 104, 0 }, { 136, 32, 29 }, { 131, 7, 146 }, { 127, 0, 228 }, { 79, 0, 229 }, { 0, 0, 0 } }; //index of color(colNo=0~13) int colorNo = 0; void Start() { int w = Screen.width; int h = Screen.height; int sx = (int)(h * 0.1); h = (int)(h * 0.8); /*Following code are omitted.*/ } ①copy&paste from colors.txt ②Write by yourself
  125. Change Color of Black Pixels (1/2) void SetColor(Texture2D texture) {

    //Do nothing if Mats are null if (bgraMat == null || binMat == null) { return; } unsafe { //Get pointer of pixel array of 2 Mats. byte* bgraPtr = bgraMat.DataPointer; byte* binPtr = binMat.DataPointer; //Calculate number of pixels of the image. int pixelCount = binMat.Width * binMat.Height; //Setting transparency and changing color. for (int i = 0; i < pixelCount; i++) { } } OpenCvSharp.Unity.MatToTexture(bgraMat, texture); } Changing color of each pixel. (See next slide.)
  126. Change Color of Black Pixels(2/2) for (int i = 0;

    i < pixelCount; i++) { //Address of blue color of i-th pixel. int bgraPos = i * 4; //If i-th pixel of binPtr is 255(white). if (binPtr[i] == 255) { bgraPtr[bgraPos + 3] = 0; } //If i-th pixel of binPtr is 0(black). else { bgraPtr[bgraPos] = colors[colorNo, 0]; //B bgraPtr[bgraPos + 1] = colors[colorNo, 1]; //G bgraPtr[bgraPos + 2] = colors[colorNo, 2]; //R bgraPtr[bgraPos + 3] = 255; } }
  127. Run Please substitute 0~12 to colorNo before you test. colorNo=3

  128. Change Color by Using Button public void ChangeColor() { colorNo++;

    colorNo %= colors.Length / 3; SetColor(capTexture); } //Setting color/alpha information to texture. void SetColor(Texture2D texture) { //Do nothing if each Mat are null. if (bgraMat == null || binMat == null) { return; } unsafe { //Getting pointer of each Mat. byte* bgraPtr = bgraMat.DataPointer; byte* binPtr = binMat.DataPointer; /*Following code are omitted.*/
  129. Calling ChangeColor() From UI ①Click ColorBtn ②Find Button ③Click +

    laying on bottom of OnClick()
  130. Calling ChangeColor() From UI Find GameObject Drag&Drop into None of

    OnClick()
  131. Calling ChangeColor() From UI ①Click No Function ②StampScript

  132. Calling ChangeColor() From UI ChangeColor()

  133. Run Capture Color Color

  134. Maximization Preview Area ①RawImage Click

  135. Maximization Preview Area Select Full Screen

  136. Maximization Preview Area Set Left,Top,Right, Bottom to 0

  137. Maximization Preview Area Un check RawImage

  138. Showing Preview After Capturing IEnumerator ImageProcessing() { canvas.SetActive(false); //Release memory

    if (bgraMat != null) { bgraMat.Release(); } if (binMat != null) { binMat.Release(); } yield return new WaitForEndOfFrame(); CreateImages(); SetColor(capTexture); canvas.SetActive(true); //Show preview area preview.enabled = true; }
  139. Run Capture

  140. Add Alpha Channel capture Binarization Procedure 2D→3D GOAL! Coloring

  141. Coloring Add Alpha Channel capture Binarization Procedure 2D→3D GOAL!

  142. Making Prefab of Textured Quad Click Quad ②Drag&Drop into QuadをAssets

  143. Making Prefab of Textured Quad Delete Quad

  144. Stamping Image in Real World. //Template object of textured quad.

    public GameObject original; void Start() { int w = Screen.width; int h = Screen.height; capRect = new UnityEngine.Rect(0, 0, w, h); capTexture = new Texture2D(w, h, TextureFormat.RGBA32, false); preview.material.mainTexture = capTexture; } //Function to putting stamp object. public void PutObject() { } Source is shown in the next page.
  145. Calculation of Physical Size of a Stamp //Getteing camera. Camera

    cam = Camera.main; //Convert left-bottom of screen into 3D space(z=0.6m) Vector3 v1 = cam.ViewportToWorldPoint(new Vector3(0, 0, 0.6f)); //Convert right-upper of screen into 3D space(z=0.6m) Vector3 v2 = cam.ViewportToWorldPoint(new Vector3(1, 1, 0.6f)); //Convert left-upper of screen into 3D space(z=0.6m) Vector3 v3 = cam.ViewportToWorldPoint(new Vector3(0, 1, 0.6f)); //Calculate physical size of stamp. float w = Vector3.Distance(v2, v3); float h = Vector3.Distance(v1, v3); /*次のページに続く*/ (0,0) (1,1) (0,1) v1 v2 v3
  146. Instantiation of Stamp in Real World /*Continued form source code

    of previous slide*/ GameObject stamp = GameObject.Instantiate(original); //Set position/rotation/size of stamp relative to camera. stamp.transform.parent = cam.transform; stamp.transform.localPosition = new Vector3(0, 0, 0.6f); stamp.transform.localRotation = Quaternion.identity; stamp.transform.localScale = new Vector3(w, h, 1); //Creating texture to apply the object instantiated above. Texture2D stampTexture = new Texture2D(capTexture.width, capTexture.height); //Setting color and applying texture. SetColor(stampTexture); stamp.GetComponent<Renderer>().material.mainTexture = stampTexture; //Detach stamp object from cameara. stamp.transform.parent = null; preview.enabled = false;
  147. Linking Object and Variable GameObject

  148. Linking Object and Variable Find Quad

  149. Linking Object and Variable Drag&Drop into Original

  150. Calling PutObject() from UI ①Click StampBtn ②Find Button ③Click +

    laying on bottom of OnClick()
  151. Calling PutObject() from UI Find GameObject Drag&Drop into None of

    OnClick()
  152. Calling PutObject() from UI ①Click No Function ②StampScript

  153. Calling PutObject() from UI PutObject()

  154. Run

  155. Setting Max Number of Stamp //Template object of textured quad

    public GameObject original; //List for holding generated stamps List<GameObject> stampList = new List<GameObject>(); void Start() { int w = Screen.width; int h = Screen.height; capRect = new UnityEngine.Rect(0, 0, w, h); capTexture = new Texture2D(w, h, TextureFormat.RGBA32, false); preview.material.mainTexture = capTexture; }
  156. Setting Max Number of Stamp public void PutObject() { /*Ommited

    above source code.*/ stamp.transform.parent = null; //Stamp is memorized and deleted by following code. stampList.Add(stamp); if (stampList.Count == 10) { DestroyImmediate(stampList[0]. GetComponent<Renderer>().material.mainTexture); DestroyImmediate(stampList[0]); stampList.RemoveAt(0); } preview.enabled = false; }
  157. None