Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Tutorial: Creation of AR Coloring with OpenCV and ARCore

Tutorial: Creation of AR Coloring with OpenCV and ARCore

Description how to create AR Coloring app by using OpenCV, ARCore and Unity.

357c9e58ecce2865f9eb748192e5143f?s=128

TakashiYoshinaga

May 12, 2019
Tweet

More Decks by TakashiYoshinaga

Other Decks in Technology

Transcript

  1. with OpenCV and ARCore Tutorial: Creation of AR Coloring

  2. Download and Installation ①Sample Data for the Tutorial http://arfukuoka.lolipop.jp/nurie/sample.zip ②ARCoreSDK(v1.8.0)

    https://github.com/google-ar/arcore-unity-sdk/releases/tag/v1.8.0 ③Unity2017.4.15f1 or later https://unity3d.com/jp/unity/qa/lts-releases?version=2017.4 ④Android SDK https://developer.android.com/studio ※Please finish setting up Android build on Unity before hand.
  3. You can see also… Source Code https://github.com/TakashiYoshinaga/AR-Coloring

  4. Goal of The Tutorial https://youtu.be/E5Jy4iunpyQ https://youtu.be/MviBGZtiv5A

  5. ARCore New marker-less AR platform which can available for Android

    devices. 【Features】 (1) Motion Tracking based on SLAM (2) Environmental Understanding (3) Light Estimation (4) Augmented Image (5) Cloud Anchor (6) Augmented Faces
  6. OpenCV Plus  Unity asset of image processing based on

    OpenCV3.  OpenCVSharp was adapted to Unity environment.  Available on Windows/Mac/Android/iOS.
  7. Procedure of Image Processing Clipping ROI Binarization Square Detection Transformation

    input output Attachment
  8. Getting Started with Unity

  9. Creation of a Unity Project New

  10. Creation of a Unity Project Create Project Name of the

    Project
  11. Adding a 3D Object Right Click 3DObject → Cube

  12. Applying a Texture to Material Change the appearance by applying

    texture image
  13. Creation of a Material File ①Assets ②Right Click

  14. Creation of a Material File ①Create ②Material

  15. Creation of a Material File ①NewMaterial ②Click here

  16. Creation of a Material File ①Legacy Shaders ②Diffuse

  17. Creation of a Material File Texture will be assigned here

    later
  18. Importing a Texture Image ①Assets ②ImportNewAsset...

  19. Importing a Texture Image ①Sample Folder ②Select ilovear & opencv

    ③Import
  20. Applying a Texture Image NewMaterial

  21. Applying a Texture Image ①See ilovear ②Drag & Drop

  22. Applying a Texture Image Image was assigned

  23. Applying a Material to 3D Object ①Click Cube ②Open Materials

    ※Click ▼
  24. Applying a Material to 3D Object ①See NewMaterial ②Drag &

    Drop it into Element0
  25. Applying a Material to 3D Object Appearance was changed

  26. Changing a Material to 3D Object ①Cube ②Open the Detail

    Information of NewMaterial
  27. Changing a Material to 3D Object ①See opencv ②Drag &

    Drop it Texture Area
  28. Changing a Material to 3D Object Appearance was changed

  29. Check Point & Next Step 【Check Point】 Appearance can be

    changed just by replacing texture file if a material to use texture was applied to3D model. 【Next Step】  Create texture from inner area of square frame.  Writing C# script to replace texture automaticaly.
  30. Save the Scene ①File ②Save Scene as...

  31. Save the Scene ①Write the name of scene ②Save

  32. Installation of ARCore SDK ①Assets ②Import Package ③Open SDK from

    Custom Package arcore-unity-sdk-v1.8.0.unitypackage
  33. Installation of ARCore SDK Import

  34. Installation of ARCore SDK GoogleARCore was added

  35. Setting Up Camera for AR Delete Main Camera

  36. Setting Up Camera for AR Assets→GoogleARCore→Prefabs

  37. Setting Up Camera for AR ①ARCore Device ②Drag & Drop

    into Hierarchy
  38. Setting Up Camera for AR ①ARCore Device ②Double Click DefaultSessionConfig

  39. Setting Up Camera for AR Switch Camera Focus Mode to

    Auto
  40. Modification of Position & Scale of a CG Click Cube

  41. Modification of Position & Scale of a CG Position: 0,

    -0.2, 0.5 Scale: All 0.1
  42. 視点 ViewPoint 0.5 Y X -0.2 Z

  43. Saving the Scene Ctrl/Command + S

  44. Build Setting ①File ②Build Settings...

  45. Build Setting ①Android ②Switch Platform

  46. Build Setting ①Internal ②Player Settings

  47. Build Setting ①Resolution and Presentation ②Select Portrait as Default Orientation

  48. Build Setting ①Other Settings ②Turn Off Multithreaded Rendering

  49. Build Setting ①Set original PackageName 例)com.fukuoka.test ②Set Minimum API Level

    as Andrid 7.0
  50. Build Setting (for Unity2018) Set Allow unsafe Code Checked

  51. Build Setting ①XR Settings ②Setting ARCore Supported Checked

  52. Saving the Scene Ctrl/Command+S

  53. Installation App Into Smartphone ①File ②Build & Run

  54. Installation App Into Smartphone ①Set a name of installer(.apk) ②Save

  55. Run

  56. From Next Slide.. Preparation for Getting Started with Image Processing

  57. Installation of OpenCV Plus Unity ①Window ②Asset Store

  58. Installation of OpenCV Plus Unity Search “OpenCV Plus”

  59. Installation of OpenCV Plus Unity Open OpenCV Plus

  60. Installation of OpenCV Plus Unity Download

  61. Installation of OpenCV Plus Unity Import

  62. Installation of OpenCV Plus Unity Import

  63. Installation of OpenCV Plus Unity OpenCV + Unity will appear

  64. Setting Up the View of Unity Editor Game This operation

    means making your modification of UI layout of application comfortable in Unity Editor.
  65. Setting Up the View of Unity Editor ①Click Free Aspect

    ②Click+ This operation means making your modification of UI layout of application comfortable in Unity Editor.
  66. Setting Up the View of Unity Editor ①Put Name ②800

    × 1280
  67. Importing Sample UI ②Import Package ①Assets ③Custom Package...

  68. Importing Sample UI ②Open ①NurieUI.unitypackage

  69. Importing Sample UI Import

  70. Importing Sample UI Scene

  71. Using Sample UI ①Canvas ②Drag & Drop into Hierarchy

  72. Using Sample UI UI was added to the Scene UI

    might not be visible the view point which you using know. But it’s not problem! Please see next page.
  73. Using Sample UI ①Double Click Canvas ②You can see whole

    UI
  74. Using Sample UI ①Click x or z ②Modify view point

    to make UI to be facing to you
  75. Role of UI Results of Image Processing Showing ROI Switch

    visibility of UI Starting capture
  76. Preparation of Writing C# Script Write click on the blank

    of Hierarchy
  77. Preparation of Writing C# Script Create Empty

  78. Preparation of Writing C# Script ①Game Object ②Add Component

  79. Preparation of Writing C# Script ①New Script ②Name of Script

    (ex. ColoringScript) ③Create and Add
  80. Preparation of Writing C# Script ①Game Object ②ColoringScript was added

  81. Preparation of Writing C# Script Double Click

  82. Importing OpenCV and UnityEngine.UI using UnityEngine; using UnityEngine.UI; using OpenCvSharp;

    using OpenCvSharp.Demo; public class ColoringScript : MonoBehaviour { // Start関数は初期化のために一度だけ実行される void Start () { cg = GameObject.Find ("Robot Kyle"); } // Update関数は毎フレーム実行される void Update () { } }
  83. Declaration of Variables public MeshRenderer target; //Rendering Setting of Cube

    public GameObject canvas; //Canvas which involves UI public RawImage viewL, viewR; //Result viewer UnityEngine.Rect capRect;//Region of screen shot Texture2D capTexture; //Texture of screenshot image Texture2D colTexture; //Result of image processing(color) Texture2D binTexture; //Result of image processing(gray) void Start () { } void Update () { } canvas viewL viewR colTexture binTexture
  84. Linking Objects and Variables ①Click GameObject ②See Target

  85. Linking Objects and Variables See Cube Drag & Drop into

    Target of ColoringScript
  86. Linking Objects and Variables See Canvas Drag & Drop into

    Canvas of ColoringScript
  87. Linking Objects and Variables See RawImage Drag & Drop into

    ViewL of ColoringScript
  88. Linking Objects and Variables View RawImage(1) Drag & Drop into

    ViewR of ColoringScript
  89. Procedure of Image Processing Clipping ROI Binarization Square Detection Transformation

    input output Attachment
  90. Procedure of Image Processing Clipping Binarization Square Detection Transformation output

    Attachment input
  91. Preparation of Screen Capture public MeshRenderer target; public GameObject canvas;

    public RawImage viewL, viewR; UnityEngine.Rect capRect; Texture2D capTexture; Texture2D colTexture; Texture2D binTexture; void Start () { int w = Screen.width; int h = Screen.height; //Definition of capture region as (0,0) to (w,h) capRect = new UnityEngine.Rect(0, 0, w, h); //Creating texture image of the size of capRect capTexture = new Texture2D(w, h, TextureFormat.RGB24, false); } width height (0,0)
  92. Making Function of Image Processing void Start () { /*Code

    was omitted in the slide.*/ } IEnumerator ImageProcessing() { canvas.SetActive(false);//Making UIs invisible yield return new WaitForEndOfFrame(); capTexture.ReadPixels(capRect, 0, 0);//Starting capturing capTexture.Apply();//Apply captured image. /*Setting texture on the coloring target object (cube)*/ target.material.mainTexture = capTexture; canvas.SetActive(true);//Making UIs visible. } public void StartCV() { StartCoroutine(ImageProcessing());//Calling coroutine. } Write!
  93. Calling StartCV() From UI ①Click Button ②Find Button ③Click +

    laying on bottom of OnClick()
  94. Calling StartCV() From UI Find GameObject Drag & Drop into

    the area written None, of OnClick()
  95. Calling StartCV() From UI ①Click No Function ②ColoringScript

  96. Calling StartCV() From UI StartCV()

  97. Ctrl/Command + S

  98. Build & Run!

  99. Run Capture

  100. Procedure of Image Processing Clipping Binarization Square Detection Transformation output

    Attachment input
  101. Procedure of Image Processing Binarization Square Detection Transformation output Attachment

    Clipping ROI input
  102. Clipping Image of Inside the ROI void Start () {

    int w = Screen.width; int h = Screen.height; //Setting up position/size of ROI int sx = (int)(w * 0.2); //Start of X int sy = (int)(h * 0.3); //Start of Y w = (int)(w * 0.6); //Width of ROI h = (int)(h * 0.4); //Height of ROi //キャプチャする領域を保持したRectを作成 capRect = new UnityEngine.Rect(0, 0, w, h); capTexture = new Texture2D(w, h, TextureFormat.RGB24, false); } capRect = new UnityEngine.Rect(sx, sy, w, h); Replace!
  103. Run Capture You can find coloring sheet (pdf) in Markers

    folder.
  104. Refactoring (1/2) IEnumerator ImageProcessing() { canvas.SetActive(false); yield return new WaitForEndOfFrame();

    //描画終了を待つ capTexture.ReadPixels(capRect, 0, 0);//キャプチャ開始 capTexture.Apply();//各画素の色をテクスチャに反映 /*下記の一行でオブジェクトにテクスチャを張り付け*/ target.material.mainTexture = capTexture; canvas.SetActive(true); } void CreateImage() { /*Cut & Paste Code of Image Creation*/ } void ShowImage() { /*Cut & Paste Code of Image Visualization*/ } Image Creation Visualization
  105. Refactoring (2/2) IEnumerator ImageProcessing() { canvas.SetActive(false); yield return new WaitForEndOfFrame();

    CreateImage(); //Image Creation ShowImage(); //Image Visualization canvas.SetActive(true); } void CreateImage() { capTexture.ReadPixels(capRect, 0, 0); capTexture.Apply(); } void ShowImage() { target.material.mainTexture = capTexture; }
  106. Procedure of Image Processing Binarization Square Detection Transformation output Attachment

    Clipping ROI input
  107. Clipping ROI input Procedure of Image Processing Square Detection Transformation

    output Attachment Binarization
  108. From now on, we use OpenCV!

  109. Binarization of Gray Scale Image 0 255 0 255 Binarization

    means splitting grayscale(0~255) into 0 or 255 by threshold. It’s very important technique to define pixels which should be processed.
  110. Preparation of Using Image with OpenCV Texture2D colTexture; Texture2D binTexture;

    //Mat:Format of image for OpenCV //bgr is for color image、bin is for binarized image Mat bgr, bin; void Start () { int w = Screen.width; int h = Screen.height; int sx = (int)(w * 0.2); int sy = (int)(h * 0.3); w = (int)(w * 0.6); h = (int)(h * 0.4); capRect = new UnityEngine.Rect(sx, sy, w, h); capTexture = new Texture2D(w, h, TextureFormat.RGB24, false); }
  111. Binarization void CreateImage() { capTexture.ReadPixels(capRect, 0, 0); capTexture.Apply(); //Conversion Texure2D

    to Mat bgr = OpenCvSharp.Unity.TextureToMat(capTexture); //Conversion Color Image to Gray Scale Image bin = bgr.CvtColor(ColorConversionCodes.BGR2GRAY); //Binarization of image with Otsu’s method. bin = bin.Threshold(100, 255, ThresholdTypes.Otsu); Cv2.BitwiseNot(bin, bin); } Color Image Gray Scale Binary Inverse
  112. Visualization of Results void ShowImage() { //Releasing memories of textures.

    if (colTexture != null) { DestroyImmediate(colTexture); } if (binTexture != null) { DestroyImmediate(binTexture); } //Conversion of Mat to Texture2D colTexture = OpenCvSharp.Unity.MatToTexture(bgr); binTexture = OpenCvSharp.Unity.MatToTexture(bin); //Attaching texture to RawImage for visualization. viewL.texture = colTexture; viewR.texture = binTexture; //スクショ画像をモデルに適用 target.material.mainTexture = colTexture; }
  113. Releasing Memories Allocated for Mat IEnumerator ImageProcessing() { canvas.SetActive(false); yield

    return new WaitForEndOfFrame(); CreateImage(); ShowImage(); //Releasing Memories allocated for two Mats bgr.Release(); bin.Release(); canvas.SetActive(true); }
  114. Run ①Put marker into ROI ②Capture ③Clipped Image and Binarized

    Image
  115. Clipping ROI input Procedure of Image Processing Square Detection Transformation

    output Attachment Binarization
  116. Binarization Clipping ROI input Procedure of Image Processing Transformation output

    Attachment Square Detection
  117. Preparation of Square Frame Detection IEnumerator ImageProcessing() { canvas.SetActive(false); yield

    return new WaitForEndOfFrame(); CreateImage(); Point[] corners; //4 corners of square will be memorized FindRect(out corners); //Square Frame Detection ShowImage(); //画像の表示 bgr.Release(); bin.Release(); canvas.SetActive(true); } void FindRect(out Point[] corners) { /*Code will be described from next page.*/ }
  118. Contour Detection //Initialization of corners corners = null; //contour points

    and hierarchy Point[][] contours; HierarchyIndex[] h; //Contour detection bin.FindContours(out contours, out h, RetrievalModes.External, ContourApproximationModes.ApproxSimple); //Finding the contour of which has the most wide area. double maxArea = 0; for(int i = 0; i < contours.Length; i++) { double area = Cv2.ContourArea(contours[i]); if (area > maxArea) { maxArea = area; corners = contours[i]; } }
  119. Visualization of the Result void FindRect(out Point[] corners) { /*Code

    is omitted in this slide.*/ double maxArea = 0; for (int i = 0; i < contours.Length; i++) { double area = Cv2.ContourArea(contours[i]); if (area > maxArea) { maxArea = area; corners = contours[i]; } } //Write a contour line of max area in bgr. if (corners != null) { bgr.DrawContours( new Point[][] { corners }, 0, Scalar.Red, 5); } }
  120. Run

  121. Polygon Approximation void FindRect(out Point[] corners) { /*Code is omitted

    in this slide*/ double maxArea = 0; for(int i = 0; i < contours.Length; i++) { //Calculate the length of contour line. double length = Cv2.ArcLength(contours[i], true); //Polygon Approximation. Point[] tmp = Cv2.ApproxPolyDP( contours[i], length * 0.01f, true); double area = Cv2.ContourArea(contours[i]); //If number of corner is 4. if (area > maxArea) { maxArea = area; corners = contours[i]; } } /*次のページに続く*/ if (tmp.Length == 4 && area > maxArea) corners = tmp;
  122. Visualization of Corners void FindRect(out Point[] corners) { /*Code is

    omitted in this slide.*/ if (corners != null) { bgr.DrawContours( new Point[][] { corners }, 0, Scalar.Red, 5); //Draw circle on the position of corner point. for(int i = 0; i < corners.Length; i++) { bgr.Circle(corners[i], 20, Scalar.Blue, 5); } } }
  123. Run 四角形以外は 認識されない

  124. Before seeing next step… void FindRect(out Point[] corners) { /*Code

    is omitted in this slide.*/ //Comment out the visualization code. /*if (corners != null) { bgr.DrawContours( new Point[][] { corners }, 0, Scalar.Red, 5); //各頂点の位置に円を描画 for(int i = 0; i < corners.Length; i++) { bgr.Circle(corners[i], 20, Scalar.Blue, 5); } }*/ }
  125. Binarization Clipping ROI input Procedure of Image Processing Transformation output

    Attachment Square Detection
  126. Square Detection Binarization Clipping ROI input Procedure of Image Processing

    output Attachment Transformation
  127. Perspective Transformation (0, 0) (255, 0) (0, 255) (255 255)

    [0] [1] [2] [3]  Deform distorted square to front view based on the result of calculation of perspective transformation matrix.
  128. Perspective Transformation IEnumerator ImageProcessing() { canvas.SetActive(false); yield return new WaitForEndOfFrame();

    CreateImage(); Point[] corners; FindRect(out corners); TransformImage(corners); //Deform distorted square. ShowImage(); //画像の表示 bgr.Release(); bin.Release(); canvas.SetActive(true); } void TransformImage(Point[] corners) { /*Code will be described from the next page.*/ }
  129. Perspective Transformation void TransformImage(Point[] corners) { //Do nothing if square

    wasn’t found. if (corners == null) return; //Input detect corners. Point2f[] input = { corners[0], corners[1], corners[2], corners[3] }; //Define corners of square image. Point2f[] square = { new Point2f(0, 0), new Point2f(0, 255), new Point2f(255, 255), new Point2f(255, 0) }; //Calculation of transformation matrix. Mat transform = Cv2.GetPerspectiveTransform(input, square); //Deform image as front view square. Cv2.WarpPerspective(bgr,bgr,transform, new Size(256, 256)); } (0, 0) (255, 0) (0, 255) (255 255) [0] [1] [2] [3]
  130. Run Result of deformation is different by the rotation of

    square frame in the ROI. Rotated
  131.  Local position of corner at [0] depends on rotation

    of square.  Sorting is necessary to obtain image of front standing view. [0] [1] [2] [3] [0] [1] [2] [3] Succeeded Failed
  132. Sorting Corner Points void TransformImage(Point[] corners) { if (corners ==

    null) return; //Sorting SortCorners(corners); Point2f[] input = { corners[0], corners[1], corners[2], corners[3] }; Point2f[] square = { new Point2f(0, 0), new Point2f(0, 255), new Point2f(255, 255), new Point2f(255, 0) }; Mat transform = Cv2.GetPerspectiveTransform(input, square); Cv2.WarpPerspective(bgr,bgr,transform, new Size(256, 256)); } void SortCorners(Point[] corners) { /*Code of sorting is described in the next page.*/ }
  133. Sorting Corner Points void SortCorners(Point[] corners) { System.Array.Sort(corners, (a, b)

    => a.X.CompareTo(b.X)); if (corners[0].Y > corners[1].Y) { corners.Swap(0, 1); } if (corners[3].Y > corners[2].Y) { corners.Swap(2, 3); } } [0] [1] [2] [3] [2] [0] [1] [3] [3] [0] [1] [2] Sort by X axis Sort [2][3] by Y axis
  134. Clip a Image Inside the Square Frame void TransformImage(Point[] corners)

    { if (corners == null) return; SortCorners(corners); Point2f[] input = { corners[0], corners[1], corners[2], corners[3] }; Point2f[] square = { new Point2f(0, 0), new Point2f(0, 255), new Point2f(255, 255), new Point2f(255, 0) }; Mat transform = Cv2.GetPerspectiveTransform(input, square); Cv2.WarpPerspective(bgr,bgr,transform, new Size(256, 256)); int s = (int)(256*0.05);//Line width of frame is 5% of square int w = (int)(256*0.9);//Width of clipping area is 90% of square OpenCvSharp.Rect innerRect = new OpenCvSharp.Rect(s, s, w, w); bgr = bgr[innerRect]; }
  135. Square Detection Binarization Clipping ROI input Procedure of Image Processing

    output Attachment Transformation
  136. Transformation Square Detection Binarization Clipping ROI input Procedure of Image

    Processing output Attachment
  137. Applying Texture to 3D Object void ShowImage() { //すでにcolTextureが存在するならいったん削除 if

    (colTexture != null) { DestroyImmediate(colTexture); } if (binTexture != null) { DestroyImmediate(binTexture); } //Matをテクスチャに変換 colTexture = OpenCvSharp.Unity.MatToTexture(bgr); binTexture = OpenCvSharp.Unity.MatToTexture(gray); //RawImageに切り抜き画像を表示 viewL.texture = colTexture; viewR.texture = binTexture; //Applying texture to target 3D object target.material.mainTexture = capTexture; //Show Canvas gain. canvas.SetActive(true); } target.material.mainTexture = colTexture; Replace
  138. Run

  139. Switching Visiblity of ROI ①Toggle ②Click + laying on the

    bottom of OnValueChanged
  140. Switching Visiblity of ROI ①See Panel ②Drag&Drop it int None

    of OnValue Changed
  141. Switching Visiblity of ROI Click No Function

  142. Switching Visiblity of ROI ①GameObject ②Click SetActive of Dynamic bool

  143. Run Turn off

  144. Repalace 3D Model

  145. Poly Download 3D model from Google Poly https://poly.google.com/

  146. Getting 3D Model Input search word here Search「Duck」

  147. Getting 3D Model Rubber Duck

  148. Getting 3D Model Download → OBJ File

  149. Getting 3D Model Unzip archive.zip

  150. Importing 3D Model ①Assets ②Drag & Drop archive folder into

    Assets
  151. Replacing 3D Model Delete Cube

  152. Replacing 3D Model Assets → archive

  153. Replacing 3D Model ①RubberDuck ②Drag&Drop into Hierarchy

  154. Modification of Position/Size of 3D Model ①RubberDuck ②Position 0 -0.25

    0.5 Rotation 0 150 0 Scale 0.03 0.03 0.03
  155. Replacing 3D Model RubberDuck →RubberDuck_mesh

  156. Replacing 3D Model GameObject

  157. Replacing 3D Model Drag&Drop RubberDuck_mesh into Target

  158. Complete You can find caroling sheet in the Markers folder.