Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Tutorial: Creation of StampAR with OpenCV and ARCore

Tutorial: Creation of StampAR with OpenCV and ARCore

This slide deck explains how to develop Stamp-AR by using OpenCV and ARCore on Unity.

TakashiYoshinaga

June 02, 2019
Tweet

More Decks by TakashiYoshinaga

Other Decks in Technology

Transcript

  1. Download and Installation ①Sample Data for the Tutorial http://arfukuoka.lolipop.jp/stampar/Sample.zip ②ARCoreSDK(v1.8.0)

    https://github.com/google-ar/arcore-unity-sdk/releases/tag/v1.8.0 ③Unity2017.4.15f1 or later https://unity3d.com/jp/unity/qa/lts-releases?version=2017.4 ④Android SDK https://developer.android.com/studio ※Please finish setting up Android build on Unity before hand.
  2. ARCore New marker-less AR platform which can available for Android

    devices. 【Features】 (1) Motion Tracking based on SLAM (2) Environmental Understanding (3) Light Estimation (4) Augmented Image (5) Cloud Anchor (6) Augmented Faces
  3. OpenCV Plus Unity  Unity asset of image processing based

    on OpenCV3.  OpenCVSharp was adapted to Unity environment.  Available on Windows/Mac/Android/iOS.
  4. Modification of View Point of Scene [←] [→] Move to

    Right/Left [↑][↓] Zoom In/Out [Alt]+Drag Rotation +Drag Move to ↑↓←→
  5. Numerical Modification of Game Object  Detailed information is shown

    in the Inspector Tab  For instance, you can translate object by changing Position ②Change Position parameter to 0 0 0 ① Quad
  6. Check Point & Idea for StampAR 【Check Point】 Appearance can

    be set just by assigning texture file if a material to use texture was applied to3D model. 【Idea】  Make a image which has alpha channel.  Change color of opaque pixels by using script.  Generate plane which is attached this image.
  7. Save the Secne ①Write the name of this scene ②Save

    Please save the scene frequently with ctrl+s after this procedure
  8. Installation of ARCore SDK ①Assets ②Import Package ③Open SDK from

    Custom Package arcore-unity-sdk-v1.8.0.unitypackage
  9. Modification of Position & Scale of a CG Double Click

    Quad Since Quad object is very smaller than default size, make the view point close to the object.
  10. Setting Up the View of Unity Editor Game This operation

    means making your modification of UI layout of application comfortable in Unity Editor.
  11. Setting Up the View of Unity Editor ①Click Free Aspect

    ②Click+ This operation means making your modification of UI layout of application comfortable in Unity Editor.
  12. Setting Up the View of Unity Editor ①Put Name ②800

    × 1280 It’s not necessary to match actual resolution of your smartphone in this tutorial.
  13. Using Sample UI UI will appear in the Scene UI

    might not be visible the view point which you using know. But it’s not problem! Please see next page.
  14. Using Sample UI ②Modify view point to make UI to

    be facing to you ①Click x or z
  15. Role of UI Start capture Change color Put stamp into

    the space Vewer of captured or processed image
  16. Importing OpenCV and UnityEngine.UI using UnityEngine; using UnityEngine.UI; using OpenCvSharp;

    using OpenCvSharp.Demo; public class StampScript : MonoBehaviour { // Start関数は初期化のために一度だけ実行される void Start () { cg = GameObject.Find ("Robot Kyle"); } // Update関数は毎フレーム実行される void Update () { } }
  17. canvas Declaration of Variables //Canvas which involves UI parts public

    GameObject canvas; //Viewer of image. public RawImage preview; //Region of screen shot. UnityEngine.Rect capRect; //Texture of screen shot image. Texture2D capTexture; void Start () { } void Update () { } preview
  18. Preaparation of Screen Capture //Canvas which involves UI parts public

    GameObject canvas; //Viewer of image. public RawImage preview; //Region of screen shot. UnityEngine.Rect capRect; //Texture of screen shot image. Texture2D capTexture; void Start () { int w = Screen.width; int h = Screen.height; //Definition of capture region as (0,0) to (w,h). capRect = new UnityEngine.Rect(0, 0, w, h); //Creating texture image of the size of capRect. capTexture = new Texture2D(w, h, TextureFormat.RGBA32, false); //Applying capTexture as texture of preview area. preview.material.mainTexture = capTexture; } width height (0,0)
  19. Making Function of Image Processing void Start () { /*Code

    was omitted in the slide.*/ } IEnumerator ImageProcessing() { canvas.SetActive(false);//Making UIs invisible yield return new WaitForEndOfFrame(); capTexture.ReadPixels(capRect, 0, 0);//Starting capture capTexture.Apply();//Apply captured image. canvas.SetActive(true);//Making UIs visible } public void StartCV() { StartCoroutine(ImageProcessing());//Calling coroutine } Write!
  20. Refactoring(1/2) IEnumerator ImageProcessing() { canvas.SetActive(false); yield return new WaitForEndOfFrame(); capTexture.ReadPixels(capRect,

    0, 0); capTexture.Apply(); canvas.SetActive(true); } void CreateImages() { /* Cut & Paste Code of Image Creation */ } Image Creation Make CreateImages function since amount of source code for image creation will be increased later.
  21. Refactoring(2/2) IEnumerator ImageProcessing() { canvas.SetActive(false); yield return new WaitForEndOfFrame(); CreateImages();

    //画像の生成 canvas.SetActive(true); } void CreateImages() { capTexture.ReadPixels(capRect, 0, 0); capTexture.Apply(); } Do not forget to call CreateImages() from ImageProcessing()
  22. Binarization of Gray Scale Image 0 255 0 255 Binarization

    means splitting grayscale(0~255) into 0 or 255 by threshold. It’s very important technique to define pixels which should be processed.
  23. Preparation of Using Image with OpenCV UnityEngine.Rect capRect; //Texture to

    store captured image. Texture2D capTexture; //Mat:Format of image for OpenCV //bgraMat is for color image with alpha channel. //binMat is for binarized image. Mat bgraMat, binMat; void Start () { int w = Screen.width; int h = Screen.height; //Definition of capture region as (0,0) to (w,h). capRect = new UnityEngine.Rect(0, 0, w, h); //Creating texture image of the size of capRect. capTexture = new Texture2D(w, h, TextureFormat.RGBA32, false); //Applying capTexture as texture of preview area. preview.material.mainTexture = capTexture; }
  24. Binarization void CreateImage() { capTexture.ReadPixels(capRect, 0, 0); capTexture.Apply(); //Conversion Texure2D

    to Mat bgraMat = OpenCvSharp.Unity.TextureToMat(capTexture); //Conversion Color Image to Gray Scale Image binMat = bgraMat.CvtColor(ColorConversionCodes.BGRA2GRAY); //Binarization of image with Otsu’s method. binMat = binMat.Threshold(100, 255, ThresholdTypes.Otsu); //Conversion Gray Scale to BGRA to change its color later. bgraMat = binMat.CvtColor(ColorConversionCodes.GRAY2BGRA); } bgraMat binMat(GrayScale) binMat (Binarized) bgraMat (B=G=R)
  25. Visualization of a Result of Binarization IEnumerator ImageProcessing() { canvas.SetActive(false);

    yield return new WaitForEndOfFrame(); CreateImages();//Creating Image SetColor(capTexture);//Setting color to capTexture canvas.SetActive(true); } //Setting color information of bgraMat to texture. void SetColor(Texture2D texture) { OpenCvSharp.Unity.MatToTexture(bgraMat, texture); }
  26. Releasing Memories Allocated for Mat IEnumerator ImageProcessing() { canvas.SetActive(false); //Releasing

    Memories allocated for two Mats if (bgraMat != null) { bgraMat.Release(); } if (binMat != null) { binMat.Release(); } yield return new WaitForEndOfFrame(); CreateImages(); SetColor(capTexture); canvas.SetActive(true); }
  27. Make White Pixels to Transparent (1/2) void SetColor(Texture2D texture) {

    //Do nothing if Mats are null if (bgraMat == null || binMat == null) { return; } unsafe { //Get pointer of pixel array of 2 Mats. byte* bgraPtr = bgraMat.DataPointer; byte* binPtr = binMat.DataPointer; //Calculate number of pixels of the image. int pixelCount = binMat.Width * binMat.Height; //Make white pixels to transparent for (int i = 0; i < pixelCount; i++) { } } OpenCvSharp.Unity.MatToTexture(bgraMat, texture); } 後ほど処理を記述
  28. Make White Pixels to Transparent (2/2) byte* bgraPtr = bgraMat.DataPointer;

    byte* binPtr = binMat.DataPointer; int pixelCount = binMat.Width * binMat.Height; for (int i = 0; i < pixelCount; i++) { //Address of blue color of i-th pixel int bgraPos = i * 4; //If i-th pixel of binPtr is 255(white). if (binPtr[i] == 255) { bgraPtr[bgraPos + 3] = 0; } //If i-th pixel of binPtr is 0(black). else { bgraPtr[bgraPos + 3] = 255; } }
  29. Data Structure of Pixel Array  Pixel data of each

    types, gray or bgra, are stored in 1-dimensional array of byte.  binPtr(gray) :  bgraPtr(color+alpha): binPtr bgraPtr Length of pixel array is n (=width*height) Length of pixel array is n*4 [0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] ・・・ ・・・ 0th pixel 1st 2nd
  30. Making Color Array //14colors. Please cut & paste from color.txt

    byte[,] colors = { { 255, 255, 255 },{ 18, 0, 230 }, { 0, 152, 243 }, { 0, 241, 255 }, { 31, 195, 143 }, { 68, 153, 0 }, { 150, 158, 0 }, { 233, 160, 0 }, { 183, 104, 0 }, { 136, 32, 29 }, { 131, 7, 146 }, { 127, 0, 228 }, { 79, 0, 229 }, { 0, 0, 0 } }; //index of color(colNo=0~13) int colorNo = 0; void Start() { int w = Screen.width; int h = Screen.height; int sx = (int)(h * 0.1); h = (int)(h * 0.8); /*Following code are omitted.*/ } ①copy&paste from colors.txt ②Write by yourself
  31. Change Color of Black Pixels (1/2) void SetColor(Texture2D texture) {

    //Do nothing if Mats are null if (bgraMat == null || binMat == null) { return; } unsafe { //Get pointer of pixel array of 2 Mats. byte* bgraPtr = bgraMat.DataPointer; byte* binPtr = binMat.DataPointer; //Calculate number of pixels of the image. int pixelCount = binMat.Width * binMat.Height; //Setting transparency and changing color. for (int i = 0; i < pixelCount; i++) { } } OpenCvSharp.Unity.MatToTexture(bgraMat, texture); } Changing color of each pixel. (See next slide.)
  32. Change Color of Black Pixels(2/2) for (int i = 0;

    i < pixelCount; i++) { //Address of blue color of i-th pixel. int bgraPos = i * 4; //If i-th pixel of binPtr is 255(white). if (binPtr[i] == 255) { bgraPtr[bgraPos + 3] = 0; } //If i-th pixel of binPtr is 0(black). else { bgraPtr[bgraPos] = colors[colorNo, 0]; //B bgraPtr[bgraPos + 1] = colors[colorNo, 1]; //G bgraPtr[bgraPos + 2] = colors[colorNo, 2]; //R bgraPtr[bgraPos + 3] = 255; } }
  33. Change Color by Using Button public void ChangeColor() { colorNo++;

    colorNo %= colors.Length / 3; SetColor(capTexture); } //Setting color/alpha information to texture. void SetColor(Texture2D texture) { //Do nothing if each Mat are null. if (bgraMat == null || binMat == null) { return; } unsafe { //Getting pointer of each Mat. byte* bgraPtr = bgraMat.DataPointer; byte* binPtr = binMat.DataPointer; /*Following code are omitted.*/
  34. Showing Preview After Capturing IEnumerator ImageProcessing() { canvas.SetActive(false); //Release memory

    if (bgraMat != null) { bgraMat.Release(); } if (binMat != null) { binMat.Release(); } yield return new WaitForEndOfFrame(); CreateImages(); SetColor(capTexture); canvas.SetActive(true); //Show preview area preview.enabled = true; }
  35. Stamping Image in Real World. //Template object of textured quad.

    public GameObject original; void Start() { int w = Screen.width; int h = Screen.height; capRect = new UnityEngine.Rect(0, 0, w, h); capTexture = new Texture2D(w, h, TextureFormat.RGBA32, false); preview.material.mainTexture = capTexture; } //Function to putting stamp object. public void PutObject() { } Source is shown in the next page.
  36. Calculation of Physical Size of a Stamp //Getteing camera. Camera

    cam = Camera.main; //Convert left-bottom of screen into 3D space(z=0.6m) Vector3 v1 = cam.ViewportToWorldPoint(new Vector3(0, 0, 0.6f)); //Convert right-upper of screen into 3D space(z=0.6m) Vector3 v2 = cam.ViewportToWorldPoint(new Vector3(1, 1, 0.6f)); //Convert left-upper of screen into 3D space(z=0.6m) Vector3 v3 = cam.ViewportToWorldPoint(new Vector3(0, 1, 0.6f)); //Calculate physical size of stamp. float w = Vector3.Distance(v2, v3); float h = Vector3.Distance(v1, v3); /*次のページに続く*/ (0,0) (1,1) (0,1) v1 v2 v3
  37. Instantiation of Stamp in Real World /*Continued form source code

    of previous slide*/ GameObject stamp = GameObject.Instantiate(original); //Set position/rotation/size of stamp relative to camera. stamp.transform.parent = cam.transform; stamp.transform.localPosition = new Vector3(0, 0, 0.6f); stamp.transform.localRotation = Quaternion.identity; stamp.transform.localScale = new Vector3(w, h, 1); //Creating texture to apply the object instantiated above. Texture2D stampTexture = new Texture2D(capTexture.width, capTexture.height); //Setting color and applying texture. SetColor(stampTexture); stamp.GetComponent<Renderer>().material.mainTexture = stampTexture; //Detach stamp object from cameara. stamp.transform.parent = null; preview.enabled = false;
  38. Run

  39. Setting Max Number of Stamp //Template object of textured quad

    public GameObject original; //List for holding generated stamps List<GameObject> stampList = new List<GameObject>(); void Start() { int w = Screen.width; int h = Screen.height; capRect = new UnityEngine.Rect(0, 0, w, h); capTexture = new Texture2D(w, h, TextureFormat.RGBA32, false); preview.material.mainTexture = capTexture; }
  40. Setting Max Number of Stamp public void PutObject() { /*Ommited

    above source code.*/ stamp.transform.parent = null; //Stamp is memorized and deleted by following code. stampList.Add(stamp); if (stampList.Count == 10) { DestroyImmediate(stampList[0]. GetComponent<Renderer>().material.mainTexture); DestroyImmediate(stampList[0]); stampList.RemoveAt(0); } preview.enabled = false; }