Let's Start AR with ARCore & Unity

Tutorial slide of ARcontents development using ARCore & Unity for beginner.

1. Let’s Start AR with ARCore and Unity for beginner
2. Name： Takashi Yoshinaga Affiliation: Institute of Systems,Information Technologies and Nanotechnologies Field of work： Application of AR for med. and edu. SNS: Twitter -> @Tks_Yoshinaga LinkedIn -> tks-yoshinaga
3. ARCore 【Features】 (1) Motion Tracking for realize marker less AR. (2) Light Estimation which can estimate light strength and color. (3) Environmental Understanding to recognize floor and wall. (4) Augmented Image to recognize predefined image marker. (5) Cloud Anchor for sharing AR experience among multiple users.
4. Covered topics in this tutorial 【Features】 (1) Motion Tracking for realize marker less AR. (2) Light Estimation which can estimate light strength and color. (3) Environmental Understanding to recognize floor and wall. (4) Augmented Image to recognize predefined image marker. (5) Cloud Anchor for sharing AR experience among multiple users.
5. Create Unity Project (1/2) Click NEW button after start Unity New
6. Create Unity Project (2/2) Click Create Project after inputting project name. 3D Create Project Project Name Save Directory
7. Importing ARCore SDK to Unity ①Assets ②Import Package → Custom Package ③arcore-unity-sdk-xxx ④Open
8. Making a New Folder for Tutorial(1/3) Right Click Asset Folder
9. Making a New Folder for Tutorial(2/3) ①Create ②Folder
10. Making a New Folder for Tutorial(3/3) Change folder name to Sample1
11. Save Current Content Open Sample1 Input a scene name (ex. sample1) Save Save this scene by Ctrl + S
12. Motion Tracking
13. Setup a Camera for ARCore(1/2) Delete MainCamera
14. Setup a Camera for ARCore(2/2) ①GoogleARCore → Prefabs ② ARCore Device ③Drag & Drop
15. Put a Virtual Object (1/2) ①GoogleARCore → Examples → Common → Prefabs ② Andy Diffuse ③Drag & Drop
16. Put a Virtual Object (2/2) ①AndyDiffuse ②Put Andy in front of user. Position (x,y,z)=(0,0,0.5)[m]
17. Build Setting ①File ② Build Settings
18. Build Setting ②Switch Platform ① Android ③Player Settings
19. Build Setting ①Input Product Name ② Other Settings ③Disable Multithreaded Rendering
20. Build Setting ①Input Package Name 例) com.yourname.test1 ② Set Minimum API Level to Android 7.0
21. Build Setting ①XR Settings ② Enable ARCore Supported
22. Build and Install ①File ② Build & Run Please connect ARCore device and PC via USB
23. Build and Install ①Input a name of apk ② Save
24. Result
25. Environmental Understanding
26. Enable Environmental Understanding (1/5) ①Sample1
27. Enable Environmental Understanding (2/5) ①Right Click ②Create ③GoogleARCore ④SessionConfig
28. Enable Environmental Understanding (3/5) Change the file name (ex. sample1)
29. Enable Environmental Understanding (4/5) ①Click sample1.asset 【Plane Finding Mode】 (1)Disabled (2)Horizontal And Vertical (3)Horizontal (4)Vertical Choose a mode from(2)~(4)
30. Enable Environmental Understanding (5/5) ②Click ARCore Device ①Sample1 ③sample1.asset ④Drag & Drop to Session Config
31. Visualize Detected Planes (1/4) ①空白を右クリック ② Create Empty
32. Visualize Detected Planes (2/4) Change the name of GameObject to Controller
33. Visualize Detected Planes (3/4) ①Click Controller ②AddComponent ③Detected Plane Generator
34. Visualize Detected Planes (4/4) ①GoogleARCore → Examples → Common → Prefabs ③DetectedPlaneVisualizer ②Controller ④Drag & Drop to Detected Plane Prefab
35. Result Horizontal Plane Vertical Plane
36. Putting Object to the Tapped Place ①Controller ②Add Component
37. Putting Object to the Tapped Place ①New Script ③Create and Add ②PutScript
38. Putting Object to the Tapped Place ①Click Controller ②Double Click PutScript
39. Script using System.Collections; using System.Collections.Generic; using UnityEngine; using GoogleARCore; public class PutScript : MonoBehaviour { public GameObject andy; //Variable to handle CG(Andy) void Start () { } void Update () { //(1) Detect tap. //(2) Transform 2D position to 3D position of real world. //(3) Put Andy there. } }
40. Script void Update () { Touch touch; if (Input.touchCount < 1 || (touch = Input.GetTouch(0)).phase != TouchPhase.Began) { return; } //Calculate touched position of detected plane. TrackableHit hit; TrackableHitFlags filter = TrackableHitFlags.PlaneWithinPolygon; if (Frame.Raycast(touch.position.x, touch.position.y, filter, out hit)) { //Move Andy to pointed position. (Next page) } } touch.position hit
41. Script if (Frame.Raycast(touch.position.x, touch.position.y, filter, out hit)) { // If pointed trackable is plane if (hit.Trackable is DetectedPlane ) { //Set the position and the angle of Andy andy.transform.position = hit.Pose.position; andy.transform.rotation = hit.Pose.rotation; andy.transform.Rotate(0, 180, 0, Space.Self); //Set the anchor to fix Andy object to real space. var anchor = hit.Trackable.CreateAnchor(hit.Pose); andy.transform.parent = anchor.transform; } }
42. Coupling Virtual Object and Variable ②AndyDiffuse ①Controller ③ Drag & Drop to Put Script’s Andy
43. Result
44. Augmented Image
45. Preparation for Next Tutorial ①Assets ③Create ②Right Click ④Folder
46. Preparation for Next Tutorial Change the folder name to Sample2
47. Preparation for Next Tutorial File→New Scene
48. Save Open Sample2 Input scene name (ex. sample2) Save Save current Scene by Ctrl+S
49. Setup a Camera for ARCore(1/2) Delete Main Camera
50. Setup a Camera for ARCore(2/2) ①GoogleARCore → Prefabs ② ARCore Device ③Drag & Drop
51. Create a Configuration File ②Right Click ③Create ④GoogleARCore ⑤SessionConfig ①Sample2
52. Create a Configuration File Re name to sample2
53. Create a Configuration File ②Click ARCore Device ①Sample2 ③sample2.asset ④Drag & Drop to Session Config
54. Next step Add images to Augmented Image Database Click sample2.asset
55. Download Maker Image http://arfukuoka.lolipop.jp/arcore/images.zip
56. Import Marker Images ②sample2 ①Open images Folder ③Drag & Drop AndyMarker & earth into sample2 folder
57. Register Images as Marker Select AndyMarker and earth together
58. Register Images as Marker ①Right click an image ②Create ③GoogleARCore ④AugmentedImage Database
59. Register Images as Marker Marker data base is generated
60. Physical Size of Marker[m] ※If width is set to 0, marker size will be estimated automatically. Quality should be over 75 for robust detection
61. Sample2.asset
62. NewDatabase.asset Drag & Drop to Augmented Image Database
63. Download 3D Model https://poly.google.com/
64. Download 3D Model Serch with the keyword “Astronaut”
65. Download 3D Model Click
66. Download 3D Model Download → OBJ File
67. Rename the folder to Astronaut after unzip, because its name is too long
68. Import Object to Unity ②sample2 ①Select Astronaut Folder ③ Drag and Drop Astronaut folder into sample2 folder
69. Import Object to Unity Astronaut is appear in unity editor
70. Script for Marker Tracking ①Right Click ②Create Empty
71. Script for Marker Tracking Rename GameObject to Controller
72. Script for Marker Tracking ①AddComponent
73. Script for Marker Tracking ①New Script ③Create and Add ②MarkerScript
74. Script for Marker Tracking ②Double Click ①Controller
75. Script using GoogleARCore; public class MarkerScript : MonoBehaviour { public GameObject andy;//Variable to handle Andy public GameObject astronaut;// Variable to handle astronaut //Dictionary to hold pairs of marker id and virtual object Dictionary<int,GameObject> markerDic = new Dictionary<int,GameObject>(); //List of detected marker in each frame. List<AugmentedImage> markers = new List<AugmentedImage>(); void Start() { } void Update() { //Couple marker and object } }
76. Coupling Variable and Virtual Object ②Controller ①GoogleARCore → Examples → Common → Prefabs ③Andy Diffuse ④Drag & Drop to Andy
77. Coupling Variable and Virtual Object ②Controller ①Sample2 → Astronaut ③Astronaut ④Drag & Drop to Astronaut
78. void Update() { //Obtain markers which tracking state was updated Session.GetTrackables<AugmentedImage> (markers,TrackableQueryFilter.Updated); //Couple or remove marker and virtual object foreach (var image in markers) { int index = image.DatabaseIndex; //marker id GameObject obj = null; markerDic.TryGetValue(index, out obj);// get virtual object with id if (image.TrackingState == TrackingState.Tracking && obj == null) { //Couple marker and virtual object. } else if (image.TrackingState == TrackingState.Stopped && obj != null) { //Remove information of marker and virtual object. } } }
79. void Update() { //Obtain markers which tracking state was updated Session.GetTrackables<AugmentedImage> (markers,TrackableQueryFilter.Updated); //Couple or remove marker and virtual object foreach (var image in markers) { int index = image.DatabaseIndex; //marker id GameObject obj = null; markerDic.TryGetValue(index, out obj); // get virtual object with id if (image.TrackingState == TrackingState.Tracking && obj == null) { //Couple marker and virtual object. } else if (image.TrackingState == TrackingState.Stopped && obj != null) { //Remove information of marker and virtual object. } } }
80. if (image.TrackingState == TrackingState.Tracking && obj == null) { Anchor anchor = image.CreateAnchor(image.CenterPose); switch (index) { case 0://AndyMarker obj = GameObject.Instantiate(andy, anchor.transform); obj.transform.localPosition = new Vector3(0, 0.02f, -0.04f); obj.transform.Rotate(-90, 0, 0, Space.Self); obj.transform.localScale =new Vector3(0.3f, 0.3f, 0.3f); markerDic.Add(index, obj); break; case 1://earth obj = GameObject.Instantiate(astronaut, anchor.transform); obj.transform.localPosition = new Vector3(0.02f, 0.04f, 0); obj.transform.Rotate(90, 0, 0, Space.Self); obj.transform.localScale = new Vector3(0.01f, 0.01f, 0.01f); markerDic.Add(index, obj); break; } }
81. void Update() { //Obtain markers which tracking state was updated Session.GetTrackables<AugmentedImage> (markers,TrackableQueryFilter.Updated); //Couple or remove marker and virtual object foreach (var image in markers){ int index = image.DatabaseIndex; //marker id GameObject obj = null; markerDic.TryGetValue(index, out obj); // get virtual object with id if (image.TrackingState == TrackingState.Tracking && obj == null) { //Coupling marker and virtual object. } else if (image.TrackingState == TrackingState.Stopped && obj != null) { markerDic.Remove(image.DatabaseIndex); Destroy(obj); } } }
82. CloudAnchor Information is shared in the same device in this tutorial. (Works as Area Learning of Google Tango)
83. You can see how the application works from following link. https://twitter.com/Tks_Yoshinaga/stat us/1006137365239173120
84. Preparation for Next Tutorial Select Sample1 and Ctrl +D
85. Preparation for Next Tutorial sample3
86. Preparation for Next Tutorial Rename them to sample3
87. Preparation for Next Tutorial Double click sample3.unity
88. Preparation for Next Tutorial Confirm sample3 is loaded
89. Preparation for Next Tutorial ①ARCoreDevice ②sample3.asset ③Drag & Drop to Session Config
90. Preparation for Next Tutorial ②Dupricate PutScript by using Ctrl+D ①Assets
91. Preparation for Next Tutorial Rename PutScript1 to AnchorScript
92. Preparation for Next Tutorial Double click AnchorScript
93. Preparation for Next Tutorial using UnityEngine; using GoogleARCore; public class AnchorScript : MonoBehaviour { public GameObject andy; // Use this for initialization void Start() { } // Update is called once per frame void Update () { Touch touch; if (Input.touchCount < 1 || (touch = Input.GetTouch(0)).phase != TouchPhase.Began) { return; } Rename class name toAnchorScript
94. Preparation for Next Tutorial ①Controller ②Click of PutScript ③Remove Component
95. Preparation for Next Tutorial ①AnchorScript
96. Preparation for Next Tutorial ①Controller ②AndyDiffuse ③Drag & Drop to Andy ofAnchor Script
97. Creating UI ①Right Click ②UI ③Button
98. Creating UI ①Double Click Canvas ②Modify the angle of view to be able to see canvas
99. Creating UI ①Canvas ②Scale With Screen Size
100. Creating UI ②Click ③top-center ①Button
101. Creating UI Input property Pos X: 90 Pos Y: -35 Height: 60
102. Creating UI ①Button→Text ②Input Host in Text
103. Creating UI ①Right click ②UI ③Button
104. Creating UI ①Button(1) ②Click ③top-center
105. Creating UI Input property Pos X: -90 Pos Y: -35 Height: 60
106. Creating UI ①Button(1)→Text ②Input Resolve in Text
107. Creating UI ①Right click ②UI ③Text
108. Creating UI ①Text ②Click ③bottom-center
109. Creating UI Input property Pos X: 0 Pos Y: 30 Width:400 Height: 60
110. UI作成 ①Set Font Size to 50 ②Set all Alignment to center
111. Enable Cloud Anchor ①ARCoreDevice ②Double Click Sample3
112. Enable Cloud Anchor Check Enable Cloud Anchor
113. Setup GCP Create
114. Setup GCP Input project name
115. Setup GCP Search with ARCore Cloud Anchor API
116. Setup GCP Enable
117. Creating API Key ①Menu ②API & Services ③Credentials
118. Creating API Key ①Create credentials ②API Key
119. Creating API Key Copy
120. Setting API Key in Unity ①Edit ②Project Settings ③ARCore
121. Setting API Key in Unity ②Close ①Paste API key
122. Script public class AnchorScript : MonoBehaviour { // Use this for initialization void Start() { } public void HostAnchor() { } public void ResolveAnchor() { } void Update () { //Detect Button tapping or screen tapping //Put Andy on a plane if user taps screen. (Not button) //save current position of Andy } } Upload Environment Information Matching the space with Cloud Anchor
123. Distinguish button tap or screen tap using UnityEngine.EventSystems; public class AnchorScript : MonoBehaviour { /*Omitted: Start,HostAnchor,ResolveAnchor*/ void Update () { if (EventSystem.current.IsPointerOverGameObject (Input.GetTouch(0).fingerId)) { return; } Touch touch; if (Input.touchCount < 1 || (touch = Input.GetTouch(0)).phase != TouchPhase.Began) { return; } TrackableHit hit; TrackableHitFlags filter = TrackableHitFlags.PlaneWithinPolygon; /*Omitted*/
124. Holding current position of Andy Anchor currentAnchor = null; void Update () { /*Ommited: Detection of screen tap */ if (Frame.Raycast(touch.position.x, touch.position.y, filter, out hit)) { if ((hit.Trackable is DetectedPlane) && Vector3.Dot(Camera.main.transform.position – hit.Pose.position,hit.Pose.rotation * Vector3.up) > 0) { //Setting position and angle of Andy andy.transform.position = hit.Pose.position; andy.transform.rotation = hit.Pose.rotation; andy.transform.Rotate(0, 180, 0, Space.Self); //Setting anchor var anchor = hit.Trackable.CreateAnchor(hit.Pose); andy.transform.parent = anchor.transform; currentAnchor = anchor; } } }
125. Holding current position of Andy using GoogleARCore; using UnityEngine.EventSystems; using GoogleARCore.CrossPlatform; using UnityEngine.UI; public class AnchorScript : MonoBehaviour { public GameObject andy; Anchor currentAnchor = null; public Text txt; //Used for display the state of saving cloud anchor bool connecting = false; //state of connecting or disconnecting to CCP void Start() { } public void HostAnchor() { } Save the environmental information and cloud anchor (See next page..)
126. Hosting current position of Andy public void HostAnchor() { if (connecting || currentAnchor == null) { return; } connecting = true; txt.text = "hosting..."; //uploading environmental information. XPSession.CreateCloudAnchor(currentAnchor).ThenAction(result => { connecting = false; if (result.Response != CloudServiceResponse.Success) { txt.text = "Please Try Again"; return; } txt.text = result.Anchor.CloudId; }); }
127. Coupling Variable an Text UI ①Select Text ②Controller ②Drag & Drop to Txt of Anchor Script
128. Hosting Cloud Anchor After Button Tap(1/4) ②+ button of On Click() ①Button
129. Hosting Cloud Anchor After Button Tap(2/4) ①Controller ②Drag & Drop to None of On Click()
130. Hosting Cloud Anchor After Button Tap(3/4) No Function
131. Hosting Cloud Anchor After Button Tap(4/4) ①AnchorScript ②HostAnchor
132. 【Test】 ① Wait for planes are detected. ② Put Andy on a plane ③ Tap the Host button. ④ Bottom text will be hosting... ⑤ ID will shown if upload is success 【Next Step】 ① Shut down and reboot this app. ② Call Cloud Anchor API with ID ③ Match space with now and before ④ Andy will appear the same position. Enable saving ID in the next step
133. Saving ID of Cloud Anchor using UnityEngine.EventSystems; using UnityEngine.UI; using System.IO; public class AnchorScript : MonoBehaviour { public GameObject andy; Anchor currentAnchor = null; public Text txt; bool connecting = false; string filePath; void Start() { filePath = Application.persistentDataPath + "/anchorID.txt"; } public void HostAnchor() { } Uploading environmental information (編集中)
134. Saving ID of Cloud Anchor public void HostAnchor() { if (connecting || currentAnchor == null) { return; } connecting = true; txt.text = "hosting..."; //環境情報をアップロード XPSession.CreateCloudAnchor(currentAnchor).ThenAction(result => { connecting = false; if (result.Response != CloudServiceResponse.Success) { txt.text = "Please Try Again"; return; } txt.text = ""; StreamWriter sw = new StreamWriter(filePath, false); sw.WriteLine(result.Anchor.CloudId); sw.Close(); }); }
135. Next Step public class AnchorScript : MonoBehaviour { // Use this for initialization void Start() { } public void HostAnchor() { } public void ResolveAnchor() { } void Update () { //Detect Button tapping or screen tapping //Put Andy on a plane if user taps screen. (Not button) //save current position of Andy } } Upload Environment Information Matching the space with Cloud Anchor
136. Load ID From Text File. public void ResolveAnchor() { if (connecting) { return; } connecting = true; // Read ID from text file FileInfo file = new FileInfo(filePath); StreamReader sr = new StreamReader(file.OpenRead()); string cloudAnchorId = sr.ReadLine(); sr.Close(); txt.text=cloudAnchorId; }
137. Loading ID After Tapping Button ②+ of On Click() ①Button(1)
138. Loading ID After Tapping Button ①Controller
139. Loading ID After Tapping Button No Function
140. Loading ID After Tapping Button ①AnchorScript ②ResolveAnchor
141. 【Test】 ① Put Andy on a detected plane. ② Tap Host button ③ Shut down the application. ④ Reboot this application. ⑤ Tap Resolve button. ⑥ ID will appear on the bottom of screen. 【Next Step】 ① Call Cloud Anchor API with ID ③ Match space with CloudAnchor. ④ Position of Andy will reproduced. ここにIDが表示される
142. Matching Space with Clouc Anchor public void ResolveAnchor() { /*Omitted: Loading ID from text file.*/ //txt.text = cloudAnchorId; txt.text ="resolving..."; XPSession.ResolveCloudAnchor(cloudAnchorId).ThenAction(result => { connecting = false; if (result.Response != CloudServiceResponse.Success) { txt.text = "Please Try Again"; return; } txt.text = ""; andy.transform.position = result.Anchor.transform.position; andy.transform.rotation = result.Anchor.transform.rotation; andy.transform.Rotate(0, 180, 0, Space.Self); andy.transform.parent = result.Anchor.transform; }); }
143. Fin.