Augmented Reality with Project Tango - Droidcon 2016 Berlin

Transcript

1. with Project Tango Dominik Helleberg
2. https://de.wikipedia.org/wiki/Erweiterte_Realit%C3%A4t http://bitstars.com/glossary/augm ented-reality/ http://www.augmentedplanet.com/2010/04/get-real-redefining-augmented-reality-games/ardronei phone.JPG Augmented Reality
3. https://www.google.de/trends/explor e#q=Augmented%20Reality&cm pt=q&tz=Etc%2FGMT-1 Augmented Reality
4. https://www.google.de/trends/explor e#q=Virtual%20reality Augmented Reality
5. http://mscorpnews.blob.core.windows.net/ncmedia/2015/01/Microsoft-HoloLens-MixedWorld-RGB.png Augmented Reality
6. https://news.microsoft.com/microsoft-hololens-press-materials/ Augmented Reality
7. https://news.microsoft.com/microsoft-hololens-press-materials/ Augmented Reality
8. http://www.magicleap.com/#/home Augmented Reality
9. https://www.youtube.com/watch?v=kPMHcanq0xM#t= 41 Augmented Reality
10. https://www.youtube.com/watch?v=CHV9xTGD4Ws Augmented Reality
11. http://www.digi-capital.com/news/wp-content/upl oads/2015/04/ARVR-Forecast.jpg Augmented Reality
12. Augmented Reality - Basics https://en.wikipedia.org/wiki/Augmented_reality Augmented reality (AR) is a live direct or indirect view of a physical, real-world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as sound, video, graphics or GPS data. -> It’s important to understand the “context”.
13. Augmented Reality - Examples By Rama - http://fr.wikipedia.org/wiki/Image:Hud_on_the_cat.jpg, Public Domain, https://commons.wikimedia.org/w/index.php?curid=129590 By Sebastian Klein/Wikipedia, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=3608534 HUD Displays Context: - Vehicle (Position, Speed, Engine)
14. Augmented Reality - Examples http://www.technologybloggers.org/gadgets/google-android-powered-glasses/ Von Tim.Reckmann - Eigenes Werk, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=32750308 Weather Forecast (Google Glass) Context: - Location (coarse) - Time - Weather forecast
15. Augmented Reality - Examples https://commons.wikimedia.org/wiki/File:Wikitude_World_Browser_@Salzburg_Old_Town_3.jpg Wikitude Locations Context: - Location (coarse) - Direction (compass / angle) - Location of POIs
16. Augmented Reality - Examples Von Meximex - Used the app and took a picture, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=23501409 Virtual objects Context: - exact, realtime location and direction of the mobile device - camera characteristics - 3D model of the virtual objects - 3D model of the real objects - Light model of the real world
17. Augmented Reality – Visual FX https://vimeo.com/155092798
18. Augmented Reality – Virtual Objects Context: - exact, realtime location and direction of the mobile device - camera characteristics - 3D model of the virtual objects - 3D model of the real objects - Light model of the real world
19. Augmented Reality – Virtual Objects Realtime location / camera chracteristics https://www.blender.org/download/demo-files/ -> BMW Demo by Mike Pan
20. Augmented Reality – VFX Realtime location / camera chracteristics Von Meximex - Used the app and took a picture, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=23501409
21. Augmented Reality – Virtual Objects Realtime location / camera chracteristics Von Meximex - Used the app and took a picture, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=23501409
22. Augmented Reality – Virtual Objects Realtime location / camera chracteristics
23. Augmented Reality – Virtual Objects Context: - exact, realtime location and direction of the mobile device - camera characteristics - 3D model of the virtual objects - 3D model of the real objects - Light model of the real world
24. Augmented Reality – Virtual Objects 3D Model https://www.blender.org/download/demo-files/ -> BMW Demo by Mike Pan
25. Augmented Reality – Visual FX 3D Model https://vimeo.com/155092798
26. Augmented Reality – Virtual Objects Context: - exact, realtime location and direction of the mobile device - camera characteristics - 3D model of the virtual objects - 3D model of the real objects - Light model of the real world
27. Augmented Reality – Virtual Objects 3D Model of the real world Von Meximex - Used the app and took a picture, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=23501409
28. Augmented Reality – Virtual Objects 3D Model of the real world https://news.microsoft.com/microsoft-hololens-press-materials/
29. Augmented Reality – Virtual Objects 3D Model of the real world https://news.microsoft.com/microsoft-hololens-press-materials/
30. Augmented Reality – Virtual Objects 3D Model of the real world https://github.com/stetro/project-tango-poc
31. Augmented Reality – Virtual Objects 3D Model of the real world https://vimeo.com/155092798
32. Augmented Reality – Virtual Objects Context: - exact, realtime location and direction of the mobile device - camera characteristics - 3D model of the virtual objects - 3D model of the real objects - Light model of the real world
33. Augmented Reality – Virtual Objects Light model
34. Augmented Reality – Virtual Objects Light model Von Meximex - Used the app and took a picture, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=23501409
35. Augmented Reality – Virtual Objects Light model https://news.microsoft.com/microsoft-hololens-press-materials/
36. Augmented Reality – Virtual Objects Light model http://gurneyjourney.blogspot.de/2012/01/reference-balls.html http://imgur.com/a/P2Nbp
37. Augmented Reality – Virtual Objects Context: - exact, realtime location and direction of the mobile device - camera characteristics - 3D model of the virtual objects - 3D model of the real objects - Light model of the real world
38. Project Tango https://www.google.com/atap/proj ect-tango/
39. https://www.google.com/atap/proj ect-tango/ http://www.androidheadlines.com/2014/02/googl es-project-tango-phone-renders-full-3d-maps-heres-look-like.html Project Tango 2014 Public announcement of “Project Tango” and release of the “peanut” prototype 2014 / 2015 “Yellowstone” Tablet released an sold 3000 times Sept 2016 Phab2 Pro
40. https://www.google.com/atap/proj ect-tango/ Project Tango • NVIDIA Tegra K1 with 192 CUDA cores • 7.02” 1920x1200 HD IPS display (323 ppi) • Dual-band Wi-Fi (2.4GHz/5GHz) • Bluetooth 4.0 • 4 GB RAM • Accelerometer • Ambient light • Barometer • Compass • aGPS • Gyroscope
41. https://www.google.com/atap/proj ect-tango/ Project Tango Infrared Laser 3D depth sensing Motion tracking camera 4 MP 2µm RGB-IR pixel sensor
42. Project Tango – Motion Tracking
43. Project Tango – Motion Tracking
44. Project Tango – Depth Sensing https://www.google.com/atap/project-tango/about-project-tango/index.html • Tango’s Depth sensor detects the range from the device to surfaces within 0.5 – 4 meters • It generates a point cloud which can be used to generate a mesh or enhance AR-Apps DEMO
45. Project Tango – Depth Sensing • BACKUP
46. Project Tango – Area Learning / indoor navigation Context: - exact, realtime location and direction of the mobile device - camera characteristics - 3D model of the virtual objects - 3D model of the real objects - Light model of the real world
47. Project Tango – Area Learning / indoor navigation https://www.google.com/atap/project-tango/about-project-tango/index.html • Tango recognizes or “learns” significant “markers” of an areas and exports them into an ADF (“Area Description File”) • A Tango app may load an ADF file and try to match visible markers against the ADF file • After “matching” on an ADF file, Tango can determine it’s position in local ADF coordinates
48. https://developers.google.com/proj ect-tango/apis/overview Project Tango – Getting started Unreal 4 engine
49. Project Tango – Getting started JAVA API Rajawali Framework Camera Objects / Textures Lights Rajawali Utils Tango Java APIs Java Support Lib Tango Service Sensor Data Application Logic Android APIs https://github.com/Rajaw ali/Rajawali
50. Project Tango – Getting started JAVA API https://github.com/Rajaw ali/Rajawali DirectionalLight light = new DirectionalLight(1, 0.2, -1); light.setColor(1, 1, 1); light.setPower(0.8f); light.setPosition(3, 2, 4); getCurrentScene().addLight(light); // Set-up a material: green with application of the light and // instructions. Material material = new Material(); material.setColor(0xff009900); try { Texture t = new Texture("instructions", R.drawable.instructions); material.addTexture(t); } catch (ATexture.TextureException e) { e.printStackTrace(); } material.setColorInfluence(0.1f); material.enableLighting(true); material.setDiffuseMethod(new DiffuseMethod.Lambert()); // Build a Cube and place it initially in the origin. mObject = new Cube(CUBE_SIDE_LENGTH); mObject.setMaterial(material); mObject.setPosition(0, 0, -3); mObject.setRotation(Vector3.Axis.Z, 180); getCurrentScene().addChild(mObject);
51. Project Tango – Getting started AR Scene
52. Project Tango – Getting started AR Scene
53. public class ARActivity extends Activity { @Bind(R.id.gl_main_surface_view) RajawaliSurfaceView mainSurfaceView; protected void onCreate(Bundle savedInstanceState) { pointCloudManager = new TangoPointCloudManager(); renderer = new SceneRenderer(this); mainSurfaceView.setSurfaceRenderer(renderer); ... } Project Tango – Getting started JAVA API
54. Project Tango – Getting started JAVA API @Override protected void onResume() { tango = new Tango(ARActivity.this, new Runnable() { @Override public void run() { try { connectTango(); connectRenderer(); isConnected = true; } catch (TangoOutOfDateException e) { } } });
55. Project Tango – Getting started JAVA API private void connectTango() { TangoConfig config = tango.getConfig(TangoConfig.CONFIG_TYPE_DEFAULT); config.putBoolean( TangoConfig.KEY_BOOLEAN_LOWLATENCYIMUINTEGRATION, true); config.putBoolean(TangoConfig.KEY_BOOLEAN_DEPTH, true); config.putBoolean(TangoConfig.KEY_BOOLEAN_COLORCAMERA, true); tango.connect(config); ...
56. Project Tango – Getting started JAVA API tango.connectListener(framePairs, new OnTangoUpdateListener() { @Override public void onPoseAvailable(TangoPoseData pose) { } @Override public void onFrameAvailable(int cameraId) { if (cameraId == TangoCameraIntrinsics.TANGO_CAMERA_COLOR) { isFrameAvailableTangoThread.set(true); mainSurfaceView.requestRender(); } } @Override public void onXyzIjAvailable(TangoXyzIjData xyzIj) { pointCloudManager.updateXyzIj(xyzIj); }
57. Project Tango – Getting started JAVA API private void connectRenderer() { renderer.getCurrentScene().registerFrameCallback(new ASceneFrameCallback() { @Override public void onPreFrame(long sceneTime, double deltaTime) { ... tango.connectTextureId( TangoCameraIntrinsics.TANGO_CAMERA_COLOR, renderer.getTextureId());
58. Project Tango – Getting started JAVA API if (isFrameAvailableTangoThread.compareAndSet(true, false)) { mRgbTimestampGlThread = tango.updateTexture(TangoCameraIntrinsics.TANGO_CAMERA_COLOR); } TangoPoseData lastFramePose = tango.getPoseAtTime(mRgbTimestampGlThread, FRAME_PAIR); if (lastFramePose.statusCode == TangoPoseData.POSE_VALID) { // Update the camera pose from the renderer renderer.updateRenderCameraPose(lastFramePose, mExtrinsics); mCameraPoseTimestamp = lastFramePose.timestamp; }
59. Project Tango – Getting started JAVA API public class SceneRenderer extends RajawaliRenderer { @Override protected void initScene() { ScreenQuad backgroundQuad = new ScreenQuad(); Material tangoCameraMaterial = new Material(); tangoCameraMaterial.setColorInfluence(0); tangoCameraTexture = new StreamingTexture("camera", (StreamingTexture.ISurfaceListener) null); tangoCameraMaterial.addTexture(mTangoCameraTexture); backgroundQuad.setMaterial(tangoCameraMaterial); getCurrentScene().addChildAt(backgroundQuad, 0);
60. Project Tango – Getting started JAVA API public void updateRenderCameraPose( TangoPoseData devicePose, DeviceExtrinsics extrinsics) { Pose cameraPose = ScenePoseCalculator.toOpenGlCameraPose(devicePose, extrinsics); getCurrentCamera().setRotation(cameraPose.getOrientation()); getCurrentCamera().setPosition(cameraPose.getPosition()); }
61. Augmented Reality with Tango DEMOs
62. Augmented Reality - Conclusion http://mscorpnews.blob.core.windows.net/ncmedia/2015/01/Microsoft-HoloLens-MixedWorld-RGB.png https://news.microsoft.com/microsoft-hololens-press-materials/
63. Augmented Reality - Conclusion https://de.wikipedia.org/wiki/Erweiterte_Realit%C3%A4t http://bitstars.com/glossary/augm ented-reality/ http://www.augmentedplanet.com/2010/04/get-real-redefining-augmented-reality-games/ardronei phone.JPG
64. Context: - exact, realtime location and direction of the mobile device - camera characteristics - 3D model of the virtual objects - 3D model of the real objects - Light model of the real world Project Tango
65. Project Tango – Lessons Learned Lessons learned - Point Cloud calculations are hard - 3D is tricky sometimes - which API to choose? - Motion Tracking works really well - ”closed” APIs make life hard - Environments change more then we recognize - Automated Testing is nearly impossible / debugging is tricky It is a LOT of fun….
66. Augmented Reality - Demos http://mashable.com/2016/02/22/project-tango-indoor-navigation/#xOEcMWsfpZqK
67. Augmented Reality - Demos
68. Augmented Reality - Demos
69. Project Tango Questions?