Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Augmented Reality with Project Tango - Droidcon 2016 Berlin

Augmented Reality with Project Tango - Droidcon 2016 Berlin

Dominik Helleberg

June 15, 2016
Tweet

Other Decks in Technology

Transcript

  1. Augmented Reality - Basics https://en.wikipedia.org/wiki/Augmented_reality Augmented reality (AR) is a

    live direct or indirect view of a physical, real-world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as sound, video, graphics or GPS data. -> It’s important to understand the “context”.
  2. Augmented Reality - Examples By Rama - http://fr.wikipedia.org/wiki/Image:Hud_on_the_cat.jpg, Public Domain,

    https://commons.wikimedia.org/w/index.php?curid=129590 By Sebastian Klein/Wikipedia, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=3608534 HUD Displays Context: - Vehicle (Position, Speed, Engine)
  3. Augmented Reality - Examples http://www.technologybloggers.org/gadgets/google-android-powered-glasses/ Von Tim.Reckmann - Eigenes Werk,

    CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=32750308 Weather Forecast (Google Glass) Context: - Location (coarse) - Time - Weather forecast
  4. Augmented Reality - Examples Von Meximex - Used the app

    and took a picture, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=23501409 Virtual objects Context: - exact, realtime location and direction of the mobile device - camera characteristics - 3D model of the virtual objects - 3D model of the real objects - Light model of the real world
  5. Augmented Reality – Virtual Objects Context: - exact, realtime location

    and direction of the mobile device - camera characteristics - 3D model of the virtual objects - 3D model of the real objects - Light model of the real world
  6. Augmented Reality – Virtual Objects Realtime location / camera chracteristics

    https://www.blender.org/download/demo-files/ -> BMW Demo by Mike Pan
  7. Augmented Reality – VFX Realtime location / camera chracteristics Von

    Meximex - Used the app and took a picture, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=23501409
  8. Augmented Reality – Virtual Objects Realtime location / camera chracteristics

    Von Meximex - Used the app and took a picture, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=23501409
  9. Augmented Reality – Virtual Objects Context: - exact, realtime location

    and direction of the mobile device - camera characteristics - 3D model of the virtual objects - 3D model of the real objects - Light model of the real world
  10. Augmented Reality – Virtual Objects Context: - exact, realtime location

    and direction of the mobile device - camera characteristics - 3D model of the virtual objects - 3D model of the real objects - Light model of the real world
  11. Augmented Reality – Virtual Objects 3D Model of the real

    world Von Meximex - Used the app and took a picture, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=23501409
  12. Augmented Reality – Virtual Objects 3D Model of the real

    world https://news.microsoft.com/microsoft-hololens-press-materials/
  13. Augmented Reality – Virtual Objects 3D Model of the real

    world https://news.microsoft.com/microsoft-hololens-press-materials/
  14. Augmented Reality – Virtual Objects 3D Model of the real

    world https://github.com/stetro/project-tango-poc
  15. Augmented Reality – Virtual Objects Context: - exact, realtime location

    and direction of the mobile device - camera characteristics - 3D model of the virtual objects - 3D model of the real objects - Light model of the real world
  16. Augmented Reality – Virtual Objects Light model Von Meximex -

    Used the app and took a picture, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=23501409
  17. Augmented Reality – Virtual Objects Context: - exact, realtime location

    and direction of the mobile device - camera characteristics - 3D model of the virtual objects - 3D model of the real objects - Light model of the real world
  18. https://www.google.com/atap/proj ect-tango/ http://www.androidheadlines.com/2014/02/googl es-project-tango-phone-renders-full-3d-maps-heres-look-like.html Project Tango 2014 Public announcement of

    “Project Tango” and release of the “peanut” prototype 2014 / 2015 “Yellowstone” Tablet released an sold 3000 times Sept 2016 Phab2 Pro
  19. https://www.google.com/atap/proj ect-tango/ Project Tango • NVIDIA Tegra K1 with 192

    CUDA cores • 7.02” 1920x1200 HD IPS display (323 ppi) • Dual-band Wi-Fi (2.4GHz/5GHz) • Bluetooth 4.0 • 4 GB RAM • Accelerometer • Ambient light • Barometer • Compass • aGPS • Gyroscope
  20. Project Tango – Depth Sensing https://www.google.com/atap/project-tango/about-project-tango/index.html • Tango’s Depth sensor

    detects the range from the device to surfaces within 0.5 – 4 meters • It generates a point cloud which can be used to generate a mesh or enhance AR-Apps DEMO
  21. Project Tango – Area Learning / indoor navigation Context: -

    exact, realtime location and direction of the mobile device - camera characteristics - 3D model of the virtual objects - 3D model of the real objects - Light model of the real world
  22. Project Tango – Area Learning / indoor navigation https://www.google.com/atap/project-tango/about-project-tango/index.html •

    Tango recognizes or “learns” significant “markers” of an areas and exports them into an ADF (“Area Description File”) • A Tango app may load an ADF file and try to match visible markers against the ADF file • After “matching” on an ADF file, Tango can determine it’s position in local ADF coordinates
  23. Project Tango – Getting started JAVA API Rajawali Framework Camera

    Objects / Textures Lights Rajawali Utils Tango Java APIs Java Support Lib Tango Service Sensor Data Application Logic Android APIs https://github.com/Rajaw ali/Rajawali
  24. Project Tango – Getting started JAVA API https://github.com/Rajaw ali/Rajawali DirectionalLight

    light = new DirectionalLight(1, 0.2, -1); light.setColor(1, 1, 1); light.setPower(0.8f); light.setPosition(3, 2, 4); getCurrentScene().addLight(light); // Set-up a material: green with application of the light and // instructions. Material material = new Material(); material.setColor(0xff009900); try { Texture t = new Texture("instructions", R.drawable.instructions); material.addTexture(t); } catch (ATexture.TextureException e) { e.printStackTrace(); } material.setColorInfluence(0.1f); material.enableLighting(true); material.setDiffuseMethod(new DiffuseMethod.Lambert()); // Build a Cube and place it initially in the origin. mObject = new Cube(CUBE_SIDE_LENGTH); mObject.setMaterial(material); mObject.setPosition(0, 0, -3); mObject.setRotation(Vector3.Axis.Z, 180); getCurrentScene().addChild(mObject);
  25. public class ARActivity extends Activity { @Bind(R.id.gl_main_surface_view) RajawaliSurfaceView mainSurfaceView; protected

    void onCreate(Bundle savedInstanceState) { pointCloudManager = new TangoPointCloudManager(); renderer = new SceneRenderer(this); mainSurfaceView.setSurfaceRenderer(renderer); ... } Project Tango – Getting started JAVA API
  26. Project Tango – Getting started JAVA API @Override protected void

    onResume() { tango = new Tango(ARActivity.this, new Runnable() { @Override public void run() { try { connectTango(); connectRenderer(); isConnected = true; } catch (TangoOutOfDateException e) { } } });
  27. Project Tango – Getting started JAVA API private void connectTango()

    { TangoConfig config = tango.getConfig(TangoConfig.CONFIG_TYPE_DEFAULT); config.putBoolean( TangoConfig.KEY_BOOLEAN_LOWLATENCYIMUINTEGRATION, true); config.putBoolean(TangoConfig.KEY_BOOLEAN_DEPTH, true); config.putBoolean(TangoConfig.KEY_BOOLEAN_COLORCAMERA, true); tango.connect(config); ...
  28. Project Tango – Getting started JAVA API tango.connectListener(framePairs, new OnTangoUpdateListener()

    { @Override public void onPoseAvailable(TangoPoseData pose) { } @Override public void onFrameAvailable(int cameraId) { if (cameraId == TangoCameraIntrinsics.TANGO_CAMERA_COLOR) { isFrameAvailableTangoThread.set(true); mainSurfaceView.requestRender(); } } @Override public void onXyzIjAvailable(TangoXyzIjData xyzIj) { pointCloudManager.updateXyzIj(xyzIj); }
  29. Project Tango – Getting started JAVA API private void connectRenderer()

    { renderer.getCurrentScene().registerFrameCallback(new ASceneFrameCallback() { @Override public void onPreFrame(long sceneTime, double deltaTime) { ... tango.connectTextureId( TangoCameraIntrinsics.TANGO_CAMERA_COLOR, renderer.getTextureId());
  30. Project Tango – Getting started JAVA API if (isFrameAvailableTangoThread.compareAndSet(true, false))

    { mRgbTimestampGlThread = tango.updateTexture(TangoCameraIntrinsics.TANGO_CAMERA_COLOR); } TangoPoseData lastFramePose = tango.getPoseAtTime(mRgbTimestampGlThread, FRAME_PAIR); if (lastFramePose.statusCode == TangoPoseData.POSE_VALID) { // Update the camera pose from the renderer renderer.updateRenderCameraPose(lastFramePose, mExtrinsics); mCameraPoseTimestamp = lastFramePose.timestamp; }
  31. Project Tango – Getting started JAVA API public class SceneRenderer

    extends RajawaliRenderer { @Override protected void initScene() { ScreenQuad backgroundQuad = new ScreenQuad(); Material tangoCameraMaterial = new Material(); tangoCameraMaterial.setColorInfluence(0); tangoCameraTexture = new StreamingTexture("camera", (StreamingTexture.ISurfaceListener) null); tangoCameraMaterial.addTexture(mTangoCameraTexture); backgroundQuad.setMaterial(tangoCameraMaterial); getCurrentScene().addChildAt(backgroundQuad, 0);
  32. Project Tango – Getting started JAVA API public void updateRenderCameraPose(

    TangoPoseData devicePose, DeviceExtrinsics extrinsics) { Pose cameraPose = ScenePoseCalculator.toOpenGlCameraPose(devicePose, extrinsics); getCurrentCamera().setRotation(cameraPose.getOrientation()); getCurrentCamera().setPosition(cameraPose.getPosition()); }
  33. Context: - exact, realtime location and direction of the mobile

    device - camera characteristics - 3D model of the virtual objects - 3D model of the real objects - Light model of the real world Project Tango
  34. Project Tango – Lessons Learned Lessons learned - Point Cloud

    calculations are hard - 3D is tricky sometimes - which API to choose? - Motion Tracking works really well - ”closed” APIs make life hard - Environments change more then we recognize - Automated Testing is nearly impossible / debugging is tricky It is a LOT of fun….