Upgrade to Pro — share decks privately, control downloads, hide ads and more …

WayFinder - Indoor Navigation with ARCore

Ryan Hodgman
October 27, 2018

WayFinder - Indoor Navigation with ARCore

In a world where Google Maps has made strangers asking for directions on the street an oddity rather than the norm, somehow we still turn to passersby for assistance with finding our way indoors. Our team tackled this pain point in an experiment with image recognition and ARCore, and accumulated a bunch of augmented reality learnings along the way. In this talk we share our experience with building a navigation solution atop Google’s AR framework.

Presentation by:
Zhenya Li - https://www.linkedin.com/in/zhenya-li-95280b104/
Ryan Hodgman - https://www.linkedin.com/in/ryanhodgman/

Ryan Hodgman

October 27, 2018
Tweet

Other Decks in Programming

Transcript

  1. How did we design it? What was challenging? How did

    we build it? What did we build?
  2. How did we design it? What was challenging? How did

    we build it? What did we build?
  3. Anchor image Realtime Database & File Storage Physical environment AR

    positioning & image capture Recording Flow Camera capture
  4. Realtime Database & File Storage Image detection Detection Flow Anchor

    image Relocalise AR world Image target Localisation algorithm Detected pose AR scene
  5. How did we design it? What was challenging? How did

    we build it? What did we build?
  6. How did we design it? What was challenging? How did

    we build it? What did we build?
  7. val modelViewMatrix = Tool.convertPose2GLMatrix(trackable.pose).data // Deal with the model view

    and projection matrices val modelViewProjection = FloatArray(16) Matrix.translateM(modelViewMatrix, 0, render.translationX, render.translationY, render.translationZ) Matrix.scaleM(modelViewMatrix, 0, render.scale, render.scale, render.scale) Matrix.multiplyMM(modelViewProjection, 0, projectionMatrix, 0, modelViewMatrix, 0) // Activate the shader program and bind the vertex/normal/tex coords GLES20.glUseProgram(shaderProgramID) GLES20.glDisable(GLES20.GL_CULL_FACE) GLES20.glVertexAttribPointer(vertexHandle, 3, GLES20.GL_FLOAT, false, 0, render.model.vertices) GLES20.glEnableVertexAttribArray(vertexHandle) // Did we find any trackables this frame? (0 until state.numTrackableResults).forEach { trackableIndex -> val trackable = state.getTrackableResult(trackableIndex) TargetRenderLoader.loadedRenders[trackable.trackable.name]?.let { render -> onTargetDetectedListener(trackable.trackable.name) if (render.model.texCoords != null) { GLES20.glVertexAttribPointer(textureCoordHandle, 2, GLES20.GL_FLOAT, false, 0, render.model.texCoords) GLES20.glEnableVertexAttribArray(textureCoordHandle) GLES20.glActiveTexture(GLES20.GL_TEXTURE0)
  8. val posterObject = Object3D() posterObject.loadModel(viroContext, modelUri, Object3D.Type.OBJ) { override fun

    onObject3DLoaded(_: Object3D, _: Object3D.Type) { val texture = BitmapFactory .decodeStream(context.assets.open(textureFileUri)) val material = Material().apply { diffuseTexture = Texture(texture, Texture.Format.RGBA8, true, true) } posterObject.geometry.materials = listOf(material) } }) anchorNode.addChildNode(posterObject)
  9. ModelRenderable.builder() .setSource(context, Uri.parse(modelUri)) .build() .thenAccept { posterModel -> val posterNode

    = Node() posterNode.setRenderable(posterModel) anchorNode.addChild(posterNode) }
  10. How did we design it? What was challenging? How did

    we build it? What did we build?
  11. How did we design it? What was challenging? How did

    we build it? What did we build?
  12. How did we design it? What was challenging? How did

    we build it? What did we build?
  13. How did we design it? What was challenging? How did

    we build it? What did we build?
  14. ...create a stable environment? ...indicate success? ...highlight offscreen content? ...maintain

    immersion? ...prevent fatigue? ...guide content placement? How do we...
  15. ...create a stable environment? ...indicate success? ...highlight offscreen content? ...maintain

    immersion? ...prevent fatigue? ...guide content placement? ...reposition content? How do we...
  16. ...create a stable environment? ...indicate success? ...highlight offscreen content? ...maintain

    immersion? ...prevent fatigue? ...guide content placement? ...reposition content? ...switch modes of interaction? How do we...
  17. ...create a stable environment? ...indicate success? ...highlight offscreen content? ...maintain

    immersion? ...prevent fatigue? ...guide content placement? ...reposition content? ...convey interaction modes? How do we...
  18. How did we design it? What was challenging? How did

    we build it? What did we build?
  19. How did we design it? What was challenging? How did

    we build it? What did we build?
  20. Illustration Nate Cooper Jack Walsh Talk Feedback Flora Wang Ankitha

    Sheeba Konrad Biernacki Amir Abdi With enormous thanks to... App Development & Design Alex Chiviliov Alex Lai Yasitha Chinthaka Amir Abdi Julius Canute Matthew Falzon Sucharitha Alli Ibramsha Sirajudeen Bindiya D’Souza Alex Waluyo Aurenia Permadi Michelle Ritoli Wiliane Chua Miffra Lee Joshua Kenzie Tony Archibald Jack Mauleekoonphairoj Yasmeen Vorha