Meta Reality

Introduction to augmented reality and the meta.io SDK given at 360iDev

Transcript

1. META REALITY iOS augmented Reality and the metaio SDK Matthew Henderson @ghostm 1 Thursday, September 13, 12
2. WHAT WE’LL TALK ABOUT • Who am I and, more importantly, what I’m not. • What is augmented reality(AR)? • What is the state of AR on iOS? • How do you do AR on iOS? • Using the metaio SDK 2 Thursday, September 13, 12
3. HI, I’M MATTHEW HENDERSON • I work for Empirical Development • I’ve been writing iOS software full time for over two and a half years • I’ve helped develop several AR apps • In my free time I write software for my own company, Haunted Robot 3 Thursday, September 13, 12
4. WHAT I’M NOT • An augmented reality expert • An OpenGL expert • A computer vision specialist 4 Thursday, September 13, 12
5. WHAT’S THE DEAL WITH AUGMENTED REALITY? 5 Thursday, September 13, 12
6. AUGMENTED REALITY • Augmented reality (AR) is a live, direct or indirect, view of a physical, real-world environment whose elements are augmented by computer-generated sensory input such as sound, video, graphics or GPS data. 6 Thursday, September 13, 12
7. AUGMENTED REALITY • AR = view of a real-world environment + something 7 Thursday, September 13, 12
8. 8 Thursday, September 13, 12
9. 9 Thursday, September 13, 12
10. 10 Thursday, September 13, 12
11. 11 Thursday, September 13, 12
12. 12 Thursday, September 13, 12
13. 13 Thursday, September 13, 12
14. VIEW OF A REAL-WORLD ENVIRONMENT? 14 Thursday, September 13, 12
15. SENSORS? 15 Thursday, September 13, 12
16. • View of a real-world environment • Cameras • Sensors • Camera input • Gyroscope • Accelerometer • GPS • Compass • Touch Screen 16 Thursday, September 13, 12
17. • View of a real-world environment • Cameras - AVFoundation.framework • Sensors • Camera input - AVFoundation.framework • Gyroscope - CoreMotion.framework • Accelerometer - CoreMotion.framework • GPS - CoreLocation.framework • Compass - CoreLocation.framework • Touch Screen - UIKit.framework 17 Thursday, September 13, 12
18. 18 Thursday, September 13, 12
19. (of what you just saw in action) LET’S SEE SOME CODE 19 Thursday, September 13, 12
20. AVCaptureSession *session = [[AVCaptureSession alloc] init]; session.sessionPreset = AVCaptureSessionPresetMedium; CALayer *viewLayer = self.vImagePreview.layer; AVCaptureVideoPreviewLayer *captureVideoPreviewLayer = [[AVCaptureVideoPreviewLayer alloc] initWithSession:session]; captureVideoPreviewLayer.frame = self.vImagePreview.bounds; [self.vImagePreview.layer addSublayer:captureVideoPreviewLayer]; AVCaptureDevice *device = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo]; NSError *error = nil; AVCaptureDeviceInput *input = [AVCaptureDeviceInput deviceInputWithDevice:device error:&error]; if (!input) { // Handle the error appropriately. } [session addInput:input]; dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{ [session startRunning]; }); Camera Input 20 Thursday, September 13, 12
21. UIAccelerometer* theAccelerometer = [UIAccelerometer sharedAccelerometer]; theAccelerometer.updateInterval = 1 / 50.0; - (void)accelerometer:(UIAccelerometer *)accelerometer didAccelerate:(UIAcceleration *)acceleration { self.accelerometerLabel.text = [NSString stringWithFormat:@"Acceleration: X=%.2f Y=%.2f Z=%.2f", acceleration.x, acceleration.y, acceleration.z]; } Accelerometer 21 Thursday, September 13, 12
22. self.locationManager = [[CLLocationManager alloc] init]; self.locationManager.desiredAccuracy = kCLLocationAccuracyBest; // setup delegate callbacks self.locationManager.delegate = self; // Start location services to get the true heading. self.locationManager.distanceFilter = 1000; self.locationManager.desiredAccuracy = kCLLocationAccuracyKilometer; [self.locationManager startUpdatingLocation]; - (void)locationManager:(CLLocationManager *)manager didUpdateToLocation:(CLLocation *)newLocation fromLocation:(CLLocation *)oldLocation { ! if (newLocation.horizontalAccuracy <= 0) { return; } ! self.latitudeLabel.text= [NSString stringWithFormat:@"%.6f", newLocation.coordinate.latitude]; ! self.longitudeLabel.text = [NSString stringWithFormat:@"%.6f", newLocation.coordinate.longitude]; ! self.altitudeLabel.text = [NSString stringWithFormat:@"%.6f", newLocation.altitude]; } GPS Location 22 Thursday, September 13, 12
23. // heading service configuration self.locationManager.headingFilter =0.5; if ([CLLocationManager headingAvailable] == NO) { // No compass is available. } else { // start the compass [self.locationManager startUpdatingHeading]; } - (void)locationManager:(CLLocationManager *)manager didUpdateHeading:(CLHeading *)newHeading { if (newHeading.headingAccuracy < 0){ return; } // Use the true heading if it is valid. CLLocationDirection currentHeading = ((newHeading.trueHeading > 0) ? newHeading.trueHeading : newHeading.magneticHeading); self.headingLabel.text = [NSString stringWithFormat:@"%.6f", currentHeading]; } Compass Heading 23 Thursday, September 13, 12
24. // Set up motionManager motionManager = [[CMMotionManager alloc] init]; motionManager.deviceMotionUpdateInterval = 1.0/60.0; opQ = [NSOperationQueue currentQueue]; if(motionManager.isDeviceMotionAvailable) { // Listen to events from the motionManager motionHandler = ^ (CMDeviceMotion *motion, NSError *error) { CMAttitude *currentAttitude = motion.attitude; self.attitudeLabel.text = [NSString stringWithFormat:@"Pitch=%.2f Roll=%.2f Yaw=%.2f", currentAttitude.pitch, currentAttitude.roll, currentAttitude.yaw]; CMRotationRate currentRotation = motion.rotationRate; self.rotationLabel.text = [NSString stringWithFormat:@"Rotation: X=%.1f Y=%.1f Z=%.1f", currentRotation.x, currentRotation.y, currentRotation.z]; }; [motionManager startDeviceMotionUpdatesToQueue:opQ withHandler:motionHandler]; } Gyroscope 24 Thursday, September 13, 12
25. THINGS TO WATCH OUT FOR • Accuracy. • Especially the compass • http://www.sundh.com/blog/2011/09/stabalize-compass-of- iphone-with-gyroscope/ • Probably a good idea to use a filter. • Devices capabilities • Can’t assume the device has all the hardware you want to use 25 Thursday, September 13, 12
26. METAIO http://www.metaio.com/products/mobile-sdk/ 26 Thursday, September 13, 12
27. METAIO • Tracking - Picture marker, markerless, markerless 3D, QR Codes • 3D Rendering engine included • Plugin connection to the Unity3D rendering engine • Location-based POI visualization engine • And more... 27 Thursday, September 13, 12
28. 28 Thursday, September 13, 12
29. METAIO 29 Thursday, September 13, 12
30. ON MY FLIGHT BACK FROM WWDC 30 Thursday, September 13, 12
31. 31 Thursday, September 13, 12
32. TURNS THIS 32 Thursday, September 13, 12
33. INTO THIS 33 Thursday, September 13, 12
34. WHAT WE NEED TO DUPLICATE THIS • Pattern: The tracking pattern/image to look for. • TrackingData: The tracking data configures the tracking component of the MobileSDK (XML file). • Model: The 3D model/geometry to be shown. • Texture: We also need a texture to make the model look better. 34 Thursday, September 13, 12
35. MARKERLESS IMAGE TRACKING • What makes a good tracking image? • Use the digital version of the target and if needed, take the reference image with the device camera that is going to be used • Select an almost square region • Select a region which is a little bit smaller than the pre-print version 35 Thursday, September 13, 12
36. EXAMPLE TRACKING IMAGE 36 Thursday, September 13, 12
37. MARKERLESS IMAGE TRACKING • Pick a good image. • Follow the directions. • Test it. • Now turn off the lights and test it with an iPod Touch. • Change your image or your configuration. 37 Thursday, September 13, 12
38. • Check your image • Not enough detail? • Subsection? • Check your configuration • Fast vs Robust • Modify the SmoothingFuser • AlphaRotation • KeepPoseForNumberOfFrames TRACKING TROUBLE? 38 Thursday, September 13, 12
39. 3D MODEL • Supported 3D formats are OBJ and MD2 (Quake 2) 39 Thursday, September 13, 12
40. TEXTURE FILE • A PNG image with the same name as the model 40 Thursday, September 13, 12
41. •UnifeyeMobileViewController •Base view controller for any metaio views •Start from the examples included with the SDK 41 Thursday, September 13, 12
42. @interface UnifeyeMobileViewController : UIViewController <UnifeyeMobileDelegate> {! metaio::IUnifeyeMobileIPhone*! unifeyeMobile;! ! ! //!< Reference to unifeye metaio::ISensorsComponent*! ! m_sensors;! ! ! // OpenGL related members (directly taken from XCode4 OpenGL example) EAGLContext *context; // our OpenGL Context BOOL animating; // are we currently animating NSInteger animationFrameInterval; // refresh interval CADisplayLink *displayLink; // pointer to our displayLink EAGLView *glView; // our OpenGL View } UnifeyeMobileViewController 42 Thursday, September 13, 12
43. // in viewDidLoad // create unifeye instance unifeyeMobile = metaio::CreateUnifeyeMobileIPhone(UNIFEYE_LICENSE); if( !unifeyeMobile ) { NSLog(@"Unifeye instance could not be created."); return; } m_sensors = metaio::CreateSensorsComponent(); if( !m_sensors ) { NSLog(@"Could not create the sensors interface."); return; } unifeyeMobile->registerSensorsComponent( m_sensors ); // register our callback method for animations unifeyeMobile->registerDelegate(self); UnifeyeMobileViewController 43 Thursday, September 13, 12
44. Was that code we just saw Obj-C++? 44 Thursday, September 13, 12
45. •UnifeyeMobileImageTrackingViewController •Subclass of UnifeyeMobileViewController 45 Thursday, September 13, 12
46. @interface UnifeyeMobileImageTrackingViewController : UnifeyeMobileViewController { ! metaio::IUnifeyeMobileGeometry* m_truckModel; metaio::IUnifeyeMobileGeometry* m_metaioMan; metaio::IUnifeyeMobileGeometry* m_moviePlane; metaio::IUnifeyeMobileGeometry* m_moviePlaneButton; } UnifeyeMobileTrackingViewController 46 Thursday, September 13, 12
47. // load our tracking configuration NSString* trackingDataFile = [[NSBundle mainBundle] pathForResource:@"TrackingData_MarkerlessFast" ofType:@"xml" inDirectory:@"Assets"]; ! if(trackingDataFile) ! { ! ! bool success = unifeyeMobile->setTrackingData([trackingDataFile UTF8String]); ! ! if( !success) ! ! ! NSLog(@"No success loading the tracking configuration"); ! } Set Up For Image Tracking 47 Thursday, September 13, 12
48. // load content NSString* metaioManModel = [[NSBundle mainBundle] pathForResource:@"metaioman" ofType:@"md2" inDirectory:@"Assets"]; ! if(metaioManModel) ! { m_metaioMan = unifeyeMobile->loadGeometry([metaioManModel UTF8String]);! ! ! if( m_metaioMan ) { // scale it a bit down m_metaioMan->setMoveScale(metaio::Vector3d(0.8,0.8,0.8)); } else { NSLog(@"error, could not load %@", metaioManModel); } } Add A Model 48 Thursday, September 13, 12
49. (www.reflectionapp.com) 49 Thursday, September 13, 12
50. ADVANCED TRACKING • 3D Pose (metaio::Pose currentPose = unifeyeMobile->getTrackingValues(1); ) • Quality of the tracked image - is it currently being tracked? • Current translation • Current rotation virtual metaio::IUnifeyeMobileGeometry* getGeometryFromScreenCoordinates( int x, int y, bool useTriangleTest = false ) = 0; 50 Thursday, September 13, 12
51. WORKING WITH YOUR MODEL m_metaioMan->setMoveScale(metaio::Vector3d(0.5,0.5,0.5)); m_metaioMan->setMoveTranslation(metaio::Vector3d(0,-55,0)); m_metaioMan->setMoveRotation(metaio::Vector3d(M_PI_2, 0, 0)); // increment rotation m_metaioMan->setMoveRotation(metaio::Vector3d(0, M_PI_2, 0), true); m_metaioMan->setVisible(true); m_metaioMan->startAnimation( "shock_down" , false); 51 Thursday, September 13, 12
52. WORKING WITH YOUR MODEL • Misfit Model 3D http://www.misfitcode.com/misfitmodel3d/ • Best application I found for viewing md2 files • Allows you to also inspect animations • Windows only :( 52 Thursday, September 13, 12
53. 53 Thursday, September 13, 12
54. PROBLEMS? • Tracking • iPod Touch in low light is a lot different than an iPhone 4S in good light • Models • Misfit Model 3D was a life saver when working with other designers 54 Thursday, September 13, 12
55. PROBLEMS? • Memory • Loaded models aren’t cheap • Transitions aren’t free 55 Thursday, September 13, 12
56. HOW I GOT A TRANSITION TO NOT CRASH -(void) tearDown{ if( m_model1 ){ m_model1->setVisible(0); unifeyeMobile->unloadGeometry(m_model1); m_model1 = NULL; } [self stopAnimation];! if( unifeyeMobile ) { unifeyeMobile->stopCamera(); } // Tear down context. if ([EAGLContext currentContext] == context) [EAGLContext setCurrentContext:nil]; // delete unifeye instance if( unifeyeMobile ) { delete unifeyeMobile; unifeyeMobile = NULL; } // delete our sensors component if( m_sensors ) { delete m_sensors; m_sensors = NULL; } } 56 Thursday, September 13, 12
57. WHERE TO GO FOR HELP • The metaio forum http://forum.metaio.com/viewforum.php?f=9 • Good chance that someone has run into your problem already. • The developers actually read the forum. • I ran into a bug which was fixed in the next update 57 Thursday, September 13, 12
58. • Thank John & Nicole for putting together such a great conference. • If you have any questions or comments you can reach me on Twitter @ghostm • Go make something awesome, and then tell me about it! THANKS 58 Thursday, September 13, 12