First Steps with Intel® RealSense SDK™

Transcript

1. First steps with the Intel® RealSense Camera Xavier Hallade, Developer Evangelist, Intel Corporation
2. 2 **Roadmap Notice: All products, computer systems, dates and figures specified are preliminary based on current expectations, and are subject to change without notice. FRONT Camera REAR Camera Intel® RealSense™ 3D Camera R200 Intel® RealSense™ Snapshot R100 Intel® RealSense™ 3D Camera F200 All-in-One Notebook Convertible Detachable Tablet 3d Camera for every Form Factor
3. USER-FACING Intel RealSense 3D Camera F200 Intel RealSense 3D Camera R200 Intel RealSense Snapshot R100 WORLD-FACING Intel® RealSense™ Cameras Redefining how we interact with our devices
4. **Roadmap Notice: All products, computer systems, dates and figures specified are preliminary based on current expectations, and are subject to change without notice. F200 Front-Facing Depth Camera 3D Face Tracking Gesture Recognition Background Removal Object Recognition and Tracking Front Facing (Laptop) Speech Recognition 3D Scene Understanding User-Facing Capabilities 3D Object Scan/Capture Rear Facing (Tablet) World-Facing Capabilities** RealSense™ SDK Framework Hand Tracking Computational Photography R200 Rear-Facing Depth Camera Augmented Reality Intel® RealSense™ SDK
5. Intel® RealSense™ F200 3D Camera 3D 2D  Platforms : 4th and 5th Generation Intel® Core Processor based Notebooks, 2 in 1 and All in Ones  Usage : Indoors user facing ; 0.2 – 1.2m  OS: Windows* 8.1/10  Technology: Coded Light Depth: VGA 60fps RGB: 1080p 30fps * Other Names and Brands may be claimed as the property of others
6. Learning & Edutainment Immersive Collaboration 3D Capture, Share & Recreate Interact Naturally Front Facing Depth Camera Usages & Experiences Gaming & Play Navigate content and perform simple actions with robust and comfortable gestures or voice Enhance gaming experience with natural language and intuitive gestures, AR and head tracking Blend physical and digital worlds for a more immersive and engaging learning & edutainment experience Scan, share, edit, augment, or 3D print objects and people Mimic real-life interactions in immersive virtual spaces and easily share or create digital content
7. Demos
8. Using the Intel® RealSense SDK™ for Windows* With C++, C#, JavaScript, and frameworks *Other names and brands may be claimed as the property of others.
9. Hands Face Speech Environment 9 Intel® RealSense™ SDK for Windows* Understands 4 basic types of input - Standalone or various permutations Categories of Input Capabilities Features Hands • Hand and Finger Tracking • Gesture Recognition • 22-point Hand and Finger Tracking • Static Poses and Dynamic Gestures Face • Face Detection and Tracking • Multiple Face Detection and tracking • 78-point Landmark Detection (facial features) • Face Recognition and Facial Expressions • Emotion Detection • Pulse Estimator (new in R2) Speech • Speech Recognition and Synthesis • Command and Control • Dictation • Text to Speech Environment • Segmentation • 3D Scanning • Augmented Reality • Background Segmentation • 3D Object / Face / Room Scanning • 2D/3D Object Tracking **Roadmap Notice: All products, computer systems, dates and figures specified are preliminary based on current expectations, and are subject to change without notice. *Other names and brands may be claimed as the property of others.
10. 10 Intel® RealSense™ SDK Gold R3 for Windows* Required Hardware A system with a minimum of a 4th generation Intel® Core™ processor including an Intel® RealSense™ 3D camera (or a peripheral camera) Required OS Microsoft Windows* 8.1/10 64-bit Desktop Mode Supported Programming Languages C++, C#, Java, JavaScript² Supported IDE Microsoft* Visual Studio 2010-2013 with service pack 1 or newer Supported Development Tools -Microsoft* .NET 4.0 Framework for C# development -Unity* PRO 4.1.0 or later for Unity game development -Processing* 2.1.2 or higher for Processing framework development -Java* JDK 1.7.0_11 or higher for Java development - Chromium/IE/Firefox in their latest version for HTML5 development The Intel® RealSense™ SDK for Windows does not support Intel® RealSense™ Snapshot, support for that is coming soon.** **Roadmap Notice: All products, computer systems, dates and figures specified are preliminary based on current expectations, and are subject to change without notice. *Other names and brands may be claimed as the property of others.
11. SDK Installation Folder C:\Program Files (x86)\Intel\RSSDK\ • bin: compiled samples • contrib: metaio* toolbox and nuance* language packs • doc: the documentation • framework: samples and wrappers for C#/JavaScript/Unity/Java/processing • props: VS2013 property sheets • sample: C++ samples
12. 12 12 Web / HTML5 Run Intel® RealSense™ technology in a browser HTML5 / JavaScript SDK Local Web-Socket Server Intel® RealSense™ SDK runtime Operating System Web Socket Communication 127.0.0.1:4181 Internet Browser Trigger face and gesture events, just like in C# applications * Other names and brands may be claimed as the property of others HTML5/JavaScript w/ SDK WebSocket server Unity* Web Player Currently supported in R2: Hand, Blob, Face Tracking, and Speech Recognition.
13. 13 13 Productivity extension for Unity* games Add Intel® RealSense™ technology to your games • A set of scripts that provides configurable actions/rules based on the capabilities provided in the SDK. • Programming: Associate your game objects with the action scripts. • Programming: Create rules. Associate your game objects with rules. * Other names and brands may be claimed as the property of others  Drag and drop programming  Write less code!
14. 14 Unity Toolkit Concepts The toolkit consist of three main concepts • Actions – MonoBehavior scripts. Actions define certain behavior on game objects. • Triggers – Define a family of interactions that can be implemented in several ways such as a Tracking Trigger. This trigger defines all interactions that track a certain object in the real world such as a hand or face. Each interaction is implemented as a Rule. Thus each trigger has it’s own set of rules. • Rules – Are specific SDK code that implements certain Unity events, actions and data types. You can use rules in one or multiple triggers. When a rule is part of a trigger, the rule and it’s configuration parameters are shown in the Unity Editor’s UI under the specific trigger(s). When attaching more than one rule to a trigger, the rules are used in an OR relationship. This means that if one of the rules fires, the trigger is triggered with the firing rule’s value. The order of which they display in the Unity Editor’s UI is the order of priority.
15. Using the PXC[M]SenseManager
16. 17 Using the PXC[M]SenseManager PXCSenseManager *psm = PXCSenseManager::CreateInstance(); psm->EnableXXX(); //XXX can be: 3DScan, 3DSeg, Emotion, Face, Hand, Stream, Tracker… not Speech PXCXXXModule xxxAnalyzer = psm->QueryXXX(); //for Face and Hand: PXCXXXData xxxData = xxxAnalyzer->CreateOutput(); PXCXXXConfiguration *xxxConfig = xxxAnalyzer->CreateActiveConfiguration(); xxxConfig->Query/Set/Disable/Enable/SubscribeYYY(…); xxxConfig->ApplyChanges(); psm->Init(); //if using callbacks set with xxxConfig->SubscribeYYY() psm->StreamFrames(/*blocking=*/true); //non-blocking will create a thread. //else, in update loop: psm->AcquireFrame(/*all=*/false,/*blocking=*/true); xxxAnalyzer->QueryYYY(); // for Face and Hand: xxxData->Update(); xxxData->QueryYYY(); xxxIsAlertFired(); psm->ReleaseFrame(); //closing: xxxConfig->Release(); psm->Release();
17. Coordinate System x y z x y World coordinates Image coordinates
18. EnableFace()
19. 20 What does the “Face” modality offer? • Best for sensing natural expressions, emotions and engagement • SDK provides APIs for: • Face Detection • Head Orientation • Expressions Tracking • Landmark Tracking • Emotion Recognition • Pulse Estimation (new in R2)
20. 21 • Accurate 3D detection and tracking • Up to 4 faces with marked rectangles for face boundaries • Developers can choose which 4 faces to detect • Only one face has landmarks • Works up to 1.2 meters Face Detection
21. 22 Expression EXPRESSION_HEAD_(TILT_LEFT|TURN_LEFT|UP|…) EXPRESSION_(SMILE|KISS|MOUTH_OPEN|TONGUE_OUT) EXPRESSION_EYES_(TURN_LEFT|UP|CLOSED_LEFT|…) EXPRESSION_BROW_(RAISER_LEFT|LOWERER_LEFT|…) PXCFaceData::Face *trackedFace = outputData->QueryFaceByIndex(i); PXCFaceData::ExpressionsData *expressionData = trackedFace->QueryExpressions(); if(expressionData!=NULL) { if (expressionData->QueryExpression(PXCFaceData::ExpressionsData::EXPRESSION_MOUTH_OPEN, &expressionResult)){ //use expressionResult.intensity } }
22. 23 • Detect how the user’s head is oriented Poses PXCFaceData::Face *trackedFace = outputData->QueryFaceByIndex(i); PXCFaceData::PoseData *poseData = trackedFace->QueryPose(); PXCFaceData::PoseEulerAngles angles; if(poseData!=NULL && poseData->QueryPoseAngles(&angles)){ // or QueryPoseQuaternion //uses angles.yaw, angles.pitch, angles.roll }
23. 24 • 3D tracking of 78 facial landmark points • Available in image and world coordinates • Supports avatar creation, emotion recognition and facial animation • Access to a subset of landmark regions Landmark tracking
24. 25 Retrieving Face Landmark Data (C#) pxcU16 numOfFaces = outputData->QueryNumberOfDetectedFaces(); for (pxcU16 i = 0; i < numOfFaces; i++) { PXCFaceData::Face *trackedFace = outputData->QueryFaceByIndex(i); PXCFaceData::LandmarksData* landmarkData = trackedFace->QueryLandmarks(); if (landmarkData != NULL) { pxcI32 numPoints = landmarkData->QueryNumPoints(); PXCFaceData::LandmarkPoint* landmarkPoints = new PXCFaceData::LandmarkPoint[numPoints]; landmarkData->QueryPoints(landmarkPoints); break; } }
25. EnableEmotion()
26. 27 • Supports 6 primary emotions – Anger, Disgust, Fear, Joy, Sadness, Surprise • Support 3 sentiments: Postive, Neutral, Negative • Returns intensity (0 to 1.0) and evidence (on a log scale) Emotion Detection int numFaces = ft.QueryNumFaces(); for (int i = 0; i < numFaces; i++) { PXCMEmotion.EmotionData data; if (ft.QueryEmotionData(i, PXCMEmotion.Emotion.EMOTION_PRIMARY_JOY, out data) >= pxcmStatus.PXCM_STATUS_NO_ERROR) { //use data.evidence and data.intensity } }
27. Demos
28. EnableHand()
29. 30 Hand Tracking Modes TRACKING_MODE_FULL_HAND VGA or HVGA (faster tracking) • closest • left most • right most • top most • bottom most • center TRACKING_MODE_EXTREMITIES Other information: palm orientation, hand openness, mass center…
30. Retrieving closest extremity Data (C++) PXCHandData::IHand* handData; PXCHandData::ExtremityData extremityData; pxcI32 numOfHands = outputData->QueryNumberOfHands(); for (pxcI32 i = 0; i < numOfHands; i++) { if (outputData->QueryHandData(PXCHandData::ACCESS_ORDER_BY_TIME, i, handData) == PXC_STATUS_NO_ERROR) { handData->QueryExtremityPoint(PXCHandData::ExtremityType::EXTREMITY_CLOSEST, extremityData); } }
31. 32 Gestures spreadfingers fist tap thumb_down thumb_up two_fingers_pinch_open v_sign full_pinch swipe_* click wave GestureData.state: GESTURE_STATE_START GESTURE_STATE_IN_PROGRESS GESTURE_STATE_END Retrieve: IsGestureFired(name, GestureData) OnFiredGesture(GestureData) Enable: PXCHandConfiguration:: EnableGesture(const pxcCHAR *name, pxcBool continuous);
32. Demos
33. Enable<Other features>()
34. Using other features from the SDK • Use PXC[M]SenseManager the same way for 3d segmentation, scan, etc. • PXCSpeechRecognition/Synthesis are used without PXCSenseManager • Look at all the samples and the documentation !
35. Speech Modality
36. Speech Modality • Speech Recognition • Grammar • List • JSGF • Dictation • Speech Synthesis
37. Recognizing pre-defined words (C#) PXCMAudioSource source = session.CreateAudioSource(); source.SetVolume(0.2f); PXCMAudioSource.DeviceInfo dinfo; source.QueryDeviceInfo(out dinfo); source.SetDevice(dinfo); PXCMSpeechRecognition sr; session.CreateImpl<PXCMSpeechRecognition>(out sr); string[] cmds = { "one", "two", "three", "stop" }; sr.BuildGrammarFromStringList(1, cmds, null); sr.SetGrammar(1); PXCMSpeechRecognition.Handler handler = new PXCMSpeechRecognition.Handler(); handler.onRecognition = OnRecognition; sr.StartRec(source, handler); while (run) {System.Threading.Thread.Sleep(500);} sr.StopRec(); sr.Dispose(); source.Dispose();
38. PXC[M]Smoother
39. PXC[M]Smoother •Stabilizer •Weighted •Quadratic* •Spring*
40. Using PXCMSmoother (C#) // Creating PXCMSmoother instance PXCMSmoother smootherFactory; senseManager.session.CreateImpl<PXCMSmoother>(out smootherFactory); // Creating 3D smoother with quadratic algorithm with smooth value of 0.8 PXCMSmoother.Smoother3D smoother3D = smootherFactory.Create3DQuadratic(0.8f); //... //... // Smoothing the value of the joint world coordinates, output to smoothed3DPoint PXCMPoint3DF32 smoothed3DPoint = smoother3D.SmoothValue(jointData.positionWorld); // Use smoothed3DPoint.x/y/z values //...
41. 42 Get Started Now!  Reserve Your Developer Kit now (includes peripheral camera, the latest SDK, and other software updates) Provided by Intel  Free Intel® RealSense™ SDK for Windows, (Version 2014, Gold)  Intel® RealSense™ Developer Kit camera available for purchase  Huge opportunity to reach customers with integrated 3D camera  Works with languages/frameworks/engines developers already use  High-level APIs for NUI beginners. Low-level APIs for NUI experts intel.com/realsense/sdk Seize the Opportunity
42. INFORMATION IN THIS DOCUMENT IS PROVIDED “AS IS”. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO THIS INFORMATION INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. Copyright © 2014 , Intel Corporation. All rights reserved. Intel, the Intel logo, Xeon, Core, VTune, Cilk, and RealSense are trademarks of Intel Corporation in the U.S. and other countries. Optimization Notice Intel’s compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Notice revision #20110804 Legal Disclaimer & Optimization Notice Copyright© 2014, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners.
43. Thank you