Intel® Perceptual Computing SDK

Презентация выступления Артема Воробьева в рамках образовательной программы HackDay#25 в Нижнем Новгороде

Transcript

1. Intel® Perceptual Computing SDK HackDay, Nizhny Novgorod, 12-14 of April
2. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. Agenda • SDK design and architecture overview • SDK Application Development • Examples
3. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. SDK Very High Level Design Drivers Module Usage Model Coordination Power Management Hardware Acceleration Privacy Notification I/O Modules Object Trackin g Applications Module Implementation Interfaces Close- Range Trackin g Face Analysi s Voice Recogniti on Module Module Module 2nd and 3rd Generation Intel® Core™ Processors
4. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. Modules overview • Software API for capturing audio and video • Close-range finger tracking and gesture recognition • Face detection, landmark detection • Voice recognition and voice synthesis • Object tracking
5. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. Game Engines and 3rd party Frameworks support • Unity* Pro v4.0.1 • Processing* v2.0.8 • openFrameworks* v0071
6. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. Privacy Notification Human perception is different when camera is constantly on. Privacy notification when SDK applications access to PII. • Personal Identifiable Information. Raise the bar so that users can be comfort.
7. SDK Application Development
8. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. SDK API Hierarchy C++ PXCSession, PXCImage, PXCAudio, PXCCapture, PXCGesture, PXCFaceAnalysis, PXCVoice C++ UtilCapture, UtilPipeline C# PXCMSession PXCMImage PXCMAudio PXCMCapture PXCMGesture PXCMFaceAnalysis PXCMVoice UtilMCapture UtilMPipeline libpxcupipeline Unity* Pro Processin g* openFramew orks* Applications Module Functionalities Module Interaction Additional Language and Framework Support
9. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. HelloWorld C++ C# class MyPipeline: public UtilPipeline { public: MyPipeline(void):UtilPipeline() { EnableGesture(); } virtual void PXCAPI OnGesture (PXCGesture::Gesture *data) { printf_s(“%d\n”,data->label); } }; int wmain(int argc, WCHAR* argv[]) { MyPipeline pipeline; pipeline.LoopFrames(); return 0; } class MyPipeline : UtilMPipeline { public MyPipeline() : base() { EnableGesture(); } public override void OnGesture (ref PXCMGesture.Gesture data) { Console.WriteLn(data.label); } }; class Program { static void Main(string[] args) { MyPipeline pipeline = new MyPipeline(); pipeline.LoopFrames(); pipeline.Dispose(); } } Enable Finger Tracking Gesture Callback Data Flow Loop
10. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. UtilPipeline-based Application Multiple processing modules receive data from single input device • Live streaming or file-based recording/playback • Synchronized image (or audio) processing UtilPipeline pp; pp.EnableImage(PXCImage::COLOR_FORMAT_RGB32); pp.EnableImage(PXCImage::COLOR_FORMAT_DEPTH); for (;;) { if (!pp.AcquireFrame(true)) break; PXCImage *color, *depth; color=pp.QueryImage(PXCImage::IMAGE_TYPE_COLOR); depth=pp.QueryImage(PXCImage::IMAGE_TYPE_DEPTH); pp.ReleaseFrame(); } pp.Close(); Configuration Queries Event Close up For each frame
11. Intel® Perceptual Computing SDK in Gaming Application: Examples
12. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. Face Tracking Pipeline Example void OgreGDC2::createScene(void) { // … <Meshes loading and other stuff // initialization goes here> mSession = PXCUPipeline_Create(); // Creating // pipeline PXCUPipeline_Init(mSession, (PXCUPipeline)PXCU_PIPELINE_FACE_LOCATION); PXCUPipeline_QueryRGBSize(mSession,&mCW,&mCH); } void OgreGDC2::destroyScene(void) { PXCUPipeline_Close(mSession); } bool OgreGDC2::frameStarted(const Ogre::FrameEvent& evt) { if(PXCUPipeline_AcquireFrame(mSession,false)) { pxcUID f; pxcU64 ts; PXCFaceAnalysis::Detection::Data fData; if(PXCUPipeline_QueryFaceID(mSession,0,&f,&ts)) { if(PXCUPipeline_QueryFaceLocationData(mSession,f,&fData)) { //Face Center in image float fcx = (fData.rectangle.x * 2.0f + fData.rectangle.w)*0.5f; float fcy = (fData.rectangle.y * 2.0f + fData.rectangle.h)*0.5f; mFaceCenter.x = 2.0f*(fcx/mCW)-1.0f; mFaceCenter.y = 2.0f*(fcy/mCH)-1.0f; } } PXCUPipeline_ReleaseFrame(mSession); } mCamera->setPosition(-mFaceCenter.x*100.0f, 0,80); mCamera->lookAt(headNode->getPosition()); return true; } Initialization Close up Processing frame
13. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. Gesture Recognition Pipeline Example void OgreGDC3::createScene(void) { // … <Meshes loading and other stuff // initialization goes here> mSession = PXCUPipeline_Create(); PXCUPipeline_Init(mSession, (PXCUPipeline)PXCU_PIPELINE_GESTURE); } void OgreGDC3::destroyScene(void) { PXCUPipeline_Close(mSession); } bool OgreGDC3::frameStarted(const Ogre::FrameEvent& evt) { if(PXCUPipeline_AcquireFrame(mSession, false)) { if(PXCUPipeline_QueryGesture(mSession, PXCGesture::GeoNode::LABEL_BODY_HAND_PRIMARY, &mGesture)) { if(mGesture.active) { switch(mGesture.label) { case PXCGesture::Gesture::LABEL_POSE_THUMB_UP: { mLightUp->setDiffuseColour( Ogre::ColourValue(1.0f,0,0)); mLightDown->setDiffuseColour( Ogre::ColourValue(0,0,0)); break; } // Other gestures handling … } } } PXCUPipeline_ReleaseFrame(mSession); } return true; } Initialization Processing frame Close up
14. Q&A
15. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. Backup
16. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. Applications with Customized Pipeline For ALL functionalities: UtilCapture to facilitate “Match I/O and Modules” • Retrieve data needs from modules • Consolidate data needs • Locate I/O that can stream the data • Configure the I/O device M1 M3 M2 I/O I/O Initialize context Close up Queries Event For each frame, from I/O to modules stream data Create I/O and modules Match I/O and modules Configure I/O modules UtilCapture
17. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. PXCSession: Module Management Multiple modules (implementations) exist in a session • During session creation, the SDK auto-loads platform modules – Preinstalled and listed under registry key HKEY_LOCAL_MACHINE/Software/Intel/PCSDK/Dispatch • At runtime, LoadModuleFromFile to load dynamic modules – UnloadModuleFromFile to unload dynamic modules Locate modules & create instances • QueryImpl to search for a module by template • CreateImpl to create module instance by template Create essential session objects • PXCScheduler: Manage asynchronous execution • PXCAccelerator: Manage context for audio/image objects
18. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. Query & Create Module Instances Module descriptor struct ImplDesc { ImplGroup group;/* implementation group */ ImplSubgroup subgroup;/* implementation sub-group */ pxcUID algorithm;/* algorithm identifier */ pxcUID iuid;/* module unique id */ ImplVersion version;/* version */ PXCAccelerator::AccelType acceleration;/* supported accelerations */ pxcU32 merit;/* merit */ pxcU32 vendor;/* vendor */ pxcUID cuids[4];/* first four interface identifiers */ pxcCHAR friendlyName[128];/* friendly name */ pxcU32 reserved[12]; }; QueryImpl & CreateImpl use descriptor as template • QueryImpl(template, index, [out] descriptor); • CreateImpl(scheduler, accelerator, template, [out] instance); Tip: Searching module by interface identifier is easiest: session->CreateImpl(PXCGesture::CUID,&instance);
19. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. Module Interface class PXCMyInterface: public PXCBase { public: PXC_CUID_OVERWRITE(PXC_UID(‘M’,’Y’,’I’,’F’)); // configurations & inquiries struct ProfileInfo { … }; virtual pxcStatus PXCAPI QueryProfile(pxcU32 idx, ProfileInfo *pinfo)=0; virtual pxcStatus PXCAPI SetProfile(ProfileInfo *pinfo)=0; // data processing virtual pxcStatus PXCAPI ProcessDataAsync(…, PXCScheduler::SyncPoint **sp)=0; }; Unique Interface Identifier used by PXCBase::QueryInterface Consistent way of querying and setting configurations Asynchronous execution returns SP for later synchronization.
20. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. Module Configurations Consistent module configuration query and set • QueryProfile to enumerate all supported configurations – QueryProfile(WORKING_PROFILE, [out] pinfo) to retrieve the current working parameters. • SetProfile to set the desired configuration. struct ProfileInfo { … }; virtual pxcStatus PXCAPI QueryProfile(pxcU32 idx, ProfileInfo *pinfo)=0; virtual pxcStatus PXCAPI SetProfile(ProfileInfo *pinfo)=0;
21. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. Module Interaction: D/M Match UtilCapture to match stream/request • Device/Module stream view • Device properties transferred back to modules • Natural to replace device with same streams – UtilCaptureFile to record and playback Input Device1 Processing Module Processing Module Input Requests Input Requests Stream1 Descriptions Input Device2 Stream2 Descriptions Input Requests Stream2 Descriptions Stream1 Descriptions Stream3 Descriptions Input Requests Module View Module View Processing Module
22. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. Locate Streams: D/M Match (1) UtilCapture *SearchForInputDevice( PXCSession *session, PXCGesture *gesture) { UtilCapture *capture=new UtilCapture(session); PXCGesture::ProfileInfo pinfo; gesture->QueryProfile(0,&pinfo); capture->LocateStreams(&pinfo.inputs); return capture; }
23. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. Locate Streams: D/M Match (2) UtilCapture *SearchForInputDevice(PXCSession *session, PXCGesture *gesture, PXCFaceAnalysis *face) { UtilCapture *capture=new UtilCapture(session); std::vector<PXCCapture::VideoStream::DataDesc*> inputs; PXCGesture::ProfileInfo ginfo; gesture->QueryProfile(0,&ginfo); inputs.push_back(&ginfo.inputs); PXCFaceAnalysis::ProfileInfo finfo; face->QueryProfile(0,&finfo); inputs.push_back(&finfo.inputs); capture->LocateStreams(inputs); return capture; }
24. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. Module Interaction: Data Delivery Threading and synchronization complexity increases quickly • Threading must be done carefully at each input source • Ideally at each module as well to maximize parallelism • Synchronization becomes difficult as #modules increases Asynchronous execution allows dynamic pipeline construction without incurring threading complexity • At the complexity of storage buffer and SP management, partially mitigated by C++ template utilities. – PXCSmartPtr<>, PXCSmartSP – PXCSmartArray<>, PXCSmartSPArray Finger Tracking Segment ation Face Detection Image Capture Audio Capture Cmd. Control Speaker Ident.
25. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. Asynchronous Execution: Basics Synchronization of asynchronous operations • Asynchronous execution pipeline – Score-boarding to resolve I/O dependency • Multiple I/O asynchronous execution pipeline – Module starts to execute when all inputs are ready. – SP1 signals when all outputs are ready. – SP1QueryPtr() returns individual output completion status. Status reporting M1 M2 M3 SP1 SP2 SP3  Synchronize() SP1 SP2 SP3 SP4 M1 SP1 M2 SP2 M3 SP3 OK OK Error Status OK Error Status Aborted Error Status Aborted Aborted
26. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. Asynchronous Execution Sample void StreamData(UtilCapture *capture, PXCGesture *gesture) { PXCSmartArray<PXCImage> images; PXCSmartSPArray sps(2); pxcStatus sts; for (;;) { sts=capture->ReadStreamAsync( images.ReleaseRefs(),sps.ReleaseRef(0)); sts=gesture->ProcessImageAsync(images,sps.ReleaseRef(1)); sps.SynchronizeEx(); // DO something with finger tracking data } }
27. SDK Module Interfaces
28. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. PXCImage/PXCAudio Abstraction of audio and video storages Format Conversion for convenience void ImageOperations(PXCImage *image) { PXCImage::ImageInfo iinfo; image->QueryInfo(&iinfo); printf_s(“%dx%d\n”,iinfo.width, iinfo.height); PXCImage::ImageData data; image->AcquireAccess(PXCImage::ACCESS_READ, PXCImage::COLOR_FORMAT_RGB32, &data); // Do something with image data // data.planes[4], data.pitches[4] image->ReleaseAccess(&data); }
29. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. PXCCapture (Audio/Video Capture) Hieratical discovery interface: • Enumerate & create capture device – QueryDevice: Query capture device names – CreateDevice: Create a capture device instance • Enumerate & select streams – QueryStream: Query stream type – CreateVideoStream: Select a video stream – CreateAudioStream: Select an audio stream • Stream operations – QueryProfile: Query stream configurations – SetProfile: Set a stream configuration – ReadStreamAsync: Read samples from the stream
30. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. PXCCapture Device Properties >SetDeviceProperty/QueryDeviceProperty Color stream properties: • CONTRAST, BRIGHTNESS, HUE, SATURATION … Depth stream properties: • DEPTH_SMOOTHING, SATURATION_VALUE, … Audio stream properties: • MIX_LEVEL Misc. properties: • ACCELEROMETER_READING
31. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. PXCGesture (Finger Tracking) class PXCGesture:public PXCBase { struct Gesture {}// pose/gesture data structure struct Blob {}// label map data structure struct GeoNode {}// geometric node data structure struct Alert {}// event data structure QueryProfile(…);// Retrieve configuration(s) SetProfile(…);// Set active configuration SubscribeGesture(Gesture::Handler);// Event setup SubscribeAlert(Alert::Handler);// Event setup ProcessImageAsync(images, …);// Data processing QueryBlobData(…);// Retrieve label map data QueryNodeData(…);// Retrieve geometric node data QueryGestureData(…);// Retrieve pose/gesture data }; Geometric nodes Label Map
32. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. PXCFaceAnalysis class PXCFaceAnalysis:public PXCBase { class Detection { QueryProfile(…); SetProfile(…); QueryData(…); }; class Landmark { QueryProfile(…); SetProfile(…); QueryLandmarkData(…); QueryPoseData(…); }; class Recognition { QueryProfile(…); SetProfile(…); CreateModel(…); }; class Attribute { QueryProfile(…); SetProfile(…); QueryData(…); }; QueryProfile(…); SetProfile(…); ProcessImageAsync(…); } Face location detection/tracking configuration and retrieve data Face landmark detection configuration and retrieve data Face attribute detection configuration and retrieve data Face recognition configuration and retrieve data Face analysis overall configuration and data processing
33. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. PXCVoiceRecognition class PXCVoiceRecognition:public PXCBase { struct Recognition {}// Recognized data structure struct Alert {}// Event data structure QueryProfile(…);// Retrieve configuration(s) SetProfile(…);// Set active configuration SubscribeRecognition(…);// Event setup SubscribeAlert(…);// Event setup CreateGrammar(…);// AddGrammar(…);// Command list construction SetGrammar(…);// DeleteGrammar(…);// ProcessAudioAsync(…);// Data processing };
34. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. PXCVoiceSynthesis class PXCVoiceSynthesis:public PXCBase { QueryProfile(…);// Retrieve configuration(s) SetProfile(…);// Set active configuration QueueSentence(…);// Queue sentence for processing ProcessAudioAsync(…);// Data processing };
35. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. PXCDVTracker class PXCDVTracker:public PXCBase { QueryProfile(…);// Retrieve configuration(s) SetProfile(…);// Set active configuration GetTargetCount(…);// ActivateTarget(…);// Retrieve object tracking data GetTargetData(…);// ProcessImageAsync(…);// Data processing };
36. Copyright© 2013, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. INFORMATION IN THIS DOCUMENT IS PROVIDED “AS IS”. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO THIS INFORMATION INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. Copyright © , Intel Corporation. All rights reserved. Intel, the Intel logo, Xeon, Xeon Phi, Core, VTune, and Cilk are trademarks of Intel Corporation in the U.S. and other countries. Optimization Notice Intel’s compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Notice revision #20110804 Legal Disclaimer & Optimization Notice Copyright© 2012, Intel Corporation. All rights reserved. *Other brands and names are the property of their respective owners. 36