Real-time Desktop Capture

Presentation by João Portela

Sometimes you need to automate desktop capture, but how do you do it? Doesn’t video encoding have a big impact on performance? Aren’t most desktop capture applications GUI based?

Doing desktop capture and encoding on windows with minimal performance impact can certainly be a challenge. Fortunately there are plenty of technologies and libraries to help us.
In this talk we'll explore what’s out there and how an open source project helped us deliver a high quality solution.

Transcript

1. Hello João Portela
2. Real-time Desktop Capture
3. What for? Our clients needed to record what our software is displaying on the monitors, and send those images to their clients. Their current solution uses dedicated hardware. They requested us to integrate this with our current software, into the same hardware that we are currently using.
6. Summary • Record 4 monitors at 1080p@60fps. • Low to medium performance impact. • Automatic disk rotation. ◦ To multiple disks at the same time (a black-box and to send to their clients) • Automatic file split (by size or time).
7. Solution Integrate FFMPEG with our software.
8. FFMPEG Pros: • Out of the box support for multiple encoders ◦ NVENC ◦ QuickSync ◦ x264 • Command line tool that can be integrated into our system. Cons: • Not very stable with QuickSync ◦ Sometimes you had to try multiple times for it to start. • Can only record at 30fps ◦ Not a deal breaker but undesirable.
9. FFMPEG - Performance Codec CPU (TOTAL) RAM (TOTAL) GPU (NVIDIA) GPU (INTEL) Video Engine Total file size Video duration 2 NVENC + 2 QUICKSYNC 40% 36% 25% 31% 27% 80.02 MB 2 mins 21 secs 2 NVENC + 2 LIBX264 71% 36% 19% 0% 26% 421.45 MB 2 mins 45 secs 2 QUICKSYNC + 2 LIBX264 100% 36% 39% 42% 0% 182.87 MB 2 mins 10 secs 3 QUICKSYNC + 1 NVENC 48% 36% 20% 46% 13% 80.09 MB 2 min 9 secs 4 LIBX264 (very fast) 100% 33% 47%@880MHZ 0% 0% 587.76 MB 2 min 33 secs 4 LIBX264 (ultra fast) 44% 33% 42%@880MHZ 0% 0% 699.48 MB 2 mins 20 secs
10. NOPE It’s never that easy.
11. FFMPEG + Cons: DirectShow can only capture one screen. GDIgrab causes mouse flicker when recording more than one screen.
12. STOP Review technology and assumptions
13. Desktop Recording - the basics • Desktop Capture • Video Encoding
14. Capture • DirectShow • Windows GDI (gdigrab in ffmpeg) • NVFBC/NVIFR ◦ NVFBC (Frame Buffer Capture) ◦ NVIFR (In-band Frame Render) • DXGI Desktop Duplication
15. Windows GDI «The Microsoft Windows graphics device interface (GDI) enables applications to use graphics and formatted text on both the video display and the printer. Windows-based applications do not access the graphics hardware directly. Instead, GDI interacts with device drivers on behalf of applications.» Old Microsoft technology for interacting with graphics devices.
16. DirectShow «Microsoft® DirectShow® is an architecture for streaming media on the Microsoft Windows® platform. DirectShow provides for high-quality capture and playback of multimedia streams.» Technology used for much more than desktop capture. Our previous ffmpeg solution created a DirectShow stream for the desktop. ffmpeg then consumed it as if it was a capture card stream.
17. NVFBC/NVIFR NVIDIA driver-level capture paths. NVFBC is what shadowplay uses. They’re amazing as they allow us to capture the screen or an application with minimal performance impact. NVFBC - NVIDIA Frame Buffer Capture Captures the framebuffer (front buffer) without any involvement from OpenGL or Direct3D. Directly from the graphics driver. NVIFR - NVIDIA In-band Frame Render Slightly more complicated and less performant than NVFBC, this can capture a single application. This requires changes at the application level.
18. NVFBC/NVIFR Unfortunately: «In new drivers NVIFR and NVFBC are now completely locked by NVIDIA for their internal tools only. Some green(dy) vendor want to artificially limit performance difference between Shadowplay and traditional video capture tools.» Alexey Nicolaychuk aka Unwinder, RivaTuner creator «NvFBC and NvIFR are private APIs that are only available to approved partners for use in remote graphics scenarios.» NVIDIA linux driver release notes.
19. DXGI Desktop Duplication DXGI: Microsoft DirectX Graphics Infrastructure «The desktop duplication API provides remote access to a desktop image for collaboration scenarios. Apps can use the desktop duplication API to access frame-by-frame updates to the desktop. Because apps receive updates to the desktop image in a DXGI surface, the apps can use the full power of the GPU to process the image updates. » • Only windows 8.1+ • Way more performant than GDI or D3D9 approaches.
20. Encoding - using H.264 • NVENC • QuickSync • X264
21. NVENC - NVIDIA encoder Pros: • Encoder used by ShadowPlay • Runs on the graphics card on dedicated video encoding hardware Cons: • Can only encode two streams at once.
22. QuickSync Pros: • Intel encoding technology. • Uses mostly iGPU ◦ Leaving main CPU and main GPU free. Cons: • Needs CPU model support • Needs motherboard support
23. x264 Software encoder. Uses a lot of CPU (multiple cores).
24. Options? • Do it from scratch • Adapt ffmpeg ◦ Add directshow sources for all monitors ◦ New frame provider using DXGI Desktop Duplication • Adapt obs-studio
25. Do it from scratch Too much work, the deadline was close.
26. Adapt ffmpeg Also, too complex. Even compiling it was an headache.
27. obs-studio • obs-studio worked great when used as a desktop application. • Out of the box support for QuickSync, NVENC and x264 • Automatically uses DXGI desktop duplication when on windows 8.1+ Obs-studio is a rewrite of obs with the intention of separating the core functionalities form the UI layer.
28. obs-studio • libobs exists separately from obs-studio ◦ It is not too easy to use since it only has inline documentation. ◦ But we can see how it is being used by the GUI • Works surprisingly well after some initial hurdles.
29. obs-cli
30. obs-cli >obs-cli.exe --help obs-cli Allowed options: -h [ --help ] Show help message --listmonitors List available monitors resolutions --listinputs List available inputs --listencoders List available encoders --listoutputs List available outputs --listaudios List available audios -m [ --monitor ] arg set monitor to be recorded -a [ --audio ] [=arg(=default)] set audio to be recorded (default to mic) -a"device_name" -e [ --encoder ] arg (=obs_x264) set encoder -v [ --vbitrate ] arg (=2500) set video bitrate. suggested values: 1200 for low, 2500 for medium, 5000 for high -o [ --output ] arg set file destination, can be set multiple times for multiple outputs
31. obs-cli - sample command obs-cli.exe --monitor 3 --audio --encoder obs_qsv11 --vbitrate 2500 --output "videoname.mp4"
32. Recorder UI
34. Performance
38. Documentation Documentation Documentation
39. ?