Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Real-time Desktop Capture

Porto Codes
September 21, 2016

Real-time Desktop Capture

Presentation by João Portela

Sometimes you need to automate desktop capture, but how do you do it? Doesn’t video encoding have a big impact on performance? Aren’t most desktop capture applications GUI based?

Doing desktop capture and encoding on windows with minimal performance impact can certainly be a challenge. Fortunately there are plenty of technologies and libraries to help us.
In this talk we'll explore what’s out there and how an open source project helped us deliver a high quality solution.

Porto Codes

September 21, 2016
Tweet

More Decks by Porto Codes

Other Decks in Programming

Transcript

  1. Hello
    João Portela

    View full-size slide

  2. Real-time Desktop Capture

    View full-size slide

  3. What for?
    Our clients needed to record what our software is displaying on the monitors,
    and send those images to their clients.
    Their current solution uses dedicated hardware. They requested us to integrate
    this with our current software, into the same hardware that we are currently
    using.

    View full-size slide

  4. Summary
    ● Record 4 monitors at 1080p@60fps.
    ● Low to medium performance impact.
    ● Automatic disk rotation.
    ○ To multiple disks at the same time (a black-box and to send to their clients)
    ● Automatic file split (by size or time).

    View full-size slide

  5. Solution
    Integrate FFMPEG with our software.

    View full-size slide

  6. FFMPEG
    Pros:
    ● Out of the box support for multiple
    encoders
    ○ NVENC
    ○ QuickSync
    ○ x264
    ● Command line tool that can be
    integrated into our system.
    Cons:
    ● Not very stable with QuickSync
    ○ Sometimes you had to try multiple times
    for it to start.
    ● Can only record at 30fps
    ○ Not a deal breaker but undesirable.

    View full-size slide

  7. FFMPEG - Performance
    Codec CPU (TOTAL) RAM (TOTAL) GPU (NVIDIA) GPU (INTEL) Video Engine Total file size Video duration
    2 NVENC +
    2 QUICKSYNC
    40% 36% 25% 31% 27% 80.02 MB 2 mins 21 secs
    2 NVENC + 2
    LIBX264
    71% 36% 19% 0% 26% 421.45 MB 2 mins 45 secs
    2 QUICKSYNC
    + 2 LIBX264
    100% 36% 39% 42% 0% 182.87 MB 2 mins 10 secs
    3 QUICKSYNC
    + 1 NVENC
    48% 36% 20% 46% 13% 80.09 MB 2 min 9 secs
    4 LIBX264
    (very fast)
    100% 33% 47%@880MHZ 0% 0% 587.76 MB 2 min 33 secs
    4 LIBX264
    (ultra fast)
    44% 33% 42%@880MHZ 0% 0% 699.48 MB 2 mins 20 secs

    View full-size slide

  8. NOPE
    It’s never that easy.

    View full-size slide

  9. FFMPEG
    + Cons:
    DirectShow can only capture one screen.
    GDIgrab causes mouse flicker when recording more than one screen.

    View full-size slide

  10. STOP
    Review technology and assumptions

    View full-size slide

  11. Desktop Recording - the basics
    ● Desktop Capture
    ● Video Encoding

    View full-size slide

  12. Capture
    ● DirectShow
    ● Windows GDI (gdigrab in ffmpeg)
    ● NVFBC/NVIFR
    ○ NVFBC (Frame Buffer Capture)
    ○ NVIFR (In-band Frame Render)
    ● DXGI Desktop Duplication

    View full-size slide

  13. Windows GDI
    «The Microsoft Windows graphics device interface (GDI) enables applications to
    use graphics and formatted text on both the video display and the printer.
    Windows-based applications do not access the graphics hardware directly.
    Instead, GDI interacts with device drivers on behalf of applications.»
    Old Microsoft technology for interacting with graphics devices.

    View full-size slide

  14. DirectShow
    «Microsoft® DirectShow® is an architecture for streaming media on the
    Microsoft Windows® platform. DirectShow provides for high-quality capture and
    playback of multimedia streams.»
    Technology used for much more than desktop capture.
    Our previous ffmpeg solution created a DirectShow stream for the desktop.
    ffmpeg then consumed it as if it was a capture card stream.

    View full-size slide

  15. NVFBC/NVIFR
    NVIDIA driver-level capture paths. NVFBC is what shadowplay uses.
    They’re amazing as they allow us to capture the screen or an application with
    minimal performance impact.
    NVFBC - NVIDIA Frame Buffer Capture
    Captures the framebuffer (front buffer) without any
    involvement from OpenGL or Direct3D. Directly from
    the graphics driver.
    NVIFR - NVIDIA In-band Frame Render
    Slightly more complicated and less performant than
    NVFBC, this can capture a single application.
    This requires changes at the application level.

    View full-size slide

  16. NVFBC/NVIFR
    Unfortunately:
    «In new drivers NVIFR and NVFBC are now completely locked by NVIDIA for their internal tools only. Some green(dy)
    vendor want to artificially limit performance difference between Shadowplay and traditional video capture tools.»
    Alexey Nicolaychuk aka Unwinder, RivaTuner creator
    «NvFBC and NvIFR are private APIs that are only available to approved partners for use in remote graphics scenarios.»
    NVIDIA linux driver release notes.

    View full-size slide

  17. DXGI Desktop Duplication
    DXGI: Microsoft DirectX Graphics Infrastructure
    «The desktop duplication API provides remote access to a desktop image for
    collaboration scenarios. Apps can use the desktop duplication API to access
    frame-by-frame updates to the desktop. Because apps receive updates to the
    desktop image in a DXGI surface, the apps can use the full power of the GPU to
    process the image updates. »
    ● Only windows 8.1+
    ● Way more performant than GDI or D3D9 approaches.

    View full-size slide

  18. Encoding - using H.264
    ● NVENC
    ● QuickSync
    ● X264

    View full-size slide

  19. NVENC - NVIDIA encoder
    Pros:
    ● Encoder used by ShadowPlay
    ● Runs on the graphics card on
    dedicated video encoding
    hardware
    Cons:
    ● Can only encode two streams at
    once.

    View full-size slide

  20. QuickSync
    Pros:
    ● Intel encoding technology.
    ● Uses mostly iGPU
    ○ Leaving main CPU and main GPU
    free.
    Cons:
    ● Needs CPU model support
    ● Needs motherboard support

    View full-size slide

  21. x264
    Software encoder.
    Uses a lot of CPU (multiple cores).

    View full-size slide

  22. Options?
    ● Do it from scratch
    ● Adapt ffmpeg
    ○ Add directshow sources for all monitors
    ○ New frame provider using DXGI Desktop Duplication
    ● Adapt obs-studio

    View full-size slide

  23. Do it from scratch
    Too much work, the deadline was close.

    View full-size slide

  24. Adapt ffmpeg
    Also, too complex. Even compiling it was an headache.

    View full-size slide

  25. obs-studio
    ● obs-studio worked great when used as a desktop application.
    ● Out of the box support for QuickSync, NVENC and x264
    ● Automatically uses DXGI desktop duplication when on windows 8.1+
    Obs-studio is a rewrite of obs with the intention of separating the core
    functionalities form the UI layer.

    View full-size slide

  26. obs-studio
    ● libobs exists separately from obs-studio
    ○ It is not too easy to use since it only has inline documentation.
    ○ But we can see how it is being used by the GUI
    ● Works surprisingly well after some initial hurdles.

    View full-size slide

  27. obs-cli
    >obs-cli.exe --help
    obs-cli Allowed options:
    -h [ --help ] Show help message
    --listmonitors List available monitors resolutions
    --listinputs List available inputs
    --listencoders List available encoders
    --listoutputs List available outputs
    --listaudios List available audios
    -m [ --monitor ] arg set monitor to be recorded
    -a [ --audio ] [=arg(=default)] set audio to be recorded (default to mic)
    -a"device_name"
    -e [ --encoder ] arg (=obs_x264) set encoder
    -v [ --vbitrate ] arg (=2500) set video bitrate. suggested values: 1200
    for low, 2500 for medium, 5000 for high
    -o [ --output ] arg set file destination, can be set multiple
    times for multiple outputs

    View full-size slide

  28. obs-cli - sample command
    obs-cli.exe --monitor 3 --audio --encoder obs_qsv11 --vbitrate 2500 --output
    "videoname.mp4"

    View full-size slide

  29. Documentation
    Documentation
    Documentation

    View full-size slide