Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Looking for a silver bullet to edit videos on A...

Looking for a silver bullet to edit videos on Android.

Seungwon Lee
LINE Plus AppDev5 Android Engineer
https://linedevday.linecorp.com/jp/2019/sessions/F2-2

LINE DevDay 2019

November 21, 2019
Tweet

More Decks by LINE DevDay 2019

Other Decks in Technology

Transcript

  1. 2019 DevDay Looking for a Silver Bullet To Edit Videos

    on Android. > Seungwon Lee > LINE Plus AppDev5 Android Engineer
  2. Agenda > Why Do We Need a Silver Bullet? >

    What Should We Know in Advance? > How Does android-transcoder Work? > What is Cruiser? > Troubleshooting
  3. Agenda > Why Do We Need a Silver Bullet? >

    What Should We Know in Advance? > How Does android-transcoder Work? > What is Cruiser? > Troubleshooting
  4. Blacklist > List of devices that had problems during video

    processing. Device A Device B Device C Device D …
  5. Blacklist > Blacklisted devices were redirected so that client-side video

    processing would not occur Bypass Device A Device B Device C Device D …
  6. Blacklisted Devices *H*-E*** *H*-0** S*-0** S*-0** R**** **** * R****

    **** * *H*-E*** V**-* *-0** 2***** *-0** *-0** E**-* E**-* F**-L** P*** *L*-* V**-* W**-* *T*-A* *L*-* V**-* ***0* M** **V-***9 **V-**9 **e*** *R *C-**F C**** C**** *R*-U*** *R*-**9 D**-***L *i*** * **2-***Y ****H
  7. Blacklisted Devices *H*-E*** *H*-0** S*-0** S*-0** R**** **** * R****

    **** * *H*-E*** V**-* *-0** 2***** *-0** *-0** E**-* E**-* F**-L** P*** *L*-* V**-* W**-* *T*-A* *L*-* V**-* ***0* M** **V-***9 **V-**9 **e*** *R *C-**F C**** C**** *R*-U*** *R*-**9 D**-***L *i*** * **2-***Y ****H **F-***0 **F-***0 **F-**9 **F-***0 **A_***0 **R-***0 **Y-*L **R-***0 **E-***0 **E-***2 **R-***0 **Y-* *E-***0 **A-**1 **A-**1 ****W *7-**0 ****W **V-***L **E-**2 **E-**4 **E-**1 **E-**3 **E-***0
  8. Android Wars: A New Hope {Episode 4) > android-transcoder made

    by Yuya Tanaka > https://github.com/ypresto/android-transcoder > Written with pure Java without codec licensing issues. > Supports OpenGL renderer
  9. Android Wars: A New Hope {Episode 4) Legacy Video Processor

    in LINE android- transcoder *A**** P* X O *A**** H***** X O *A**** P* X O *A**** H***** X X *A**** M***** O O > android-transcoder made by Yuya Tanaka > https://github.com/ypresto/android-transcoder > Written with pure Java without codec licensing issues. > Supports OpenGL renderer
  10. Agenda > Why Do We Need a Silver Bullet? >

    What Should We Know in Advance? > How Does android-transcoder Work? > What is Cruiser? > Troubleshooting
  11. Audio Track (AAC) Video Track (H.264) MP4 Container Video Format(Container)

    Reference: - https://en.wikipedia.org/wiki/Video_file_format - https://en.wikipedia.org/wiki/Codec
  12. AAC Audio H.264 Video MP4 Container Vorbis Audio VP8 Video

    3GP Container Encoder (AAC) Decoder (Vorbis) Friend A Mine Transcoding Encoder (H.264) Decoder (VP8) Vorbis, VP8 (X) AAC, H.264 (O)
  13. Agenda > Why Do We Need a Silver Bullet? >

    What Should We Know in Advance? > How Does android-transcoder Work? > What is Cruiser? > Troubleshooting
  14. android-transcoder Audio Decoder Video Decoder Audio Encoder Video Encoder Muxer

    Video OpenGL Renderer Extractor Audio Track Video Track Video Container Audio Track Video Track Video Container Reference: https://github.com/ypresto/android-transcoder
  15. android-transcoder Audio Decoder Video Decoder Audio Encoder Video Encoder Muxer

    Video OpenGL Renderer Extractor Audio Track Video Track Video Container Audio Track Video Track Video Container Reference: https://github.com/ypresto/android-transcoder
  16. android-transcoder Audio Decoder Video Decoder Audio Encoder Video Encoder Muxer

    Video OpenGL Renderer Extractor Audio Track Video Track Video Container Audio Track Video Track Video Container Extracting, Decoding Rendering Encoding, Muxing Reference: https://github.com/ypresto/android-transcoder
  17. android-transcoder Audio Decoder Video Decoder Audio Encoder Video Encoder Muxer

    Video OpenGL Renderer Extractor Audio Track Video Track Video Container Audio Track Video Track Video Container Extracting, Decoding Rendering Encoding, Muxing Reference: https://github.com/ypresto/android-transcoder
  18. How to Extract/Decode a Video Extractor android.media.MediaExtractor Audio Decoder Video

    Decoder Audio Encoder Video Encoder Muxer Video OpenGL Renderer Audio Track Video Track Video Container Audio Track Video Track Video Container Reference: https://github.com/ypresto/android-transcoder
  19. > Query the metadata of each track > Read the

    encoded data stream > Demux data streams into each track android.media.MediaExtractor Media Extractor Audio info: AAC Video info: H.264 Audio Track (AAC) Video Track (H.264) MP4 Container
  20. > Query the metadata of each track > Read the

    encoded data stream > Demux data streams into each track android.media.MediaExtractor Media Extractor Audio info: AAC Video info: H.264 Audio Track (AAC) Video Track (H.264) MP4 Container Audio Decoder Video Decoder
  21. How to Extract Video and Audio Media Extractor Time Audio

    Track (AAC) Video Track (H.264) MP4 Container val size = mediaExtractor.readSampleData(byteBuf, offset) val presentationTimeUs = mediaExtractor.getSampleTime()
  22. How to Extract Video and Audio Data Media Extractor mediaExtractor.advance()

    val size = mediaExtractor.readSampleData(byteBuf, offset) End of Stream Time Audio Track (AAC) Video Track (H.264) MP4 Container End of Stream … …
  23. How to Decode a Video Audio Decoder Video Decoder Audio

    Encoder Video Encoder Muxer Video OpenGL Renderer Extractor android.media.MediaCodec Audio Track Video Track Video Container Audio Track Video Track Video Container Reference: https://github.com/ypresto/android-transcoder
  24. How to Decode a Video > Wrapper class for media

    codecs (i.e. encoder and decoder) > Use together with android.media.MediaExtractor, android.media.MediaMuxer AAC Audio Decoder H.264 Video Decoder
  25. How to Extract/Decode a Video Extracting, Decoding Rendering Encoding, Muxing

    Audio Decoder Video Decoder Audio Encoder Video Encoder Muxer Video OpenGL Renderer Extractor Audio Track Video Track Video Container Audio Track Video Track Video Container Reference: https://github.com/ypresto/android-transcoder
  26. How to Extract/Decode a Video Video Decoder Media Extractor Audio

    Track (AAC) Video Track (H.264) MP4 Container
  27. How to Extract/Decode a Video Video Decoder Media Extractor Audio

    Track (AAC) Video Track (H.264) MP4 Container Empty Input Buffer
  28. How to Extract/Decode a Video Video Decoder Media Extractor Audio

    Track (AAC) Video Track (H.264) MP4 Container Empty Input Buffer Fill Input Buffer
  29. How to Extract/Decode a Video Video Decoder Output Surface Media

    Extractor Audio Track (AAC) Video Track (H.264) MP4 Container Empty Input Buffer Fill Input Buffer Video Renderer
  30. How to Extract/Decode a Video Video Decoder Output Surface Media

    Extractor Audio Track (AAC) Video Track (H.264) MP4 Container Video Renderer onFrameAvailable Empty Input Buffer Fill Input Buffer
  31. How to Extract/Decode a Video Video Decoder Output Surface Media

    Extractor Audio Track (AAC) Video Track (H.264) MP4 Container Video Renderer onFrameAvailable Released Buffer in Surface Empty Input Buffer Fill Input Buffer
  32. Rendering Audio Decoder Video Decoder Audio Encoder Video Encoder Muxer

    Video OpenGL Renderer Extractor Audio Track Video Track Video Container Audio Track Video Track Video Container Extracting, Decoding Rendering Encoding, Muxing Reference: https://github.com/ypresto/android-transcoder
  33. How to Render a Video Audio Decoder Video Decoder Audio

    Encoder Video Encoder Muxer Video OpenGL Renderer Extractor Audio Track Video Track Video Container Audio Track Video Track Video Container Extracting, Decoding Rendering Encoding, Muxing Output Surface Input Surface Frame Buffer Reference: https://github.com/ypresto/android-transcoder
  34. How to Render a Video Audio Decoder Video Decoder Audio

    Encoder Video Encoder Muxer Video OpenGL Renderer Extractor Audio Track Video Track Video Container Audio Track Video Track Video Container Extracting, Decoding Rendering Encoding, Muxing Output Surface Input Surface Frame Buffer Render Reference: https://github.com/ypresto/android-transcoder
  35. How to Render a Video Audio Decoder Video Decoder Audio

    Encoder Video Encoder Muxer Video OpenGL Renderer Extractor Audio Track Video Track Video Container Audio Track Video Track Video Container Extracting, Decoding Rendering Encoding, Muxing Output Surface Input Surface Frame Buffer Render Render Reference: https://github.com/ypresto/android-transcoder
  36. How to Render a Video > OutputSurface and InputSurface will

    be bound with the current OpenGL environment Output Surface from Decoder FrameBuffer Input Surface to Encoder
  37. How to Render a Video > Pixel data of a

    video frame is drawn into OutputSurface of the decoder 720 X 1280 Output Surface from Decoder FrameBuffer Input Surface to Encoder
  38. How to Render a Video Output Surface from Decoder FrameBuffer

    Input Surface to Encoder > We can reduce a video frame by drawing it on FrameBuffer in OpenGL 720 X 1280 540 X 960
  39. How to Render a Video > We pass the resized

    frame to InputSurface of the encoder 540 X 960 720 X 1280 540 X 960 Output Surface from Decoder FrameBuffer Input Surface to Encoder
  40. How to Encode/Mux a Video Audio Decoder Video Decoder Audio

    Encoder Video Encoder Muxer Video OpenGL Renderer Extractor Audio Track Video Track Video Container Audio Track Video Track Video Container Extracting, Decoding Rendering Encoding, Muxing Reference: https://github.com/ypresto/android-transcoder
  41. Output Buffer How to Extract/Decode a Video Video OpenGL Renderer

    Request Encoding Input Surface Video Encoder
  42. Muxer How to Extract/Decode a Video Video OpenGL Renderer Request

    Encoding Audio Track Video Track Input Surface Output Buffer Release Output Buffer Video Encoder
  43. How to Encode/Mux a Video Muxer android.media.MediaMuxer Audio Decoder Video

    Decoder Audio Encoder Video Encoder Muxer Video OpenGL Renderer Extractor Audio Track Video Track Video Container Audio Track Video Track Video Container Reference: https://github.com/ypresto/android-transcoder
  44. android.media.MediaMuxer > Mux encoded streams of audio/video > Supports MP4,

    WebM and 3GP files AAC Encoder H.264 Encoder Media Muxer Audio Track (AAC) Video Track (H.264) Video Container Reference: - https://developer.android.com/reference/android/media/MediaMuxer - https://developer.android.com/guide/topics/media/media-formats
  45. android.media.MediaMuxer Media Muxer mediaMuxer.writeSampleData(trackIndex, byteBuf, bufferInfo) -> Audio Track (AAC)

    Video Track (H.264) Video Container End of stream Time End of stream … …
  46. How Does android-transcoder Work Audio Decoder Video Decoder Audio Encoder

    Video Encoder Muxer Video OpenGL Renderer Extractor Audio Track Video Track Video Container Audio Track Video Track Video Container Reference: https://github.com/ypresto/android-transcoder
  47. Audio Features Didn’t Exist Audio sampler Audio Decoder Video Decoder

    Audio Encoder Video Encoder Muxer Video OpenGL Renderer Extractor Audio Track Video Track Video Container Audio Track Video Track Video Container
  48. Overlaying Stickers Didn’t Exist Audio sampler Overlay Sticker Audio Decoder

    Video Decoder Audio Encoder Video Encoder Muxer Video OpenGL Renderer Extractor Audio Track Video Track Video Container Audio Track Video Track Video Container
  49. Agenda > Why Do We Need a Silver Bullet? >

    What Should We Know in Advance? > How Does android-transcoder Work? > What is Cruiser? > Troubleshooting
  50. Added Features to Cruiser v1.0 Container MP4 Video Codec H.264

    Bitrate 2 Mbit/s Resolution 720*1280 Trim range 1200 ms ~ 250300 ms Audio Codec AAC Sample Rate 44.1 kHz ... ... > Adjust sampling rate of audio > Add APIs for a trancoding policy including codec, resolution, trim ranges, and etc.
  51. Audio Sampler Audio Sampler > Adjust sampling rate of audio

    Audio Decoder Video Decoder Audio Encoder Video Encoder Muxer Video OpenGL Renderer Extractor Audio Track Video Track Video Container Audio Track Video Track Video Container
  52. Audio Sampler Audio Sampler AAC Decoder AAC Encoder > Based

    on libsamplerate made by erikd > https://github.com/erikd/libsamplerate > A library for performing sample rate conversion of audio data 320 kbit/s 96 kbit/s
  53. Cruiser v1.0 Features > Adjusts video size and sampling audio

    with a simple policy for transcoding > Trims video Container MP4 Video Codec H.264 Bitrate 2 Mbit/s Resolution 720*1280 Trim range 1200 ms ~ 250300 ms Audio Codec AAC Samplerate 44.1 kHz
  54. Cruiser v1.0 Features > Adjusts video size and sampling audio

    with a simple policy for transcoding > Trims video > Mutes audio X Container MP4 Video Codec H.264 Bitrate 2 Mbit/s Resolution 720*1280 Trim range 1200 ms ~ 250300 ms Audio Codec AAC Samplerate 44.1 kHz
  55. Cruiser v1.1 Main Feature Audio sampler Overlay Sticker Audio Decoder

    Video Decoder Audio Encoder Video Encoder Muxer Video OpenGL Renderer Extractor Audio Track Video Track Video Container Audio Track Video Track Video Container
  56. How Does Cruiser Overlay A Sticker > OutputSurface and InputSurface

    will bind with the current OpenGL environment Output Surface from Decoder FrameBuffer Input Surface to Encoder
  57. How Does Cruiser Overlay A Sticker > Pixel data of

    a video frame is copied into OutputSurface of the decoder Output Surface from Decoder FrameBuffer Input Surface to Encoder
  58. How Does Cruiser Overlay A Sticker > We load a

    sticker as a Bitmap instance and replace it to a texture that OpenGL can recognize Teuxture Output Surface from Decoder FrameBuffer Input Surface to Encoder
  59. How Does Cruiser Overlay A Sticker > We can draw

    additionally overlay a sticker on FrameBuffer in OpenGL Output Surface from Decoder FrameBuffer Input Surface to Encoder
  60. How Does Cruiser Overlay A Sticker > We pass this

    frame to InputSurface of the encoder Output Surface from Decoder FrameBuffer Input Surface to Encoder
  61. How Does Cruiser Overlay A Sticker > Like this, We

    can overlay serval stickers on a video frame Output Surface from Decoder FrameBuffer Input Surface to Encoder
  62. Blacklisted Devices *H*-E*** *H*-0** S*-0** S*-0** R**** **** * R****

    **** * *H*-E*** V**-* *-0** 2***** *-0** *-0** E**-* E**-* F**-L** P*** *L*-* V**-* W**-* *T*-A* *L*-* V**-* ***0* M** **V-***9 **V-**9 **e*** *R *C-**F C**** C**** *R*-U*** *R*-**9 D**-***L *i*** * **2-***Y ****H **F-***0 **F-***0 **F-**9 **F-***0 **A_***0 **R-***0 **Y-*L **R-***0 **E-***0 **E-***2 **R-***0 **Y-* *E-***0 **A-**1 **A-**1 ****W *7-**0 ****W **V-***L **E-**2 **E-**4 **E-**1 **E-**3 **E-***0
  63. *H*-E*** *H*-0** S*-0** S*-0** R**** **** * R**** **** *

    *H*-E*** V**-* *-0** 2***** *-0** *-0** E**-* E**-* F**-L** P*** *L*-* V**-* W**-* *T*-A* *L*-* V**-* ***0* M** **V-***9 **V-**9 **e*** *R *C-**F C**** C**** *R*-U*** *R*-**9 D**-***L *i*** * **2-***Y ****H **F-***0 **F-***0 **F-**9 **F-***0 **A_***0 **R-***0 **Y-*L **R-***0 **E-***0 **E-***2 **R-***0 **Y-* *E-***0 **A-**1 **A-**1 ****W *7-**0 ****W **V-***L **E-**2 **E-**4 **E-**1 **E-**3 **E-***0 Blacklist On Cruiser X
  64. *H*-E*** *H*-0** S*-0** S*-0** R**** **** * R**** **** *

    *H*-E*** V**-* *-0** 2***** *-0** *-0** E**-* E**-* F**-L** P*** *L*-* V**-* W**-* *T*-A* *L*-* V**-* ***0* M** **V-***9 **V-**9 **e*** *R *C-**F C**** C**** *R*-U*** *R*-**9 D**-***L *i*** * **2-***Y ****H **F-***0 **F-***0 **F-**9 **F-***0 **A_***0 **R-***0 **Y-*L **R-***0 **E-***0 **E-***2 **R-***0 **Y-* *E-***0 **A-**1 **A-**1 ****W *7-**0 ****W **V-***L **E-**2 **E-**4 **E-**1 **E-**3 **E-***0 Blacklist On Cruiser X !
  65. Agenda > Why Do We Need a Silver Bullet? >

    What Should We Know in Advance? > How Does android-transcoder Work? > What is Cruiser? > Troubleshooting
  66. Troubleshooting Video issues Quality issue Orientation issue Audio issue 0

    bit/s Issue Out of Sync Audio and Video Issue Audio Distortion Issue
  67. Troubleshooting Video Issues > Quality Issue Orientation Issue Audio Issue

    0 bit/s Issue Out of Sync Audio and Video Issue Audio Distortion Issue
  68. Low Quality Due to Video Profiles Video Encoder Cruiser High

    Profile Video > Don’t leave a device to choose a video profile High Profile Video Encoder Cruiser Base Profile Video Base Profile
  69. Troubleshooting Video Issues Quality Issue > Orientation Issue Audio Issue

    0 bit/s Issue Out of Sync Audio and Video Issue Audio Distortion Issue
  70. Video Rotated Issue > Don’t set the orientation value of

    a video to an encoder Video Encoder Muxer 90 Orientation = 90 Extractor 90 X
  71. Troubleshooting Video Issues Quality Issue Orientation Issue Audio Issue >

    0 bit/s Issue Out of Sync Audio and Video Issue Audio Distortion Issue
  72. 0 bit/s Issue 128 kbit/s 96 kbit/s 96 kbit/s Audio

    Format Video Format Container Audio Decoder Audio Encoder Muxer Extractor Audio Down Sampler Audio Format Container Video Format 96 kbit/s 48 kbit/s 48 kbit/s 48 kbit/s 48 kbit/s Audio Codec AAC Sample Rate 44.1 kHz Bitrate 96 kbit/s
  73. 0 bit/s Issue > Don’t forget to check 0 values

    0 kbit/s 0 kbit/s Audio Format Video Format Container Audio Decoder Audio Encoder Muxer Extractor Audio Down Sampler Audio Format Container Video Format 0 kbit/s 0 kbit/s
  74. 0 bit/s Issue > Don’t forget to check 0 values

    0 kbit/s 96 kbit/s 96 kbit/s Audio Format Video Format Container Audio Decoder Audio Encoder Muxer Extractor Audio Down Sampler Audio Format Container Video Format 96 kbit/s
  75. Troubleshooting Video Issues Quality Issue Orientation Issue Audio Issue 0

    bit/s Issue > Out of Sync Audio and Video Issue Audio Distortion Issue
  76. Waiting of Muxer Initialization Muxer Audio Track Video Track Output

    Buffer Video Encoder X MUXER_STATE_UNINITIALIZED
  77. Waiting of Muxer Initialization Muxer Audio Track Video Track Queue

    Video Encoder Output Buffer Release Output Buffer Shallow copy MUXER_STATE_UNINITIALIZED
  78. Waiting of Muxer Initialization Muxer Audio Track Video Track Video

    Encoder MUXER_STATE_STARTED Output Buffer Release Output buffer Pass Output Buffer Queue
  79. Waiting of Muxer Initialization Muxer Audio Track Video Track Queue

    Video Encoder MUXER_STATE_STARTED Output Buffer Release Output buffer
  80. Waiting of Muxer Initialization Muxer Audio Track Video Track Video

    Encoder MUXER_STATE_STARTED Output Buffer Release Output buffer Pass Output Buffer Queue
  81. Out of Sync Issue Muxer Audio Track Video Track Pass

    Output Buffer Queue Video Encoder MUXER_STATE_STARTED Output Buffer Release Output buffer Shallow copy
  82. Out of Sync Issue Muxer Audio Track Video Track Pass

    Output Buffer Queue Video Encoder MUXER_STATE_STARTED Output Buffer Release Output Buffer Deep copy > Don’t rely on release timing of the device
  83. Troubleshooting Video Issues Quality Issue Orientation Issue Audio Issue 0

    bit/s Issue Out of Sync Audio and Video Issue > Audio Distortion Issue
  84. Audio Distortion Problem After Trimming Extract the bitrate File Analysis

    123 kbit/s Extract the bitrate Extractor 24 kbit/s
  85. Audio Distortion Problem Audio Format Video Format Container Audio Decoder

    Audio Encoder Muxer Extractor Audio Down Sampler Audio Format Container Video Format > We transcoded the audio track and resampled the ByteBuffer of the audio > We don’t need to resample this problematic audio track 24 kbit/s (123 kbit/s) 24 kbit/s (123 kbit/s) 24 kbit/s (123 kbit/s) 24 kbit/s (123 kbit/s)
  86. Audio Distortion Problem > Cruiser can detect this issue by

    extracting 24 kbit/s > Cruiser bypasses the resampling process if it has 24 kbit/s Audio Format Video Format Container Audio Decoder Audio Encoder Muxer Extractor Audio Down Sampler Audio Format Container Video Format X X X Audio Stream Bypass 24 kbit/s (123 kbit/s) 24 kbit/s (123 kbit/s)
  87. Troubleshooting Video issues Quality issue Orientation issue Audio issue 0

    bit/s Issue Out of Sync Audio and Video Issue Audio Distortion Issue
  88. Blacklisted Devices *H*-E*** *H*-0** S*-0** S*-0** R**** **** * R****

    **** * *H*-E*** V**-* *-0** 2***** *-0** *-0** E**-* E**-* F**-L** P*** *L*-* V**-* W**-* *T*-A* *L*-* V**-* ***0* M** **V-***9 **V-**9 **e*** *R *C-**F C**** C**** *R*-U*** *R*-**9 D**-***L *i*** * **2-***Y ****H **F-***0 **F-***0 **F-**9 **F-***0 **A_***0 **R-***0 **Y-*L **R-***0 **E-***0 **E-***2 **R-***0 **Y-* *E-***0 **A-**1 **A-**1 ****W *7-**0 ****W **V-***L **E-**2 **E-**4 **E-**1 **E-**3 **E-***0 X
  89. Cruiser v1.2 features > Make a video with a static

    image Cruiser v1.3 features > Crop and rotate a video What‘s Next? Cruiser v2.0 features > Each component runs on multiple threads