Upgrade to Pro — share decks privately, control downloads, hide ads and more …

VIDEO EDITING ON ANDROID

VIDEO EDITING ON ANDROID

Nothing better than sharing visual content in the sleekest form possible. Which is why more and more apps now require implementing offline video editing features. Ever tried synchronizing the usage of FFmpeg and ExoPlayer on Android? If not, why should you?

76655285ce09413e5e739ecb60add953?s=128

Michal Jenicek

January 23, 2020
Tweet

Transcript

  1. Michal Jenicek, Software Engineer at STRV VIDEO EDITING I would

    like to share experiences with video editing
  2. 2 We have been working on the project called Opkix.

    Opkix is about IoT device looking like the egg with two cameras.
  3. 3 Google Pixel 3 - Black 357 x 714 px

    Google Pixel 3 - Black 357 x 714 px Opkix device has it’s own application for firmware and video management. You can manage the egg by using WiFi and BLE.
  4. 4 Google Pixel 3 - Black 357 x 714 px

    Google Pixel 3 - Black 357 x 714 px User want’s also the ability to combine clips and create final move.
  5. 5 01 CUSTOM VIDEO EDITING FEATURES

  6. 6 WYSIWYG • Visible result of every step Real-Time •

    Don’t wait for each step processing Offline • Use just WiFi & BLE FEATURE REQUIREMENTS WYSIWYG - What you see is what you get. User knows how edit step affect the video. REAL-TIME - See the the result immediately (don’t wait for real video processing) OFFLINE - app is communicating just via WiFi or BLE with the egg.
  7. 7 EXPORTABLE • Ability to export final movie • Export

    shouldn’t block the app FEATURE REQUIREMENTS
  8. 8 • Media object is defining all edits. • View

    layer represents media object in real time. • Export layer exports final movie. IMPLEMENTATION STRATEGY We know the requirements, let’s define clear implementation strategy. media object - edits can be simply persisted without need to adjust source video file
  9. 9 iOS • Platform native media object IMPLEMENTATION STRATEGY iOS

    - media object with ability to adjust behaviour.
  10. 10 iOS • Platform native media object • Platform native

    view layer IMPLEMENTATION STRATEGY iOS - media object with ability to adjust behaviour, be played.
  11. 11 iOS • Platform native media object • Platform native

    view layer • Platform native export layer IMPLEMENTATION STRATEGY iOS - media object with ability to adjust behaviour, be played and is also exportable.
  12. 12 iOS • Platform native media object • Platform native

    view layer • Platform native export layer Android • Platform native media object IMPLEMENTATION STRATEGY ANDROID REALITY iOS - media object with ability to adjust behaviour, be played and is also exportable.
  13. 13 iOS • Platform native media object • Platform native

    view layer • Platform native export layer Android • Platform native media object • Platform native view layer IMPLEMENTATION STRATEGY ANDROID REALITY iOS - media object with ability to adjust behaviour, be played and is also exportable.
  14. 14 iOS • Platform native media object • Platform native

    view layer • Platform native export layer Android • Platform native media object • Platform native view layer • Platform native export layer IMPLEMENTATION STRATEGY ANDROID REALITY iOS - media object with ability to adjust behaviour, be played and is also exportable.
  15. 15 iOS • Platform native media object • Platform native

    view layer • Platform native export layer Android • Platform native media object • Platform native view layer • Platform native export layer • Fake them all! IMPLEMENTATION STRATEGY ANDROID REALITY iOS - media object with ability to adjust behaviour, be played and is also exportable. Android - has to solve the framework itself.
  16. • MediaPlayer (view layer) 16 CHOOSE THE FAKE TOOLS MediaPlayer

    MediaPlayer is simply blackbox. We didn’t pick it.
  17. ❌ MediaPlayer • Android Media API (Extractor, Muxer, …) 17

    CHOOSE THE FAKE TOOLS android.media MediaPlayer android.media Android Media API is complicated without good samples. We tried android.media for different task and we decided not to use it this time.
  18. ❌ MediaPlayer ❌ Android Media API (Extractor, Muxer, …) •

    Commercial edit libraries (export layer) 18 CHOOSE THE FAKE TOOLS android.media MediaPlayer We didn’t want to use any paid solutions.
  19. ✅ Kotlin data class (media object) ❌ MediaPlayer ❌ Android

    Media API (Extractor, Muxer, …) ❌ Commercial edit libraries 19 CHOOSE THE FAKE TOOLS android.media MediaPlayer
  20. ✅ Kotlin data class (media object) ✅ ExoPlayer (view layer)

    ❌ MediaPlayer ❌ Android Media API (Extractor, Muxer, …) ❌ Commercial edit libraries 20 CHOOSE THE FAKE TOOLS android.media MediaPlayer ExoPlayer Open Source library by Google. Good documentation including samples and solutions on stackoverflow.
  21. ✅ Kotlin data class (media object) ✅ ExoPlayer (view layer)

    ✅ FFmpeg (export layer) ❌ MediaPlayer ❌ Android Media API (Extractor, Muxer, …) ❌ Commercial edit libraries 21 CHOOSE THE FAKE TOOLS android.media MediaPlayer FFmpeg ExoPlayer Powerful opensource library with broad community. Usable also on Android.
  22. • ExoPlayer (view layer) • FFmpeg (export layer) 22 FAKE

    TOOLS FFmpeg ExoPlayer Usage is straight forward.
  23. • ExoPlayer (view layer) • FFmpeg (export layer) 23 FAKE

    TOOLS FFmpeg ExoPlayer FFmpeg is not targeted to Android. It’s usage on Android might be less obvious. Let’s talk it a bit.
  24. 24 02 FFMPEG ON ANDROID

  25. 25 • Audio/Video processing library FFMPEG IS

  26. 26 • Audio/Video processing library • C language FFMPEG IS

  27. 27 • Audio/Video processing library • C language • Opensource

    FFMPEG IS
  28. 28 • Audio/Video processing library • C language • Opensource

    FFMPEG IS read and write media streams decode and encode media streams scale images utility functions for all other libraries libavformat libavcodec libswscale libavutil FFmpeg stands for 4 core libraries ( it takes about 16MB ).
  29. 29 • Audio/Video processing library • C language • Opensource

    • Lot of external libraries • x264 • x265 • xvidcore • vid.stab • … FFMPEG IS read and write media streams decode and encode media streams scale images utility functions for all other libraries libavformat libavcodec libswscale libavutil There is lot of external libraries available.
  30. 30 • Audio/Video processing library • C language • Opensource

    • Lot of external libraries • x264 • x265 • xvidcore • vid.stab • … FFMPEG IS read and write media streams decode and encode media streams scale images utility functions for all other libraries libavformat libavcodec libswscale libavutil FFmpeg offers thre tools.
  31. 31 • Audio/Video processing library • C language • Opensource

    • Lot of external libraries • x264 • x265 • xvidcore • vid.stab • … FFMPEG STARTS ON 16MB SIZE read and write media streams decode and encode media streams scale images utility functions for all other libraries libavformat libavcodec libswscale libavutil To write basic video editor the ffmpeg tool and few external libraries is enough. For basic video editor it takes about 20MB.
  32. 32 Build FFmpeg yourself • Compile • Connect extension libraries

    • Integrate (JNI) FFMPEG ON ANDROID You have options to get build of FFmpeg on Android
  33. 33 Build FFmpeg yourself • Compile • Connect extension libraries

    • Integrate (JNI) Use existing prebuilt library • TANERSENER (mobile-ffmpeg) • … FFMPEG ON ANDROID
  34. 34 Build FFmpeg yourself • Compile • Connect extension libraries

    • Integrate (JNI) Use existing prebuilt library • TANERSENER (mobile-ffmpeg) • … FFMPEG ON ANDROID Specific function • Function in C language (on top of FFmpeg build) • Expose function via JNI Specific function - good for exposing your code as a library.
  35. 35 Build FFmpeg yourself • Compile • Connect extension libraries

    • Integrate (JNI) Use existing prebuilt library • TANERSENER (mobile-ffmpeg) • … FFMPEG ON ANDROID Generic function • Runtime.getRuntime().exec(…) • Expose generic call via JNI Specific function • Function in C language (on top of FFmpeg build) • Expose function via JNI Send set of ffmpeg commands as a string.
  36. 36 Build FFmpeg yourself • Compile • Connect extension libraries

    • Integrate (JNI) Use existing prebuilt library • TANERSENER (mobile-ffmpeg) • … FFMPEG ON ANDROID OUR CHOICE Generic function • Runtime.getRuntime().exec(…) • Expose generic call via JNI Specific function • Function in C language (on top of FFmpeg build) • Expose function via JNI
  37. 37 FFMPEG TANERSENER TO THE RESCUE • https://github.com/tanersener/mobile-ffmpeg

  38. 38 FFMPEG TANERSENER TO THE RESCUE • Support many FFmpeg

    releases (Active development) https://github.com/tanersener/mobile-ffmpeg
  39. 39 FFMPEG TANERSENER TO THE RESCUE • Support many FFmpeg

    releases (Active development) • 32 external libraries https://github.com/tanersener/mobile-ffmpeg
  40. 40 FFMPEG TANERSENER TO THE RESCUE • Support many FFmpeg

    releases (Active development) • 32 external libraries • x86_64 support https://github.com/tanersener/mobile-ffmpeg
  41. 41 FFMPEG TANERSENER TO THE RESCUE • Support many FFmpeg

    releases (Active development) • 32 external libraries • x86_64 support • API Level 16+ https://github.com/tanersener/mobile-ffmpeg
  42. 42 FFMPEG TANERSENER TO THE RESCUE • Support many FFmpeg

    releases (Active development) • 32 external libraries • x86_64 support • API Level 16+ • MobileFFmpeg wrapper library https://github.com/tanersener/mobile-ffmpeg
  43. 43 FFMPEG TANERSENER TO THE RESCUE • Support many FFmpeg

    releases (Active development) • 32 external libraries • x86_64 support • API Level 16+ • MobileFFmpeg wrapper library • 8 types of pre-built library variants (or fork the custom one) https://github.com/tanersener/mobile-ffmpeg
  44. 44 FFMPEG TANERSENER TO THE RESCUE • Support many FFmpeg

    releases (Active development) • 32 external libraries • x86_64 support • API Level 16+ • Generic mobileFFmpeg wrapper library • 8 types of pre-built library variants (or fork the custom one) https://github.com/tanersener/mobile-ffmpeg
  45. 45 03 IMPLEMENT FEATURES

  46. 46 < FEATURE > < LAYER > Piece of code

    to show the main trick.
  47. 47 TIMELINE EXOPLAYER val mediaSourceList: List<ClippingMediaSource> = timelineList.map { ClippingMediaSource(

    ProgressiveMediaSource.Factory(videoDataSourceFactory) .createMediaSource(Uri.parse(it.videoPath)), it.startPositionInMillis, it.endPositionInMillis) } exoPlayer.prepare( ConcatenatingMediaSource(isAtomic = true, *mediaSourceList.toTypedArray()) )
  48. 48 TIMELINE EXOPLAYER val mediaSourceList: List<ClippingMediaSource> = timelineList.map { ClippingMediaSource(

    ProgressiveMediaSource.Factory(videoDataSourceFactory) .createMediaSource(Uri.parse(it.videoPath)), it.startPositionInMillis, it.endPositionInMillis) } exoPlayer.prepare( ConcatenatingMediaSource(isAtomic = true, *mediaSourceList.toTypedArray()) )
  49. 49 TIMELINE EXOPLAYER val mediaSourceList: List<ClippingMediaSource> = timelineList.map { ClippingMediaSource(

    ProgressiveMediaSource.Factory(videoDataSourceFactory) .createMediaSource(Uri.parse(it.videoPath)), it.startPositionInMillis, it.endPositionInMillis) } exoPlayer.prepare( ConcatenatingMediaSource(isAtomic = true, *mediaSourceList.toTypedArray()) )
  50. 50 TIMELINE EXOPLAYER val mediaSourceList: List<ClippingMediaSource> = timelineList.map { ClippingMediaSource(

    ProgressiveMediaSource.Factory(videoDataSourceFactory) .createMediaSource(Uri.parse(it.videoPath)), it.startPositionInMillis, it.endPositionInMillis) } exoPlayer.prepare( ConcatenatingMediaSource(isAtomic = true, *mediaSourceList.toTypedArray()) )
  51. 51 TIMELINE EXOPLAYER val mediaSourceList: List<ClippingMediaSource> = timelineList.map { ClippingMediaSource(

    ProgressiveMediaSource.Factory(videoDataSourceFactory) .createMediaSource(Uri.parse(it.videoPath)), it.startPositionInMillis, it.endPositionInMillis) } exoPlayer.prepare( ConcatenatingMediaSource(isAtomic = true, *mediaSourceList.toTypedArray()) )
  52. 52 TIMELINE FFMPEG ffmpeg -i input1.mp4 -c copy -f mpegts

    intermediate1.ts ffmpeg -i input2.mp4 -c copy -f mpegts intermediate2.ts ffmpeg -i "concat:intermediate1.ts|intermediate2.ts" -c copy output.mp4 Concat protocol
  53. 53 TIMELINE FFMPEG ffmpeg -i input1.mp4 -c copy -f mpegts

    intermediate1.ts ffmpeg -i input2.mp4 -c copy -f mpegts intermediate2.ts ffmpeg -i "concat:intermediate1.ts|intermediate2.ts" -c copy output.mp4 Concat protocol
  54. 54 TIMELINE FFMPEG ffmpeg -i input1.mp4 -c copy -f mpegts

    intermediate1.ts ffmpeg -i input2.mp4 -c copy -f mpegts intermediate2.ts ffmpeg -i "concat:intermediate1.ts|intermediate2.ts" -c copy output.mp4 Concat protocol mpegts - converts mp4 to MPEG-2 transport stream, which is concatenable.
  55. 55 TIMELINE FFMPEG ffmpeg -i input1.mp4 -c copy -f mpegts

    intermediate1.ts ffmpeg -i input2.mp4 -c copy -f mpegts intermediate2.ts ffmpeg -i "concat:intermediate1.ts|intermediate2.ts" -c copy output.mp4 Concat protocol mpegts - converts mp4 to MPEG-2 transport stream, which is concatenable.
  56. 56 TIMELINE FFMPEG ffmpeg -i input1.mp4 -c copy -f mpegts

    intermediate1.ts ffmpeg -i input2.mp4 -c copy -f mpegts intermediate2.ts ffmpeg -i "concat:intermediate1.ts|intermediate2.ts" -c copy output.mp4 Concat protocol mpegts - converts mp4 to MPEG-2 transport stream, which is concatenable.
  57. 57 TIMELINE FFMPEG ffmpeg -i input1.mp4 -c copy -f mpegts

    intermediate1.ts ffmpeg -i input2.mp4 -c copy -f mpegts intermediate2.ts ffmpeg -i "concat:intermediate1.ts|intermediate2.ts" -c copy output.mp4 Concat protocol mpegts - converts mp4 to MPEG-2 transport stream, which is concatenable.
  58. 58 TIMELINE FFMPEG ffmpeg -i input1.mp4 -c copy -f mpegts

    intermediate1.ts ffmpeg -i input2.mp4 -c copy -f mpegts intermediate2.ts ffmpeg -i "concat:intermediate1.ts|intermediate2.ts" -c copy output.mp4 Concat protocol mpegts - converts mp4 to MPEG-2 transport stream, which is concatenable.
  59. 59 TIMELINE FFMPEG ffmpeg -i input1.mp4 -c copy -f mpegts

    intermediate1.ts ffmpeg -i input2.mp4 -c copy -f mpegts intermediate2.ts ffmpeg -i "concat:intermediate1.ts|intermediate2.ts" -c copy output.mp4 Concat protocol mpegts - converts mp4 to MPEG-2 transport stream, which is concatenable. Concat protocol is fast since it doesn’t require RE-ENCODING.
  60. 60 TIMELINE FFMPEG ffmpeg -i input1.mp4 -c copy -f mpegts

    intermediate1.ts ffmpeg -i input2.mp4 -c copy -f mpegts intermediate2.ts ffmpeg -i "concat:intermediate1.ts|intermediate2.ts" -c copy output.mp4 Concat protocol It’s usage is limited by using intermediate files (you have to handle them and it is another space required for processing). Another downside is hard to show overall progress.
  61. 61 TIMELINE FFMPEG ffmpeg -i input1.mp4 -i input2.mov -i input3.ts

    \ -filter_complex “[0:v:0][0:a:0][1:v:0][1:a:0][2:v:0][2:a:0] concat=n=2:v:1:a=1[outv][outa]" \ -map "[outv]" -map "[outa]" output.mp4 Concat filter
  62. 62 TIMELINE FFMPEG ffmpeg -i input1.mp4 -i input2.mov -i input3.ts

    \ -filter_complex “[0:v:0][0:a:0][1:v:0][1:a:0][2:v:0][2:a:0] concat=n=2:v:1:a=1[outv][outa]" \ -map "[outv]" -map "[outa]" output.mp4 Concat filter video and audio stream of input 1
  63. 63 TIMELINE FFMPEG ffmpeg -i input1.mp4 -i input2.mov -i input3.ts

    \ -filter_complex “[0:v:0][0:a:0][1:v:0][1:a:0][2:v:0][2:a:0] concat=n=2:v:1:a=1[outv][outa]" \ -map "[outv]" -map "[outa]" output.mp4 Concat filter video and audio stream of input 2
  64. 64 TIMELINE FFMPEG ffmpeg -i input1.mp4 -i input2.mov -i input3.ts

    \ -filter_complex “[0:v:0][0:a:0][1:v:0][1:a:0][2:v:0][2:a:0] concat=n=2:v:1:a=1[outv][outa]" \ -map "[outv]" -map "[outa]" output.mp4 Concat filter video and audio stream of input 3
  65. 65 TIMELINE FFMPEG ffmpeg -i input1.mp4 -i input2.mov -i input3.ts

    -filter_complex “[0:v:0][0:a:0][1:v:0][1:a:0][2:v:0][2:a:0] concat=n=2:v:1:a=1[outv][outa]" \ -map "[outv]" -map "[outa]" output.mp4 Concat filter Concat filter is transposing all inputs to streams and you can write all operations as one command.
  66. 66 TIMELINE FFMPEG ffmpeg -i input1.mp4 -i input2.mov -i input3.ts

    -filter_complex “[0:v:0][0:a:0][1:v:0][1:a:0][2:v:0][2:a:0] concat=n=2:v:1:a=1[outv][outa]" \ -map "[outv]" -map "[outa]" output.mp4 Concat filter Concat filter is transposing all inputs to streams and you can write all operations as one command.
  67. 67 TIMELINE FFMPEG ffmpeg -i input1.mp4 -i input2.mov -i input3.ts

    \ -filter_complex “[0:v:0][0:a:0][1:v:0][1:a:0][2:v:0][2:a:0] concat=n=2:v:1:a=1[outv][outa]" \ -map "[outv]" -map "[outa]" output.mp4 Concat filter Concat filter is transposing all inputs to streams and you can write all operations as one command. This is great to show overall progress. It can combine different video file types. Even though it requires re-encoding, so it’s quite slow, we picked this as usable solution. Export of course requires processing on foreground service not to block the app.
  68. 68 TRIM EXOPLAYER val mediaSourceList: List<ClippingMediaSource> = timelineList.map { ClippingMediaSource(

    ProgressiveMediaSource.Factory(videoDataSourceFactory) .createMediaSource(Uri.parse(it.videoPath)), it.startPositionInMillis, it.endPositionInMillis) } exoPlayer.prepare( ConcatenatingMediaSource(isAtomic = true, *mediaSourceList.toTypedArray()) ) Trim is ensured by parameters start and end of media source
  69. 69 TRIM EXOPLAYER val mediaSourceList: List<ClippingMediaSource> = timelineList.map { ClippingMediaSource(

    ProgressiveMediaSource.Factory(videoDataSourceFactory) .createMediaSource(Uri.parse(it.videoPath)), it.startPositionInMillis, it.endPositionInMillis) } exoPlayer.prepare( ConcatenatingMediaSource(isAtomic = true, *mediaSourceList.toTypedArray()) ) Code including TRIM and CONCAT
  70. 70 TRIM FFMPEG ffmpeg -i input1.mp4 -i input2.mov -filter_complex [0:v:0]trim=0.0:20.624,

    setpts=PTS-STARTPTS [video00] [1:v:0]trim=0.0:21.625, setpts=PTS-STARTPTS [video10] [0:a:0]atrim=0.0:20.624, asetpts=PTS-STARTPTS [audio00] [1:a:0]atrim=0.0:21.625, asetpts=PTS-STARTPTS [audio10] [video00][video10][audio00][audio10] concat=n=2:v:1:a=1 [outv] [outa] -map "[outv]" -map "[outa]" output.mp4
  71. 71 TRIM FFMPEG ffmpeg -i input1.mp4 -i input2.mov -filter_complex [0:v:0]trim=0.0:20.624,

    setpts=PTS-STARTPTS [video00] [1:v:0]trim=0.0:21.625, setpts=PTS-STARTPTS [video10] [0:a:0]atrim=0.0:20.624, asetpts=PTS-STARTPTS [audio00] [1:a:0]atrim=0.0:21.625, asetpts=PTS-STARTPTS [audio10] [video00][video10][audio00][audio10] concat=n=2:v:1:a=1 [outv] [outa] -map "[outv]" -map "[outa]" output.mp4
  72. 72 TRIM FFMPEG ffmpeg -i input1.mp4 -i input2.mov -filter_complex [0:v:0]trim=0.0:20.624,

    setpts=PTS-STARTPTS [video00] [1:v:0]trim=0.0:21.625, setpts=PTS-STARTPTS [video10] [0:a:0]atrim=0.0:20.624, asetpts=PTS-STARTPTS [audio00] [1:a:0]atrim=0.0:21.625, asetpts=PTS-STARTPTS [audio10] [video00][video10][audio00][audio10] concat=n=2:v:1:a=1 [outv] [outa] -map "[outv]" -map "[outa]" output.mp4
  73. 73 ROTATE/ZOOM/MOVE EXOPLAYER import android.graphics.Matrix val matrix = Matrix() matrix.setScale(video.zoom,

    video.zoom, pivotPointX, pivotPointY) matrix.postScale(cropScale, cropScale, pivotPointX, pivotPointY) matrix.postRotate(video.rotation, pivotPointX, pivotPointY) matrix.postTranslate(translationX * cropScale, translationY * cropScale) (binding.playerView.videoSurfaceView as TextureView).setTransform(matrix)
  74. 74 ROTATE/ZOOM/MOVE EXOPLAYER import android.graphics.Matrix val matrix = Matrix() matrix.setScale(video.zoom,

    video.zoom, pivotPointX, pivotPointY) matrix.postScale(cropScale, cropScale, pivotPointX, pivotPointY) matrix.postRotate(video.rotation, pivotPointX, pivotPointY) matrix.postTranslate(translationX * cropScale, translationY * cropScale) (binding.playerView.videoSurfaceView as TextureView).setTransform(matrix)
  75. 75 ROTATE/ZOOM/MOVE EXOPLAYER import android.graphics.Matrix val matrix = Matrix() matrix.setScale(video.zoom,

    video.zoom, pivotPointX, pivotPointY) matrix.postScale(cropScale, cropScale, pivotPointX, pivotPointY) matrix.postRotate(video.rotation, pivotPointX, pivotPointY) matrix.postTranslate(translationX * cropScale, translationY * cropScale) (binding.playerView.videoSurfaceView as TextureView).setTransform(matrix)
  76. 76 ROTATE/ZOOM/MOVE EXOPLAYER import android.graphics.Matrix val matrix = Matrix() matrix.setScale(video.zoom,

    video.zoom, pivotPointX, pivotPointY) matrix.postScale(cropScale, cropScale, pivotPointX, pivotPointY) matrix.postRotate(video.rotation, pivotPointX, pivotPointY) matrix.postTranslate(translationX * cropScale, translationY * cropScale) (binding.playerView.videoSurfaceView as TextureView).setTransform(matrix)
  77. 77 ROTATE/ZOOM/MOVE EXOPLAYER import android.graphics.Matrix val matrix = Matrix() matrix.setScale(video.zoom,

    video.zoom, pivotPointX, pivotPointY) matrix.postScale(cropScale, cropScale, pivotPointX, pivotPointY) matrix.postRotate(video.rotation, pivotPointX, pivotPointY) matrix.postTranslate(translationX * cropScale, translationY * cropScale) (binding.playerView.videoSurfaceView as TextureView).setTransform(matrix)
  78. 78 ZOOM/MOVE FFMPEG ffmpeg -i input1.mp4 -filter_complex [0:v:0] scale=w=(1.0*max(iw*1080/ih\,1920)):h=(1.0*max(1080\,ih*1920/iw)), crop=w=1920:h=1080:x=(iw-ow)/2-((iw*0.5)/1920.0):y=(ih-oh)/2-((ih*0.0)/1080.0)

    [video00] [0:a:0]anull [audio00] [video00][audio00] concat=n=1:v:1:a=1 [outv][outa] -map "[outv]" -map "[outa]" output.mp4 1.0 ${zoom} 0.5 ${translation.translationX} 0.0 ${translation.translationY}
  79. 79 ZOOM/MOVE FFMPEG ffmpeg -i input1.mp4 -filter_complex [0:v:0] scale=w=(1.0*max(iw*1080/ih\,1920)):h=(1.0*max(1080\,ih*1920/iw)), crop=w=1920:h=1080:x=(iw-ow)/2-((iw*0.5)/1920.0):y=(ih-oh)/2-((ih*0.0)/1080.0)

    [video00] [0:a:0]anull [audio00] [video00][audio00] concat=n=1:v:1:a=1 [outv][outa] -map "[outv]" -map "[outa]" output.mp4 1.0 ${zoom} 0.5 ${translation.translationX} 0.0 ${translation.translationY}
  80. 80 ZOOM/MOVE FFMPEG ffmpeg -i input1.mp4 -filter_complex [0:v:0] scale=w=(1.0*max(iw*1080/ih\,1920)):h=(1.0*max(1080\,ih*1920/iw)), crop=w=1920:h=1080:x=(iw-ow)/2-((iw*0.5)/1920.0):y=(ih-oh)/2-((ih*0.0)/1080.0)

    [video00] [0:a:0]anull [audio00] [video00][audio00] concat=n=1:v:1:a=1 [outv][outa] -map "[outv]" -map "[outa]" output.mp4 1.0 ${zoom} 0.5 ${translation.translationX} 0.0 ${translation.translationY}
  81. 81 ROTATE FFMPEG ffmpeg -i input1.mp4 -filter_complex [0:v:0] rotate=90*PI/180:ow=min(iw,ih)/sqrt(2):oh=ow:c=none [video00]

    [0:a:0]anull [audio00] [video00][audio00] concat=n=1:v:1:a=1 [outv][outa] -map "[outv]" -map "[outa]" output.mp4 ScaleType.CROP Rotate input video clockwise, expressed as number of radians. c=none used when no backgeround is ever shown to improve performance
  82. 82 ROTATE FFMPEG ffmpeg -i input1.mp4 -filter_complex [0:v:0] rotate=90*PI/180:ow=min(iw,ih)/sqrt(2):oh=ow:c=none [video00]

    [0:a:0]anull [audio00] [video00][audio00] concat=n=1:v:1:a=1 [outv][outa] -map "[outv]" -map "[outa]" output.mp4 ScaleType.CROP Rotate input video clockwise, expressed as number of radians. c=none used when no backgeround is ever shown to improve performance
  83. 83 ROTATE FFMPEG ffmpeg -i input1.mp4 -filter_complex [0:v:0] rotate=90*PI/180:ow=min(iw,ih)/sqrt(2):oh=ow:c=none [video00]

    [0:a:0]anull [audio00] [video00][audio00] concat=n=1:v:1:a=1 [outv][outa] -map "[outv]" -map "[outa]" output.mp4 ScaleType.CROP Rotate input video clockwise, expressed as number of radians. c=none used when no backgeround is ever shown to improve performance
  84. 84 ROTATE FFMPEG ffmpeg -i input1.mp4 -filter_complex [0:v:0] rotate=90*PI/180:ow=min(iw,ih)/sqrt(2):oh=ow:c=none [video00]

    [0:a:0]anull [audio00] [video00][audio00] concat=n=1:v:1:a=1 [outv][outa] -map "[outv]" -map "[outa]" output.mp4 ScaleType.CROP Rotate input video clockwise, expressed as number of radians. c=none used when no backgeround is ever shown to improve performance
  85. 85 ROTATE FFMPEG ffmpeg -i input1.mp4 -filter_complex [0:v:0] rotate=90*PI/180:ow=min(iw,ih)/sqrt(2):oh=ow:c=none [video00]

    [0:a:0]anull [audio00] [video00][audio00] concat=n=1:v:1:a=1 [outv][outa] -map "[outv]" -map "[outa]" output.mp4 ScaleType.CROP Rotate input video clockwise, expressed as number of radians. c=none used when no backgeround is ever shown to improve performance
  86. 86 COLOR FILTER EXOPLAYER <com.google.android.exoplayer2.ui.PlayerView app:surface_type=“texture_view” />

  87. 87 COLOR FILTER EXOPLAYER 1) val contentFrame = binding.root.findViewById<View> (com.google.android.exoplayer2.ui.R.id.exo_content_frame)

    as AspectRatioFrameLayout 2) val textureView = GLTextureView(context) contentFrame.addView(textureView) 3) val exoPlayerRenderer = ExoPlayerRenderer(context, textureView, viewModel.getPlayer()) textureView.setRenderer(exoPlayerRenderer) 4) exoPlayerRenderer.setTransform(Matrix()) <com.google.android.exoplayer2.ui.PlayerView app:surface_type=“texture_view” />
  88. 88 COLOR FILTER EXOPLAYER 1) val contentFrame = binding.root.findViewById<View> (com.google.android.exoplayer2.ui.R.id.exo_content_frame)

    as AspectRatioFrameLayout 2) val textureView = GLTextureView(context) contentFrame.addView(textureView) 3) val exoPlayerRenderer = ExoPlayerRenderer(context, textureView, viewModel.getPlayer()) textureView.setRenderer(exoPlayerRenderer) 4) exoPlayerRenderer.setTransform(Matrix()) <com.google.android.exoplayer2.ui.PlayerView app:surface_type=“texture_view” />
  89. 89 COLOR FILTER EXOPLAYER 1) val contentFrame = binding.root.findViewById<View> (com.google.android.exoplayer2.ui.R.id.exo_content_frame)

    as AspectRatioFrameLayout 2) val textureView = GLTextureView(context) contentFrame.addView(textureView) 3) val exoPlayerRenderer = ExoPlayerRenderer(context, textureView, viewModel.getPlayer()) textureView.setRenderer(exoPlayerRenderer) 4) exoPlayerRenderer.setTransform(Matrix()) <com.google.android.exoplayer2.ui.PlayerView app:surface_type=“texture_view” />
  90. 90 COLOR FILTER EXOPLAYER 1) val contentFrame = binding.root.findViewById<View> (com.google.android.exoplayer2.ui.R.id.exo_content_frame)

    as AspectRatioFrameLayout 2) val textureView = GLTextureView(context) contentFrame.addView(textureView) 3) val exoPlayerRenderer = ExoPlayerRenderer(context, textureView, viewModel.getPlayer()) textureView.setRenderer(exoPlayerRenderer) 4) exoPlayerRenderer.setTransform(Matrix()) <com.google.android.exoplayer2.ui.PlayerView app:surface_type=“texture_view” />
  91. 91 COLOR FILTER EXOPLAYER 1) val contentFrame = binding.root.findViewById<View> (com.google.android.exoplayer2.ui.R.id.exo_content_frame)

    as AspectRatioFrameLayout 2) val textureView = GLTextureView(context) contentFrame.addView(textureView) 3) val exoPlayerRenderer = ExoPlayerRenderer(context, textureView, viewModel.getPlayer()) textureView.setRenderer(exoPlayerRenderer) 4) exoPlayerRenderer.setTransform(Matrix()) <com.google.android.exoplayer2.ui.PlayerView app:surface_type=“texture_view” /> With this mechanism you can apply any OpenGL program you want.
  92. 92 COLOR FILTER FFMPEG VideoEffectType.NONE -> "" VideoEffectType.BLACK_WHITE -> "hue=s=0"

    VideoEffectType.VIGNETTE -> "vignette=PI/4" VideoEffectType.SEPIA -> "colorchannelmixer=.393:.769:.189:0:.349:.686:.168:0:.272:.534:.131" VideoEffectType.INVERT -> "negate" VideoEffectType.GAMMA -> "eq=gamma=5.0" ffmpeg -i input1.mp4 -filter_complex [0:v:0] colorchannelmixer=.393:.769:.189:0:.349:.686:.168:0:.272:.534:.131 [video00] [0:a:0]anull[audio00] [video00][audio00] concat=n=1:v:1:a=1 [outv][outa] -map "[outv]" -map "[outa]" output.mp4 FFmpeg offers operations over the color channel matrix. Sample of SEPIA effect.
  93. 93 COLOR FILTER FFMPEG VideoEffectType.NONE -> "" VideoEffectType.BLACK_WHITE -> "hue=s=0"

    VideoEffectType.VIGNETTE -> "vignette=PI/4" VideoEffectType.SEPIA -> "colorchannelmixer=.393:.769:.189:0:.349:.686:.168:0:.272:.534:.131" VideoEffectType.INVERT -> "negate" VideoEffectType.GAMMA -> "eq=gamma=5.0" ffmpeg -i input1.mp4 -filter_complex [0:v:0] colorchannelmixer=.393:.769:.189:0:.349:.686:.168:0:.272:.534:.131 [video00] [0:a:0]anull[audio00] [video00][audio00] concat=n=1:v:1:a=1 [outv][outa] -map "[outv]" -map "[outa]" output.mp4 In therms of well known filters, we can use existing tools of ffmpeg.
  94. 94 SPEED EXOPLAYER simpleExoPlayer.playbackParameters = PlaybackParameters(currentItem.speedMultiplier, _ , _ )

  95. 95 SPEED FFMPEG ffmpeg -i input1.mp4 -filter_complex [0:v:0] setpts=${1/speedMultiplier}*PTS[video00] [0:a:0]

    atempo=${speedMultiplier} [audio00] [video00][audio00] concat=n=1:v:1:a=1 [outv][outa] -map "[outv]" -map "[outa]" output.mp4 setpts= change PTS (presentation timestamp) of the input frames. setpts=0.5 fast motion setpts=2.0 slow motion atempo=0.8 means 80% atempo is just speedUp asetpts is dropping (audio) frames
  96. 96 SPEED FFMPEG ffmpeg -i input1.mp4 -filter_complex [0:v:0] setpts=${1/speedMultiplier}*PTS[video00] [0:a:0]

    atempo=${speedMultiplier} [audio00] [video00][audio00] concat=n=1:v:1:a=1 [outv][outa] -map "[outv]" -map "[outa]" output.mp4 setpts= change PTS (presentation timestamp) of the input frames. setpts=0.5 fast motion setpts=2.0 slow motion atempo=0.8 means 80% atempo is just speedUp asetpts is dropping (audio) frames
  97. 97 SPEED FFMPEG ffmpeg -i input1.mp4 -filter_complex [0:v:0] setpts=${1/speedMultiplier}*PTS[video00] [0:a:0]

    atempo=${speedMultiplier} [audio00] [video00][audio00] concat=n=1:v:1:a=1 [outv][outa] -map "[outv]" -map "[outa]" output.mp4 setpts= change PTS (presentation timestamp) of the input frames. setpts=0.5 fast motion setpts=2.0 slow motion atempo=0.8 means 80% atempo is just speedUp asetpts is dropping (audio) frames
  98. 98 VOLUME EXOPLAYER simpleExoPlayer.volume = currentItem.volumeMultiplier

  99. 99 VOLUME EXOPLAYER simpleExoPlayer.volume = currentItem.volumeMultiplier The platform doesn't provide

    support for volume greater than 1.0f (amplification). It could be implemented as an AudioProcessor that allows the application to set a volume multiplier for each channel. ChannelMappingAudioProcessor (ExoPlayer/issues/2659)
  100. 100 VOLUME EXOPLAYER simpleExoPlayer.volume = currentItem.volumeMultiplier The platform doesn't provide

    support for volume greater than 1.0f (amplification). It could be implemented as an AudioProcessor that allows the application to set a volume multiplier for each channel. ChannelMappingAudioProcessor (ExoPlayer/issues/2659)
  101. 101 VOLUME EXOPLAYER simpleExoPlayer.volume = currentItem.volumeMultiplier The platform doesn't provide

    support for volume greater than 1.0f (amplification). It could be implemented as an AudioProcessor that allows the application to set a volume multiplier for each channel. ChannelMappingAudioProcessor (ExoPlayer/issues/2659)
  102. 102 VOLUME FFMPEG ffmpeg -i input1.mp4 -filter_complex [0:v:0] input1[video00] [0:a:0]

    volume=${volumeMultiplier}[audio00] [video00][audio00] concat=n=1:v:1:a=1 [outv][outa] -map "[outv]" -map "[outa]" output.mp4 The default value for volume is "1.0" = unity gain Amplification (values over 1.0) works well too.
  103. 103 VOLUME FFMPEG ffmpeg -i input1.mp4 -filter_complex [0:v:0] input1[video00] [0:a:0]

    volume=${volumeMultiplier}[audio00] [video00][audio00] concat=n=1:v:1:a=1 [outv][outa] -map "[outv]" -map "[outa]" output.mp4 The default value for volume is "1.0" = unity gain Amplification (values over 1.0) works well too.
  104. 104 TEXT EXOPLAYER fun StaticLayout.draw(canvas: Canvas?, x: Float, y: Float,

    scale: Float) { canvas?.withTranslation(x, y) { canvas.withScale(x=scale, y=scale, pivotX=0f, pivotY=0f) { draw(this) } } } class TextCanvasView : View { override fun onDraw(canvas: Canvas?) { super.onDraw(canvas) layout.draw(canvas, translationX, translationY, scale) } } It’s about drawing text to static layout
  105. 105 TEXT EXOPLAYER fun StaticLayout.draw(canvas: Canvas?, x: Float, y: Float,

    scale: Float) { canvas?.withTranslation(x, y) { canvas.withScale(x=scale, y=scale, pivotX=0f, pivotY=0f) { draw(this) } } } class TextCanvasView : View { override fun onDraw(canvas: Canvas?) { super.onDraw(canvas) layout.draw(canvas, translationX, translationY, scale) } }
  106. 106 TEXT EXOPLAYER fun StaticLayout.draw(canvas: Canvas?, x: Float, y: Float,

    scale: Float) { canvas?.withTranslation(x, y) { canvas.withScale(x=scale, y=scale, pivotX=0f, pivotY=0f) { draw(this) } } } class TextCanvasView : View { override fun onDraw(canvas: Canvas?) { super.onDraw(canvas) layout.draw(canvas, translationX, translationY, scale) } }
  107. 107 TEXT FFMPEG ffmpeg -i input1.mp4 -filter_complex [0:v:0] drawtext=fontfile=Chewy.ttf :fontsize=100

    :fontcolor=white :x=\(w-text_w\)/2 :y=\(h-text_h\)/2 :text=Text [video00] [0:a:0] anull[audio00] [video00][audio00] concat=n=1:v:1:a=1 [outv][outa] -map "[outv]" -map "[outa]" output.mp4 Title based text is strongly dependent on characters supported by font. There is bad support for placeholder font in FFmpeg. Text on android canvas might differ from text on final movie - when special characters are not supported by font.
  108. 108 TEXT FFMPEG ffmpeg -i input1.mp4 -filter_complex [0:v:0] drawtext=fontfile=Chewy.ttf :fontsize=100

    :fontcolor=white :x=\(w-text_w\)/2 :y=\(h-text_h\)/2 :text=Text [video00] [0:a:0] anull[audio00] [video00][audio00] concat=n=1:v:1:a=1 [outv][outa] -map "[outv]" -map "[outa]" output.mp4
  109. 109 TEXT FFMPEG ffmpeg -i input1.mp4 -filter_complex [0:v:0]null[video00] movie=bitmap_text.png[text00] [video00][text00]overlay=x=0:y=0,

    setpts=PTS-STARTPTS[videotext00] [0:a:0] anull[audio00] [videotext00][audio00] concat=n=1:v:1:a=1 [outv][outa] -map "[outv]" -map "[outa]" output.mp4
  110. 110 TEXT FFMPEG ffmpeg -i input1.mp4 -filter_complex [0:v:0]null[video00] movie=bitmap_text.png[text00] [video00][text00]overlay=x=0:y=0,

    setpts=PTS-STARTPTS[videotext00] [0:a:0] anull[audio00] [videotext00][audio00] concat=n=1:v:1:a=1 [outv][outa] -map "[outv]" -map "[outa]" output.mp4 Overlay 0,0 (left upper corner) expects that bitmap is in the video resolution.
  111. 111 TEXT FFMPEG ffmpeg -i input1.mp4 -filter_complex [0:v:0]null[video00] movie=bitmap_text.png[text00] [video00][text00]overlay=x=0:y=0,

    setpts=PTS-STARTPTS[videotext00] [0:a:0] anull[audio00] [videotext00][audio00] concat=n=1:v:1:a=1 [outv][outa] -map "[outv]" -map "[outa]" output.mp4
  112. 112 BACKGROUND MUSIC EXOPLAYER private val exoPlayer: SimpleExoPlayer private val

    exoPlayerBackground: SimpleExoPlayer exoPlayerBackground.prepare( ProgressiveMediaSource.Factory(dataSourceFactory) .createMediaSource(Uri.parse(requireNotNull(currentBackgroundMusic).path)) ) exoPlayer.addListener(object : Player.EventListener { override fun onPlayerStateChanged(playWhenReady: Boolean, playbackState: Int) { when (exoPlayer.playbackState) { Player.STATE_READY -> { if (!exoPlayer.playWhenReady) { exoPlayerBackground.playWhenReady = false } else { exoPlayerBackground.playWhenReady = currentBackgroundMusic != null } } } } })
  113. 113 BACKGROUND MUSIC EXOPLAYER private val exoPlayer: SimpleExoPlayer private val

    exoPlayerBackground: SimpleExoPlayer exoPlayerBackground.prepare( ProgressiveMediaSource.Factory(dataSourceFactory) .createMediaSource(Uri.parse(requireNotNull(currentBackgroundMusic).path)) ) exoPlayer.addListener(object : Player.EventListener { override fun onPlayerStateChanged(playWhenReady: Boolean, playbackState: Int) { when (exoPlayer.playbackState) { Player.STATE_READY -> { if (!exoPlayer.playWhenReady) { exoPlayerBackground.playWhenReady = false } else { exoPlayerBackground.playWhenReady = currentBackgroundMusic != null } } } } })
  114. 114 BACKGROUND MUSIC EXOPLAYER private val exoPlayer: SimpleExoPlayer private val

    exoPlayerBackground: SimpleExoPlayer exoPlayerBackground.prepare( ProgressiveMediaSource.Factory(dataSourceFactory) .createMediaSource(Uri.parse(requireNotNull(currentBackgroundMusic).path)) ) exoPlayer.addListener(object : Player.EventListener { override fun onPlayerStateChanged(playWhenReady: Boolean, playbackState: Int) { when (exoPlayer.playbackState) { Player.STATE_READY -> { if (!exoPlayer.playWhenReady) { exoPlayerBackground.playWhenReady = false } else { exoPlayerBackground.playWhenReady = currentBackgroundMusic != null } } } } }) Synchronize players.
  115. 115 BACKGROUND MUSIC EXOPLAYER private val exoPlayer: SimpleExoPlayer private val

    exoPlayerBackground: SimpleExoPlayer exoPlayerBackground.prepare( ProgressiveMediaSource.Factory(dataSourceFactory) .createMediaSource(Uri.parse(requireNotNull(currentBackgroundMusic).path)) ) exoPlayer.addListener(object : Player.EventListener { override fun onPlayerStateChanged(playWhenReady: Boolean, playbackState: Int) { when (exoPlayer.playbackState) { Player.STATE_READY -> { if (!exoPlayer.playWhenReady) { exoPlayerBackground.playWhenReady = false } else { exoPlayerBackground.playWhenReady = currentBackgroundMusic != null } } } } }) Synchronize players.
  116. 116 BACKGROUND MUSIC EXOPLAYER private val exoPlayer: SimpleExoPlayer private val

    exoPlayerBackground: SimpleExoPlayer exoPlayerBackground.prepare( ProgressiveMediaSource.Factory(dataSourceFactory) .createMediaSource(Uri.parse(requireNotNull(currentBackgroundMusic).path)) ) exoPlayer.addListener(object : Player.EventListener { override fun onPlayerStateChanged(playWhenReady: Boolean, playbackState: Int) { when (exoPlayer.playbackState) { Player.STATE_READY -> { if (!exoPlayer.playWhenReady) { exoPlayerBackground.playWhenReady = false } else { exoPlayerBackground.playWhenReady = currentBackgroundMusic != null } } } } })
  117. 117 BACKGROUND MUSIC FFMPEG ffmpeg -i input1.mp4 -filter_complex [0:v:0]null[video00][0:a:0]anull[audio00] amovie=

    background_music.mp3:loop=0,asepts=PTS-STARTPTS,volume=0.3[outm] [video00][audio00] concat=n=1:v:1:a=1 [outv][outav] [outav][outm]amerge,asepts=PTS-STARTPTS[outa] -map "[outv]" -map "[outa]" output.mp4
  118. 118 BACKGROUND MUSIC FFMPEG ffmpeg -i input1.mp4 -filter_complex [0:v:0]null[video00][0:a:0]anull[audio00] amovie=

    background_music.mp3:loop=0,asepts=PTS-STARTPTS,volume=0.3[outm] [video00][audio00] concat=n=1:v:1:a=1 [outv][outav] [outav][outm]amerge,asepts=PTS-STARTPTS[outa] -map "[outv]" -map "[outa]" output.mp4
  119. 119 BACKGROUND MUSIC FFMPEG ffmpeg -i input1.mp4 -filter_complex [0:v:0]null[video00][0:a:0]anull[audio00] amovie=

    background_music.mp3:loop=0,asepts=PTS-STARTPTS,volume=0.3[outm] [video00][audio00] concat=n=1:v:1:a=1 [outv][outav] [outav][outm]amerge,asepts=PTS-STARTPTS[outa] -map "[outv]" -map "[outa]" output.mp4 Do nothing with video and audio stream of input 0.
  120. 120 03 LAST BUT NOT LEAST

  121. 121 AUDIO FOCUS import com.google.android.exoplayer2.C import com.google.android.exoplayer2.ExoPlayerFactory import com.google.android.exoplayer2.audio.AudioAttributes exoPlayer.setAudioAttributes(

    AudioAttributes.Builder() .setUsage(C.USAGE_MEDIA) .setContentType(C.CONTENT_TYPE_MOVIE).build(), handleAudioFocus = true) C.CONTENT_TYPE_MOVIE //reduce volume while notification is playing C.CONTENT_TYPE_SPEECH //pause player while notification is playing
  122. 122 AUDIO FOCUS import com.google.android.exoplayer2.C import com.google.android.exoplayer2.ExoPlayerFactory import com.google.android.exoplayer2.audio.AudioAttributes exoPlayer.setAudioAttributes(

    AudioAttributes.Builder() .setUsage(C.USAGE_MEDIA) .setContentType(C.CONTENT_TYPE_MOVIE).build(), handleAudioFocus = true) C.CONTENT_TYPE_MOVIE //reduce volume while notification is playing C.CONTENT_TYPE_SPEECH //pause player while notification is playing
  123. 123 AUDIO FOCUS import com.google.android.exoplayer2.C import com.google.android.exoplayer2.ExoPlayerFactory import com.google.android.exoplayer2.audio.AudioAttributes exoPlayer.setAudioAttributes(

    AudioAttributes.Builder() .setUsage(C.USAGE_MEDIA) .setContentType(C.CONTENT_TYPE_MOVIE).build(), handleAudioFocus = true) C.CONTENT_TYPE_MOVIE //reduce volume while notification is playing C.CONTENT_TYPE_SPEECH //pause player while notification is playing
  124. 124 AUDIO FOCUS import com.google.android.exoplayer2.C import com.google.android.exoplayer2.ExoPlayerFactory import com.google.android.exoplayer2.audio.AudioAttributes exoPlayer.setAudioAttributes(

    AudioAttributes.Builder() .setUsage(C.USAGE_MEDIA) .setContentType(C.CONTENT_TYPE_MOVIE).build(), handleAudioFocus = true) C.CONTENT_TYPE_MOVIE //reduce volume while notification is playing C.CONTENT_TYPE_SPEECH //pause player while notification is playing In case of more exoPlayers playing at the same you should mark just one (the main) as audioFocus handler. Otherwise they will affect each other :-)
  125. 125 Hopefully some ideas mentioned here will help or inspire

    you.
  126. Michal Jenicek / michal.jenicek@strv.com THANK YOU!

  127. QUESTIONS