DevFest Florida - Video Processing on Android

An increasing number of social media apps such as Instagram, Snapchat and Messenger are using video features. With modern Android devices having enough processing power to run video processing algorithms locally, there's no need to send videos through expensive internet bandwidth to be processed on the backend. Overlays, transcoding and cropping are just a few of the challenges you face when working with video on-device. In this talk I will go through my experience experimenting with built-in as well as third party libraries for video manipulation on Android. We will look at the benefits of two leading media processing libraries, and how you can leverage them to enhance Android's native MediaCodec API to accomplish these tasks. You will walk away with a head start on how to tackle the most common challenges with videos on Android.

Transcript

1. V I D E O P R O C E S S I N G O N A N D R O I D N A M R ATA B A N D E K A R #DevFestFL @namratacodes
2. W H O A M I • OANDA • raywenderlich.com • Ginger!
3. http://bit.do/NamDevFest S L I D E S #DevFestFL @namratacodes
4. #DevFestFL @namratacodes V I D E O P R O C E S S I N G I N T R O D U C T I O N T O
5. #DevFestFL @namratacodes C O N TA I N E R V I D E O S T R E A M A U D I O S T R E A M DEMUXING DECODING V I D E O F R A M E S A U D I O S A M P L E S
6. #DevFestFL @namratacodes C O N TA I N E R V I D E O F R A M E S A U D I O S A M P L E S ENCODING MUXING V I D E O S T R E A M A U D I O S T R E A M
7. M O B I L E A P P S P O P U L A R U S E I N #DevFestFL @namratacodes
8. C O M M O N F E AT U R E S • Special video effects • Merging videos • Cropping and trimming #DevFestFL @namratacodes
9. M Y S E C O N D J O B ! #DevFestFL @namratacodes
10. W E AT H E R G I F #DevFestFL @namratacodes
11. F E AT U R E L I S T ? #DevFestFL @namratacodes
12. #DevFestFL @namratacodes
13. I O S W E AT H E R G I F #DevFestFL @namratacodes
14. V I D E O E D I T I N G O N D E V I C E #DevFestFL @namratacodes
15. A N D R O I D S U P P O R T #DevFestFL @namratacodes • MediaCodec • MediaExtractor • MediaMuxer
16. #DevFestFL @namratacodes
17. W H AT N E X T ? #DevFestFL @namratacodes
18. F F M P E G #DevFestFL @namratacodes
19. W H Y F F M P E G ? • Swiss Army Knife for video • Filters • Subtitles #DevFestFL @namratacodes
20. F F M P E G O N A N D R O I D • Executable binary • Shared object library #DevFestFL @namratacodes
21. F F M P E G E X E C U TA B L E B U I L D I N G T H E #DevFestFL @namratacodes
22. D E P E N D E N C I E S • autoconf • automake • libtool #DevFestFL @namratacodes
23. G E T T I N G S TA R T E D Step 1: Download and unpack NDK #DevFestFL @namratacodes
24. #DevFestFL @namratacodes G E T T I N G S TA R T E D Step 2: Download FFmpeg source code https://github.com/FFmpeg/FFmpeg
25. #DevFestFL @namratacodes G E T T I N G S TA R T E D Step 3: Download sources for other libraries like yasm and libvpx
26. Step 4: Put FFmpeg and other libraries under ndk/sources #DevFestFL @namratacodes G E T T I N G S TA R T E D
27. C O N F I G U R I N G … #DevFestFL @namratacodes
28. ./configure \ --target-os="$TARGET_OS" \ --arch="$NDK_ABI" \ --sysroot="$NDK_SYSROOT" \ \ --enable-decoders --enable-encoders \ --enable-muxers --enable-demuxers \ --enable-filters \ \ --enable-yasm \ --enable-libx264 \ \ C O N F I G U R I N G #DevFestFL @namratacodes
29. ./configure \ --target-os="$TARGET_OS" \ --arch="$NDK_ABI" \ --sysroot="$NDK_SYSROOT" \ \ --enable-decoders --enable-encoders \ --enable-muxers --enable-demuxers \ --enable-filters \ \ --enable-yasm \ --enable-libx264 \ \ #DevFestFL @namratacodes
30. ./configure \ --target-os="$TARGET_OS" \ --arch="$NDK_ABI" \ --sysroot="$NDK_SYSROOT" \ \ --enable-decoders --enable-encoders \ --enable-muxers --enable-demuxers \ --enable-filters \ \ --enable-yasm \ --enable-libx264 \ \ #DevFestFL @namratacodes
31. ./configure \ --target-os="$TARGET_OS" \ --arch="$NDK_ABI" \ --sysroot="$NDK_SYSROOT" \ \ --enable-decoders --enable-encoders \ --enable-muxers --enable-demuxers \ --enable-filters \ \ --enable-yasm \ --enable-libx264 \ \ #DevFestFL @namratacodes
32. ./configure \ --target-os="$TARGET_OS" \ --arch="$NDK_ABI" \ --sysroot="$NDK_SYSROOT" \ \ --enable-decoder=h263 \ --enable-decoder=h264 \ --enable-decoder=mpeg4 \ --enable-decoder=libvpx_vp8 \ --enable-encoder=libvpx_vp8 \ \ --enable-yasm \ --enable-libx264 \ \ #DevFestFL @namratacodes
33. C O M P I L I N G make -j4 make install #DevFestFL @namratacodes
34. F F M P E G E X E C U TA B L E I N Y O U R A N D R O I D P R O J E C T #DevFestFL @namratacodes
35. C H A N G E P E R M I S S I O N S try { File f = new File(ctx.getDir("bin", 0), "FFmpeg"); if (f.exists()) { f.delete(); } copyRawFile(ctx, R.raw.ffmpeg, f); // Change the permissions String filePath = f.getCanonicalPath(); Runtime.getRuntime().exec("chmod 0755 "+ filePath).waitFor(); } catch (Exception e) { String errorMsg = e.getLocalizedMessage(); Log.e(TAG, "installBinary failed: "+errorMsg); return null; } #DevFestFL @namratacodes
36. private int execProcess(List<String> cmds) { File dir = new File(ffmpegBin).getParentFile(); ProcessBuilder pb = new ProcessBuilder(cmds); pb.directory(dir); Process process = pb.start(); int exitVal = process.waitFor(); return exitVal; } #DevFestFL @namratacodes P R O C E S S B U I L D E R
37. ArrayList<String> cmd = new ArrayList<String>(); cmd.add(mFfmpegBin); cmd.add("-y"); cmd.add("-i"); cmd.add(new File(inputVideo.path).getCanonicalPath()); cmd.add("-vf"); cmd.add("movie="+watermarkImg.path+" [logo]; [in] scale="+width+":"+height+" " +"[scaled]; [scaled] crop=“+newWidth+":"+newHeight+" [cropped]; [cropped][logo] overlay=0:0 [out]"); result.path = outputPath; result.mimeType = "video/mp4"; cmd.add(new File(result.path).getCanonicalPath()); execFFMPEG(cmd, sc); #DevFestFL @namratacodes F I LT E R C H A I N I N G
38. P E R F O R M A N C E #DevFestFL @namratacodes
39. S H A R E D O B J E C T L I B R A RY B U I L D I N G F F M P E G A S A #DevFestFL @namratacodes
40. S E T U P F O R F F M P E G . S O Step 1: JNI project same folder as NDK #DevFestFL @namratacodes
41. LOCAL_PATH := $(call my-dir) include $(CLEAR_VARS) LOCAL_MODULE := videokit ANDROID_LIB := -landroid LOCAL_CFLAGS := -I$(NDK)/sources/ffmpeg LOCAL_SRC_FILES := videokit.c ffmpeg.c cmdutils.c LOCAL_SHARED_LIBRARIES := libavcodec libavutil libavfilter include $(BUILD_SHARED_LIBRARY) $(call import-module,ffmpeg/android/$(CPU)) #DevFestFL @namratacodes S E T U P F O R F F M P E G . S O Step 2: Android.mk file
42. APP_OPTIM := release APP_PLATFORM := $(PLATFORM) APP_ABI := $(ABI) NDK_TOOLCHAIN_VERSION=4.9 APP_STL := stlport_shared #DevFestFL @namratacodes S E T U P F O R F F M P E G . S O Step 3: Application.mk file
43. #DevFestFL @namratacodes S E T U P F O R F F M P E G . S O Step 4: Place native code in app/jni
44. N D K M A G I C ndk-build #DevFestFL @namratacodes
45. public class VideoKit { static { System.loadLibrary("ffmpeg"); } // pass commands to ffmpeg private native int run(int loglevel, String[] args); } L O A D S H A R E D L I B R A RY #DevFestFL @namratacodes
46. W E AT H E R G I F D E M O #DevFestFL @namratacodes
47. #DevFestFL @namratacodes
48. L I C E N S I N G #DevFestFL @namratacodes
49. 1. Compile FFmpeg without “enable- gpl” and without “enable-nonfree” . . . 18. Make sure your program is not using any GPL libraries (notably libx264) #DevFestFL @namratacodes L I C E N S E C O M P L I A N C E
50. —-enable-gpl —-enable-nonfree #DevFestFL @namratacodes L I C E N S E C O M P L I A N C E
51. • H.264 encoding • VP8 • VP9 #DevFestFL @namratacodes L I C E N S E C O M P L I A N C E
52. • Dynamic linking • FFmpeg source • Configure script #DevFestFL @namratacodes L I C E N S E C O M P L I A N C E
53. #DevFestFL @namratacodes L I C E N S E C O M P L I A N C E
54. #DevFestFL @namratacodes L I C E N S E C O M P L I A N C E
55. F F M P E G P R O S • Modular • Support for codecs • Versatile • Well documented #DevFestFL @namratacodes
56. • Licensing • Slow • Large #DevFestFL @namratacodes F F M P E G L I M I TAT I O N S
57. M P 4 PA R S E R #DevFestFL @namratacodes
58. M P 4 PA R S E R F E AT U R E S • Concatenation • Trimming • Muxing • Demuxing #DevFestFL @namratacodes
59. RxMp4Parser.concatenateInto( //The output where the resulting Movie object should be stored output, //Cropped video RxMp4Parser.crop(f, 8.5f, 13f), //The entire video RxMp4Parser.from(f) ) .subscribe(new Action<File>() { @Override public void call(File file) { mProgressDialog.dismiss(); } }, new Action<Throwable>() { @Override public void call(Throwable throwable) { Toast.makeText(MainActivity.this, "Concatenation failed! " + throwable.getMessage(), Toast.LENGTH_SHORT).show(); } } ); M P 4 PA R S E R U S A G E #DevFestFL @namratacodes
60. M P 4 PA R S E R D E M O #DevFestFL @namratacodes
61. #DevFestFL @namratacodes
62. Features\Library F F M P E G R X M P 4 PA R S E R Muxing, Demuxing Trimming, Concatenating Codec support Scaling, Cropping, Overlaying Easy to integrate (Java library, licensing) Performance #DevFestFL @namratacodes
63. R E S O U R C E S • https://github.com/guardianproject/android-ffmpeg • https://github.com/Yelp/ffmpeg-android • https://github.com/inFullMobile/videokit-ffmpeg- android • https://www.ffmpeg.org/legal.html • https://github.com/sannies/mp4parser • https://github.com/TeamWanari/RxMp4Parser #DevFestFL @namratacodes
64. T H A N K S A N D C R E D I T S • Dhaval Giani • Godfrey Nolan • GDG Toronto • Toronto Android Developers Meetup • PyLadies Toronto • OANDA #DevFestFL @namratacodes
65. O A N D A I S H I R I N G ! jobs.oanda.com #DevFestFL @namratacodes
66. T H A N K Y O U ! Q U E S T I O N S ? https://github.com/namrata-b/talks/ #DevFestFL @namratacodes