When digital meets physical world - Applying Computer Vision to reduce paperwork

Desde OnTruck estamos transformando el sector del transporte por carretera en Europa aplicando soluciones distintas a todo lo que existía hasta el momento. Es por ello que nos encontramos con problemas particulares que no tienen soluciones de referencia. En esta charla queremos hablar de cómo solucionamos uno de estos problemas integrando Computer Vision en nuestras apps.

Presentación en el marco del Madrid Android Developer Group (MADG)

Ponentes
Álvar Montes
App Developer desde 2009. Ha participado en más de 50 apps con diferentes roles: desde Project Manager en proyectos de banca (Santander) y telecomunicaciones (Telefónica) hasta Desarrollador Indie (iPlaya) pasando por Mobile Team Lead (La Nevera Roja, Food Panda y OnTruck).

Ha intentado hacerte más fácil aparcar en Madrid con Apparcamos, donde fue Cofounder & CTO y actualmente está trabajando para cambiar el transporte de mercancías en Europa con OnTruck.

Iván Martínez
Iván Martínez lleva desarrollando apps Android desde 2010, y actualmente forma parte del equipo de ingeniería de OnTruck. Empezó creando apps y juegos en su propia empresa para más tarde pasar a formar parte del equipo de Android devs de bq. También hace sus pinitos en IoT y Machine Learning.

Transcript

2. Sponsored by ONTRUCK
4. #MADGvision
5. ONTRUCK When digital meets the physical world Applying Computer Vision to reduce paperwork Álvar Montes - Mobile Team Lead Iván Martínez - Android Engineer
6. Index A bit of context The paper problem The solution OpenCV Results and nexts steps What is OnTruck From 0 to 1000 When digital meets the physical world Integration Implementation Performance #MADGvision
7. OnTruck A bit of context 1
8. Servicio sencillo para empresas y transportistas Driver does the job 97% punctuality Client requests shipping It takes under 2 minutes The offer is sent to drivers 1.600 vehicles in our platform Driver accepts job 98% acceptance rate 18 seconds to assign a job #MADGvision
11. From 0 to 1000
12. Our first version… was not an app Do we need to build an App? Or can we use a Telegram channel as App V0? #MADGvision
13. The beginning is always the hardest A very small team A lot of unknowns Everything had to be built #MADGvision
14. OnTruck Android App Today
15. Now is time to build… but very close to the user
17. The paper problem 2
18. When digital meets the physical world POD - The document that certifies the delivery of an order. - It must be signed by the recipient of the load. - If it’s not delivered or is not legible, our Ops people will have to do some manual work. - Automating everything is key. #MADGvision
19. We have a problem ✓ ✗ #MADGvision
20. How bad is it? Applying Machine Learning to reduce paperwork on the truck industry https://nerds.ontruck.com #MADGvision
21. First iterations, little results #MADGvision
22. From instructing to helping: multiple possible solutions Machine learning 3rd party doc. scanning SDKs Computer vision #MADGvision
23. Machine learning PROs CONs ✓ ✗ We already had a trained ML model. ✗ Increases the size of the apps. Only helps with pictures already taken. ✗ It’s not a help but an evaluation. ✓ We had already validated this option. #MADGvision
24. 3rd party doc. scanning SDKs PROs CONs ✗ Cost money. Not cheap. All work with custom pricing, but it is around 10K€ /year. Adds a new external dependency to our system. ✗ They maintain their own camera allowing customization of the in-camera experience. ✓ The corner detection is done in real time during the camera experience. ✓ #MADGvision
25. Computer vision PROs CONs ✗ No real time recognition. The corner detection is done after the photo is taken. ✗ Increases the size of the apps. It works! ✓ Tackling the problem this way, we really help the driver to take better pictures. ✓ #MADGvision
26. Our decision - Validating focus and brightness of each picture. - Finding image corners. - Cropping and adjusting image based on selected corners. - Improving image quality. #MADGvision
27. The solution OpenCV 3
28. OpenCV Project Open Source Computer Vision Library - https://github.com/opencv/opencv - It’s free for commercial use - Written in C++ - C++, C, Python, Java and MATLAB interfaces Documentation - https://opencv.org/platforms/android/ - Section “Android Development with OpenCV” - A bit outdated... #MADGvision
29. Use cases Image Enhancement #MADGvision
30. Use cases Image Restoration #MADGvision
31. Use cases Image/Video object detection #MADGvision
32. Use cases Real-time gesture recognition #MADGvision
33. Integration in Android app
34. OpenCV4Android SDK - https://sourceforge.net/projects/opencvlibrary/files/opencv-android/ - Samples, doc… - SDK with two parts: - Java wrapper - Native libs for each architecture #MADGvision
35. Integration Dependency type: - External (maven) - Library module Initialization strategy: - Async - Static Two decisions to be made. #MADGvision
36. Dependency type: External (maven) - https://github.com/steveliles/opencv-android - Quick to validate your idea - Unofficial ✓ Fast ✗ Fixed OpenCV version ✗ Not intended for production #MADGvision
37. Dependency type: Library Module 1. Create opencv android lib module in your project 2. Download and extract OpenCV4Android SDK 3. Copy contents of sdk/java/src into opencv module src/main/java 4. Copy contents of sdk/native/libs into opencv module src/main/jniLibs (only for static initialization) 5. Copy sdk/java/org/opencv/engine/OpenCVEngineInterface.aidl into opencv module src/main/aidl/org/opencv/engine/ ✓ Choose OpenCV version ✗ Another module #MADGvision
38. Integration options Dependency type: - External (maven) - Library module Initialization strategy: - Async - Static Two decisions to be made. ✓ #MADGvision
39. Initialization strategy: Async - Java-only OpenCV dependency - External APK manages the native libs: OpenCV Manager ✓ Smaller APK ✓ Target-specific native libs ✗ User experience ✗ Extra APK #MADGvision
40. Initialization strategy: Static - Java + Native OpenCV dependency - Stand-alone solution ✓ Great UX ✗ Decide architecture support ✗ Bigger APK #MADGvision
41. Initialization strategy: Static - Architectures support - Avoid multi-apk (releases hell) - Supporting armeabi-v7a and x86 is enough - You can check on Google Play console - ScanBot and Anyline did the same choice - In your app’s build.gradle: #MADGvision
42. Initialization strategy: Static - APK impact Integrated version - OpenCV 3.1.0 Disk impact - 8.7 MB Method impact - 900 methods #MADGvision
43. Integration options Dependency type: - External (maven) - Library module Initialization strategy: - Async - Static Two decisions to be made. ✓ ✓ #MADGvision
44. Implementation
45. The code - ImageProcessor class - Initialize OpenCV within the constructor - Set of methods implemented using OpenCV Java wrapper - We invoke them on a background thread - Use of OpenCV is exactly the same as on iOS - Focus and Brightness thresholds come from the BE #MADGvision
46. The code #MADGvision
47. The code #MADGvision
48. The code #MADGvision
49. The code #MADGvision
50. Performance
51. Performance: tips - https://opencv.org/platforms/android/android-best-practices.html - Avoid excessive memory allocations (MAT creation) - Initialize MATs with size and depth when possible - Beware the loops - Avoid processing big images - Free resources ASAP: MAT.release( ) #MADGvision
52. Performance: profile Focus, brightness, corners Crop & enhance - Use Android Studio profiler - Check that the native memory is freed after use #MADGvision
53. Results and next steps 4
54. The data #MADGvision
55. Conclusions Apps are difficult: The Digital World meets The Real World We need tools to measure Be Lean. Small iterations done right will help Technical research time is key to validate the solutions OpenCV is a great tool with a lot of possibilities #MADGvision
56. The future? Tensor Flow model with MobileNet: 4 Mb Fast. Optimized for Mobile Devices Already integrated on an Android app Similar accuracy to Caronte #MADGvision
57. We are looking for Android Engineers! Email us at alvar@ontruck.com - ivan@ontruck.com