Supersonic image recognition
for your mobile apps
Cédric @deltheil
Paris Tech Talks #6
Moodstocks
Tech co-founder
Slide 2
Slide 2 text
Mobile Image Recognition
Your Mobile App
Reference Images + ID-s
Image Recognition Platform
query
match ID = 1234
index
user
developer
ID = 1234
Slide 3
Slide 3 text
The old way: thin client
Mobile App
Image Recognition API
client
server
reference images
send JPEG image
via HTTP
match ID or nil
Slide 4
Slide 4 text
Drawback: network latencies!
Slide 5
Slide 5 text
Our approach: thick client
Mobile App
Moodstocks API
client
server
reference images
up to millions images
Moodstocks SDK
image signatures
up to 10k images
query match ID or nil
server-side search
for large DB
sync
Slide 6
Slide 6 text
Benefit: speed
a typical search runs in a few ms
Slide 7
Slide 7 text
Moodstocks SDK
Mobile App
Wrapper
Core Library
Objective-C
Java
C
Slide 8
Slide 8 text
Moodstocks SDK
Mobile App
Wrapper
Core Library
we will now focus
on this part
Slide 9
Slide 9 text
Design Goal #1
Portability
• core library written in C99
• +99% code in common between iOS / Android
• e.g supporting Intel CPU-s on Android was… no effort!
Slide 10
Slide 10 text
Design Goal #2
Efficiency
• in-memory data structures for maximum speed
• leverage NEON capabilities (SIMD) where applicable
• a typical search runs in ~ 30 ms / 10k images (iPhone 5S)
Slide 11
Slide 11 text
Design Goal #3
Low footprint
• 100% home made, lightweight implementation
• implement & embed only what we need!
• around 800 kB for the iOS ARMv7s release build (-Os)
Slide 12
Slide 12 text
in 1/2 day
Our choices pay off
Slide 13
Slide 13 text
Core Library Architecture
Public Header File
Internal Modules
3rd Party Libraries
Image
Recognition
Pipeline
Slide 14
Slide 14 text
Public Header File
Internal Modules
3rd Party Libraries
Image
Recognition
Pipeline
Slide 15
Slide 15 text
Image Recognition Pipeline
• full-stack: the whole recognition occurs on-device!
• robust to camera noises (blur, lighting, …)
• supports 1D and 2D barcodes too
Image Recognition Pipeline
Slide 16
Slide 16 text
Public Header File
Internal Modules
3rd Party Libraries
Image
Recognition
Pipeline
Slide 17
Slide 17 text
Internal Modules (1/4)
• master-slave sync engine for image signatures
• efficient: works with diff-s & data bundles (less requests)
• fail-safe: retries + the client always stays consistent
Sync Storage
Logs API client
Slide 18
Slide 18 text
Internal Modules (2/4)
Sync Storage
Logs API client
• collects search results & metadata (OS, etc) along time
• a long-running thread posts batches every 30s
• tolerant to network errors -> HTTP 409 (Already Exists)
Slide 19
Slide 19 text
Internal Modules (3/4)
• persists image signatures on-disk
• persists search results events on-disk
• used as a temp on-disk buffer at sync time
Sync Storage
Logs API client
Slide 20
Slide 20 text
Internal Modules (4/4)
• used for server-side recognition (send image via HTTP)
• includes a pool to re-use connections
• also used to post logs batches to the server (HTTP)
Sync Storage
Logs API client
Slide 21
Slide 21 text
Public Header File
Internal Modules
3rd Party Libraries
Image
Recognition
Pipeline
Slide 22
Slide 22 text
3rd Party Libraries (1/4)
libcurl Tokyo Cabinet jpec msgpack
• the +500M users file transfer library
• multi API -> multiple transfers in the same thread
• we use it as the backend of our sync engine + HTTP reqs.
Slide 23
Slide 23 text
3rd Party Libraries (2/4)
libcurl Tokyo Cabinet jpec msgpack
• key / value embedded database (hash / table / B+tree)
• … and also a fantastic C library (see tcutil.h)
• we use it as the backend of our storage engine
Slide 24
Slide 24 text
3rd Party Libraries (3/4)
libcurl Tokyo Cabinet jpec msgpack
• our home-made, tiny JPEG encoder (600 LOC-s)
• we open sourced it -> github.com/Moodstocks/jpec
• we use it for server-side requests (send image)
Slide 25
Slide 25 text
3rd Party Libraries (4/4)
libcurl Tokyo Cabinet jpec msgpack
• efficient binary serialization format
• we use it to pack the data we synchronize
Slide 26
Slide 26 text
Thanks!
Questions? Comments? cedric AT moodstocks DOT com | @deltheil
Slide 27
Slide 27 text
Building
• we use simple Makefile-s and shell scripts
• we cross-compile w/ the iOS and Android toolchains
• we combine all the arch-s into a fat static lib (armv7, …)
Slide 28
Slide 28 text
Testing
• we write unit tests with ct (Easy Unit Testing) by @krarick
• we run them on device! e.g jailbreaked iPhone + openssh
• we use a mock to test the sync engine
Slide 29
Slide 29 text
Quality
• no warnings: we compile with -Wall -Werror
• static analysis with Clang Static Analyzer
• memory leak checks with Valgrind