Solving “Where’s Wally?” with Turi Create - Florian Ludot

Solving “Where’s Wally?” with Turi Create - Florian Ludot

Solving “Where’s Wally?” with Turi Create (iOS)

Source code used for my talk: Solving “Where’s Wally?” with Turi Create

This project uses the Machine Learning framework Turi Create in order to solve the game “Where’s Wally?”.

GitHub repository: https://github.com/FlorianLdt/WallyML

The video of the talk can be found here: https://youtu.be/GYGceQlZorE

6c1033d9c6fbc5b25a19d6b534d98fe6?s=128

Florian Ludot

November 29, 2018
Tweet

Transcript

  1. 2.

    • Former SAP HR Analyst @ Air France " •

    Co-Founder & iOS Engineer @ Avanssion # $ • First year working on iOS WHOAMI 2 Florian Ludot
  2. 10.

    • First Machine Learning project • Always struggled on this

    game… So? • Not a lot of presentations/tutorials about Object Detection (OD) • CreateML does not work for Object Detection • Learning is way more efficient when having fun 5
  3. 11.

    6

  4. 12.

    7 As the series goes on, Wally progressively becomes harder

    to find. 1st book: 0.99 cm2 2nd book: 0.8 cm2 3rd book: 0.33 cm2 From 4th to 7th books: 0.2 cm2 He has also been surrounded by more other characters, from 225 on the first book's first page to about 850 on the last book's first page.
  5. 13.

    8

  6. 17.

    You don't have to be a machine learning expert to

    add recommendations, object detection, image classification, image similarity or activity classification to your app. • Easy-to-use: Focus on tasks instead of algorithms • Visual: Built-in, streaming visualizations to explore your data • Flexible: Supports text, images, audio, video and sensor data • Fast and Scalable: Work with large datasets on a single machine • Ready To Deploy: Export models to Core ML for use in iOS, macOS, watchOS, and tvOS apps • Supported Platform: macOS 10.12+, Linux (with glibc 2.12+), Windows 10 (via WSL) “ ” 12
  7. 21.

    Label Images Image Classification Recognize objects within images Object Detection

    Find similar images Image Similarity Create stylized avatar / profile images Style Transfer Personalize choices of users Recommender Detect an activity using sensors Activity Classification Analyze sentiment of messages Text Classifier Predict a label Classifiers Predict numeric values Regression Group similar datapoint together Clustering 15
  8. 22.

    15 Label Images Image Classification Recognize objects within images Object

    Detection Find similar images Image Similarity Create stylized avatar / profile images Style Transfer Personalize choices of users Recommender Detect an activity using sensors Activity Classification Analyze sentiment of messages Text Classifier Predict a label Classifiers Predict numeric values Regression Group similar datapoint together Clustering
  9. 23.

    Data - What data do you need? • A lot

    of Wally’s pictures • The coordinates (x, y, height, width) in pixels of Wally in those pictures 16
  10. 26.

    path,annotations images/1.jpg,[{"coordinates": {"height": …, "width": …, "x": …, "y": …},

    "label": “…”}] 17 y: 547px width: 39px height: 68px x: 724px
  11. 27.

    path,annotations images/1.jpg,[{"coordinates": {"height": 68, "width": 39, "x": 724, "y": 547},

    "label": “waldo”}] 17 y: 547px width: 39px height: 68px x: 724px
  12. 29.

    Unfortunately • A small amount of good data is available

    
 (If the entire picture is small, Wally will be even smaller) • Only 7 “Where’s Wally?” official books have be released Improve the model Deteriorate the model 19
  13. 30.

    20 Second method - Creating my own “Where’s Wally?” puzzle

    by cropping original images “If your problem is a that you only have few images, why not creating yours?” - a clever person
  14. 31.

    21

  15. 32.

    22

  16. 33.

    23

  17. 34.
  18. 35.
  19. 36.

    25 • (original images) = 36 • 1st method =

    • 2nd method (256) = = = 2,304 • 2nd method (256 + 128) = = 11,520 4x( 1024 256 ) 2 4(x(4)2) 4(x(4)2 + x(8)2) x x Method review But there is still one problem…
  20. 37.

    26 Third method - Creating my own “Where’s Wally?” puzzle

    by cropping original images and reassembling them “You train the model with small images but the real ones are way bigger, it can possibly explain this issue” - an other clever person
  21. 38.
  22. 39.
  23. 40.
  24. 41.
  25. 42.
  26. 43.

    • (original images) = 36 • 1st method = •

    2nd method (256) = = = 2,304 • 2nd method (256 + 128) = = 11,520 • 3rd method (256) = = 31 4(x(4)2 + x(8)2) x x … before inserting Wally’s face randomly on the image 6.3055 × 1053 4x( 1024 256 ) 2 4(x(4)2) Method review 230416
  27. 44.

    Model - Create the model • Image folder (10,000 images)

    ✅ • annotations.csv file (10,000 lines) ✅ # • Use Turi Create API to create the model 32
  28. 47.

    Deploy - Add CoreML model to your app • Create

    the Core ML model in one line • ⚠ include_non_maximum_suppression=False for iOS 11 compatibility 35
  29. 53.

    40 // Engine States // Enable the engine to start

    with an image of type CIImage or CVPixelBuffer
  30. 57.

    44 // Intersection-over-Union // Non-maximum suppression algorithm to remove multiple

    similar predictions associated with a single object instance.
  31. 60.

    Wrap up • Turi Create is a really great and

    easy to use ML Framework where no ML skills are required • Easy integration with CoreML • Good result despite a difficult problem • Provide a handy set of visualization tools • Data preparation is key and can be very time consuming • No model retraining feature • No way to output a model layer 47
  32. 61.

    Where to go from here • WallyML is open source:

    https://github.com/FlorianLdt/WallyML • Compare with other ML frameworks like TensorFlow, Keras, … • Using the “pure negative” technique may improve model’s precision • Have fun 48
  33. 65.

    Where to go from here 48 • WallyML is open

    source: https://github.com/FlorianLdt/WallyML • Compare with other ML frameworks like TensorFlow, Keras, … • Using the “pure negative” technique may improve model’s precision • Have fun
  34. 66.

    References • Turi Create documentation: 
 https://github.com/apple/turicreate • Turi Create

    (Object Detection): 
 https://apple.github.io/turicreate/docs/userguide/object_detection/ • Python Pillow Library:
 https://github.com/python-pillow/Pillow • Repository containing "Where's Wally?" images for training: 
 https://github.com/tadejmagajna/HereIsWally 49
  35. 67.

    Further reading • Finding Waldo Using Semantic Segmentation & Tiramisu:

    
 https://hackernoon.com/wheres-waldo-terminator-edition-8b3bd0805741 • Tensorflow project that includes a model for solving Where's Wally puzzles: 
 https://github.com/tadejmagajna/HereIsWally • How to Find Wally with a Neural Network: 
 https://towardsdatascience.com/how-to-find-wally-neural-network- eddbb20b0b90 • How Turi Create is Disrupting the Machine Learning Landscape: 
 https://towardsdatascience.com/how-turi-create-is-disrupting-the- machine-learning-landscape-37b562f01eab 50