Solving “Where’s Wally?” with Turi Create - Florian Ludot

Solving “Where’s Wally?” with Turi Create - Florian Ludot

Solving “Where’s Wally?” with Turi Create (iOS)

Source code used for my talk: Solving “Where’s Wally?” with Turi Create

This project uses the Machine Learning framework Turi Create in order to solve the game “Where’s Wally?”.

GitHub repository: https://github.com/FlorianLdt/WallyML

The video of the talk can be found here: https://youtu.be/GYGceQlZorE

6c1033d9c6fbc5b25a19d6b534d98fe6?s=128

Florian Ludot

November 29, 2018
Tweet

Transcript

  1. @florianldt 1 Solving “Where’s Wally?” with Turi Create

  2. • Former SAP HR Analyst @ Air France " •

    Co-Founder & iOS Engineer @ Avanssion # $ • First year working on iOS WHOAMI 2 Florian Ludot
  3. 3 Why solving “Where’s Wally?”? @florianldt

  4. 4 “Where’s Wally?”

  5. “Where’s Wally?” & 4

  6. “Where’s Wally?” & 4 Waldo?” '

  7. “Where’s Wally?” & 4 Waldo?” ' ウォーリー?” $

  8. “Where’s Wally?” & 4 Waldo?” ' ウォーリー?” $ ਘܻ?” (

    威利利?” ) Charlie?” " …
  9. “Where’s Wally?” 4

  10. • First Machine Learning project • Always struggled on this

    game… So? • Not a lot of presentations/tutorials about Object Detection (OD) • CreateML does not work for Object Detection • Learning is way more efficient when having fun 5
  11. 6

  12. 7 As the series goes on, Wally progressively becomes harder

    to find. 1st book: 0.99 cm2 2nd book: 0.8 cm2 3rd book: 0.33 cm2 From 4th to 7th books: 0.2 cm2 He has also been surrounded by more other characters, from 225 on the first book's first page to about 850 on the last book's first page.
  13. 8

  14. 9 Introduction to Turi Create @florianldt

  15. https://developer.apple.com/wwdc18/712 10

  16. https://github.com/apple/turicreate 11

  17. You don't have to be a machine learning expert to

    add recommendations, object detection, image classification, image similarity or activity classification to your app. • Easy-to-use: Focus on tasks instead of algorithms • Visual: Built-in, streaming visualizations to explore your data • Flexible: Supports text, images, audio, video and sensor data • Fast and Scalable: Work with large datasets on a single machine • Ready To Deploy: Export models to Core ML for use in iOS, macOS, watchOS, and tvOS apps • Supported Platform: macOS 10.12+, Linux (with glibc 2.12+), Windows 10 (via WSL) “ ” 12
  18. 5 steps for creating Core ML models Task Data Evaluate

    Deploy Model 13
  19. Task - What are you trying to do? • Finding

    Wally 14
  20. • Finding Wally 14 Task - What are you trying

    to do?
  21. Label Images Image Classification Recognize objects within images Object Detection

    Find similar images Image Similarity Create stylized avatar / profile images Style Transfer Personalize choices of users Recommender Detect an activity using sensors Activity Classification Analyze sentiment of messages Text Classifier Predict a label Classifiers Predict numeric values Regression Group similar datapoint together Clustering 15
  22. 15 Label Images Image Classification Recognize objects within images Object

    Detection Find similar images Image Similarity Create stylized avatar / profile images Style Transfer Personalize choices of users Recommender Detect an activity using sensors Activity Classification Analyze sentiment of messages Text Classifier Predict a label Classifiers Predict numeric values Regression Group similar datapoint together Clustering
  23. Data - What data do you need? • A lot

    of Wally’s pictures • The coordinates (x, y, height, width) in pixels of Wally in those pictures 16
  24. 17 Initial method - Finding manually Wally on the original

    images
  25. y: 547px x: 724px width: 39px height: 68px 17

  26. path,annotations images/1.jpg,[{"coordinates": {"height": …, "width": …, "x": …, "y": …},

    "label": “…”}] 17 y: 547px width: 39px height: 68px x: 724px
  27. path,annotations images/1.jpg,[{"coordinates": {"height": 68, "width": 39, "x": 724, "y": 547},

    "label": “waldo”}] 17 y: 547px width: 39px height: 68px x: 724px
  28. • Create an annotations file (csv and json supported) 18

    • 36 images • 43 lines
  29. Unfortunately • A small amount of good data is available

    
 (If the entire picture is small, Wally will be even smaller) • Only 7 “Where’s Wally?” official books have be released Improve the model Deteriorate the model 19
  30. 20 Second method - Creating my own “Where’s Wally?” puzzle

    by cropping original images “If your problem is a that you only have few images, why not creating yours?” - a clever person
  31. 21

  32. 22

  33. 23

  34. 24 …

  35. 24 …

  36. 25 • (original images) = 36 • 1st method =

    • 2nd method (256) = = = 2,304 • 2nd method (256 + 128) = = 11,520 4x( 1024 256 ) 2 4(x(4)2) 4(x(4)2 + x(8)2) x x Method review But there is still one problem…
  37. 26 Third method - Creating my own “Where’s Wally?” puzzle

    by cropping original images and reassembling them “You train the model with small images but the real ones are way bigger, it can possibly explain this issue” - an other clever person
  38. 27 …

  39. 28 …

  40. 29 …

  41. 30 …

  42. 30 …

  43. • (original images) = 36 • 1st method = •

    2nd method (256) = = = 2,304 • 2nd method (256 + 128) = = 11,520 • 3rd method (256) = = 31 4(x(4)2 + x(8)2) x x … before inserting Wally’s face randomly on the image 6.3055 × 1053 4x( 1024 256 ) 2 4(x(4)2) Method review 230416
  44. Model - Create the model • Image folder (10,000 images)

    ✅ • annotations.csv file (10,000 lines) ✅ # • Use Turi Create API to create the model 32
  45. 33 Turi Create is a Python library

  46. Evaluate - Is the model any good? 34

  47. Deploy - Add CoreML model to your app • Create

    the Core ML model in one line • ⚠ include_non_maximum_suppression=False for iOS 11 compatibility 35
  48. Our Core ML Model 36

  49. Simple model integration in Xcode 37

  50. Simple model integration in Xcode 37

  51. 38 WallyMLEngine @florianldt

  52. Overview 39

  53. 40 // Engine States // Enable the engine to start

    with an image of type CIImage or CVPixelBuffer
  54. 41 // Launch the detection request

  55. 42 // Process detection request result

  56. 43 // Set proper WallyMLEngine.State return value

  57. 44 // Intersection-over-Union // Non-maximum suppression algorithm to remove multiple

    similar predictions associated with a single object instance.
  58. In action 45

  59. 0 Demo 0 46 @florianldt

  60. Wrap up • Turi Create is a really great and

    easy to use ML Framework where no ML skills are required • Easy integration with CoreML • Good result despite a difficult problem • Provide a handy set of visualization tools • Data preparation is key and can be very time consuming • No model retraining feature • No way to output a model layer 47
  61. Where to go from here • WallyML is open source:

    https://github.com/FlorianLdt/WallyML • Compare with other ML frameworks like TensorFlow, Keras, … • Using the “pure negative” technique may improve model’s precision • Have fun 48
  62. 48 https://github.com/FlorianLdt/WallyML

  63. 48 https://github.com/FlorianLdt/DeepLearningImageGenerator

  64. 48 https://github.com/FlorianLdt/TFannotations2TC

  65. Where to go from here 48 • WallyML is open

    source: https://github.com/FlorianLdt/WallyML • Compare with other ML frameworks like TensorFlow, Keras, … • Using the “pure negative” technique may improve model’s precision • Have fun
  66. References • Turi Create documentation: 
 https://github.com/apple/turicreate • Turi Create

    (Object Detection): 
 https://apple.github.io/turicreate/docs/userguide/object_detection/ • Python Pillow Library:
 https://github.com/python-pillow/Pillow • Repository containing "Where's Wally?" images for training: 
 https://github.com/tadejmagajna/HereIsWally 49
  67. Further reading • Finding Waldo Using Semantic Segmentation & Tiramisu:

    
 https://hackernoon.com/wheres-waldo-terminator-edition-8b3bd0805741 • Tensorflow project that includes a model for solving Where's Wally puzzles: 
 https://github.com/tadejmagajna/HereIsWally • How to Find Wally with a Neural Network: 
 https://towardsdatascience.com/how-to-find-wally-neural-network- eddbb20b0b90 • How Turi Create is Disrupting the Machine Learning Landscape: 
 https://towardsdatascience.com/how-turi-create-is-disrupting-the- machine-learning-landscape-37b562f01eab 50
  68. • WallyML: https://github.com/FlorianLdt/WallyML • Github: https://github.com/FlorianLdt • Twitter: @florianldt Thank

    you 51