Play Rock-Paper-Scissors-Spock-Lizard with your Android Things

Transcript

1. @Xebiconfr #Xebicon18 @bonbonking Jouer à Rock-Paper-Scissors-Spock-Lizard avec son Android Things Qian Jin | Technical Officer | qjin@xebia.fr Xebia France 1
2. @Xebiconfr #Xebicon18 @bonbonking Qian Jin Google Developer Expert IoT Technical Officer @ Xebia France @bonbonking 2
3. @Xebiconfr #Xebicon18 @bonbonking Play Rock-Paper-Scissors-Spock-Lizard "Wacky, Whimsical Machinery with cardboard and duct tape" 3
4. @Xebiconfr #Xebicon18 @bonbonking 4
5. @Xebiconfr #Xebicon18 @bonbonking Motivation behind the shoebox project 5
6. @Xebiconfr #Xebicon18 @bonbonking 6 Image Source: https://www.flaticon.com/packs/internet-of-things-45 Cloud Data Centers thousands Fog Nodes millions Edge Devices billions
7. @Xebiconfr #Xebicon18 @bonbonking 7 • Latency • Availability • Bandwidth • Security/Privacy
8. @Xebiconfr #Xebicon18 @bonbonking Explained in xkcd way 8
9. @Xebiconfr #Xebicon18 @bonbonking 9
10. @Xebiconfr #Xebicon18 @bonbonking • Data collection tool (raw data to train the ML model) • Hardware components (robot arms + game control) • Application running on Android Things ◦ Applicative: game rules, play logic and interactions ◦ Artificial Intelligence: recognize opponent's gesture 10
11. @Xebiconfr #Xebicon18 @bonbonking Raw Image Crowdsourcing 11
12. @Xebiconfr #Xebicon18 @bonbonking • Web mobile application: Node.js + Express • Hosting/Deployment: Firebase Hosting / Function / Storage 12
13. @Xebiconfr #Xebicon18 @bonbonking 13
14. @Xebiconfr #Xebicon18 @bonbonking 14
15. @Xebiconfr #Xebicon18 @bonbonking 15 Thank you my fellow Xebians <3
16. @Xebiconfr #Xebicon18 @bonbonking • Raw images data close to reality are hard to obtain • GDPR? Not compliant… • Reusable for future projects • Feed the model training pipeline 16
17. @Xebiconfr #Xebicon18 @bonbonking Build your own robot 17
18. @Xebiconfr #Xebicon18 @bonbonking 18
19. @Xebiconfr #Xebicon18 @bonbonking 19
20. @Xebiconfr #Xebicon18 @bonbonking 20
21. @Xebiconfr #Xebicon18 @bonbonking 21
22. @Xebiconfr #Xebicon18 @bonbonking 22
23. @Xebiconfr #Xebicon18 @bonbonking 23
24. @Xebiconfr #Xebicon18 @bonbonking Pulse Width Modulation (PWM) is a common method used to apply a proportional control signal to an external device using a digital output pin. (e.g. servo motors, LCD display). It is a technique for getting analog results with digital means. • Frequency (expressed in Hz): describes how often the output pulse repeats. • Period is the time each cycle takes and is the inverse of frequency. • Duty cycle (expressed as a percentage) describes the width of the pulse within that frequency window. 24
25. @Xebiconfr #Xebicon18 @bonbonking 25
26. @Xebiconfr #Xebicon18 @bonbonking 26
27. @Xebiconfr #Xebicon18 @bonbonking 27
28. @Xebiconfr #Xebicon18 @bonbonking 28
29. @Xebiconfr #Xebicon18 @bonbonking 29
30. @Xebiconfr #Xebicon18 @bonbonking • Android Things Starter Kit (NXP i.MX7D): ~200€ • Or Raspberry Pi (~40€) + Camera Module (~25€) + Display • 16 Channel Servo Driver (~12€) • Power Supply or battery (~5€) • 5 servo motors (~12€) • Cardboard + woodsticks (~0€) • Glue Gun (~10€) • Electronique jumper + Resistors + LED + Button + Breadboard (~10€) 30
31. @Xebiconfr #Xebicon18 @bonbonking Make your device smarter 31
32. @Xebiconfr #Xebicon18 @bonbonking 32
33. @Xebiconfr #Xebicon18 @bonbonking 33
34. @Xebiconfr #Xebicon18 @bonbonking 34
35. @Xebiconfr #Xebicon18 @bonbonking 35
36. @Xebiconfr #Xebicon18 @bonbonking 36 Paper: 0.80 Scissors: 0.15 Rock: 0.05
37. @Xebiconfr #Xebicon18 @bonbonking 37
38. @Xebiconfr #Xebicon18 @bonbonking 38
39. @Xebiconfr #Xebicon18 @bonbonking 39 Paper: 0.10 Scissors: 0.15 Rock: 0.75
40. @Xebiconfr #Xebicon18 @bonbonking 40 Paper: 0.10 Scissors: 0.15 Rock: 0.75 Prediction Error
41. @Xebiconfr #Xebicon18 @bonbonking 41 Paper: 0.10 Scissors: 0.15 Rock: 0.75 Prediction Error
42. @Xebiconfr #Xebicon18 @bonbonking 42 Paper: 0.10 Scissors: 0.15 Rock: 0.75 Prediction Error
43. @Xebiconfr #Xebicon18 @bonbonking 43 Paper: 0.80 Scissors: 0.15 Rock: 0.05
44. @Xebiconfr #Xebicon18 @bonbonking 44 Paper: 0.20 Scissors: 0.75 Rock: 0.05
45. @Xebiconfr #Xebicon18 @bonbonking 45 Visualisation of Inception v3 Model Architecture
46. @Xebiconfr #Xebicon18 @bonbonking 46 Keep all weights identical except these ones
47. @Xebiconfr #Xebicon18 @bonbonking 47 Image Credit: https://xkcd.com/1987/
48. @Xebiconfr #Xebicon18 @bonbonking 48 Image Credit: https://xkcd.com/1838/
49. @Xebiconfr #Xebicon18 @bonbonking 49
50. @Xebiconfr #Xebicon18 @bonbonking 50
51. @Xebiconfr #Xebicon18 @bonbonking 51
52. @Xebiconfr #Xebicon18 @bonbonking 52 python -m scripts.retrain \ --bottleneck_dir=tf_files/bottlenecks \ --how_many_training_steps=4000 \ --model_dir=tf_files/models/ \ --summaries_dir=tf_files/training_summaries/mobilenet_0.50_224 \ --output_graph=tf_files/handgame_graph.pb \ --output_labels=tf_files/handgame_labels.txt \ --architecture=mobilenet_0.50_224 \ --image_dir=tf_files/gesture_photos
53. @Xebiconfr #Xebicon18 @bonbonking 53 tflite_convert \ --graph_def_file=tf_files/handgame_graph.pb \ --output_file=tf_files/handgame_graph.lite \ --input_format=TENSORFLOW_GRAPHDEF \ --output_format=TFLITE \ --input_shape=1,224,224,3 \ --input_array=input \ --output_array=final_result \ --inference_type=FLOAT \ --input_type=FLOAT
54. @Xebiconfr #Xebicon18 @bonbonking 54
55. @Xebiconfr #Xebicon18 @bonbonking Piece everything together 55
56. @Xebiconfr #Xebicon18 @bonbonking 56
57. @Xebiconfr #Xebicon18 @bonbonking 57
58. @Xebiconfr #Xebicon18 @bonbonking 58 Android SDK Android Studio Play Service Firebase Google Cloud Platform
59. @Xebiconfr #Xebicon18 @bonbonking 59
60. @Xebiconfr #Xebicon18 @bonbonking 60
61. @Xebiconfr #Xebicon18 @bonbonking
62. @Xebiconfr #Xebicon18 @bonbonking 62
63. @Xebiconfr #Xebicon18 @bonbonking The Inter-Integrated Circuit (IIC or I2C) bus connects simple peripheral devices with small data payloads. Sensors and actuators are common use cases for I2C. Examples include accelerometers, thermometers, LCD displays, and motor drivers. I2C is a synchronous serial interface, relies on a shared clock signal to synchronize data transfer between devices. I2C devices connect using a 3-Wire interface consisting of: • SCL: Shared clock signal • SDA: Shared data line • GND: Common ground reference 63
64. @Xebiconfr #Xebicon18 @bonbonking 64
65. @Xebiconfr #Xebicon18 @bonbonking
66. @Xebiconfr #Xebicon18 @bonbonking 66
67. @Xebiconfr #Xebicon18 @bonbonking Live Demo? My robot is still a bit silly... 67
68. @Xebiconfr #Xebicon18 @bonbonking An entire weekend (of burning finger and slapped face)! My robot doesn't respect (yet) Asimov's Three Laws of Robotics 68
69. @Xebiconfr #Xebicon18 @bonbonking Long Live Chifumi robot 69
70. @Xebiconfr #Xebicon18 @bonbonking • Model training pipeline with user feedback & photo upload • Dynamic model downloading (via API or Firebase ML kit) • Better game control • Better UX • Leaderboard 70
71. @Xebiconfr #Xebicon18 @bonbonking 71
72. @Xebiconfr #Xebicon18 @bonbonking • Artificial neural network: https://en.wikipedia.org/wiki/Artificial_neural_network • Deep Learning: https://en.wikipedia.org/wiki/Deep_learning • Convolutional Neural Network: https://en.wikipedia.org/wiki/Convolutional_neural_network • TensorFlow for Poets: https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/ • TensorFlow for Poets 2: Optimize for Mobile: https://codelabs.developers.google.com/codelabs/tensorflow-for-poets-2/ • TensorFlow Glossary: https://www.tensorflow.org/versions/r0.12/resources/glossary • Medium article: Android meets Machine Learning: https://medium.com/xebia-france/android-meets-machine-learning-part-1-from-tensorflow-mobile-li te-to-ml-kit-4c7e6bc8eee3 72
73. @Xebiconfr #Xebicon18 @bonbonking Merci ! Come play against chifumi robot at the stand :) Github Repo: https://github.com/jinqian/at-rock-paper-scissors-lizard-spock