GDG DevFest 2019 - "Deep Neural Networks for Video Applications"

Deep neural networks can be used to convert real-time video into actionable data which can be used to trigger real-time anomaly alerts and optimize complex business processes.

In addition to commercial applications, deep learning can be used to analyze large amounts of video recorded from the point of view of animals to study complex behavior patterns otherwise impossible to analyze.

This talk will give an intuition for how deep neural networks work and anyone can get started using them for video applications. Several real-world examples will be presented with code examples in python and detail on how to train and deploy models using Google Cloud Platform

Feel free to ask me questions on Twitter: @alxcnwy

#numberboost #deeplearning #machinelearning #deeplearningforvideo #convolutionalneuralnetworks #recurrentneuralnetworks #centroidtracking #objectdetection #deepfakes #poseestimation #videomachinelearning #numberboost

Talk details here: https://devfest.co.za/schedule/2019-11-30?sessionId=412

Transcript

1. DEEP NEURAL NETWORKS Alex Conway alex @ numberboost.com Google Developer Group Cape Town #DevFest 2019 Neither confidential nor proprietary - please distribute ;) for Video Applications
2. Note about these slides The original version of this presentation is over 500 MB (lots of videos) which is too big to upload but there are links to most of the videos in this compressed PDF version * *you might have to download the PDF to click on the links
3. 2016 MultiChoice Innovation Competition 1st Prize Winners 2017 Mercedes-Benz Innovation Competition 1st Prize Winners 2018 Lloyd’s Register Innovation Competition 1st Prize Winners 2019 NTT & Dimension Data Innovation Competition 1st Prize Winners
4. HANDS UP!
5. https://www.youtube.com/watch?v=Gz0QZP2RKWA
9. https://twitter.com/goodfellow_ian/status/1084973596236144640
10. 10 https://twitter.com/quasimondo/status/1100016467213516801
11. https://twitter.com/jonathanfly/status/1185022363848658945?s=21
12. https:/ /arxiv.org/abs/1508.06576 CONTENT IMAGE STYLE IMAGE STYLE TRANSFER OUTPUT + =
13. https://github.com/junyanz/CycleGAN 13
14. https://news.developer.nvidia.com/ai-can-transform-anyone-into-a-professional-dancer/
15. https://motherboard.vice.com/en_us/article/gydydm/gal-gadot-fake-ai-porn
16. https://www.youtube.com/watch?v=MVBe6_o4cMI
17. https://twitter.com/XHNews/status/1098173090448629760
18. https://www.youtube.com/watch?v=EkfnjAeHFAk
20. https://www.youtube.com/watch?v=aE1kA0Jy0Xg
21. https://www.youtube.com/watch?v=xhp47v5OBXQ
23. https://twitter.com/x0rz/status/1104744170529439744
39. f (video) = useful data
41. f (video) = clip label
42. f (video) = frame label
43. f (video) = object count
44. f (video) = object activity
45. f (video) = object poses
46. f (video) = facial expressions
47. f (video) = higher res video
48. f (video) = video with new faces
49. Neural Networks Crash Course
50. Basically watch this incredible video series by 3blue1brown: www.3blue1brown.com/neural-networks
60. input output Neural Network
61. http://playground.tensorflow.org http://playground.tensorflow.org
62. . ⋯ . ⋮ ⋱ ⋮ . ⋯ . . . . … . 13.37 . ⋯ . ⋮ ⋱ ⋮ . ⋯ . . ⋯ . ⋮ ⋱ ⋮ . ⋯ .
63. width 28px Height 28px
65. image tensor 500 x 500 x 3 = 750’000 3-dimensions: width, height, colour
66. video tensor (60 second clip @ 24fps) 500 x 500 x 3 x 24 x 60 = 1’080’000’000 4-dimensions: width, height, colour, time
67. “How much error increases when we increase this weight” HOW DOES A NEURAL NETWORK LEARN? New weight = Old weight Learning rate - Gradient of Error with respect to Weight ( ) x
68. GRADIENT DESCENT http://scs.ryerson.ca/~aharley/neural-networks/
75. Convolutional Neural Networks (CNNs)
76. https://github.com/fchollet/keras/blob/master/examples/mnist_cnn.py (99.25% test accuracy in 192 seconds and 46 lines of code)
78. Convolutional Neural Network 1. Local receptive fields 2. Shared weights 3. Subsampling
79. http://setosa.io/ev/image-kernels
80. 80 http://deeplearning.net/software/theano/tutorial/conv_arithmetic.html
81. 81 VGGNet
83. 83 Zeiler, M.D. and Fergus, R., 2014, September. Visualizing and understanding convolutional networks. In European conference on computer vision (pp. 818-833).
84. 84 Zeiler, M.D. and Fergus, R., 2014, September. Visualizing and understanding convolutional networks. In European conference on computer vision (pp. 818-833).
85. Convolutional Nets Learn Hierarchical Features 85
86. SUBSAMPLING aka “POOLING” 86
87. 87 VGGNet
89. we need labelled training data
90. 14,197,122 images, 21841 synsets indexed ILSVRC: 1‘200‘000 images, 1000 categories ImageNet
92. 92 ImageNet
93. 93 ImageNet
94. IMAGENET TOP-5 ERROR RATE Traditional Image Processing Methods AlexNet 8 Layers ZFNet 8 Layers GoogLeNet 22 Layers ResNet 152 Layers SENet Ensamble TSNet Ensamble
96. 96 https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html
97. = f ) CNN ( class probabilities . “ ” . “ ” . “ ” … . “” 1000 rows
98. = f ) ( . “ ” . “ ” . “ ” … . “”
99. = f ) ( “ ” with probability 0.5037
100. Example: Use CNN to Classify Product Images 100 https://github.com/alexcnwy/ DeepLearning4ComputerVision
101. https://www.youtube.com/watch?v=X4Q6C915sUY
103. TRANSFER LEARNING
104. Using a CNN as a Feature Extractor Feature Extractor (“ENCODER”) Classifier
105. Extracting Features from an Image
106. = f ) CNN ( feature vector . . . … . “encoding”
107. Using a CNN as a Feature Extractor Feature Extractor (“ENCODER”) Classifier
108. Adding a New Classifier
109. Fine-tuning A CNN To Solve A New Problem 96.3% accuracy in under 2 minutes for classifying products into categories (WITH ONLY 3467 TRAINING IMAGES!!1!)
110. https://www.pyimagesearch.com/2019/06/03/fine- tuning-with-keras-and-deep-learning/
111. IMAGE & VIDEO MODERATION TODO 111
112. Object Detection
114. List of bounding boxes + classes = f ) , , , , , , , , … , , , , Object Detection Model (
119. https://www.youtube.com/watch?v=VOC3huqHrss
120. 1.5 million object instances 80 object categories http://cocodataset.org
121. DEMO (HOLD THUMBS)
123. https://towardsdatascience.com/how-to-train-your-own-object- detector-with-tensorflows-object-detector-api-bec72ecfe1d9
124. https://github.com/tensorflow/models/blob/master/research /object_detection/g3doc/detection_model_zoo.md
125. https://github.com/tzutalin/labelImg CUSTOM OBJECT DETECTION
133. CNN … P(A) = 0.005 P(B) = 0.002 P(C) = 0.98 P(9) = 0.001 P(0) = 0.03
144. https://www.reddit.com/r/southafrica/comments/asl4n5/when_a_l ittle_is_just_not_enough/
146. Passenger Counting Person Tracking Vanish in Door? Person Detector Door Detector Passenger Count Video Frame Stream
147. ID #1 OBJECT TRACKING ID #2 https://www.pyimagesearch.com/2018/07/23/simple-object-tracking-with-opencv/ Start by assigning each object in first frame an ID Centroid shown as dot in center of each object
148. OBJECT TRACKING Objects in frame t shown in green, Objects in frame t+1 shown in red
149. OBJECT TRACKING For each object in frame t+1, compute the Euclidean distance between its centroid and the centroid of every object in frame t
150. OBJECT TRACKING Assign object in frame t+1 the ID of nearest object from frame t provided distance less than threshold distance ID #1 ID #1 ID #2 ID #2
151. ID #1 ID #2 ID #1 ID #2 ID #3 OBJECT TRACKING If no object from frame t-1 within threshold distance then assign new ID
152. https://www.youtube.com/watch?v=FfU22I-_dI4
153. https://www.youtube.com/watch?v=NW-rXqCl7us
154. Recurrent Neural Networks (RNNs)
156. CNN CNN CNN Rugby Rugby CNN CNN CNN Soccer Soccer Rugby Rugby
157. Video Clip Model = Frame CNN + Prediction Voting Rugby
159. CNN CNN CNN Rugby Rugby CNN CNN CNN Rugby Rugby Rugby Rugby RNN RNN RNN RNN RNN RNN
160. CNN Encoder Rugby Rugby Rugby Rugby Rugby Rugby RNN RNN RNN RNN RNN RNN
161. CNN Encoder Rugby Rugby Rugby Rugby Rugby Rugby RNN RNN RNN RNN RNN RNN
162. CNN Encoder Pass Run Catch Run Run Score RNN RNN RNN RNN RNN RNN
163. 163 http://colah.github.io/posts/2015-08-Understanding-LSTMs/ Recurrent Neural Networks (RNNs)
164. 164 Recurrent Neural Networks (RNNs) http://colah.github.io/posts/2015-08-Understanding-LSTMs/
165. feature vector = f ) . . . … . . . . … . . . . … . . . . … . , , , … RNN ( CNN frame encodings clip / frame prediction
167. Frame CNN model test accuracy = 71% Video CNN + RNN model test accuracy = 93%
170. https://i.imgur.com/mGXdpdp.gifv
172. Frame-level Action Recognition (7 classes)
173. Frame CNN model test accuracy = 51% Video CNN + RNN model test accuracy = 87%
175. 175 https://github.com/alxcnwy/Deep-Neural- Networks-for-Video-Classification
176. MORE (CRAZY) APPLICATIONS
177. XXX 177 https://www.youtube.com/watch?v=UeheTiBJ0Io VIDEO Q&A
178. XXX 178 https://www.youtube.com/watch?v=UeheTiBJ0Io VIDEO Q&A
179. 179 https://www.youtube.com/watch?v=UeheTiBJ0Io VIDEO Q&A
181. XXX 181 https://www.youtube.com/watch?v=UeheTiBJ0Io VIDEO Q&A
182. Pose Estimation https://www.youtube.com/watch?v=pW6nZXeWlGM
183. https://www.youtube.com/watch?v=pW6nZXeWlGM Pose Estimation
184. https://www.youtube.com/watch?v=pW6nZXeWlGM Pose Estimation
185. https://www.youtube.com/watch?v=pW6nZXeWlGM Pose Estimation
186. https://github.com/CMU-Perceptual-Computing-Lab/openpose
187. https://www.affectiva.com/product/affectiva- automotive-ai-for-driver-monitoring-solutions/ Distracted Driving Detection
188. Few-Shot Adversarial Learning of Realistic Neural Talking Head Models https://www.youtube.com/watch?v=p1b5aiTrGzY
189. https://www.youtube.com/watch?v=p1b5aiTrGzY Few-Shot Adversarial Learning of Realistic Neural Talking Head Models
190. Network 1: CNN embedder compresses faces & landmarks to vector Few-Shot Adversarial Learning of Realistic Neural Talking Head Models
191. Network 2: Generator takes landmarks and synthesizes photo Few-Shot Adversarial Learning of Realistic Neural Talking Head Models
192. Network 3: Discriminator learns to tell apart real and synthesized photos Few-Shot Adversarial Learning of Realistic Neural Talking Head Models
193. https://www.youtube.com/watch?v=8AZBuyEuDqc
194. REMEMBER
197. f (input) = output
198. f (video) = useful data
199. f (video) =
200. Producers vs. Consumers
201. Deep Learning Indaba http://www.deeplearningindaba.com Jeremy Howard & Rachel Thomas http://course.fast.ai Andrej Karpathy’s Class on Computer Vision http://cs231n.github.io Richard Socher’s Class on NLP (great RNN resource) http://web.stanford.edu/class/cs224n/ Keras docs https://keras.io/ GREAT FREE RESOURCES
202. THANK YOU! @alxcnwy alex @ numberboost.com