Deep Learning on iOS #360iDev

2017年8月13日〜16日にアメリカ合衆国コロラド州デンバーにて開催された「360|iDev 2017」にて登壇して際の発表資料です。

ブログ： コロラド州デンバーで開催されたiOSカンファレンス「360|iDev 2017」に登壇した話

Transcript

1. 360|iDev 2017 Shuichi Tsutsumi @shu223 Deep Learning on iOS
3. Overview • How to implement “Deep Learning” on iOS Metal Performance Shaders (MPSCNN) Accelerate (BNNS) Core ML Vision Your App
4. Why so exciting? !
5. AIphaGo Cancer Detection Self-driving Car
6. AutoDraw
7. Pose Estimation http://qiita.com/nnn112358/items/a4490d85dac5827db53b
8. Frontal View Synthesis
9. AutoHair
10. AutoDraw Pose Estimation Frontal View AutoHair
11. AutoDraw Pose Estimation Frontal View AutoHair This evolutional tech works on iOS"
12. Deep Learning ON iOS = works on iOS devices
13. (Demo)
14. Image Result 60 times / sec xUsers # #
16. TOOTHPASTE
17. !
18. Train Inference Trained Params iOS ML frameworks
19. Implementation
20. """Core ML"""
22. The largest hall was 100% full for the Core ML session.
23. ✋
24. Metal Performance Shaders (MPSCNN) Accelerate (BNNS) Core ML Vision Your App iOS 11 iOS 10
25. Metal Performance Shaders (MPSCNN) Accelerate (BNNS) Core ML Vision Your App iOS 11 iOS 10
26. Your App Metal Performance Shaders (MPSCNN) Accelerate (BNNS) iOS 10 • Optimized for GPU (by ) and CPU (by Accelerate) • Available for iOS 10, too • Basically any ML tools can be used to train the models Still works!
27. Metal Performance Shaders (MPSCNN) Accelerate (BNNS) Your App GPU CPU
28. MPSCNN 900 results Core ML 160,000 results
29. 3 steps to implement w/ MPSCNN
30. How to implement w/ MPSCNN Step 1: Create the model
31. Train Inference Trained Params -Which tools can be used for the training? -What kind of formats can be used to pass the pre-trained params?
32. Which ML tools can be used? : ANY What model format can be used? : ANY Model (Trained Params) dat Train ML Tools
33. Which ML tools can be used? : ANY What model format can be used? : ANY Model (Trained Params) dat Train ML Tools hdf5 •The “.dat” files are common binary files. - Not specific for iOS or MPS. •Contains the trained params - Weights / Biases •Any other format can be used as long as it can be read by the iOS app. - c.f. hdf5
34. Which ML tools can be used? : ANY What model format can be used? : ANY Model (Trained Params) dat Train ML Tools hdf5 •Any tools which can train CNN, and export the params
35. How to implement w/ MPSCNN Step 2: Implement the network
36. MPSCNNConvolution MPSCNNFullyConnected MPSCNNPooling MPSCNNConvolution MPSCNNConvolution MPSCNNPooling
37. MPSCNNConvolution MPSCNNFullyConnected MPSCNNPooling MPSCNNConvolution MPSCNNConvolution MPSCNNPooling • Almost same name -> Easy to find • Complicated Maths or GPU optimization are encapsulated Classes corresponding to each CNN layers are provided
38. How to implement w/ MPSCNN Step 3: Implement the inference
39. Input image MPSImage Result (UInt, etc.) CNN • Implemented in Step 2 • Trained params (created in Step 1) are loaded. Do something
40. • Step 1: Create the model - Any ML tools can be used - Any file format can be used for the trained params • Step 2: Implement the network - Classes corresponding to each CNN layer are provided • Step 3: Implement the inference - Input to the CNN, and output from the CNN
41. Demo : Logo detection
43. Increased by 70 times
44. Trained Params dat f.write(session.run(w_conv1_p).tobytes())
45. (Demo)
47. ⚠
48. CNN for the logo detection
49. GoogLeNet (Inception v3)
50. GoogLeNet (Inception v3) Apple’s implementation w/ MPSCNN Inception3Net.swift
52. 2,000 lines& 0OMZ
    GPS
    UIF
    $//
54. Core ML
55. Development Flow w/ MPSCNN ML Tools Train some format Trained Params dat Extract Parse dat MPSCNNConvolution MPSCNNFullyConnected Implement Network 2,000 lines& App
56. Development Flow w/ Core ML ML Tools Train some format Trained Params dat Extract Parse dat MPSCNNConvolution MPSCNNFullyConnected Implement Network App 2,000 lines& 1) Convert w/ coremltools 2) Drag & Drop some format Generate
57. Input image MPSImage Result (UInt, etc.) CNN Do something let size = MTLSize(width: inputWidth, height: inputHeight let region = MTLRegion(origin: MTLOrigin(x: 0, y: 0, z: 0 size: size) network.srcImage.texture.replace( region: region, mipmapLevel: 0, slice: 0, withBytes: context.data!, bytesPerRow: inputWidth, bytesPerImage: 0) Need to know Metal to use MPSCNN let origin = MTLOrigin(x: 0, y: 0, z: 0) let size = MTLSize(width: 1, height: 1, depth: 1) finalLayer.texture.getBytes(&(result_half_array[4*i]), bytesPerRow: MemoryLayout<UIn bytesPerImage: MemoryLayout<U from: MTLRegion(origin: origi size: size), mipmapLevel: 0, slice: i)
58. MPSCNN Accelerate (BNNS) Core ML Vision Your App
59. Input image MPSImage Results CNN Do something let ciImage = CIImage(cvPixelBuffer: imageBuffer) let handler = VNImageRequestHandler(ciImage: ciImage) try! handler.perform([self.coremlRequest]) Don’t need to touch Metal to use Vision guard let results = request.results as? [VNClassificationObservation] else { return } guard let best = results.first?.identifier else { return
60. MPSCNN Accelerate (BNNS) Your App
61. MPSCNN Accelerate (BNNS) Core ML Vision Your App
62. MPSCNN Accelerate (BNNS) Core ML Vision Your App How about BNNS?
63. Deep Learning on CPU?!
64. CPU: 40 days GPU: 6 days Deep Learning on CPU?!
65. Asked @ Lab in WWDC17
66. “How can we utilize both MPSCNN and BNNS?”
67. ' “Basically use MPSCNN.”
68. “OK, but, when should I choose BNNS?”
69. ' “I don’t know.”
70. • He added “watchOS might be a case on that you should use BNNS.” • Because watchOS doesn’t support MPSCNN, but supports BNNS. (I haven’t tried yet.)
71. My current understanding: • The cost for passing data between CPU and GPU is not small • When the network is small, the CPU <-> GPU cost might be bigger than the benefit of parallel processing. BNNS might be better when the network is small.
72. Recap • Why is “Deep Learning on iOS” exciting? • How to implement “Deep Learning” on iOS - w/ MPSCNN (iOS 10) - w/ Core ML & Vision (iOS 11) - When to choose BNNS MPSCNN BNNS Core ML Vision
73. Thank you! https://github.com/shu223