Machine Intelligence at Google Scale

What is Neural Network? Why is Deep Learning so important? What are the challenges for introducing those technologies to production services? In this session, we will look at the answers for those questions and how Google has been successfully deploying large scale neural network on services such as Google Photos, Android and Google Search. Also, we will introduce the new Google Cloud products such as Cloud Vision API, Speech API, TensorFlow and Cloud ML that allows developers to take advantage of the power of Google's machine intelligence with scalable and fully managed services.

「より使いやすい機械学習へ：Googleの機械学習テクノロジー」
ニューラルネットワークとは何か？　ディープラーニングはなぜ注目を集めているのか？　実開発での利用の課題は？　このセッションでは、これらの疑問について議論し、さらにGoogleフォトやAndroid、Google検索等で利用されているGoogleにおける大規模なニューラルネットワーク導入事例を紹介します。また後半では、Cloud Vision API、Speech API、TensorFlow、CloudMLなど、Googleが提供するスケーラブルでフルマネージドのクラウドサービスについて解説し、Googleの機械学習技術を活用する方法を紹介します。

Transcript

1. Machine Intelligence at Google Scale
   ML APIs, TensorFlow and Cloud ML
   

   View Slide

2. +Kazunori Sato
   @kazunori_279
   Kaz Sato
   Staff Developer Advocate
   Tech Lead for Data & Analytics
   Cloud Platform, Google Inc.
   

   View Slide

3. What we’ll cover
   What is Neural Network and Deep Learning
   Machine Learning use cases at Google services
   Externalizing the power with ML APIs
   TensorFlow: the open source library for ML
   TensorFlow in the Wild
   Distributed training and prediction with Cloud ML
   

   View Slide

4. What is Neural Network
   and Deep Learning
   

   View Slide

5. Neural Network is a function that can learn
   

   View Slide

6. x
   n
   > b?
   w
   1
   w
   n
   x
   2
   x
   1
   Inspired by the behavior of biological neurons
   

   View Slide

7. How do you
   classify them?
   

   View Slide

8. weights
   bias
   (threshold)
   Programmers need to specify the parameters
   

   View Slide

9. Let’s see how neural network solves the problem
   

   View Slide

10. The computer tries to find
    the best parameters
    A neuron classifies a data point into two kinds
    

    View Slide

11. Gradient Descent: adjusting the params
    gradually to reduce errors
    From: Andrew Ng
    

    View Slide

12. How do you
    classify them?
    

    View Slide

13. What we see What the computer “sees”
    

    View Slide

14. 28 x 28 gray scale image =
    784 numbers
    

    View Slide

15. input vector
    (pixel data)
    output vector
    (probability)
    

    View Slide

16. How do you
    classify them?
    

    View Slide

17. More neurons = More features to extract
    

    View Slide

18. Hidden Layers:
    mapping inputs to
    a feature space,
    classifying with
    a hyperplane
    From: Neural Networks, Manifolds, and Topology, colah's blog
    

    View Slide

19. How about this?
    

    View Slide

20. More hidden layers = More hierarchies of features
    

    View Slide

21. How about this?
    

    View Slide

22. We need to go deeper neural network
    From: Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations, Honglak Lee et al.
    

    View Slide

23. From: mNeuron: A Matlab Plugin to Visualize Neurons from Deep Models, Donglai Wei et. al.
    

    View Slide

24. Machine Learning use cases
    at Google services
    

    View Slide

25. 25
    signal
    for Search ranking,
    out of hundreds
    improvement
    to ranking quality
    in 2+ years
    #3 #1
    Search
    machine learning for search engines
    RankBrain: a deep neural network for search ranking
    

    View Slide

26. View Slide

27. WaveNet by
    Google DeepMind
    

    View Slide

28. 28
    [glacier]
    Google Photos
    28
    

    View Slide

29. 29
    Smart reply
    in Inbox by Gmail
    10%
    of all responses
    sent on mobile
    

    View Slide

30. http://googleresearch.blogspot.com/2016/05/announcing-syntaxnet-worlds-most.html
    

    View Slide

31. Saved Data Center cooling energy for 40%
    Improved Power Usage Effectiveness (PUE) for 15%
    

    View Slide

32. 32
    Android
    Apps
    Gmail
    Maps
    Photos
    Speech
    Search
    Translation
    YouTube
    and many others ...
    Used across products:
    2012 2013 2014 2015
    Deep Learning usage at Google
    

    View Slide

33. Externalizing the power
    with ML APIs
    

    View Slide

34. TensorFlow Cloud Machine Learning ML API
    Easy-to-Use, for non-ML engineers
    Customizable, for Data Scientists
    Machine Learning products from Google
    

    View Slide

35. Image analysis with pre-trained models
    No Machine Learning skill required
    REST API: receives an image and returns a JSON
    General Availability
    Cloud Vision API
    

    View Slide

36. Confidential & Proprietary
    Google Cloud Platform 36
    Faces
    Faces, facial landmarks, emotions
    OCR
    Read and extract text, with
    support for > 10 languages
    Label
    Detect entities from furniture to
    transportation
    Logos
    Identify product logos
    Landmarks & Image Properties
    Detect landmarks & dominant
    color of image
    Safe Search
    Detect explicit content - adult,
    violent, medical and spoof
    

    View Slide

37. 37
    37
    Demo
    

    View Slide

38. Pre-trained models. No ML skill required
    REST API: receives audio and returns texts
    Supports 80+ languages
    Streaming or non-streaming
    Public Beta - cloud.google.com/speech
    Cloud Speech API
    

    View Slide

39. Confidential & Proprietary
    Google Cloud Platform 39
    Features
    Automatic Speech Recognition (ASR)
    powered by deep learning neural
    networking to power your
    applications like voice search or
    speech transcription.
    Recognizes over 80
    languages and variants
    with an extensive
    vocabulary.
    Returns partial
    recognition results
    immediately, as they
    become available.
    Filter inappropriate
    content in text results.
    Audio input can be captured by an application’s
    microphone or sent from a pre-recorded audio
    file. Multiple audio file formats are supported,
    including FLAC, AMR, PCMU and linear-16.
    Handles noisy audio from many
    environments without requiring
    additional noise cancellation.
    Audio files can be uploaded in the
    request and, in future releases,
    integrated with Google Cloud
    Storage.
    Automatic Speech Recognition Global Vocabulary Inappropriate Content
    Filtering
    Streaming Recognition
    Real-time or Buffered Audio Support Noisy Audio Handling Integrated API
    

    View Slide

40. 40
    40
    Demo
    

    View Slide

41. Pre-trained models. No ML skill required
    REST API: receives text and returns analysis results
    Supports English, Spanish and Japanese
    Public Beta - cloud.google.com/natural-language
    Cloud Natural Language API
    

    View Slide

42. Confidential & Proprietary
    Google Cloud Platform 42
    Features
    Extract sentence, identify parts of
    speech and create dependency parse
    trees for each sentence.
    Identify entities and label by types such
    as person, organization, location, events,
    products and media.
    Understand the overall sentiment of a
    block of text.
    Syntax Analysis Entity Recognition
    Sentiment Analysis
    

    View Slide

43. 43
    43
    Demo
    

    View Slide

44. TensorFlow:
    An open source library for
    Machine Intelligence
    

    View Slide

45. Google's open source library for
    machine intelligence
    tensorflow.org launched in Nov 2015
    Used by many production ML projects
    What is TensorFlow?
    

    View Slide

46. Sharing our tools with researchers and developers
    around the world
    repository
    for “machine learning”
    category on GitHub
    #1
    Released in Nov. 2015
    From: http://deliprao.com/archives/168
    

    View Slide

47. Before
    Hire Data Scientists
    ↓
    Understand the math model
    ↓
    Impl with programming code
    ↓
    Train with single GPU
    ↓
    Build a GPU cluster
    ↓
    Train with the GPU cluster
    ↓
    Build a prediction server
    or Impl mobile/IoT prediction
    After
    Easy network design and impl
    ↓
    Train with single machine
    ↓
    Train on the cloud
    ↓
    Prediction on the cloud
    or mobile/IoT devices
    many people
    stuck here
    

    View Slide

48. # define the network
    import tensorflow as tf
    x = tf.placeholder(tf.float32, [None, 784])
    W = tf.Variable(tf.zeros([784, 10]))
    b = tf.Variable(tf.zeros([10]))
    y = tf.nn.softmax(tf.matmul(x, W) + b)
    # define a training step
    y_ = tf.placeholder(tf.float32, [None, 10])
    xent = -tf.reduce_sum(y_*tf.log(y))
    step =
    tf.train.GradientDescentOptimizer(0.01).minimize(xent)
    

    View Slide

49. TensorBoard: visualization tool
    

    View Slide

50. Portable and Scalable
    Training on:
    Mac/Windows
    GPU server
    GPU cluster / Cloud
    Prediction on:
    Android and iOS
    RasPi and TPU
    

    View Slide

51. Distributed Training with TensorFlow
    

    View Slide

52. TensorFlow in the Wild
    (or democratization of deep learning)
    

    View Slide

53. TensorFlow
    powered
    Fried Chicken
    Nugget Server
    From: http://www.rt-net.jp/karaage1/
    

    View Slide

54. TensorFlow powered Cucumber Sorter
    From: http://workpiles.com/2016/02/tensorflow-cnn-cucumber/
    

    View Slide

55. Autonomous
    Driving
    of RasPi car
    with Inception 3
    on TensorFlow
    From:
    https://github.com/zxzhijia/GoPiGo-Driven-by-Te
    nsorflow
    

    View Slide

56. TV popstar classifier
    with 95% accuracy
    From: http://memo.sugyan.com/entry/2016/06/14/220624
    

    View Slide

57. TensorFlow＋
    RasPi for
    sorting garbages
    From:
    https://techcrunch.com/2016/09/13/auto-trash-sorts-
    garbage-automatically-at-the-techcrunch-disrupt-hacka
    thon/
    

    View Slide

58. TensorFlow +
    Drones
    for counting trucks
    From: http://www.brainpad.co.jp/news/2016/09/02/3454
    

    View Slide

59. From: http://otoro.net/
    Generative Arts with TensorFlow
    

    View Slide

60. Distributed Training and
    Prediction with Cloud ML
    

    View Slide

61. From: Andrew Ng
    The Bigger, The Better
    

    View Slide

62. The Challenge: Computing Power
    DNN requires large training datasets
    Large models doesn't fit into a GPU
    Requires try-and-errors to find the
    best design, configs and params
    ↓
    Need to spend a few days or
    weeks to finish a training
    

    View Slide

63. GPUs run at nanoseconds
    GPU cluster needs microsec network
    

    View Slide

64. View Slide

65. Enterprise
    Google Cloud is
    The Datacenter as a Computer
    

    View Slide

66. Jupiter network
    10 GbE x 100 K = 1 Pbps
    Consolidates servers with
    microsec latency
    

    View Slide

67. Borg
    No VMs, pure containers
    10K - 20K nodes per Cell
    DC-scale job scheduling
    CPUs, mem, disks and IO
    

    View Slide

68. Distributed Training
    with TensorFlow
    by data parallelism
    split data,
    share model
    

    View Slide

69. ● CPU/GPU scheduling
    ● Communications
    ○ Local, RPC, RDMA
    ○ 32/16/8 bit quantization
    ● Cost-based optimization
    ● Fault tolerance
    Distributed Systems for Large Neural Network
    

    View Slide

70. What's the scalability of Google Brain?
    "Large Scale Distributed Systems for Training Neural
    Networks", NIPS 2015
    ○ Inception / ImageNet: 40x with 50 GPUs
    ○ RankBrain: 300x with 500 nodes
    

    View Slide

71. Fully managed distributed training and prediction
    Supports custom TensorFlow graphs
    Integrated with Cloud Dataflow and Cloud Datalab
    Limited Preview - cloud.google.com/ml
    Cloud Machine Learning (Cloud ML)
    

    View Slide

72. 72
    72
    Ready to use Machine
    Learning models
    Use your own data to
    train models
    Cloud
    Vision API
    Cloud
    Speech API
    Cloud
    Translate API
    Cloud Machine Learning
    Develop - Model - Test
    Google
    BigQuery
    Stay
    Tuned….
    Cloud
    Storage
    Cloud
    Datalab
    NEW
    Alpha
    GA Beta
    GA
    Alpha
    GA
    GA
    

    View Slide

73. Tensor Processing Unit
    ASIC for TensorFlow
    Designed by Google
    10x better perf / watt
    latency and efficiency
    bit quantization
    

    View Slide

74. TPU on Production
    RankBrain
    AlphaGo
    Google Photos
    Speech
    and more
    

    View Slide

75. Thank you!
    

    View Slide

76. Links & Resources
    Large Scale Distributed Systems for Training Neural Networks, Jeff Dean and
    Oriol Vinals
    Cloud Vision API: cloud.google.com/vision
    Cloud Speech API: cloud.google.com/speech
    TensorFlow: tensorflow.org
    Cloud Machine Learning: cloud.google.com/ml
    Cloud Machine Learning: demo video
    

    View Slide