What does your Instagram say about you? Exploring Google Cloud Vision

Prairie.code() Des Moines, IA September 12th 2019

Explore application development with Google Cloud Vision API, which can evaluate a photo and extract categories with what is going in it. From there, the sky's the limit with exploring the detected labels. We'll check out different photo sharing accounts, and see what we can know about them all through their photos. This will use Google Cloud Platform, and Ruby, with JSON formatted labels that are translated into tag cloud with what labels are being displayed through the images and shows a great cross-section of different technologies.

Transcript

1. @tech_christine @ryanhos
   What does your
   Christine Seeman and Ryan Hochstetler
   say about you?
   

   View Slide

2. @tech_christine @ryanhos
   What is in an image?
   

   View Slide

3. @tech_christine @ryanhos
   What is in an image?
   

   View Slide

4. @tech_christine @ryanhos
   What do you focus on?
   

   View Slide

5. @tech_christine @ryanhos
   What do you focus on?
   

   View Slide

6. @tech_christine @ryanhos
   What is in it that you see?
   

   View Slide

7. @tech_christine @ryanhos
   What is in it that you see?
   

   View Slide

8. @tech_christine @ryanhos
   But what does an
   application see in them?
   

   View Slide

9. @tech_christine @ryanhos
   Meet Google Cloud
   Vision
   

   View Slide

10. @tech_christine @ryanhos
    

    View Slide

11. @tech_christine @ryanhos
    

    View Slide

12. @tech_christine @ryanhos
    Little bit different
    vision
    

    View Slide

13. @tech_christine @ryanhos
    Cloud Vision API
    Landmarks
    Face detection Image labeling
    Optical character recognition
    Explicit content detection
    Logo detection
    Web entities
    Crop hint detection
    

    View Slide

14. @tech_christine @ryanhos
    But how can we use it?
    

    View Slide

15. @tech_christine @ryanhos
    Let's look at the client library
    

    View Slide

16. @tech_christine @ryanhos
    Let's combine
    accounts with
    

    View Slide

17. @tech_christine @ryanhos
    

    View Slide

18. @tech_christine @ryanhos
    So that got us from this
    

    View Slide

19. @tech_christine @ryanhos
    To this
    

    View Slide

20. @tech_christine @ryanhos
    Aggregate the results into histograms
    Labels
    Landmarks
    Logos
    

    View Slide

21. @tech_christine @ryanhos
    Then finally the tag cloud
    

    View Slide

22. @tech_christine @ryanhos
    Let's explore the
    different detections
    

    View Slide

23. @tech_christine @ryanhos
    Face detection
    

    View Slide

24. @tech_christine @ryanhos
    What’s a “gnathion”?
    

    View Slide

25. @tech_christine @ryanhos
    Sentiment Detection
    Probably should be called “Sentiment Best Guestimation”
    

    View Slide

26. @tech_christine @ryanhos
    Joy
    

    View Slide

27. @tech_christine @ryanhos
    Surprise
    

    View Slide

28. @tech_christine @ryanhos
    Anger?
    

    View Slide

29. @tech_christine @ryanhos
    Sorrow?
    

    View Slide

30. @tech_christine @ryanhos
    Image Composition
    

    View Slide

31. @tech_christine @ryanhos
    Label detection
    

    View Slide

32. @tech_christine @ryanhos
    @expertvagabond
    

    View Slide

33. @tech_christine @ryanhos
    @expertvagabond
    

    View Slide

34. @tech_christine @ryanhos
    @ripleyandrue
    

    View Slide

35. @tech_christine @ryanhos
    @ripleyandrue
    

    View Slide

36. @tech_christine @ryanhos
    Safe Search detection
    

    View Slide

37. @tech_christine @ryanhos
    Yes it knows if it is a hotdog
    

    View Slide

38. @tech_christine @ryanhos
    ...and when it's not
    

    View Slide

39. @tech_christine @ryanhos
    Seems accurate…
    

    View Slide

40. @tech_christine @ryanhos
    Oh, there might be a bit of bias…
    

    View Slide

41. @tech_christine @ryanhos
    Violence is difficult too…
    

    View Slide

42. @tech_christine @ryanhos
    Racy but not violent?!?
    

    View Slide

43. @tech_christine @ryanhos
    Doctored images are hit and miss
    We chose not to include the image of a former US
    president photoshopped to look like a dictator and
    war criminal.
    Google labeled it “Possibly” a spoof; i.e. altered to be
    funny or offensive
    

    View Slide

44. @tech_christine @ryanhos
    Text detection
    

    View Slide

45. @tech_christine @ryanhos
    @letterfolk
    

    View Slide

46. @tech_christine @ryanhos
    @letterfolk
    MY
    BRAIN
    HAS
    TO0
    MANY
    TABS
    OPEN
    Dechtlatte
    THIS WEEK
    pt de C
    CAit s pg
    

    View Slide

47. @tech_christine @ryanhos
    @letterfolk
    

    View Slide

48. @tech_christine @ryanhos
    @letterfolk
    YOURE NOT THE
    BOSS OF ME..
    IWHISPER UNDER
    MY BREATH AS I
    CLEAN UP ALL OF
    MY CHILDREN'S
    MESSES.
    TRE
    

    View Slide

49. @tech_christine @ryanhos
    @letterfolk
    

    View Slide

50. @tech_christine @ryanhos
    @letterfolk
    Seth
    Elizabeth
    Mat
    IT'S BEGINNING TO
    amazon Prime
    LOOK A LOT LIKE...
    vmazon Prime
    echo
    I GOT MY MONEY'S
    WORTH FROM MY
    1t\nare en
    a ton Prime
    AMAZON PRIME
    MEMBERSHIP
    Nany
    echo
    eciro
    

    View Slide

51. @tech_christine @ryanhos
    Logo detection
    

    View Slide

52. @tech_christine @ryanhos
    

    View Slide

53. @tech_christine @ryanhos
    @nike
    

    View Slide

54. @tech_christine @ryanhos
    @nike
    

    View Slide

55. @tech_christine @ryanhos
    @nike
    1. nike
    2. nike plus
    3. nike azul
    4. blue nike
    5. nike store
    

    View Slide

56. @tech_christine @ryanhos
    @christine_seeman
    

    View Slide

57. @tech_christine @ryanhos
    @christine_seeman
    

    View Slide

58. @tech_christine @ryanhos
    @christine_seeman
    1. etelä suomen sanomat
    2. rogue status
    3. american horror story
    4. delta skymiles
    5. deník
    !"!"
    

    View Slide

59. @tech_christine @ryanhos
    Landmark detection
    

    View Slide

60. @tech_christine @ryanhos
    @visit_nebraska
    

    View Slide

61. @tech_christine @ryanhos
    @iloveny
    

    View Slide

62. @tech_christine @ryanhos
    @iloveny
    

    View Slide

63. @tech_christine @ryanhos
    @expertvagabond
    

    View Slide

64. @tech_christine @ryanhos
    Have you been to London, NE?
    

    View Slide

65. @tech_christine @ryanhos
    The British Museum there is lovely
    

    View Slide

66. @tech_christine @ryanhos
    British Museum, alternate photo
    

    View Slide

67. @tech_christine @ryanhos
    So much alike
    

    View Slide

68. @tech_christine @ryanhos
    This wasn’t even a building…
    

    View Slide

69. @tech_christine @ryanhos
    But what does my
    instagram say about me?
    

    View Slide

70. @tech_christine @ryanhos
    

    View Slide

71. @tech_christine @ryanhos
    @christineseeman
    

    View Slide

72. @tech_christine @ryanhos
    Blooper cloud
    

    View Slide

73. @tech_christine @ryanhos
    Cloud AutoML Vision
    •Train ML models to classify
    images
    •Use your own defined labels.
    •Graphical user interface to
    train, evaluate, improve, and
    deploy models
    

    View Slide

74. @tech_christine @ryanhos
    @visit_nebraska
    

    View Slide

75. @tech_christine @ryanhos
    Vs @visitcalifornia
    

    View Slide

76. @tech_christine @ryanhos
    Vs @iloveny
    

    View Slide

77. @tech_christine @ryanhos
    Sydney, Australia

    Bangkok, Thailand
    Paris, France
    

    View Slide

78. @tech_christine @ryanhos
    

    View Slide

79. @tech_christine @ryanhos
    Evaluating Model Accuracy
    

    View Slide

80. @tech_christine @ryanhos
    Confusion Matrix
    

    View Slide

81. @tech_christine @ryanhos
    Skyline Predictions
    

    View Slide

82. @tech_christine @ryanhos
    More training required…
    

    View Slide

83. @tech_christine @ryanhos
    Urban Scenery Predictions
    

    View Slide

84. @tech_christine @ryanhos
    Outdoor Scenery Predictions
    

    View Slide

85. @tech_christine @ryanhos
    Where to go from here?
    What to do next with Google Cloud Vision
    

    View Slide

86. @tech_christine @ryanhos
    Explore bias in Machine
    Learning?
    

    View Slide

87. @tech_christine @ryanhos
    How much will this cost you?
    For us, about $65
    

    View Slide

88. @tech_christine @ryanhos
    905.33!
    

    View Slide

89. @tech_christine @ryanhos
    $40 for the Training Compute Hours

    

    View Slide

90. @tech_christine @ryanhos
    Technology Used
    •Ruby (No Rails)
    •Rspec
    •Ruby Vision API Client Libraries
    •Google Cloud Storage
    •Rmagick (Ruby binding to Imagemagick)
    •MagicCloud tag cloud gem
    

    View Slide

91. @tech_christine @ryanhos
    All the code
    https://github.com/hi-christine/cloud-vision-insta
    

    View Slide

92. @tech_christine @ryanhos
    Reference Links
    •https://cloud.google.com/vision/
    •https://cloud.google.com/vision/docs/
    •https://googleapis.github.io/google-cloud-ruby/docs/
    •https://github.com/GoogleCloudPlatform/ruby-docs-
    samples/blob/master/vision/quickstart.rb
    •https://github.com/zverok/magic_cloud
    

    View Slide

93. @tech_christine @ryanhos
    All the instagram accounts*
    @ripleyandrue
    @visit_nebraska
    @wolffolins
    @expertvagabond
    @iloveny
    @letterfolk
    @myraswim
    @visitcalifornia
    @nationalportraitgallery
    @nike
    * and none were harmed in the making of this talk
    

    View Slide

94. @tech_christine @ryanhos
    getflywheel.com/about/careers
    

    View Slide

95. @tech_christine @ryanhos
    Thank you for attending!
    

    View Slide

96. @tech_christine @ryanhos
    For your evaluation consideration...
    

    View Slide

97. @tech_christine @ryanhos
    Bonus Code Time
    

    View Slide

98. @tech_christine @ryanhos
    image_annotator_client.rb
    

    View Slide

99. @tech_christine @ryanhos
    image_annotator_client.rb
    

    View Slide

100. @tech_christine @ryanhos
     image_annotator_client.rb
     

     View Slide

101. @tech_christine @ryanhos
     image_annotator_client.rb
     

     View Slide

102. @tech_christine @ryanhos
     analysis_reader.rb
     

     View Slide

103. @tech_christine @ryanhos
     analysis_reader.rb
     

     View Slide

104. @tech_christine @ryanhos
     raster_annotator.rb
     

     View Slide

105. @tech_christine @ryanhos
     raster_annotator.rb
     

     View Slide