What does your Instagram say about you? Exploring Google Cloud Vision

Explore application development with Google Cloud Vision API which can categorize photos, detect objects, identify landmarks, and extract corporate logos from images. From there, the sky's the limit with exploring the detected labels. We'll check out different photo sharing accounts, and see what we can know about them all through their photos. This will use Google Cloud Platform, Ruby, with JSON formatted labels that are translated into a tag cloud with what activities are being displayed through the images and shows a cross-section of different technologies.

Transcript

1. @tech_christine @ryanhos
   What does your
   Christine Seeman and Ryan Hochstetler
   say about you?
   

   View Slide

2. @tech_christine @ryanhos
   What is in an image?
   

   View Slide

3. @tech_christine @ryanhos
   What is in an image?
   

   View Slide

4. @tech_christine @ryanhos
   What do you focus on?
   

   View Slide

5. @tech_christine @ryanhos
   What do you focus on?
   

   View Slide

6. @tech_christine @ryanhos
   What is in it that you see?
   

   View Slide

7. @tech_christine @ryanhos
   What is in it that you see?
   

   View Slide

8. @tech_christine @ryanhos
   But what does an
   application see in them?
   

   View Slide

9. @tech_christine @ryanhos
   Meet Google Cloud
   Vision
   

   View Slide

10. @tech_christine @ryanhos
    

    View Slide

11. @tech_christine @ryanhos
    

    View Slide

12. @tech_christine @ryanhos
    Little bit different
    vision
    

    View Slide

13. @tech_christine @ryanhos
    Cloud Vision API
    Landmarks
    Face detection Image labeling
    Optical character recognition
    Explicit content detection
    Logo detection
    Object localization
    Crop hint detection
    

    View Slide

14. @tech_christine @ryanhos
    But how can we use it?
    

    View Slide

15. @tech_christine @ryanhos
    Let's look at the client library
    

    View Slide

16. @tech_christine @ryanhos
    Let's combine
    accounts with
    

    View Slide

17. @tech_christine @ryanhos
    

    View Slide

18. @tech_christine @ryanhos
    Onto the code
    

    View Slide

19. @tech_christine @ryanhos
    image_annotator_client.rb
    

    View Slide

20. @tech_christine @ryanhos
    image_annotator_client.rb
    

    View Slide

21. @tech_christine @ryanhos
    image_annotator_client.rb
    

    View Slide

22. @tech_christine @ryanhos
    image_annotator_client.rb
    

    View Slide

23. @tech_christine @ryanhos
    So that got us from this
    

    View Slide

24. @tech_christine @ryanhos
    To this
    

    View Slide

25. @tech_christine @ryanhos
    analysis_reader.rb
    

    View Slide

26. @tech_christine @ryanhos
    analysis_reader.rb
    

    View Slide

27. @tech_christine @ryanhos
    After histogram generation
    Labels
    Landmarks
    Logos
    

    View Slide

28. @tech_christine @ryanhos
    Then finally the tag cloud
    

    View Slide

29. @tech_christine @ryanhos
    Let's explore the
    different detections
    

    View Slide

30. @tech_christine @ryanhos
    Face detection
    

    View Slide

31. @tech_christine @ryanhos
    What’s a “gnathion”?
    

    View Slide

32. @tech_christine @ryanhos
    raster_annotator.rb
    

    View Slide

33. @tech_christine @ryanhos
    raster_annotator.rb
    

    View Slide

34. @tech_christine @ryanhos
    Sentiment Detection
    Probably should be called “Sentiment Best Guestimation”
    

    View Slide

35. @tech_christine @ryanhos
    Joy
    

    View Slide

36. @tech_christine @ryanhos
    Surprise
    

    View Slide

37. @tech_christine @ryanhos
    Anger?
    

    View Slide

38. @tech_christine @ryanhos
    Sorrow?
    

    View Slide

39. @tech_christine @ryanhos
    Image Composition
    

    View Slide

40. @tech_christine @ryanhos
    Label detection
    

    View Slide

41. @tech_christine @ryanhos
    @expertvagabond
    

    View Slide

42. @tech_christine @ryanhos
    @expertvagabond
    

    View Slide

43. @tech_christine @ryanhos
    @ripleyandrue
    

    View Slide

44. @tech_christine @ryanhos
    @ripleyandrue
    

    View Slide

45. @tech_christine @ryanhos
    Safe Search detection
    

    View Slide

46. @tech_christine @ryanhos
    Yes it knows if it is a hotdog
    

    View Slide

47. @tech_christine @ryanhos
    ...and when it's not
    

    View Slide

48. @tech_christine @ryanhos
    Seems accurate…
    

    View Slide

49. @tech_christine @ryanhos
    Oh, there might be a bit of bias…
    

    View Slide

50. @tech_christine @ryanhos
    Violence is difficult too…
    

    View Slide

51. @tech_christine @ryanhos
    Racy but not violent?!?
    

    View Slide

52. @tech_christine @ryanhos
    Doctored images are hit and miss
    We chose not to include the image of a former US
    president photoshopped to look like a dictator and
    war criminal.
    Google labeled it “Possibly” a spoof; i.e. altered to be
    funny or offensive
    

    View Slide

53. @tech_christine @ryanhos
    Text detection
    

    View Slide

54. @tech_christine @ryanhos
    @letterfolk
    

    View Slide

55. @tech_christine @ryanhos
    @letterfolk
    "MY\nBRAIN\nHAS\nTO0\nMANY\nTABS\nOPEN\nDechtlatte\nTHIS WEEK\npt de C\nCAit s pg\n"
    MY
    BRAIN
    HAS
    TO0
    MANY
    TABS
    OPEN
    Dechtlatte
    THIS WEEK
    pt de C
    CAit s pg
    

    View Slide

56. @tech_christine @ryanhos
    @letterfolk
    

    View Slide

57. @tech_christine @ryanhos
    @letterfolk
    YOURE NOT THE\nBOSS OF ME..\nIWHISPER UNDER\nMY BREATH AS I\nCLEAN UP ALL OF\nMY
    CHILDREN'S\nMESSES.\nTRE\n
    YOURE NOT THE
    BOSS OF ME..
    IWHISPER UNDER
    MY BREATH AS I
    CLEAN UP ALL OF
    MY CHILDREN'S
    MESSES.
    TRE
    

    View Slide

58. @tech_christine @ryanhos
    @letterfolk
    

    View Slide

59. @tech_christine @ryanhos
    @letterfolk
    Seth\nElizabeth\nMat\nIT'S BEGINNING TO\namazon Prime\nLOOK A LOT LIKE...\nvmazon
    Prime\necho\nI GOT MY MONEY'S\nWORTH FROM MY\n1t\nare en\na ton Prime\nAMAZON
    PRIME\nMEMBERSHIP\nNany\necho\neciro\n
    Seth
    Elizabeth
    Mat
    IT'S BEGINNING TO
    amazon Prime
    LOOK A LOT LIKE...
    vmazon Prime
    echo
    I GOT MY MONEY'S
    WORTH FROM MY
    1t\nare en
    a ton Prime
    AMAZON PRIME
    MEMBERSHIP
    Nany
    echo
    eciro
    

    View Slide

60. @tech_christine @ryanhos
    Logo detection
    

    View Slide

61. @tech_christine @ryanhos
    

    View Slide

62. @tech_christine @ryanhos
    @nike
    

    View Slide

63. @tech_christine @ryanhos
    @nike
    

    View Slide

64. @tech_christine @ryanhos
    @nike
    1. nike
    2. nike plus
    3. nike azul
    4. blue nike
    5. nike store
    

    View Slide

65. @tech_christine @ryanhos
    @christine_seeman
    

    View Slide

66. @tech_christine @ryanhos
    @christine_seeman
    

    View Slide

67. @tech_christine @ryanhos
    @christine_seeman
    1. etelä suomen sanomat
    2. rogue status
    3. american horror story
    4. delta skymiles
    5. deník
    !"!"
    

    View Slide

68. @tech_christine @ryanhos
    Landmark detection
    

    View Slide

69. @tech_christine @ryanhos
    @visit_nebraska
    

    View Slide

70. @tech_christine @ryanhos
    @iloveny
    

    View Slide

71. @tech_christine @ryanhos
    @iloveny
    

    View Slide

72. @tech_christine @ryanhos
    @expertvagabond
    

    View Slide

73. @tech_christine @ryanhos
    Have you been to London, NE?
    

    View Slide

74. @tech_christine @ryanhos
    The British Museum there is lovely
    

    View Slide

75. @tech_christine @ryanhos
    British Museum, alternate photo
    

    View Slide

76. @tech_christine @ryanhos
    So much alike
    

    View Slide

77. @tech_christine @ryanhos
    This wasn’t even a building…
    

    View Slide

78. @tech_christine @ryanhos
    But what does my
    instagram say about me?
    

    View Slide

79. @tech_christine @ryanhos
    

    View Slide

80. @tech_christine @ryanhos
    @christineseeman
    

    View Slide

81. @tech_christine @ryanhos
    Blooper cloud
    

    View Slide

82. @tech_christine @ryanhos
    Where to go from here?
    What to do next with Google Cloud Vision
    

    View Slide

83. @tech_christine @ryanhos
    @visit_nebraska
    

    View Slide

84. @tech_christine @ryanhos
    Vs @visitcalifornia
    

    View Slide

85. @tech_christine @ryanhos
    Vs @iloveny
    

    View Slide

86. @tech_christine @ryanhos
    Can we teach a machine
    to know if a pic is from
    NY vs CA vs NE
    

    View Slide

87. @tech_christine @ryanhos
    Explore bias in Machine
    Learning?
    

    View Slide

88. @tech_christine @ryanhos
    How much will this cost you?
    For us, about $65
    

    View Slide

89. @tech_christine @ryanhos
    Technology Used
    •Ruby (No Rails)
    •Rspec
    •Ruby Vision API Client Libraries
    •Google Cloud Storage
    •Rmagick (Ruby binding to Imagemagick)
    •MagicCloud tag cloud gem
    

    View Slide

90. @tech_christine @ryanhos
    Reference Links
    •https://cloud.google.com/vision/
    •https://cloud.google.com/vision/docs/
    •https://googleapis.github.io/google-cloud-ruby/docs/
    •https://github.com/GoogleCloudPlatform/ruby-docs-
    samples/blob/master/vision/quickstart.rb
    •https://github.com/zverok/magic_cloud
    

    View Slide

91. @tech_christine @ryanhos
    All the instagram accounts*
    @ripleyandrue
    @visit_nebraska
    @wolffolins
    @expertvagabond
    @iloveny
    @letterfolk
    @myraswim
    @visitcalifornia
    @nationalportraitgallery
    @nike
    * and none were harmed in the making of this talk
    

    View Slide

92. @tech_christine @ryanhos
    getflywheel.com/about/careers
    

    View Slide

93. @tech_christine @ryanhos
    Thank you for attending!
    

    View Slide

94. @tech_christine @ryanhos
    For your evaluation consideration...
    

    View Slide