Upgrade to Pro — share decks privately, control downloads, hide ads and more …

What does your Instagram say about you? Exploring Google Cloud Vision

Christine
September 12, 2019

What does your Instagram say about you? Exploring Google Cloud Vision

Prairie.code() Des Moines, IA September 12th 2019

Explore application development with Google Cloud Vision API, which can evaluate a photo and extract categories with what is going in it. From there, the sky's the limit with exploring the detected labels. We'll check out different photo sharing accounts, and see what we can know about them all through their photos. This will use Google Cloud Platform, and Ruby, with JSON formatted labels that are translated into tag cloud with what labels are being displayed through the images and shows a great cross-section of different technologies.

Christine

September 12, 2019
Tweet

More Decks by Christine

Other Decks in Technology

Transcript

  1. @tech_christine @ryanhos
    What does your
    Christine Seeman and Ryan Hochstetler
    say about you?

    View full-size slide

  2. @tech_christine @ryanhos
    What is in an image?

    View full-size slide

  3. @tech_christine @ryanhos
    What is in an image?

    View full-size slide

  4. @tech_christine @ryanhos
    What do you focus on?

    View full-size slide

  5. @tech_christine @ryanhos
    What do you focus on?

    View full-size slide

  6. @tech_christine @ryanhos
    What is in it that you see?

    View full-size slide

  7. @tech_christine @ryanhos
    What is in it that you see?

    View full-size slide

  8. @tech_christine @ryanhos
    But what does an
    application see in them?

    View full-size slide

  9. @tech_christine @ryanhos
    Meet Google Cloud
    Vision

    View full-size slide

  10. @tech_christine @ryanhos

    View full-size slide

  11. @tech_christine @ryanhos

    View full-size slide

  12. @tech_christine @ryanhos
    Little bit different
    vision

    View full-size slide

  13. @tech_christine @ryanhos
    Cloud Vision API
    Landmarks
    Face detection Image labeling
    Optical character recognition
    Explicit content detection
    Logo detection
    Web entities
    Crop hint detection

    View full-size slide

  14. @tech_christine @ryanhos
    But how can we use it?

    View full-size slide

  15. @tech_christine @ryanhos
    Let's look at the client library

    View full-size slide

  16. @tech_christine @ryanhos
    Let's combine
    accounts with

    View full-size slide

  17. @tech_christine @ryanhos

    View full-size slide

  18. @tech_christine @ryanhos
    So that got us from this

    View full-size slide

  19. @tech_christine @ryanhos
    To this

    View full-size slide

  20. @tech_christine @ryanhos
    Aggregate the results into histograms
    Labels
    Landmarks
    Logos

    View full-size slide

  21. @tech_christine @ryanhos
    Then finally the tag cloud

    View full-size slide

  22. @tech_christine @ryanhos
    Let's explore the
    different detections

    View full-size slide

  23. @tech_christine @ryanhos
    Face detection

    View full-size slide

  24. @tech_christine @ryanhos
    What’s a “gnathion”?

    View full-size slide

  25. @tech_christine @ryanhos
    Sentiment Detection
    Probably should be called “Sentiment Best Guestimation”

    View full-size slide

  26. @tech_christine @ryanhos
    Joy

    View full-size slide

  27. @tech_christine @ryanhos
    Surprise

    View full-size slide

  28. @tech_christine @ryanhos
    Anger?

    View full-size slide

  29. @tech_christine @ryanhos
    Sorrow?

    View full-size slide

  30. @tech_christine @ryanhos
    Image Composition

    View full-size slide

  31. @tech_christine @ryanhos
    Label detection

    View full-size slide

  32. @tech_christine @ryanhos
    @expertvagabond

    View full-size slide

  33. @tech_christine @ryanhos
    @expertvagabond

    View full-size slide

  34. @tech_christine @ryanhos
    @ripleyandrue

    View full-size slide

  35. @tech_christine @ryanhos
    @ripleyandrue

    View full-size slide

  36. @tech_christine @ryanhos
    Safe Search detection

    View full-size slide

  37. @tech_christine @ryanhos
    Yes it knows if it is a hotdog

    View full-size slide

  38. @tech_christine @ryanhos
    ...and when it's not

    View full-size slide

  39. @tech_christine @ryanhos
    Seems accurate…

    View full-size slide

  40. @tech_christine @ryanhos
    Oh, there might be a bit of bias…

    View full-size slide

  41. @tech_christine @ryanhos
    Violence is difficult too…

    View full-size slide

  42. @tech_christine @ryanhos
    Racy but not violent?!?

    View full-size slide

  43. @tech_christine @ryanhos
    Doctored images are hit and miss
    We chose not to include the image of a former US
    president photoshopped to look like a dictator and
    war criminal.
    Google labeled it “Possibly” a spoof; i.e. altered to be
    funny or offensive

    View full-size slide

  44. @tech_christine @ryanhos
    Text detection

    View full-size slide

  45. @tech_christine @ryanhos
    @letterfolk

    View full-size slide

  46. @tech_christine @ryanhos
    @letterfolk
    MY
    BRAIN
    HAS
    TO0
    MANY
    TABS
    OPEN
    Dechtlatte
    THIS WEEK
    pt de C
    CAit s pg

    View full-size slide

  47. @tech_christine @ryanhos
    @letterfolk

    View full-size slide

  48. @tech_christine @ryanhos
    @letterfolk
    YOURE NOT THE
    BOSS OF ME..
    IWHISPER UNDER
    MY BREATH AS I
    CLEAN UP ALL OF
    MY CHILDREN'S
    MESSES.
    TRE

    View full-size slide

  49. @tech_christine @ryanhos
    @letterfolk

    View full-size slide

  50. @tech_christine @ryanhos
    @letterfolk
    Seth
    Elizabeth
    Mat
    IT'S BEGINNING TO
    amazon Prime
    LOOK A LOT LIKE...
    vmazon Prime
    echo
    I GOT MY MONEY'S
    WORTH FROM MY
    1t\nare en
    a ton Prime
    AMAZON PRIME
    MEMBERSHIP
    Nany
    echo
    eciro

    View full-size slide

  51. @tech_christine @ryanhos
    Logo detection

    View full-size slide

  52. @tech_christine @ryanhos

    View full-size slide

  53. @tech_christine @ryanhos
    @nike

    View full-size slide

  54. @tech_christine @ryanhos
    @nike

    View full-size slide

  55. @tech_christine @ryanhos
    @nike
    1. nike
    2. nike plus
    3. nike azul
    4. blue nike
    5. nike store

    View full-size slide

  56. @tech_christine @ryanhos
    @christine_seeman

    View full-size slide

  57. @tech_christine @ryanhos
    @christine_seeman

    View full-size slide

  58. @tech_christine @ryanhos
    @christine_seeman
    1. etelä suomen sanomat
    2. rogue status
    3. american horror story
    4. delta skymiles
    5. deník
    !"!"

    View full-size slide

  59. @tech_christine @ryanhos
    Landmark detection

    View full-size slide

  60. @tech_christine @ryanhos
    @visit_nebraska

    View full-size slide

  61. @tech_christine @ryanhos
    @iloveny

    View full-size slide

  62. @tech_christine @ryanhos
    @iloveny

    View full-size slide

  63. @tech_christine @ryanhos
    @expertvagabond

    View full-size slide

  64. @tech_christine @ryanhos
    Have you been to London, NE?

    View full-size slide

  65. @tech_christine @ryanhos
    The British Museum there is lovely

    View full-size slide

  66. @tech_christine @ryanhos
    British Museum, alternate photo

    View full-size slide

  67. @tech_christine @ryanhos
    So much alike

    View full-size slide

  68. @tech_christine @ryanhos
    This wasn’t even a building…

    View full-size slide

  69. @tech_christine @ryanhos
    But what does my
    instagram say about me?

    View full-size slide

  70. @tech_christine @ryanhos

    View full-size slide

  71. @tech_christine @ryanhos
    @christineseeman

    View full-size slide

  72. @tech_christine @ryanhos
    Blooper cloud

    View full-size slide

  73. @tech_christine @ryanhos
    Cloud AutoML Vision
    •Train ML models to classify
    images
    •Use your own defined labels.
    •Graphical user interface to
    train, evaluate, improve, and
    deploy models

    View full-size slide

  74. @tech_christine @ryanhos
    @visit_nebraska

    View full-size slide

  75. @tech_christine @ryanhos
    Vs @visitcalifornia

    View full-size slide

  76. @tech_christine @ryanhos
    Vs @iloveny

    View full-size slide

  77. @tech_christine @ryanhos
    Sydney, Australia

    Bangkok, Thailand
    Paris, France

    View full-size slide

  78. @tech_christine @ryanhos

    View full-size slide

  79. @tech_christine @ryanhos
    Evaluating Model Accuracy

    View full-size slide

  80. @tech_christine @ryanhos
    Confusion Matrix

    View full-size slide

  81. @tech_christine @ryanhos
    Skyline Predictions

    View full-size slide

  82. @tech_christine @ryanhos
    More training required…

    View full-size slide

  83. @tech_christine @ryanhos
    Urban Scenery Predictions

    View full-size slide

  84. @tech_christine @ryanhos
    Outdoor Scenery Predictions

    View full-size slide

  85. @tech_christine @ryanhos
    Where to go from here?
    What to do next with Google Cloud Vision

    View full-size slide

  86. @tech_christine @ryanhos
    Explore bias in Machine
    Learning?

    View full-size slide

  87. @tech_christine @ryanhos
    How much will this cost you?
    For us, about $65

    View full-size slide

  88. @tech_christine @ryanhos
    905.33!

    View full-size slide

  89. @tech_christine @ryanhos
    $40 for the Training Compute Hours


    View full-size slide

  90. @tech_christine @ryanhos
    Technology Used
    •Ruby (No Rails)
    •Rspec
    •Ruby Vision API Client Libraries
    •Google Cloud Storage
    •Rmagick (Ruby binding to Imagemagick)
    •MagicCloud tag cloud gem

    View full-size slide

  91. @tech_christine @ryanhos
    All the code
    https://github.com/hi-christine/cloud-vision-insta

    View full-size slide

  92. @tech_christine @ryanhos
    Reference Links
    •https://cloud.google.com/vision/
    •https://cloud.google.com/vision/docs/
    •https://googleapis.github.io/google-cloud-ruby/docs/
    •https://github.com/GoogleCloudPlatform/ruby-docs-
    samples/blob/master/vision/quickstart.rb
    •https://github.com/zverok/magic_cloud

    View full-size slide

  93. @tech_christine @ryanhos
    All the instagram accounts*
    @ripleyandrue
    @visit_nebraska
    @wolffolins
    @expertvagabond
    @iloveny
    @letterfolk
    @myraswim
    @visitcalifornia
    @nationalportraitgallery
    @nike
    * and none were harmed in the making of this talk

    View full-size slide

  94. @tech_christine @ryanhos
    getflywheel.com/about/careers

    View full-size slide

  95. @tech_christine @ryanhos
    Thank you for attending!

    View full-size slide

  96. @tech_christine @ryanhos
    For your evaluation consideration...

    View full-size slide

  97. @tech_christine @ryanhos
    Bonus Code Time

    View full-size slide

  98. @tech_christine @ryanhos
    image_annotator_client.rb

    View full-size slide

  99. @tech_christine @ryanhos
    image_annotator_client.rb

    View full-size slide

  100. @tech_christine @ryanhos
    image_annotator_client.rb

    View full-size slide

  101. @tech_christine @ryanhos
    image_annotator_client.rb

    View full-size slide

  102. @tech_christine @ryanhos
    analysis_reader.rb

    View full-size slide

  103. @tech_christine @ryanhos
    analysis_reader.rb

    View full-size slide

  104. @tech_christine @ryanhos
    raster_annotator.rb

    View full-size slide

  105. @tech_christine @ryanhos
    raster_annotator.rb

    View full-size slide