Exploring the World Using Cloud Vision & Twilio

62b2249a42d624dc93357931f0f5d2f1?s=47 juliaferraioli
September 20, 2016

Exploring the World Using Cloud Vision & Twilio

We take photos for many reasons: to capture moments, because something's funny or pretty, because we're curious about something and want to know more. Luckily, there's a whole area of machine learning dedicated to understanding images.

We’ll explore how we can use machine learning ﹘ without the math ﹘ with Twilio and the pictures that we take to gain a greater understanding of the world around us. We'll dig into various ways that this fusion can be used to make people's lives richer and better.

Accompanying blog post: http://www.blog.juliaferraioli.com/2016/02/exploring-world-using-vision-twilio.html

62b2249a42d624dc93357931f0f5d2f1?s=128

juliaferraioli

September 20, 2016
Tweet

Transcript

  1. Exploring the World Using Cloud Vision & Twilio Julia Ferraioli

    @juliaferraioli Software Engineer Google Open Source Programs Office
  2. ★ All code is licensed under the Apache License, Version

    2.0 bit.ly/apache-v2
  3. @juliaferraioli Other stuff I like to do

  4. People think of machine learning like this

  5. CC image courtesy of M.Kemal: https://flic.kr/p/bYAsCs

  6. But really, machine learning is this

  7. → ⚙ → ❓ data algorithm insight

  8. But that means we can do cool stuff

  9. @juliaferraioli Like this

  10. @juliaferraioli Or this

  11. But what about the data?

  12. You need a bunch of it

  13. @juliaferraioli Machine learning models as a Service ⚙ Models pre-trained

    on extensive amounts of curated data
  14. Google Cloud Vision • Detect faces, landmarks, logos, text, and

    more • Perform sentiment analysis • Straightforward REST API • Works on a base64-encoded image • Connects to Google Cloud Storage
  15. The primary place I take pictures is on my phone

  16. ...and I want it to be similar to existing Q&A

    methods...
  17. Oh, I know! Twilio!

  18. @juliaferraioli Lifecycle of a picture

  19. @juliaferraioli Receiving the MMS @app.route("/", methods=['GET', 'POST']) def receive_message(): for

    i in range(int(request.values.get('NumMedia', None))): media_url = request.values.get('MediaUrl%i' % i, None) image = requests.get(media_url).content labels = get_labels(image) resp = construct_message(labels) return str(resp)
  20. @juliaferraioli Receiving the MMS @app.route("/", methods=['GET', 'POST']) def receive_message(): for

    i in range(int(request.values.get('NumMedia', None))): media_url = request.values.get('MediaUrl%i' % i, None) image = requests.get(media_url).content labels = get_labels(image) resp = construct_message(labels) return str(resp)
  21. @juliaferraioli Processing the image def get_labels(image, num_retries=3, max_results=3): # Set

    up the service that can access the API # Prepare the image for the API # Construct the request # Send it off to the API # Return the labels
  22. @juliaferraioli Authenticating with the API def get_labels(image, num_retries=3, max_results=3): #

    Set up the service that can access the API http = httplib2.Http() credentials = GoogleCredentials.get_application_default().create_scoped( ['https://www.googleapis.com/auth/cloud-platform']) credentials.authorize(http) service = discovery.build('vision', 'v1', http=http, discoveryServiceUrl=DISCOVERY_URL)
  23. @juliaferraioli Encoding the image def get_labels(image, num_retries=3, max_results=3): # Prepare

    the image for the API image_content = base64.b64encode(image)
  24. @juliaferraioli Crafting the request def get_labels(image, num_retries=3, max_results=3): # Construct

    the request service_request = service.images().annotate( body={'requests': [{ 'image': { 'content': image_content, }, 'features': [{ 'type': 'LABEL_DETECTION', 'maxResults': max_results, }] }] })
  25. @juliaferraioli Executing the request def get_labels(image, num_retries=3, max_results=3): # Send

    it off to the API response = service_request.execute(num_retries=num_retries)
  26. @juliaferraioli Returning the labels def get_labels(image, num_retries=3, max_results=3): # Return

    the labels if('responses' in response and 'labelAnnotations' in response['responses'][0]): # Hey, the API found something! return response['responses'][0]['labelAnnotations'] else: # Sigh, no dice this time :-( return []
  27. CC image courtesy of John Morgan: https://flic.kr/p/5XbG1m

  28. @juliaferraioli Receiving the MMS @app.route("/", methods=['GET', 'POST']) def receive_message(): for

    i in range(int(request.values.get('NumMedia', None))): media_url = request.values.get('MediaUrl%i' % i, None) image = requests.get(media_url).content labels = get_labels(image) response = construct_message(labels) return str(response)
  29. @juliaferraioli Sending back the results def construct_message(labels): label_desc = ""

    # Go through labels and turn them into text of the response # Turn the string into a twiml response
  30. @juliaferraioli Sending back the results def construct_message(labels): # Go through

    labels and turn them into text of the response for i in range(len(labels)): # We've got an answer! Let's tell them about it label_desc += 'Score is %s for %s\n' % (labels[i]['score'], labels[i]['description'])
  31. @juliaferraioli Sending back the results def construct_message(labels): # Turn the

    string into a twiml response resp = twilio.twiml.Response() resp.message(label_desc)
  32. @juliaferraioli Responding to the MMS @app.route("/", methods=['GET', 'POST']) def receive_message():

    for i in range(int(request.values.get('NumMedia', None))): media_url = request.values.get('MediaUrl%i' % i, None) image = requests.get(media_url).content labels = get_labels(image) resp = construct_message(labels) return str(resp)
  33. Demo time!

  34. @juliaferraioli

  35. @juliaferraioli What’s that? We found some labels for your image:

    Score is 0.90518606 for cephalopod Score is 0.79291707 for marine invertebrates Score is 0.78696847 for octopus
  36. @juliaferraioli CC image courtesy of John Morgan: https://flic.kr/p/5XbG1m

  37. @juliaferraioli What’s that? We found some labels for your image:

    Score is 0.59691668 for brand Score is 0.58166391 for writing
  38. What? NO DICE?!?

  39. This is cool, but...

  40. Why?

  41. @juliaferraioli Why? ¯\_(ツ)_/¯ • Aid tourists around the world with

    landmark detection • Build a bot to sort your mail and text you the senders with OCR • Run a simple Q&A service for people with low vision • Help folks analyze interactions using sentiment analysis • Boost conference scanners to recognize logos on business cards • … and lots more
  42. Learn more • Google Cloud Vision developer documentation: http://bit.ly/gcp-vision •

    Code from today: http://bit.ly/twilio-vision • Code from today running on Kubernetes (!!!): http://bit.ly/twilio-vision-k8s
  43. Thanks!