Exploring the World Using Cloud Vision & Twilio

Slide 1

Slide 1 text

Exploring the World Using Cloud Vision & Twilio Julia Ferraioli @juliaferraioli Software Engineer Google Open Source Programs Office

Slide 2

Slide 2 text

★ All code is licensed under the Apache License, Version 2.0 bit.ly/apache-v2

Slide 3

Slide 3 text

@juliaferraioli Other stuff I like to do

Slide 4

Slide 4 text

People think of machine learning like this

Slide 5

Slide 5 text

CC image courtesy of M.Kemal: https://flic.kr/p/bYAsCs

Slide 6

Slide 6 text

But really, machine learning is this

Slide 7

Slide 7 text

→ ⚙ → ❓ data algorithm insight

Slide 8

Slide 8 text

But that means we can do cool stuff

Slide 9

Slide 9 text

@juliaferraioli Like this

Slide 10

Slide 10 text

@juliaferraioli Or this

Slide 11

Slide 11 text

But what about the data?

Slide 12

Slide 12 text

You need a bunch of it

Slide 13

Slide 13 text

@juliaferraioli Machine learning models as a Service ⚙ Models pre-trained on extensive amounts of curated data

Slide 14

Slide 14 text

Google Cloud Vision ● Detect faces, landmarks, logos, text, and more ● Perform sentiment analysis ● Straightforward REST API ● Works on a base64-encoded image ● Connects to Google Cloud Storage

Slide 15

Slide 15 text

The primary place I take pictures is on my phone

Slide 16

Slide 16 text

...and I want it to be similar to existing Q&A methods...

Slide 17

Slide 17 text

Oh, I know! Twilio!

Slide 18

Slide 18 text

@juliaferraioli Lifecycle of a picture

Slide 19

Slide 19 text

@juliaferraioli Receiving the MMS @app.route("/", methods=['GET', 'POST']) def receive_message(): for i in range(int(request.values.get('NumMedia', None))): media_url = request.values.get('MediaUrl%i' % i, None) image = requests.get(media_url).content labels = get_labels(image) resp = construct_message(labels) return str(resp)

Slide 20

Slide 20 text

Slide 21

Slide 21 text

@juliaferraioli Processing the image def get_labels(image, num_retries=3, max_results=3): # Set up the service that can access the API # Prepare the image for the API # Construct the request # Send it off to the API # Return the labels

Slide 22

Slide 22 text

@juliaferraioli Authenticating with the API def get_labels(image, num_retries=3, max_results=3): # Set up the service that can access the API http = httplib2.Http() credentials = GoogleCredentials.get_application_default().create_scoped( ['https://www.googleapis.com/auth/cloud-platform']) credentials.authorize(http) service = discovery.build('vision', 'v1', http=http, discoveryServiceUrl=DISCOVERY_URL)

Slide 23

Slide 23 text

@juliaferraioli Encoding the image def get_labels(image, num_retries=3, max_results=3): # Prepare the image for the API image_content = base64.b64encode(image)

Slide 24

Slide 24 text

@juliaferraioli Crafting the request def get_labels(image, num_retries=3, max_results=3): # Construct the request service_request = service.images().annotate( body={'requests': [{ 'image': { 'content': image_content, }, 'features': [{ 'type': 'LABEL_DETECTION', 'maxResults': max_results, }] }] })

Slide 25

Slide 25 text

@juliaferraioli Executing the request def get_labels(image, num_retries=3, max_results=3): # Send it off to the API response = service_request.execute(num_retries=num_retries)

Slide 26

Slide 26 text

@juliaferraioli Returning the labels def get_labels(image, num_retries=3, max_results=3): # Return the labels if('responses' in response and 'labelAnnotations' in response['responses'][0]): # Hey, the API found something! return response['responses'][0]['labelAnnotations'] else: # Sigh, no dice this time :-( return []

Slide 27

Slide 27 text

CC image courtesy of John Morgan: https://flic.kr/p/5XbG1m

Slide 28

Slide 28 text

Slide 29

Slide 29 text

@juliaferraioli Sending back the results def construct_message(labels): label_desc = "" # Go through labels and turn them into text of the response # Turn the string into a twiml response

Slide 30

Slide 30 text

@juliaferraioli Sending back the results def construct_message(labels): # Go through labels and turn them into text of the response for i in range(len(labels)): # We've got an answer! Let's tell them about it label_desc += 'Score is %s for %s\n' % (labels[i]['score'], labels[i]['description'])

Slide 31

Slide 31 text

@juliaferraioli Sending back the results def construct_message(labels): # Turn the string into a twiml response resp = twilio.twiml.Response() resp.message(label_desc)

Slide 32

Slide 32 text

@juliaferraioli Responding to the MMS @app.route("/", methods=['GET', 'POST']) def receive_message(): for i in range(int(request.values.get('NumMedia', None))): media_url = request.values.get('MediaUrl%i' % i, None) image = requests.get(media_url).content labels = get_labels(image) resp = construct_message(labels) return str(resp)

Slide 33

Slide 33 text

Demo time!

Slide 34

Slide 34 text

@juliaferraioli

Slide 35

Slide 35 text

@juliaferraioli What’s that? We found some labels for your image: Score is 0.90518606 for cephalopod Score is 0.79291707 for marine invertebrates Score is 0.78696847 for octopus

Slide 36

Slide 36 text

@juliaferraioli CC image courtesy of John Morgan: https://flic.kr/p/5XbG1m

Slide 37

Slide 37 text

@juliaferraioli What’s that? We found some labels for your image: Score is 0.59691668 for brand Score is 0.58166391 for writing

Slide 38

Slide 38 text

What? NO DICE?!?

Slide 39

Slide 39 text

This is cool, but...

Slide 40

Slide 40 text

Why?

Slide 41

Slide 41 text

@juliaferraioli Why? ¯\_(ツ)_/¯ ● Aid tourists around the world with landmark detection ● Build a bot to sort your mail and text you the senders with OCR ● Run a simple Q&A service for people with low vision ● Help folks analyze interactions using sentiment analysis ● Boost conference scanners to recognize logos on business cards ● … and lots more

Slide 42

Slide 42 text

Learn more ● Google Cloud Vision developer documentation: http://bit.ly/gcp-vision ● Code from today: http://bit.ly/twilio-vision ● Code from today running on Kubernetes (!!!): http://bit.ly/twilio-vision-k8s

Slide 43

Slide 43 text

Thanks!