Tweet
Tweet

Slide 1

Slide 1 text

James Thomas. Developer Advocate @ IBM ☁ @THOMASJ Serverless Machine Learning with

Slide 2

Slide 2 text

@THOMASJ

Slide 3

Slide 3 text

@THOMASJ

Slide 4

Slide 4 text

“CAN I WRITE SOME CODE TO DO THIS?” @THOMASJ

Slide 5

Slide 5 text

@THOMASJ

Slide 6

Slide 6 text

TWITTER
 SEARCH Check Search Results Register Search Twitter Search Face Recognition API Gateway Web Client ARCHITECTURE Auth0 Twitter API TensorFlow.js ~550 LOC JavaScript @THOMASJ

Slide 7

Slide 7 text

TWITTER
 SEARCH Check Search Results Register Search Twitter Search Face Recognition API Gateway Web Client Auth0 Twitter API TensorFlow.js ~550 LOC JavaScript @THOMASJ {“query”: “serverless”} {“job_id”: 12345} {“query”: “serverless”}

Slide 8

Slide 8 text

TWITTER
 SEARCH Check Search Results Register Search Twitter Search Face Recognition API Gateway Web Client Auth0 Twitter API TensorFlow.js ~550 LOC JavaScript @THOMASJ {“job_id”: 12345} {“url”: “…”}

Slide 9

Slide 9 text

TWITTER
 SEARCH Check Search Results Register Search Twitter Search Face Recognition API Gateway Web Client Auth0 Twitter API TensorFlow.js ~550 LOC JavaScript @THOMASJ {“job_id”: 12345}

Slide 10

Slide 10 text

TWITTER
 SEARCH Check Search Results Register Search Twitter Search Face Recognition API Gateway Web Client ARCHITECTURE Auth0 Twitter API TensorFlow.js ~550 LOC JavaScript @THOMASJ

Slide 11

Slide 11 text

@THOMASJ

Slide 12

Slide 12 text

@THOMASJ

Slide 13

Slide 13 text

@THOMASJ

Slide 14

Slide 14 text

IS THIS THE END? $ sls create -t openwhisk-nodejs $ npm install @tensorflow/tfjs-node $ sls deploy @THOMASJ

Slide 15

Slide 15 text

✋ NOT SO FAAS-T! @THOMASJ

Slide 16

Slide 16 text

MAKE IT WORK MAKE IT RIGHT MAKE IT FAST @THOMASJ

Slide 17

Slide 17 text

$ sls create -t openwhisk-nodejs $ npm install @tensorflow/tfjs-node $ sls deploy ERROR: DEPLOYMENT PACKAGE TOO LARGE. @THOMASJ Not so fast…

Slide 18

Slide 18 text

$ du -hd0 node_modules 243M node_modules @THOMASJ Not so fast…

Slide 19

Slide 19 text

libtensorflow.so = 142M @THOMASJ Not so fast…

Slide 20

Slide 20 text

let cold_start = false const library = 'libtensorflow.so' if (cold_start) { const data = from_object_store(library) write_to_fs(library, data) cold_start = true } // rest of function code… WORK AROUND? @THOMASJ

Slide 21

Slide 21 text

PRO-TIP:
 BUILD CUSTOM TF-JS RUNTIME @THOMASJ

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

FROM openwhisk/action-nodejs-v8:latest RUN npm install @tensorflow/tfjs-node ^^ Dockerfile $ docker build -t username/tfjs . $ docker push username/tfjs CUSTOM RUNTIMES @THOMASJ

Slide 24

Slide 24 text

service: machine-learning provider: name: openwhisk functions: classification: handler: lib.handler image: "username/tfjs" SERVERLESS.YAML @THOMASJ

Slide 25

Slide 25 text

$ sls create -t openwhisk-nodejs $ npm install @tensorflow/tfjs-node $ sls deploy SUCCESS! @THOMASJ

Slide 26

Slide 26 text

SINGLE CONFIGURATION FILE (JSON). ( MULTIPLE WEIGHTS SHARDS (BINARY). LOADING TENSORFLOW.JS MODELS @THOMASJ

Slide 27

Slide 27 text

// Pre-trained image recognition model. const mobilenet = require('@tensorflow-models/ mobilenet') // Load model from Google over HTTP. const model = await mobilenet.load() @THOMASJ MODEL LOADING

Slide 28

Slide 28 text

ReferenceError: fetch is not defined @THOMASJ // Pre-trained image recognition model. const mobilenet = require('@tensorflow-models/ mobilenet') // Load model from Google over HTTP. const model = await mobilenet.load()

Slide 29

Slide 29 text

// Pre-trained image recognition model. const mobilenet = require('@tensorflow-models/ mobilenet') // Make HTTP client available in runtime global.fetch = require('node-fetch') // Load model from Google over HTTP. const model = await mobilenet.load() ReferenceError: fetch is not defined @THOMASJ

Slide 30

Slide 30 text

const img = document.getElementById('img'); // Classify the image. const predictions = await model.classify(img); @THOMASJ MODEL PREDICTIONS

Slide 31

Slide 31 text

const img = document.getElementById('img'); // Classify the image. const predictions = await model.classify(img); @THOMASJ ReferenceError: document is not defined

Slide 32

Slide 32 text

@THOMASJ

Slide 33

Slide 33 text

TRANSLATE TO RGB VALUES pixel 1 = #f3fa0a pixel 2 = #ea0f16 pixel 3 = #120f0a pixel 4 = #644fab … @THOMASJ

Slide 34

Slide 34 text

const jpeg = require('jpeg-js') const buffer = fs.readFileSync('pug.jpg') const jpegData = jpeg.decode(jpegData, true) console.log(jpegData) /* { width: 320, height: 180, data: { '0': 91, '1': 64, ... } } // typed array */ @THOMASJ RGB values

Slide 35

Slide 35 text

PRO-TIP:
 IGNORE THE ALPHA CHANNEL @THOMASJ

Slide 36

Slide 36 text

const jpeg = require('jpeg-js') const buffer = fs.readFileSync('pug.jpg') const image = jpeg.decode(buffer, true) 
 // remove alpha channel from RGBA values const rgb = image.data.filter((el, i) => (++i % 4)) @THOMASJ

Slide 37

Slide 37 text

const jpeg = require('jpeg-js') const buffer = fs.readFileSync('pug.jpg') const image = jpeg.decode(buffer, true) const rgb = image.data.filter((el, i) => (++i % 4)) // convert rgb values to typed array const values = new Int32Array(rgb) // shape of tensor is image dimensions const shape = [image.height, image.width, 3] // input tensor from rgb values & image dimensions const input = tf.tensor3d(values, shape, ‘int32’) const predictions = await model.classify(input) @THOMASJ

Slide 38

Slide 38 text

MAKE IT WORK MAKE IT RIGHT MAKE IT FAST @THOMASJ

Slide 39

Slide 39 text

$ sls invoke -f classifier -p input data {"result": …} @THOMASJ

Slide 40

Slide 40 text

$ sls invoke -f classifier -p input data {"result": …} $ sls invoke -f classifier -p input data {"result": …} @THOMASJ

Slide 41

Slide 41 text

$ sls invoke -f classifier -p input data {"result": …} $ sls invoke -f classifier -p input data {"result": …} $ sls invoke -f classifier -p input data {"result": …} @THOMASJ

Slide 42

Slide 42 text

$ sls invoke -f classifier -p input data {"result": …} $ sls invoke -f classifier -p input data {"result": …} $ sls invoke -f classifier -p input data {"result": …} $ sls invoke -f classifier -p input data {"error": "OUT OF MEMORY"} @THOMASJ

Slide 43

Slide 43 text

const jpeg = require('jpeg-js') const buffer = fs.readFileSync('pug.jpg') const image = jpeg.decode(buffer, true) const rgb = image.data.filter((el, i) => (++i % 4)) const values = new Int32Array(rgb) const shape = [image.height, image.width, 3] const input = tf.tensor3d(values, shape, ‘int32’) const predictions = await model.classify(input) Allocates native memory NOT garbage collected @THOMASJ

Slide 44

Slide 44 text

PRO-TIP:
 USE DISPOSE() TO RELEASE MEMORY @THOMASJ

Slide 45

Slide 45 text

const jpeg = require('jpeg-js') const buffer = fs.readFileSync('pug.jpg') const image = jpeg.decode(buffer, true) const rgb = image.data.filter((el, i) => (++i % 4)) const values = new Int32Array(rgb) const shape = [image.height, image.width, 3] const input = tf.tensor3d(values, shape, ‘int32’) const predictions = await model.classify(input) input.dispose() @THOMASJ

Slide 46

Slide 46 text

MAKE IT WORK MAKE IT RIGHT MAKE IT FAST @THOMASJ

Slide 47

Slide 47 text

How can we improve this? ❄ COLD START ❄ (~8 seconds) WARM START (~5 seconds) @THOMASJ

Slide 48

Slide 48 text

❄ COLD START ❄ WARM START INITIALISATION 1200 ms 0 ms MODEL LOADING 3200 ms 2000 ms IMAGE LOADING 500 ms x 2 500 ms x 2 FACE DETECTION 700 - 900 ms x 2 700 - 900 ms x 2 EVERYTHING ELSE 1000 ms 500 ms TOTAL DURATION ~ 8 seconds ~ 5 seconds

Slide 49

Slide 49 text

FROM openwhisk/action-nodejs-v8:latest RUN npm install @tensorflow/tfjs-node COPY weights weights ADDING WEIGHTS

Slide 50

Slide 50 text

// load weight files from filesystem global.fetch = async (file) => { return { json: () => JSON.parse(fs.readFileSync(file, 'utf8')), arrayBuffer: () => fs.readFileSync(file) } }

Slide 51

Slide 51 text

❄ COLD START ❄ WARM START INITIALISATION 1200 ms 0 ms MODEL LOADING 2700 ms 1500 ms IMAGE LOADING 500 ms x 2 500 ms x 2 FACE DETECTION 700 - 900 ms x 2 700 - 900 ms x 2 EVERYTHING ELSE 1000 ms 500 ms TOTAL DURATION ~ 7.5 seconds ~ 4.5 seconds

Slide 52

Slide 52 text

@THOMASJ

Slide 53

Slide 53 text

const faceapi = require(‘face-api.js') let LOADED = false exports.load = async location => { if (!LOADED) { await faceapi.loadFaceDetectionModel(location) await faceapi.loadFaceRecognitionModel(location) await faceapi.loadFaceLandmarkModel(location) LOADED = true } return faceapi } @THOMASJ MODEL CACHING

Slide 54

Slide 54 text

❄ COLD START ❄ WARM START INITIALISATION 1200 ms 0 ms MODEL LOADING 2700 ms 0 ms IMAGE LOADING 500 ms x 2 500 ms x 2 FACE DETECTION 700 - 900 ms x 2 700 - 900 ms x 2 EVERYTHING ELSE 1000 ms 500 ms TOTAL DURATION ~ 7.5 seconds ~ 3 seconds

Slide 55

Slide 55 text

const faces = await models.allFaces(input, min_score) const face = faces[0] console.log(face.descriptor) // face descriptor > Float32Array (length: 128) [-0.1298077404499054, 0.08730170130729675, 0.03973294794559479, 0.03567018359899521, -0.09620543569326401, 0.03842385113239288, ... Cache Values in Redis @THOMASJ

Slide 56

Slide 56 text

COLD START ❄ + CACHE WARM START INITIALISATION 1200 ms 1200 ms 0 ms MODEL LOADING 2700 ms 2700 ms 0 ms IMAGE LOADING 500 ms x 2 500ms 500 ms FACE DETECTION 700 - 900 ms x 2 700 - 900 ms 700 - 900 ms EVERYTHING ELSE 1000 ms 1000 ms 500 ms TOTAL DURATION ~ 7.5 seconds ~ 6 seconds ~ 2.5 seconds

Slide 57

Slide 57 text

❄ COLD START ❄ (~8 seconds)
 (~7.5 - 6 seconds) WARM START (~5 seconds) (~ 2.5 seconds) cold start savings = ~6% - 25% warm start savings = ~50% @THOMASJ

Slide 58

Slide 58 text

MAKE IT WORK MAKE IT RIGHT MAKE IT FAST @THOMASJ

Slide 59

Slide 59 text

in the @THOMASJ

Slide 60

Slide 60 text

@THOMASJ

Slide 61

Slide 61 text

PRE-TRAINED MODELS @THOMASJ

Slide 62

Slide 62 text

PRE-TRAINED MODELS COST @THOMASJ

Slide 63

Slide 63 text

PRE-TRAINED MODELS COST 1 PERFORMANCE @THOMASJ @THOMASJ

Slide 64

Slide 64 text

PRE-TRAINED MODELS COST 1 PERFORMANCE EASE OF USE @THOMASJ

Slide 65

Slide 65 text

PRE-TRAINED MODELS COST 1 PERFORMANCE EASE OF USE ( TRAINING @THOMASJ

Slide 66

Slide 66 text

@THOMASJ

Slide 67

Slide 67 text

PRE-TRAINED MODELS COST 1 PERFORMANCE EASE OF USE ( TRAINING @THOMASJ

Slide 68

Slide 68 text

CONCLUSION TF.JS + SERVERLESS = SCALABLE ML IN THE CLOUD @THOMASJ

Slide 69

Slide 69 text

@THOMASJ

Slide 70

Slide 70 text

QUESTIONS? @THOMASJ jamesthom.as jthomas openwhisk.org bluemix.net/openwhisk