Scaling ML embedding models to serve a billion queries

Slide 1

Slide 1 text

Scaling embedding models to serve a billion queries Senthilkumar Gopal @sengopal

Slide 2

Slide 2 text

Slide 3

Slide 3 text

© 2022 eBay. All rights reserved. 3 Search @ eBay How can we discover items without describing them? This is a problem across many domains where search is a core functionality. Question to ponder can we provide users with the ability to “discover” through visual cues instead? https://unsplash.com/photos/2oUiUu5QAys

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

© 2022 eBay. All rights reserved. 6 So what is an embedding then? https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html Represents Semantic Similarity Similarity (sofa, couch) A real word example [R 768 ]

Slide 7

Slide 7 text

© 2022 eBay. All rights reserved. 7 So what is an embedding then? Large Language Models - GPT 3 [175 B] ○ 45 TB text data - Wikipedia and books Neural network learns word associations from a large corpus. ○ Detects synonymous words. ○ Suggests words for a partial sentence. https://en.wikipedia.org/wiki/Word2vec https://ruder.io/word-embeddings-1/ You shall know a word by the company it keeps - (Firth, J. R. 1957:11)

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Slide 12

Slide 12 text

© 2022 eBay. All rights reserved. 12 How do we “learn” an embedding? Devlin, Jacob, et al. "Bert: Pre-training of deep bidirectional transformers for language understanding." arXiv preprint arXiv:1810.04805 (2018). https://en.wikipedia.org/wiki/Convolutional_neural_network Text Encoder Image Encoder R 768 That's how an embedding looks like!!!

Slide 13

Slide 13 text

© 2022 eBay. All rights reserved. 13 Why do we need ANN? Exhaustive search curse of dimensionality ANN Approximate Nearest Neighbours All problems start with SCALE https://en.wikipedia.org/wiki/Proximity_analysis#/media/File:Euclidean_Voronoi_diagram.svg http://ann-benchmarks.com/index.html#algorithms

Slide 14

Slide 14 text

Slide 15

Slide 15 text

© 2022 eBay. All rights reserved. 15 How does this function? Display all inventory matching my visual appeal Is this a orange throw kilim pillow? Quickly pivot to entire inventory using a visual ﬁrst cue

Slide 16

Slide 16 text

Data Engineering

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Slide 20

Slide 20 text

Slide 21

Slide 21 text

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Slide 24

Slide 24 text

© 2022 eBay. All rights reserved. 24 Challenges with Apache Airflow Challenge Solution Multiple Spark versions Define task level parameters Multiple Docker image versions Python virtual environment packages Different platforms, zones, and network flakiness Retries, system monitoring Apache Airflow Logo

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Slide 27

Slide 27 text

© 2022 eBay. All rights reserved. 27 Model Drift • Seasonality • Aging of the models Actions • Metrics monitoring • Downstream evaluation • Retraining Evolution Data Drift • Data Integrity • Data pipelines Actions • Fault tolerance • Monitoring of time, cpu, memory, disk