## Slide 1

### Slide 1 text

1 Theory and Applications of Graph-based Search Yusuke Matsui The University of Tokyo CVPR 2023 Tutorial on Neural Search in Action

## Slide 2

### Slide 2 text

2 Yusuke Matsui ✓ Image retrieval ✓ Large-scale indexing http://yusukematsui.me Lecturer (Assistant Professor), the University of Tokyo, Japan @utokyo_bunny ARM 4-bit PQ [Matsui+, ICASSP 22] Image Retrieval in the Wild [Matsui+, CVPR 20, tutorial] @matsui528

## Slide 3

### Slide 3 text

3 ➢ Background ➢ Graph-based search ✓ Basic (construction and search) ✓ Observation ✓ Properties ➢ Representative works ✓ HNSW, NSG, NGT, Vamana ➢ Discussion

## Slide 4

### Slide 4 text

4 ➢ Background ➢ Graph-based search ✓ Basic (construction and search) ✓ Observation ✓ Properties ➢ Representative works ✓ HNSW, NSG, NGT, Vamana ➢ Discussion

## Slide 5

### Slide 5 text

Search 𝒙1 , 𝒙2 , … , 𝒙𝑁 𝒙𝑛 ∈ ℝ𝐷 5 ➢𝑁 𝐷-dim database vectors: 𝒙𝑛 𝑛=1 𝑁 Nearest Neighbor Search; NN

## Slide 6

### Slide 6 text

0.23 3.15 0.65 1.43 Search 0.20 3.25 0.72 1.68 𝒒 ∈ ℝ𝐷 𝒙74 argmin 𝑛∈ 1,2,…,𝑁 𝒒 − 𝒙𝑛 2 2 Result 𝒙1 , 𝒙2 , … , 𝒙𝑁 𝒙𝑛 ∈ ℝ𝐷 6 ➢𝑁 𝐷-dim database vectors: 𝒙𝑛 𝑛=1 𝑁 ➢Given a query 𝒒, find the closest vector from the database ➢One of the fundamental problems in computer science ➢Solution: linear scan, 𝑂 𝑁𝐷 , slow  Nearest Neighbor Search; NN Often, argmax + inner product is also considered. Don’t care in this talk.

## Slide 7

### Slide 7 text

0.23 3.15 0.65 1.43 Search 0.20 3.25 0.72 1.68 𝒒 ∈ ℝ𝐷 𝒙74 argmin 𝑛∈ 1,2,…,𝑁 𝒒 − 𝒙𝑛 2 2 Result 𝒙1 , 𝒙2 , … , 𝒙𝑁 𝒙𝑛 ∈ ℝ𝐷 Approximate Nearest Neighbor Search; ANN ➢Faster search ➢Don’t necessarily have to be exact neighbors ➢Trade off: runtime, accuracy, and memory-consumption 7

## Slide 8

### Slide 8 text

0.23 3.15 0.65 1.43 Search 0.20 3.25 0.72 1.68 𝒒 ∈ ℝ𝐷 𝒙74 argmin 𝑛∈ 1,2,…,𝑁 𝒒 − 𝒙𝑛 2 2 Result 𝒙1 , 𝒙2 , … , 𝒙𝑁 𝒙𝑛 ∈ ℝ𝐷 Approximate Nearest Neighbor Search; ANN ➢Faster search ➢Don’t necessarily have to be exact neighbors ➢Trade off: runtime, accuracy, and memory-consumption 8 ➢ In this talk, suppose: 𝑁 < 109 ➢ All data can be loaded on memory

## Slide 9

### Slide 9 text

9 Real-world use cases 1: multimodal search Image are from: https://github.com/haltakov/natural-language-image-search Credit: Photos by Genton Damian, bruce mars, Dalal Nizam, and Richard Burlton on Unsplash

## Slide 10

### Slide 10 text

10 Real-world use cases 1: multimodal search Image are from: https://github.com/haltakov/natural-language-image-search Credit: Photos by Genton Damian, bruce mars, Dalal Nizam, and Richard Burlton on Unsplash 𝒙1 CLIP Image Encoder

## Slide 11

### Slide 11 text

11 Real-world use cases 1: multimodal search Image are from: https://github.com/haltakov/natural-language-image-search Credit: Photos by Genton Damian, bruce mars, Dalal Nizam, and Richard Burlton on Unsplash 𝒙1 , 𝒙2 , CLIP Image Encoder

## Slide 12

### Slide 12 text

12 Real-world use cases 1: multimodal search Image are from: https://github.com/haltakov/natural-language-image-search Credit: Photos by Genton Damian, bruce mars, Dalal Nizam, and Richard Burlton on Unsplash 𝒙1 , 𝒙2 , … , 𝒙𝑁 … CLIP Image Encoder

## Slide 13

### Slide 13 text

13 Real-world use cases 1: multimodal search Image are from: https://github.com/haltakov/natural-language-image-search Credit: Photos by Genton Damian, bruce mars, Dalal Nizam, and Richard Burlton on Unsplash 0.23 3.15 0.65 1.43 Search 𝒙1 , 𝒙2 , … , 𝒙𝑁 CLIP Text Encoder … CLIP Image Encoder “Two dogs playing in the snow”

## Slide 14

### Slide 14 text

14 Real-world use cases 1: multimodal search Image are from: https://github.com/haltakov/natural-language-image-search Credit: Photos by Genton Damian, bruce mars, Dalal Nizam, and Richard Burlton on Unsplash “Two dogs playing in the snow” 0.23 3.15 0.65 1.43 Search 0.20 3.25 0.72 1.68 argmin 𝒒 − 𝒙𝑛 2 2 Result 𝒙1 , 𝒙2 , … , 𝒙𝑁 CLIP Text Encoder … CLIP Image Encoder

## Slide 15

### Slide 15 text

15 Real-world use cases 1: multimodal search Image are from: https://github.com/haltakov/natural-language-image-search Credit: Photos by Genton Damian, bruce mars, Dalal Nizam, and Richard Burlton on Unsplash “Two dogs playing in the snow” 0.23 3.15 0.65 1.43 Search 0.20 3.25 0.72 1.68 argmin 𝒒 − 𝒙𝑛 2 2 Result 𝒙1 , 𝒙2 , … , 𝒙𝑁 CLIP Text Encoder … CLIP Image Encoder ➢ Encoder determines the upper bound of the accuracy of the system ➢ ANN determines a trade-off between accuracy, runtime, and memory

## Slide 16

### Slide 16 text

16 Real-world use cases 2: LLM + embedding Texts are from: https://github.com/openai/openaicookbook/blob/main/examples/Question_answering_using_embeddings.ipynb Icon credit: https://ja.wikipedia.org/wiki/ChatGPT "Who won curling gold at the 2022 Winter Olympics?" ChatGPT 3.5 (trained in 2021)

## Slide 17

### Slide 17 text

17 Real-world use cases 2: LLM + embedding Texts are from: https://github.com/openai/openaicookbook/blob/main/examples/Question_answering_using_embeddings.ipynb Icon credit: https://ja.wikipedia.org/wiki/ChatGPT "Who won curling gold at the 2022 Winter Olympics?" ChatGPT 3.5 (trained in 2021) “I'm sorry, but as an AI language model, I don't have information about the future events.” Ask 

## Slide 18

### Slide 18 text

18 Real-world use cases 2: LLM + embedding Texts are from: https://github.com/openai/openaicookbook/blob/main/examples/Question_answering_using_embeddings.ipynb Icon credit: https://ja.wikipedia.org/wiki/ChatGPT "Who won curling gold at the 2022 Winter Olympics?" ChatGPT 3.5 (trained in 2021)

## Slide 19

### Slide 19 text

19 Real-world use cases 2: LLM + embedding Texts are from: https://github.com/openai/openaicookbook/blob/main/examples/Question_answering_using_embeddings.ipynb Icon credit: https://ja.wikipedia.org/wiki/ChatGPT "Who won curling gold at the 2022 Winter Olympics?" ChatGPT 3.5 (trained in 2021) “Damir Sharipzyanov¥n¥n=Career…” “Lviv bid for the 2022 Winter…” … “Chinami Yoshida¥n¥n==Personal…”

## Slide 20

### Slide 20 text

20 Real-world use cases 2: LLM + embedding 𝒙1 , Texts are from: https://github.com/openai/openaicookbook/blob/main/examples/Question_answering_using_embeddings.ipynb Icon credit: https://ja.wikipedia.org/wiki/ChatGPT "Who won curling gold at the 2022 Winter Olympics?" ChatGPT 3.5 (trained in 2021) “Damir Sharipzyanov¥n¥n=Career…” “Lviv bid for the 2022 Winter…” … “Chinami Yoshida¥n¥n==Personal…” Text Encoder

## Slide 21

### Slide 21 text

21 Real-world use cases 2: LLM + embedding 𝒙1 , 𝒙2 , Texts are from: https://github.com/openai/openaicookbook/blob/main/examples/Question_answering_using_embeddings.ipynb Icon credit: https://ja.wikipedia.org/wiki/ChatGPT "Who won curling gold at the 2022 Winter Olympics?" ChatGPT 3.5 (trained in 2021) “Damir Sharipzyanov¥n¥n=Career…” “Lviv bid for the 2022 Winter…” … “Chinami Yoshida¥n¥n==Personal…” Text Encoder

## Slide 22

### Slide 22 text

22 Real-world use cases 2: LLM + embedding 𝒙1 , 𝒙2 , … , 𝒙𝑁 Texts are from: https://github.com/openai/openaicookbook/blob/main/examples/Question_answering_using_embeddings.ipynb Icon credit: https://ja.wikipedia.org/wiki/ChatGPT "Who won curling gold at the 2022 Winter Olympics?" ChatGPT 3.5 (trained in 2021) “Damir Sharipzyanov¥n¥n=Career…” “Lviv bid for the 2022 Winter…” … “Chinami Yoshida¥n¥n==Personal…” Text Encoder

## Slide 23

### Slide 23 text

23 Real-world use cases 2: LLM + embedding 0.23 3.15 0.65 1.43 𝒙1 , 𝒙2 , … , 𝒙𝑁 Texts are from: https://github.com/openai/openaicookbook/blob/main/examples/Question_answering_using_embeddings.ipynb Icon credit: https://ja.wikipedia.org/wiki/ChatGPT "Who won curling gold at the 2022 Winter Olympics?" ChatGPT 3.5 (trained in 2021) “Damir Sharipzyanov¥n¥n=Career…” “Lviv bid for the 2022 Winter…” … Text Encoder “Chinami Yoshida¥n¥n==Personal…” Text Encoder

## Slide 24

### Slide 24 text

24 Real-world use cases 2: LLM + embedding 0.23 3.15 0.65 1.43 0.20 3.25 0.72 1.68 argmin 𝒒 − 𝒙𝑛 2 2 Result 𝒙1 , 𝒙2 , … , 𝒙𝑁 Texts are from: https://github.com/openai/openaicookbook/blob/main/examples/Question_answering_using_embeddings.ipynb Icon credit: https://ja.wikipedia.org/wiki/ChatGPT "Who won curling gold at the 2022 Winter Olympics?" ChatGPT 3.5 (trained in 2021) Search “Damir Sharipzyanov¥n¥n=Career…” “Lviv bid for the 2022 Winter…” … Text Encoder “Chinami Yoshida¥n¥n==Personal…” Text Encoder “List of 2022 Winter Olympics medal winners…”

## Slide 25

### Slide 25 text

25 Real-world use cases 2: LLM + embedding 0.23 3.15 0.65 1.43 0.20 3.25 0.72 1.68 argmin 𝒒 − 𝒙𝑛 2 2 Result 𝒙1 , 𝒙2 , … , 𝒙𝑁 Texts are from: https://github.com/openai/openaicookbook/blob/main/examples/Question_answering_using_embeddings.ipynb Icon credit: https://ja.wikipedia.org/wiki/ChatGPT "Who won curling gold at the 2022 Winter Olympics?" Search Update “Damir Sharipzyanov¥n¥n=Career…” “Lviv bid for the 2022 Winter…” … Text Encoder “Chinami Yoshida¥n¥n==Personal…” Text Encoder “Who won curling gold at the 2022 Winter Olympics? Use the bellow articles: List of 2022 Winter Olympics medal winners…” “List of 2022 Winter Olympics medal winners…” ChatGPT 3.5 (trained in 2021)

## Slide 26

### Slide 26 text

26 Real-world use cases 2: LLM + embedding 0.23 3.15 0.65 1.43 0.20 3.25 0.72 1.68 argmin 𝒒 − 𝒙𝑛 2 2 Result 𝒙1 , 𝒙2 , … , 𝒙𝑁 Texts are from: https://github.com/openai/openaicookbook/blob/main/examples/Question_answering_using_embeddings.ipynb Icon credit: https://ja.wikipedia.org/wiki/ChatGPT "Who won curling gold at the 2022 Winter Olympics?" “Niklas Edin, Oskar Eriksson, …” Search Update ☺ “Damir Sharipzyanov¥n¥n=Career…” “Lviv bid for the 2022 Winter…” … Text Encoder “Chinami Yoshida¥n¥n==Personal…” Text Encoder “Who won curling gold at the 2022 Winter Olympics? Use the bellow articles: List of 2022 Winter Olympics medal winners…” “List of 2022 Winter Olympics medal winners…” ChatGPT 3.5 (trained in 2021)

## Slide 27

### Slide 27 text

27 Real-world use cases 2: LLM + embedding 0.23 3.15 0.65 1.43 0.20 3.25 0.72 1.68 argmin 𝒒 − 𝒙𝑛 2 2 Result 𝒙1 , 𝒙2 , … , 𝒙𝑁 Texts are from: https://github.com/openai/openaicookbook/blob/main/examples/Question_answering_using_embeddings.ipynb Icon credit: https://ja.wikipedia.org/wiki/ChatGPT "Who won curling gold at the 2022 Winter Olympics?" “Niklas Edin, Oskar Eriksson, …” Search Update ☺ “Damir Sharipzyanov¥n¥n=Career…” “Lviv bid for the 2022 Winter…” … Text Encoder “Chinami Yoshida¥n¥n==Personal…” Text Encoder “Who won curling gold at the 2022 Winter Olympics? Use the bellow articles: List of 2022 Winter Olympics medal winners…” “List of 2022 Winter Olympics medal winners…” ChatGPT 3.5 (trained in 2021) Embedding+ANN is the current easiest way to provide knowledge to LLM

## Slide 28

### Slide 28 text

28 Real-world use cases 2: LLM + embedding 0.23 3.15 0.65 1.43 0.20 3.25 0.72 1.68 argmin 𝒒 − 𝒙𝑛 2 2 Result 𝒙1 , 𝒙2 , … , 𝒙𝑁 Texts are from: https://github.com/openai/openaicookbook/blob/main/examples/Question_answering_using_embeddings.ipynb Icon credit: https://ja.wikipedia.org/wiki/ChatGPT "Who won curling gold at the 2022 Winter Olympics?" “Niklas Edin, Oskar Eriksson, …” Search Update ☺ “Damir Sharipzyanov¥n¥n=Career…” “Lviv bid for the 2022 Winter…” … Text Encoder “Chinami Yoshida¥n¥n==Personal…” Text Encoder “Who won curling gold at the 2022 Winter Olympics? Use the bellow articles: List of 2022 Winter Olympics medal winners…” “List of 2022 Winter Olympics medal winners…” ChatGPT 3.5 (trained in 2021) Embedding+ANN is the current easiest way to provide knowledge to LLM https://em-content.zobj.net/thumbs/120/twitter/322/thinking-face_1f914.png Vector DB???

## Slide 29

### Slide 29 text

29 Three levels of technology Milvus Pinecone Qdrant ScaNN (4-bit PQ) [Guo+, ICML 2020] Algorithm ➢ Scientific paper ➢ Math ➢ Often, by researchers Library ➢ Implementations of algorithms ➢ Usually, a search function only ➢ By researchers, developers, etc Service (e.g., vector DB) ➢ Library + (handling metadata, serving, scaling, IO, CRUD, etc) ➢ Usually, by companies Product Quantization + Inverted Index (PQ, IVFPQ) [Jégou+, TPAMI 2011] Hierarchical Navigable Small World (HNSW) [Malkov+, TPAMI 2019] Weaviate Vertex AI Matching Engine faiss NMSLIB hnswlib Vald ScaNN jina

## Slide 30

### Slide 30 text

Three levels of technology 30 Milvus Pinecone Qdrant ScaNN (4-bit PQ) [Guo+, ICML 2020] Algorithm ➢ Scientific paper ➢ Math ➢ Often, by researchers Library ➢ Implementations of algorithms ➢ Usually, a search function only ➢ By researchers, developers, etc Service (e.g., vector DB) ➢ Library + (handling metadata, serving, scaling, IO, CRUD, etc) ➢ Usually, by companies Weaviate Vertex AI Matching Engine NMSLIB hnswlib Vald ScaNN jina Product Quantization + Inverted Index (PQ, IVFPQ) [Jégou+, TPAMI 2011] Hierarchical Navigable Small World (HNSW) [Malkov+, TPAMI 2019] faiss One library may implement multiple algorithms  “I benchmarked faiss” ☺ “I benchmarked PQ in faiss”

## Slide 31

### Slide 31 text

Three levels of technology 31 Milvus Pinecone Qdrant ScaNN (4-bit PQ) [Guo+, ICML 2020] Algorithm ➢ Scientific paper ➢ Math ➢ Often, by researchers Library ➢ Implementations of algorithms ➢ Usually, a search function only ➢ By researchers, developers, etc Service (e.g., vector DB) ➢ Library + (handling metadata, serving, scaling, IO, CRUD, etc) ➢ Usually, by companies Product Quantization + Inverted Index (PQ, IVFPQ) [Jégou+, TPAMI 2011] Weaviate Vertex AI Matching Engine Vald ScaNN jina Hierarchical Navigable Small World (HNSW) [Malkov+, TPAMI 2019] faiss NMSLIB hnswlib One algorithm may be implemented in multiple libraries

## Slide 32

### Slide 32 text

Three levels of technology 32 Milvus Pinecone Qdrant Algorithm ➢ Scientific paper ➢ Math ➢ Often, by researchers Library ➢ Implementations of algorithms ➢ Usually, a search function only ➢ By researchers, developers, etc Service (e.g., vector DB) ➢ Library + (handling metadata, serving, scaling, IO, CRUD, etc) ➢ Usually, by companies Product Quantization + Inverted Index (PQ, IVFPQ) [Jégou+, TPAMI 2011] Hierarchical Navigable Small World (HNSW) [Malkov+, TPAMI 2019] Weaviate Vertex AI Matching Engine faiss NMSLIB hnswlib Vald jina ScaNN (4-bit PQ) [Guo+, ICML 2020] ScaNN Often, one library = one algorithm

## Slide 33

### Slide 33 text

Three levels of technology 33 Pinecone Qdrant ScaNN (4-bit PQ) [Guo+, ICML 2020] Algorithm ➢ Scientific paper ➢ Math ➢ Often, by researchers Library ➢ Implementations of algorithms ➢ Usually, a search function only ➢ By researchers, developers, etc Service (e.g., vector DB) ➢ Library + (handling metadata, serving, scaling, IO, CRUD, etc) ➢ Usually, by companies Product Quantization + Inverted Index (PQ, IVFPQ) [Jégou+, TPAMI 2011] Vertex AI Matching Engine NMSLIB Vald ScaNN jina Weaviate Milvus faiss hnswlib Hierarchical Navigable Small World (HNSW) [Malkov+, TPAMI 2019] One service may use some libraries … or re-implement algorithms from scratch (e.g., by Go)

## Slide 34

### Slide 34 text

34 Three levels of technology Milvus Pinecone Qdrant ScaNN (4-bit PQ) [Guo+, ICML 2020] Algorithm ➢ Scientific paper ➢ Math ➢ Often, by researchers Library ➢ Implementations of algorithms ➢ Usually, a search function only ➢ By researchers, developers, etc Service (e.g., vector DB) ➢ Library + (handling metadata, serving, scaling, IO, CRUD, etc) ➢ Usually, by companies Product Quantization + Inverted Index (PQ, IVFPQ) [Jégou+, TPAMI 2011] Hierarchical Navigable Small World (HNSW) [Malkov+, TPAMI 2019] Weaviate Vertex AI Matching Engine faiss NMSLIB hnswlib Vald ScaNN jina This talk mainly focuses algorithms

## Slide 35

### Slide 35 text

35 𝑁 109 106 billion-scale million-scale Locality Sensitive Hashing (LSH) Tree / Space Partitioning Graph traversal 0.34 0.22 0.68 0.71 0 1 0 0 ID: 2 ID: 123 0.34 0.22 0.68 0.71 Space partition Data compression ➢ k-means ➢ PQ/OPQ ➢ Graph traversal ➢ etc… ➢ Raw data ➢ Scalar quantization ➢ PQ/OPQ ➢ etc… Look-up-based Hamming-based Linear-scan by Asymmetric Distance … Linear-scan by Hamming distance Inverted index + data compression For raw data: Acc. ☺, Memory:  For compressed data: Acc. , Memory: ☺

## Slide 36

### Slide 36 text

36 𝑁 109 106 billion-scale million-scale Locality Sensitive Hashing (LSH) Tree / Space Partitioning Graph traversal 0.34 0.22 0.68 0.71 0 1 0 0 ID: 2 ID: 123 0.34 0.22 0.68 0.71 Space partition Data compression ➢ k-means ➢ PQ/OPQ ➢ Graph traversal ➢ etc… ➢ Raw data ➢ Scalar quantization ➢ PQ/OPQ ➢ etc… Look-up-based Hamming-based Linear-scan by Asymmetric Distance … Linear-scan by Hamming distance Inverted index + data compression For raw data: Acc. ☺, Memory:  For compressed data: Acc. , Memory: ☺ Today’s my topic

## Slide 37

### Slide 37 text

37 𝑁 109 106 billion-scale million-scale Locality Sensitive Hashing (LSH) Tree / Space Partitioning Graph traversal 0.34 0.22 0.68 0.71 0 1 0 0 ID: 2 ID: 123 0.34 0.22 0.68 0.71 Space partition Data compression ➢ k-means ➢ PQ/OPQ ➢ Graph traversal ➢ etc… ➢ Raw data ➢ Scalar quantization ➢ PQ/OPQ ➢ etc… Look-up-based Hamming-based Linear-scan by Asymmetric Distance … Linear-scan by Hamming distance Inverted index + data compression For raw data: Acc. ☺, Memory:  For compressed data: Acc. , Memory: ☺ See my previous tutorial at CVPR20 https://speakerdeck.com/matsui_528/cvpr 20-tutorial-billion-scale-approximate- nearest-neighbor-search Today’s my topic

## Slide 38

### Slide 38 text

38 𝑁 109 106 billion-scale million-scale Locality Sensitive Hashing (LSH) Tree / Space Partitioning Graph traversal 0.34 0.22 0.68 0.71 0 1 0 0 ID: 2 ID: 123 0.34 0.22 0.68 0.71 Space partition Data compression ➢ k-means ➢ PQ/OPQ ➢ Graph traversal ➢ etc… ➢ Raw data ➢ Scalar quantization ➢ PQ/OPQ ➢ etc… Look-up-based Hamming-based Linear-scan by Asymmetric Distance … Linear-scan by Hamming distance Inverted index + data compression For raw data: Acc. ☺, Memory:  For compressed data: Acc. , Memory: ☺ See my previous tutorial at CVPR20 https://speakerdeck.com/matsui_528/cvpr 20-tutorial-billion-scale-approximate- nearest-neighbor-search See Martin’s next presentation! Today’s my topic

## Slide 39

### Slide 39 text

39 ➢ Background ➢ Graph-based search ✓ Basic (construction and search) ✓ Observation ✓ Properties ➢ Representative works ✓ HNSW, NSG, NGT, Vamana ➢ Discussion

## Slide 40

### Slide 40 text

40 Graph search ➢ De facto standard if all data can be loaded on memory ➢ Fast and accurate for real-world data ➢ Important for billion-scale situation as well ✓ Graph-search is a building block for billion-scale systems Images are from [Malkov+, Information Systems, 2013] ➢ Traverse graph towards the query ➢ Seems intuitive, but not so much easy to understand ➢ Review the algorithm carefully

## Slide 41

### Slide 41 text

41 Graph search ➢ De facto standard if all data can be loaded on memory ➢ Fast and accurate for real-world data ➢ Important for billion-scale situation as well ✓ Graph-search is a building block for billion-scale systems Images are from [Malkov+, Information Systems, 2013] ➢ Traverse graph towards the query ➢ Seems intuitive, but not so much easy to understand ➢ Review the algorithm carefully The purpose of this tutorial is to make graph search not a black box

## Slide 42

### Slide 42 text

42 Construction Images are from [Malkov+, Information Systems, 2013] and [Subramanya+, NeruIPS 2019] Increment approach Refinement approach ➢ Add a new item to the current graph incrementally ➢ Iteratively refine an initial graph

## Slide 43

### Slide 43 text

43 Construction Images are from [Malkov+, Information Systems, 2013] and [Subramanya+, NeruIPS 2019] Increment approach Refinement approach ➢ Add a new item to the current graph incrementally ➢ Iteratively refine an initial graph

## Slide 44

### Slide 44 text

44 Images are from [Malkov+, Information Systems, 2013] ➢Each node is a database vector 𝒙13 Graph of 𝒙1 , … , 𝒙90 Construction: incremental approach

## Slide 45

### Slide 45 text

45 ➢Each node is a database vector ➢Given a new database vector, 𝒙13 𝒙91 Graph of 𝒙1 , … , 𝒙90 Images are from [Malkov+, Information Systems, 2013] Construction: incremental approach

## Slide 46

### Slide 46 text

46 ➢Each node is a database vector ➢Given a new database vector, create new edges to neighbors 𝒙13 𝒙91 Graph of 𝒙1 , … , 𝒙90 Images are from [Malkov+, Information Systems, 2013] Construction: incremental approach

## Slide 47

### Slide 47 text

47 ➢Each node is a database vector ➢Given a new database vector, create new edges to neighbors 𝒙13 𝒙91 Graph of 𝒙1 , … , 𝒙90 Images are from [Malkov+, Information Systems, 2013] Construction: incremental approach

## Slide 48

### Slide 48 text

48 ➢Each node is a database vector ➢Given a new database vector, create new edges to neighbors 𝒙13 𝒙91 Graph of 𝒙1 , … , 𝒙90 Images are from [Malkov+, Information Systems, 2013] Construction: incremental approach ➢ Prune edges if some node have too many edges ➢ Several strategies (e.g., RNG-pruning)

## Slide 49

### Slide 49 text

49 Construction Images are from [Malkov+, Information Systems, 2013] and [Subramanya+, NeruIPS 2019] Increment approach Refinement approach ➢ Add a new item to the current graph incrementally ➢ Iteratively refine an initial graph

## Slide 50

### Slide 50 text

50 Construction: refinement approach Images are from [Subramanya+, NeruIPS 2019] ➢ Create an initial graph (e.g., random graph or approx. kNN graph) ➢ Refine it iteratively (pruning/adding edges)

## Slide 51

### Slide 51 text

51 Construction: refinement approach Images are from [Subramanya+, NeruIPS 2019] ➢ Create an initial graph (e.g., random graph or approx. kNN graph) ➢ Refine it iteratively (pruning/adding edges) ➢ Need to be moderately sparse (otherwise the graph traverse is slow) ➢ Some “long” edges are required for shortcut

## Slide 52

### Slide 52 text

52 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M ➢ Given a query vector Candidates (size = 3) Close to the query Name each node for explanation

## Slide 53

### Slide 53 text

53 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M ➢ Given a query vector ➢ Start from an entry point (e.g., ) Candidates (size = 3) Close to the query M

## Slide 54

### Slide 54 text

54 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M ➢ Given a query vector ➢ Start from an entry point (e.g., ). Record the distance to q. Candidates (size = 3) Close to the query M M 23.1

## Slide 55

### Slide 55 text

55 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query 23.1 M

## Slide 56

### Slide 56 text

56 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query M 23.1 1st iteration

## Slide 57

### Slide 57 text

57 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M ➢ Pick up the unchecked best candidate ( ) Candidates (size = 3) Close to the query M M 23.1 Best Best

## Slide 58

### Slide 58 text

58 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M ➢ Pick up the unchecked best candidate ( ). Check it. Candidates (size = 3) Close to the query M M 23.1 Best Best check!

## Slide 59

### Slide 59 text

59 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query 23.1 Best ➢ Pick up the unchecked best candidate ( ). Check it. ➢ Find the connected points. Best M M check!

## Slide 60

### Slide 60 text

60 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query 23.1 Best ➢ Pick up the unchecked best candidate ( ). Check it. ➢ Find the connected points. ➢ Record the distances to q. N M M check!

## Slide 61

### Slide 61 text

61 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query Best ➢ Pick up the unchecked best candidate ( ). Check it. ➢ Find the connected points. ➢ Record the distances to q. N M J 11.1 N 15.3 K 19.4 M 23.1 check!

## Slide 62

### Slide 62 text

62 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query Best ➢ Pick up the unchecked best candidate ( ). Check it. ➢ Find the connected points. ➢ Record the distances to q. N M J 11.1 N 15.3 K 19.4 M 23.1

## Slide 63

### Slide 63 text

63 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query Best ➢ Pick up the unchecked best candidate ( ). Check it. ➢ Find the connected points. ➢ Record the distances to q. ➢ Maintain the candidates (size=3) N M J 11.1 N 15.3 K 19.4 M 23.1

## Slide 64

### Slide 64 text

64 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query Best ➢ Pick up the unchecked best candidate ( ). Check it. ➢ Find the connected points. ➢ Record the distances to q. ➢ Maintain the candidates (size=3) N M J 11.1 N 15.3 K 19.4

## Slide 65

### Slide 65 text

65 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N J 11.1 N 15.3 K 19.4

## Slide 66

### Slide 66 text

66 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N J 11.1 N 15.3 K 19.4 2nd iteration

## Slide 67

### Slide 67 text

67 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N J 11.1 N 15.3 K 19.4 ➢ Pick up the unchecked best candidate ( ) J Best Best

## Slide 68

### Slide 68 text

68 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N J 11.1 N 15.3 K 19.4 ➢ Pick up the unchecked best candidate ( ). Check it. J Best Best check!

## Slide 69

### Slide 69 text

69 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N J 11.1 N 15.3 K 19.4 ➢ Pick up the unchecked best candidate ( ). Check it. ➢ Find the connected points. J Best Best check!

## Slide 70

### Slide 70 text

70 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N J 11.1 N 15.3 K 19.4 ➢ Pick up the unchecked best candidate ( ). Check it. ➢ Find the connected points. ➢ Record the distances to q. J Best 13.2 9.7 check! Already visited

## Slide 71

### Slide 71 text

71 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N ➢ Pick up the unchecked best candidate ( ). Check it. ➢ Find the connected points. ➢ Record the distances to q. J Best 13.2 9.7 J 11.1 N 15.3 B 2.3 G 3.5 I 9.7 F 10.2 L 13.2 check! Already visited

## Slide 72

### Slide 72 text

72 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N ➢ Pick up the unchecked best candidate ( ). Check it. ➢ Find the connected points. ➢ Record the distances to q. J Best J 11.1 N 15.3 B 2.3 G 3.5 I 9.7 F 10.2 L 13.2

## Slide 73

### Slide 73 text

73 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N ➢ Pick up the unchecked best candidate ( ). Check it. ➢ Find the connected points. ➢ Record the distances to q. ➢ Maintain the candidates (size=3) J Best J 11.1 N 15.3 B 2.3 G 3.5 I 9.7 F 10.2 L 13.2

## Slide 74

### Slide 74 text

74 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N ➢ Pick up the unchecked best candidate ( ). Check it. ➢ Find the connected points. ➢ Record the distances to q. ➢ Maintain the candidates (size=3) J Best B 2.3 G 3.5 I 9.7

## Slide 75

### Slide 75 text

75 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N B 2.3 G 3.5 I 9.7

## Slide 76

### Slide 76 text

76 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N B 2.3 G 3.5 I 9.7 3rd iteration

## Slide 77

### Slide 77 text

77 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N B 2.3 G 3.5 I 9.7 Best Best ➢ Pick up the unchecked best candidate ( ) B

## Slide 78

### Slide 78 text

78 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N B 2.3 G 3.5 I 9.7 Best Best ➢ Pick up the unchecked best candidate ( ). Check it. B check!

## Slide 79

### Slide 79 text

79 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N B 2.3 G 3.5 I 9.7 Best Best ➢ Pick up the unchecked best candidate ( ). Check it. ➢ Find the connected points. B check!

## Slide 80

### Slide 80 text

80 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N B 2.3 G 3.5 I 9.7 Best ➢ Pick up the unchecked best candidate ( ). Check it. ➢ Find the connected points. ➢ Record the distances to q. B 0.5 2.1 check!

## Slide 81

### Slide 81 text

81 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N Best ➢ Pick up the unchecked best candidate ( ). Check it. ➢ Find the connected points. ➢ Record the distances to q. B 0.5 2.1 C 0.5 D 2.1 A 3.6 B 2.3 G 3.5 I 9.7 check!

## Slide 82

### Slide 82 text

82 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N ➢ Pick up the unchecked best candidate ( ). Check it. ➢ Find the connected points. ➢ Record the distances to q. B C 0.5 D 2.1 A 3.6 B 2.3 G 3.5 I 9.7 Best

## Slide 83

### Slide 83 text

83 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N ➢ Pick up the unchecked best candidate ( ). Check it. ➢ Find the connected points. ➢ Record the distances to q. ➢ Maintain the candidates (size=3) B C 0.5 D 2.1 A 3.6 B 2.3 G 3.5 I 9.7 Best

## Slide 84

### Slide 84 text

84 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N ➢ Pick up the unchecked best candidate ( ). Check it. ➢ Find the connected points. ➢ Record the distances to q. ➢ Maintain the candidates (size=3) B C 0.5 D 2.1 B 2.3 Best

## Slide 85

### Slide 85 text

85 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N C 0.5 D 2.1 B 2.3

## Slide 86

### Slide 86 text

86 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N C 0.5 D 2.1 B 2.3 4th iteration

## Slide 87

### Slide 87 text

87 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N C 0.5 D 2.1 B 2.3 ➢ Pick up the unchecked best candidate ( ). C Best Best

## Slide 88

### Slide 88 text

88 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N C 0.5 D 2.1 B 2.3 ➢ Pick up the unchecked best candidate ( ). Check it. C Best Best check!

## Slide 89

### Slide 89 text

89 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N C 0.5 D 2.1 B 2.3 ➢ Pick up the unchecked best candidate ( ). Check it. ➢ Find the connected points. C Best Best check!

## Slide 90

### Slide 90 text

90 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N C 0.5 D 2.1 B 2.3 ➢ Pick up the unchecked best candidate ( ). Check it. ➢ Find the connected points. ➢ Record the distances to q. C Best check! Already visited Already visited Already visited Already visited

## Slide 91

### Slide 91 text

91 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N C 0.5 D 2.1 B 2.3 ➢ Pick up the unchecked best candidate ( ). Check it. ➢ Find the connected points. ➢ Record the distances to q. ➢ Maintain the candidates (size=3) C Best

## Slide 92

### Slide 92 text

92 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N C 0.5 D 2.1 B 2.3

## Slide 93

### Slide 93 text

93 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N C 0.5 D 2.1 B 2.3 5th iteration

## Slide 94

### Slide 94 text

94 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N C 0.5 D 2.1 B 2.3 ➢ Pick up the unchecked best candidate ( ). D Best Best

## Slide 95

### Slide 95 text

95 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N C 0.5 D 2.1 B 2.3 ➢ Pick up the unchecked best candidate ( ). Check it. D Best Best check!

## Slide 96

### Slide 96 text

96 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N C 0.5 D 2.1 B 2.3 ➢ Pick up the unchecked best candidate ( ). Check it. ➢ Find the connected points. D Best Best check!

## Slide 97

### Slide 97 text

97 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N C 0.5 D 2.1 B 2.3 ➢ Pick up the unchecked best candidate ( ). Check it. ➢ Find the connected points. ➢ Record the distances to q. D Best check! Already visited Already visited

## Slide 98

### Slide 98 text

98 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N C 0.5 D 2.1 B 2.3 ➢ Pick up the unchecked best candidate ( ). Check it. ➢ Find the connected points. ➢ Record the distances to q. D Best check! Already visited Already visited H 3.9

## Slide 99

### Slide 99 text

99 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N C 0.5 D 2.1 B 2.3 ➢ Pick up the unchecked best candidate ( ). Check it. ➢ Find the connected points. ➢ Record the distances to q. D Best H 3.9

## Slide 100

### Slide 100 text

100 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N C 0.5 D 2.1 B 2.3 ➢ Pick up the unchecked best candidate ( ). Check it. ➢ Find the connected points. ➢ Record the distances to q. ➢ Maintain the candidates (size=3) D Best H 3.9

## Slide 101

### Slide 101 text

101 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N C 0.5 D 2.1 B 2.3 ➢ Pick up the unchecked best candidate ( ). Check it. ➢ Find the connected points. ➢ Record the distances to q. ➢ Maintain the candidates (size=3) D Best

## Slide 102

### Slide 102 text

102 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N C 0.5 D 2.1 B 2.3 ➢ All candidates are checked. Finish. ➢ Here, is the closet to the query ( ) C

## Slide 103

### Slide 103 text

103 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N C 0.5 D 2.1 B 2.3 C Final output 1: Candidates ➢ You can pick up topk results ➢ All candidates are checked. Finish. ➢ Here, is the closet to the query ( )

## Slide 104

### Slide 104 text

104 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N C 0.5 D 2.1 B 2.3 C Final output 1: Candidates ➢ You can pick up topk results ➢ All candidates are checked. Finish. ➢ Here, is the closet to the query ( ) Final output 2: Checked items ➢ i.e., search path

## Slide 105

### Slide 105 text

105 Search Images are from [Malkov+, Information Systems, 2013] A B C D E F G H I J K L N M Candidates (size = 3) Close to the query N C 0.5 D 2.1 B 2.3 ➢ All candidates are checked. Finish. ➢ Here, is the closet to the query ( ) C Final output 1: Candidates ➢ You can pick up topk results Final output 2: Checked items ➢ i.e., search path Final output 3: Visit flag ➢ For each item, visited or not

## Slide 106

### Slide 106 text

106 ➢ Background ➢ Graph-based search ✓ Basic (construction and search) ✓ Observation ✓ Properties ➢ Representative works ✓ HNSW, NSG, NGT, Vamana ➢ Discussion

## Slide 107

### Slide 107 text

107 Observation: runtime ➢ Item comparison takes time; 𝑂 𝐷 ➢ The overall runtime ~ #item_comparison ∼ length_of_search_path * average_outdegree 𝒒 ∈ ℝ𝐷 𝒙13 ∈ ℝ𝐷 start query start query start query 1st path 2nd path 3rd path 2.1 1.9 outdegree = 1 outdegree = 2 outdegree = 2 #item_comparison = 3 * (1 + 2 + 2)/3 = 5 2.4

## Slide 108

### Slide 108 text

108 Observation: runtime ➢ Item comparison takes time; 𝑂 𝐷 ➢ The overall runtime ~ #item_comparison ∼ length_of_search_path * average_outdegree 𝒒 ∈ ℝ𝐷 𝒙13 ∈ ℝ𝐷 start query start query start query 1st path 2nd path 3rd path 2.1 1.9 outdegree = 1 outdegree = 2 outdegree = 2 #item_comparison = 3 * (1 + 2 + 2)/3 = 5 2.4 To accelerate the search, (1) How to shorten the search path? ➢ E.g., long edge (shortcut), hierarchical structure (2) How to sparsify the graph? ➢ E.g., deleting redundant edges

## Slide 109

### Slide 109 text

109 A D C B query Observation: candidate size E start Candidates (size = 1) C A D C B query E start Candidates (size = 3) C D E size = 1: Greedy search size > 1: Beam search ➢ Larger candidate size, better but slower results ➢ Online parameter to control the trade-off ➢ Called “ef” in HNSW Fast. But stuck in a local minimum Slow. But find a better solution

## Slide 110

### Slide 110 text

110 Pseudo code ➢ All papers have totally different pseudo code  ➢ Principles are the same. But small details are different. ➢ Hint: Explicitly state the data structure or not NSG [Cong+, VLDB 19] DiskANN [Subramanya+, NeurIPS 19] Learning to route [Baranchuk+, ICML 19]

## Slide 111

### Slide 111 text

111 Pseudo code ➢ All papers have totally different pseudo code  ➢ Principles are the same. But small details are different. ➢ Hint: Explicitly state the data structure or not NSG [Cong+, VLDB 19] DiskANN [Subramanya+, NeurIPS 19] Learning to route [Baranchuk+, ICML 19] Sort the array explicitly Candidates are stored in a set Candidates are stored in a heap; automatically sorted Candidates are stored in an array When need to sort, say “closest L points”

## Slide 112

### Slide 112 text

112 Pseudo code ➢ All papers have totally different pseudo code  ➢ Principles are the same. But small details are different. ➢ Hint: Explicitly state the data structure or not NSG [Cong+, VLDB 19] DiskANN [Subramanya+, NeurIPS 19] Learning to route [Baranchuk+, ICML 19] Just “check” Checked items are stored in a set (“visit” in this code means “check” in our notation)

## Slide 113

### Slide 113 text

113 Pseudo code ➢ All papers have totally different pseudo code  ➢ Principles are the same. But small details are different. ➢ Hint: Explicitly state the data structure or not NSG [Cong+, VLDB 19] DiskANN [Subramanya+, NeurIPS 19] Learning to route [Baranchuk+, ICML 19] Visited item are simply said to be “visited”; implying an additional hidden data structure (array) Visited items are stored in a set

## Slide 114

### Slide 114 text

114 Pseudo code ➢ All papers have totally different pseudo code  ➢ Principles are the same. But small details are different. ➢ Hint: Explicitly state the data structure or not NSG [Cong+, VLDB 19] DiskANN [Subramanya+, NeurIPS 19] Learning to route [Baranchuk+, ICML 19] Termination condition??

## Slide 115

### Slide 115 text

115 Pseudo code ➢ All papers have totally different pseudo code  ➢ Principles are the same. But small details are different. ➢ Hint: Explicitly state the data structure or not NSG [Cong+, VLDB 19] DiskANN [Subramanya+, NeurIPS 19] Learning to route [Baranchuk+, ICML 19] My explanation was based on NSG, but with slight modifications for simplicity: ➢ Candidates are stored in an automatically-sorted array ➢ Termination condition is “all candidates are checked”

## Slide 116

### Slide 116 text

116 Pseudo code ➢ All papers have totally different pseudo code  ➢ Principles are the same. But small parts are very different ➢ Hint: Explicitly state the data structure or not NSG [Cong+, VLDB 19] DiskANN [Subramanya+, NeurIPS 19] Learning to route [Baranchuk+, ICML 19] Formal (?) definition would be helpful for everyone

## Slide 117

### Slide 117 text

117 ➢ Background ➢ Graph-based search ✓ Basic (construction and search) ✓ Observation ✓ Properties ➢ Representative works ✓ HNSW, NSG, NGT, Vamana ➢ Discussion

## Slide 118

### Slide 118 text

118 Base graph Images are from an excellent survey paper [Wang+, VLDB 2021] ➢ Although there are many graph algorithms, there exists four base graphs. ➢ These base graphs are (1) slow to be constructed, and (2) often too dense ➢ Each algorithm often improves one of the base graphs

## Slide 119

### Slide 119 text

119 Base graph Images are from an excellent survey paper [Wang+, VLDB 2021] ➢ Although there are many graph algorithms, there exists four base graphs. ➢ These base graphs are (1) slow to be constructed, and (2) often too dense ➢ Each algorithm often improves one of the base graphs Principal: ➢ Not too dense: Search is slow for dense graph ➢ But moderately dense: Each points should be reachable

## Slide 120

### Slide 120 text

120 Base graph Images are from an excellent survey paper [Wang+, VLDB 2021] ➢ Although there are many graph algorithms, there exists four base graphs. ➢ These base graphs are (1) slow to be constructed, and (2) often too dense ➢ Each algorithm often improves one of the base graphs Famous Delaunay graph ☺ Always reaches the correct answer  Almost fully connected when 𝐷 is large Principal: ➢ Not too dense: Search is slow for dense graph ➢ But moderately dense: Each points should be reachable

## Slide 121

### Slide 121 text

121 Base graph Images are from an excellent survey paper [Wang+, VLDB 2021] ➢ Although there are many graph algorithms, there exists four base graphs. ➢ These base graphs are (1) slow to be constructed, and (2) often too dense ➢ Each algorithm often improves one of the base graphs Relative Neighborhood Graph (RNG) [Toussaint, PR 80] ➢ Consider 𝑥 and 𝑦. There must be no points in the “lune” ➢ Can cut off redundant edges ➢ Not famous in general, but widely used in ANN ➢ Will review again later Principal: ➢ Not too dense: Search is slow for dense graph ➢ But moderately dense: Each points should be reachable

## Slide 122

### Slide 122 text

122 Base graph Images are from an excellent survey paper [Wang+, VLDB 2021] ➢ Although there are many graph algorithms, there exists four base graphs. ➢ These base graphs are (1) slow to be constructed, and (2) often too dense ➢ Each algorithm often improves one of the base graphs K Nearest Neighbor Graph ☺ Can limit the number of neighbor (K at most), enforcing a sparsity  No guaranty for the connectivity Principal: ➢ Not too dense: Search is slow for dense graph ➢ But moderately dense: Each points should be reachable

## Slide 123

### Slide 123 text

123 Base graph Images are from an excellent survey paper [Wang+, VLDB 2021] ➢ Although there are many graph algorithms, there exists four base graphs. ➢ These base graphs are (1) slow to be constructed, and (2) often too dense ➢ Each algorithm often improves one of the base graphs Minimum Spanning Tree (MST) ☺ Ensure the global connectivity. Low degree.  Lack of shortcuts Principal: ➢ Not too dense: Search is slow for dense graph ➢ But moderately dense: Each points should be reachable

## Slide 124

### Slide 124 text

124 Graph search algorithms Images are from an excellent survey paper [Wang+, VLDB 2021] ➢ Lots of algorithms ➢ The basic structure is same: (1) designing a good graph + (2) beam search

## Slide 125

### Slide 125 text

125 The initial seed matters Start here? Start here? v.s. ➢ Starting from a good seed ➡ Shorter path ➡ Faster search ➢ Finding a good seed is also an ANN problem ➢ Solve a small ANN problem by tree [NST; Iwasaki+, arXiv 18], hash [Effana; Fu+, arXiv 16] or LSH [LGTM; Arai+, DEXA 21]

## Slide 126

### Slide 126 text

126 Edge selection: RNG-pruning A When inserting A, where to edge? A All neighbors? ➢ Too many edges ➢ Slow for search A Top-K? ➢ Not reachable ➢ Low accuracy. A   ☺ Probably connected So we don’t need this RNG-pruning: Moderate number of edges

## Slide 127

### Slide 127 text

127 C B D A Given A, make edges to B, C, D, and E? ? ? ? E Edge selection: RNG-pruning

## Slide 128

### Slide 128 text

128 C B D A E Edge selection: RNG-pruning

## Slide 129

### Slide 129 text

129 B D A Find the nearest one to A C E Edge selection: RNG-pruning

## Slide 130

### Slide 130 text

130 C B D A ➢ For all neighbors of A, compare and ➢ If is the shortest, make an edge Find the nearest one to A E Edge selection: RNG-pruning

## Slide 131

### Slide 131 text

131 C B D A ➢ For all neighbors of A, compare and ➢ If is the shortest, make an edge Find the nearest one to A This time, there are no neighbors. So let’s make an edge E Edge selection: RNG-pruning

## Slide 132

### Slide 132 text

132 C B D A done E Edge selection: RNG-pruning

## Slide 133

### Slide 133 text

133 C B D A Find the 2nd nearest one to A done E Edge selection: RNG-pruning

## Slide 134

### Slide 134 text

134 C B D A Find the 2nd nearest one to A ➢ For all neighbors of A, compare and ➢ If is the shortest, make an edge done E Edge selection: RNG-pruning

## Slide 135

### Slide 135 text

Edge selection: RNG-pruning 135 C B D A Find the 2nd nearest one to A ➢ For all neighbors of A, compare and ➢ If is the shortest, make an edge done Shortest! Not make an edge E

## Slide 136

### Slide 136 text

136 C B D A done done E Edge selection: RNG-pruning

## Slide 137

### Slide 137 text

Edge selection: RNG-pruning 137 C B D A done done Find the 3rd nearest one to A E

## Slide 138

### Slide 138 text

Edge selection: RNG-pruning 138 C B D A done done Find the 3rd nearest one to A ➢ For all neighbors of A, compare and ➢ If is the shortest, make an edge E

## Slide 139

### Slide 139 text

Edge selection: RNG-pruning 139 C B D A done done Find the 3rd nearest one to A ➢ For all neighbors of A, compare and ➢ If is the shortest, make an edge Shortest! Make an edge E

## Slide 140

### Slide 140 text

140 C B D A done done done E Edge selection: RNG-pruning

## Slide 141

### Slide 141 text

Edge selection: RNG-pruning 141 C B D A done done done E Find the 4th nearest one to A

## Slide 142

### Slide 142 text

Edge selection: RNG-pruning 142 C B D A done done done E Find the 4th nearest one to A ➢ For all neighbors of A, compare and ➢ If is the shortest, make an edge

## Slide 143

### Slide 143 text

Edge selection: RNG-pruning 143 C B D A done done done E Find the 4th nearest one to A ➢ For all neighbors of A, compare and ➢ If is the shortest, make an edge Shortest! Not make an edge

## Slide 144

### Slide 144 text

144 C B D A done done done E done Edge selection: RNG-pruning

## Slide 145

### Slide 145 text

145 C B D A done done done E done ➢ RNG-pruning is an effective edge-pruning technique, and used in several algorithms Pros: Implementation is easy Cons: Require many distance computations Edge selection: RNG-pruning

## Slide 146

### Slide 146 text

146 ➢ Background ➢ Graph-based search ✓ Basic (construction and search) ✓ Observation ✓ Properties ➢ Representative works ✓ HNSW, NSG, NGT, Vamana ➢ Discussion

## Slide 147

### Slide 147 text

147 Hierarchical Navigable Small World; HNSW [Malkov and Yashunin, TPAMI, 2019] ➢ Construct the graph hierarchically [Malkov and Yashunin, TPAMI, 2019] ➢ Fix #edge per node by RNG-pruning ➢ The most famous algorithm; works very well in real world Search on a coarse graph Move to the same node on a finer graph Repeat

## Slide 148

### Slide 148 text

148 ➢ Used in various services ✓ milvus, weaviate, qdrant, vearch, elasticsearch, OpenSearch, vespa, redis, Lucene… ➢ Three famous implementations ✓ NMSLIB (the original implementation) ✓ hnswlib (light-weight implementation from NMSLIB) ✓ Faiss (re-implemented version by the faiss team) Hierarchical Navigable Small World; HNSW [NMSLIB] https://github.com/nmslib/nmslib [hnswlib] https://github.com/nmslib/hnswlib [Faiss] https://github.com/facebookresearch/faiss/blob/main/faiss/IndexHNSW.h

## Slide 149

### Slide 149 text

149 https://www.facebook.com/groups/faissusers/posts/917143142043306/?comment_id=917533385337615&reply_comment_id=920542105036743 Any implementation difference between NMSLIB, hnswlib, and faiss-hnsw? My view on the implementation differences (I might forgot something): 1) nmslib’s HNSW requires internal index conversion step (from nmslib’s format to an internal one) to have good performance, and after the conversion the index cannot be updated with new elements. nmslib also has a simple "graph diversification" postprocessing after building the index (controlled by the "post" parameter) and sophisticated queue optimizations which makes it a bit faster compared to other implementations. Another advantage of nmslib is out-of-the box support for large collection of distance functions, including some exotic distances. 2) hnswlib is a header-only C++ library reimplementation of nmslib's hnsw. It does not have the index conversion step, thus - the Pros (compared to nmslib): much more memory efficient and faster at build time. It also supports index insertions, element updates (with incremental graph rewiring - added recently) and fake deletions (mark elements as deleted to avoid returning them during the graph traversal). Cons (compared to nnmslib): It is a tad slower than nmslib due to lack of graph postprocessing and queue optimization; out-of-the box version supports only 3 distance functions, compared to many distance functions in nmslib. Overall, I've tried to keep hnswlib as close as possible to a distributed index (hence no index postprocessing). 3) Faiss hnsw is a different reimplementation. It has its own algorithmic features, like having the first elements in the upper layers on the structure (opposed to random in other implementations). It is a bit more memory efficient compared to hnswlib with raw vectors and optimized for batch processing. Due to the latter it is noticeably slower at single query processing (opposed to nmslib or hnswlib) and generally a bit slower for batch queries (the last time I’ve tested, but there were exceptions). The implementation also supports incremental insertions (also preferably batched), quantized data and two-level encoding, which makes it much less memory hungry and the overall best when memory is a big concern. Yury Malkov (the author of HNSW paper) Discussion from Faiss User Forum in FB Note that this discussion was in 2020 and the libraries have been updated a lot since then

## Slide 150

### Slide 150 text

150 ➢ See the following excellent blog posts for more details https://www.pinecone.io/learn/hnsw/ James Briggs, PINECONE, Faiss: The Missing Manual, 6. Hierarchical Navigable Small Worlds (HNSW) Hierarchical Navigable Small World; HNSW https://zilliz.com/blog/hierarchical- navigable-small-worlds-HNSW Frank Liu, zilliz, Vector Database 101, Hierarchical Navigable Small Worlds (HNSW) https://towardsdatascience.com /ivfpq-hnsw-for-billion-scale- similarity-search-89ff2f89d90e Peggy Chang, IVFPQ + HNSW for Billion-scale Similarity Search

## Slide 151

### Slide 151 text

151 Navigating Spreading-out Graph (NSG) ➢ Monotonic RNG ➢ In some cases, slightly better than HNSW ➢ Used in Alibaba’s Taobao RNG Monotonic RNG ➢ Recall the def. of RNG is “no point in a lune” ➢ The path “p -> q” is ling Monotonic RNG can make more edges [Fu+, VLDB 19] Images are from [Fu+, VLDB 19]

## Slide 152

### Slide 152 text

152 Navigating Spreading-out Graph (NSG) ➢ The original implementation: ➢ Implemented in faiss as well ➢ If you’re using faiss-hnsw and need a little bit more performance with the same interface, worth trying NSG https://github.com/ZJULearning/nsg IndexHNSWFlat(int d, int M, MetricType metric) IndexNSGFlat(int d, int R, MetricType metric) [Fu+, VLDB 19]

## Slide 153

### Slide 153 text

153 Neighborhood Graph and Tree (NGT) ➢ Make use of range search for construction ➢ Obtain a seed via VP-tree ➢ Current best methods in ann-benchmarks are NGT-based algorithms ➢ Quantization is natively available ➢ Repository: ➢ From Yahoo Japan ➢ Used in Vald [Iwasaki+, arXiv 18] Image are from the original repository https://github.com/yahoojapan/NGT

## Slide 154

### Slide 154 text

154 DiskANN (Vamana) ➢ Vamana: Graph-based search algorithm ➢ DiskANN: Disk-friendly search system using Vamana ➢ From MSR India ➢ Good option for huge data (not the main focus of this talk, though) ➢ The same team is actively developing interesting functionalites ✓ Data update: FreshDiskANN [Singh+, arXiv 21] ✓ Filter: Filtered-DiskANN [Gollapudi+, WWW 23] [Subramanya+, NeurIPS 19] https://github.com/microsoft/DiskANN

## Slide 155

### Slide 155 text

155 ➢ Background ➢ Graph-based search ✓ Basic (construction and search) ✓ Observation ✓ Properties ➢ Representative works ✓ HNSW, NSG, NGT, Vamana ➢ Discussion

## Slide 156

### Slide 156 text

156 Just NN? Vector DB? ➢ Vector DB companies say “Vector DB is cool” ➢ My own idea: ➢ Which vector DB? ➡ No conclusions! ➢ If you need a clean & well designed API, I recommend taking a look at docarray in Jina AI (see Han’s talk today!) ✓ https://weaviate.io/blog/vector-library-vs-vector-database ✓ https://codelabs.milvus.io/vector-database-101-what-is-a-vector-database/index#2 ✓ https://zilliz.com/learn/what-is-vector-database Try the simplest numpy–only search Slow? Try fast algorithm such as HNSW in faiss Try Vector DB If speed is the only concern, just use libraries

## Slide 157

### Slide 157 text

157 Useful resources ➢ Several companies have very useful blog series ➢ Pinecone Blog ✓ https://www.pinecone.io/learn/ ➢ Weaviate Blog ✓ https://weaviate.io/blog ➢ Jina AI Blog ✓ https://jina.ai/news/ ➢ Zilliz Blog ✓ https://zilliz.com/blog ➢ Romain Beaumont Blog ✓ https://rom1504.medium.com/

## Slide 158

### Slide 158 text

158 Progress in the last three years ➢ Three years have passed since my previous tutorial at CVPR 2020 ➢ What progress in the last three years in the ANN field? Y. Matasui, “Billion-scale Approximate Nearest Neighbor Search”, CVPR 2020 Tutorial ➢ Slide: https://speakerdeck.com/matsui_528/cvpr20-tutorial-billion-scale- approximate-nearest-neighbor-search ➢ Video: https://youtu.be/SKrHs03i08Q

## Slide 159

### Slide 159 text

159 Progress in the last three years ➢ Three years have passed since my previous tutorial at CVPR 2020 ➢ What progress in the last three years in the ANN field? Y. Matasui, “Billion-scale Approximate Nearest Neighbor Search”, CVPR 2020 Tutorial ➢ Slide: https://speakerdeck.com/matsui_528/cvpr20-tutorial-billion-scale- approximate-nearest-neighbor-search ➢ Video: https://youtu.be/SKrHs03i08Q ➢ The basic framework is still same (HNSW and IVFPQ!) ➢ HNSW is still de facto standard; although several papers claim they perform better ➢ Disk-based systems are getting attention ➢ Vector DB has gained rapid popularity for LLM applications. ➢ Because of LLM, we should suppose D as ~1000 (not ~100) ➢ GPU-ANN is powerful, but less widespread than I expected; CPUs are more convenient for LLM ➢ Competitions (SISAP and bigann-benchmarks) ➢ New billion-scale datasets ➢ A breakthrough algorithm that goes beyond graph-based methods awaits.

## Slide 160

### Slide 160 text

160 ➢ Background ➢ Graph-based search ✓ Basic (construction and search) ✓ Observation ✓ Properties ➢ Representative works ✓ HNSW, NSG, NGT, Vamana ➢ Discussion

## Slide 161

### Slide 161 text

161 ◼ [Jégou+, TPAMI 2011] H. Jégou+, “Product Quantization for Nearest Neighbor Search”, IEEE TPAMI 2011 ◼ [Guo+, ICML 2020] R. Guo+, “Accelerating Large-Scale Inference with Anisotropic Vector Quantization”, ICML 2020 ◼ [Malkov+, TPAMI 2019] Y. Malkov+, “Efficient and Robust Approximate Nearest Neighbor search using Hierarchical Navigable Small World Graphs,” IEEE TPAMI 2019 ◼ [Malkov+, IS 13] Y, Malkov+, “Approximate Nearest Neighbor Algorithm based on Navigable Small World Graphs”, Information Systems 2013 ◼ [Fu+, VLDB 19] C. Fu+, “Fast Approximate Nearest Neighbor Search With The Navigating Spreading-out Graphs”, 2019 ◼ [Subramanya+, NeurIPS 19] S. J. Subramanya+, “DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node”, NeurIPS 2019 ◼ [Baranchuk+, ICML 19] D. Baranchuk+, “Learning to Route in Similarity Graphs” ◼ [Wang+, VLDB 21] M. Wang+, “A Comprehensive Survey and Experimental Comparison of Graph-Based Approximate Nearest Neighbor Search”, VLDB 2021 ◼ [Toussaint, PR 80] G. T. Toussaint, “The Relative Neighbouhood Graph of A Finite Planar Set”, Pattern Recognition 1980 ◼ [Fu+, arXiv 16] C. Fu and D. Cai, “Efanna: An Extremely Fast Approximate Nearest Neighbor Search Algorithm based on knn Graph”, arXiv 2016 ◼ [Arai+, DEXA 21] Y. Arai+, “LGTM: A Fast and Accurate kNN Search Algorithm in High-Dimensional Spaces”, DEXA 2021 ◼ [Iwasaki+, arXiv 18] M. Iwasaki and D. Miyazaki, “Optimization if Indexing Based on k-Nearest Neighbor Graph for Proximity Search in High-dimensional Data”, arXiv 2018 ◼ [Singh+, arXiv 21] A. Singh+, “FreshDiskANN: A Fast and Accurate Graph-Based ANN Index for Streaming Similarity Search”, arXiv 2021 ◼ [Gollapudi+, WWW 23] S. Gollapudi+, “Filtered-DiskANN: Graph Algorithms for Approximate Nearest Neighbor Search with Filters”, WWW 2023 Reference

## Slide 162

### Slide 162 text

162 Reference ◼ [Pinecone] https://www.pinecone.io/ ◼ [Milvus] https://milvus.io/ ◼ [Qdrant] https://qdrant.tech/ ◼ [Weaviate] https://weaviate.io/ ◼ [Vertex AI Matching Engine] https://cloud.google.com/vertex-ai/docs/matching-engine ◼ [Vald] https://vald.vdaas.org/ ◼ [Vearch] https://vearch.github.io/ ◼ [Elasticsearch] https://www.elastic.co/jp/blog/introducing-approximate-nearest-neighbor-search-in-elasticsearch-8-0 ◼ [OpenSearch] https://opensearch.org/docs/latest/search-plugins/knn/approximate-knn/ ◼ [Vespa] https://vespa.ai/ ◼ [Redis] https://redis.com/solutions/use-cases/vector-database/ ◼ [Lucene] https://lucene.apache.org/core/9_1_0/core/org/apache/lucene/util/hnsw/HnswGraphSearcher.html ◼ [SISAP] SISAP 2023 Indexing Challenge https://sisap-challenges.github.io/ ◼ [Bigann-benchmarks] Billion-Scale Approximate Nearest Neighbor Search Challenge: NeurIPS'21 competition track https://big-ann-benchmarks.com/

## Slide 163

### Slide 163 text

163 Thank you! Time Session Presenter 13:30 – 13:40 Opening Yusuke Matsui 13:40 – 14:30 Theory and Applications of Graph-based Search Yusuke Matsui 14:30 – 15:20 A Survey on Approximate Nearest Neighbors in a Billion-Scale Settings Martin Aumüller 15:20 – 15:30 Break 15:30 – 16:20 Query Language for Neural Search in Practical Applications Han Xiao Acknowledgements ➢ I would like to express my deep gratitude to Prof. Daichi Amagata, Naoki Ono, and Tomohiro Kanaumi for reviewing the contents of this tutorial and providing valuable feedback. ➢ This work was supported by JST AIP Acceleration Research JPMJCR23U2, Japan.