Locality Sensitive Hashing at Lyst

Transcript

1. Speeding up search with locality sensitive hashing. by Maciej Kula

2. Hi, I’m Maciej Kula. @maciej_kula

3. We collect the world of
 fashion into a customisable
 shopping

   experience.

4. Given a point, find other points close to it. Nearest

   neighbour search… 4
5. None

6. At Lyst we use it for… 1.) Image Search 2.)

   Recommendations 6

7. Convert image to points in space (vectors) & use nearest

   neighbour search to get similar images. 1. Image Search (-0.3, 2.1, 0.5)

8. Super useful for deduplication & search.

9. Convert products and users to points in space & use

   nearest neighbour search to get related products for the user. 2. Recommendations user = (-0.3, 2.1, 0.5) product = (5.2, 0.3, -0.5)

10. Great, but…

11. 11 80 million We have images

12. 12 9 million We have products

13. Exhaustive nearest neighbour search is too slow.

14. Locality sensitive hashing to the rescue! Use a hash table.

    Pick a hash function that puts similar points in the same bucket. Only search within the bucket.

15. We use Random Projection Forests

16. Partition by splitting on random vectors

17. Partition by splitting on random vectors

18. Partition by splitting on random vectors

19. Partition by splitting on random vectors

20. Partition by splitting on random vectors

21. Points to note Keep splitting until the nodes are small

    enough. Median splits give nicely balanced trees. Build a forest of trees.

22. Why do we need a forest? Some partitions split the

    true neighbourhood of a point. Because partitions are random, other trees will not repeat the error. Build more trees to trade off query speed for precision.

23. LSH in Python annoy, Python wrapper for C++ code. LSHForest,

    part of scikit-learn FLANN, an auto-tuning ANN index

24. But… LSHForest is slow. FLANN is a pain to deploy.

    annoy is great, but can’t add points to an existing index.

25. So we wrote our own.

26. github.com/lyst/rpforest pip install rpforest

27. rpforest Quite fast. Allows adding new items to the index.

    Does not require us to store points in memory.

28. We use it in conjunction with PostgreSQL Send the query

    point to the ANN index. Get ANN row ids back Plug them into postgres for filtering Final scoring done in postgres using C extensions.

29. Side note: postgres is awesome. Arrays & custom functions in

    C

30. Gives us a fast and reliable ANN service 100x speed-up

    with 0.6 10-NN precision Allows us to serve real-time results All on top of a real database.

31. thank you @maciej_kula