annoy4s

Transcript

1. annoy4s @pishen
2. spotify/annoy Approximate Nearest Neighbors Oh Yeah
3. Linked from spotify/annoy
4. pishen/annoy4s version 0.1.0
5. Annoy C++ library Python client Go client Lua client Scala client spotify/annoy annoy4s
6. Pros • Fully supports all the functionality of Annoy (indexing/querying, Euclidean/Angular). • Didn't rewrite the code, utilized the optimized C++ code provided by Annoy. • Easy parallelized by Scala. queries.par.map(q => annoy.query(q)) • JVM with C++ native code is fast and type-safe. • Annoy itself is fast (10x faster than lsh4s).
7. Cons • Platform dependent. Need to compile the C++ code (for now) if you're not using linux-x86-64. > compileNative > publish • May not be as simple as lsh4s when broadcasting the index onto each worker in Spark. • My C++ skill is poor, as well as my JN* knowledge.
8. How to use it? https://github.com/pishen/annoy4s
9. How does it call the C++ library? https://github.com/java-native-access/jna
10. JNA in one page libraryDependencies += "net.java.dev.jna" % "jna" % "4.2.2" src/main/cpp/annoyjava.cpp src/main/scala/annoy4s/AnnoyLibrary.scala src/main/resources/linux-x86-64/libannoy.so functions mapping compile call
11. annoyjava.cpp AnnoyLibrary.scala
12. AnnoyIndexInterface<int, float> *createEuclidean(int f) { return new AnnoyIndex<int, float, Euclidean, Kiss64Random>(f); } val annoy: Pointer = lib.createEuclidean(64) memory address space JVM -Xmx2G AnnoyIndex annoy
13. void getNnsByItem(AnnoyIndexInterface<int, float> *ptr, int item, int n, int search_k, int *result, float *distances){ vector<int> resultV; vector<float> distancesV; ptr->get_nns_by_item(item, n, search_k, &resultV, &distancesV); std::copy(resultV.begin(), resultV.end(), result); std::copy(distancesV.begin(), distancesV.end(), distances); } val result = Array.fill(10)(-1) val distances = Array.fill(10)(-1.0f) lib.getNnsByItem(annoy, item, 10, -1, result, distances) under GC's control free automatically
14. Browse the code to see more details: https://github.com/pishen/annoy4s