ETS versus Elasticsearch for Queryable Caching

If your data sets aren't sufficiently large enough while building your OTP applications, consider reaching for built-in functionality of ETS.

Transcript

1. QUERYABLE CACHING ETS VERSUS ELASTICSEARCH @schainks
2. HOW DO YOU SEARCH? Without “The Google”!
3. HELLO WORLD! ➕
7. FILTERING AUTO COMPLETE FULL TEXT SEARCH BACKGROUND
8. ELASTICSEARCH ⬆ Performant BACKGROUND
9. ELASTICSEARCH ⬆ Performant ⬆ “Queryable Cache BACKGROUND CACHE
10. ELASTICSEARCH Operational Costs BACKGROUND CACHE
11. ELASTICSEARCH OPERATIONAL COSTS CACHE Yet Another DSL Integration Time Money Version Management Configuration Gotchas
12. ELASTICSEARCH OPERATIONAL COSTS CACHE Yet Another DSL Integration Time Money Version Management Configuration Gotchas
13. ELASTICSEARCH OPERATIONAL COSTS CACHE Yet Another DSL Integration Time Money Version Management Configuration Gotchas
14. ELASTICSEARCH OPERATIONAL COSTS CACHE Yet Another DSL Integration Time Money Version Management Configuration Gotchas
15. ELASTICSEARCH OPERATIONAL COSTS CACHE Yet Another DSL Integration Time Money Version Management Configuration Gotchas
17. ERLANG TERM STORAGE
18. ETS FEATURES
19. ETS FEATURES Built into OTP (:ets)
20. ETS FEATURES Built into OTP (:ets) K/V table containing tuples
21. ETS FEATURES Built into OTP (:ets) K/V table containing tuples No garbage collection
22. ETS FEATURES Built into OTP (:ets) K/V table containing tuples No garbage collection Concurrently accessible data (“serialized”)
23. USING ETS USE CASES Presets Cached data Message buffer Atomic Counters
24. USING ITS USE CASES Cached data
25. USING ETS REQUIREMENTS Where does our data come from? How much data do we have? What’s our R/W frequency? How often does this data change?
26. USING ETS REQUIREMENTS Transactional DB How much data do we have? What’s our R/W frequency? How often does this data change?
27. USING ETS REQUIREMENTS Transactional DB Fits in RAM What’s our R/W frequency? How often does this data change?
28. USING ETS REQUIREMENTS Transactional DB Fits in RAM Read heavy How often does this data change?
29. USING ETS REQUIREMENTS Transactional DB Fits in RAM Read heavy Infrequent
30. QUERYABLE CACHING WITH ETS
32. LET’S GET CLONING FILTERING AUTO COMPLETE FULL TEXT SEARCH
33. %Fonts.Font{ name: “Futura”, classification: “sans-serif”, description: “Futura is a geometric sans-serif typeface designed by Paul Renner and released in 1927.” }
34. INDEX AND LOOKUP > table = :ets.new(:fonts_lookup, [:set, :public]) > :ets.insert(table, {f.name, f.classification, f.description}) > :ets_insert(table, {"acumin", “sans-serif", "Balanced and rational quality."}) > :ets.lookup(table, "futura") [{"futura", "sans-serif"}]
35. match/2, match_object/1 > :ets.match(:fonts_lookup, {:"$1", "sans-serif"}) [["acumin"], [“futura"]] > :ets.match_object(table, {:"$1", "sans-serif"}) [{"futura", "sans-serif"}] MATCHING OBJECTS IN AN ETS TABLE
36. fun2ms/1 > fun = :ets.fun2ms( fn {n, c, d} when c == "sans-serif" -> n end) [{ {:”$1", :"$2"}, [{:==, :"$2", "sans-serif"}], [:”$1”] }] > :ets.select(table, fun) ["futura"] ETS MATCH SPECIFICATION
37. fun2ms/1 > fun = :ets.fun2ms( fn {n, c, d} when c == "sans-serif" -> n end) [{ {:”$1", :"$2"}, [{:==, :"$2", "sans-serif"}], [:”$1”] }] > :ets.select(table, fun) ["futura"] ETS MATCH SPECIFICATION
38. fun2ms/1 > fun = :ets.fun2ms( fn {n, c, d} when c == "sans-serif" -> n end) [{ {:”$1", :"$2"}, [{:==, :"$2", "sans-serif"}], [:”$1”] }] > :ets.select(table, fun) ["futura"] ETS MATCH SPECIFICATION
39. fun2ms/1 > fun = :ets.fun2ms( fn {n, c, d} when c == "sans-serif" -> n end) [{ {:”$1", :"$2"}, [{:==, :"$2", "sans-serif"}], [:”$1”] }] > :ets.select(table, fun) ["futura"] ETS MATCH SPECIFICATION
40. FILTERING ETSO: AN ETS ADAPTER FOR ECTO Use Ecto schemas with ETS tables Handles ETS table supervision Understands match specifications
41. fun2ms/1 > fun = :ets.fun2ms( fn {n, c, d} when c == "sans-serif" -> n end) [{ {:”$1", :"$2"}, [{:==, :"$2", "sans-serif"}], [:”$1”] }] > :ets.select(table, fun) ["futura"] ETS MATCH SPECIFICATION
42. GITHUB.COM/EVADNE/ETSO Model.Font |> where([x], x.classification == “sans-serif”) |> select([x], x.name) |> Repo.all()
43. LET’S GET CLONING FILTERING AUTO COMPLETE FULL TEXT SEARCH
44. AUTOCOMPLETE iex(19)> :ets.fun2ms(fn({key, "serif"} = el) -> key end) [{{:"$1", "serif"}, [], [:"$1"]}] iex(20)> :ets.select(table, fun) ["futura", "adobe garamond"]
45. LET’S GET CLONING FILTERING AUTO COMPLETE FULL TEXT SEARCH
47. FULL TEXT SEARCH elib1 Lucene Redisearch Riak
48. REAL WORLD CONCERNS Performance Ranking / Search engine features International support The Business
49. OPERATIONAL CONCERNS NOT COVERED High availability architecture Synchronization Index changes
50. BUT WAIT, THERE’S MORE
51. persistent_term Much Faster than ETS for fetching whole terms Garbage collect on update / delete No filtering Good for NIF references
52. THANK YOU! Q&A
53. THANK YOU! Q&A NOW HIRING
54. REFERENCES 1. Armstrong, Joe. Elib1 - https://github.com/joearms/elib1 2. Bekker, Ward. A Basic Full Text Search Server in Erlang - http://blog.equanimity.nl/blog/2011/10/07/a-basic- full-text-search-server-in-erlang/ 3. Herbert, Fred. Learn You Some Erlang For Great Good. ETS - https://learnyousomeerlang.com/ets 4. Nelson, Jay. Avoiding Single Process Bottlenecks By Using Ets Concurrency. Erlang Factory, 6 Mar. 2014. http:// www.erlang-factory.com/static/upload/media/1394716488140115jaynelson.pdf, https://www.youtube.com/ watch?v=XrkY9WRY8p0 5. Pena, Maria. Beam Extreme: Don't Do This At Home - https://www.youtube.com/watch?v=_ePnlegZqgI 6. Implementing Autocomplete in Elixir Using ETS? https://elixirforum.com/t/implementing-autocomplete-in-elixir- using-ets/8651/2