Bloom Filters: A Look Into Ruby

Transcript

1. B L O O M F I LT E R S or: that one time I was hella bored
2. Bloom Filters Or: How I Learned To Stop Procrastinating And Benchmark The Code
3. THE A MASTERPIECE OF MODERN HORROR FiLTERiNG
4. 2016: a space-efficient odyssey An epic drama of boredom and exploration
5. B L O O M F I LT E R S or: that one time I was hella bored
6. “a bloom filter is a space-efficient probabilistic data structure, conceived by Burton Howard Bloom in 1970 (…) a query returns either "possibly in set" or "definitely not in set"” - Wikipedia, 2016
7. bloom filter
8. bloom filter do you have the element 3?
9. bloom filter yeah, probably
10. bloom filter do you have the element 4?
11. bloom filter I most certainly do not
12. bloom filter I most certainly do not “Why do people even like this thing?”
13. add ‘subvisual’
14. hash(‘subvisual’)
15. add ‘rubyconf’
16. hash(‘rubyconf’)
17. test ‘subvisual’
18. hash(‘subvisual’) all are 1?
19. test ‘subvisual’ true
20. test ‘office’
21. all are 1? hash(‘office’)
22. test ‘office’ false
23. test ‘mirrorconf’
24. hash(‘mirrorconf’) all are 1?
25. test ‘mirrorconf’ true
26. test and add play with hash functions get to say smart stuff like “so I wrote this bloom filter”
27. diving into it with Ruby
28. module DumbFilter end
29. module DumbFilter class Array def initialize @data = [] end end end
30. module DumbFilter class Array def add(str) @data << str end end end
31. module DumbFilter class Array def test(str) @data.include? str end end end
32. you don’t play with hash functions sequential access space wastefulness
33. module DumbFilter class Hash def initialize @data = {} end end end
34. module DumbFilter class Hash def add(str) @data[str] = true end end end
35. module DumbFilter class Hash def test(str) @data[str] end end end
36. you kinda play with hash functions instant access
37. “a bloom filter is a space-efficient probabilistic data structure, conceived by Burton Howard Bloom in 1970 (…) a query returns either "possibly in set" or "definitely not in set"” - Wikipedia, 2016
38. /peterc/bitarray
39. def initialize(size: 1024) @bits = BitArray.new(size) @fnv = FNV.new @size = size end
40. def add(str) @bits[i(str)] = 1 end def i(str) @fnv.fnv1a_64(str) % @size end
41. def test(str) @bits[i(str)] == 1 end
42. you do play with hash functions instant access space-efficient small universe == more collisions
43. def initialize(size: 1024, iterations: 3) @bits = BitArray.new(size) @size = size @seeds = seed(iterations) end
44. def initialize(size: 1024, iterations: 3) @bits = BitArray.new(size) @size = size @seeds = seed(iterations) end
45. def initialize(size: 1024, iterations: 3) @bits = BitArray.new(size) @size = size @seeds = seed(iterations) end
46. def seed(nr) (1..nr).each_with_object([]) do |n, s| s << SecureRandom.hex(3).to_i(16) end end
47. def hash(str, seed) MurmurHash3::V32.str_hash(str, seed) end
48. def i(str) @seeds.map { |s| hash(str, s) % @size } end
49. def add(str) set i(str) end def set(indexes) indexes.each { |i| @bits[i] = 1 } end
50. def test(str) get i(str) end def get(indexes) indexes.all? { |i| @bits[i] == 1 } end
51. demo (yes, yet another goddamned Rails blog app)
54. test-drive
55. 5 million random inserts probabilistic universe of 10 million 5 million random accesses /igrigorik/bloomfilter-rb
56. fnv is really slow ruby string hashing is optimized bloomfilter-rb uses C extensions
57. Collision counting ruby’s hash is not probabilistic nor space-efficient “what about bf_v2’s poor result?”
58. you do play with hash functions instant access space-efficient small universe == more collisions
59. Collision counting: 1024 bits & 300 entries m(bits)/n(entries) * ln(2) optimal number of hash functions:
60. in the field
61. Article tailoring - Quora & Medium Type-ahead queries — Facebook I/O Filter — Apache HBase Malicious URL Check — bit.ly Checking node communications in IoT sensors
62. B L O O M F I LT E R S or: that one time I was hella bored