In the Back of Your Mind

Some simple examples of how to pipeline data through beanstalkd or 0MQ, using Dominic memory encoding as an example.

Transcript

1. IN THE BACK OF YOUR MIND beanstalkd and 0MQ through the Dominic System
3. THE DOMINIC SYSTEM
4. LEARN HOW TO IMPROVE YOUR MIND A book I am currently reading
5. MEMORY TECHNIQUES
6. MEMORY TECHNIQUES • The book includes various techniques for improving memory
7. MEMORY TECHNIQUES • The book includes various techniques for improving memory • It helps to translate abstract data (the brain is weak with) into vivid images (a strength of the brain)
8. MEMORY TECHNIQUES • The book includes various techniques for improving memory • It helps to translate abstract data (the brain is weak with) into vivid images (a strength of the brain) • The Dominic System is used to translate numbers into characters and actions • It can help you memorize large numbers
9. SINGLE DIGITS
10. SINGLE DIGITS • Just count through the alphabet • But replace F with S • And skip to N when you hit 9
11. SINGLE DIGITS • Just count through the alphabet • But replace F with S • And skip to N when you hit 9 1 2 3 4 5 6 7 8 9 0 A B C D E S G H N O
12. DOUBLE DIGITS
13. DOUBLE DIGITS • Convert digit pairs into letters • 15 = AE, 23 = BC, 27 = BG, 39 = CN, 80 = HO, 80 = HA
14. DOUBLE DIGITS • Convert digit pairs into letters • 15 = AE, 23 = BC, 27 = BG, 39 = CN, 80 = HO, 80 = HA • Use the letters as initials for a character • AE = Albert Einstein, BG = Bill Gates, CN = Chuck Norris
15. DOUBLE DIGITS • Convert digit pairs into letters • 15 = AE, 23 = BC, 27 = BG, 39 = CN, 80 = HO, 80 = HA • Use the letters as initials for a character • AE = Albert Einstein, BG = Bill Gates, CN = Chuck Norris • Or use letters to indicate characters in other ways • BC = Jesus Christ, HO = Santa Claus, HA = Julia Sweeney
16. IMAGERY IS KEY
17. IMAGERY IS KEY • Each digit pair should create an image in your mind
18. IMAGERY IS KEY • Each digit pair should create an image in your mind • The image need to be a character doing their signature action • Chuck Norris performing a roundhouse kick • Santa Claus delivering presents • Julia Sweeney dressing asexually
19. FOUR DIGIT NUMBERS
20. FOUR DIGIT NUMBERS • To handle four digit numbers, use the first two for the character and the second two as the action • 2339 = BCCN = Jesus Christ performing a roundhouse kick • 8081 = HOHA = Santa Claus dressing asexually
21. FOUR DIGIT NUMBERS • To handle four digit numbers, use the first two for the character and the second two as the action • 2339 = BCCN = Jesus Christ performing a roundhouse kick • 8081 = HOHA = Santa Claus dressing asexually • You’re up to a 10,000 number memory at this point!
22. LARGER NUMBERS
23. LARGER NUMBERS • “Chunk” up bigger numbers into two and four digit groups
24. LARGER NUMBERS • “Chunk” up bigger numbers into two and four digit groups • Build wacky stories using those characters and actions • 27808039 = 2780 8039 = BGHO HOCN = Bill Gates started delivering presents, but this angered Santa Claus who gave him a roundhouse kick.
25. A CAST OF CHARACTERS
26. A CAST OF CHARACTERS http://ron.ludism.org/mnemonics_public.txt … 40 = DO = Fred Flintstone, breaking rocks in the quarry (Yabba dabba DOo!) 41 = DA = Douglas Adams, putting a Babelfish into his ear (_The Hitch Hiker's Guide to the Galaxy_) 42 = DB = PERSONAL 43 = DC = George Washington, chopping down cherry tree (Washington DC) 44 = DD = Danny Dunn (fictional character), descending in a bathysphere 45 = DE = PERSONAL 46 = DS = Ivan Stang (a.k.a. Doug Smith), ranting on a soapbox 47 = DG = PERSONAL 48 = DH = Douglas Hofstadter, recursively holding a picture of himself 49 = DN = Dan (_Roseanne_ TV show), wearing sunglasses to switch on Christmas lights …
27. A CAST OF CHARACTERS http://ron.ludism.org/mnemonics_public.txt … 40 = DO = Fred Flintstone, breaking rocks in the quarry (Yabba dabba DOo!) 41 = DA = Douglas Adams, putting a Babelfish into his ear (_The Hitch Hiker's Guide to the Galaxy_) 42 = DB = PERSONAL 43 = DC = George Washington, chopping down cherry tree (Washington DC) 44 = DD = Danny Dunn (fictional character), descending in a bathysphere 45 = DE = PERSONAL 46 = DS = Ivan Stang (a.k.a. Doug Smith), ranting on a soapbox 47 = DG = PERSONAL 48 = DH = Douglas Hofstadter, recursively holding a picture of himself 49 = DN = Dan (_Roseanne_ TV show), wearing sunglasses to switch on Christmas lights …
29. BEANSTALKD (VIA STALKER)
30. BEANSTALKD
31. BEANSTALKD • A simple but highly effective message queue
32. BEANSTALKD • A simple but highly effective message queue • I’ve used it many times, including to do moves in Go vs Go
33. BEANSTALKD • A simple but highly effective message queue • I’ve used it many times, including to do moves in Go vs Go • It’s definitely one of my favorite tools for background jobs
34. CHARACTER LOOKUPS
35. CHARACTER LOOKUPS require "open-uri" def lookup_table return @lookup_table if defined? @lookup_table @lookup_table = { } open("http://ron.ludism.org/mnemonics_public.txt") do |page| page.each do |line| if line =~ /\A(\d{2})\s*=\s*[A-Z]{2}\s*=\s*(.+)/ @lookup_table[$1] = $2 elsif line =~ /\A\s+(.+)/ @lookup_table.values.last << " #{$1}" end end end @lookup_table end
36. CHARACTER LOOKUPS require "open-uri" def lookup_table return @lookup_table if defined? @lookup_table @lookup_table = { } open("http://ron.ludism.org/mnemonics_public.txt") do |page| page.each do |line| if line =~ /\A(\d{2})\s*=\s*[A-Z]{2}\s*=\s*(.+)/ @lookup_table[$1] = $2 elsif line =~ /\A\s+(.+)/ @lookup_table.values.last << " #{$1}" end end end @lookup_table end
37. CHARACTER LOOKUPS require "open-uri" def lookup_table return @lookup_table if defined? @lookup_table @lookup_table = { } open("http://ron.ludism.org/mnemonics_public.txt") do |page| page.each do |line| if line =~ /\A(\d{2})\s*=\s*[A-Z]{2}\s*=\s*(.+)/ @lookup_table[$1] = $2 elsif line =~ /\A\s+(.+)/ @lookup_table.values.last << " #{$1}" end end end @lookup_table end
38. ENCODING CHUNKS
39. ENCODING CHUNKS def encode(n, as = :both) if n.size == 4 and as == :both "#{encode(n[0..1], :character)} #{encode(n[2..3], :action)}." elsif as == :both "#{encode(n, :character)} #{encode(n, :action)}." elsif as == :character lookup_table[n][/\A\s*([^,]+(?:\([^)]+\))?)/, 1] elsif as == :action lookup_table[n][/\A\s*[^,]+(?:\([^)]+\))?,\s*(.+?)\s*\z/, 1] end end
40. ENCODING CHUNKS def encode(n, as = :both) if n.size == 4 and as == :both "#{encode(n[0..1], :character)} #{encode(n[2..3], :action)}." elsif as == :both "#{encode(n, :character)} #{encode(n, :action)}." elsif as == :character lookup_table[n][/\A\s*([^,]+(?:\([^)]+\))?)/, 1] elsif as == :action lookup_table[n][/\A\s*[^,]+(?:\([^)]+\))?,\s*(.+?)\s*\z/, 1] end end
41. ENCODING CHUNKS def encode(n, as = :both) if n.size == 4 and as == :both "#{encode(n[0..1], :character)} #{encode(n[2..3], :action)}." elsif as == :both "#{encode(n, :character)} #{encode(n, :action)}." elsif as == :character lookup_table[n][/\A\s*([^,]+(?:\([^)]+\))?)/, 1] elsif as == :action lookup_table[n][/\A\s*[^,]+(?:\([^)]+\))?,\s*(.+?)\s*\z/, 1] end end
42. ENCODING NUMBERS
43. ENCODING NUMBERS job "number.encode" do |args| number = args["number"].to_s.delete("^0-9") number = "0#{number}" if (number.size % 2).nonzero? chunks = [4] * (number.size / 4) + [2] * (number.size % 4 / 2) encodings = [ ] chunks.permutation.to_a.uniq.each do |grouping| encodings << number.match(/\A#{grouping.map { |g| "(.{#{g}})" }.join}\z/) .captures .map { |n| encode(n) } .join(" ") .tr("_", '"') end Stalker.enqueue( "number.display", number: args["number"], encodings: encodings ) end
44. ENCODING NUMBERS job "number.encode" do |args| number = args["number"].to_s.delete("^0-9") number = "0#{number}" if (number.size % 2).nonzero? chunks = [4] * (number.size / 4) + [2] * (number.size % 4 / 2) encodings = [ ] chunks.permutation.to_a.uniq.each do |grouping| encodings << number.match(/\A#{grouping.map { |g| "(.{#{g}})" }.join}\z/) .captures .map { |n| encode(n) } .join(" ") .tr("_", '"') end Stalker.enqueue( "number.display", number: args["number"], encodings: encodings ) end
45. ENCODING NUMBERS job "number.encode" do |args| number = args["number"].to_s.delete("^0-9") number = "0#{number}" if (number.size % 2).nonzero? chunks = [4] * (number.size / 4) + [2] * (number.size % 4 / 2) encodings = [ ] chunks.permutation.to_a.uniq.each do |grouping| encodings << number.match(/\A#{grouping.map { |g| "(.{#{g}})" }.join}\z/) .captures .map { |n| encode(n) } .join(" ") .tr("_", '"') end Stalker.enqueue( "number.display", number: args["number"], encodings: encodings ) end
46. ENCODING NUMBERS job "number.encode" do |args| number = args["number"].to_s.delete("^0-9") number = "0#{number}" if (number.size % 2).nonzero? chunks = [4] * (number.size / 4) + [2] * (number.size % 4 / 2) encodings = [ ] chunks.permutation.to_a.uniq.each do |grouping| encodings << number.match(/\A#{grouping.map { |g| "(.{#{g}})" }.join}\z/) .captures .map { |n| encode(n) } .join(" ") .tr("_", '"') end Stalker.enqueue( "number.display", number: args["number"], encodings: encodings ) end
47. DISPLAYING ENCODINGS
48. DISPLAYING ENCODINGS job "number.display" do |args| Array(args["encodings"]).each do |encoded| puts "#{args['number']}:" puts encoded puts end end
49. QUEUING NUMBERS
50. QUEUING NUMBERS abort "USAGE: #{$PROGRAM_NAME} NUMBER" if ARGV.empty? require "stalker" Stalker.enqueue("number.encode", number: ARGV.shift)
51. RUNNING JOBS
52. RUNNING JOBS $ beanstalkd
53. RUNNING JOBS $ beanstalkd $ ruby encode_number.rb 405-285-0536
54. RUNNING JOBS $ beanstalkd $ ruby encode_number.rb 405-285-0536 $ rvm 1.9.2 $ RUBYLIB=. stalk jobs.rb number.encode [2010-12-08 13:07:24 -0600] Working 1 jobs: [ number.encode ] [2010-12-08 13:11:18 -0600] -> number.encode (number=405-285-0536) [2010-12-08 13:11:19 -0600] -> number.encode finished in 510ms
55. RUNNING JOBS $ beanstalkd $ ruby encode_number.rb 405-285-0536 $ rvm 1.9.2 $ RUBYLIB=. stalk jobs.rb number.encode [2010-12-08 13:07:24 -0600] Working 1 jobs: [ number.encode ] [2010-12-08 13:11:18 -0600] -> number.encode (number=405-285-0536) [2010-12-08 13:11:19 -0600] -> number.encode finished in 510ms
56. RUNNING JOBS $ beanstalkd $ ruby encode_number.rb 405-285-0536 $ rvm 1.9.2 $ RUBYLIB=. stalk jobs.rb number.encode [2010-12-08 13:07:24 -0600] Working 1 jobs: [ number.encode ] [2010-12-08 13:11:18 -0600] -> number.encode (number=405-285-0536) [2010-12-08 13:11:19 -0600] -> number.encode finished in 510ms
57. RUNNING JOBS $ beanstalkd $ ruby encode_number.rb 405-285-0536 $ rvm 1.9.2 $ RUBYLIB=. stalk jobs.rb number.encode [2010-12-08 13:07:24 -0600] Working 1 jobs: [ number.encode ] [2010-12-08 13:11:18 -0600] -> number.encode (number=405-285-0536) [2010-12-08 13:11:19 -0600] -> number.encode finished in 510ms
58. RUNNING JOBS $ beanstalkd $ ruby encode_number.rb 405-285-0536 $ rvm 1.9.2 $ RUBYLIB=. stalk jobs.rb number.encode [2010-12-08 13:07:24 -0600] Working 1 jobs: [ number.encode ] [2010-12-08 13:11:18 -0600] -> number.encode (number=405-285-0536) [2010-12-08 13:11:19 -0600] -> number.encode finished in 510ms
59. RUNNING JOBS $ beanstalkd $ ruby encode_number.rb 405-285-0536 $ rvm 1.9.2 $ RUBYLIB=. stalk jobs.rb number.encode [2010-12-08 13:07:24 -0600] Working 1 jobs: [ number.encode ] [2010-12-08 13:11:18 -0600] -> number.encode (number=405-285-0536) [2010-12-08 13:11:19 -0600] -> number.encode finished in 510ms
60. RUNNING JOBS $ beanstalkd $ ruby encode_number.rb 405-285-0536 $ rvm 1.9.2 $ RUBYLIB=. stalk jobs.rb number.encode [2010-12-08 13:07:24 -0600] Working 1 jobs: [ number.encode ] [2010-12-08 13:11:18 -0600] -> number.encode (number=405-285-0536) [2010-12-08 13:11:19 -0600] -> number.encode finished in 510ms $ RUBYLIB=. stalk jobs.rb number.display … 405-285-0536: Fred Flintstone breaking rocks in the quarry (Yabba dabba DOo!). Stuart Little (tiny fictional mouse wearing street clothes) releasing helium balloons (Newtonmas, HE HE HE!). J.R.R. Tolkien stepping into the Wardrobe. …
61. FOR MORE INFORMATION See Railscasts #243
62. FOR MORE INFORMATION See Railscasts #243
63. BEANSTALKD PROS
64. BEANSTALKD PROS • Extremely lightweight (especially if you drop the Rails stack)
65. BEANSTALKD PROS • Extremely lightweight (especially if you drop the Rails stack) • Ridiculously fast
66. BEANSTALKD PROS • Extremely lightweight (especially if you drop the Rails stack) • Ridiculously fast • Designed to be a queue
67. BEANSTALKD PROS • Extremely lightweight (especially if you drop the Rails stack) • Ridiculously fast • Designed to be a queue • “Tubes” make it trivial to divide work or pipeline jobs
68. BEANSTALKD PROS • Extremely lightweight (especially if you drop the Rails stack) • Ridiculously fast • Designed to be a queue • “Tubes” make it trivial to divide work or pipeline jobs • Really easy to spread across ruby versions and environments
69. BEANSTALKD CONS
70. BEANSTALKD CONS • Not ideal for durable queuing needs (though it does have a binlog)
71. BEANSTALKD CONS • Not ideal for durable queuing needs (though it does have a binlog) • Missing niceties like logging (though Stalker adds logging above beanstalkd)
72. BEANSTALKD CONS • Not ideal for durable queuing needs (though it does have a binlog) • Missing niceties like logging (though Stalker adds logging above beanstalkd) • Needs job introspection
74. 0MQ (AKA ZEROMQ OR ZMQ)
75. 0MQ
76. 0MQ • A high-level socket API over an asynchronous message queue • Request and reply, publish subscribe, and pipeline models
77. 0MQ • A high-level socket API over an asynchronous message queue • Request and reply, publish subscribe, and pipeline models • This was my first time using it (I was curious about it) • I don’t think it’s well suited to this task
78. THE END OF THE PIPELINE
79. THE END OF THE PIPELINE require "zmq" zmq = ZMQ::Context.new up = zmq.socket(ZMQ::UPSTREAM) up.bind("tcp://127.0.0.1:5000") while message = up.recv puts message end
80. THE END OF THE PIPELINE require "zmq" zmq = ZMQ::Context.new up = zmq.socket(ZMQ::UPSTREAM) up.bind("tcp://127.0.0.1:5000") while message = up.recv puts message end
81. THE END OF THE PIPELINE require "zmq" zmq = ZMQ::Context.new up = zmq.socket(ZMQ::UPSTREAM) up.bind("tcp://127.0.0.1:5000") while message = up.recv puts message end
82. THE END OF THE PIPELINE require "zmq" zmq = ZMQ::Context.new up = zmq.socket(ZMQ::UPSTREAM) up.bind("tcp://127.0.0.1:5000") while message = up.recv puts message end
83. NUMBER LOOKUPS
84. NUMBER LOOKUPS # same lookup_table() as before… def lookup lookup_table.each do |number, character_and_action| return number if yield character_and_action end nil end def character_number(msg) lookup { |ca| msg.start_with? ca[/\A\s*([^,]+(?:\([^)]+\))?)/, 1] } end def action_number(msg) lookup { |ca| msg.end_with? ca[/\A\s*[^,]+(?:\([^)]+\))?,\s*(.+?)\s*\z/, 1] } end
85. NUMBER LOOKUPS # same lookup_table() as before… def lookup lookup_table.each do |number, character_and_action| return number if yield character_and_action end nil end def character_number(msg) lookup { |ca| msg.start_with? ca[/\A\s*([^,]+(?:\([^)]+\))?)/, 1] } end def action_number(msg) lookup { |ca| msg.end_with? ca[/\A\s*[^,]+(?:\([^)]+\))?,\s*(.+?)\s*\z/, 1] } end
86. A DECODER
87. A DECODER def decode(message) message.to_s.strip.split(/\.(?:\s{2}|\n|\z)/).map { |chunk| chunk.gsub!(/\s+/, " ") character = character_number(chunk) action = action_number(chunk) character == action ? character : "#{character}#{action}" }.join end
88. THE MIDDLE PIPE
89. THE MIDDLE PIPE require "zmq" zmq = ZMQ::Context.new up = zmq.socket(ZMQ::UPSTREAM) down = zmq.socket(ZMQ::DOWNSTREAM) up.bind("tcp://127.0.0.1:5001") down.connect("tcp://127.0.0.1:5000") while message = up.recv down.send(decode(message)) end
90. THE MIDDLE PIPE require "zmq" zmq = ZMQ::Context.new up = zmq.socket(ZMQ::UPSTREAM) down = zmq.socket(ZMQ::DOWNSTREAM) up.bind("tcp://127.0.0.1:5001") down.connect("tcp://127.0.0.1:5000") while message = up.recv down.send(decode(message)) end
91. THE MIDDLE PIPE require "zmq" zmq = ZMQ::Context.new up = zmq.socket(ZMQ::UPSTREAM) down = zmq.socket(ZMQ::DOWNSTREAM) up.bind("tcp://127.0.0.1:5001") down.connect("tcp://127.0.0.1:5000") while message = up.recv down.send(decode(message)) end
92. THE MIDDLE PIPE require "zmq" zmq = ZMQ::Context.new up = zmq.socket(ZMQ::UPSTREAM) down = zmq.socket(ZMQ::DOWNSTREAM) up.bind("tcp://127.0.0.1:5001") down.connect("tcp://127.0.0.1:5000") while message = up.recv down.send(decode(message)) end
93. THE PIPELINE OPENING
94. THE PIPELINE OPENING require "zmq" zmq = ZMQ::Context.new down = zmq.socket(ZMQ::DOWNSTREAM) down.connect("tcp://127.0.0.1:5001") down.send(ARGF.read)
95. A DATA PIPELINE IN ACTION
96. A DATA PIPELINE IN ACTION $ ruby queue_message.rb Fred Flintstone breaking rocks in the quarry (Yabba dabba DOo!). Stuart Little (tiny fictional mouse wearing street clothes) releasing helium balloons (Newtonmas, HE HE HE!). J.R.R. Tolkien stepping into the Wardrobe. ^d
97. A DATA PIPELINE IN ACTION $ ruby queue_message.rb Fred Flintstone breaking rocks in the quarry (Yabba dabba DOo!). Stuart Little (tiny fictional mouse wearing street clothes) releasing helium balloons (Newtonmas, HE HE HE!). J.R.R. Tolkien stepping into the Wardrobe. ^d $ ruby decoder.rb
98. A DATA PIPELINE IN ACTION $ ruby queue_message.rb Fred Flintstone breaking rocks in the quarry (Yabba dabba DOo!). Stuart Little (tiny fictional mouse wearing street clothes) releasing helium balloons (Newtonmas, HE HE HE!). J.R.R. Tolkien stepping into the Wardrobe. ^d $ ruby decoder.rb $ ruby decoder_display.rb 4052850536
99. 0MQ PROS
100. 0MQ PROS • Asynchronous messaging primitives for constructing networks
101. 0MQ PROS • Asynchronous messaging primitives for constructing networks • Extremely versatile (can be made to fit most any model)
102. 0MQ PROS • Asynchronous messaging primitives for constructing networks • Extremely versatile (can be made to fit most any model) • Supports many to many connections
103. 0MQ PROS • Asynchronous messaging primitives for constructing networks • Extremely versatile (can be made to fit most any model) • Supports many to many connections • Automatically load balanced
104. 0MQ CONS
105. 0MQ CONS • Very low-level (really just a platform to build on) • Missing all the conveniences: logging, message formats, etc.
106. 0MQ CONS • Very low-level (really just a platform to build on) • Missing all the conveniences: logging, message formats, etc. • Best for asynchronous networks you design and build
108. THANKS!