Heroku Postgres

Talk delivered at Burlington Ruby Conf, 07/28/2012

Transcript

1. Heroku Postgres The Tale of Conceiving and Building a Leading Cloud Database Service 1 Harold Giménez @hgimenez 1 Saturday, July 28, 12
2. 2 2 Saturday, July 28, 12
3. Heroku origins 3 3 Saturday, July 28, 12
4. focus on rails 4 4 Saturday, July 28, 12
5. rails apps need a database 5 5 Saturday, July 28, 12
6. web apps need a database 6 6 Saturday, July 28, 12
7. thankfully postgres was chosen 7 7 Saturday, July 28, 12
8. otherwise I wouldn’t be here 8 8 Saturday, July 28, 12
9. “let’s make a production grade postgres service” 9 9 Saturday, July 28, 12
10. 10 10 Saturday, July 28, 12
11. (hopefully yours) 11 11 Saturday, July 28, 12
12. 12 Heroku Postgres v.0.pre.alpha • A sinatra app implementing the heroku addons API • create servers • install postgres service • create databases for users - a “Resource” • Sequel talks to postgres • stem talks to AWS 12 Saturday, July 28, 12
13. Two main entities 13 13 Saturday, July 28, 12
14. Resource 14 { database: ‘d4f9wdf02’, port: 5432, username: ‘uf0wjasdf’, password: ‘pf14fhjas’, created_at: ‘2012-05-02’, state: ‘available’ } 14 Saturday, July 28, 12
15. Server 15 { elastic_ip: ‘192.168.0.1’, instance_id: ‘i-2efjoiads’, ami: ‘pg-prod’, availability_zone: ‘us-east-1’, created_at: ‘2012-05-02’, state: ‘booting’ } 15 Saturday, July 28, 12
16. ...and a thin admin web interface 16 erb templates in sinatra endpoint 16 Saturday, July 28, 12
17. We are just an add-on 17 17 Saturday, July 28, 12
18. 18 18 Saturday, July 28, 12
19. 19 we run on 19 Saturday, July 28, 12
20. the simplest thing that could possibly work, but no less 20 20 Saturday, July 28, 12
21. We’ve come a long way since then 21 21 Saturday, July 28, 12
22. Workflow and Monitoring 22 22 Saturday, July 28, 12
23. draw inspiration from gaming 23 23 Saturday, July 28, 12
24. 24 class Resource def feel observations.create( Feeler.new(self).current_environment ) end end class Feeler def current_environment { service_available?: service_available?, open_connections: open_connections, row_count: row_count, table_count: table_count, seq_scans: seq_scans, index_scans: index_scans } end end monitoring 24 Saturday, July 28, 12
25. 25 class Resource include Stateful state :available do unless service_available? transition :unavailable end end end resource = Resource.new resource.transition :available resource.feel resource.tick puts resource.state # ‘unavailable’ workflow 25 Saturday, July 28, 12
26. 26 module Stateful def self.included(base) base.extend ClassMethods end module ClassMethods def state(name, &block) states[name] = block end def states; @states ||= {}; end end def tick self.instance_eval( &self.class.states[self.state.to_sym] ) end def transition(state) # log and assign new state end end workflow 26 Saturday, July 28, 12
27. 27 27 Saturday, July 28, 12
28. 28 resource.feel resource.tick Need to do this all the time 28 Saturday, July 28, 12
29. 29 db1 db2 db3 db4 db5 db6 db7 db8 db9 ... dbn db1.feel db1.tick 29 Saturday, July 28, 12
30. 30 db2 db3 db4 db5 db6 db7 db8 db9 ... dbn db1 db2.feel db2.tick enqueue(db1) 30 Saturday, July 28, 12
31. 31 QueueClassic http://github.com/ryandotsmith/queue_classic 31 Saturday, July 28, 12
32. Durability & Availability 32 32 Saturday, July 28, 12
33. 33 33 Saturday, July 28, 12
34. 34 34 Saturday, July 28, 12
35. 35 35 Saturday, July 28, 12
36. 36 36 Saturday, July 28, 12
37. Continuous Protection 37 • Write-Ahead Log segments shipped to durable storage every 60 seconds • We can replay these logs on a new server to recover your data • https://github.com/heroku/WAL-E 37 Saturday, July 28, 12
38. Need a more flexible object model 38 38 Saturday, July 28, 12
39. 39 timeline 39 Saturday, July 28, 12
40. 40 participant 40 Saturday, July 28, 12
41. 41 41 Saturday, July 28, 12
42. 42 42 Saturday, July 28, 12
43. 43 resource 43 Saturday, July 28, 12
44. 44 follower 44 Saturday, July 28, 12
45. 45 fork 45 Saturday, July 28, 12
46. 46 disaster 46 Saturday, July 28, 12
47. 47 47 Saturday, July 28, 12
48. 48 recovery 48 Saturday, July 28, 12
49. big project 49 49 Saturday, July 28, 12
50. lots of moving parts 50 50 Saturday, July 28, 12
51. long test suite 51 51 Saturday, July 28, 12
52. modularize and build APIs 52 52 Saturday, July 28, 12
53. 53 53 Saturday, July 28, 12
54. gain in agility 54 54 Saturday, July 28, 12
55. composable services 55 55 Saturday, July 28, 12
56. independently scalable 56 56 Saturday, July 28, 12
57. Logging and Metrics 57 57 Saturday, July 28, 12
58. log generation 58 58 Saturday, July 28, 12
59. 59 59 Saturday, July 28, 12
60. logs are event streams 60 60 Saturday, July 28, 12
61. how should you log? 61 61 Saturday, July 28, 12
62. 62 post “/work” do puts “starting to do work” worker = Worker.new(params) begin worker.lift_things_up worker.put_them_down rescue WorkerError => e puts “Fail :( #{e.message}” status 500 end puts “done doing work” status 200 end 62 Saturday, July 28, 12
63. 63 $ heroku logs --tail 2012-07-28T02:43:35 [web.4] starting to do work 2012-07-28T02:43:35 [web.4] Fail :( invalid worker, nothing to do 2012-07-28T02:43:35 heroku[router] POST myapp.com/work dyno=web.4 queue=0 wait=0ms service=14ms status=500 bytes=643 63 Saturday, July 28, 12
64. 64 bad logging good logging 64 Saturday, July 28, 12
65. 65 post “/work” do log(create_work: true, request_id: uuid) do worker = Worker.new(params.merge(uuid: uuid)) begin worker.lift_things_up worker.put_them_down rescue WorkerError => e log_exception(e, create_work: true) end end end helpers do def uuid SecureRandom.uuid end end 65 Saturday, July 28, 12
66. 66 require ‘scrolls’ module App module Logs extend self def log(data, &block) Scrolls.log(with_env(data), &block) end def log_exception(exception, data, &block) Scrolls.log_exception(with_env(data), &block) end def with_env(hash) { environment: ENV[‘RACK_ENV’] }.merge(data) end end end 66 Saturday, July 28, 12
67. 67 $ heroku logs --tail 2012-07-28T02:43:35 [web.4] create_work request_id=afe2-f0d at=start 2012-07-28T02:43:35 [web.4] create_work request_id=afe2-f0d at=exception message=invalid worker, nothing to do 2012-07-28T02:43:35 [web.4] create_work request_id=afe2-f0d at=finish elapsed=53 2012-07-28T02:43:35 heroku[router] POST myapp.com/work dyno=web.4 queue=0 wait=0ms service=14ms status=500 bytes=643 67 Saturday, July 28, 12
68. 68 log consumption 68 Saturday, July 28, 12
69. (this is the fun part) 69 69 Saturday, July 28, 12
70. 70 70 Saturday, July 28, 12
71. 71 select * from events; 71 Saturday, July 28, 12
72. 72 72 Saturday, July 28, 12
73. 73 good logging metrics alerts 73 Saturday, July 28, 12
74. current tooling 74 • still using sequel and sinatra • fog displaced stem • backbone.js for web UIs • fernet for auth tokens, valcro for simple validations, QueueClassic for job queues • Wal-e for durability • python, go and bash in some subsystems 74 Saturday, July 28, 12
75. • managing databases is hard • start simple • extract (and share) reusable code • separate concerns into services • use the right tool (framework, library, language) • learn to love your event stream, metrics for everything 75 lessons 75 Saturday, July 28, 12
76. TRUNCATE TABLE talk; 76 @hgimenez @herokupostgres Thank you! 76 Saturday, July 28, 12
77. select a.body from answers a inner join questions q on a.id = q.answer_id; 77 77 Saturday, July 28, 12