Using Heroku to build Heroku Postgres

Introduces how Heroku works and how the primitives around processes, dynos and apps are used to build a complex system consisting of a constellation of applications collaborating with each other to run the Heroku Postgres platform.

Presented on November 21st 2013 at Dreamforce in SF.

Transcript

1. Using Heroku
   to build Heroku Postgres Harold Giménez @hgmnz Dreamforce 2013 +
2. What is Heroku?
3. We run your code in the cloud
4. app logic with code, not clicks
5. Use the tools you know
6. Use the tools you love
7. Use the tools that make you happy
8. Use the tools that make you productive
9. Use the tools that make you productive
10. your app is the center of the universe
11. Creating demo-app... done, region is us http://demo-app.herokuapp.com/ | git@heroku.com:demo- app.git Git remote heroku added $ $ heroku apps:create demo-app ----> Heroku receiving push ----> Rails app detected ----> Compiled slug size is 8.0MB http://demo-app.herokuapp.com deployed to Heroku $ git push heroku master
12. Tell us how to run your code
13. web: bundle exec thin start -p $PORT -e $RACK_ENV worker: bundle exec rake worker --trace clock: bundle exec bin/clock $ $ cat Procfile foreman start 18:08:22 web.1 | started with pid 3371 18:08:22 worker.1 | started with pid 3372 18:08:22 clock.1 | started with pid 3373
14. And scale it right up!
15. Scaling web dynos... done, now running 10 Scaling worker dynos... done, now running 5 $ $ heroku ps:scale web=10 worker=5 -a demo-app
16. Attach some add-ons addons.heroku.com
17. Adding loggly on demo-app… done, v161 (free) Use `heroku addons:docs loggly` to view documentation. $ $ heroku addons:add loggly
18. Heroku Postgres
20. Heroku Origins
21. focus on Rails
22. Rails apps need a database
23. web apps need a database
24. thankfully postgres was chosen
25. otherwise I wouldn’t be here
26. “let’s build a production grade database service”
28. Heroku Postgres iron.io loggly you Heroku Add-ons
29. Adding heroku-postgresql on demo-app… done, v129 (free) Attached as HEROKU_POSTGRESQL_ORANGE_URL Database has been created and is available Use `heroku addons:docs heroku-postgresql` to view documentation. $ $ heroku addons:add heroku-postgresql --app demo-app Plan: Dev Status: available Connections: 1 PG Version: 9.2.4 Data Size: 6.3 MB Tables: 0 Rows: 0/10000 (In compliance) Fork/Follow: Unsupported heroku pg:info -a demo-app
30. Heroku Postgres v0 A sinatra app implementing the heroku addons API create servers install postgres service create database for user - a “Resource” Sequel talks to postgres stem talks to AWS
31. the simplest thing that could possibly work, but no less
32. two main entities
33. Resource { database: ‘d3lwi9ef2’, port: 5432, username: ‘u23f8doife9’, password: ‘dfwefujp’, created_at: ‘2012-05-02’, state: ‘available’ }
34. Server { ip: ‘192.168.0.1’ instance_id: ‘i-2fidj3c8’, ami: ‘pg-prod’, availability_zone: ‘us-east-1a’ created_at: ‘2012-05-02’, state: ‘booting’ }
35. and a thin web interface erb templates 
 served by sinatra app
36. we’ve come a long way since then
37. Workflow and Monitoring
38. draw inspiration from game programming
39. observe environment finite state machine
40. available creating uncertain unavailable deprovisioning deprovisioned
41. class Resource def feel observations.create( Feeler.new(self).current_environment ) end end ! class Feeler def current_environment { services_available?: service_available?, connections: connections, row_count: row_count, table_count: table_count, seq_scans: seq_scans, index_scans: index_scans } end end
42. class Resource include Stateful ! state :available do unless service_available? transition :uncertain end end end ! resource = Resource.find resource.transition :available resource.feel resource.tick puts resource.state # => :uncertain
43. module Stateful def self.included(base) base.extend ClassMethods end ! module ClassMethods def state(name, &block) state[name] = block end def states; @states ||={}; end end ! def tick self.instance_eval( &self.class.states[self.state.to_sym] ) end def transition(state); end end
44. resource.feel resource.tick Need to do this all the time
45. db1 db2 db3 db4 db5 db6 db7 db8 dbn W orkers db1.feel
 db1.tick
46. db4 db3 db2 db1 dbn Enqueue
47. Durability and Availability
48. INSERT INTO … 1. Write to WAL 2. Keep it in memory 4. Flush to disk 3. Respond to client
49. Ship WAL at least every 60s multi-datacenter storage
50. fork
51. follower
52. need a more flexible object model
53. timeline T0
54. participant
55. participant
56. participant
57. participant
58. participant
59. resource
60. followers
61. fork
62. disaster
63. disaster
64. recovery STONITH
65. big project
66. lots of moving parts long test suite
67. modularize and build APIs
68. move away from the large monolithic application
69. and into a constellation of apps talking to each other
70. Dashboard Dataclips Resources Heroku Admin UI Participants
 Timelines
 Servers CLI infra Monitoring
71. gain in agility
72. composable services
73. independently scalable
74. independently testable
75. current tooling
76. Rails Sinatra sequel fog ruby python bash go coffeescript backbone.js Dashboard Dataclips Resources Heroku Admin UI Participants
 Timelines
 Servers CLI infra Monitoring
77. Lessons managing databases is hard start simple extract and share code separate concerns into services
78. Thanks! @hgmnz @herokupostgres +