Using Heroku to build Heroku Postgres

Slide 1

Slide 1 text

Using Heroku to build Heroku Postgres Harold Giménez @hgmnz Dreamforce 2013 +

Slide 2

Slide 2 text

What is Heroku?

Slide 3

Slide 3 text

We run your code in the cloud

Slide 4

Slide 4 text

app logic with code, not clicks

Slide 5

Slide 5 text

Use the tools you know

Slide 6

Slide 6 text

Use the tools you love

Slide 7

Slide 7 text

Use the tools that make you happy

Slide 8

Slide 8 text

Use the tools that make you productive

Slide 9

Slide 9 text

Use the tools that make you productive

Slide 10

Slide 10 text

your app is the center of the universe

Slide 11

Slide 11 text

Creating demo-app... done, region is us http://demo-app.herokuapp.com/ | [email protected]:demo- app.git Git remote heroku added $ $ heroku apps:create demo-app ----> Heroku receiving push ----> Rails app detected ----> Compiled slug size is 8.0MB http://demo-app.herokuapp.com deployed to Heroku $ git push heroku master

Slide 12

Slide 12 text

Tell us how to run your code

Slide 13

Slide 13 text

web: bundle exec thin start -p $PORT -e $RACK_ENV worker: bundle exec rake worker --trace clock: bundle exec bin/clock $ $ cat Procfile foreman start 18:08:22 web.1 | started with pid 3371 18:08:22 worker.1 | started with pid 3372 18:08:22 clock.1 | started with pid 3373

Slide 14

Slide 14 text

And scale it right up!

Slide 15

Slide 15 text

Scaling web dynos... done, now running 10 Scaling worker dynos... done, now running 5 $ $ heroku ps:scale web=10 worker=5 -a demo-app

Slide 16

Slide 16 text

Attach some add-ons addons.heroku.com

Slide 17

Slide 17 text

Adding loggly on demo-app… done, v161 (free) Use `heroku addons:docs loggly` to view documentation. $ $ heroku addons:add loggly

Slide 18

Slide 18 text

Heroku Postgres

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

Heroku Origins

Slide 21

Slide 21 text

focus on Rails

Slide 22

Slide 22 text

Rails apps need a database

Slide 23

Slide 23 text

web apps need a database

Slide 24

Slide 24 text

thankfully postgres was chosen

Slide 25

Slide 25 text

otherwise I wouldn’t be here

Slide 26

Slide 26 text

“let’s build a production grade database service”

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

Heroku Postgres iron.io loggly you Heroku Add-ons

Slide 29

Slide 29 text

Adding heroku-postgresql on demo-app… done, v129 (free) Attached as HEROKU_POSTGRESQL_ORANGE_URL Database has been created and is available Use `heroku addons:docs heroku-postgresql` to view documentation. $ $ heroku addons:add heroku-postgresql --app demo-app Plan: Dev Status: available Connections: 1 PG Version: 9.2.4 Data Size: 6.3 MB Tables: 0 Rows: 0/10000 (In compliance) Fork/Follow: Unsupported heroku pg:info -a demo-app

Slide 30

Slide 30 text

Heroku Postgres v0 A sinatra app implementing the heroku addons API create servers install postgres service create database for user - a “Resource” Sequel talks to postgres stem talks to AWS

Slide 31

Slide 31 text

the simplest thing that could possibly work, but no less

Slide 32

Slide 32 text

two main entities

Slide 33

Slide 33 text

Resource { database: ‘d3lwi9ef2’, port: 5432, username: ‘u23f8doife9’, password: ‘dfwefujp’, created_at: ‘2012-05-02’, state: ‘available’ }

Slide 34

Slide 34 text

Server { ip: ‘192.168.0.1’ instance_id: ‘i-2fidj3c8’, ami: ‘pg-prod’, availability_zone: ‘us-east-1a’ created_at: ‘2012-05-02’, state: ‘booting’ }

Slide 35

Slide 35 text

and a thin web interface erb templates   served by sinatra app

Slide 36

Slide 36 text

we’ve come a long way since then

Slide 37

Slide 37 text

Workflow and Monitoring

Slide 38

Slide 38 text

draw inspiration from game programming

Slide 39

Slide 39 text

observe environment finite state machine

Slide 40

Slide 40 text

available creating uncertain unavailable deprovisioning deprovisioned

Slide 41

Slide 41 text

class Resource def feel observations.create( Feeler.new(self).current_environment ) end end ! class Feeler def current_environment { services_available?: service_available?, connections: connections, row_count: row_count, table_count: table_count, seq_scans: seq_scans, index_scans: index_scans } end end

Slide 42

Slide 42 text

class Resource include Stateful ! state :available do unless service_available? transition :uncertain end end end ! resource = Resource.find resource.transition :available resource.feel resource.tick puts resource.state # => :uncertain

Slide 43

Slide 43 text

module Stateful def self.included(base) base.extend ClassMethods end ! module ClassMethods def state(name, &block) state[name] = block end def states; @states ||={}; end end ! def tick self.instance_eval( &self.class.states[self.state.to_sym] ) end def transition(state); end end

Slide 44

Slide 44 text

resource.feel resource.tick Need to do this all the time

Slide 45

Slide 45 text

db1 db2 db3 db4 db5 db6 db7 db8 dbn W orkers db1.feel  db1.tick

Slide 46

Slide 46 text

db4 db3 db2 db1 dbn Enqueue

Slide 47

Slide 47 text

Durability and Availability

Slide 48

Slide 48 text

INSERT INTO … 1. Write to WAL 2. Keep it in memory 4. Flush to disk 3. Respond to client

Slide 49

Slide 49 text

Ship WAL at least every 60s multi-datacenter storage

Slide 50

Slide 50 text

fork

Slide 51

Slide 51 text

follower

Slide 52

Slide 52 text

need a more flexible object model

Slide 53

Slide 53 text

timeline T0

Slide 54

Slide 54 text

participant

Slide 55

Slide 55 text

participant

Slide 56

Slide 56 text

participant

Slide 57

Slide 57 text

participant

Slide 58

Slide 58 text

participant

Slide 59

Slide 59 text

resource

Slide 60

Slide 60 text

followers

Slide 61

Slide 61 text

fork

Slide 62

Slide 62 text

disaster

Slide 63

Slide 63 text

disaster

Slide 64

Slide 64 text

recovery STONITH

Slide 65

Slide 65 text

big project

Slide 66

Slide 66 text

lots of moving parts long test suite

Slide 67

Slide 67 text

modularize and build APIs

Slide 68

Slide 68 text

move away from the large monolithic application

Slide 69

Slide 69 text

and into a constellation of apps talking to each other

Slide 70

Slide 70 text

Dashboard Dataclips Resources Heroku Admin UI Participants  Timelines  Servers CLI infra Monitoring

Slide 71

Slide 71 text

gain in agility

Slide 72

Slide 72 text

composable services

Slide 73

Slide 73 text

independently scalable

Slide 74

Slide 74 text

independently testable

Slide 75

Slide 75 text

current tooling

Slide 76

Slide 76 text

Rails Sinatra sequel fog ruby python bash go coffeescript backbone.js Dashboard Dataclips Resources Heroku Admin UI Participants  Timelines  Servers CLI infra Monitoring

Slide 77

Slide 77 text

Lessons managing databases is hard start simple extract and share code separate concerns into services

Slide 78

Slide 78 text

Thanks! @hgmnz @herokupostgres +