Redis at Heroku

Given at RedisConf 2015

Transcript

1. Redis at Heroku
   Harold Giménez
   @hgmnz
   redisconf 2015 +
   

   View Slide

2. Redis at Heroku
   git deployment
   

   View Slide

3. proxy
   git push heroku master
   worker
   worker
   worker
   build service
   repos
   git push
   user
   

   View Slide

4. proxy
   git push heroku master
   worker
   worker
   worker
   build service
   repos
   build app
   breezing-swiftly-152
   user
   

   View Slide

5. proxy
   git push heroku master
   worker
   worker
   worker
   build service
   repos
   I got this
   user
   

   View Slide

6. proxy
   git push heroku master
   worker
   worker
   worker
   build service
   repos
   user
   

   View Slide

7. proxy
   git push heroku master
   worker
   worker
   worker
   build service
   repos
   here’s my
   hostname
   and port
   user
   

   View Slide

8. proxy
   git push heroku master
   worker
   worker
   worker
   build service
   repos
   user
   

   View Slide

9. Redis at Heroku
   Logs
   

   View Slide

10. heroku logs --tail -a hgmnz
    2010-09-16T15:13:46.677020+00:00 app[web.1]: Processing
    PostController#list (for 208.39.138.12 at 2010-09-16
    15:13:46) [GET]
    2010-09-16T15:13:46.677023+00:00 app[web.1]: Rendering
    template within layouts/application
    2010-09-16T15:13:46.698234+00:00 app[web.1]: Completed in
    74ms (View: 31, DB: 40) | 200 OK [http://
    myapp.heroku.com/]
    2010-09-16T15:13:46.723498+00:00 heroku[router]: at=info
    method=GET path="/posts" host=myapp.herokuapp.com"
    fwd="204.204.204.204" dyno=web.1 connect=1ms service=18ms
    status=200 bytes=975
    2010-09-16T15:13:47.893472+00:00 app[worker.1]: 2 jobs
    processed at 16.6761 j/s, 0 failed ...
    $
    

    View Slide

11. app
    dyno 1
    dyno 2
    dyno 3
    postgres
    primary
    postgres
    follower
    redis
    logplex
    config
    heroku logs --tail
    Logplex
    app
    log messages
    

    View Slide

12. Logplex canaries
    msg producer
    msg producer
    msg producer
    msg producer
    logging
    pipeline
    msg set
    message loss?
    latency?
    

    View Slide

13. Redis at Heroku
    Orchestration
    

    View Slide

14. Process orchestration
    Scaling web dynos... done, now running 10
    Scaling worker dynos... done, now running 5
    $
    $ heroku ps:scale web=10 worker=5 -a demo-app
    process
    manager
    Messages sent to any Redis to boot,

    kill, restart processes
    dynos
    

    View Slide

15. Web dyno idling
    heroku router
    cluster
    keys with a
    1h expiry
    HTTP traffic
    process
    manager
    which apps have had

    HTTP traffic

    in the last 24h?
    

    View Slide

16. Redis at Heroku
    Database Lifecycle
    

    View Slide

17. observe environment
    finite state machine
    

    View Slide

18. available
    creating
    uncertain
    unavailable
    deprovisioning
    deprovisioned
    

    View Slide

19. class Resource
    def feel
    observations.create(
    Feeler.new(self).current_environment
    )
    end
    end
    class Feeler
    def current_environment
    {
    services_available?: service_available?,
    connections: connections,
    row_count: row_count,
    table_count: table_count,
    seq_scans: seq_scans,
    index_scans: index_scans
    }
    end
    end
    

    View Slide

20. class Resource
    include Stateful
    state :available do
    unless service_available?
    transition :uncertain
    end
    end
    end
    resource = Resource.find
    resource.transition :available
    resource.fetch_observation
    resource.tick
    puts resource.state
    # => :uncertain
    

    View Slide

21. module Stateful
    def self.included(base)
    base.extend ClassMethods
    end
    module ClassMethods
    def state(name, &block)
    state[name] = block
    end
    def states; @states ||={}; end
    end
    def tick
    self.instance_eval(
    &self.class.states[self.state.to_sym]
    )
    end
    def transition(state); end
    end
    

    View Slide

22. resource.fetch_observation
    resource.tick
    Need to do this all the time
    

    View Slide

23. db1 db2 db3 db4 db5 db6 db7 db8 dbn
    W
    orkers
    db1.fetch_observation

    db1.tick
    

    View Slide

24. db
    db3
    db2
    db1
    dbn
    Enqueue
    

    View Slide

25. Redis at Heroku
    Database Monitoring
    

    View Slide

26. Database Observations and Monitoring
    supervisor
    worker 1
    worker 2
    worker 3
    worker 4
    worker N
    I’m available
    for work
    

    View Slide

27. supervisor
    worker 1
    worker 2
    worker 3
    worker 4
    worker N
    Worker 1
    monitor this DB
    Database Observations and Monitoring
    

    View Slide

28. supervisor
    worker 1
    worker 2
    worker 3
    worker 4
    worker N
    Database Observations and Monitoring
    monitor
    

    View Slide

29. supervisor
    worker 1
    worker 2
    worker 3
    worker 4
    worker N
    Database Observations and Monitoring
    

    View Slide

30. supervisor
    worker 1
    worker 2
    worker 3
    worker 4
    worker N
    I can’t reach
    this DB
    Database Observations and Monitoring
    

    View Slide

31. supervisor
    worker 1
    worker 2
    worker 3
    worker 4
    worker N
    Worker 4
    monitor this DB
    Database Observations and Monitoring
    

    View Slide

32. supervisor
    worker 1
    worker 2
    worker 3
    worker 4
    worker N
    Database Observations and Monitoring
    monitor
    

    View Slide

33. worker
    Database Monitoring Last Observation cache
    postgres
    Heroku Postgres
    automation service
    GET /last_observation
    HSET db1 db_size 4213
    HSET db1 cache_hit_rate 98
    

    View Slide

34. Redis at Heroku
    Other use cases
    

    View Slide

35. Router stats
    router node
    router node
    router node
    router node
    router node
    router node
    router node
    requests
    errors
    

    View Slide

36. pubsub + websockets
    dyno 1
    dyno 2
    dyno 3
    pubsub
    browser
    websockets
    

    View Slide

37. I asked: what’s good and bad
    about Redis?
    

    View Slide

38. good bad
    •Fast
    •Simple
    •Powerful primitives
    •Expires on any data type
    •Many use cases, single infra
    •replication protocol is
    “surprisingly elegant”
    •No SSL support
    •Rotating creds is a pain
    •SPOF
    •Single threaded(ish)
    

    View Slide

39. Thanks!
    @hgmnz
    +
    

    View Slide