$30 off During Our Annual Pro Sale. View Details »

Services and Rails: The Shit They Don't Tell You

Services and Rails: The Shit They Don't Tell You

Building services and integrating them into Rails is hard. We want smaller Rails apps and nicely encapsulated services, but services introduce complexity. If you go overboard in the beginning, you're doing extra work and getting some of it wrong. If you wait too long, you've got a mess.

At Yammer, we constantly clean up the mess that worked well in the early days, but has become troublesome to maintain and scale. We pull things out of the core Rails app, stand them up on their own, and make sure they work well and are fast. With 20+ services, we've learned some lessons along the way. Services that seem clean in the beginning can turn into development environment nightmares. Temporary double-dispatching solutions turn into developer confusion. Monitoring one app turns into monitoring a suite of apps and handling failure between them.

This talk looks at our mistakes and solutions, the tradeoffs, and how we're able to keep moving quickly. Having services and a smaller Rails codebase makes for scalable development teams, happier engineers, and predictable production environments. Getting there is full of hard decisions -- sometimes we’re right, sometimes we fuck it up, but we usually have a story to tell.

Brian Morton

March 03, 2013
Tweet

More Decks by Brian Morton

Other Decks in Programming

Transcript

  1. Serv%ces and Ra%ls
    The Sh&t They
    Don’t T
    ell You

    View Slide

  2. Who &s th&s guy?

    View Slide

  3. But first…

    View Slide

  4. View Slide

  5. View Slide

  6. •  We have a huge fuck&ng Ra&ls app
    •  300+ models, 200 controllers
    •  20+ CVM serv&ces
    •  Some serv&ces do over a b&ll&on
    requests per day
    Alr&ght, let’s cont&nue

    View Slide

  7. …but we still have
    a huge Rails app.

    View Slide

  8. Service Oriented
    Architectures

    View Slide

  9. Components that
    scale individually.

    View Slide

  10. Reusab&l&ty

    View Slide

  11. •  Eas&er to scale out serv&ce
    •  Known and pred&ctable usage and
    performance patterns
    •  Monol&th&c arch&tecture means
    resources for everyth&ng
    Scalab&l&ty

    View Slide

  12. •  Smaller and more focused
    •  Encapsulated concerns
    •  Push updates &ndependently
    •  Change out everyth&ng w&thout tell&ng
    anyone*
    Loose coupl&ng

    View Slide

  13. Codebases that scale
    organizationally.

    View Slide

  14. •  Enabled by loose coupl&ng
    •  Ass&gn a team to the serv&ce
    •  Two teams coord&nate/agree on APQs
    •  Use dummy components
    D&str&buted execut&on

    View Slide

  15. View Slide

  16. Conway’s Law

    View Slide

  17. “Organ'zat'ons wh'ch des'gn systems ... are
    constra'ned to produce des'gns wh'ch are
    cop'es of the commun'cat'on structures of
    these organ'zat'ons.”
    Conway’s Law
    Melv&n Conway

    View Slide

  18. The Messag&ng Team

    View Slide

  19. •  Dec&des on &nterface and &mplements
    •  Has s&loed knowledge of system
    •  Not most &mportant th&ng to be
    work&ng on
    •  Do we keep creat&ng feature teams?
    The Messag&ng Team

    View Slide

  20. Ra&ls and Core Serv&ces Teams

    View Slide

  21. Cross@functional teams

    View Slide

  22. Example cross-funct&onal team

    View Slide

  23. •  Sp&ns up, learns doma&n
    •  Takes advantage of d&str&buted
    execut&on
    •  Team coord&nates on APQ between
    serv&ces and the cl&ent
    Cross-funct&onal Teams

    View Slide

  24. •  Have some cost w&th not hav&ng
    s&loed “experts”
    •  Be careful not to couple the APQ
    &mplementat&on to the cl&ent
    •  After proZect &s completed, we st&ll
    have to support the feature
    Tradeoffs

    View Slide

  25. View Slide

  26. Put them all beh&nd Ra&ls

    View Slide

  27. Or talk d&rectly to a serv&ce

    View Slide

  28. There are lots of ways to do &t

    View Slide

  29. View Slide

  30. Act&veRecord &s awesome
    •  Callbacks and hooks
    •  Val&dat&ons
    •  State mach&nes
    •  Cache &nval&dat&on*

    View Slide

  31. Ugh, Act&veRecord
    •  Ugh, callbacks and hooks
    •  Ugh, val&dat&ons
    •  Ugh, state mach&nes
    •  Ugh, cache &nval&dat&on

    View Slide

  32. •  Don’t use Act&veRecord
    •  Use your serv&ces as &ndexes – Zust
    store the QDs
    •  Move the data so the serv&ce owns &t
    What are my opt&ons?

    View Slide

  33. View Slide

  34. Mov&ng data
    •  Chances are you can’t have downt&me
    to move data
    •  What &f the serv&ce doesn’t work as
    &ntended?
    •  Roll&ng back &s hard, but you need a
    backup plan

    View Slide

  35. •  Backf&ll data to serv&ce
    •  Wr&te data to database and POST to
    serv&ce
    •  Serv&ce can be mon&tored and
    prof&led
    •  Move to the serv&ce &ncrementally
    Double d&spatch&ng

    View Slide

  36. Now you have
    duplicate data.

    View Slide

  37. The old way is
    comfortable.

    View Slide

  38. View Slide

  39. View Slide

  40. •  Close to product&on
    •  Runn&ng serv&ces locally
    •  Keep up w&th rap&dly chang&ng
    serv&ces
    Development env&ronment

    View Slide

  41. View Slide

  42. Deploy&ng serv&ces
    •  Need a system that allows you to add
    new serv&ces qu&ckly
    •  Easy to deploy each serv&ce
    •  Ma&nta&n stable and pre-release
    packages for development envs

    View Slide

  43. View Slide

  44. Mon&tor&ng and alert&ng
    •  Performance mon&tor&ng
    •  Capac&ty plann&ng and queue depths
    •  Reachab&l&ty from dependent cl&ents

    View Slide

  45. Standardized tools

    View Slide

  46. Dropwizard
    dropwizard.codahale.com

    View Slide

  47. View Slide

  48. Complex systems fail

    View Slide

  49. Serv&ce tradeoffs
    •  Handle unava&lab&l&ty
    •  Transact&ons aren’t free
    •  APQs are much harder to change
    •  There are no atom&c deploys

    View Slide

  50. Always reevaluate
    your costs and their
    viability.

    View Slide

  51. Organization that
    supports building
    services.

    View Slide

  52. Tools that allow you
    to keep moving fast.

    View Slide

  53. Be ready to be wrong.

    View Slide

  54. Brian Morton / @brianxq3
    eng.yammer.com

    View Slide