Save 37% off PRO during our Black Friday Sale! »

Services and Rails: The Shit They Don't Tell You

Services and Rails: The Shit They Don't Tell You

Building services and integrating them into Rails is hard. We want smaller Rails apps and nicely encapsulated services, but services introduce complexity. If you go overboard in the beginning, you're doing extra work and getting some of it wrong. If you wait too long, you've got a mess.

At Yammer, we constantly clean up the mess that worked well in the early days, but has become troublesome to maintain and scale. We pull things out of the core Rails app, stand them up on their own, and make sure they work well and are fast. With 20+ services, we've learned some lessons along the way. Services that seem clean in the beginning can turn into development environment nightmares. Temporary double-dispatching solutions turn into developer confusion. Monitoring one app turns into monitoring a suite of apps and handling failure between them.

This talk looks at our mistakes and solutions, the tradeoffs, and how we're able to keep moving quickly. Having services and a smaller Rails codebase makes for scalable development teams, happier engineers, and predictable production environments. Getting there is full of hard decisions -- sometimes we’re right, sometimes we fuck it up, but we usually have a story to tell.


Brian Morton

March 03, 2013


  1. Serv%ces and Ra%ls The Sh&t They Don’t T ell You

  2. Who &s th&s guy?

  3. But first…

  4. None
  5. None
  6. •  We have a huge fuck&ng Ra&ls app •  300+

    models, 200 controllers •  20+ CVM serv&ces •  Some serv&ces do over a b&ll&on requests per day Alr&ght, let’s cont&nue
  7. …but we still have a huge Rails app.

  8. Service Oriented Architectures

  9. Components that scale individually.

  10. Reusab&l&ty

  11. •  Eas&er to scale out serv&ce •  Known and pred&ctable

    usage and performance patterns •  Monol&th&c arch&tecture means resources for everyth&ng Scalab&l&ty
  12. •  Smaller and more focused •  Encapsulated concerns •  Push

    updates &ndependently •  Change out everyth&ng w&thout tell&ng anyone* Loose coupl&ng
  13. Codebases that scale organizationally.

  14. •  Enabled by loose coupl&ng •  Ass&gn a team to

    the serv&ce •  Two teams coord&nate/agree on APQs •  Use dummy components D&str&buted execut&on
  15. None
  16. Conway’s Law

  17. “Organ'zat'ons wh'ch des'gn systems ... are constra'ned to produce des'gns

    wh'ch are cop'es of the commun'cat'on structures of these organ'zat'ons.” Conway’s Law Melv&n Conway
  18. The Messag&ng Team

  19. •  Dec&des on &nterface and &mplements •  Has s&loed knowledge

    of system •  Not most &mportant th&ng to be work&ng on •  Do we keep creat&ng feature teams? The Messag&ng Team
  20. Ra&ls and Core Serv&ces Teams

  21. Cross@functional teams

  22. Example cross-funct&onal team

  23. •  Sp&ns up, learns doma&n •  Takes advantage of d&str&buted

    execut&on •  Team coord&nates on APQ between serv&ces and the cl&ent Cross-funct&onal Teams
  24. •  Have some cost w&th not hav&ng s&loed “experts” • 

    Be careful not to couple the APQ &mplementat&on to the cl&ent •  After proZect &s completed, we st&ll have to support the feature Tradeoffs
  25. None
  26. Put them all beh&nd Ra&ls

  27. Or talk d&rectly to a serv&ce

  28. There are lots of ways to do &t

  29. None
  30. Act&veRecord &s awesome •  Callbacks and hooks •  Val&dat&ons • 

    State mach&nes •  Cache &nval&dat&on*
  31. Ugh, Act&veRecord •  Ugh, callbacks and hooks •  Ugh, val&dat&ons

    •  Ugh, state mach&nes •  Ugh, cache &nval&dat&on
  32. •  Don’t use Act&veRecord •  Use your serv&ces as &ndexes

    – Zust store the QDs •  Move the data so the serv&ce owns &t What are my opt&ons?
  33. None
  34. Mov&ng data •  Chances are you can’t have downt&me to

    move data •  What &f the serv&ce doesn’t work as &ntended? •  Roll&ng back &s hard, but you need a backup plan
  35. •  Backf&ll data to serv&ce •  Wr&te data to database

    and POST to serv&ce •  Serv&ce can be mon&tored and prof&led •  Move to the serv&ce &ncrementally Double d&spatch&ng
  36. Now you have duplicate data.

  37. The old way is comfortable.

  38. None
  39. None
  40. •  Close to product&on •  Runn&ng serv&ces locally •  Keep

    up w&th rap&dly chang&ng serv&ces Development env&ronment
  41. None
  42. Deploy&ng serv&ces •  Need a system that allows you to

    add new serv&ces qu&ckly •  Easy to deploy each serv&ce •  Ma&nta&n stable and pre-release packages for development envs
  43. None
  44. Mon&tor&ng and alert&ng •  Performance mon&tor&ng •  Capac&ty plann&ng and

    queue depths •  Reachab&l&ty from dependent cl&ents
  45. Standardized tools

  46. Dropwizard

  47. None
  48. Complex systems fail

  49. Serv&ce tradeoffs •  Handle unava&lab&l&ty •  Transact&ons aren’t free • 

    APQs are much harder to change •  There are no atom&c deploys
  50. Always reevaluate your costs and their viability.

  51. Organization that supports building services.

  52. Tools that allow you to keep moving fast.

  53. Be ready to be wrong.

  54. Brian Morton / @brianxq3