Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Services and Rails: The Shit They Don't Tell You

Services and Rails: The Shit They Don't Tell You

Building services and integrating them into Rails is hard. We want smaller Rails apps and nicely encapsulated services, but services introduce complexity. If you go overboard in the beginning, you're doing extra work and getting some of it wrong. If you wait too long, you've got a mess.

At Yammer, we constantly clean up the mess that worked well in the early days, but has become troublesome to maintain and scale. We pull things out of the core Rails app, stand them up on their own, and make sure they work well and are fast. With 20+ services, we've learned some lessons along the way. Services that seem clean in the beginning can turn into development environment nightmares. Temporary double-dispatching solutions turn into developer confusion. Monitoring one app turns into monitoring a suite of apps and handling failure between them.

This talk looks at our mistakes and solutions, the tradeoffs, and how we're able to keep moving quickly. Having services and a smaller Rails codebase makes for scalable development teams, happier engineers, and predictable production environments. Getting there is full of hard decisions -- sometimes we’re right, sometimes we fuck it up, but we usually have a story to tell.

Brian Morton

March 03, 2013
Tweet

More Decks by Brian Morton

Other Decks in Programming

Transcript

  1. •  We have a huge fuck&ng Ra&ls app •  300+

    models, 200 controllers •  20+ CVM serv&ces •  Some serv&ces do over a b&ll&on requests per day Alr&ght, let’s cont&nue
  2. •  Eas&er to scale out serv&ce •  Known and pred&ctable

    usage and performance patterns •  Monol&th&c arch&tecture means resources for everyth&ng Scalab&l&ty
  3. •  Smaller and more focused •  Encapsulated concerns •  Push

    updates &ndependently •  Change out everyth&ng w&thout tell&ng anyone* Loose coupl&ng
  4. •  Enabled by loose coupl&ng •  Ass&gn a team to

    the serv&ce •  Two teams coord&nate/agree on APQs •  Use dummy components D&str&buted execut&on
  5. “Organ'zat'ons wh'ch des'gn systems ... are constra'ned to produce des'gns

    wh'ch are cop'es of the commun'cat'on structures of these organ'zat'ons.” Conway’s Law Melv&n Conway
  6. •  Dec&des on &nterface and &mplements •  Has s&loed knowledge

    of system •  Not most &mportant th&ng to be work&ng on •  Do we keep creat&ng feature teams? The Messag&ng Team
  7. •  Sp&ns up, learns doma&n •  Takes advantage of d&str&buted

    execut&on •  Team coord&nates on APQ between serv&ces and the cl&ent Cross-funct&onal Teams
  8. •  Have some cost w&th not hav&ng s&loed “experts” • 

    Be careful not to couple the APQ &mplementat&on to the cl&ent •  After proZect &s completed, we st&ll have to support the feature Tradeoffs
  9. Ugh, Act&veRecord •  Ugh, callbacks and hooks •  Ugh, val&dat&ons

    •  Ugh, state mach&nes •  Ugh, cache &nval&dat&on
  10. •  Don’t use Act&veRecord •  Use your serv&ces as &ndexes

    – Zust store the QDs •  Move the data so the serv&ce owns &t What are my opt&ons?
  11. Mov&ng data •  Chances are you can’t have downt&me to

    move data •  What &f the serv&ce doesn’t work as &ntended? •  Roll&ng back &s hard, but you need a backup plan
  12. •  Backf&ll data to serv&ce •  Wr&te data to database

    and POST to serv&ce •  Serv&ce can be mon&tored and prof&led •  Move to the serv&ce &ncrementally Double d&spatch&ng
  13. •  Close to product&on •  Runn&ng serv&ces locally •  Keep

    up w&th rap&dly chang&ng serv&ces Development env&ronment
  14. Deploy&ng serv&ces •  Need a system that allows you to

    add new serv&ces qu&ckly •  Easy to deploy each serv&ce •  Ma&nta&n stable and pre-release packages for development envs
  15. Mon&tor&ng and alert&ng •  Performance mon&tor&ng •  Capac&ty plann&ng and

    queue depths •  Reachab&l&ty from dependent cl&ents
  16. Serv&ce tradeoffs •  Handle unava&lab&l&ty •  Transact&ons aren’t free • 

    APQs are much harder to change •  There are no atom&c deploys