5 ways we screwed up Microservices

Lessons learned while adopting Microservices at Yammer

Transcript

1. Hi!  My  name  is  Gonzalo  and  we’re  going  to  go  over  some  challenges  we  had  while   adop:ng  an  SOA  architecture.   1  
5. This  is  our  main  page   5  
6. We  use  a  lot  of  services,  they  are  micro.  And  that’s  it.   6  
7. The  end.   7  
8. Well,  not  exactly,  we  had  some  early  mistakes   8  
9. *  We  didn’t  use  Circuit  breakers  at  the  start,  so  we  didn't  have  Back  Pressure.   *  We  had  unbounded  queues  (Which  can  cause  RabbitMQ  to  run  out  of  memory)   9  
10. *  We  Thought  all  services  had  the  same  storage  requirements  and  created  a  shared   AP  storage  that  could  get  saturated  by  a  single  service...   *  We  are  not  a  db  company,  so  our  custom  sharing/rou:ng  logic  wasn’t  the  best.   *  Our  consistent  hashing  didn’t  spread  par::ons  enough  and  created  hotspots.   *  Sharing  storage  also  complicates  tracking  state  changes  and  makes  database   migra:ons  a  nightmare.   10  
11. *  JSON  is  awesome  and  lightweight…  But  it's  schema-­‐less,  so  you  can't  deprecate  old   endpoints  easily  and  you  will  need  to  create  new  endpoints  for  adding  new  required   fields.   •  Integra:on  tes:ng  is  vital   11  
12. *  Created  some  Mini-­‐Monoliths   *  Model  dependencies  are  bad,  Services  dependencies  are  ten  :mes  worse  (SLA   degrades  exponen:ally  as  your  dependencies  increase)  If  you  have  two  services  at   99.99%  availability  now  you  are  down  to  99.95%  (Note:  These  numbers  are  made  up,   but  some:mes  it  is  that  bad.)     *  Graceful  degrada:on  should  not  be  an  aferthought.   *  We  didn’t  enforce  strict  boundaries  which  created  dependencies  on  their  internals.   This  creates  fragile  services.   12  
13. *  We  didn’t  have  consistent  metrics   *  Our  dashboards  were  slow   *  We  didn’t  have  aggregated  logs   *  Tracing  issues  across  services  is  hard.   13  
16. Microservices  help  you  deploy  fixes  faster  and  scale  beger.   16  
18. Building  Microservices  by  Sam  Newman   18  
20. And  watch  this  Big  Ruby  talk  by  Brian  Morton.   hgps://www.youtube.com/watch?v=GuJ49PNBsn8     20