From Monolith to Microservices and back again. Twice.

The benefits of a microservice architecture to a growing organisation have been discussed by many, many people at conferences, in books, and on video. As an industry, we’re good at waxing lyrical about the gains in agility, productivity and reliability that such an architecture can provide. The problem is that very few organisations get this right, certainly not the first time, and certainly not without some serious pain along the way.

At ShopKeep we’ve been down this road. We’ve tried, and failed, to break our monolith apart and move towards the wonderful world of microservice nirvana. This talk is an adventure in failure and eventual recovery. I explore the transitions we made, the reasons why, the bad decisions, and the silly mistakes we made along the way. It’s not hopeless, there is light at the end of the tunnel, and hope for a glorious microservicey future for all of us.

Sharing our experiences of failure helps us all to build better, more reliable systems.

Transcript

1. From Monolith to Microservices and back again. Twice.
2. $whoami • Paddy Carey • Lead Engineer, ShopKeep • I do meetup stuff • @paddycarey everywhere that matters
3. $whoami • Paddy Carey • Lead Engineer, ShopKeep • I do meetup stuff (sometimes) • @paddycarey everywhere that matters
6. Monoliths
7. Microservices
8. In the beginning there was Rails ...and it was good
9. Scaling up goes a long way
10. Scaling out goes even further
15. Attempt #1
16. “Hercules”
17. Goals • Break out reporting functionality in a new service • Use the tools we already know to build something familiar • Build something scalable that'll allow us to grow our reporting functionality along with our user base for the next $NUM years
18. Architecture/Decisions • Rails/Sidekiq stack • EC2/Capistrano • Amazon Redshift • Lives behind our existing monolith • Invisible to callers/clients
19. Get your ducks in a row Lesson #1
21. Be prepared to fail Lesson #2
22. Back to the drawing board
23. Attempt #2
24. “Platform”
25. Goals • Break out reporting functionality in a new service whilst providing a foundation for ShopKeep's future growth on the backend. • Use the best-in-class tools to build something scalable and manageable • Build something scalable that'll allow us to grow our reporting functionality along with our user base for the next $NUM years
26. Architecture/Decisions • Go microservices • Radical decomposition • Heterogeneous datastores ◦ PostgreSQL ◦ Cassandra ◦ Elasticsearch ◦ Kinesis for shuffling data around • Protobufs everywhere • Docker/Marathon/Mesos (originally we used Deis) • New set of client-facing APIs • Lives behind our existing monolith
27. Seriously, get your ducks in a row Lesson #3
28. Don't throw the baby out with the bathwater Lesson #4
29. Beware the distributed monolith Lesson #5
31. Don't bite off more than you can chew Lesson #6
32. Don't build in isolation Lesson #7
33. Know your system Lesson #8
34. Shared schemas can be more hindrance than help Lesson #9
35. Use a real API gateway Lesson #10
36. Have clearly defined failure modes Lesson #11
37. How did we find ourselves back here again?
38. Taking a step back
39. Attempt #3
40. “Rogue”
41. Goals • Break out reporting functionality in a new service that can consolidate our previous attempts at this functionality. • Initially replace only our existing reports. • Use the best-in-class tools to build something scalable and manageable • Build something scalable that'll allow us to grow our reporting functionality along with our user base for the next $NUM years • Prototype multiple approaches to validate our thinking before committing to a long intensive build
42. Architecture/Decisions • A small handful of Python services • A message queue for shuffling data between systems asynchronously • CQRS(ish) • Initially SNS/SQS based (now Kafka)
43. Start small Lesson #12
44. Vertical slicing allows for easier decoupling Lesson #13
45. Sometimes plain, boring and predictable works best Lesson #14
46. Understanding your constraints Lesson #15
47. Set precedent by example, not by decree Lesson #16
48. Everything should be replaceable (even you) Lesson #17
49. Resilience is key Lesson #18
50. Automate all the things Lesson #19
51. Get it right once, and the rest comes easy Lesson #20
52. Conclusion • Microservices are hard • You'll probably get it wrong the first time ◦ that's totally OK, so long as you learn from it ◦ in fact, unless you make these mistakes yourself, you're unlikely to appreciate the subtlety and pain of some of the decisions you're making • Don't be afraid of failure, embrace it, iterate where you need to • You can always start over
53. Ta muchly! Questions? @paddycarey