Monolith to Microservices: Lessons From The Trenches

Video: https://www.youtube.com/watch?v=T09xgDrRw08

Hootsuite is undergoing an architectural transformation. We're moving from our large monolithic PHP app to a microservices architecture. Microservices hold the promise of faster performance scaling, reduced technical debt, higher reliability, and increased speed of innovation, however the journey to get there isn't easy. How do you carve out parts of your app, rebuild them as services, and then switch to using those services, all while the plane is still flying--without any downtime and without your users noticing.

In this talk I'll define what exactly a microservices architecture looks like, discuss its pros and cons, and speak about how to best partition your app into services. We'll then go over lessons learned at Hootsuite as we've carved up our own app into microservices; what worked and what fell flat on its face.

Talk given at Pacific Northwest PHP Conference on September 16th, 2015.

Transcript

1. Monolith to Microservices: Lessons From The Trenches Pacific Northwest PHP Conference September 12th, 2015 Luke Kysow - @lkysow
3. >whoami
5. @lkysow @lkysow linkedin.com/in/lkysow
6. Agenda
13. Scale • well over 10M users • 5000 requests/second • 4 million messages sent per day • Main application written in PHP, 60 front end servers all deployed on AWS, classic Memcached + MySQL • over 200 servers in our service layer • [ server diagram]
14. • two years ago ◦ classic startup story ◦ hard 5 year battle of adding features and beating competitors ◦ hockey stick growth ◦ 75 developers ◦ huge PHP monolith Set the stage 2 Years Ago
15. [ Server diagram ] LOC? Different services [get bill’s diagram with cron/job servers] Architecture 2 years ago
19. Need a plan [ picture of a blue print or a plan ]
20. • A microservice is a single deployable artifact that can be deployed independently • microservices communicate only via an API • have their own DB’s • subset of SOA What are microservices?
21. Monolith diagram • shows publishing, streams, and team management all in one box • single deployable artifact across multiple nodes Example of what our microservices would look like
22. [dashboard as front-end for services] instead the dashboard will be the front-end for a suite of services that would service those needs Example of what our microservices would look like
23. We’re not the only ones [ list of companies ] Why microservices?
24. Speed • not necessarily performance • but innovation, agility, and development Why microservices?
25. Speed • we want 100 engineers working in parallel, not in sequence Why microservices?
26. Speed • build pipeline for the dashboard Why microservices?
27. Speed of development • microservices are deployed independently • bugs only slow down a subset of your developers & features Why microservices?
28. Technical debt • development gets slowed down by dependencies between components • in a monolith corners inevitably get cut and components boundaries are • results in technical debt so it’s slower to develop ◦ example of gearman worker and typehinting • faster to build greenfield services Why microservices?
29. More innovation • use a new language? • new version of a library everything else depends on? • speed Why microservices?
30. Easier to scale • have one endpoint that’s being called a lot? • with a monolith you can only scale all your servers up • with microservices you just scale that one service • speed Why microservices?
31. Reliability • microservices isolate failures Why microservices?
32. Speed • don’t hold back your developers through a monolithic deploy pipeline • ammeliorate technical debt by enforcing component boundaries • innovate and try new technologies • scale • higher reliability Why microservices?
33. Back to Hootsuite [ we have a plan but how do we get started? ]
34. • Step 1: build a small prototype for a non-critical feature • built a pdf service ◦ that worked ◦ used our service template with brokers and app workers Back to Hootsuite
35. • Step 2: map out domain objects/bounded contexts • example ◦ members ◦ organizations ◦ teams Back to Hootsuite
36. • Figure out dependencies ◦ [map of teams belonging to organizations, teams being made of members] Back to Hootsuite
37. • member service was the key ◦ root dependency for other domain objects ◦ incredibly important service, would ensure we built the best service we could ◦ high volume, would teach us a lot of lessons ◦ highly integrated to the monolith, would show us what it’s like to pull it out Back to Hootsuite
38. • member service plan ◦ split out DB ◦ create simple CRUD REST service ◦ monolith would call that service ◦ the next service we wrote could use it Back to Hootsuite
39. • migration strategy ◦ new db ◦ double writes Member Service
40. • migration strategy ◦ controlled by dark launching ◦ if (Feature::isEnabled(‘FEATURE_NAME’)) { ◦ $member = newService(); ◦ } else { ◦ $member = oldService(); ◦ } Member Service
41. [ screen shot of dark launch admin panel ] Member Service
42. • circuit breaker ◦ [ code snippet] ◦ if ($numErrorsInLastMin > $errorThreshold) { ◦ Logger::log(“falling back to dashboard DB”) ◦ $member = oldMethod(); ◦ } Member Service
43. • public function log($message) { • $memberId = getMemberId(); • Logger::log(“$message, for member: $memberId” • } Circuit Breaker issues...
44. • PHP SDK’s for clients ◦ no issues with versioning Member Service
45. Member Service • Monitoring? ◦ graphite ◦ elasticsearch, logstash, kibana ◦ [ picture of graphite and kibana]
46. • Success! • [diagram of final architecture] Member Service
47. [ strike forces ] Conway’s Law
48. [ specialists ] Conway’s Law
49. [ split teams across product areas ] Conway’s Law
50. Need to enable teams to build new features as services
51. SOM • wrote a full API layer on top of the existing monolith, affectionately termed the “service oriented monolith” • faster than building out all the services for those domain objects • enabled more services to be built • when we had the time to move them to real services, consumers just had to switch their endpoints Hootsuite
52. • Since SOM
53. Event Bus? Hootsuite
54. • continuing to scale with microservices • issues ◦ snowflake deployment pipelines across services ◦ ops tooling for provisioning servers, security groups needs to catch up ◦ constant tension between building features on the monolith vs. as new services Conclusion
55. • more complexity ◦ dealing with distributed systems, eventual consistency • need excellent devops ◦ deployments, monitoring, logging • restructure your development organization ◦ teams need to own their stack Why not microservices?
56. • split on domains • split out DB • Re-organize teams to map to product areas • API’s on the monolith (service oriented monolith) Strategies • dark launching • circuit breakers • SDK’s • graphite • kibana (ELK) Tools