What Tech Leaders Must Know About Microservices

If your team practices DevOps, it's only a matter of time before they say "We want to move to microservices".

How do we, the tech leaders, arm ourselves with enough knowledge to make good decisions on the move to microservices. Many times engineers will emphatically push for this move because microservices represent the bleeding edge. They do not always assess the value of this change in the same way we do.

Using our real life experiences at Comcast, we will answer the following questions to help tech leaders make good decisions:

* What are microservices, exactly?
* What are the main reasons to move to microservices?
* What are reasons to NOT move to microservices?
* What maintenance considerations should my team prepare for?
* Where are the hidden costs that come with microservices?

At the end of this talk, you will have an appropriate level of understanding of microservices and will be prepared to make informed decisions with your team.

1. @leslieachapman| @michaelswinslow Michael Winslow Senior Director - Comcast I still code for management tasks What Tech Leaders Must Know About Microservices @michaelswinslow /michaelswinslow
2. @leslieachapman| @michaelswinslow Some background: 1. Michael was on a DEV team that built web services using Spring MVC 2. Catherine was on the Splunk Platform team (logging, observability, etc.) 3. At some point, the team and leadership decided to pursue Microservices 4. Things worked out well, but it was not always sunshine Now you have a choice: • Hear about the lessons your team learned! (red) • Hear about how great your team is at making microservices! (blue)
3. @leslieachapman| @michaelswinslow @leslieachapman| @michaelswinslow
4. @leslieachapman| @michaelswinslow @leslieachapman| @michaelswinslow
6. @leslieachapman| @michaelswinslow So, Microservices…
7. @leslieachapman| @michaelswinslow Is There a Need To Move to Microservices? Are you Netflix? No no Start
8. @leslieachapman| @michaelswinslow This is a MONOLITH Service Repository (DAO) Controller
9. @leslieachapman| @michaelswinslow This is a MONOLITH Service Repository (DAO) eligibility user device eligibility user device eligibility user device Controller eligibility user device
10. @leslieachapman| @michaelswinslow This is a MICROSERVICE Controller device Service Repository
11. @leslieachapman| @michaelswinslow Service Repository This is a MICROSERVICE Service Repository Service Repository Controller device Controller user Controller eligibility
12. @leslieachapman| @michaelswinslow Service Repository This is a MICROSERVICE Service Repository Service Repository Controller device Controller user Controller eligibility Note: Large organizations may dedicate a small team to each microservice. A team that has a great deal of developers who can work independently may benefit from this model.
13. @leslieachapman| @michaelswinslow Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway
14. @leslieachapman| @michaelswinslow Scale / Service Discovery
15. @leslieachapman| @michaelswinslow Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway “Oh no! For some reason requests to the ELIGIBILITY API have gone way up! At least 3x the traffic of any other API!”
16. @leslieachapman| @michaelswinslow Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway Controller eligibility Service Repository Controller eligibility Service Repository Discovery
17. @leslieachapman| @michaelswinslow Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway Controller eligibility Service Repository Controller eligibility Service Repository Discovery 1 2 3
18. @leslieachapman| @michaelswinslow Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway Controller eligibility Service Repository Controller eligibility Service Repository Discovery Note: Elasticity is one of the major reasons to move to microservices. If your traffic tends to increase and decrease dramatically, you may find value. Especially in the cloud.
19. @leslieachapman| @michaelswinslow Fault Tolerance / Circuit Breaker
20. @leslieachapman| @michaelswinslow Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway Controller eligibility Service Repository Controller eligibility Service Repository Discovery
21. @leslieachapman| @michaelswinslow Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway Controller eligibility Service Repository Controller eligibility Service Repository Discovery Retry 3/3
22. @leslieachapman| @michaelswinslow HYSTRIX Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway Controller eligibility Service Repository Discovery Retry 3/3
23. @leslieachapman| @michaelswinslow Summary: 1. Monolith vs Microservice 2. API Gateway - Zuul 3. Service Discovery - Eureka 4. Software Load Balancer – Ribbon 5. Circuit Breaker - Hystrix Now you have a choice: • That was amazing! Now tell us what your team learned! (red) • That was amazing! Now tell us what your team learned! (blue)
24. @leslieachapman| @michaelswinslow “Being able to scale and distribute your application resources is certainly alluring. However, this doesn't come for free. Complexity begets complexity. Soon tools are needed to manage the complexity. Then tools are needed to manage the tools.” – Ryan Emerle, Principal Engineer
25. @leslieachapman| @michaelswinslow Sample: We need to call our API to determine if a particular USERis ELIGIBLE to purchase a DEVICE
26. @leslieachapman| @michaelswinslow Service Repository (DAO) eligibility user device eligibility user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog -- -- -- -- process boundary syslog entry
27. @leslieachapman| @michaelswinslow Service Repository (DAO) eligibility user device eligibility user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 -- -- --
28. @leslieachapman| @michaelswinslow MONOLITH MICROSERVICES syslog applog syslog applog 03 -- 08 -- Note: Increased number of process boundaries also creates more surface area for security attacks during runtime.
29. @leslieachapman| @michaelswinslow So What? Microservices inflate the System Logs a little (and security risk). It’s totally worth it!
30. @leslieachapman| @michaelswinslow Maybe… but we’re not finished yet.
31. @leslieachapman| @michaelswinslow Execution Errors easy to track?
32. @leslieachapman| @michaelswinslow Service Repository (DAO) eligibility user device eligibility user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 -- 08 --
33. @leslieachapman| @michaelswinslow Service Repository (DAO) eligibility user device eligibility user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 -- 08 -- java.lang.NullPointerException: variable username is null at com.stacktrace.StackTraceExample.getUser(UserRepository.java:13) at com.stacktrace.StackTraceExample.loadUserByUserName(UserService.java:9) at com.stacktrace.StackTraceExample.fetchUser(EligibilityService.java:1012) at com.stacktrace.StackTraceExample.evaluateEligibility(EligibilityService.java:5) at com.stacktrace.StackTraceExample.evaluateEligibility(EligibilityController.java:27) STACK TRACE for Monolith
34. @leslieachapman| @michaelswinslow Service Repository (DAO) eligibility user device eligibility user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 -- 08 -- java.lang.NullPointerException: variable username is null at com.stacktrace.StackTraceExample.getUser(UserRepository.java:13) at com.stacktrace.StackTraceExample.loadUserByUserName(UserService.java:9) at com.stacktrace.StackTraceExample.fetchUser(EligibilityService.java:1012) at com.stacktrace.StackTraceExample.evaluateEligibility(EligibilityService.java:5) at com.stacktrace.StackTraceExample.evaluateEligibility(EligibilityController.java:27) STACK TRACE for Monolith
35. @leslieachapman| @michaelswinslow Service Repository (DAO) eligibility user device eligibility user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 -- Stack trace has visibility of everything in the Monolith
36. @leslieachapman| @michaelswinslow Service Repository (DAO) eligibility user device eligibility user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 --
37. @leslieachapman| @michaelswinslow Service Repository (DAO) eligibility user device eligibility user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 -- java.lang.NullPointerException: variable username is null at com.stacktrace.StackTraceExample.getUser(UserRepository.java:13) at com.stacktrace.StackTraceExample.loadUserByUserName(UserService.java:9) at com.stacktrace.StackTraceExample.loadUserByUserName(UserController.java:127) STACK TRACE for Microservice
38. @leslieachapman| @michaelswinslow Service Repository (DAO) eligibility user device eligibility user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 -- Stack trace only has visibility of the current microservice
39. @leslieachapman| @michaelswinslow Service Repository (DAO) eligibility user device eligibility user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 -- All context of where the request started is lost
40. @leslieachapman| @michaelswinslow Distributed Tracing and the logs that come with it [Dapper] [Zipkin] [Money] [OpenTelemetry] @pauljamescleary
41. @leslieachapman| @michaelswinslow “Microservices did not introduce the need for distributed tracing, but they certainly increased the need.” – Anthony Orapallo, Engineering Leader
42. @leslieachapman| @michaelswinslow Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 07 { "correlationId": 1, "method":"EligibilityController.evaluateEligibility", "level":"INFO"…} { "correlationId": 1, "method":"EligibilityService.evaluateEligibility", "level":"INFO"…} { "correlationId": 1, "method":"EligibilityService.fetchUser", "level":"INFO"…} { "correlationId": 1, "method":“UserController.fetchUser", "level":"INFO"…} { "correlationId": 1, "method":“UserService.loadUserByUsername", "level":"INFO"…} { "correlationId": 1, "method":“UserRepository.getUser", "level":"INFO"…} … { "correlationId": 1, "method": “UserRepository.getUser" "level": “ERR", "message": "variable username is null", "timestamp": 1385053862.3072, "_user_id": 9001, } correlationId = 1
43. @leslieachapman| @michaelswinslow Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway { "correlationId": 1, "method":"EligibilityController.evaluateEligibility", "level":”INFO"…} { "correlationId": 1, "method":"EligibilityService.evaluateEligibility", "level":”INFO"…} { "correlationId": 1, "method":"EligibilityService.fetchUser", "level":”INFO"…} { "correlationId": 1, "method":“UserController.fetchUser", "level":”INFO"…} { "correlationId": 1, "method":“UserService.loadUserByUsername", "level":”INFO"…} { "correlationId": 1, "method":“UserRepository.getUser", "level":”INFO"…} … { "correlationId": 1, "method": “UserRepository.getUser" "level": “ERR", “message": "variable username is null", "timestamp": 1385053862.3072, "_user_id": 9001, } MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 07 0 200 400 600 800 1000 1200 1400 1600 1 2 3 4 5 6 7 8 9 LOG ENTRIES # OF REQUESTS Log File Sizes Monolith Microservices 4x
44. @leslieachapman| @michaelswinslow What you did for ONE you now have to do for MANY
45. @leslieachapman| @michaelswinslow
46. @leslieachapman| @michaelswinslow
47. @leslieachapman| @michaelswinslow code review + CI CD to all environments firewalls + observability security scans
48. @leslieachapman| @michaelswinslow Suggestions: 1. Do not create all of your microservices at once 2. Monitor the usage to find your most utilized endpoints and use that to prioritize your migration 3. Have a template for creating new microservices and USE IT!
49. @leslieachapman| @michaelswinslow Change Management Team wants their release notes
50. @leslieachapman| @michaelswinslow
51. @leslieachapman| @michaelswinslow Version Control and Compatibility can be a nightmare
52. @leslieachapman| @michaelswinslow Release 1.5.7 A RELEASE version now consists of a configuration of many microservices. Each microservice version increments independently from the RELEASE and other microservices.
53. @leslieachapman| @michaelswinslow “…when you publish a new version of a mobile app, the old version doesn’t just magically go away.” – Mark Dappollone, Senior Android Developer Dappollone, M (2018). Future Proofing Mobile Applicationsfrom https://medium.com/datadriveninvestor/future-proofing-mobile-applications-56f37e80cd4c
54. @leslieachapman| @michaelswinslow Burnout is real
55. @leslieachapman| @michaelswinslow Note that the World Health Organization now classifies burn-out as an official medical disorder. World Health Organization (2019). QD85 Burn-Out from https://icd.who.int/browse11/l-m/en#/http://id.who.int/icd/entity/129180281
56. @leslieachapman| @michaelswinslow Recap with DOs and DON’Ts: DO have a reason to move to microservices (Resiliency, Cost-Optimization, etc.) DON'T practice RDD (Resume-Driven Development) DO communicate with all teams about monitoring, telemetry and distributed tracing DON'T count on traditional logging techniques (Stack Trace) DO peel off one microservice at a time (Strangler Application pattern) DON'T refactor everything at once DO understand that all teams are impacted (Operations, Release Mgmt, Security, etc.) DON'T say “we're doing microservices!” #YOLO DO staff appropriately and monitor employee health DON'T ignore the human impact of your decision
57. @leslieachapman| @michaelswinslow Recap with DOs and DON’Ts: DO encourage DevOps teams to pursue microservices if there are benefits DON'T underestimate the time, effort and money involved
58. @leslieachapman| @michaelswinslow Michael Winslow Senior Director - Comcast I still code for management tasks What Tech Leaders Must Know About Microservices @michaelswinslow /michaelswinslow
59. @leslieachapman| @michaelswinslow Helpful Links 1. Netflix Zuul - https://github.com/Netflix/zuul 2. Netflix Eureka - https://github.com/Netflix/eureka 3. Netflix Ribbon - https://github.com/Netflix/ribbon 4. Netflix Hystrix - https://github.com/Netflix/hystrix 5. Netflix Vizsceral - https://github.com/Netflix/vizceral 6. Google Dapper – https://ai.google/research/pubs/pub36356 7. Comcast Money - https://github.com/Comcast/money 8. Twitter Zipkin - https://zipkin.apache.org/ Tom Martin, Splunk – Zipkin and Splunk Mark Dappalone, Comcast – FutureProofing Your Mobile Applications