Operating Microservices

Listen closely. Close your eyes. I know you hear it coming. It's the microservices buzz. Well, by now it is more of a quake. Catherine Rodriguez and Michael Winslow have been involved with 2 enterprise level microservices implementations. One was greenfield and the other was a migration from a FAT service. It turns out that the 2 easiest things to do when moving to microservices are to DECIDE to do it and to WRITE THE CODE. If you stick around after that, you better be ready for a ride!

Michael will address considerations like scaling, testing and releasing your microservices.
Catherine will cover logging and traceability.

Transcript

1. @catherinerod27 | @michaelswinslow Michael Winslow Core Applications - Comcast Spaces > Tabs Operating Microservices @michaelswinslow /michaelswinslow Catherine Rodriguez Core Applications - Comcast I have a twin sister /catherinerodriguez1 @CatherineRod27
2. @catherinerod27 | @michaelswinslow Some background: 1. Michael was on a DEV team that built web services using Spring MVC 2. Catherine was on the Splunk Platform team (logging, observability, etc.) 3. At some point, the team and leadership decided to pursue Microservices 4. Things worked out well, but it was not always sunshine Now you have a choice: • Hear about the lessons your team learned! (red) • Hear about how great your team is at making microservices! (blue)
3. @catherinerod27 | @michaelswinslow @catherinerod27 | @michaelswinslow
4. @catherinerod27 | @michaelswinslow @catherinerod27 | @michaelswinslow
5. @catherinerod27 | @michaelswinslow So, Microservices…
6. @catherinerod27 | @michaelswinslow Is There a Need To Move to Microservices? Are you Netflix? No no Start
7. @catherinerod27 | @michaelswinslow This is a MONOLITH Service Repository (DAO) Controller
8. @catherinerod27 | @michaelswinslow This is a MONOLITH Service Repository (DAO) eligibility user device eligibility user device eligibility user device Controller eligibility user device
9. @catherinerod27 | @michaelswinslow This is a MICROSERVICE Controller device Service Repository
10. @catherinerod27 | @michaelswinslow Service Repository This is a MICROSERVICE Service Repository Service Repository Controller device Controller user Controller eligibility
11. @catherinerod27 | @michaelswinslow Service Repository This is a MICROSERVICE Service Repository Service Repository Controller device Controller user Controller eligibility Note: Large organizations may dedicate a small team to each microservice. A team that has a great deal of developers who can work independently may benefit from this model.
12. @catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway
13. @catherinerod27 | @michaelswinslow Scale / Service Discovery
14. @catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway “Oh no! For some reason requests to the ELIGIBILITY API have gone way up! At least 3x the traffic of any other API!”
15. @catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway Controller eligibility Service Repository Controller eligibility Service Repository Discovery
16. @catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway Controller eligibility Service Repository Controller eligibility Service Repository Discovery 1 2 3
17. @catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway Controller eligibility Service Repository Controller eligibility Service Repository Discovery Note: Elasticity is one of the major reasons to move to microservices. If your traffic tends to increase and decrease dramatically, you may find value. Especially in the cloud.
18. @catherinerod27 | @michaelswinslow Fault Tolerance / Circuit Breaker
19. @catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway Controller eligibility Service Repository Controller eligibility Service Repository Discovery
20. @catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway Controller eligibility Service Repository Controller eligibility Service Repository Discovery Retry 3/3
21. @catherinerod27 | @michaelswinslow HYSTRIX Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway Controller eligibility Service Repository Discovery Retry 3/3
22. @catherinerod27 | @michaelswinslow Summary: 1. Monolith vs Microservice 2. API Gateway - Zuul 3. Service Discovery - Eureka 4. Software Load Balancer – Ribbon 5. Circuit Breaker - Hystrix Now you have a choice: • That was amazing! Now tell us what your team learned! (red) • That was amazing! Now tell us what your team learned! (blue)
23. @catherinerod27 | @michaelswinslow “Being able to scale and distribute your application resources is certainly alluring. However, this doesn't come for free. Complexity begets complexity. Soon tools are needed to manage the complexity. Then tools are needed to manage the tools.” – Ryan Emerle, Principal Engineer
24. @catherinerod27 | @michaelswinslow Sample: We need to call our API to determine if a particular USER is ELIGIBLE to purchase a DEVICE
25. @catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog -- -- -- -- process boundary syslog entry
26. @catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 -- -- --
27. @catherinerod27 | @michaelswinslow MONOLITH MICROSERVICES syslog applog syslog applog 03 -- 08 -- Note: Increased number of process boundaries also creates more surface area for security attacks during runtime.
28. @catherinerod27 | @michaelswinslow So What? Microservices inflate the System Logs a little (and security risk). It’s totally worth it!
29. @catherinerod27 | @michaelswinslow Maybe… but we’re not finished yet.
30. @catherinerod27 | @michaelswinslow Execution Errors easy to track?
31. @catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 -- 08 --
32. @catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 -- 08 -- java.lang.NullPointerException: variable username is null at com.stacktrace.StackTraceExample.getUser(UserRepository.java:13) at com.stacktrace.StackTraceExample.loadUserByUserName(UserService.java:9) at com.stacktrace.StackTraceExample.fetchUser(EligibilityService.java:1012) at com.stacktrace.StackTraceExample.evaluateEligibility(EligibilityService.java:5) at com.stacktrace.StackTraceExample.evaluateEligibility(EligibilityController.java:27) STACK TRACE for Monolith
33. @catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 -- 08 -- java.lang.NullPointerException: variable username is null at com.stacktrace.StackTraceExample.getUser(UserRepository.java:13) at com.stacktrace.StackTraceExample.loadUserByUserName(UserService.java:9) at com.stacktrace.StackTraceExample.fetchUser(EligibilityService.java:1012) at com.stacktrace.StackTraceExample.evaluateEligibility(EligibilityService.java:5) at com.stacktrace.StackTraceExample.evaluateEligibility(EligibilityController.java:27) STACK TRACE for Monolith
34. @catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 -- Stack trace has visibility of everything in the Monolith
35. @catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 --
36. @catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 -- java.lang.NullPointerException: variable username is null at com.stacktrace.StackTraceExample.getUser(UserRepository.java:13) at com.stacktrace.StackTraceExample.loadUserByUserName(UserService.java:9) at com.stacktrace.StackTraceExample.loadUserByUserName(UserController.java:127) STACK TRACE for Microservice
37. @catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 -- Stack trace only has visibility of the current microservice
38. @catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 -- All context of where the request started is lost
39. @catherinerod27 | @michaelswinslow Distributed Tracing and the logs that come with it [Dapper] [Zipkin] [Money] [OpenTelemetry] @pauljamescleary
40. @catherinerod27 | @michaelswinslow “Microservices did not introduce the need for distributed tracing, but they certainly increased the need.” – Anthony Orapallo, Engineering Leader
41. @catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 07 { "correlationId": 1, "method":"EligibilityController.evaluateEligibility", "level":"INFO"…} { "correlationId": 1, "method":"EligibilityService.evaluateEligibility", "level":"INFO"…} { "correlationId": 1, "method":"EligibilityService.fetchUser", "level":"INFO"…} { "correlationId": 1, "method":“UserController.fetchUser", "level":"INFO"…} { "correlationId": 1, "method":“UserService.loadUserByUsername", "level":"INFO"…} { "correlationId": 1, "method":“UserRepository.getUser", "level":"INFO"…} … { "correlationId": 1, "method": “UserRepository.getUser" "level": “ERR", "message": "variable username is null", "timestamp": 1385053862.3072, "_user_id": 9001, } correlationId = 1
42. @catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway { "correlationId": 1, "method":"EligibilityController.evaluateEligibility", "level":”INFO"…} { "correlationId": 1, "method":"EligibilityService.evaluateEligibility", "level":”INFO"…} { "correlationId": 1, "method":"EligibilityService.fetchUser", "level":”INFO"…} { "correlationId": 1, "method":“UserController.fetchUser", "level":”INFO"…} { "correlationId": 1, "method":“UserService.loadUserByUsername", "level":”INFO"…} { "correlationId": 1, "method":“UserRepository.getUser", "level":”INFO"…} … { "correlationId": 1, "method": “UserRepository.getUser" "level": “ERR", “message": "variable username is null", "timestamp": 1385053862.3072, "_user_id": 9001, } MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 07 0 200 400 600 800 1000 1200 1400 1600 1 2 3 4 5 6 7 8 9 LOG ENTRIES # OF REQUESTS Log File Sizes Monolith Microservices 4x
43. @catherinerod27 | @michaelswinslow What you did for ONE you now have to do for MANY
44. @catherinerod27 | @michaelswinslow
45. @catherinerod27 | @michaelswinslow
46. @catherinerod27 | @michaelswinslow code review + CI CD to all environments firewalls + observability security scans
47. @catherinerod27 | @michaelswinslow Suggestions: 1. Do not create all of your microservices at once 2. Monitor the usage to find your most utilized endpoints and use that to prioritize your migration 3. Have a template for creating new microservices and USE IT!
48. @catherinerod27 | @michaelswinslow Change Management Team wants their release notes
49. @catherinerod27 | @michaelswinslow
50. @catherinerod27 | @michaelswinslow Version Control and Compatibility can be a nightmare @michaelswinslow
51. @catherinerod27 | @michaelswinslow Release 1.5.7 A RELEASE version now consists of a configuration of many microservices. Each microservice version increments independently from the RELEASE and other microservices.
52. @catherinerod27 | @michaelswinslow “…when you publish a new version of a mobile app, the old version doesn’t just magically go away.” – Mark Dappollone, Senior Android Developer Dappollone, M (2018). Future Proofing Mobile Applications from https://medium.com/datadriveninvestor/future-proofing-mobile-applications-56f37e80cd4c
53. @catherinerod27 | @michaelswinslow Burnout is real
54. @catherinerod27 | @michaelswinslow Note that the World Health Organization now classifies burn-out as an official medical disorder. World Health Organization (2019). QD85 Burn-Out from https://icd.who.int/browse11/l-m/en#/http://id.who.int/icd/entity/129180281
55. @catherinerod27 | @michaelswinslow Recap with DOs and DON’Ts: DO have a reason to move to microservices (Resiliency, Cost-Optimization, etc.) DON'T practice RDD (Resume-Driven Development) DO communicate with all teams about monitoring, telemetry and distributed tracing DON'T count on traditional logging techniques (Stack Trace) DO peel off one microservice at a time (Strangler Application pattern) DON'T refactor everything at once DO understand that all teams are impacted (Operations, Release Mgmt, Security, etc.) DON'T say “we're doing microservices!” #YOLO DO staff appropriately and monitor employee health DON'T ignore the human impact of your decision
56. @catherinerod27 | @michaelswinslow Recap with DOs and DON’Ts: DO encourage DevOps teams to pursue microservices if there are benefits DON'T underestimate the time, effort and money involved
57. @catherinerod27 | @michaelswinslow Michael Winslow Core Applications - Comcast Spaces > Tabs @michaelswinslow /michaelswinslow Catherine Rodriguez Core Applications - Comcast I have a twin sister /catherinerodriguez1 @CatherineRod27 Thank You!
58. @catherinerod27 | @michaelswinslow Helpful Links 1. Netflix Zuul - https://github.com/Netflix/zuul 2. Netflix Eureka - https://github.com/Netflix/eureka 3. Netflix Ribbon - https://github.com/Netflix/ribbon 4. Netflix Hystrix - https://github.com/Netflix/hystrix 5. Netflix Vizsceral - https://github.com/Netflix/vizceral 6. Google Dapper – https://ai.google/research/pubs/pub36356 7. Comcast Money - https://github.com/Comcast/money 8. Twitter Zipkin - https://zipkin.apache.org/ Tom Martin, Splunk – Zipkin and Splunk Mark Dappalone, Comcast – FutureProofing Your Mobile Applications