Operating Microservices

Operating Microservices

Listen closely. Close your eyes. I know you hear it coming. It's the microservices buzz. Well, by now it is more of a quake. Catherine Rodriguez and Michael Winslow have been involved with 2 enterprise level microservices implementations. One was greenfield and the other was a migration from a FAT service. It turns out that the 2 easiest things to do when moving to microservices are to DECIDE to do it and to WRITE THE CODE. If you stick around after that, you better be ready for a ride!

Michael will address considerations like scaling, testing and releasing your microservices.
Catherine will cover logging and traceability.

D2985b36fb510e4b37ec87d7b9ac979b?s=128

Michael Scott Winslow

June 03, 2019
Tweet

Transcript

  1. @catherinerod27 | @michaelswinslow Michael Winslow Core Applications - Comcast Spaces

    > Tabs Operating Microservices @michaelswinslow /michaelswinslow Catherine Rodriguez Core Applications - Comcast I have a twin sister /catherinerodriguez1 @CatherineRod27
  2. @catherinerod27 | @michaelswinslow Some background: 1. Michael was on a

    DEV team that built web services using Spring MVC 2. Catherine was on the Splunk Platform team (logging, observability, etc.) 3. At some point, the team and leadership decided to pursue Microservices 4. Things worked out well, but it was not always sunshine Now you have a choice: • Hear about the lessons your team learned! (red) • Hear about how great your team is at making microservices! (blue)
  3. @catherinerod27 | @michaelswinslow @catherinerod27 | @michaelswinslow

  4. @catherinerod27 | @michaelswinslow @catherinerod27 | @michaelswinslow

  5. @catherinerod27 | @michaelswinslow So, Microservices…

  6. @catherinerod27 | @michaelswinslow Is There a Need To Move to

    Microservices? Are you Netflix? No no Start
  7. @catherinerod27 | @michaelswinslow This is a MONOLITH Service Repository (DAO)

    Controller
  8. @catherinerod27 | @michaelswinslow This is a MONOLITH Service Repository (DAO)

    eligibility user device eligibility user device eligibility user device Controller eligibility user device
  9. @catherinerod27 | @michaelswinslow This is a MICROSERVICE Controller device Service

    Repository
  10. @catherinerod27 | @michaelswinslow Service Repository This is a MICROSERVICE Service

    Repository Service Repository Controller device Controller user Controller eligibility
  11. @catherinerod27 | @michaelswinslow Service Repository This is a MICROSERVICE Service

    Repository Service Repository Controller device Controller user Controller eligibility Note: Large organizations may dedicate a small team to each microservice. A team that has a great deal of developers who can work independently may benefit from this model.
  12. @catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service

    Repository Controller eligibility Service Repository API Gateway
  13. @catherinerod27 | @michaelswinslow Scale / Service Discovery

  14. @catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service

    Repository Controller eligibility Service Repository API Gateway “Oh no! For some reason requests to the ELIGIBILITY API have gone way up! At least 3x the traffic of any other API!”
  15. @catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service

    Repository Controller eligibility Service Repository API Gateway Controller eligibility Service Repository Controller eligibility Service Repository Discovery
  16. @catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service

    Repository Controller eligibility Service Repository API Gateway Controller eligibility Service Repository Controller eligibility Service Repository Discovery 1 2 3
  17. @catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service

    Repository Controller eligibility Service Repository API Gateway Controller eligibility Service Repository Controller eligibility Service Repository Discovery Note: Elasticity is one of the major reasons to move to microservices. If your traffic tends to increase and decrease dramatically, you may find value. Especially in the cloud.
  18. @catherinerod27 | @michaelswinslow Fault Tolerance / Circuit Breaker

  19. @catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service

    Repository Controller eligibility Service Repository API Gateway Controller eligibility Service Repository Controller eligibility Service Repository Discovery
  20. @catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service

    Repository Controller eligibility Service Repository API Gateway Controller eligibility Service Repository Controller eligibility Service Repository Discovery Retry 3/3
  21. @catherinerod27 | @michaelswinslow HYSTRIX Controller device Service Repository Controller user

    Service Repository Controller eligibility Service Repository API Gateway Controller eligibility Service Repository Discovery Retry 3/3
  22. @catherinerod27 | @michaelswinslow Summary: 1. Monolith vs Microservice 2. API

    Gateway - Zuul 3. Service Discovery - Eureka 4. Software Load Balancer – Ribbon 5. Circuit Breaker - Hystrix Now you have a choice: • That was amazing! Now tell us what your team learned! (red) • That was amazing! Now tell us what your team learned! (blue)
  23. @catherinerod27 | @michaelswinslow “Being able to scale and distribute your

    application resources is certainly alluring. However, this doesn't come for free. Complexity begets complexity. Soon tools are needed to manage the complexity. Then tools are needed to manage the tools.” – Ryan Emerle, Principal Engineer
  24. @catherinerod27 | @michaelswinslow Sample: We need to call our API

    to determine if a particular USER is ELIGIBLE to purchase a DEVICE
  25. @catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility

    user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog -- -- -- -- process boundary syslog entry
  26. @catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility

    user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 -- -- --
  27. @catherinerod27 | @michaelswinslow MONOLITH MICROSERVICES syslog applog syslog applog 03

    -- 08 -- Note: Increased number of process boundaries also creates more surface area for security attacks during runtime.
  28. @catherinerod27 | @michaelswinslow So What? Microservices inflate the System Logs

    a little (and security risk). It’s totally worth it!
  29. @catherinerod27 | @michaelswinslow Maybe… but we’re not finished yet.

  30. @catherinerod27 | @michaelswinslow Execution Errors easy to track?

  31. @catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility

    user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 -- 08 --
  32. @catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility

    user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 -- 08 -- java.lang.NullPointerException: variable username is null at com.stacktrace.StackTraceExample.getUser(UserRepository.java:13) at com.stacktrace.StackTraceExample.loadUserByUserName(UserService.java:9) at com.stacktrace.StackTraceExample.fetchUser(EligibilityService.java:1012) at com.stacktrace.StackTraceExample.evaluateEligibility(EligibilityService.java:5) at com.stacktrace.StackTraceExample.evaluateEligibility(EligibilityController.java:27) STACK TRACE for Monolith
  33. @catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility

    user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 -- 08 -- java.lang.NullPointerException: variable username is null at com.stacktrace.StackTraceExample.getUser(UserRepository.java:13) at com.stacktrace.StackTraceExample.loadUserByUserName(UserService.java:9) at com.stacktrace.StackTraceExample.fetchUser(EligibilityService.java:1012) at com.stacktrace.StackTraceExample.evaluateEligibility(EligibilityService.java:5) at com.stacktrace.StackTraceExample.evaluateEligibility(EligibilityController.java:27) STACK TRACE for Monolith
  34. @catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility

    user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 -- Stack trace has visibility of everything in the Monolith
  35. @catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility

    user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 --
  36. @catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility

    user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 -- java.lang.NullPointerException: variable username is null at com.stacktrace.StackTraceExample.getUser(UserRepository.java:13) at com.stacktrace.StackTraceExample.loadUserByUserName(UserService.java:9) at com.stacktrace.StackTraceExample.loadUserByUserName(UserController.java:127) STACK TRACE for Microservice
  37. @catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility

    user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 -- Stack trace only has visibility of the current microservice
  38. @catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility

    user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 -- All context of where the request started is lost
  39. @catherinerod27 | @michaelswinslow Distributed Tracing and the logs that come

    with it [Dapper] [Zipkin] [Money] [OpenTelemetry] @pauljamescleary
  40. @catherinerod27 | @michaelswinslow “Microservices did not introduce the need for

    distributed tracing, but they certainly increased the need.” – Anthony Orapallo, Engineering Leader
  41. @catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service

    Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 07 { "correlationId": 1, "method":"EligibilityController.evaluateEligibility", "level":"INFO"…} { "correlationId": 1, "method":"EligibilityService.evaluateEligibility", "level":"INFO"…} { "correlationId": 1, "method":"EligibilityService.fetchUser", "level":"INFO"…} { "correlationId": 1, "method":“UserController.fetchUser", "level":"INFO"…} { "correlationId": 1, "method":“UserService.loadUserByUsername", "level":"INFO"…} { "correlationId": 1, "method":“UserRepository.getUser", "level":"INFO"…} … { "correlationId": 1, "method": “UserRepository.getUser" "level": “ERR", "message": "variable username is null", "timestamp": 1385053862.3072, "_user_id": 9001, } correlationId = 1
  42. @catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service

    Repository Controller eligibility Service Repository API Gateway { "correlationId": 1, "method":"EligibilityController.evaluateEligibility", "level":”INFO"…} { "correlationId": 1, "method":"EligibilityService.evaluateEligibility", "level":”INFO"…} { "correlationId": 1, "method":"EligibilityService.fetchUser", "level":”INFO"…} { "correlationId": 1, "method":“UserController.fetchUser", "level":”INFO"…} { "correlationId": 1, "method":“UserService.loadUserByUsername", "level":”INFO"…} { "correlationId": 1, "method":“UserRepository.getUser", "level":”INFO"…} … { "correlationId": 1, "method": “UserRepository.getUser" "level": “ERR", “message": "variable username is null", "timestamp": 1385053862.3072, "_user_id": 9001, } MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 07 0 200 400 600 800 1000 1200 1400 1600 1 2 3 4 5 6 7 8 9 LOG ENTRIES # OF REQUESTS Log File Sizes Monolith Microservices 4x
  43. @catherinerod27 | @michaelswinslow What you did for ONE you now

    have to do for MANY
  44. @catherinerod27 | @michaelswinslow

  45. @catherinerod27 | @michaelswinslow

  46. @catherinerod27 | @michaelswinslow code review + CI CD to all

    environments firewalls + observability security scans
  47. @catherinerod27 | @michaelswinslow Suggestions: 1. Do not create all of

    your microservices at once 2. Monitor the usage to find your most utilized endpoints and use that to prioritize your migration 3. Have a template for creating new microservices and USE IT!
  48. @catherinerod27 | @michaelswinslow Change Management Team wants their release notes

  49. @catherinerod27 | @michaelswinslow

  50. @catherinerod27 | @michaelswinslow Version Control and Compatibility can be a

    nightmare @michaelswinslow
  51. @catherinerod27 | @michaelswinslow Release 1.5.7 A RELEASE version now consists

    of a configuration of many microservices. Each microservice version increments independently from the RELEASE and other microservices.
  52. @catherinerod27 | @michaelswinslow “…when you publish a new version of

    a mobile app, the old version doesn’t just magically go away.” – Mark Dappollone, Senior Android Developer Dappollone, M (2018). Future Proofing Mobile Applications from https://medium.com/datadriveninvestor/future-proofing-mobile-applications-56f37e80cd4c
  53. @catherinerod27 | @michaelswinslow Burnout is real

  54. @catherinerod27 | @michaelswinslow Note that the World Health Organization now

    classifies burn-out as an official medical disorder. World Health Organization (2019). QD85 Burn-Out from https://icd.who.int/browse11/l-m/en#/http://id.who.int/icd/entity/129180281
  55. @catherinerod27 | @michaelswinslow Recap with DOs and DON’Ts: DO have

    a reason to move to microservices (Resiliency, Cost-Optimization, etc.) DON'T practice RDD (Resume-Driven Development) DO communicate with all teams about monitoring, telemetry and distributed tracing DON'T count on traditional logging techniques (Stack Trace) DO peel off one microservice at a time (Strangler Application pattern) DON'T refactor everything at once DO understand that all teams are impacted (Operations, Release Mgmt, Security, etc.) DON'T say “we're doing microservices!” #YOLO DO staff appropriately and monitor employee health DON'T ignore the human impact of your decision
  56. @catherinerod27 | @michaelswinslow Recap with DOs and DON’Ts: DO encourage

    DevOps teams to pursue microservices if there are benefits DON'T underestimate the time, effort and money involved
  57. @catherinerod27 | @michaelswinslow Michael Winslow Core Applications - Comcast Spaces

    > Tabs @michaelswinslow /michaelswinslow Catherine Rodriguez Core Applications - Comcast I have a twin sister /catherinerodriguez1 @CatherineRod27 Thank You!
  58. @catherinerod27 | @michaelswinslow Helpful Links 1. Netflix Zuul - https://github.com/Netflix/zuul

    2. Netflix Eureka - https://github.com/Netflix/eureka 3. Netflix Ribbon - https://github.com/Netflix/ribbon 4. Netflix Hystrix - https://github.com/Netflix/hystrix 5. Netflix Vizsceral - https://github.com/Netflix/vizceral 6. Google Dapper – https://ai.google/research/pubs/pub36356 7. Comcast Money - https://github.com/Comcast/money 8. Twitter Zipkin - https://zipkin.apache.org/ Tom Martin, Splunk – Zipkin and Splunk Mark Dappalone, Comcast – FutureProofing Your Mobile Applications