Operating Microservices

Slide 1

Slide 1 text

@catherinerod27 | @michaelswinslow Michael Winslow Core Applications - Comcast Spaces > Tabs Operating Microservices @michaelswinslow /michaelswinslow Catherine Rodriguez Core Applications - Comcast I have a twin sister /catherinerodriguez1 @CatherineRod27

Slide 2

Slide 2 text

@catherinerod27 | @michaelswinslow Some background: 1. Michael was on a DEV team that built web services using Spring MVC 2. Catherine was on the Splunk Platform team (logging, observability, etc.) 3. At some point, the team and leadership decided to pursue Microservices 4. Things worked out well, but it was not always sunshine Now you have a choice: • Hear about the lessons your team learned! (red) • Hear about how great your team is at making microservices! (blue)

Slide 3

Slide 3 text

@catherinerod27 | @michaelswinslow @catherinerod27 | @michaelswinslow

Slide 4

Slide 4 text

@catherinerod27 | @michaelswinslow @catherinerod27 | @michaelswinslow

Slide 5

Slide 5 text

@catherinerod27 | @michaelswinslow So, Microservices…

Slide 6

Slide 6 text

@catherinerod27 | @michaelswinslow Is There a Need To Move to Microservices? Are you Netflix? No no Start

Slide 7

Slide 7 text

@catherinerod27 | @michaelswinslow This is a MONOLITH Service Repository (DAO) Controller

Slide 8

Slide 8 text

@catherinerod27 | @michaelswinslow This is a MONOLITH Service Repository (DAO) eligibility user device eligibility user device eligibility user device Controller eligibility user device

Slide 9

Slide 9 text

@catherinerod27 | @michaelswinslow This is a MICROSERVICE Controller device Service Repository

Slide 10

Slide 10 text

@catherinerod27 | @michaelswinslow Service Repository This is a MICROSERVICE Service Repository Service Repository Controller device Controller user Controller eligibility

Slide 11

Slide 11 text

@catherinerod27 | @michaelswinslow Service Repository This is a MICROSERVICE Service Repository Service Repository Controller device Controller user Controller eligibility Note: Large organizations may dedicate a small team to each microservice. A team that has a great deal of developers who can work independently may benefit from this model.

Slide 12

Slide 12 text

@catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway

Slide 13

Slide 13 text

@catherinerod27 | @michaelswinslow Scale / Service Discovery

Slide 14

Slide 14 text

@catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway “Oh no! For some reason requests to the ELIGIBILITY API have gone way up! At least 3x the traffic of any other API!”

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

@catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway Controller eligibility Service Repository Controller eligibility Service Repository Discovery Note: Elasticity is one of the major reasons to move to microservices. If your traffic tends to increase and decrease dramatically, you may find value. Especially in the cloud.

Slide 18

Slide 18 text

@catherinerod27 | @michaelswinslow Fault Tolerance / Circuit Breaker

Slide 19

Slide 19 text

Slide 20

Slide 20 text

Slide 21

Slide 21 text

@catherinerod27 | @michaelswinslow HYSTRIX Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway Controller eligibility Service Repository Discovery Retry 3/3

Slide 22

Slide 22 text

@catherinerod27 | @michaelswinslow Summary: 1. Monolith vs Microservice 2. API Gateway - Zuul 3. Service Discovery - Eureka 4. Software Load Balancer – Ribbon 5. Circuit Breaker - Hystrix Now you have a choice: • That was amazing! Now tell us what your team learned! (red) • That was amazing! Now tell us what your team learned! (blue)

Slide 23

Slide 23 text

@catherinerod27 | @michaelswinslow “Being able to scale and distribute your application resources is certainly alluring. However, this doesn't come for free. Complexity begets complexity. Soon tools are needed to manage the complexity. Then tools are needed to manage the tools.” – Ryan Emerle, Principal Engineer

Slide 24

Slide 24 text

@catherinerod27 | @michaelswinslow Sample: We need to call our API to determine if a particular USER is ELIGIBLE to purchase a DEVICE

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Slide 27

Slide 27 text

@catherinerod27 | @michaelswinslow MONOLITH MICROSERVICES syslog applog syslog applog 03 -- 08 -- Note: Increased number of process boundaries also creates more surface area for security attacks during runtime.

Slide 28

Slide 28 text

@catherinerod27 | @michaelswinslow So What? Microservices inflate the System Logs a little (and security risk). It’s totally worth it!

Slide 29

Slide 29 text

@catherinerod27 | @michaelswinslow Maybe… but we’re not finished yet.

Slide 30

Slide 30 text

@catherinerod27 | @michaelswinslow Execution Errors easy to track?

Slide 31

Slide 31 text

Slide 32

Slide 32 text

@catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 -- 08 -- java.lang.NullPointerException: variable username is null at com.stacktrace.StackTraceExample.getUser(UserRepository.java:13) at com.stacktrace.StackTraceExample.loadUserByUserName(UserService.java:9) at com.stacktrace.StackTraceExample.fetchUser(EligibilityService.java:1012) at com.stacktrace.StackTraceExample.evaluateEligibility(EligibilityService.java:5) at com.stacktrace.StackTraceExample.evaluateEligibility(EligibilityController.java:27) STACK TRACE for Monolith

Slide 33

Slide 33 text

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Slide 36

Slide 36 text

@catherinerod27 | @michaelswinslow Service Repository (DAO) eligibility user device eligibility user device Controller eligibility user device Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 -- java.lang.NullPointerException: variable username is null at com.stacktrace.StackTraceExample.getUser(UserRepository.java:13) at com.stacktrace.StackTraceExample.loadUserByUserName(UserService.java:9) at com.stacktrace.StackTraceExample.loadUserByUserName(UserController.java:127) STACK TRACE for Microservice

Slide 37

Slide 37 text

Slide 38

Slide 38 text

Slide 39

Slide 39 text

@catherinerod27 | @michaelswinslow Distributed Tracing and the logs that come with it [Dapper] [Zipkin] [Money] [OpenTelemetry] @pauljamescleary

Slide 40

Slide 40 text

@catherinerod27 | @michaelswinslow “Microservices did not introduce the need for distributed tracing, but they certainly increased the need.” – Anthony Orapallo, Engineering Leader

Slide 41

Slide 41 text

@catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 07 { "correlationId": 1, "method":"EligibilityController.evaluateEligibility", "level":"INFO"…} { "correlationId": 1, "method":"EligibilityService.evaluateEligibility", "level":"INFO"…} { "correlationId": 1, "method":"EligibilityService.fetchUser", "level":"INFO"…} { "correlationId": 1, "method":“UserController.fetchUser", "level":"INFO"…} { "correlationId": 1, "method":“UserService.loadUserByUsername", "level":"INFO"…} { "correlationId": 1, "method":“UserRepository.getUser", "level":"INFO"…} … { "correlationId": 1, "method": “UserRepository.getUser" "level": “ERR", "message": "variable username is null", "timestamp": 1385053862.3072, "_user_id": 9001, } correlationId = 1

Slide 42

Slide 42 text

@catherinerod27 | @michaelswinslow Controller device Service Repository Controller user Service Repository Controller eligibility Service Repository API Gateway { "correlationId": 1, "method":"EligibilityController.evaluateEligibility", "level":”INFO"…} { "correlationId": 1, "method":"EligibilityService.evaluateEligibility", "level":”INFO"…} { "correlationId": 1, "method":"EligibilityService.fetchUser", "level":”INFO"…} { "correlationId": 1, "method":“UserController.fetchUser", "level":”INFO"…} { "correlationId": 1, "method":“UserService.loadUserByUsername", "level":”INFO"…} { "correlationId": 1, "method":“UserRepository.getUser", "level":”INFO"…} … { "correlationId": 1, "method": “UserRepository.getUser" "level": “ERR", “message": "variable username is null", "timestamp": 1385053862.3072, "_user_id": 9001, } MONOLITH MICROSERVICES syslog applog syslog applog 03 01 08 07 0 200 400 600 800 1000 1200 1400 1600 1 2 3 4 5 6 7 8 9 LOG ENTRIES # OF REQUESTS Log File Sizes Monolith Microservices 4x

Slide 43

Slide 43 text

@catherinerod27 | @michaelswinslow What you did for ONE you now have to do for MANY

Slide 44

Slide 44 text

@catherinerod27 | @michaelswinslow

Slide 45

Slide 45 text

@catherinerod27 | @michaelswinslow

Slide 46

Slide 46 text

@catherinerod27 | @michaelswinslow code review + CI CD to all environments firewalls + observability security scans

Slide 47

Slide 47 text

@catherinerod27 | @michaelswinslow Suggestions: 1. Do not create all of your microservices at once 2. Monitor the usage to find your most utilized endpoints and use that to prioritize your migration 3. Have a template for creating new microservices and USE IT!

Slide 48

Slide 48 text

@catherinerod27 | @michaelswinslow Change Management Team wants their release notes

Slide 49

Slide 49 text

@catherinerod27 | @michaelswinslow

Slide 50

Slide 50 text

@catherinerod27 | @michaelswinslow Version Control and Compatibility can be a nightmare @michaelswinslow

Slide 51

Slide 51 text

@catherinerod27 | @michaelswinslow Release 1.5.7 A RELEASE version now consists of a configuration of many microservices. Each microservice version increments independently from the RELEASE and other microservices.

Slide 52

Slide 52 text

@catherinerod27 | @michaelswinslow “…when you publish a new version of a mobile app, the old version doesn’t just magically go away.” – Mark Dappollone, Senior Android Developer Dappollone, M (2018). Future Proofing Mobile Applications from https://medium.com/datadriveninvestor/future-proofing-mobile-applications-56f37e80cd4c

Slide 53

Slide 53 text

@catherinerod27 | @michaelswinslow Burnout is real

Slide 54

Slide 54 text

@catherinerod27 | @michaelswinslow Note that the World Health Organization now classifies burn-out as an official medical disorder. World Health Organization (2019). QD85 Burn-Out from https://icd.who.int/browse11/l-m/en#/http://id.who.int/icd/entity/129180281

Slide 55

Slide 55 text

@catherinerod27 | @michaelswinslow Recap with DOs and DON’Ts: DO have a reason to move to microservices (Resiliency, Cost-Optimization, etc.) DON'T practice RDD (Resume-Driven Development) DO communicate with all teams about monitoring, telemetry and distributed tracing DON'T count on traditional logging techniques (Stack Trace) DO peel off one microservice at a time (Strangler Application pattern) DON'T refactor everything at once DO understand that all teams are impacted (Operations, Release Mgmt, Security, etc.) DON'T say “we're doing microservices!” #YOLO DO staff appropriately and monitor employee health DON'T ignore the human impact of your decision

Slide 56

Slide 56 text

@catherinerod27 | @michaelswinslow Recap with DOs and DON’Ts: DO encourage DevOps teams to pursue microservices if there are benefits DON'T underestimate the time, effort and money involved

Slide 57

Slide 57 text

@catherinerod27 | @michaelswinslow Michael Winslow Core Applications - Comcast Spaces > Tabs @michaelswinslow /michaelswinslow Catherine Rodriguez Core Applications - Comcast I have a twin sister /catherinerodriguez1 @CatherineRod27 Thank You!

Slide 58

Slide 58 text

@catherinerod27 | @michaelswinslow Helpful Links 1. Netflix Zuul - https://github.com/Netflix/zuul 2. Netflix Eureka - https://github.com/Netflix/eureka 3. Netflix Ribbon - https://github.com/Netflix/ribbon 4. Netflix Hystrix - https://github.com/Netflix/hystrix 5. Netflix Vizsceral - https://github.com/Netflix/vizceral 6. Google Dapper – https://ai.google/research/pubs/pub36356 7. Comcast Money - https://github.com/Comcast/money 8. Twitter Zipkin - https://zipkin.apache.org/ Tom Martin, Splunk – Zipkin and Splunk Mark Dappalone, Comcast – FutureProofing Your Mobile Applications