Mesh, not just for the club (anymore)

Presented at Maritime DevCon 2019 https://maritimedevcon.ca/

A mesh is defined as an “interlaced structure” and this exactly describes what we are building when we create distributed systems. The pieces talk to and depend on one another. Managing, monitoring, and controlling the ether between our services is a daunting task and one that can lead to failure scenarios if mismanaged. In this talk, Josh Comer will explore how adding a Service Mesh to our systems can not only make them more resilient but flexible and transparent. With first-hand stories of distributed failure, Josh will discuss how a mesh could have avoided these scenarios. Hopefully, at the end, you will agree that if you could only take one technology to a distributed desert island, it would be a Service Mesh.

Transcript

1. #MaritimeDevCon - 08/06/19 - @jjcomer not just for the club ME
2. #MaritimeDevCon - 08/06/19 - @jjcomer First some terms
3. #MaritimeDevCon - 08/06/19 - @jjcomer What is a distributed system A grouping of related, yet independently resourced, processes. Ok… but why? • Resilience • Performance • Economics • … and more
4. #MaritimeDevCon - 08/06/19 - @jjcomer Microservices Strict(ish) set of definitions NO SHARING! 12 Factor Apps are better: https://12factor.net/ Fun fact: “If your services must be deployed in a certain order, you have a monolith”
5. #MaritimeDevCon - 08/06/19 - @jjcomer What are the problems we face Coordination, coordination, coordination Oh and coordination Deployments without downtime Resilience Configuration Coordination Observability
6. #MaritimeDevCon - 08/06/19 - @jjcomer Side Pitch #1
7. #MaritimeDevCon - 08/06/19 - @jjcomer Agnostic Configuration
8. #MaritimeDevCon - 08/06/19 - @jjcomer Agnostic configuration... the dream Create build artifacts that can be used anywhere The environment is the environment! • Pull vs Push (prometheus) • Docker logging drivers (log to stdout) • Environment aware URLs ◦ http://serviceb vs http://env1.us.serviceb • Constant secret keys ◦ Requesting “DB_PASSWORD” will return the correct password for context ◦ Limits blast radius
9. #MaritimeDevCon - 08/06/19 - @jjcomer And now the definition
10. #MaritimeDevCon - 08/06/19 - @jjcomer Mesh definition From the Dictionary: Noun: a weblike pattern or construction Verb: to coordinate closely Also a fabric which makes great club wear
11. #MaritimeDevCon - 08/06/19 - @jjcomer Service mesh definition “A service mesh is a configurable, low‑latency infrastructure layer designed to handle a high volume of network‑based interprocess communication among application infrastructure services using application programming interfaces (APIs).” -nginx site
12. #MaritimeDevCon - 08/06/19 - @jjcomer Let’s take a look
13. #MaritimeDevCon - 08/06/19 - @jjcomer Data Plane Control Plane Service Service Service Service Service Discovery Telemetry
14. #MaritimeDevCon - 08/06/19 - @jjcomer Service A Instance Service C Instance Service B Instance M M M Mesh Control Plane
15. #MaritimeDevCon - 08/06/19 - @jjcomer Service Zoom out a bit... Load Balancer Instance 1 Instance 2 Instance 3 Requestor
16. #MaritimeDevCon - 08/06/19 - @jjcomer Service B Zoom in a bit... Mesh node (Service A) Instance 1 Instance 2 Instance 3 Service A Instance M M M
17. #MaritimeDevCon - 08/06/19 - @jjcomer Case Studies
18. #MaritimeDevCon - 08/06/19 - @jjcomer Case study #1 -- Architecture Diagrams You are trying to put together on boarding materials... but there are NO DIAGRAMS. The platform is too large for any one person to know all service relationships and you need this information to be accurate (forever).
19. #MaritimeDevCon - 08/06/19 - @jjcomer Discoverability The mesh controls all communication It knows exactly who talks to whom Can glean extra information: latency and frequency Only drawback is that the communication must happen once
20. #MaritimeDevCon - 08/06/19 - @jjcomer Case study #2 -- retry failure The worst outage ever Microservice API → 4 layers deep with 5*retries at each level Load overwhelmed the second layer which lead to 5^3 retries per request Ended up needing an entire restart of the platform to flush the queues Client Service A Service B Service C
21. #MaritimeDevCon - 08/06/19 - @jjcomer Resilience All communication goes through the mesh Control plane manages distributed circuit breakers Sidebar: Request time should be started at the edge of the platform. Each level down just has a slice of the total request time Client Service A Service B Service C 30s 10s 3s
22. #MaritimeDevCon - 08/06/19 - @jjcomer Case study #2.5 -- retry resolution As part of the outage a defect was found in the retry logic, which prevented us from disabling retries This meant a change was required to resolve the defect A lot of disruption to teams and a complete redeployment
23. #MaritimeDevCon - 08/06/19 - @jjcomer Externalization of dependencies Services don’t need to implement retry logic Only need to write logic for error states Ability to dramatically change networking implementation without changing service code
24. #MaritimeDevCon - 08/06/19 - @jjcomer Case study #3 -- Security The security team finally noticed the platform you have been building. They would like to know why your data plane is in the clear… Well now you need to encrypt all traffic. This means securely setting up an internal CA and issuing certs to each service instance (with responsible expirations). Services need to change their code to install the CA, use TLS, and verify incoming connection. Oh and do a complete redeploy. Not to mention bootstrapping into TLS mode.
25. #MaritimeDevCon - 08/06/19 - @jjcomer Security All communication goes through the service mesh 0 Trust Networking! Mesh supports mutual TLS (including authorization) between nodes No service changes required! Service A Instance Service B Instance M M Secure (via Mesh) Clear (localhost) Clear (localhost)
26. #MaritimeDevCon - 08/06/19 - @jjcomer Case study #4 -- new feature roll out I want to release a new feature to my customers in a safe manner. There can’t be any downtime and if things go wrong I would like a small blast radius. Feature flags are ok, but they can create spaghetti code and need to be removed after the fact (intentional tech debt).
27. #MaritimeDevCon - 08/06/19 - @jjcomer Traffic distribution • All traffic goes through the service mesh! • Provides ability to shape traffic ◦ Blue/Green deploys ◦ Percentage based routing ◦ Tenant based routing ◦ Ghost traffic • If things go wrong, only a small fraction of requests were affected • No service changes required to support new deployment models
28. #MaritimeDevCon - 08/06/19 - @jjcomer Service A Instance Service B’ Instance Service B Instance M M M All requests to Service B are mirrored to B’ Responses from B’ and automatically discarded
29. #MaritimeDevCon - 08/06/19 - @jjcomer There must be some downsides...
30. #MaritimeDevCon - 08/06/19 - @jjcomer Latency A service mesh will add latency No way to avoid this Any extra hops will incur latency Are the benefits worth it?
31. #MaritimeDevCon - 08/06/19 - @jjcomer Resource cost A proxy is required for every instance of a service While efficient, they still require memory, CPU, and bandwidth Mesh CPU Memory Istio 0.6 vCPU/1000r/s 50M/proxy Linkerd 2.0 0.2-1.5 vCPU 128M-2G
32. #MaritimeDevCon - 08/06/19 - @jjcomer Additional maintenance • Requires integration with multiple systems ◦ Service discovery ◦ Orchestration layer • Easier to maintain than embedded network code
33. #MaritimeDevCon - 08/06/19 - @jjcomer Where to start
34. #MaritimeDevCon - 08/06/19 - @jjcomer Pick a Service Mesh
35. #MaritimeDevCon - 08/06/19 - @jjcomer Tangent #1
36. #MaritimeDevCon - 08/06/19 - @jjcomer We love a good battle
37. #MaritimeDevCon - 08/06/19 - @jjcomer OS?
38. #MaritimeDevCon - 08/06/19 - @jjcomer Editors
39. #MaritimeDevCon - 08/06/19 - @jjcomer Editors
40. #MaritimeDevCon - 08/06/19 - @jjcomer Orchestration
41. #MaritimeDevCon - 08/06/19 - @jjcomer There can be only one The battle for orchestration is over CNCF is here to save the day K8S won! Others will continue to exist due to prior adoption Most cloud native solutions are targeting k8s first or only
42. #MaritimeDevCon - 08/06/19 - @jjcomer Which Service Mesh?
43. #MaritimeDevCon - 08/06/19 - @jjcomer Options available
44. #MaritimeDevCon - 08/06/19 - @jjcomer Mesh Darwinism Allow the best to survive… Unlike orchestration, many can co-exist Landscape has already churned: Istio → Linkerd 2.0 Investment is high to sink into particular adoption, how can we allow easy churn?
45. #MaritimeDevCon - 08/06/19 - @jjcomer Open Standards!
46. #MaritimeDevCon - 08/06/19 - @jjcomer Service Mesh Interface SMI
47. #MaritimeDevCon - 08/06/19 - @jjcomer SMI for the win Hot off the Press! Microsoft, Linkerd, HashiCorp, Solo.io, Kinvolk, and Weaveworks Removes the lock-in Allows adoption to be a one time cost to service teams Data and Control Planes can be changed without changing service code
48. #MaritimeDevCon - 08/06/19 - @jjcomer SMI -- What does it cover? Traffic Policy -- Identity and Security Traffic Telemetry -- Key metrics Traffic Management -- Controlling distribution https://smi-spec.io/ * * Only for k8s
49. #MaritimeDevCon - 08/06/19 - @jjcomer Side Pitch #2
50. #MaritimeDevCon - 08/06/19 - @jjcomer Telemetry OpenTelemetry -- https://opentelemetry.io/ Merger of OpenTracing and OpenCensus Finally! Covers Tracing and Metrics -- Long term plan to include logging One library to rule all observability! * Not k8s specific
51. #MaritimeDevCon - 08/06/19 - @jjcomer In summary Service Mesh add an abstraction between your services and how they communicate Consistent telemetry increases discoverability and observability Distributed error handling can allow the system to breath and recover 0 Trust Networking helps secure your infrastructure OPEN STANDARDS!!!!!
52. #MaritimeDevCon - 08/06/19 - @jjcomer whoami Josh Comer Software Architect -- SRE @Cvent (we are hiring ) Here in Fredericton!