$30 off During Our Annual Pro Sale. View Details »

Tales of deploying Istio Ingress

Tales of deploying Istio Ingress

This talk presents Workday’s journey towards deploying Istio Ingress to our Public Cloud environments. As we transferred our services from our legacy ingress to our new Istio ingress solution, a number of platform and application-layer issues surfaced. This talk presents how browser policy, HSTS, cookie stickiness, and headers can break applications, how we debugged those issues, and how we resolved them. Attendees can expect to learn some common and less common pitfalls of updating platform and infrastructure, the tools and techniques to triage them, and how they can impact the underlying applications.

Pauline Lallinec

May 11, 2021
Tweet

More Decks by Pauline Lallinec

Other Decks in Programming

Transcript

  1. None
  2. Tales of deploying Istio Ingress --- Continuous Lifecycle London 2021

    Pauline Lallinec
  3. Tales of deploying Istio Ingress --- 1. Intro 2. What

    is Istio? What is Envoy? 3. Istio resources overview 4. Workday infrastructure change overview 5. Rollout plans 6. Lessons learnt
  4. Data centers in Asia, Canada, Europe, USA 95.14% of transactions

    had a response time of less than 1 second 195 billion transactions FY2020 Service uptime > 99.98% Workday community = 45 million workers
  5. Software Engineer - DevOps Non-stop karaoke machine @plallin Workday +

    k8s platform = Scylla Public + Private cloud (Asia, NA, Europe) 5 teams, 2 continents
  6. Workday + Public Infrastructure = Pi Aka. Infrastructure Public Cloud

    2 teams, 2 continents Software Engineer - DevOps Non-stop karaoke machine @plallin
  7. Tales of deploying Istio Ingress --- 1. Intro 2. What

    is Istio? What is Envoy? 3. Istio resources overview 4. Workday infrastructure change overview 5. Rollout plans 6. Lessons learnt
  8. What is Istio? Istio is an open-source platform-independent service mesh

    that provides traffic management, policy enforcement, and telemetry collection. • Developed by Google + IBM + Lyft • Uses the Envoy proxy • It lets you manage your ingress resources as a set of Kubernetes resources • It runs inside the Kubernetes cluster • Istio ingress =/= Istio Mesh ◦ Istio Ingress: manages external traffic to pods ◦ Istio Mesh: manages pod-to-pod traffic
  9. What is Envoy? Envoy is a high-performance proxy designed for

    cloud-native applications • Originally developed by Lyft in C++ • Open source • Cloud-agnostic • Manage all inbound and outbound traffic • In Istio, Envoy proxies are deployed as sidecar containers
  10. What is Envoy? Envoy Features: • Dynamic service discovery •

    Load balancing • TLS termination • HTTP/2 and gRPC proxies • Circuit breakers • Health checks • Staged rollouts with %-based traffic split • Fault injection • & more
  11. None
  12. Tales of deploying Istio Ingress --- 1. Intro 2. What

    is Istio? What is Envoy? 3. Istio resources overview 4. Workday infrastructure change overview 5. Rollout plans 6. Lessons learnt
  13. Istio Resources • high level routing rules are configured via

    custom resources • We will focus on 3 of them: • Istio gateway Describes a load balancer operating at the edge of the mesh receiving incoming or outgoing HTTP/TCP connections • Virtual services A set of traffic routing rules to apply when a host is addressed • Destination rules Defines policies that apply to traffic after routing has occurred (such as load balancing, connection pool size, etc)
  14. Istio Resources • Istio is logically split into a control

    plane and a data plane • Istiod: the control plane ◦ Use the CRDs to convert high level routing rules into Envoy-specific configurations ◦ Push Envoy configuration to Envoy proxy sidecars ◦ Discovery + certificate management (out of scope for this talk) • Envoy sidecars: the data plane ◦ Receive and routes traffic as per the configuration received from the control plane • Istio ingress gateway (part of the data plane) ◦ runs an Envoy proxy sidecar ◦ Routes ingress traffic according to the ingress configuration
  15. Istio Resources

  16. Istio Resources: gateway apiVersion: networking.istio.io/v1alpha3 kind: Gateway metadata: name: my-gateway

    namespace: some-config-namespace spec: selector: app: my-gateway-controller servers: - port: number: 443 name: https-443 protocol: HTTPS hosts: - uk.bookinfo.com - eu.bookinfo.com tls: mode: SIMPLE # enables HTTPS on this port serverCertificate : /etc/certs/servercert.pem privateKey: /etc/certs/privatekey.pem
  17. Istio Resources: virtual service apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name:

    reviews-route spec: gateway: - my-gateway hosts: - uk.bookinfo.com http: - name: "booking-routes" match: - uri: prefix: "/booking" - uri: prefix: "/user-profile" route: - destination: host: bookings.my-namespace.svc.cluster.local port-number: 8080
  18. Istio Resources: destination rule apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name:

    bookinfo spec: host: bookings.my-namespace.svc.cluster.local trafficPolicy : loadBalancer: simple: LEAST_CONN
  19. Tales of deploying Istio Ingress --- 1. Intro 2. What

    is Istio? What is Envoy? 3. Istio resources overview 4. Workday infrastructure change overview 5. Rollout plans 6. Lessons learnt
  20. What is Istio Replacing? Past Ingress solution Current Ingress solution

    Future Ingress solution Internet Internet Internet Legacy load balancer Legacy load balancer AWS Load Balancer Istio Istio EC2 instances
  21. What is Istio Replacing? Past Ingress solution Current Ingress solution

    Future Ingress solution Internet Internet Internet Legacy load balancer Legacy load balancer AWS Load Balancer Istio Istio EC2 instances This talk
  22. What is Istio Replacing? Past Ingress solution Current Ingress solution

    Future Ingress solution Internet Internet Internet Legacy load balancer Legacy load balancer AWS Load Balancer Istio Istio EC2 instances This talk Future
  23. Why are we implementing Istio Ingress? Dev and Prod parity

    • Legacy Load balancer too costly to run in dev • Dev load balancer = AWS Application Load balancer • This represents a delta between dev and prod
  24. Why are we implementing Istio Ingress? Dev and Prod parity

    • Legacy Load balancer too costly to run in dev • Dev load balancer = AWS Application Load balancer • This represents a delta between dev and prod Reduce coupling with infrastructure team • The legacy load balancer: managed by the Infrastructure team. • Applications run on platforms managed by the platform team • Istio: also managed by the platform team
  25. Why are we implementing Istio Ingress? Dev and Prod parity

    • Legacy Load balancer too costly to run in dev • Dev load balancer = AWS Application Load balancer • This represents a delta between dev and prod Reduce coupling with infrastructure team • The legacy load balancer: managed by the Infrastructure team. • Applications run on platforms managed by the platform team • Istio: also managed by the platform team Istio is under Kubernetes control • Easier for platform team to manage • Stepping stone for the implementation for Istio Mesh.
  26. Tales of deploying Istio Ingress --- 1. Intro 2. What

    is Istio? What is Envoy? 3. Istio resources overview 4. Workday infrastructure change overview 5. Rollout plans 6. Lessons learnt
  27. Environments overview Development Legacy Internet AWS ALB EC2 New Ingress

    infrastructure Internet AWS NLB Istio
  28. Environments overview Staging, preprod, prod Legacy Internet Legacy LB EC2

    (including routing) New ingress infrastructure Internet Legacy LB Istio (No routing)
  29. Rollout plan in development Step 1: “Dark launch” “Dark launching

    allows development teams to test the efficacy of new, production-ready features without releasing them to an entire user base.” (launchdarkly.com) • Deploy Istio Ingress with no traffic • Add network load balancer to route traffic to Istio • Implement a test ingress service to ensure traffic flows as expected
  30. Rollout plan in development Step 2: pre-flight checks • Review

    current configuration • Translate them into Istio configuration • Ask SME to review configuration
  31. Rollout plan in development Step 3: Progressive rollout • Retire

    AWS application load balancers one by one • Route traffic to Istio instead • Ask SME to run regression tests
  32. Rollout plan in production Phased release Pair with SMEs to

    test the change and ensure there is no regressions Regression tests Forward a subset of the traffic to Istio Move subset of traffic in staging And repeat the process until all subsets of the network are forwarded to Istio Roll out in further environments
  33. Tales of deploying Istio Ingress --- 1. Intro 2. What

    is Istio? What is Envoy? 3. Istio resources overview 4. Workday infrastructure change overview 5. Rollout plans 6. Lessons learnt
  34. Lesson #1: Make allies & educate them Identifies your key

    stakeholders If possible, make a list of key allies so you have escalation points straight away when needed Educate them about the change The more they understand about the change, the more they will be able to test the change Make allies out of them This will allow you to roll out faster by reducing friction / resistance to change as well as making sure they understand the scope of the change and tested it
  35. Lesson #2: Double-check the configuration with your SMEs Service SME

    should double-check the configuration Send a copy of the new configuration to the relevant team and ask them to verify it Verify the permissions As well as the routes, ask them to verify the permissions against the route as well (e.g. HEAD, GET, POST, DELETE) Prepare for the future Depending on the company size, eventually you might need teams to own their own configuration, so it’s a good idea to make them familiar with it
  36. Lesson #3: Accept there will be “unknown unknowns” Legacy is

    everywhere It’s likely the legacy piece of infrastructure has some feature that too many teams depend on and that needs to be ported, replaced, or mitigated in your new infrastructure Cutting edge is not everywhere! Give yourself & your team the time to learn, find issues, and mitigate them Examples: the “pinback feature”
  37. Example: “pinback feature” Can I get: example.com/internalURI ? No!

  38. Example: “pinback feature” Can I get: example.com/internalURI ? No! Can

    I get: example.com/internalURI ? OK!
  39. Example: “pinback feature” Can I get: example.com/internalURI ? No! Can

    I get: example.com/internalURI ? Also no!
  40. Istio Resources: gateway apiVersion: networking.istio.io/v1alpha3 kind: Gateway metadata: name: my-gateway

    namespace: some-config-namespace spec: selector: app: my-gateway-controller servers: - port: number: 443 name: https-443 protocol: HTTPS hosts: - example.com tls: mode: SIMPLE # enables HTTPS on this port serverCertificate : /etc/certs/servercert.pem privateKey: /etc/certs/privatekey.pem
  41. Istio Resources: virtual service apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name:

    example spec: gateway: - my-gateway hosts: - example.com http: - name: "booking-routes" match: - uri: prefix: "/foo" - uri: prefix: "/bar" route: - destination: host: example.my-namespace.svc.cluster.local port-number: 8080
  42. Example: “pinback feature” Can I get: example.com/internalURI ? No! Can

    I get: example.my-namespace.svc.cluster.local/internalURI? OK!
  43. Lesson #4: Verify your assumptions The behavior of your new

    infrastructure might not be the one you expect it to have. Check your assumptions to avoid surprises.
  44. Lesson #4: Verify your assumptions The behavior of your new

    infrastructure might not be the one you expect it to have. Check your assumptions to avoid disappointments. Example: sticky cookies in Istio
  45. Lesson #4: Verify your assumptions The behavior of your new

    infrastructure might not be the one you expect it to have. Check your assumptions to avoid disappointments. Example: sticky cookies in Istio Assumptions: • Path will be defaulted to “/” • This will make my sticky cookie available on the entire website, ensuring stickiness
  46. Lesson #4: Verify your assumptions The behavior of your new

    infrastructure might not be the one you expect it to have. Check your assumptions to avoid disappointments. Example: sticky cookies in Istio Assumptions • Path will be defaulted to “/” • This will make my sticky cookie available on the entire website, ensuring stickiness Reality • Path is defaulted to the URI being hit • If you go to example.com/foo, the cookie is set at “/foo” • Which means, if you go to “example.com/bar”, stickiness is not guaranteed.
  47. Lesson #5: Be ready to develop new solutions (Again) Legacy

    is everywhere Whereby the legacy infrastructure you are replacing is well installed in your company, it might have features / custom tooling that your new infrastructure can’t replace Cutting edge is not everywhere :( Be ready for the possibility that you might have to develop a new software, or get new pieces of infrastructure, in order to maintain some functionality Example: Development of a PrivateLink operator for cluster-to-cluster comm, to ensure isolation of some traffic from the Internet AWS DC 1 AWS DC 2 Internet
  48. Lesson #5: Be ready to develop new solutions (Again) Legacy

    is everywhere Whereby the legacy infrastructure you are replacing is well installed in your company, it might have features / custom tooling that your new infrastructure can’t replace Cutting edge is not everywhere :( Be ready for the possibility that you might have to develop a new software, or get new pieces of infrastructure, in order to maintain some functionality Example: Development of a PrivateLink operator for cluster-to-cluster comm, to ensure isolation of some traffic from the Internet AWS DC 1 AWS DC 2 PrivateLink
  49. Lesson #6: Don’t carry over legacy (And again) Legacy is

    everywhere If the infrastructure you are replacing is old, it’s likely there is piece of legacy involved! Don’t carry them over to your new infra! Take this opportunity to address legacy Liaise with the team whose infrastructure you’re replacing and take notes of the pieces of legacy to address Similarly, liaise with the service teams to ensure their configuration is up to date and that you’re not porting legacy to your new app Example: HTTP Headers overflow
  50. Example: HTTP header overflow

  51. Example: HTTP header overflow

  52. Example: HTTP header overflow

  53. Example: HTTP header overflow Why am I writing so many

    headers? I should fix that...
  54. Lesson #7: Test at scale • Test at scale, with

    the same service redundancy and performance as you would expect in your current production environment • Test the performance of your current infrastructure as a baseline • Record the performance of the new infrastructure to ensure no regression, or mitigate prior to rollout
  55. Lesson #8: Be ready for failure • Defensive approach: Assume

    you will have to revert the change and make it as easy as possible to do that • Prepare a rollback plan in advance and make all stakeholders familiar with it • Ultimate goal: minimize disruption!
  56. Lesson #9: Take all opportunities to learn about your new

    tool One last time: legacy is everywhere; cutting edge is not! If you are replacing legacy infrastructure with new age infrastructure, it’s likely there isn’t as much knowledge about it yet. Take all opportunities you can to learn about it! • experiment with it, participate in debugging issues, read the documentation etc • Document the bugs so that team members who could not participate in debugging can familiarize themselves with it • Share what you can with the wider community!
  57. Tales of deploying Istio Ingress --- Summary Before developing a

    new solution • Identify stakeholders to make this change successful and make them your allies • Double check the current configuration to make sure you start with a correct one • Accept there will be unknown unknowns While rolling out the solution • Verify your assumptions • Be willing to develop new solutions • Don’t carry over legacy Get ready for production rollout • Test at scale • Be prepared for failures At all times, take all the opportunities to learn!
  58. This presentation features not only my work, but my entire

    team’s work, and therefore I would like to recognize their contribution :-) Thank you Scylla + Fabrication Team INF + Pi Team Slide not included in the presentation
  59. Farouq Cathal Adrian Sathish David Rob Lucas Declan Dave John

    Jamie
  60. Tales of deploying Istio Ingress --- Thank you! Learn more

    more about engineering at Workday! medium.com/workday-engineering Learn more about opportunities at Workday! workday.com/careers Learn more about me! @plallin plallin.dev