© Hitachi, Ltd. 2021. All rights reserved. Benefits and Usage Notes of Istio based on One year of Our Practice CloudNative Days Kansai 2019 Hitachi, Ltd. R&D Group Takaya Ide 2019.11.28

© Hitachi, Ltd. 2021. All rights reserved. Objectives of this session ▪ Service Mesh and Istio are getting a lot of attention ▪ On the other hand, there are only a few cases of Istio use in Japan ▪ Hitachi, Ltd. has been using Istio in its internal systems for a year now1 ▪ We will share our knowledge of Istio from our practice, including its benefits, problems, and how to deal with it. 1 1. Since 2018 (Added in May 2021)

© Hitachi, Ltd. 2021. All rights reserved. Contents 2 1. Why Service Mesh? 2. What is Istio 3. Use Cases of Istio in the Company 4. What we learned through operation 5. Conclusion

© Hitachi, Ltd. 2021. All rights reserved. Why Service Mesh 3

© Hitachi, Ltd. 2021. All rights reserved. Microservice Architecture ▪ An architectural style composes an apps as a collection of small services ▪ Each service is developed and operated by each autonomous small team ▪ Advantages ✓ Easy to develop on a large scale ✓ Easy to change ✓ Easy to scale ✓ Easy to localize failure 4 Microservices UI Auth Svc1 Svc2 Svc3

© Hitachi, Ltd. 2021. All rights reserved. Growth of Microservices As your business grows, your system grows ▪ Increase services and service teams ▪ Each service will change from time to time ▪ Each service has different language and quality 5 Death Star Diagram

© Hitachi, Ltd. 2021. All rights reserved. Operating Microservices How to operate a large Microservices? ▪ How about routing between services? ▪ How about access control? ▪ How do you observe the entire system? “Each service team solves the issues” IS NOT GOOD ▪ Large man-hours for development and operation ▪ Specifications and quality will be disparate. 6 Need a solution that is transparent to each service and consistent across the system → Service Mesh

© Hitachi, Ltd. 2021. All rights reserved. Istio 7

© Hitachi, Ltd. 2021. All rights reserved. What is Istio? 8

© Hitachi, Ltd. 2021. All rights reserved. Overview of Istio Istio is OSS service mesh developed by Google and IBM ▪ High functionality and high name recognition ▪ Feature Comparison: INNOQ Service Mesh Comparison https://servicemesh.es/ Values ▪ Provides operating features with ✓ Transparent from each service ✓ Consistent across entire the service 9

© Hitachi, Ltd. 2021. All rights reserved. Behavior of Istio 10 Pod Pod Service Container Container Service Ingress Traffic in Kubernetes reaches containers in Pods via Service

© Hitachi, Ltd. 2021. All rights reserved. Pod Pod Container Container Behavior of Istio 11 Service Service Proxy Proxy Proxy Istio inserts a transparent proxy server in each Pod Traffic in Kubernetes reaches containers in Pods via Service

© Hitachi, Ltd. 2021. All rights reserved. Behavior of Istio 12 Pod Pod Proxy Proxy Container Container Service Proxy Service Control Plane Istio’s Control Plane controls each proxy dynamically Istio inserts a transparent proxy server in each Pod Traffic in Kubernetes reaches containers in Pods via Service

© Hitachi, Ltd. 2021. All rights reserved. Istio undertake the endpoints, so the traffic is controlled transparently from the container Istio’s Control Plane controls each proxy dynamically Istio inserts a transparent proxy server in each Pod Traffic in Kubernetes reaches containers in Pods via Service Behavior of Istio Pod Pod Proxy Proxy Container Container Service Service Proxy 13 Control Plane Routing Monitoring Access Ctrl Late Limit etc.

© Hitachi, Ltd. 2021. All rights reserved. Istio undertake the endpoints, so the traffic is controlled transparently from the container Istio’s Control Plane control each proxy dynamically Istio inserts a transparent proxy server in each Pod Traffic in Kubernetes reaches containers in Pods via Service Behavior of Istio Pod Pod Proxy Proxy Container Container Service Service Proxy 14 Routing Monitoring Access Ctrl Late Limit etc. Control Plane

© Hitachi, Ltd. 2021. All rights reserved. Istio’s Control Plane1 Control plane named istiod has the features below ▪ Pilot ✓ Delivery configs to proxies ✓ Service Discovery ▪ Citadel ✓ TLS key management ▪ Galley ✓ Control Kubernetes 15 https://istio.io/docs/concepts/what-is-istio/ 1. Updated the figure and description in May 2021

© Hitachi, Ltd. 2021. All rights reserved. Istio undertake the endpoints, so the traffic is controlled transparently from the container Istio’s Control Plane control each proxy dynamically Istio inserts a transparent proxy server in each Pod Traffic in Kubernetes reaches containers in Pods via Service Behavior of Istio Pod Pod Proxy Container Container Service Service Proxy 16 Control Plane Routing Monitoring Access Ctrl Late Limit etc. Proxy

© Hitachi, Ltd. 2021. All rights reserved. Envoy Proxy ▪ L4/L7 proxies developed by Lyft ▪ Update config dynamically via API ▪ Actual use in production services ▪ Many Service Meshes use it as a data plane ▪ CNCF Graduated (Nov. 2018) https://github.com/Proxyproxy/artwork 17

© Hitachi, Ltd. 2021. All rights reserved. Features of Istio Traffic Management Service Discovery, Dynamic Request Routing, Load Balancing, Mirroring, Splitting, Shaping, Timeout, Retry, CORS, Circuit Braking, Chaos Engineering Security Management Authorization, Authentication, Access Control, TLS key management, etc. Observability Monitoring, Logging, Distributed Tracing 18

© Hitachi, Ltd. 2021. All rights reserved. Traffic Management Dynamic Request Routing ▪ Dynamic traffic control based on policy ▪ Available values for routing ✓ Kubernetes attributes : Namespace, Service, Label... ✓ Traffic attributes : URI, HTTP Method, HTTP Header... ▪ Controlled by VirtualService and DestinationRule resources Resilience ▪ Retry, Timeout, Circuit Breaking, Redirect, CORS, etc. ▪ Controls that used to be configured for individual services can now be applied on the platform side 19

© Hitachi, Ltd. 2021. All rights reserved. Canary Release ▪ 90% to v1 app ▪ 10% to v2 app Proxy Container version: v1 Proxy Container version: v2 app Example | Policy based Traffic Control 20 https://istio.io/blog/2017/0.1-canary/

© Hitachi, Ltd. 2021. All rights reserved. apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: canary Spec: hosts: - app http: - route: - destination: host: app subset: v1 weight: 90 - destination: host: app subset: v2 weight: 10 Canary Release ▪ 90% to v1 app ▪ 10% to v2 app Proxy Container version: v1 Proxy Container version: v2 app Example | Policy based Traffic Control 21 apiVersion: networking.ist kind: DestinationRule metadata: name: canary Spec: host: app subsets: - name: v1 labels: version: v1 - name: v2 labels: version: v2 https://istio.io/blog/2017/0.1-canary/

© Hitachi, Ltd. 2021. All rights reserved. apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: canary Spec: hosts: - app http: - route: - destination: host: app subset: v1 weight: 90 - destination: host: app subset: v2 weight: 10 Canary Release ▪ 90% to v1 app ▪ 10% to v2 app Proxy Container version: v1 Proxy Container version: v2 app Example | Policy based Traffic Control 22 apiVersion: networking.ist kind: DestinationRule metadata: name: canary Spec: host: app subsets: - name: v1 labels: version: v1 - name: v2 labels: version: v2 https://istio.io/blog/2017/0.1-canary/ To Label Label Specify the percentage

© Hitachi, Ltd. 2021. All rights reserved. apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: canary Spec: hosts: - app http: - match: - headers: cookie: regex: "^(.*?;)?(email=[^;]*@foobar.com)(;.*)?$" route: - destination: host: app subset: v1 weight: 90 - destination: host: app subset: v2 weight: 10 - route: - destination: host: app subset: v1 Canary Release ▪ Only traffic with a foobar.com ▪ 90% to v1, 10% to v2 Example2 | Policy based Traffic Control 23 https://istio.io/blog/2017/0.1-canary/ apiVersion: networking.istio.io/v1alph kind: DestinationRule metadata: name: canary Spec: host: app subsets: - name: v1 labels: version: v1 - name: v2 labels: version: v2 Proxy Container version: v1 Proxy Container version: v2 app bob@foobar.com andi@foobar.com

© Hitachi, Ltd. 2021. All rights reserved. apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: canary Spec: hosts: - app http: - match: - headers: cookie: regex: "^(.*?;)?(email=[^;]*@foobar.com)(;.*)?$" route: - destination: host: app subset: v1 weight: 90 - destination: host: app subset: v2 weight: 10 - route: - destination: host: app subset: v1 Canary Release ▪ Only traffic with a foobar.com ▪ 90% to v1, 10% to v2 Example2 | Policy based Traffic Control 24 https://istio.io/blog/2017/0.1-canary/ apiVersion: networking.istio.io/v1alph kind: DestinationRule metadata: name: canary Spec: host: app subsets: - name: v1 labels: version: v1 - name: v2 labels: version: v2 Proxy Container version: v1 Proxy Container version: v2 app bob@foobar.com andi@foobar.com Cookie includes @foobar.com (NOTE: Storing addresses in cookies is not secure) not to be included

© Hitachi, Ltd. 2021. All rights reserved. apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: canary Spec: hosts: - app http: - match: - headers: mode: exact: experimental route: - destination: host: app subset: v1 weight: 90 - destination: host: app subset: v2 weight: 10 - route: - destination: host: app subset: v1 Canary Release ▪ Only traffic with mode experimental by HTTP header ▪ 90% to v1, 10% to v2 Example2 | Policy based Traffic Control 25 https://istio.io/blog/2017/0.1-canary/ apiVersion: networking.istio.io/v1alph kind: DestinationRule metadata: name: canary Spec: host: app subsets: - name: v1 labels: version: v1 - name: v2 labels: version: v2 Proxy Container version: v1 Proxy Container version: v2 app mode=experimental

© Hitachi, Ltd. 2021. All rights reserved. apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: canary Spec: hosts: - app http: - match: - headers: mode: exact: experimental route: - destination: host: app subset: v1 weight: 90 - destination: host: app subset: v2 weight: 10 - route: - destination: host: app subset: v1 Canary Release ▪ Only traffic with mode experimental by HTTP header ▪ 90% to v1, 10% to v2 Example2 | Policy based Traffic Control 26 https://istio.io/blog/2017/0.1-canary/ apiVersion: networking.istio.io/v1alph kind: DestinationRule metadata: name: canary Spec: host: app subsets: - name: v1 labels: version: v1 - name: v2 labels: version: v2 Proxy Container version: v1 Proxy Container version: v2 app mode=experimental HTTP headers includes “mode: experimental” not to be included

© Hitachi, Ltd. 2021. All rights reserved. Security Management Inter-service Authentication ▪ mTLS: Both server and client authenticate each other with certificates Authorization ▪ Perform L7-level access control ▪ Attributes of Kuberetes and traffic are available ▪ NOTE: Kubernetes attributes requires “source is via proxy" and "mTLS" ▪ Health check from Kubelet is dropped because it is not through a proxy → To avoid this, enable “rewriteAppHTTPProbers” 27

© Hitachi, Ltd. 2021. All rights reserved. Observability ▪ Integration with observability tools ✓ Prometheus, Grafana, Jaeger, Kiali, etc. ▪ NOTE: Beware of increasing system complexity and resource usage 28 Image source Grafana: https://istio.io/docs/tasks/observability/metrics/using-istio-dashboard/ Jaeger: https://istio.io/docs/tasks/observability/distributed-tracing/jaeger/ kiali: https://github.com/kiali/kiali Grafana Kiali Jaeger

© Hitachi, Ltd. 2021. All rights reserved. Use Cases of Istio in the Company 29

© Hitachi, Ltd. 2021. All rights reserved. Our System ▪ Internal service provides a development environment ✓ Provides web editor, backend services, etc. ✓ Backend services are developed autonomously by each team ▪ Kubernetes-based. 100+ Nodes, 2000+ Pods 30 Kubernetes Web Editor Service 1 Service 2 App Use User Service 1 Team Service 2 Team Platform Team Develop Manage Kubernetes Develop

© Hitachi, Ltd. 2021. All rights reserved. Motivation to use Istio 1. Manage inter-service traffic with ✓ Flexible routing ✓ Access Control ✓ Collecting metrics 2. Do not disturb backend team 3. Istio Trend Surges 31 Introduce Istio for verification purposes as well Operated it in an internal production env. for 1.5 year

© Hitachi, Ltd. 2021. All rights reserved. What we learned through operation 32

© Hitachi, Ltd. 2021. All rights reserved. Impressions of Use Useful, but the operational cost is heavy1 ▪ Are there use cases that can pay for the operational cost ? ▪ Do you have an operational system in place ? 33 There are other perspectives • System complexity • Resource consumption • Additional latency? 1. This was the opinion in 2019. the operational cost now is improved (Added in May 2021)

© Hitachi, Ltd. 2021. All rights reserved. Impressions of Use Useful, but the operational cost is heavy1 ▪ Are there use cases that can pay for the operational cost ? ▪ Do you have an operational system in place ? 34 There are other perspectives • System complexity • Resource consumption • Additional latency? 1. This was the opinion in 2019. the operational cost now is improved (Added in May 2021)

© Hitachi, Ltd. 2021. All rights reserved. Benefits of Istio 1. Transparency from services 2. Kubernetes native 3. An active community 35

© Hitachi, Ltd. 2021. All rights reserved. Benefit 1｜Transparency from services No need to modify individual services ▪ Useful when using applications that are difficult to modify, such as OSS or existing products. ▪ Auto-Injection feature allows for automatic proxy insertion. ▪ Communication costs with the service team are also low (supposedly). Flexible and sophisticated traffic control ▪ Routing and access control by attributes of traffic and Kubernetes ▪ Controlling retries, timeouts, CORS, etc. that previously had to be implemented by each service 36

© Hitachi, Ltd. 2021. All rights reserved. Benefit 2｜Kubernetes native All management is completed by custom resources ▪ We can use Kubernetes ecosystems such as Helm Istio itself also runs on Kubernetes ▪ Availability with replica and scalability with HPA ▪ NOTE: Beaware of the impact of Control Plane's load Declarative network management ▪ Manifests becomes the definition of the network configuration ▪ Network management with a GitOps 37

© Hitachi, Ltd. 2021. All rights reserved. Benefit 3｜An active community Large and active community ▪ Average response time for issues: 5 days 1 ▪ Bi-weekly level patch updates ▪ Stable quality since v1.1 A lot of information ▪ Official Documents, Issues, Books, Blogs ▪ However, they also become obsolete quickly. 38 1 Istio Project Dashboards https://istio.teststats.cncf.io/d/11/issues-age-by-repository-group?orgId=1

© Hitachi, Ltd. 2021. All rights reserved. Impressions of Use Useful, but the operational cost is heavy1 ▪ Are there use cases that can pay for the operational cost ? ▪ Do you have an operational system in place ? 39 There are other perspectives • System complexity • Resource consumption • Additional latency? 1. This was the opinion in 2019. the operational cost now is improved (Added in May 2021)

© Hitachi, Ltd. 2021. All rights reserved. Usage Notes of Istio 1. Operating Difficulty 2. Update every 3 months 3. The boundary between services and platforms 4. Memory usage of proxy 5. Migration of authentication and authorization 40

© Hitachi, Ltd. 2021. All rights reserved. Note 1｜Operating Difficulty Istio integrates traffic control, security and observability ▪ A lot of flexible features with a lot of custom resources ▪ Hard to ensure consistency among services ▪ Many demands for knowledge make operations dependent on personal skill Difficulty in isolating the cause of a failure ▪ Physics, Kubernetes, App,... many factors are involved ▪ Proxy Timeout frames Istio as the criminal ✓ Destination Apps is down → Proxy returns a timeout ✓ Network is down → Proxy returns timeout 41

© Hitachi, Ltd. 2021. All rights reserved. Solution 1｜ Internal Istio Information Sharing Wiki ▪ Create an internal wiki to operating Istio (No plan to open it to the public) ▪ Content ✓ Overview ✓ Verification of Istio’s behavior ✓ Past failures ✓ Debugging tips ✓ etc. 42

© Hitachi, Ltd. 2021. All rights reserved. Note 2｜Update every 3 months Istio releases every 3 months and support for only one previous version1 ▪ Need to keep up with version upgrades ▪ The man-hours for verification and migration are significant. It is better to have one more person dedicated to this task Revisit configuration for updates ▪ Changes in authentication and authorization methods ▪ Changes of resource consumption 43 1. Istio Build & Release Cadence https://istio.io/about/release-cadence/

© Hitachi, Ltd. 2021. All rights reserved. Solution 2｜Istio-Operator Operator, which automates the operation of Istio1 ▪ Currently, it's mainly installation and updates ✓ Provides a high-level API that hides the complexity of installation ✓ Improved robustness by applying validation and deprecated policies ▪ Discussion of multi-cluster collaboration is underway 44 1. Content as of 2019

© Hitachi, Ltd. 2021. All rights reserved. Note 3｜Boundaries between services and platforms Duplication of function between services and Istio ▪ Both on the app and Istio have functions such as timeout, retry, CORS ▪ Forcing Istio will cause a backlash from service teams ▪ “I don’t use Istio’s feature because we have already implemented it” ▪ “I use an OSS requires HTTPS. Don’t terminate HTTPS with a proxy” Who can manage Istio ▪ Some Istio manifest is applied to the entire cluster (e.g., DestinationRule) ▪ Service teams can deploy this at will → Cluster failure ▪ But some configs (such as routing) should be handled by service teams 45

© Hitachi, Ltd. 2021. All rights reserved. Solution 3｜Helm chart ▪ Package the Istio configuration as a Helm Chart ✓ Only the functions that you want the service team to operate configuration items (values). ✓ Port number and protocol to be published ✓ Timeout, Retry, etc. ▪ The service team can configure features via the chart 46

© Hitachi, Ltd. 2021. All rights reserved. Note 4 | Memory usage of proxy ▪ How does each proxy know the destination ? → Each proxy has an address list of all proxy calculated in Control Plane ▪ In our env., the list grows linier to the number of pods ✓ The large number of Pods resulted in a bloated list ✓ At that time, each proxy used 750MB memory by 100 Pods ✓ Memory usage in the entire cluster : 1000 x 750MB = 750GB ✓ OOM Killer will crash Pods and Nodes ✓ In addition, the trend of resource usage changes by version → Performance evaluation is required for each Istio update for sizing 47

© Hitachi, Ltd. 2021. All rights reserved. Solution 4｜istio-bench, our benchmark tool Tool for calculating resource usage based on the number of pods https://github.com/Hitachi/istio-bench ▪ Deploy 50 dummy Pods and monitor resource usage of Istio ▪ Iterate above and calculate the approximation equation of resource usage 48

© Hitachi, Ltd. 2021. All rights reserved. Solution 4｜(Reference) Changes in proxy memory usage1 49 ▪ Memory usage of each proxy at 1000 Pods ▪ Drastically reduced in v1.0 and slowly reducing since then. 771MB 268MB 163MB 145MB 127MB 120MB 163MB 163MB 0 200 400 600 800 1000 1.0.2 1.0.7 1.1.7 1.2.9 1.3.2 1.4.5 1.5.0 1.6.2 Memory usage[MB] Istio version 1. Updated the chart in April 2020

© Hitachi, Ltd. 2021. All rights reserved. Solution 4｜SideCar Resource ▪ Resources for direct control of proxies ▪ the size of the destination list can be fixed by limiting the destinations ▪ The level of abstraction is finer than others. Pending introduction 50 apiVersion: networking.istio.io/v1alpha3 kind: Sidecar metadata: name: clusters-reducer namespace: default Spec: egress: - hosts: - "default/*" - "istio-system/*" Sidecar https://istio.io/docs/reference/config/networking/sidecar/

© Hitachi, Ltd. 2021. All rights reserved. Note 5 | Migration of authentication and authorization Applying Istio's functions to the entire system is a bit risky. But Applying Istio only partially complicates AuthN/AuthZ ▪ Authentication ✓ mTLS: Both client and server authenticate each other ✓ Pods out of Istio mesh cannot use mTLS ✓ Need to consider whether Istio is already applied to peer ▪ Authorization ✓ Using K8s attributes in AuthZ require mTLS on both peers ✓ Need to consider the AuthZ method for each service out of Istio mesh 51

© Hitachi, Ltd. 2021. All rights reserved. Solution 5｜Organize authentication patterns Organize conditions to communication according to whether Istio is applied and reflect them in the Helm Chart described above. 52 # Source Destination Conditions to communication 1 Istio Applied Istio Applied Both sender and receiver must have TLS settings Enabled or Disabled. 2 Istio Applied ｰ The TLS setting is disabled on the source 3 ｰ Istio Applied The TLS setting on the destination is disabled or PERMISSIVE mode 4 ｰ ｰ Unconditional But no access control can be applied

© Hitachi, Ltd. 2021. All rights reserved. Solution 5｜Share common errors and their causes Summarize common errors and their causes 53 # Error message Causes 1 upstream connect error or disconnect/reset before header ✓ Proxy is down or not working ✓ Proxy is not PID1, so it is not be restarted 2 Connection reset by peer ✓ Problem with authentication ✓ One peer is mTLS and the other is plain 3 403 access denied ✓ Inconsistent AuthZ or the sender is not via a proxy ✓ If the receiver proxy log with debug level includes a namespace data, the sender go through a proxy

© Hitachi, Ltd. 2021. All rights reserved. Solution 5｜Automatic mutual TLS (Alpha) ▪ Features introduced in Istio 1.4 ▪ Automatically configure the source mTLS setting ✓ Both of peers have a proxy → mTLS ✓ Neither has a proxy → Plain text ▪ The mTLS setting on the destination is set manually 54 Automatic mutual TLS https://istio.io/docs/tasks/security/authentication/auto-mtls/ Pod Container Proxy Pod Container Proxy mTLS Pod Container Proxy Pod Container Plain rext

© Hitachi, Ltd. 2021. All rights reserved. Impressions of Use Useful, but the operational cost is heavy1 ▪ Are there use cases that can pay for the operational cost ? ▪ Do you have an operational system in place ? 55 There are other perspectives • System complexity • Resource consumption • Additional latency? 1. This was the opinion in 2019. the operational cost now is improved (Added in May 2021)

© Hitachi, Ltd. 2021. All rights reserved. Conclusion Istio is for Microservices operations ▪ Transparent and consistent communication management from the service ▪ With functions of communication management, security management, and observability Istio is Useful, but heavy operational load ▪ High functionality and flexible traffic control. Kubernetes native, etc. ▪ However, the operational cost is heavy, and practices are still developing. ▪ Is there a use case that pays for the cost ? 56

© Hitachi, Ltd. 2021. All rights reserved. Trademarks ▪ Istio is a registered trademark or trademark of Google LLC in the United States and other countries. ▪ Envoy Proxy is a registered trademark or trademark of The Linux Foundation in the United States and/or other countries. ▪ Kubernetes is a registered trademark or trademark of The Linux Foundation in the United States and/or other countries. ▪ Prometheus is a registered trademark or trademark of The Linux Foundation in the United States and/or other countries. ▪ Grafana is a registered trademark or trademark of Grafana Labs in the United States and/or other countries. ▪ Jaeger is a registered trademark or trademark of The Linux Foundation in the United States and/or other countries. ▪ Kiali is a registered trademark or trademark of Red Hat, Inc. in the United States and/or other countries. ▪ All other company names, product names, service names, and other proper nouns are trademarks or registered trademarks of their respective companies. ▪ TM and meta marks are not indicated in the text and figures in this presentation. 57

No content