Benefits and Usage Notes of Istio based on One year of Our Practice

© Hitachi, Ltd. 2021. All rights reserved.
Benefits and Usage Notes of Istio
based on One year of Our Practice
CloudNative Days Kansai 2019
Hitachi, Ltd. R&D Group
Takaya Ide
2019.11.28

View Slide

Objectives of this session
▪ Service Mesh and Istio are getting a lot of attention
▪ On the other hand, there are only a few cases of Istio use in Japan
▪ Hitachi, Ltd. has been using Istio in its internal systems for a year now1
▪ We will share our knowledge of Istio from our practice,
including its benefits, problems, and how to deal with it.
1
1. Since 2018 (Added in May 2021)

View Slide

Contents
2
1. Why Service Mesh?
2. What is Istio
3. Use Cases of Istio in the Company
4. What we learned through operation
5. Conclusion

View Slide

Why Service Mesh
3

View Slide

Microservice Architecture
▪ An architectural style composes an apps as a collection of small services
▪ Each service is developed and operated by each autonomous small team
▪ Advantages
✓ Easy to develop on a large scale
✓ Easy to change
✓ Easy to scale
✓ Easy to localize failure
4
Microservices
UI
Auth
Svc1
Svc2
Svc3

View Slide

Growth of Microservices
As your business grows, your system grows
▪ Increase services and service teams
▪ Each service will change from time to time
▪ Each service has different language and quality
5
Death Star Diagram

View Slide

Operating Microservices
How to operate a large Microservices?
▪ How about routing between services?
▪ How about access control?
▪ How do you observe the entire system?
“Each service team solves the issues” IS NOT GOOD
▪ Large man-hours for development and operation
▪ Specifications and quality will be disparate.
6
Need a solution that is transparent to each service and consistent across the system
→ Service Mesh

View Slide

Istio
7

View Slide

What is Istio?
8

View Slide

Overview of Istio
Istio is OSS service mesh developed by Google and IBM
▪ High functionality and high name recognition
▪ Feature Comparison:
INNOQ Service Mesh Comparison https://servicemesh.es/
Values
▪ Provides operating features with
✓ Transparent from each service
✓ Consistent across entire the service
9

View Slide

Behavior of Istio
10
Pod
Pod
Service
Container Container
Service
Ingress
Traffic in Kubernetes reaches
containers in Pods via Service

View Slide

Pod
Pod
Container Container
Behavior of Istio
11
Service
Service
Proxy
Proxy Proxy
Istio inserts a transparent proxy
server in each Pod

View Slide

Behavior of Istio
12
Pod
Pod
Proxy Proxy
Container Container
Service
Proxy
Service
Control
Plane
Istio’s Control Plane controls each
proxy dynamically
server in each Pod

View Slide

Istio undertake the endpoints, so
the traffic is controlled
transparently from the container
Istio’s Control Plane controls each
proxy dynamically
server in each Pod
Behavior of Istio
Pod
Pod
Proxy Proxy
Container Container
Service
Service
Proxy
13
Control
Plane
Routing
Monitoring
Access Ctrl
Late Limit
etc.

View Slide

Istio’s Control Plane control each
proxy dynamically
server in each Pod
Behavior of Istio
Pod
Pod
Proxy Proxy
Container Container
Service
Service
Proxy
14
Routing
Monitoring
Access Ctrl
Late Limit
etc.
Control
Plane

View Slide

Istio’s Control Plane1
Control plane named istiod has
the features below
▪ Pilot
✓ Delivery configs to proxies
✓ Service Discovery
▪ Citadel
✓ TLS key management
▪ Galley
✓ Control Kubernetes
15
https://istio.io/docs/concepts/what-is-istio/
1. Updated the figure and description in May 2021

View Slide

Istio’s Control Plane control each
proxy dynamically
server in each Pod
Behavior of Istio
Pod
Pod
Proxy
Container Container
Service
Service
Proxy
16
Control
Plane
Routing
Monitoring
Access Ctrl
Late Limit
etc.
Proxy

View Slide

Envoy Proxy
▪ L4/L7 proxies developed by Lyft
▪ Update config dynamically via API
▪ Actual use in production services
▪ Many Service Meshes use it as a
data plane
▪ CNCF Graduated (Nov. 2018)
https://github.com/Proxyproxy/artwork 17

View Slide

Features of Istio
Traffic Management
Service Discovery, Dynamic Request Routing,
Load Balancing, Mirroring, Splitting, Shaping,
Timeout, Retry, CORS, Circuit Braking, Chaos Engineering
Security Management
Authorization, Authentication,
Access Control, TLS key management, etc.
Observability
Monitoring, Logging, Distributed Tracing
18

View Slide

Traffic Management
Dynamic Request Routing
▪ Dynamic traffic control based on policy
▪ Available values for routing
✓ Kubernetes attributes : Namespace, Service, Label...
✓ Traffic attributes : URI, HTTP Method, HTTP Header...
▪ Controlled by VirtualService and DestinationRule resources
Resilience
▪ Retry, Timeout, Circuit Breaking, Redirect, CORS, etc.
▪ Controls that used to be configured for individual services can now be
applied on the platform side
19

View Slide

Canary Release
▪ 90% to v1 app
▪ 10% to v2 app
Proxy
Container
version: v1
Proxy
Container
version: v2
app
Example | Policy based Traffic Control
20
https://istio.io/blog/2017/0.1-canary/

View Slide

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: canary
Spec:
hosts:
- app
http:
- route:
- destination:
host: app
subset: v1
weight: 90
- destination:
host: app
subset: v2
weight: 10
Canary Release
▪ 90% to v1 app
▪ 10% to v2 app
Proxy
Container
version: v1
Proxy
Container
version: v2
app
21
apiVersion: networking.ist
kind: DestinationRule
metadata:
name: canary
Spec:
host: app
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2

View Slide

metadata:
name: canary
Spec:
hosts:
- app
http:
- route:
- destination:
host: app
subset: v1
weight: 90
- destination:
host: app
subset: v2
weight: 10
Canary Release
▪ 90% to v1 app
▪ 10% to v2 app
Proxy
Container
version: v1
Proxy
Container
version: v2
app
22
apiVersion: networking.ist
metadata:
name: canary
Spec:
host: app
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
To Label
Label
Specify the
percentage

View Slide

metadata:
name: canary
Spec:
hosts:
- app
http:
- match:
- headers:
cookie:
regex: "^(.*?;)?(email=[^;]*@foobar.com)(;.*)?$"
route:
- destination:
host: app
subset: v1
weight: 90
- destination:
host: app
subset: v2
weight: 10
- route:
- destination:
host: app
subset: v1
Canary Release
▪ Only traffic with a foobar.com
▪ 90% to v1, 10% to v2
Example2 | Policy based Traffic Control
23
apiVersion: networking.istio.io/v1alph
metadata:
name: canary
Spec:
host: app
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
Proxy
Container
version: v1
Proxy
Container
version: v2
app
[email protected]
[email protected]

View Slide

metadata:
name: canary
Spec:
hosts:
- app
http:
- match:
- headers:
cookie:
regex: "^(.*?;)?(email=[^;]*@foobar.com)(;.*)?$"
route:
- destination:
host: app
subset: v1
weight: 90
- destination:
host: app
subset: v2
weight: 10
- route:
- destination:
host: app
subset: v1
Canary Release
▪ Only traffic with a foobar.com
▪ 90% to v1, 10% to v2
24
metadata:
name: canary
Spec:
host: app
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
Proxy
Container
version: v1
Proxy
Container
version: v2
app
[email protected]
[email protected]
Cookie includes @foobar.com
(NOTE: Storing addresses in
cookies is not secure)
not to be included

View Slide

metadata:
name: canary
Spec:
hosts:
- app
http:
- match:
- headers:
mode:
exact: experimental
route:
- destination:
host: app
subset: v1
weight: 90
- destination:
host: app
subset: v2
weight: 10
- route:
- destination:
host: app
subset: v1
Canary Release
▪ Only traffic with mode
experimental by HTTP header
▪ 90% to v1, 10% to v2
25
metadata:
name: canary
Spec:
host: app
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
Proxy
Container
version: v1
Proxy
Container
version: v2
app
mode=experimental

View Slide

metadata:
name: canary
Spec:
hosts:
- app
http:
- match:
- headers:
mode:
exact: experimental
route:
- destination:
host: app
subset: v1
weight: 90
- destination:
host: app
subset: v2
weight: 10
- route:
- destination:
host: app
subset: v1
Canary Release
▪ Only traffic with mode
experimental by HTTP header
▪ 90% to v1, 10% to v2
26
metadata:
name: canary
Spec:
host: app
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
Proxy
Container
version: v1
Proxy
Container
version: v2
app
mode=experimental
HTTP headers includes
“mode: experimental”
not to be included

View Slide

Security Management
Inter-service Authentication
▪ mTLS: Both server and client authenticate each other with certificates
Authorization
▪ Perform L7-level access control
▪ Attributes of Kuberetes and traffic are available
▪ NOTE: Kubernetes attributes requires “source is via proxy" and "mTLS"
▪ Health check from Kubelet is dropped because it is not through a proxy
→ To avoid this, enable “rewriteAppHTTPProbers”
27

View Slide

Observability
▪ Integration with observability tools
✓ Prometheus, Grafana, Jaeger, Kiali, etc.
▪ NOTE: Beware of increasing system complexity and resource usage
28
Image source
Grafana: https://istio.io/docs/tasks/observability/metrics/using-istio-dashboard/
Jaeger: https://istio.io/docs/tasks/observability/distributed-tracing/jaeger/
kiali: https://github.com/kiali/kiali
Grafana Kiali
Jaeger

View Slide

Use Cases of Istio in the Company
29

View Slide

Our System
▪ Internal service provides a development environment
✓ Provides web editor, backend services, etc.
✓ Backend services are developed autonomously by each team
▪ Kubernetes-based. 100+ Nodes, 2000+ Pods
30
Kubernetes
Web
Editor
Service
1
Service
2
App
Use
User
Service 1 Team
Service 2 Team
Platform Team
Develop
Manage
Kubernetes
Develop

View Slide

Motivation to use Istio
1. Manage inter-service traffic with
✓ Flexible routing
✓ Access Control
✓ Collecting metrics
2. Do not disturb backend team
3. Istio Trend Surges
31
Introduce Istio for verification
purposes as well
Operated it in an internal
production env. for 1.5 year

View Slide

What we learned through operation
32

View Slide

Impressions of Use
Useful, but
the operational cost is heavy1
▪ Are there use cases that can pay for the operational cost ?
▪ Do you have an operational system in place ?
33
There are other perspectives
• System complexity
• Resource consumption
• Additional latency?
1. This was the opinion in 2019. the operational cost now is improved (Added in May 2021)

View Slide

Impressions of Use
Useful, but
34

View Slide

Benefits of Istio
1. Transparency from services
2. Kubernetes native
3. An active community
35

View Slide

Benefit 1｜Transparency from services
No need to modify individual services
▪ Useful when using applications that are difficult to modify,
such as OSS or existing products.
▪ Auto-Injection feature allows for automatic proxy insertion.
▪ Communication costs with the service team are also low (supposedly).
Flexible and sophisticated traffic control
▪ Routing and access control by attributes of traffic and Kubernetes
▪ Controlling retries, timeouts, CORS, etc.
that previously had to be implemented by each service
36

View Slide

Benefit 2｜Kubernetes native
All management is completed by custom resources
▪ We can use Kubernetes ecosystems such as Helm
Istio itself also runs on Kubernetes
▪ Availability with replica and scalability with HPA
▪ NOTE: Beaware of the impact of Control Plane's load
Declarative network management
▪ Manifests becomes the definition of the network configuration
▪ Network management with a GitOps
37

View Slide

Benefit 3｜An active community
Large and active community
▪ Average response time for issues: 5 days 1
▪ Bi-weekly level patch updates
▪ Stable quality since v1.1
A lot of information
▪ Official Documents, Issues, Books, Blogs
▪ However, they also become obsolete quickly.
38
1 Istio Project Dashboards
https://istio.teststats.cncf.io/d/11/issues-age-by-repository-group?orgId=1

View Slide

Impressions of Use
Useful, but
39

View Slide

Usage Notes of Istio
1. Operating Difficulty
2. Update every 3 months
3. The boundary between services and platforms
4. Memory usage of proxy
5. Migration of authentication and authorization
40

View Slide

Note 1｜Operating Difficulty
Istio integrates traffic control, security and observability
▪ A lot of flexible features with a lot of custom resources
▪ Hard to ensure consistency among services
▪ Many demands for knowledge make operations dependent on personal skill
Difficulty in isolating the cause of a failure
▪ Physics, Kubernetes, App,... many factors are involved
▪ Proxy Timeout frames Istio as the criminal
✓ Destination Apps is down → Proxy returns a timeout
✓ Network is down → Proxy returns timeout
41

View Slide

Solution 1｜ Internal Istio Information Sharing Wiki
▪ Create an internal wiki to
operating Istio (No plan to
open it to the public)
▪ Content
✓ Overview
✓ Verification of
Istio’s behavior
✓ Past failures
✓ Debugging tips
✓ etc.
42

View Slide

Note 2｜Update every 3 months
Istio releases every 3 months and support for only one previous version1
▪ Need to keep up with version upgrades
▪ The man-hours for verification and migration are significant.
It is better to have one more person dedicated to this task
Revisit configuration for updates
▪ Changes in authentication and authorization methods
▪ Changes of resource consumption
43
1. Istio Build & Release Cadence https://istio.io/about/release-cadence/

View Slide

Solution 2｜Istio-Operator
Operator, which automates the operation of Istio1
▪ Currently, it's mainly installation and updates
✓ Provides a high-level API that hides
the complexity of installation
✓ Improved robustness by applying
validation and deprecated policies
▪ Discussion of multi-cluster collaboration is underway
44
1. Content as of 2019

View Slide

Note 3｜Boundaries between services and platforms
Duplication of function between services and Istio
▪ Both on the app and Istio have functions such as timeout, retry, CORS
▪ Forcing Istio will cause a backlash from service teams
▪ “I don’t use Istio’s feature because we have already implemented it”
▪ “I use an OSS requires HTTPS. Don’t terminate HTTPS with a proxy”
Who can manage Istio
▪ Some Istio manifest is applied to the entire cluster (e.g., DestinationRule)
▪ Service teams can deploy this at will → Cluster failure
▪ But some configs (such as routing) should be handled by service teams
45

View Slide

Solution 3｜Helm chart
▪ Package the Istio configuration as a Helm Chart
✓ Only the functions that you want the service team to operate
configuration items (values).
✓ Port number and protocol to be published
✓ Timeout, Retry, etc.
▪ The service team can configure features via the chart
46

View Slide

Note 4 | Memory usage of proxy
▪ How does each proxy know the destination ?
→ Each proxy has an address list of all proxy calculated in Control Plane
▪ In our env., the list grows linier to the number of pods
✓ The large number of Pods resulted in a bloated list
✓ At that time, each proxy used 750MB memory by 100 Pods
✓ Memory usage in the entire cluster : 1000 x 750MB = 750GB
✓ OOM Killer will crash Pods and Nodes
✓ In addition, the trend of resource usage changes by version
→ Performance evaluation is required for each Istio update for sizing
47

View Slide

Solution 4｜istio-bench, our benchmark tool
Tool for calculating resource usage based on the number of pods
https://github.com/Hitachi/istio-bench
▪ Deploy 50 dummy Pods and monitor resource usage of Istio
▪ Iterate above and calculate the approximation equation of resource usage
48

View Slide

Solution 4｜(Reference) Changes in proxy memory usage1
49
▪ Memory usage of each proxy at 1000 Pods
▪ Drastically reduced in v1.0 and slowly reducing since then.
771MB
268MB
163MB
145MB 127MB 120MB 163MB 163MB
0
200
400
600
800
1000
1.0.2 1.0.7 1.1.7 1.2.9 1.3.2 1.4.5 1.5.0 1.6.2
Memory usage[MB]
Istio version
1. Updated the chart in April 2020

View Slide

Solution 4｜SideCar Resource
▪ Resources for direct control of proxies
▪ the size of the destination list can be fixed by limiting the destinations
▪ The level of abstraction is finer than others. Pending introduction
50
kind: Sidecar
metadata:
name: clusters-reducer
namespace: default
Spec:
egress:
- hosts:
- "default/*"
- "istio-system/*"
Sidecar https://istio.io/docs/reference/config/networking/sidecar/

View Slide

Note 5 | Migration of authentication and authorization
Applying Istio's functions to the entire system is a bit risky.
But Applying Istio only partially complicates AuthN/AuthZ
▪ Authentication
✓ mTLS: Both client and server authenticate each other
✓ Pods out of Istio mesh cannot use mTLS
✓ Need to consider whether Istio is already applied to peer
▪ Authorization
✓ Using K8s attributes in AuthZ require mTLS on both peers
✓ Need to consider the AuthZ method for each service out of Istio mesh
51

View Slide

Solution 5｜Organize authentication patterns
Organize conditions to communication according to whether Istio is applied
and reflect them in the Helm Chart described above.
52
# Source Destination Conditions to communication
1
Istio
Applied
Istio
Applied
Both sender and receiver must have TLS settings
Enabled or Disabled.
2
Istio
Applied
ｰ The TLS setting is disabled on the source
3 ｰ
Istio
Applied
The TLS setting on the destination is disabled
or PERMISSIVE mode
4 ｰｰ
Unconditional
But no access control can be applied

View Slide

Solution 5｜Share common errors and their causes
Summarize common errors and their causes
53
# Error message Causes
1
upstream connect error
or disconnect/reset
before header
✓ Proxy is down or not working
✓ Proxy is not PID1, so it is not be restarted
2
Connection reset by
peer
✓ Problem with authentication
✓ One peer is mTLS and the other is plain
3 403 access denied
✓ Inconsistent AuthZ or the sender is not via a proxy
✓ If the receiver proxy log with debug level includes a
namespace data, the sender go through a proxy

View Slide

Solution 5｜Automatic mutual TLS (Alpha)
▪ Features introduced in Istio 1.4
▪ Automatically configure the source mTLS setting
✓ Both of peers have a proxy → mTLS
✓ Neither has a proxy → Plain text
▪ The mTLS setting on the destination is set manually
54
Automatic mutual TLS
https://istio.io/docs/tasks/security/authentication/auto-mtls/
Pod
Container
Proxy
Pod
Container
Proxy
mTLS
Pod
Container
Proxy
Pod
Container
Plain rext

View Slide

Impressions of Use
Useful, but
55

View Slide

Conclusion
Istio is for Microservices operations
▪ Transparent and consistent communication management from the service
▪ With functions of communication management, security management,
and observability
Istio is Useful, but heavy operational load
▪ High functionality and flexible traffic control. Kubernetes native, etc.
▪ However, the operational cost is heavy, and practices are still developing.
▪ Is there a use case that pays for the cost ?
56

View Slide

Trademarks
▪ Istio is a registered trademark or trademark
of Google LLC in the United States and other
countries.
▪ Envoy Proxy is a registered trademark or
trademark of The Linux Foundation in the
United States and/or other countries.
▪ Kubernetes is a registered trademark or
▪ Prometheus is a registered trademark or
▪ Grafana is a registered trademark or
trademark of Grafana Labs in the United
States and/or other countries.
▪ Jaeger is a registered trademark or
▪ Kiali is a registered trademark or trademark
of Red Hat, Inc. in the United States and/or
other countries.
▪ All other company names, product names,
service names, and other proper nouns are
trademarks or registered trademarks of their
respective companies.
▪ TM and meta marks are not indicated in the
text and figures in this presentation.
57

View Slide

View Slide

Benefits and Usage Notes of Istio based on One year of Our Practice

Benefits and Usage Notes of Istio based on One year of Our Practice

id

More Decks by id

Other Decks in Technology

Featured

Transcript