$30 off During Our Annual Pro Sale. View Details »

Benefits and Usage Notes of Istio based on One year of Our Practice

id
May 17, 2021

Benefits and Usage Notes of Istio based on One year of Our Practice

Original slide

Movie (in Japanese)

---

CloudNative Days Kansai 2019
Istio, a service mesh implementation, has been attracting attention for operating microservices, which are becoming more complex every day.

However, despite Istio's popularity and functionality, it is still difficult to implement Istio in actual services due to the complexity of its specifications and lack of practices.

At Hitachi, we have been operating microservices using Istio on a 2000+Pod Kubernetes cluster for a year to provide an in-house development environment.

In this talk, I will introduce some of the practices we have learned through this operation, such as problems and countermeasures when using Istio.

id

May 17, 2021
Tweet

More Decks by id

Other Decks in Technology

Transcript

  1. © Hitachi, Ltd. 2021. All rights reserved.
    Benefits and Usage Notes of Istio
    based on One year of Our Practice
    CloudNative Days Kansai 2019
    Hitachi, Ltd. R&D Group
    Takaya Ide
    2019.11.28

    View Slide

  2. © Hitachi, Ltd. 2021. All rights reserved.
    Objectives of this session
    ▪ Service Mesh and Istio are getting a lot of attention
    ▪ On the other hand, there are only a few cases of Istio use in Japan
    ▪ Hitachi, Ltd. has been using Istio in its internal systems for a year now1
    ▪ We will share our knowledge of Istio from our practice,
    including its benefits, problems, and how to deal with it.
    1
    1. Since 2018 (Added in May 2021)

    View Slide

  3. © Hitachi, Ltd. 2021. All rights reserved.
    Contents
    2
    1. Why Service Mesh?
    2. What is Istio
    3. Use Cases of Istio in the Company
    4. What we learned through operation
    5. Conclusion

    View Slide

  4. © Hitachi, Ltd. 2021. All rights reserved.
    Why Service Mesh
    3

    View Slide

  5. © Hitachi, Ltd. 2021. All rights reserved.
    Microservice Architecture
    ▪ An architectural style composes an apps as a collection of small services
    ▪ Each service is developed and operated by each autonomous small team
    ▪ Advantages
    ✓ Easy to develop on a large scale
    ✓ Easy to change
    ✓ Easy to scale
    ✓ Easy to localize failure
    4
    Microservices
    UI
    Auth
    Svc1
    Svc2
    Svc3

    View Slide

  6. © Hitachi, Ltd. 2021. All rights reserved.
    Growth of Microservices
    As your business grows, your system grows
    ▪ Increase services and service teams
    ▪ Each service will change from time to time
    ▪ Each service has different language and quality
    5
    Death Star Diagram

    View Slide

  7. © Hitachi, Ltd. 2021. All rights reserved.
    Operating Microservices
    How to operate a large Microservices?
    ▪ How about routing between services?
    ▪ How about access control?
    ▪ How do you observe the entire system?
    “Each service team solves the issues” IS NOT GOOD
    ▪ Large man-hours for development and operation
    ▪ Specifications and quality will be disparate.
    6
    Need a solution that is transparent to each service and consistent across the system
    → Service Mesh

    View Slide

  8. © Hitachi, Ltd. 2021. All rights reserved.
    Istio
    7

    View Slide

  9. © Hitachi, Ltd. 2021. All rights reserved.
    What is Istio?
    8

    View Slide

  10. © Hitachi, Ltd. 2021. All rights reserved.
    Overview of Istio
    Istio is OSS service mesh developed by Google and IBM
    ▪ High functionality and high name recognition
    ▪ Feature Comparison:
    INNOQ Service Mesh Comparison https://servicemesh.es/
    Values
    ▪ Provides operating features with
    ✓ Transparent from each service
    ✓ Consistent across entire the service
    9

    View Slide

  11. © Hitachi, Ltd. 2021. All rights reserved.
    Behavior of Istio
    10
    Pod
    Pod
    Service
    Container Container
    Service
    Ingress
    Traffic in Kubernetes reaches
    containers in Pods via Service

    View Slide

  12. © Hitachi, Ltd. 2021. All rights reserved.
    Pod
    Pod
    Container Container
    Behavior of Istio
    11
    Service
    Service
    Proxy
    Proxy Proxy
    Istio inserts a transparent proxy
    server in each Pod
    Traffic in Kubernetes reaches
    containers in Pods via Service

    View Slide

  13. © Hitachi, Ltd. 2021. All rights reserved.
    Behavior of Istio
    12
    Pod
    Pod
    Proxy Proxy
    Container Container
    Service
    Proxy
    Service
    Control
    Plane
    Istio’s Control Plane controls each
    proxy dynamically
    Istio inserts a transparent proxy
    server in each Pod
    Traffic in Kubernetes reaches
    containers in Pods via Service

    View Slide

  14. © Hitachi, Ltd. 2021. All rights reserved.
    Istio undertake the endpoints, so
    the traffic is controlled
    transparently from the container
    Istio’s Control Plane controls each
    proxy dynamically
    Istio inserts a transparent proxy
    server in each Pod
    Traffic in Kubernetes reaches
    containers in Pods via Service
    Behavior of Istio
    Pod
    Pod
    Proxy Proxy
    Container Container
    Service
    Service
    Proxy
    13
    Control
    Plane
    Routing
    Monitoring
    Access Ctrl
    Late Limit
    etc.

    View Slide

  15. © Hitachi, Ltd. 2021. All rights reserved.
    Istio undertake the endpoints, so
    the traffic is controlled
    transparently from the container
    Istio’s Control Plane control each
    proxy dynamically
    Istio inserts a transparent proxy
    server in each Pod
    Traffic in Kubernetes reaches
    containers in Pods via Service
    Behavior of Istio
    Pod
    Pod
    Proxy Proxy
    Container Container
    Service
    Service
    Proxy
    14
    Routing
    Monitoring
    Access Ctrl
    Late Limit
    etc.
    Control
    Plane

    View Slide

  16. © Hitachi, Ltd. 2021. All rights reserved.
    Istio’s Control Plane1
    Control plane named istiod has
    the features below
    ▪ Pilot
    ✓ Delivery configs to proxies
    ✓ Service Discovery
    ▪ Citadel
    ✓ TLS key management
    ▪ Galley
    ✓ Control Kubernetes
    15
    https://istio.io/docs/concepts/what-is-istio/
    1. Updated the figure and description in May 2021

    View Slide

  17. © Hitachi, Ltd. 2021. All rights reserved.
    Istio undertake the endpoints, so
    the traffic is controlled
    transparently from the container
    Istio’s Control Plane control each
    proxy dynamically
    Istio inserts a transparent proxy
    server in each Pod
    Traffic in Kubernetes reaches
    containers in Pods via Service
    Behavior of Istio
    Pod
    Pod
    Proxy
    Container Container
    Service
    Service
    Proxy
    16
    Control
    Plane
    Routing
    Monitoring
    Access Ctrl
    Late Limit
    etc.
    Proxy

    View Slide

  18. © Hitachi, Ltd. 2021. All rights reserved.
    Envoy Proxy
    ▪ L4/L7 proxies developed by Lyft
    ▪ Update config dynamically via API
    ▪ Actual use in production services
    ▪ Many Service Meshes use it as a
    data plane
    ▪ CNCF Graduated (Nov. 2018)
    https://github.com/Proxyproxy/artwork 17

    View Slide

  19. © Hitachi, Ltd. 2021. All rights reserved.
    Features of Istio
    Traffic Management
    Service Discovery, Dynamic Request Routing,
    Load Balancing, Mirroring, Splitting, Shaping,
    Timeout, Retry, CORS, Circuit Braking, Chaos Engineering
    Security Management
    Authorization, Authentication,
    Access Control, TLS key management, etc.
    Observability
    Monitoring, Logging, Distributed Tracing
    18

    View Slide

  20. © Hitachi, Ltd. 2021. All rights reserved.
    Traffic Management
    Dynamic Request Routing
    ▪ Dynamic traffic control based on policy
    ▪ Available values for routing
    ✓ Kubernetes attributes : Namespace, Service, Label...
    ✓ Traffic attributes : URI, HTTP Method, HTTP Header...
    ▪ Controlled by VirtualService and DestinationRule resources
    Resilience
    ▪ Retry, Timeout, Circuit Breaking, Redirect, CORS, etc.
    ▪ Controls that used to be configured for individual services can now be
    applied on the platform side
    19

    View Slide

  21. © Hitachi, Ltd. 2021. All rights reserved.
    Canary Release
    ▪ 90% to v1 app
    ▪ 10% to v2 app
    Proxy
    Container
    version: v1
    Proxy
    Container
    version: v2
    app
    Example | Policy based Traffic Control
    20
    https://istio.io/blog/2017/0.1-canary/

    View Slide

  22. © Hitachi, Ltd. 2021. All rights reserved.
    apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
    name: canary
    Spec:
    hosts:
    - app
    http:
    - route:
    - destination:
    host: app
    subset: v1
    weight: 90
    - destination:
    host: app
    subset: v2
    weight: 10
    Canary Release
    ▪ 90% to v1 app
    ▪ 10% to v2 app
    Proxy
    Container
    version: v1
    Proxy
    Container
    version: v2
    app
    Example | Policy based Traffic Control
    21
    apiVersion: networking.ist
    kind: DestinationRule
    metadata:
    name: canary
    Spec:
    host: app
    subsets:
    - name: v1
    labels:
    version: v1
    - name: v2
    labels:
    version: v2
    https://istio.io/blog/2017/0.1-canary/

    View Slide

  23. © Hitachi, Ltd. 2021. All rights reserved.
    apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
    name: canary
    Spec:
    hosts:
    - app
    http:
    - route:
    - destination:
    host: app
    subset: v1
    weight: 90
    - destination:
    host: app
    subset: v2
    weight: 10
    Canary Release
    ▪ 90% to v1 app
    ▪ 10% to v2 app
    Proxy
    Container
    version: v1
    Proxy
    Container
    version: v2
    app
    Example | Policy based Traffic Control
    22
    apiVersion: networking.ist
    kind: DestinationRule
    metadata:
    name: canary
    Spec:
    host: app
    subsets:
    - name: v1
    labels:
    version: v1
    - name: v2
    labels:
    version: v2
    https://istio.io/blog/2017/0.1-canary/
    To Label
    Label
    Specify the
    percentage

    View Slide

  24. © Hitachi, Ltd. 2021. All rights reserved.
    apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
    name: canary
    Spec:
    hosts:
    - app
    http:
    - match:
    - headers:
    cookie:
    regex: "^(.*?;)?(email=[^;]*@foobar.com)(;.*)?$"
    route:
    - destination:
    host: app
    subset: v1
    weight: 90
    - destination:
    host: app
    subset: v2
    weight: 10
    - route:
    - destination:
    host: app
    subset: v1
    Canary Release
    ▪ Only traffic with a foobar.com
    ▪ 90% to v1, 10% to v2
    Example2 | Policy based Traffic Control
    23
    https://istio.io/blog/2017/0.1-canary/
    apiVersion: networking.istio.io/v1alph
    kind: DestinationRule
    metadata:
    name: canary
    Spec:
    host: app
    subsets:
    - name: v1
    labels:
    version: v1
    - name: v2
    labels:
    version: v2
    Proxy
    Container
    version: v1
    Proxy
    Container
    version: v2
    app
    [email protected]
    [email protected]

    View Slide

  25. © Hitachi, Ltd. 2021. All rights reserved.
    apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
    name: canary
    Spec:
    hosts:
    - app
    http:
    - match:
    - headers:
    cookie:
    regex: "^(.*?;)?(email=[^;]*@foobar.com)(;.*)?$"
    route:
    - destination:
    host: app
    subset: v1
    weight: 90
    - destination:
    host: app
    subset: v2
    weight: 10
    - route:
    - destination:
    host: app
    subset: v1
    Canary Release
    ▪ Only traffic with a foobar.com
    ▪ 90% to v1, 10% to v2
    Example2 | Policy based Traffic Control
    24
    https://istio.io/blog/2017/0.1-canary/
    apiVersion: networking.istio.io/v1alph
    kind: DestinationRule
    metadata:
    name: canary
    Spec:
    host: app
    subsets:
    - name: v1
    labels:
    version: v1
    - name: v2
    labels:
    version: v2
    Proxy
    Container
    version: v1
    Proxy
    Container
    version: v2
    app
    [email protected]
    [email protected]
    Cookie includes @foobar.com
    (NOTE: Storing addresses in
    cookies is not secure)
    not to be included

    View Slide

  26. © Hitachi, Ltd. 2021. All rights reserved.
    apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
    name: canary
    Spec:
    hosts:
    - app
    http:
    - match:
    - headers:
    mode:
    exact: experimental
    route:
    - destination:
    host: app
    subset: v1
    weight: 90
    - destination:
    host: app
    subset: v2
    weight: 10
    - route:
    - destination:
    host: app
    subset: v1
    Canary Release
    ▪ Only traffic with mode
    experimental by HTTP header
    ▪ 90% to v1, 10% to v2
    Example2 | Policy based Traffic Control
    25
    https://istio.io/blog/2017/0.1-canary/
    apiVersion: networking.istio.io/v1alph
    kind: DestinationRule
    metadata:
    name: canary
    Spec:
    host: app
    subsets:
    - name: v1
    labels:
    version: v1
    - name: v2
    labels:
    version: v2
    Proxy
    Container
    version: v1
    Proxy
    Container
    version: v2
    app
    mode=experimental

    View Slide

  27. © Hitachi, Ltd. 2021. All rights reserved.
    apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
    name: canary
    Spec:
    hosts:
    - app
    http:
    - match:
    - headers:
    mode:
    exact: experimental
    route:
    - destination:
    host: app
    subset: v1
    weight: 90
    - destination:
    host: app
    subset: v2
    weight: 10
    - route:
    - destination:
    host: app
    subset: v1
    Canary Release
    ▪ Only traffic with mode
    experimental by HTTP header
    ▪ 90% to v1, 10% to v2
    Example2 | Policy based Traffic Control
    26
    https://istio.io/blog/2017/0.1-canary/
    apiVersion: networking.istio.io/v1alph
    kind: DestinationRule
    metadata:
    name: canary
    Spec:
    host: app
    subsets:
    - name: v1
    labels:
    version: v1
    - name: v2
    labels:
    version: v2
    Proxy
    Container
    version: v1
    Proxy
    Container
    version: v2
    app
    mode=experimental
    HTTP headers includes
    “mode: experimental”
    not to be included

    View Slide

  28. © Hitachi, Ltd. 2021. All rights reserved.
    Security Management
    Inter-service Authentication
    ▪ mTLS: Both server and client authenticate each other with certificates
    Authorization
    ▪ Perform L7-level access control
    ▪ Attributes of Kuberetes and traffic are available
    ▪ NOTE: Kubernetes attributes requires “source is via proxy" and "mTLS"
    ▪ Health check from Kubelet is dropped because it is not through a proxy
    → To avoid this, enable “rewriteAppHTTPProbers”
    27

    View Slide

  29. © Hitachi, Ltd. 2021. All rights reserved.
    Observability
    ▪ Integration with observability tools
    ✓ Prometheus, Grafana, Jaeger, Kiali, etc.
    ▪ NOTE: Beware of increasing system complexity and resource usage
    28
    Image source
    Grafana: https://istio.io/docs/tasks/observability/metrics/using-istio-dashboard/
    Jaeger: https://istio.io/docs/tasks/observability/distributed-tracing/jaeger/
    kiali: https://github.com/kiali/kiali
    Grafana Kiali
    Jaeger

    View Slide

  30. © Hitachi, Ltd. 2021. All rights reserved.
    Use Cases of Istio in the Company
    29

    View Slide

  31. © Hitachi, Ltd. 2021. All rights reserved.
    Our System
    ▪ Internal service provides a development environment
    ✓ Provides web editor, backend services, etc.
    ✓ Backend services are developed autonomously by each team
    ▪ Kubernetes-based. 100+ Nodes, 2000+ Pods
    30
    Kubernetes
    Web
    Editor
    Service
    1
    Service
    2
    App
    Use
    User
    Service 1 Team
    Service 2 Team
    Platform Team
    Develop
    Manage
    Kubernetes
    Develop

    View Slide

  32. © Hitachi, Ltd. 2021. All rights reserved.
    Motivation to use Istio
    1. Manage inter-service traffic with
    ✓ Flexible routing
    ✓ Access Control
    ✓ Collecting metrics
    2. Do not disturb backend team
    3. Istio Trend Surges
    31
    Introduce Istio for verification
    purposes as well
    Operated it in an internal
    production env. for 1.5 year

    View Slide

  33. © Hitachi, Ltd. 2021. All rights reserved.
    What we learned through operation
    32

    View Slide

  34. © Hitachi, Ltd. 2021. All rights reserved.
    Impressions of Use
    Useful, but
    the operational cost is heavy1
    ▪ Are there use cases that can pay for the operational cost ?
    ▪ Do you have an operational system in place ?
    33
    There are other perspectives
    • System complexity
    • Resource consumption
    • Additional latency?
    1. This was the opinion in 2019. the operational cost now is improved (Added in May 2021)

    View Slide

  35. © Hitachi, Ltd. 2021. All rights reserved.
    Impressions of Use
    Useful, but
    the operational cost is heavy1
    ▪ Are there use cases that can pay for the operational cost ?
    ▪ Do you have an operational system in place ?
    34
    There are other perspectives
    • System complexity
    • Resource consumption
    • Additional latency?
    1. This was the opinion in 2019. the operational cost now is improved (Added in May 2021)

    View Slide

  36. © Hitachi, Ltd. 2021. All rights reserved.
    Benefits of Istio
    1. Transparency from services
    2. Kubernetes native
    3. An active community
    35

    View Slide

  37. © Hitachi, Ltd. 2021. All rights reserved.
    Benefit 1|Transparency from services
    No need to modify individual services
    ▪ Useful when using applications that are difficult to modify,
    such as OSS or existing products.
    ▪ Auto-Injection feature allows for automatic proxy insertion.
    ▪ Communication costs with the service team are also low (supposedly).
    Flexible and sophisticated traffic control
    ▪ Routing and access control by attributes of traffic and Kubernetes
    ▪ Controlling retries, timeouts, CORS, etc.
    that previously had to be implemented by each service
    36

    View Slide

  38. © Hitachi, Ltd. 2021. All rights reserved.
    Benefit 2|Kubernetes native
    All management is completed by custom resources
    ▪ We can use Kubernetes ecosystems such as Helm
    Istio itself also runs on Kubernetes
    ▪ Availability with replica and scalability with HPA
    ▪ NOTE: Beaware of the impact of Control Plane's load
    Declarative network management
    ▪ Manifests becomes the definition of the network configuration
    ▪ Network management with a GitOps
    37

    View Slide

  39. © Hitachi, Ltd. 2021. All rights reserved.
    Benefit 3|An active community
    Large and active community
    ▪ Average response time for issues: 5 days 1
    ▪ Bi-weekly level patch updates
    ▪ Stable quality since v1.1
    A lot of information
    ▪ Official Documents, Issues, Books, Blogs
    ▪ However, they also become obsolete quickly.
    38
    1 Istio Project Dashboards
    https://istio.teststats.cncf.io/d/11/issues-age-by-repository-group?orgId=1

    View Slide

  40. © Hitachi, Ltd. 2021. All rights reserved.
    Impressions of Use
    Useful, but
    the operational cost is heavy1
    ▪ Are there use cases that can pay for the operational cost ?
    ▪ Do you have an operational system in place ?
    39
    There are other perspectives
    • System complexity
    • Resource consumption
    • Additional latency?
    1. This was the opinion in 2019. the operational cost now is improved (Added in May 2021)

    View Slide

  41. © Hitachi, Ltd. 2021. All rights reserved.
    Usage Notes of Istio
    1. Operating Difficulty
    2. Update every 3 months
    3. The boundary between services and platforms
    4. Memory usage of proxy
    5. Migration of authentication and authorization
    40

    View Slide

  42. © Hitachi, Ltd. 2021. All rights reserved.
    Note 1|Operating Difficulty
    Istio integrates traffic control, security and observability
    ▪ A lot of flexible features with a lot of custom resources
    ▪ Hard to ensure consistency among services
    ▪ Many demands for knowledge make operations dependent on personal skill
    Difficulty in isolating the cause of a failure
    ▪ Physics, Kubernetes, App,... many factors are involved
    ▪ Proxy Timeout frames Istio as the criminal
    ✓ Destination Apps is down → Proxy returns a timeout
    ✓ Network is down → Proxy returns timeout
    41

    View Slide

  43. © Hitachi, Ltd. 2021. All rights reserved.
    Solution 1| Internal Istio Information Sharing Wiki
    ▪ Create an internal wiki to
    operating Istio (No plan to
    open it to the public)
    ▪ Content
    ✓ Overview
    ✓ Verification of
    Istio’s behavior
    ✓ Past failures
    ✓ Debugging tips
    ✓ etc.
    42

    View Slide

  44. © Hitachi, Ltd. 2021. All rights reserved.
    Note 2|Update every 3 months
    Istio releases every 3 months and support for only one previous version1
    ▪ Need to keep up with version upgrades
    ▪ The man-hours for verification and migration are significant.
    It is better to have one more person dedicated to this task
    Revisit configuration for updates
    ▪ Changes in authentication and authorization methods
    ▪ Changes of resource consumption
    43
    1. Istio Build & Release Cadence https://istio.io/about/release-cadence/

    View Slide

  45. © Hitachi, Ltd. 2021. All rights reserved.
    Solution 2|Istio-Operator
    Operator, which automates the operation of Istio1
    ▪ Currently, it's mainly installation and updates
    ✓ Provides a high-level API that hides
    the complexity of installation
    ✓ Improved robustness by applying
    validation and deprecated policies
    ▪ Discussion of multi-cluster collaboration is underway
    44
    1. Content as of 2019

    View Slide

  46. © Hitachi, Ltd. 2021. All rights reserved.
    Note 3|Boundaries between services and platforms
    Duplication of function between services and Istio
    ▪ Both on the app and Istio have functions such as timeout, retry, CORS
    ▪ Forcing Istio will cause a backlash from service teams
    ▪ “I don’t use Istio’s feature because we have already implemented it”
    ▪ “I use an OSS requires HTTPS. Don’t terminate HTTPS with a proxy”
    Who can manage Istio
    ▪ Some Istio manifest is applied to the entire cluster (e.g., DestinationRule)
    ▪ Service teams can deploy this at will → Cluster failure
    ▪ But some configs (such as routing) should be handled by service teams
    45

    View Slide

  47. © Hitachi, Ltd. 2021. All rights reserved.
    Solution 3|Helm chart
    ▪ Package the Istio configuration as a Helm Chart
    ✓ Only the functions that you want the service team to operate
    configuration items (values).
    ✓ Port number and protocol to be published
    ✓ Timeout, Retry, etc.
    ▪ The service team can configure features via the chart
    46

    View Slide

  48. © Hitachi, Ltd. 2021. All rights reserved.
    Note 4 | Memory usage of proxy
    ▪ How does each proxy know the destination ?
    → Each proxy has an address list of all proxy calculated in Control Plane
    ▪ In our env., the list grows linier to the number of pods
    ✓ The large number of Pods resulted in a bloated list
    ✓ At that time, each proxy used 750MB memory by 100 Pods
    ✓ Memory usage in the entire cluster : 1000 x 750MB = 750GB
    ✓ OOM Killer will crash Pods and Nodes
    ✓ In addition, the trend of resource usage changes by version
    → Performance evaluation is required for each Istio update for sizing
    47

    View Slide

  49. © Hitachi, Ltd. 2021. All rights reserved.
    Solution 4|istio-bench, our benchmark tool
    Tool for calculating resource usage based on the number of pods
    https://github.com/Hitachi/istio-bench
    ▪ Deploy 50 dummy Pods and monitor resource usage of Istio
    ▪ Iterate above and calculate the approximation equation of resource usage
    48

    View Slide

  50. © Hitachi, Ltd. 2021. All rights reserved.
    Solution 4|(Reference) Changes in proxy memory usage1
    49
    ▪ Memory usage of each proxy at 1000 Pods
    ▪ Drastically reduced in v1.0 and slowly reducing since then.
    771MB
    268MB
    163MB
    145MB 127MB 120MB 163MB 163MB
    0
    200
    400
    600
    800
    1000
    1.0.2 1.0.7 1.1.7 1.2.9 1.3.2 1.4.5 1.5.0 1.6.2
    Memory usage[MB]
    Istio version
    1. Updated the chart in April 2020

    View Slide

  51. © Hitachi, Ltd. 2021. All rights reserved.
    Solution 4|SideCar Resource
    ▪ Resources for direct control of proxies
    ▪ the size of the destination list can be fixed by limiting the destinations
    ▪ The level of abstraction is finer than others. Pending introduction
    50
    apiVersion: networking.istio.io/v1alpha3
    kind: Sidecar
    metadata:
    name: clusters-reducer
    namespace: default
    Spec:
    egress:
    - hosts:
    - "default/*"
    - "istio-system/*"
    Sidecar https://istio.io/docs/reference/config/networking/sidecar/

    View Slide

  52. © Hitachi, Ltd. 2021. All rights reserved.
    Note 5 | Migration of authentication and authorization
    Applying Istio's functions to the entire system is a bit risky.
    But Applying Istio only partially complicates AuthN/AuthZ
    ▪ Authentication
    ✓ mTLS: Both client and server authenticate each other
    ✓ Pods out of Istio mesh cannot use mTLS
    ✓ Need to consider whether Istio is already applied to peer
    ▪ Authorization
    ✓ Using K8s attributes in AuthZ require mTLS on both peers
    ✓ Need to consider the AuthZ method for each service out of Istio mesh
    51

    View Slide

  53. © Hitachi, Ltd. 2021. All rights reserved.
    Solution 5|Organize authentication patterns
    Organize conditions to communication according to whether Istio is applied
    and reflect them in the Helm Chart described above.
    52
    # Source Destination Conditions to communication
    1
    Istio
    Applied
    Istio
    Applied
    Both sender and receiver must have TLS settings
    Enabled or Disabled.
    2
    Istio
    Applied
    ー The TLS setting is disabled on the source
    3 ー
    Istio
    Applied
    The TLS setting on the destination is disabled
    or PERMISSIVE mode
    4 ー ー
    Unconditional
    But no access control can be applied

    View Slide

  54. © Hitachi, Ltd. 2021. All rights reserved.
    Solution 5|Share common errors and their causes
    Summarize common errors and their causes
    53
    # Error message Causes
    1
    upstream connect error
    or disconnect/reset
    before header
    ✓ Proxy is down or not working
    ✓ Proxy is not PID1, so it is not be restarted
    2
    Connection reset by
    peer
    ✓ Problem with authentication
    ✓ One peer is mTLS and the other is plain
    3 403 access denied
    ✓ Inconsistent AuthZ or the sender is not via a proxy
    ✓ If the receiver proxy log with debug level includes a
    namespace data, the sender go through a proxy

    View Slide

  55. © Hitachi, Ltd. 2021. All rights reserved.
    Solution 5|Automatic mutual TLS (Alpha)
    ▪ Features introduced in Istio 1.4
    ▪ Automatically configure the source mTLS setting
    ✓ Both of peers have a proxy → mTLS
    ✓ Neither has a proxy → Plain text
    ▪ The mTLS setting on the destination is set manually
    54
    Automatic mutual TLS
    https://istio.io/docs/tasks/security/authentication/auto-mtls/
    Pod
    Container
    Proxy
    Pod
    Container
    Proxy
    mTLS
    Pod
    Container
    Proxy
    Pod
    Container
    Plain rext

    View Slide

  56. © Hitachi, Ltd. 2021. All rights reserved.
    Impressions of Use
    Useful, but
    the operational cost is heavy1
    ▪ Are there use cases that can pay for the operational cost ?
    ▪ Do you have an operational system in place ?
    55
    There are other perspectives
    • System complexity
    • Resource consumption
    • Additional latency?
    1. This was the opinion in 2019. the operational cost now is improved (Added in May 2021)

    View Slide

  57. © Hitachi, Ltd. 2021. All rights reserved.
    Conclusion
    Istio is for Microservices operations
    ▪ Transparent and consistent communication management from the service
    ▪ With functions of communication management, security management,
    and observability
    Istio is Useful, but heavy operational load
    ▪ High functionality and flexible traffic control. Kubernetes native, etc.
    ▪ However, the operational cost is heavy, and practices are still developing.
    ▪ Is there a use case that pays for the cost ?
    56

    View Slide

  58. © Hitachi, Ltd. 2021. All rights reserved.
    Trademarks
    ▪ Istio is a registered trademark or trademark
    of Google LLC in the United States and other
    countries.
    ▪ Envoy Proxy is a registered trademark or
    trademark of The Linux Foundation in the
    United States and/or other countries.
    ▪ Kubernetes is a registered trademark or
    trademark of The Linux Foundation in the
    United States and/or other countries.
    ▪ Prometheus is a registered trademark or
    trademark of The Linux Foundation in the
    United States and/or other countries.
    ▪ Grafana is a registered trademark or
    trademark of Grafana Labs in the United
    States and/or other countries.
    ▪ Jaeger is a registered trademark or
    trademark of The Linux Foundation in the
    United States and/or other countries.
    ▪ Kiali is a registered trademark or trademark
    of Red Hat, Inc. in the United States and/or
    other countries.
    ▪ All other company names, product names,
    service names, and other proper nouns are
    trademarks or registered trademarks of their
    respective companies.
    ▪ TM and meta marks are not indicated in the
    text and figures in this presentation.
    57

    View Slide

  59. View Slide