Integrating Service Mesh with Kubernetes-based connected vehicle platform

Integrating Service Mesh with Kubernetes-based connected vehicle platform

July Tech Festa 2020

5bc984d78def65048378c439d77c379f?s=128

Tomoya Amachi

July 25, 2020
Tweet

Transcript

  1. June 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Integrating Service Mesh with Kubernetes-based connected vehicle platform DENSO CORPORATION Digital Innovation Div JunKai Yong
 Tomoya Amachi / @tomoyamachi
  2. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Agenda 2 Introduction of DENSO and our team What we are building Technical details and demonstration
  3. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Introduction JunKai Yong/@yjk_miloaddict 3 Software Engineer@DENSO CORPORATION Digital Innovation Div. Graduated from Nanyang Technological University (NTU) in May 2018 Joined DENSO in Oct 2018. Working on IoT and Service-Mesh related topics
  4. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Introduction Tomoya Amachi/@tomoyamachi 4 Software Engineer Joined misaki project as part of CREATIONLINE team
  5. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. About DENSO 5 https://www.denso.com/global/en/
  6. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. 6 As a global manufacturer of automotive parts, DENSO is expanding its business operations around the world. DENSO supplies a wide range of products and systems as a Tier 1* supplier trusted by the world’s automobile manufacturers. World’s No.2 Consolidated net sales of 5.2 trillion yen. About DENSO
  7. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Our team DENSO + Creationline = Our team! 7
  8. June 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Abstracts of the Misaki
  9. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. 9 Why it is difficult to develop vehicle applications? Applications on vehicle is becoming more and more important! Develop application on vehicles
  10. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Develop application on vehicles 10 There are many ECU(Electric control unit)s in a car. rade r accessmete r rain sensor camera battery sensor ECU Difficult to Develop - Need to learn about embedded systems.
  11. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Difficult to Develop - Need to learn about embedded systems. 11 There are many ECU(Electric control unit)s in a car. Not using linux in most cases. CPU/Memory resources are limited and distributed. Develop application on vehicles
  12. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. No easy way to update applications on many cars simultaneously. 12 It's not always possible to run the application on your PC as it is. Develop application on vehicles Difficult to Develop - Need to learn about embedded systems.
  13. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Network connection can be unstable Because cars are moving around! 13 If connection breaks.. Cars are like highly distributed system. Developers have to deal with network disconnection for their apps. (retries/store data locally…) Develop application on vehicles Difficult to Develop - Unstable Network.
  14. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. But application on vehicle is becoming more and more important! 14 It’s important to handle data on edge and and keep updating the application to minimize communications expenses. e.g: Picture / Video / Traffic Data / Metadata The Global Connected Car Market size is projected to reach USD 212.7 billion by 2027, from an estimated value of USD 42.6 billion in 2019, at a CAGR of 22.3%. Current vehicles are like feature phone at 2000s. Smart "vehicle" era is coming soon. "Applications" will be a selling point of smart vehicles same as smart phone. Develop application on vehicles https://www.marketsandmarkets.com/Market-Reports/connected-car-market-102580117.html
  15. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. What we are building? Kubernetes based connected vehicle platform! 15 Minimize the need to learn about embedded systems. Easy to deploy/update application to vehicles. Remove the need for application developers to handle network issues. It's still a prototype.
  16. June 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Architecture Overview
  17. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Architecture Overview 17 App Edge Cloud App K8s Agent VPN VPN Applying Manifest Choose HELM Chart Generate K8s Manifest List Share Ideal Manifest b/w Edge and Cloud DigitalTwin API UI K8s Agent
  18. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Why Kubernetes? Flexibility of resources 18 radar accessmete r rain sensor battery sensor ECU ECUs will unified into several High Performance ECU in the future. We can use it as k8s worker node.
  19. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Flexibility of resources 19 radar accessmete r rain sensor battery sensor High Performance ECU Easy to orchestrate and manage ECU resources. These ECUs will be separated from the basic operation of the car (running, stopping, turning) Why Kubernetes?
  20. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. High performance ECU: NVIDIA Jetson AGX Xavier Developer Kit 20 https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-xavier-nx/ Why Kubernetes?
  21. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. 21 Edge A Cloud High Spec Car We can use resources on cloud. We can run same applications even if edge resources are not enough We can run container image in different vehicles and cloud. Flexibility of resources B C VPN A Low Spec Car B C VPN Why Kubernetes?
  22. June 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Technical Details
  23. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Components of Misaki Misaki 23 Misaki Orchestrator Misaki Service mesh
  24. June 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Misaki Orchestrator For more information: https://youtu.be/2x7jQTBUT5w
  25. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Technical details about Misaki orchestrator 25 App Edge Cloud App K8s Agent VPN Apply Manifest To Cluster Choose HELM Chart Generate K8s Manifest List Share Ideal Manifest b/w Edge and Cloud DigitalTwin API UI 4 3 2 1 4 components
  26. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. 26 Misaki orchestrator Applications Repository Data Processcor Data Uploader Sensory data APP1 APP2 APP3 APP4 APP1 APP2 APP3 APP4 https://www.flaticon.com/authors/vitaly- gorbachev https://www.flaticon.com/authors/ xnimrodx https://www.flaticon.com/authors/good- ware How it works:
  27. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. 27 Demo: Misaki orchestrator Video scenario GPS manager Can manager Deleted Service: CAN-uploader-for-prius-phv Pod 3 Pod 4 Pod 1 Pod 2 Local Broker Redis to Ambass GPS manager Can manager New service deployed: via-containers-for-prius Pod 3 Pod 4 Pod 1 Pod 2 Local Broker Redis to Ambass Pod 5 File uploader Pod 6 Ring Buffer Pod 7 Steady video Pod 8 Video segment
  28. June 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Misaki Service-Mesh
  29. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. What is Service Mesh? 29 Traffic control (load balancing, routing rules, failovers etc...) Policy control Secure communications Service monitoring (metrics, logs, traces) https://www.abhishek-tiwari.com/a-sidecar-for-your-service-mesh/ A dedicated infrastructure layer mainly for microservices
  30. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Service Mesh Architecture 30 https://servicemesh.es/
  31. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Why Service Mesh? 31 App Proxy Service-mesh decouples network concerns from applications. https://philcalcado.com/img/service-mesh/mesh1.png https://qiita.com/Ladicle/items/4ba57078128d6affadd5 Application developers can focus on their business values/logics Applications no longer needs to implement distributed system practices (i.e timeouts, service discoveries etc.)
  32. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Technical Detail about Misaki Service Mesh Overview 32 Misaki Control Plane Misaki Data Plane
  33. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Technical Detail about Misaki Service Mesh Architecture 33 • Provides policies and configurations to each Envoy • Connects individual envoys into a distributed system. Routing the application to appropriate service(s) Envoy: Cloud node Edge node Queue Filter Queue Filter Misaki Control Plane App A App B App C Proxy
  34. June 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Misaki Control Plane
  35. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Technical Detail about Misaki Control Plane What is needed to deploy an app? 35 K8s deployment manifest Envoy Custom Resource Definitions (CRD)
  36. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Technical Detail about Misaki Control Plane Deployment Manifest 36 Create, modify or delete K8s resources (i.e pods, deployment) Commonly written in .yaml - App - Envoy side-car - Control plane configmap - Initialization settings
  37. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Technical Detail about Misaki Control Plane Custom Resource Definition (CRD) 37 Custom k8s API endpoint that stores a collection of API objects I.E route CRD contains route CRD contains all the route information values
  38. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Technical Detail about Misaki Control Plane Misaki Control Plane 38 Custom k8s controller Assign appropriate settings to envoy side-car of newly deployed application, or when changes are made to an existing application
  39. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Technical Detail about Misaki Control Plane Misaki Control Plane 39 App C Proxy New k8s deployment Misaki Control Plane App C CRD Obtain Pod.Spec information 1 Retrieve App C’s CRD Generates envoy setting snapshot and sets it to the envoy side-car of the new app 3 2 App B Proxy App A Proxy
  40. June 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Misaki Data Plane
  41. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. What’s going on normal service-mesh Data Plane? 41 https://www.slideshare.net/ThomasGraf5/accelerating-envoy-and-istio-with-cilium-and-the-linux-kernel
  42. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. What’s going on Misaki Data Plane? 42 Buffering HTTP Request from application when network disconnect or not applicable network interface(3g, 4g, wifi etc). Re-sends the HTTP Request/streaming-data to server when network available again. Store HTTP Request from application if network did not recover during buffering duration.
  43. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Prototype Outbounds on Misaki Data Plane 43 volume App Proxy Proxy Server Online 1 Offline 2 volume App Proxy Proxy Server Name resolve in /etc/hosts Recover 3 volume App Proxy Proxy Server
  44. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Problems 44 Server footprint is not small Not supported Dynamic DNS resolve Network security problems
  45. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Current Outbounds on Misaki Data Plane 45 volume App Proxy volume App Proxy Local DNS Local DNS Online 1 Offline 2 Recover during buffering time 3 App Proxy Local DNS
  46. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Current Outbounds on Misaki Data Plane 46 Recover after buffering time 4 volume App Proxy Local DNS Replay Async response volume App Proxy Local DNS Replay Next request 5
  47. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Troubles 47 Envoy does not suppose offline communications. We need to customize Envoy code (written by C++11). Network security problems Create an original DNS Resolver (based on tomoyamachi/go-dnsmasq) All component images need to build for cross platforms(ARM, AMD etc).
  48. June 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Summary
  49. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Summary 49 We developed a k8s-based connected vehicle platform prototype Misaki service-mesh removes the need for developers to handle network problems But there is still many challenges to be solved.
  50. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Challenges 50 Misaki service-mesh currently only support simple HTTP/1.1 and HTTP/2.0. We want it to support more protocols (WebSocket, gRPC etc) Misaki control plane needs to be more flexible Our journey is just beginning! Minimizing cpu/memory usage on edge devices.
  51. May 2020 / Digital Innovation, Engineering Research & Development ŏ

    DENSO CORPORATION All Rights Reserved. Visit us 51 misaki-io currently empty, but we have a plan to make it open source this year Thank You