DENSO CORPORATION All Rights Reserved. Integrating Service Mesh with Kubernetes-based connected vehicle platform DENSO CORPORATION Digital Innovation Div JunKai Yong Tomoya Amachi / @tomoyamachi
DENSO CORPORATION All Rights Reserved. Introduction JunKai Yong/@yjk_miloaddict 3 Software Engineer@DENSO CORPORATION Digital Innovation Div. Graduated from Nanyang Technological University (NTU) in May 2018 Joined DENSO in Oct 2018. Working on IoT and Service-Mesh related topics
DENSO CORPORATION All Rights Reserved. 6 As a global manufacturer of automotive parts, DENSO is expanding its business operations around the world. DENSO supplies a wide range of products and systems as a Tier 1* supplier trusted by the world’s automobile manufacturers. World’s No.2 Consolidated net sales of 5.2 trillion yen. About DENSO
DENSO CORPORATION All Rights Reserved. 9 Why it is difficult to develop vehicle applications? Applications on vehicle is becoming more and more important! Develop application on vehicles
DENSO CORPORATION All Rights Reserved. Develop application on vehicles 10 There are many ECU(Electric control unit)s in a car. rade r accessmete r rain sensor camera battery sensor ECU Difficult to Develop - Need to learn about embedded systems.
DENSO CORPORATION All Rights Reserved. Difficult to Develop - Need to learn about embedded systems. 11 There are many ECU(Electric control unit)s in a car. Not using linux in most cases. CPU/Memory resources are limited and distributed. Develop application on vehicles
DENSO CORPORATION All Rights Reserved. No easy way to update applications on many cars simultaneously. 12 It's not always possible to run the application on your PC as it is. Develop application on vehicles Difficult to Develop - Need to learn about embedded systems.
DENSO CORPORATION All Rights Reserved. Network connection can be unstable Because cars are moving around! 13 If connection breaks.. Cars are like highly distributed system. Developers have to deal with network disconnection for their apps. (retries/store data locally…) Develop application on vehicles Difficult to Develop - Unstable Network.
DENSO CORPORATION All Rights Reserved. But application on vehicle is becoming more and more important! 14 It’s important to handle data on edge and and keep updating the application to minimize communications expenses. e.g: Picture / Video / Traffic Data / Metadata The Global Connected Car Market size is projected to reach USD 212.7 billion by 2027, from an estimated value of USD 42.6 billion in 2019, at a CAGR of 22.3%. Current vehicles are like feature phone at 2000s. Smart "vehicle" era is coming soon. "Applications" will be a selling point of smart vehicles same as smart phone. Develop application on vehicles https://www.marketsandmarkets.com/Market-Reports/connected-car-market-102580117.html
DENSO CORPORATION All Rights Reserved. What we are building? Kubernetes based connected vehicle platform! 15 Minimize the need to learn about embedded systems. Easy to deploy/update application to vehicles. Remove the need for application developers to handle network issues. It's still a prototype.
DENSO CORPORATION All Rights Reserved. Why Kubernetes? Flexibility of resources 18 radar accessmete r rain sensor battery sensor ECU ECUs will unified into several High Performance ECU in the future. We can use it as k8s worker node.
DENSO CORPORATION All Rights Reserved. Flexibility of resources 19 radar accessmete r rain sensor battery sensor High Performance ECU Easy to orchestrate and manage ECU resources. These ECUs will be separated from the basic operation of the car (running, stopping, turning) Why Kubernetes?
DENSO CORPORATION All Rights Reserved. 21 Edge A Cloud High Spec Car We can use resources on cloud. We can run same applications even if edge resources are not enough We can run container image in different vehicles and cloud. Flexibility of resources B C VPN A Low Spec Car B C VPN Why Kubernetes?
DENSO CORPORATION All Rights Reserved. 26 Misaki orchestrator Applications Repository Data Processcor Data Uploader Sensory data APP1 APP2 APP3 APP4 APP1 APP2 APP3 APP4 https://www.flaticon.com/authors/vitaly- gorbachev https://www.flaticon.com/authors/ xnimrodx https://www.flaticon.com/authors/good- ware How it works:
DENSO CORPORATION All Rights Reserved. 27 Demo: Misaki orchestrator Video scenario GPS manager Can manager Deleted Service: CAN-uploader-for-prius-phv Pod 3 Pod 4 Pod 1 Pod 2 Local Broker Redis to Ambass GPS manager Can manager New service deployed: via-containers-for-prius Pod 3 Pod 4 Pod 1 Pod 2 Local Broker Redis to Ambass Pod 5 File uploader Pod 6 Ring Buffer Pod 7 Steady video Pod 8 Video segment
DENSO CORPORATION All Rights Reserved. What is Service Mesh? 29 Traffic control (load balancing, routing rules, failovers etc...) Policy control Secure communications Service monitoring (metrics, logs, traces) https://www.abhishek-tiwari.com/a-sidecar-for-your-service-mesh/ A dedicated infrastructure layer mainly for microservices
DENSO CORPORATION All Rights Reserved. Why Service Mesh? 31 App Proxy Service-mesh decouples network concerns from applications. https://philcalcado.com/img/service-mesh/mesh1.png https://qiita.com/Ladicle/items/4ba57078128d6affadd5 Application developers can focus on their business values/logics Applications no longer needs to implement distributed system practices (i.e timeouts, service discoveries etc.)
DENSO CORPORATION All Rights Reserved. Technical Detail about Misaki Service Mesh Architecture 33 • Provides policies and configurations to each Envoy • Connects individual envoys into a distributed system. Routing the application to appropriate service(s) Envoy: Cloud node Edge node Queue Filter Queue Filter Misaki Control Plane App A App B App C Proxy
DENSO CORPORATION All Rights Reserved. Technical Detail about Misaki Control Plane What is needed to deploy an app? 35 K8s deployment manifest Envoy Custom Resource Definitions (CRD)
DENSO CORPORATION All Rights Reserved. Technical Detail about Misaki Control Plane Deployment Manifest 36 Create, modify or delete K8s resources (i.e pods, deployment) Commonly written in .yaml - App - Envoy side-car - Control plane configmap - Initialization settings
DENSO CORPORATION All Rights Reserved. Technical Detail about Misaki Control Plane Custom Resource Definition (CRD) 37 Custom k8s API endpoint that stores a collection of API objects I.E route CRD contains route CRD contains all the route information values
DENSO CORPORATION All Rights Reserved. Technical Detail about Misaki Control Plane Misaki Control Plane 38 Custom k8s controller Assign appropriate settings to envoy side-car of newly deployed application, or when changes are made to an existing application
DENSO CORPORATION All Rights Reserved. Technical Detail about Misaki Control Plane Misaki Control Plane 39 App C Proxy New k8s deployment Misaki Control Plane App C CRD Obtain Pod.Spec information 1 Retrieve App C’s CRD Generates envoy setting snapshot and sets it to the envoy side-car of the new app 3 2 App B Proxy App A Proxy
DENSO CORPORATION All Rights Reserved. What’s going on normal service-mesh Data Plane? 41 https://www.slideshare.net/ThomasGraf5/accelerating-envoy-and-istio-with-cilium-and-the-linux-kernel
DENSO CORPORATION All Rights Reserved. What’s going on Misaki Data Plane? 42 Buffering HTTP Request from application when network disconnect or not applicable network interface(3g, 4g, wifi etc). Re-sends the HTTP Request/streaming-data to server when network available again. Store HTTP Request from application if network did not recover during buffering duration.
DENSO CORPORATION All Rights Reserved. Prototype Outbounds on Misaki Data Plane 43 volume App Proxy Proxy Server Online 1 Offline 2 volume App Proxy Proxy Server Name resolve in /etc/hosts Recover 3 volume App Proxy Proxy Server
DENSO CORPORATION All Rights Reserved. Current Outbounds on Misaki Data Plane 45 volume App Proxy volume App Proxy Local DNS Local DNS Online 1 Offline 2 Recover during buffering time 3 App Proxy Local DNS
DENSO CORPORATION All Rights Reserved. Current Outbounds on Misaki Data Plane 46 Recover after buffering time 4 volume App Proxy Local DNS Replay Async response volume App Proxy Local DNS Replay Next request 5
DENSO CORPORATION All Rights Reserved. Troubles 47 Envoy does not suppose offline communications. We need to customize Envoy code (written by C++11). Network security problems Create an original DNS Resolver (based on tomoyamachi/go-dnsmasq) All component images need to build for cross platforms(ARM, AMD etc).
DENSO CORPORATION All Rights Reserved. Summary 49 We developed a k8s-based connected vehicle platform prototype Misaki service-mesh removes the need for developers to handle network problems But there is still many challenges to be solved.
DENSO CORPORATION All Rights Reserved. Challenges 50 Misaki service-mesh currently only support simple HTTP/1.1 and HTTP/2.0. We want it to support more protocols (WebSocket, gRPC etc) Misaki control plane needs to be more flexible Our journey is just beginning! Minimizing cpu/memory usage on edge devices.
tomoyamachi
mats16
satoshin
sinedied
korodroid
widstkyi
karukaru808
1ftseabass
chandi
maekawa123
kolinz
sayokomiura
bkeepers
shpigford
searls
reverentgeek
samanthasiow
destraynor
lynnandtonic
bryan
tanoku
kastner
deanohume
revolveconf
