Integrating Service Mesh with Kubernetes-based connected vehicle platform

June 2020 / Digital Innovation, Engineering Research & Development
ŏ DENSO CORPORATION All Rights Reserved.
Integrating Service Mesh
with Kubernetes-based
connected vehicle platform
DENSO CORPORATION Digital Innovation Div
JunKai Yong 
Tomoya Amachi / @tomoyamachi

View Slide

May 2020 / Digital Innovation, Engineering Research & Development
Agenda
2
Introduction of DENSO and our team
What we are building
Technical details and demonstration

View Slide

Introduction
JunKai Yong/@yjk_miloaddict
3
Software Engineer@DENSO CORPORATION
Digital Innovation Div.
Graduated from Nanyang Technological University
(NTU) in May 2018
Joined DENSO in Oct 2018.
Working on IoT and Service-Mesh related
topics

View Slide

Introduction
Tomoya Amachi/@tomoyamachi
4
Software Engineer
Joined misaki project as part of CREATIONLINE team

View Slide

About DENSO
5
https://www.denso.com/global/en/

View Slide

6
As a global manufacturer of automotive parts, DENSO is expanding its
business operations around the world.
DENSO supplies a wide range of products and systems
as a Tier 1* supplier trusted by the world’s automobile
manufacturers.
World’s No.2
Consolidated net sales of 5.2 trillion yen.
About DENSO

View Slide

Our team
DENSO + Creationline = Our team!
7

View Slide

Abstracts of the Misaki

View Slide

9
Why it is difficult to develop vehicle
applications?
Applications on vehicle is becoming more and more important!
Develop application
on vehicles

View Slide

Develop application
on vehicles
10
There are many ECU(Electric control unit)s in a
car.
rade
r
accessmete
r
rain
sensor
camera
battery
sensor
ECU
Difficult to Develop - Need to learn about embedded systems.

View Slide

11
There are many ECU(Electric control unit)s in a
car.
Not using linux in most
cases.
CPU/Memory resources are limited and
distributed.
Develop application
on vehicles

View Slide

No easy way to update applications on many cars
simultaneously.
12
It's not always possible to run the application on your
PC as it is.
Develop application
on vehicles

View Slide

Network connection can be unstable
Because cars are moving around!
13
If connection
breaks..
Cars are like highly distributed
system.
Developers have to deal with
network
disconnection for their apps.
(retries/store data locally…)
Develop application
on vehicles
Difficult to Develop - Unstable Network.

View Slide

But application on vehicle is becoming more and more important!
14
It’s important to handle data on edge and and keep updating the application
to minimize communications expenses.
e.g: Picture / Video / Traffic Data / Metadata
The Global Connected Car Market size is projected to reach USD 212.7 billion by
2027, from an estimated value of USD 42.6 billion in 2019, at a CAGR of 22.3%.
Current vehicles are like feature phone at 2000s. Smart "vehicle" era is coming
soon. "Applications" will be a selling point of smart vehicles same as smart phone.
Develop application
on vehicles
https://www.marketsandmarkets.com/Market-Reports/connected-car-market-102580117.html

View Slide

What we are building?
Kubernetes based connected vehicle platform!
15
Minimize the need to learn about embedded
systems.
Easy to deploy/update application to vehicles.
Remove the need for application developers to
handle network issues.
It's still a
prototype.

View Slide

Architecture Overview

View Slide

Architecture Overview
17
App
Edge Cloud
App
K8s
Agent
VPN
VPN
Applying
Manifest
Choose
HELM
Chart
Generate
K8s
Manifest
List
Share
Ideal
Manifest
b/w Edge and Cloud
DigitalTwin API UI
K8s
Agent

View Slide

Why Kubernetes?
Flexibility of resources
18
radar
accessmete
r
rain
sensor
battery
sensor
ECU
ECUs will unified into several High Performance ECU in the future. We can use it
as k8s worker node.

View Slide

19
radar
accessmete
r
rain
sensor
battery
sensor
High Performance
ECU
Easy to orchestrate and manage ECU resources.
These ECUs will be separated from the basic operation of the car (running,
stopping, turning)
Why Kubernetes?

View Slide

High performance ECU:
NVIDIA Jetson AGX Xavier Developer Kit
20
https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-xavier-nx/
Why Kubernetes?

View Slide

21
Edge
A
Cloud
High Spec Car
We can use resources on cloud.
We can run same applications
even if edge resources are not enough
We can run container image
in different vehicles and cloud.
B C
VPN
A
Low Spec Car
B C
VPN
Why Kubernetes?

View Slide

Technical Details

View Slide

Components of Misaki
Misaki
23
Misaki Orchestrator
Misaki Service mesh

View Slide

Misaki Orchestrator
For more information:
https://youtu.be/2x7jQTBUT5w

View Slide

Technical details about
Misaki orchestrator
25
App
Edge Cloud
App
K8s
Agent
VPN
Apply
Manifest
To
Cluster
Choose
HELM
Chart
Generate
K8s
Manifest
List
Share
Ideal
Manifest
b/w Edge and Cloud
DigitalTwin API UI
4 3 2 1
4 components

View Slide

26
Misaki orchestrator
Applications

Repository
Data
Processcor
Data
Uploader
Sensory data
APP1
APP2
APP3
APP4
APP1
APP2
APP3
APP4
https://www.flaticon.com/authors/vitaly-
gorbachev
https://www.flaticon.com/authors/
xnimrodx
https://www.flaticon.com/authors/good-
ware
How it works:

View Slide

27
Demo: Misaki orchestrator Video scenario
GPS manager
Can manager
Deleted Service:
CAN-uploader-for-prius-phv
Pod 3 Pod 4
Pod 1 Pod 2
Local Broker Redis to Ambass
GPS manager
Can manager
New service deployed:
via-containers-for-prius
Pod 3
Pod 4
Pod 1 Pod 2
Local Broker
Redis to Ambass
Pod 5
File uploader
Pod 6
Ring Buffer
Pod 7
Steady video
Pod 8
Video
segment

View Slide

Misaki Service-Mesh

View Slide

What is Service
Mesh?
29
Traffic control (load balancing, routing rules, failovers etc...)
Policy control
Secure communications
Service monitoring (metrics, logs, traces)
https://www.abhishek-tiwari.com/a-sidecar-for-your-service-mesh/
A dedicated infrastructure layer mainly for microservices

View Slide

Service Mesh
Architecture
30
https://servicemesh.es/

View Slide

Why Service Mesh?
31
App Proxy
Service-mesh decouples network concerns
from applications.
https://philcalcado.com/img/service-mesh/mesh1.png
https://qiita.com/Ladicle/items/4ba57078128d6affadd5
Application developers can focus on their
business values/logics
Applications no longer needs to implement
distributed system practices (i.e timeouts, service
discoveries etc.)

View Slide

Technical Detail about
Misaki Service Mesh
Overview
32
Misaki Control Plane
Misaki Data Plane

View Slide

Misaki Service Mesh
Architecture
33
• Provides policies
and configurations
to each Envoy
• Connects individual
envoys into a
distributed system.
Routing the application
to appropriate service(s)
Envoy:
Cloud node
Edge node
Queue
Filter
Queue
Filter
Misaki Control
Plane
App A App B
App C
Proxy

View Slide


View Slide

What is needed to deploy an app?
35
K8s deployment manifest
Envoy Custom Resource Definitions (CRD)

View Slide

Deployment Manifest
36
Create, modify or delete K8s resources
(i.e pods, deployment)
Commonly written in .yaml
- App
- Envoy side-car
- Control plane configmap
- Initialization settings

View Slide

Custom Resource Definition (CRD)
37
Custom k8s API endpoint that stores a collection of API
objects
I.E route CRD contains route CRD contains all the route information values

View Slide

38
Custom k8s controller
Assign appropriate settings to envoy side-car of newly deployed application,
or when changes are made to an existing application

View Slide

39
App C Proxy
New k8s
deployment
Misaki Control
Plane
App C
CRD
Obtain Pod.Spec
information
1
Retrieve App C’s CRD
Generates envoy setting
snapshot and sets it to the
envoy side-car of the new app
3
2
App B Proxy
App A Proxy

View Slide

Misaki Data Plane

View Slide

What’s going on
normal service-mesh Data Plane?
41
https://www.slideshare.net/ThomasGraf5/accelerating-envoy-and-istio-with-cilium-and-the-linux-kernel

View Slide

What’s going on
Misaki Data Plane?
42
Buffering HTTP Request from application when network disconnect or not
applicable network interface(3g, 4g, wifi etc).
Re-sends the HTTP Request/streaming-data to server when network available
again.
Store HTTP Request from application if network did not recover during buffering
duration.

View Slide

Prototype Outbounds
on Misaki Data Plane
43
volume
App Proxy Proxy Server
Online
1 Offline
2
volume
Name resolve
in /etc/hosts
Recover
3
volume

View Slide

Problems
44
Server footprint is not small
Not supported Dynamic DNS resolve
Network security problems

View Slide

Current Outbounds
45
volume
App Proxy
volume
App Proxy
Local DNS Local DNS
Online
1 Offline
2 Recover during buffering time
3
App Proxy
Local DNS

View Slide

Current Outbounds
46
Recover after buffering time
4
volume
App Proxy
Local DNS
Replay
Async response
volume
App Proxy
Local DNS
Replay
Next request
5

View Slide

Troubles
47
Envoy does not suppose offline communications.
We need to customize Envoy code (written by C++11).
Network security problems
Create an original DNS Resolver (based on tomoyamachi/go-dnsmasq)
All component images need to build for cross platforms(ARM, AMD etc).

View Slide

Summary

View Slide

Summary
49
We developed a k8s-based connected vehicle platform prototype
Misaki service-mesh removes the need for developers to
handle network problems
But there is still many challenges to be
solved.

View Slide

Challenges
50
Misaki service-mesh currently only support simple HTTP/1.1 and HTTP/2.0.
We want it to support more protocols (WebSocket, gRPC etc)
Misaki control plane needs to be more flexible
Our journey is just beginning!
Minimizing cpu/memory usage on edge devices.

View Slide

Visit us
51
misaki-io
currently empty, but
we have a plan to
make it open source
this year
Thank You

View Slide

Integrating Service Mesh with Kubernetes-based connected vehicle platform

Integrating Service Mesh with Kubernetes-based connected vehicle platform

Tomoya Amachi

More Decks by Tomoya Amachi

Other Decks in Technology

Featured

Transcript