E2E Observability for Connected Vehicle Services Including 5G Cellular Network U-Plane Troubles
I gave this talk at OSSJ2023 (Open Source Summit Japan 2023).
The primary distribution site of this presentation is:
https://sched.co/1Tyrm Talk video is also available there.
Introduction – Overview of E2E Communication of Connected Vehicles – Challenges & Motivation Related Works / Former Works Problem Statement Resolutions and Ideas POC Future Works Summary 2
of E2E Communication of Connected Vehicles Trend : CASE – Connected, Autonomous, Shared/Service, Electric Data : Real-time/Batch processing, Data Accumulation CAN(Sensor data) , Camera data, … Server Side Architecture : Public Cloud, On-premise, Edge Offload… = Hybrid Cloud Connectivity : Cellular, WiFi, V2X, … 3 Internal Developers Customers Wi-Fi Cellular Mobile NW Backbone NW DynamicMap Data Processing(CAN/Camera) Data Accumulation Edge Offload Hybrid Cloud OTA, Center Driven Control, .. Edge Offload
and Motivation Challenges We can address Issues/Troubles in Vehicle and/or Center side. But… On Network Troubles, there are not many things we can do. We receive lots of customer calls claiming, but… It’s not easy to figure out even WHERE the network trouble occurred Motivation If we work with Network Service Providers(MNO), we can: Make the Connected Vehicle Service more robust Reduce E2E Service Level Recovery Time 4
from Connected Vehicles Service side On a trouble, Connected Vehicle Service Providers want to identify: Areas where vehicle communication is affected (Where) Vehicles that get affected from the trouble (Which) What kind of communication trouble? (What/How) Intermittent or Persistent trouble, for example When the trouble occurred and is expected to be resolved (When) 5 Excerpt from KubeConNA 2023 Presentation, “E2E Observability for Connected Vehicle Service via Distributed Tracing” Requirements from Mobile Network Operator side Systems with large number of users constantly connected Provided for IoT devices such as connected car Accountability for the failure impact and the extent to which customers were affected when the failure occurred
Activities – Related Works/Former Works Various Collaborations 6 Open Collaboration Partner Collaboration Open Source AGL, OpenChain, etc. AECC, 3GPP/ETSI, ISO, ASAM, Academic Organizations… KDDI, various vendors/servicers
Works AECC : Automotive Edge Computing Consortium (https://aecc.org) An Open Collaborative Activity for promoting Edge Computing focusing on Automotive Use Cases, with POCs 7 Vehicle System Connected Car Vehicle System Connected Car Vehicle System Connected Car Vehicle System Connected Car Cellular Network Edge Server Pre- processing Cloud Edge Server Pre- processing Moving logs Biometric Sensors Cameras Cruising Data Sensor Data Control Data Intelligent Driving Enterprise Network CAN
Works (2) – Application Layer Observability A POC System Overview of Connected Vehicles Service gNB#1 W A N Location #1 (on-premise) Location #2 (on-premise) Public Cloud #1 CAN Camera DynamicMap Others… LB Ops Subsystem Pseudo Vehicle Pseudo Vehicle (generator) Edge#1(Region #2) Auth GW Dispatcher Offload Process Slice#1 (Lat.) Auth GW For Fallback W A N Data Accumulation Edge#2(Region #2) Auth GW Dispatcher Offload Process UE#1 Slice#2 (B/W) UPF#2 UPF#1 Dedicated Line Pseudo Vehicle (generator) UE#1 Slice#1 (Lat.) Slice#2 (B/W) OCI /etc. Public Cloud#2 UPF#2 UPF#1 Fail Over gNB#2 gNB#2 gNB#1 Orchestr ation gNB#1 Dedicated Line/VPN Dedicated Line/VPN Dedicated Line 9
Works (2) – Application Layer Observability A POC System Overview of Connected Vehicles Service 10 gNB#1 W A N Location #1 (on-premise) Location #2 (on-premise) Public Cloud #1 CAN Camera DynamicMap Others… LB Ops Subsystem Peeudo Vehiclea Pseudo Vehicle (generator) Edge#1(Region #2) Auth GW Dispatcher Offload Process Slice#1 (Lat.) Auth GW For Fallback W A N Data Accumulation Edge#2(Region #2) Auth GW Dispatcher Offload Process UE#1 Slice#2 (B/W) UPF#2 UPF#1 Dedicated Line Pseudo Vehicle (generator) UE#1 Slice#1 (Lat.) Slice#2 (B/W) OCI /etc. Public Cloud#2 UPF#2 UPF#1 Fail Over gNB#2 gNB#2 gNB#1 Orchestr ation gNB#1 Dedicated Line/VPN Dedicated Line/VPN Dedicated Line Boxes with red dashed line are Subsystems in question
Works (2) – Application Layer Observability A Simplified View of a part of the Upper Layer System: Object Detection for movie data from vehicle onboard cameras Application : mTLS (NGINX) w/ OCSP + Spark + Kafka + TensorFlow + S3(compat. storage) Infra: Kubernetes(1.27.x), OpenStack(Yoga), Ubuntu 22.04(LTS) Ops Subsystem: Prometheus + ElasticSearch + Jaeger/OpenTelemetry object- detection-api movie-kafka divide-video image-kafka detect-image detect-service generator VMs pygen mTLS Front (ingress) Camera Data Processing Center Accumlation Center (*) Each pod is consists of multiple containers 11 Object Detection Subsystem (k8s) Ops Subsystem Metrics Log Trace Front Subsystem (k8s) Multiple Components running distributed manner Not so easy to monitor
Works (2) – Application Layer Observability An example of Distributed Trace of a POC application to process Camera data 13 Object Detection Subsystem Camera Data Upload Event TRIP : from Engine Start to Stop Camera Data Upload Operation Camera Data Upload Completion handler Invokes Object Detection Subsystem Asynchronously Current Effort: Trace Based Anomaly Detection & Prediction (To Appear Somewhere in ‘24)
Statement – Break Down Problems to be Addressed Application Layer Troubles are OK, But Need to address Network Troubles (including U-Plane) Assumptions MNO does health monitoring of C/U-Plane NFs User (Connected Vehicle Servicer) Side knows: Identifiers(IMSI(SUPI)…), Current Location, Route Plan (e.g., Destination, Schedule), … Motivations/Reasons Detailed Requirements If we can get U-Plane trouble information with UE granularity earlier, we can take Proactive Actions, sending/retrieving necessary information/command in prior, for example. Even if it’s difficult to forecast failures/troubles, failure location and vehicle location is normally apart. Thus, we can take Actions Proactively. Recap(slide5) : Requirements from Connected Vehicles Service side On a trouble, Connected Vehicle Service Providers want to identify: Areas where vehicle communication is affected (Where) Vehicles that get affected from the trouble (Which) What kind of communication trouble? (What/How) When the trouble was happened and is expected to be resolved (When) 14
U-Plane Quick Overview Many NF (Network Function) components NRF, AMF, SMF, UPF, gNodeB, UE, … Two Phases until up/down link Traffic (1) UE Registration Mostly C-Plane Addressed by the KDDI Team – Have a look at their slide deck (2) PDU Session Establishment U-Plane entities (UPF/gNodeB) are configured by C-Plane entities (AMF/SMF) 15 3GPP TS 23.502 version 15.3.0 Release 15
and Resolution TODO Get list of PDU Sessions per UPF/gNodeB basis, and extract affected our own SUPI(IMSI)s from them Initial Ideas (1) Get PDU Session Information (with SUPI(IMSI)) from NFs via REST API Could be called this function as a kind of NEF? (2) Use OpenTelemetry Automatic Instrumentation Two choices – OpenTelemetry otel-go-instrumentation (https://github.com/open-telemetry/opentelemetry- go-instrumentation) – Grafana Labs Beyla Automatic Instrumentation (https://github.com/grafana/beyla) KDDI Team took Manual Instrumentation approach to address non-HTTP 5GS communication(NGAP/PFCP) 16
and Resolution Reality (1) Free5GC NFs implementations do not return sufficient information (2) Golang (in which Free5GC is written) Automatic Instrumentation has limitations Even HTTP (not NGAP/PFCP) could not be instrumented enough Current Status / (Work Around) Resolution Monitor Log files of NFs (especially AMF and SMF), and extract necessary information in realtime A custom Log Parser of Free5GC NFs which can emit trace data with SUPI(IMSI) and to be correlated later 17
– Evaluation Result After receiving a UPF failure, confirmed to receive a set of the failed UPF identifier and affected UE identifiers (SUPI/IMSI) mapping information within a second Practically, this time lag should depend on heartbeat timeout of NFs Current Limitations Cannot handle gNodeB information (at the moment) In case of Free5GC + UERANSIM: – gNodeB failure can be recovered via gNodeB Hand-Over – UPF failures require InitRegistration again Cannot handle tiered UPF (w/I-UPF) topology (at the moment) 25
In-Progress & Next Step 1. (Work In-Progress) Blush-up Design and Implementation of the Prototype Resolve restrictions, dashboard, Integration with other Observability Stack Or try other 5GS stack implementations (OAI etc.) 2. (Middle Term) Explore more sophisticated Information Retrieval (& Workflow) Best way is to rely on Standardized API of 3GPP/ETSI (if possible) 3. (Middle Term) Improve Performance & Scalability Up to millions of vehicles out of tens of millions of MNO subscribers Failure Notification Latency (less than a minute) 4. (Long Term) Propose to Telco Equipment Vendors/Integrators/MNOs Explore 3GPP spec. and Contribute to Free5GC or try other 5GS Open Source Stacks 26
Proved it’s possible to Notify Users UE granularity U-Plane Failure Information With the notification, Connected Vehicle Service Providers can take actions Proactively if the failure is ahead of the of Vehicles Without any modifications to 3GPP protocol This time, watched log lines sequence and extracted UE/U-Plane Function mapping (but to be improved more) In General Observability via Distributed Tracing is quite Useful and Important Contributed 5GS Open Source Stack (and others) via reporting & posting fixes 27
Span Links – Record Correlation across OpenTelemetry Spans In case not a direct child, Links can be used to record indirect relationship (Such as Causal relationship) https://opentelemetry.io/docs/concepts/signals/traces/#span-links 28 Trace (1) Trace (2) Point Other Trace(s) as Link(s) References (List of Links) Useful for aggregating traces across system domains (e.g. Network and Application)
RLS : Radio Link Simulation An encapsulation of Cellular Traffic over UDP UERANSIM internal protocol Wireshark dissector bundled 29 https://github.com/louisroyer/RLS-wireshark-dissector RLS of UERANSIM (over UDP) is dissected and displayed Application Traffic through UE using the Assigned IP address from 5GS
thatsdone
qbsupportinfo
netmarkjp
masatoshi0205
caarlos0
onishi
laysakura
ojima_h
tanakarian
soh9834
pal4de
tnoho
taikis
ammeep
reverentgeek
bryan
addyosmani
palkan
tammyeverts
62gerente
lynnandtonic
eileencodes
nonsquared
garrettdimon
