Share
Embed
E2E Observability for Connected Vehicle Services Including 5G Cellular Network U-Plane Troubles
I gave this talk at OSSJ2023 (Open Source Summit Japan 2023).
The primary distribution site of this presentation is:
https://sched.co/1Tyrm
Talk video is also available there.
thatsdone
December 06, 2023
Transcript
30 slides
-
#OSSummit E2E Observability for Connected Vehicle Services including 5G Cellular
Network U-Plane Troubles Masanori Itoh, Toyota Motor Corporation Kota Endo, KDDI Corporation -
Copyright © 2023 TOYOTA MOTOR CORPORATION All rights reserved. Contents
Introduction – Overview of E2E Communication of Connected Vehicles – Challenges & Motivation Related Works / Former Works Problem Statement Resolutions and Ideas POC Future Works Summary 2 -
Copyright © 2023 TOYOTA MOTOR CORPORATION All rights reserved. Overview
of E2E Communication of Connected Vehicles Trend : CASE – Connected, Autonomous, Shared/Service, Electric Data : Real-time/Batch processing, Data Accumulation CAN(Sensor data) , Camera data, … Server Side Architecture : Public Cloud, On-premise, Edge Offload… = Hybrid Cloud Connectivity : Cellular, WiFi, V2X, … 3 Internal Developers Customers Wi-Fi Cellular Mobile NW Backbone NW DynamicMap Data Processing(CAN/Camera) Data Accumulation Edge Offload Hybrid Cloud OTA, Center Driven Control, .. Edge Offload -
Copyright © 2023 TOYOTA MOTOR CORPORATION All rights reserved. Challenges
and Motivation Challenges We can address Issues/Troubles in Vehicle and/or Center side. But… On Network Troubles, there are not many things we can do. We receive lots of customer calls claiming, but… It’s not easy to figure out even WHERE the network trouble occurred Motivation If we work with Network Service Providers(MNO), we can: Make the Connected Vehicle Service more robust Reduce E2E Service Level Recovery Time 4 -
Copyright © 2023 TOYOTA MOTOR CORPORATION All rights reserved. Requirements
from Connected Vehicles Service side On a trouble, Connected Vehicle Service Providers want to identify: Areas where vehicle communication is affected (Where) Vehicles that get affected from the trouble (Which) What kind of communication trouble? (What/How) Intermittent or Persistent trouble, for example When the trouble occurred and is expected to be resolved (When) 5 Excerpt from KubeConNA 2023 Presentation, “E2E Observability for Connected Vehicle Service via Distributed Tracing” Requirements from Mobile Network Operator side Systems with large number of users constantly connected Provided for IoT devices such as connected car Accountability for the failure impact and the extent to which customers were affected when the failure occurred -
Copyright © 2023 TOYOTA MOTOR CORPORATION All rights reserved. Our
Activities – Related Works/Former Works Various Collaborations 6 Open Collaboration Partner Collaboration Open Source AGL, OpenChain, etc. AECC, 3GPP/ETSI, ISO, ASAM, Academic Organizations… KDDI, various vendors/servicers -
Copyright © 2023 TOYOTA MOTOR CORPORATION All rights reserved. Related
Works AECC : Automotive Edge Computing Consortium (https://aecc.org) An Open Collaborative Activity for promoting Edge Computing focusing on Automotive Use Cases, with POCs 7 Vehicle System Connected Car Vehicle System Connected Car Vehicle System Connected Car Vehicle System Connected Car Cellular Network Edge Server Pre- processing Cloud Edge Server Pre- processing Moving logs Biometric Sensors Cameras Cruising Data Sensor Data Control Data Intelligent Driving Enterprise Network CAN -
Copyright © 2023 TOYOTA MOTOR CORPORATION All rights reserved. Former
Works (1) – 5GS C-Plane Observability “E2E Observability for Connected Vehicle Service via Distributed Tracing” (KubeCon NA 2023@Chicago) Ref : https://kccncna2023.sched.com/event/1R2oh Focus Points & Results 5GS C-Plane side Observability for Trouble Shooting using Distributed Tracing Point: Correlate 5G UE Identifier and Vehicle Identifier in C-Plane 5GS Stack / Observability Stack / Infra – Free5GC / UERANSIM / ONAP » https://free5gc.org/ » https://github.com/aligungr/UERANSIM » https://www.onap.org/ – OpenTelemetry, Grafana Loki/Tempo… » https://opentelemetry.io/ » https://grafana.com/oss/tempo/ – kubernetes / OpenStack… 8 -
Copyright © 2023 TOYOTA MOTOR CORPORATION All rights reserved. Former
Works (2) – Application Layer Observability A POC System Overview of Connected Vehicles Service gNB#1 W A N Location #1 (on-premise) Location #2 (on-premise) Public Cloud #1 CAN Camera DynamicMap Others… LB Ops Subsystem Pseudo Vehicle Pseudo Vehicle (generator) Edge#1(Region #2) Auth GW Dispatcher Offload Process Slice#1 (Lat.) Auth GW For Fallback W A N Data Accumulation Edge#2(Region #2) Auth GW Dispatcher Offload Process UE#1 Slice#2 (B/W) UPF#2 UPF#1 Dedicated Line Pseudo Vehicle (generator) UE#1 Slice#1 (Lat.) Slice#2 (B/W) OCI /etc. Public Cloud#2 UPF#2 UPF#1 Fail Over gNB#2 gNB#2 gNB#1 Orchestr ation gNB#1 Dedicated Line/VPN Dedicated Line/VPN Dedicated Line 9 -
Copyright © 2023 TOYOTA MOTOR CORPORATION All rights reserved. Former
Works (2) – Application Layer Observability A POC System Overview of Connected Vehicles Service 10 gNB#1 W A N Location #1 (on-premise) Location #2 (on-premise) Public Cloud #1 CAN Camera DynamicMap Others… LB Ops Subsystem Peeudo Vehiclea Pseudo Vehicle (generator) Edge#1(Region #2) Auth GW Dispatcher Offload Process Slice#1 (Lat.) Auth GW For Fallback W A N Data Accumulation Edge#2(Region #2) Auth GW Dispatcher Offload Process UE#1 Slice#2 (B/W) UPF#2 UPF#1 Dedicated Line Pseudo Vehicle (generator) UE#1 Slice#1 (Lat.) Slice#2 (B/W) OCI /etc. Public Cloud#2 UPF#2 UPF#1 Fail Over gNB#2 gNB#2 gNB#1 Orchestr ation gNB#1 Dedicated Line/VPN Dedicated Line/VPN Dedicated Line Boxes with red dashed line are Subsystems in question -
Copyright © 2023 TOYOTA MOTOR CORPORATION All rights reserved. Former
Works (2) – Application Layer Observability A Simplified View of a part of the Upper Layer System: Object Detection for movie data from vehicle onboard cameras Application : mTLS (NGINX) w/ OCSP + Spark + Kafka + TensorFlow + S3(compat. storage) Infra: Kubernetes(1.27.x), OpenStack(Yoga), Ubuntu 22.04(LTS) Ops Subsystem: Prometheus + ElasticSearch + Jaeger/OpenTelemetry object- detection-api movie-kafka divide-video image-kafka detect-image detect-service generator VMs pygen mTLS Front (ingress) Camera Data Processing Center Accumlation Center (*) Each pod is consists of multiple containers 11 Object Detection Subsystem (k8s) Ops Subsystem Metrics Log Trace Front Subsystem (k8s) Multiple Components running distributed manner Not so easy to monitor -
Copyright © 2023 TOYOTA MOTOR CORPORATION All rights reserved.
Observability Solution : OpenTelemetry https://openetelemetry.io/ Conceptual Components Aggregator, Collector, Pipeline, … Trace/Span, …, Metrics, Log Data Flow North-South : Export Trace data – Application Trace Aggregator (through Collectors) East-West : Pass Caller Information: Context Propagation – Application (Caller) Application (Callee) » A Span has to know its parent span (Caller) – ContextPropagation over HTTP is standardized at W3C (traceparent format) Inter Trace Correlation : Links (Not directly making a span as a child of caller, but add reference) Practical Design Point How to Instrument: Emit Trace Span and Propagate Context Information – Manual (w/Modifications to Applications) or Automatic (w/o Modifications to Applications) Collector Former Works (2) – Application Layer Observability Collector Aggregator Application #1 (Emits Trace Spans) Aggregator (non OTEL) Application #2 (Emits Trace Spans) 12 Pipeline Receiver Exporter Processor Instrumentation Instrumentation Context Propagation -
Copyright © 2023 TOYOTA MOTOR CORPORATION All rights reserved. Former
Works (2) – Application Layer Observability An example of Distributed Trace of a POC application to process Camera data 13 Object Detection Subsystem Camera Data Upload Event TRIP : from Engine Start to Stop Camera Data Upload Operation Camera Data Upload Completion handler Invokes Object Detection Subsystem Asynchronously Current Effort: Trace Based Anomaly Detection & Prediction (To Appear Somewhere in ‘24) -
Copyright © 2023 TOYOTA MOTOR CORPORATION All rights reserved. Problem
Statement – Break Down Problems to be Addressed Application Layer Troubles are OK, But Need to address Network Troubles (including U-Plane) Assumptions MNO does health monitoring of C/U-Plane NFs User (Connected Vehicle Servicer) Side knows: Identifiers(IMSI(SUPI)…), Current Location, Route Plan (e.g., Destination, Schedule), … Motivations/Reasons Detailed Requirements If we can get U-Plane trouble information with UE granularity earlier, we can take Proactive Actions, sending/retrieving necessary information/command in prior, for example. Even if it’s difficult to forecast failures/troubles, failure location and vehicle location is normally apart. Thus, we can take Actions Proactively. Recap(slide5) : Requirements from Connected Vehicles Service side On a trouble, Connected Vehicle Service Providers want to identify: Areas where vehicle communication is affected (Where) Vehicles that get affected from the trouble (Which) What kind of communication trouble? (What/How) When the trouble was happened and is expected to be resolved (When) 14 -
Copyright © 2023 TOYOTA MOTOR CORPORATION All rights reserved. 5GS
U-Plane Quick Overview Many NF (Network Function) components NRF, AMF, SMF, UPF, gNodeB, UE, … Two Phases until up/down link Traffic (1) UE Registration Mostly C-Plane Addressed by the KDDI Team – Have a look at their slide deck (2) PDU Session Establishment U-Plane entities (UPF/gNodeB) are configured by C-Plane entities (AMF/SMF) 15 3GPP TS 23.502 version 15.3.0 Release 15 -
Copyright © 2023 TOYOTA MOTOR CORPORATION All rights reserved. Ideas
and Resolution TODO Get list of PDU Sessions per UPF/gNodeB basis, and extract affected our own SUPI(IMSI)s from them Initial Ideas (1) Get PDU Session Information (with SUPI(IMSI)) from NFs via REST API Could be called this function as a kind of NEF? (2) Use OpenTelemetry Automatic Instrumentation Two choices – OpenTelemetry otel-go-instrumentation (https://github.com/open-telemetry/opentelemetry- go-instrumentation) – Grafana Labs Beyla Automatic Instrumentation (https://github.com/grafana/beyla) KDDI Team took Manual Instrumentation approach to address non-HTTP 5GS communication(NGAP/PFCP) 16 -
Copyright © 2023 TOYOTA MOTOR CORPORATION All rights reserved. Ideas
and Resolution Reality (1) Free5GC NFs implementations do not return sufficient information (2) Golang (in which Free5GC is written) Automatic Instrumentation has limitations Even HTTP (not NGAP/PFCP) could not be instrumented enough Current Status / (Work Around) Resolution Monitor Log files of NFs (especially AMF and SMF), and extract necessary information in realtime A custom Log Parser of Free5GC NFs which can emit trace data with SUPI(IMSI) and to be correlated later 17 -
Copyright © 2023 TOYOTA MOTOR CORPORATION All rights reserved.
5GS Stack Free5GC v3.3.0 w/go 1.21.1 + gtp5g UERANSIM git+3a96298 (’23/5/9) Ubuntu 22.04.1(amd64) Deployment 2 Slices per Tracking Area 2 Tracking Areas Simple VM deployment No containerization (for now) POC – Setup UPF#1-1 VM (free5gc-poc-up-11) 192.168.1.11) VM(free5gc-poc-ue-1) 192.168.63.1 uesimtun0 VM(free5gc-poc-cp-1) 192.168.0.11 NRF SMF#1 uesimtun1 SNAT boundary (iptables MASQUERADE) SMF#2 VM (free5gc- poc-cp-2) AMF UE#1 NRF Static NAT boundary (Floating IP) UPF#1-2 gNB#1-2 gNB#1-1 .1.211 upfgtp 18 upfgtp VM#3 (free5gc-poc-up-12) 192.168.1.12) .1.111 N11 : HTTP N3 : GTP-U (udp 2152) N4 : PFCP (udp 8805) N2 : NGAP (sctp 38412) Radio Link Simulation (udp 4997) Tracking Area #1 Tracking Area #2 Tracking Area #3 : : .1.212 .1.112 VM (free5gc- poc-cp-3) -
Copyright © 2023 TOYOTA MOTOR CORPORATION All rights reserved.
Procedure(1) Follow UERANSIM/Free5GC documentation Procedure(2) (TIP 0) (1) Create UEs without PDU Sessions (Initial Registration Only) Use a configuration with NULL ‘sessions:’ and ‘default-nssai:’ section (2) Initiate PDU Session Establishment Request (per slice) Using nr-cli, invoke PDU Session Establishment request TIP: Use double quotation(“) to group sub-command – https://github.com/aligungr/UERANSIM/wiki/Usage POC – Procedure 19 $ nr-cli imsi-208930000012345 --exec “ps-establish IPv4 0x01 0x010203 internet11” # 0x01 : sst (Slice and Service Type) # 0x010203 : sd (Slice Differenciator) # Internet11 : dnn UE configuration example --- supi: 'imsi- 208930000000003' mcc: '208' mnc: '93' (snip) gnbSearchList: - 192.168.1.211 (snip) sessions: configured-nssai: - sst: 0x01 sd: 0x010203 - sst: 0x01 sd: 0x112233 default-nssai: integrity: (snip) Make these sections empty Make these sections empty -
Copyright © 2023 TOYOTA MOTOR CORPORATION All rights reserved. POC
– Analysis Analysis – Log Based Analysis User Plane Function and UE association is managed by SMF 20 $ grep -E ¥(PduSess¥|GIN¥) 20231127-1-free5gc-pdu-session-success-smf.log time="2023-11-27T16:04:22.923623330+09:00" level="info" msg="Receive Create SM Context Request" CAT="PduSess" NF="SMF" time="2023-11-27T16:04:22.927806025+09:00" level="info" msg="In HandlePDUSessionSMContextCreate" CAT="PduSess" NF="SMF" time="2023-11-27T16:04:22.928808304+09:00" level="trace" msg="State[InActive] -> State[InActive]" CAT="PduSess" NF="SMF" pdu_session_id="1" supi="imsi-208930000000003" time="2023-11-27T16:04:22.930953619+09:00" level="trace" msg="State[InActive] -> State[ActivePending]" CAT="PduSess" NF="SMF" pdu_session_id="1" supi="imsi-208930000000003" time="2023-11-27T16:04:22.931518587+09:00" level="debug" msg="S-NSSAI[sst: 1, sd: 010203] DNN[internet11]" CAT="PduSess" NF="SMF" pdu_session_id="1" supi="imsi-208930000000003" time="2023-11-27T16:04:22.938366848+09:00" level="info" msg="Send NF Discovery Serving UDM Successfully" CAT="PduSess" NF="SMF" pdu_session_id="1" supi="imsi-208930000000003" time="2023-11-27T16:04:22.970142403+09:00" level="trace" msg="Send NF Discovery Serving AMF successfully" CAT="PduSess" NF="SMF" pdu_session_id="1" supi="imsi-208930000000003" time="2023-11-27T16:04:22.971051987+09:00" level="trace" msg="findPSAandAllocUeIP" CAT="PduSess" NF="SMF" pdu_session_id="1" supi="imsi-208930000000003" time="2023-11-27T16:04:22.975311711+09:00" level="info" msg="Allocated PDUAdress[10.241.0.1]" CAT="PduSess" NF="SMF" pdu_session_id="1" supi="imsi-208930000000003" time="2023-11-27T16:04:23.028520457+09:00" level="debug" msg="Install SessionRule[SessRuleId-1]: &{AuthSessAmbr:0xc00005ac80 AuthDefQos:0xc00005aca0 SessRuleId:SessRuleId-1 RefUmData: RefCondData:}" CAT="PduSess" NF="SMF" pdu_session_id="1" supi="imsi-208930000000003" time="2023-11-27T16:04:23.029753661+09:00" level="info" msg="Has no pre-config route" CAT="PduSess" NF="SMF" pdu_session_id="1" supi="imsi-208930000000003" time="2023-11-27T16:04:23.030725335+09:00" level="trace" msg="In AllocateLocalSEIDForDataPath" CAT="PduSess" NF="SMF" time="2023-11-27T16:04:23.031665952+09:00" level="trace" msg="NodeIDtoIP: 192.168.1.111" CAT="PduSess" NF="SMF" time="2023-11-27T16:04:23.032432749+09:00" level="trace" msg="In ActivateTunnelAndPDR" CAT="PduSess" NF="SMF" time="2023-11-27T16:04:23.033283688+09:00" level="trace" msg="DataPath Meta Information¥nActivated: false¥nIsDefault Path: true¥nHas Braching Point: false¥nDestination IP: ¥nDestination Port: ¥nDataPath Routing Information¥n1th Node in the Path¥nCurrent UPF IP: 192.168.1.111¥nPrevious UPF IP: None¥nNext UPF IP: None¥n" CAT="PduSess" NF="SMF" time="2023-11-27T16:04:23.035178231+09:00" level="trace" msg="Current DP Node IP: 192.168.1.111" CAT="PduSess" NF="SMF" time="2023-11-27T16:04:23.037023448+09:00" level="warning" msg="No Create URR" CAT="PduSess" NF="SMF" time="2023-11-27T16:04:23.038764297+09:00" level="trace" msg="Current DP Node IP: 192.168.1.111" CAT="PduSess" NF="SMF" time="2023-11-27T16:04:23.039538734+09:00" level="trace" msg="Before DLPDR OuterHeaderCreation" CAT="PduSess" NF="SMF" time="2023-11-27T16:04:23.041489662+09:00" level="info" msg="| 201 | 192.168.0.11 | POST | /nsmf-pdusession/v1/sm-contexts | " CAT="GIN" NF="SMF" time="2023-11-27T16:04:23.041964494+09:00" level="info" msg="Sending PFCP Session Establishment Request" CAT="PduSess" NF="SMF" time="2023-11-27T16:04:23.043417128+09:00" level="trace" msg="[SMF] Send SendPfcpSessionEstablishmentRequest" CAT="PduSess" NF="SMF" time="2023-11-27T16:04:23.043899656+09:00" level="trace" msg="Send to addr 192.168.1.111:8805" CAT="PduSess" NF="SMF" time="2023-11-27T16:04:23.066085696+09:00" level="info" msg="Received PFCP Session Establishment Accepted Response" CAT="PduSess" NF="SMF" time="2023-11-27T16:04:23.099668947+09:00" level="trace" msg="State[ActivePending] -> State[Active]" CAT="PduSess" NF="SMF" pdu_session_id="1" supi="imsi-208930000000003" time="2023-11-27T16:04:23.112868164+09:00" level="info" msg="Receive Update SM Context Request" CAT="PduSess" NF="SMF" time="2023-11-27T16:04:23.114583438+09:00" level="trace" msg="State[Active] -> State[ModificationPending]" CAT="PduSess" NF="SMF" pdu_session_id="1" supi="imsi-208930000000003" time="2023-11-27T16:04:23.116560933+09:00" level="trace" msg="State[ModificationPending] -> State[PFCPModification]" CAT="PduSess" NF="SMF" pdu_session_id="1" supi="imsi-208930000000003" time="2023-11-27T16:04:23.117479738+09:00" level="trace" msg="In case PFCPModification" CAT="PduSess" NF="SMF" pdu_session_id="1" supi="imsi-208930000000003" time="2023-11-27T16:04:23.129293784+09:00" level="info" msg="Received PFCP Session Modification Accepted Response from AN UPF" CAT="PduSess" NF="SMF" time="2023-11-27T16:04:23.130259009+09:00" level="trace" msg="In case SessionUpdateSuccess" CAT="PduSess" NF="SMF" pdu_session_id="1" supi="imsi-208930000000003" time="2023-11-27T16:04:23.131363584+09:00" level="trace" msg="State[PFCPModification] -> State[Active]" CAT="PduSess" NF="SMF" pdu_session_id="1" supi="imsi-208930000000003" time="2023-11-27T16:04:23.132523645+09:00" level="info" msg="| 200 | 192.168.0.11 | POST | /nsmf-pdusession/v1/sm-contexts/urn:uuid:474a7f9e-f223-4981-a748-cbe82a1b5a06/modify | " CAT="GIN" NF="SMF" -
Copyright © 2023 TOYOTA MOTOR CORPORATION All rights reserved. POC
– Analysis Analysis – TIPs for Log analysis TIP 1 Common Tags: time, level, msg, CAT (CATegory), NF (Network Function) TIP 2 Additional Tags: pdu_session, supi (=IMSI) for CATegory=“PDUSess” log lines TIP 3 Category “GIN”? Is HTTP Access Log (GIN is a Golang REST API Framework) 21 time="2023-11-27T16:04:22.923623330+09:00" level="info" msg="Receive Create SM Context Request" CAT="PduSess" NF="SMF" time="2023-11-27T16:04:22.928808304+09:00" level="trace" msg="State[InActive] -> State[InActive]" CAT="PduSess" NF="SMF" pdu_session_id="1" supi="imsi- 208930000000003" time="2023-11-27T16:04:23.041489662+09:00" level="info" msg="| 201 | 192.168.0.11 | POST | /nsmf-pdusession/v1/sm-contexts | " CAT="GIN" NF="SMF" -
Copyright © 2023 TOYOTA MOTOR CORPORATION All rights reserved. POC
– Analysis Analysis – Log Lines of Interest UPF Facility Identifier (Name/IP) and UE Identifier (SUPI/IMSI) correspondence can be found in SMF logs 22 level="trace" msg="findPSAandAllocUeIP" CAT="PduSess" NF="SMF" pdu_session_id="1" supi="imsi-208930000000003" level="debug" msg="check start UPF: UPF1-1" CAT="CTX" NF="SMF" level="debug" msg="check start UEIPPool(10.241.0.0/17)" CAT="CTX" NF="SMF" level="info" msg="Allocated UE IP address: 10.241.0.1" CAT="CTX" NF="SMF" level="info" msg="Selected UPF: UPF1-1" CAT="CTX" NF="SMF" level="info" msg="Allocated PDUAdress[10.241.0.1]" CAT="PduSess" NF="SMF" pdu_session_id="1" supi="imsi-208930000000003“ level="trace" msg="NodeIDtoIP: 192.168.1.111" CAT="PduSess" NF="SMF" Selected UPF name for the UE An excerpt from SMF log with ‘log level = trace’. timestamp (time=“YYMM…”) column is omitted for simplicity. Allocated UE IP Address UE Identifier, SUPI (IMSI) (optional) UPF (Underlay) IP address -
Copyright © 2023 TOYOTA MOTOR CORPORATION All rights reserved.
Monitoring System Prototype – Log Monitor + Trace Shipper Watch SMF log files, extract PDU Session - UPF mapping and send it out as a Trace span. The mapping is queried to identify blast-radius (affected UEs) on U-Plane failures. POC – Monitoring System Prototype 23 Trace Aggregator / Analyzer Establish PDU Session Log Watch Tool (Trace Shipper) Vehicle Mobile Network SMF Send Trace (w/SUPI) Log Time Send Trace (w/UPF+SUPI) Watch Write AMF … PDU Session Setup (not online) Init Registration Correlate Traces using SUPI, and Identify Vehicle/Location etc. U-Plane Failure On U-Plane failures, Lookup Trace Aggregator and extract suffered UEs (=Vehicles) list DB: VIN IMEI/IMSI DB: Vehicle Loc. / etc. Keep Alive -
Copyright © 2023 TOYOTA MOTOR CORPORATION All rights reserved. POC
– Monitoring System Prototype Log Watch Tool Example Output 24 {"time": "2023-12-05T07:48:25.411976661+09:00", "level": "info", "msg": "Allocated UE IP address: 10.241.0.6", "NF": "SMF", "CAT": "CTX"} {"time": "2023-12-05T07:48:25.412720710+09:00", "level": "info", "msg": "Selected UPF: UPF1-1", "NF": "SMF", "CAT": "CTX"} {"time": "2023-12-05T07:48:25.413436458+09:00", "level": "info", "msg": "Allocated PDUAdress[10.241.0.6]", "NF": "SMF", "CAT": "PduSess", "pdu_session_id": "2", "supi": "imsi-208930000000002"} # Found U-Plane/PDU Session mapping: UPF: UPF1-1 SUPI: imsi-208930000000002 pdu_session_id: 2 IP(UE): 10.241.0.6 { "name": null, "context": { "trace_id": "0x4419c8cca82561d8b52116276b202ee8", "span_id": "0x396829beeaa73eae", "trace_state": "[]" }, "kind": "SpanKind.INTERNAL", "parent_id": null, "start_time": "2023-12-05T02:35:34.698170Z", "end_time": "2023-12-05T02:35:34.698250Z", "status": { "status_code": "OK" }, "attributes": { "supi": "imsi-208930000000002", "pdu_session_id": "2", "ue_ip": "10.241.0.6", "selected_upf": "UPF1-1" }, "events": [], "links": [], "resource": { "attributes": { "service.name": "free5gc-parse.py" }, "schema_url": "" } } OpenTelemetry Trace/Span Data (By ConsoleExporter, normally exported to remote server) Mapping information of UE and UPF is exported as trace attributes. Processed (parsed) Free5GC SMF Log Lines -
Copyright © 2023 TOYOTA MOTOR CORPORATION All rights reserved. POC
– Evaluation Result After receiving a UPF failure, confirmed to receive a set of the failed UPF identifier and affected UE identifiers (SUPI/IMSI) mapping information within a second Practically, this time lag should depend on heartbeat timeout of NFs Current Limitations Cannot handle gNodeB information (at the moment) In case of Free5GC + UERANSIM: – gNodeB failure can be recovered via gNodeB Hand-Over – UPF failures require InitRegistration again Cannot handle tiered UPF (w/I-UPF) topology (at the moment) 25 -
Copyright © 2023 TOYOTA MOTOR CORPORATION All rights reserved. Works
In-Progress & Next Step 1. (Work In-Progress) Blush-up Design and Implementation of the Prototype Resolve restrictions, dashboard, Integration with other Observability Stack Or try other 5GS stack implementations (OAI etc.) 2. (Middle Term) Explore more sophisticated Information Retrieval (& Workflow) Best way is to rely on Standardized API of 3GPP/ETSI (if possible) 3. (Middle Term) Improve Performance & Scalability Up to millions of vehicles out of tens of millions of MNO subscribers Failure Notification Latency (less than a minute) 4. (Long Term) Propose to Telco Equipment Vendors/Integrators/MNOs Explore 3GPP spec. and Contribute to Free5GC or try other 5GS Open Source Stacks 26 -
Copyright © 2023 TOYOTA MOTOR CORPORATION All rights reserved. Summary
Proved it’s possible to Notify Users UE granularity U-Plane Failure Information With the notification, Connected Vehicle Service Providers can take actions Proactively if the failure is ahead of the of Vehicles Without any modifications to 3GPP protocol This time, watched log lines sequence and extracted UE/U-Plane Function mapping (but to be improved more) In General Observability via Distributed Tracing is quite Useful and Important Contributed 5GS Open Source Stack (and others) via reporting & posting fixes 27 -
Copyright © 2023 TOYOTA MOTOR CORPORATION All rights reserved. Appendix
Span Links – Record Correlation across OpenTelemetry Spans In case not a direct child, Links can be used to record indirect relationship (Such as Causal relationship) https://opentelemetry.io/docs/concepts/signals/traces/#span-links 28 Trace (1) Trace (2) Point Other Trace(s) as Link(s) References (List of Links) Useful for aggregating traces across system domains (e.g. Network and Application) -
Copyright © 2023 TOYOTA MOTOR CORPORATION All rights reserved. Appendix
RLS : Radio Link Simulation An encapsulation of Cellular Traffic over UDP UERANSIM internal protocol Wireshark dissector bundled 29 https://github.com/louisroyer/RLS-wireshark-dissector RLS of UERANSIM (over UDP) is dissected and displayed Application Traffic through UE using the Assigned IP address from 5GS