Me Masanori Itoh Affiliation TOYOTA MOTOR CORPORATION Digital Information and Communication Dept. InfoTech Div. Open Source Program Group (Toyota OSPO) Works R&D for Connected Vehicle Systems • E2E Observability, Standardization, Diagnostics, … OSPO Operations Keywords Operating System, Cloud Infrastructure, etc. https://github.com/thatsdone https://www.linkedin.com/in/masanori-itoh-6401603/
カメラ画像データに対する物体検知処理のPOCアプリにOTELを適用した例 9 Object Detection Subsystem Camera Data Upload Event TRIP : from Engine Start to Stop Camera Data Upload Operation Camera Data Upload Completion handler Invokes Object Detection Subsystem Asynchronously センタ側の制御に従っ てカメラ画像をアップ ロードし、非同期に物 体検出処理を実行
トラブル発生時に特定したいこと: Areas where vehicle communication is affected (どこで) Vehicles that get affected from the trouble (どれ(＝どの車両)) What kind of communication trouble? (何が/どう) Intermittent or Persistent trouble, for example When the trouble occurred and is expected to be resolved (いつ) 11 Excerpt from KubeConNA 2023 Presentation, “E2E Observability for Connected Vehicle Service via Distributed Tracing” Requirements from Mobile Network Operator side Systems with large number of users constantly connected Provided for IoT devices such as connected car Accountability for the failure impact and the extent to which customers were affected when the failure occurred
– 詳細 Problems to be Addressed Application Layer Troubles are OK, But Need to address Network Troubles (including U-Plane) Assumptions MNO does health monitoring of C/U-Plane NFs User (Connected Vehicle Servicer) Side knows: Identifiers(IMSI(SUPI)…), Current Location, Route Plan (e.g., Destination, Schedule), … Motivations/Reasons Detailed Requirements If we can get U-Plane trouble information with UE granularity earlier, we can take Proactive Actions, sending/retrieving necessary information/command in prior, for example. Even if it’s difficult to forecast failures/troubles, failure location and vehicle location is normally apart. Thus, we can take Actions Proactively. 12
– 理想 実現したいこと U-Plane機器(中継装置/基地局)単位で、PDUセッションの情報と、端末 の識別子を取り出せればよい 初期アイデア (1) Get PDU Session Information (with SUPI(IMSI)) from NFs via REST API Could be called this function as a kind of NEF? (2) Use OpenTelemetry Automatic Instrumentation Two choices – OpenTelemetry otel-go-instrumentation (https://github.com/open- telemetry/opentelemetry-go-instrumentation) – Grafana Labs Beyla Automatic Instrumentation (https://github.com/grafana/beyla) KDDI Team took Manual Instrumentation approach to address non- HTTP 5GS communication(NGAP/PFCP) 14
– 現実 現実 (1) Free5GC NFs implementations do not return sufficient information (2) Golang (in which Free5GC is written) Automatic Instrumentation has limitations Even HTTP (not NGAP/PFCP) could not be instrumented enough 実際やってみたこと(Work Around) NF(特に AMF と SMF)が出力するログを監視し、リアルタイムに必要 な情報を取り出し、OTLPで送り出すツールを作成 A custom Log Parser of Free5GC NFs which can emit trace data with SUPI(IMSI) and to be correlated later 15
– 評価 結果 中継装置(UPF)で疑似故障発生後、確かに影響を受ける中継装置 (UPF)と端末(UE)の対応情報を受信できた どの程度リアルタイムに検知できるかは、NW基盤の監視の仕組みの設定 (≒通常は生存検知間隔)に依存 Current Limitations Cannot handle gNodeB information (at the moment) In case of Free5GC + UERANSIM: – gNodeB failure can be recovered via gNodeB Hand-Over – UPF failures require InitRegistration again Cannot handle tiered UPF (w/I-UPF) topology (at the moment) 23
In-Progress & Next Step 1. (Work In-Progress) Blush-up Design and Implementation of the Prototype Resolve restrictions, dashboard, Integration with other Observability Stack Or try other 5GS stack implementations (OAI etc.) 2. (Middle Term) Explore more sophisticated Information Retrieval (& Workflow) Best way is to rely on Standardized API of 3GPP/ETSI (if possible) 3. (Middle Term) Improve Performance & Scalability Up to millions of vehicles out of tens of millions of MNO subscribers Failure Notification Latency (less than a minute) 4. (Long Term) Propose to Telco Equipment Vendors/Integrators/MNOs Explore 3GPP spec. and Contribute to Free5GC or try other 5GS Open Source Stacks 24
Proved it’s possible to Notify Users UE granularity U-Plane Failure Information With the notification, Connected Vehicle Service Providers can take actions Proactively if the failure is ahead of the of Vehicles Without any modifications to 3GPP protocol This time, watched log lines sequence and extracted UE/U-Plane Function mapping (but to be improved more) In General Observability via Distributed Tracing is quite Useful and Important Contributed 5GS Open Source Stack (and others) via reporting & posting fixes 25
Span Links – Record Correlation across OpenTelemetry Spans In case not a direct child, Links can be used to record indirect relationship (Such as Causal relationship) https://opentelemetry.io/docs/concepts/signals/traces/#span-links 33 Trace (1) Trace (2) Point Other Trace(s) as Link(s) References (List of Links) Useful for aggregating traces across system domains (e.g. Network and Application)
thatsdone
sansantech
logica0419
jollyjoester
gtnao
travelcarecenter
biwashi
sansan33
yokawasa
nakasho
masatokinugawa
murabayashi
bluesmoon
smashingmag
csswizardry
paigeccino
productmarketing
cassininazir
geoffreycrofte
hatefulcrawdad
mongodb
sstephenson
trallard
marcelosomers
