Mitigating the Latency-Accuracy Trade-off in Mobile Data Analytics Systems

Mitigating the Latency-Accuracy Trade-
off in Mobile Data Analytics Systems
Anand Iyer ⋆, Li Erran Li⬩, Mosharaf Chowdhury✢, Ion Stoica ⋆
⋆UC Berkeley ⬩Fudan University/Pony.ai ✢University of Michigan
MobiCom, November 1, 2018

View Slide

§ Many emerging domains
Mobile Data Analytics Very Popular

View Slide

§ Many emerging domains
Common Goal: Understand user/entity behavior
Mobile Data Analytics Very Popular

View Slide

Mobile Data Analytics

View Slide

Uplink SINR >
-11.75
RSRQ > -16.5
RSRQ
Available?
Success Drop
Uplink SINR >
-5.86
CQI > 5.875
Drop
Drop
Yes No
Yes No
No
Yes
Success
No
No
Yes
Yes
Success

View Slide

Uplink SINR >
-11.75
RSRQ > -16.5
RSRQ
Available?
Success Drop
Uplink SINR >
-5.86
CQI > 5.875
Drop
Drop
Yes No
Yes No
No
Yes
Success
No
No
Yes
Yes
Success
Tasks operate on data ingested in (near)
real-time for low-latency decisions

View Slide

Uplink SINR >
-11.75
RSRQ > -16.5
RSRQ
Available?
Success Drop
Uplink SINR >
-5.86
CQI > 5.875
Drop
Drop
Yes No
Yes No
No
Yes
Success
No
No
Yes
Yes
Success
Tasks operate on data ingested in (near)
real-time for low-latency decisions
Model/predict per-user/per-entity behavior

View Slide

Latency-Accuracy Trade-off
Data collection latency
Model Accuracy

View Slide

Model Accuracy
Statistically
insignificant

View Slide

Model Accuracy
Statistically
insignificant
High latency

View Slide

Model Accuracy
Statistically
insignificant
High latency
1 hour latency for
94% accuracy

View Slide

Model Accuracy
Statistically
insignificant
High latency
1 hour latency for
94% accuracy
Staleness enforces
short interval analyses

View Slide

Model Accuracy
Statistically
insignificant
High latency
1 hour latency for
94% accuracy
Staleness enforces
Highest achieved
accuracy ~66%

View Slide

Model Accuracy
Statistically
insignificant
High latency
1 hour latency for
94% accuracy
Staleness enforces
Highest achieved
accuracy ~66%
Need to update models frequently

View Slide

Mitigating Latency-Accuracy Trade-off
Model Accuracy

View Slide

Model Accuracy
Efficient Task
Formulations

View Slide

Model Accuracy
Efficient Task
Formulations
Intelligent Data
Grouping

View Slide

Hybrid Multi-Task Learning
Model Accuracy
Efficient Task
Formulations
Intelligent Data
Grouping

View Slide

PCA based partitioning
Hybrid Multi-Task Learning
Model Accuracy
Efficient Task
Formulations
Intelligent Data
Grouping

View Slide

Cellular RAN Performance Diagnostics

View Slide

Goal: Diagnose problems using data collected at base stations
Cellular RAN Performance Diagnostics

View Slide

Base stations vary widely in data
��
��
��
��
��
��
��
��
��
� � � � � ��
��
��
��

View Slide

Base stations vary widely in data
��
��
��
��
��
��
��
��
��
� � � � � ��
��
��
��
Many base
stations do not
collect enough
data in small
intervals

View Slide

Latency-Accuracy Trade-off in RANs
0
20
40
60
80
100
0 1 2 3 4 5 6 7 8
Accuracy (%)
Data Collection Latency (minutes)
Random Forest Lasso Regression
10 60
(Call drops) (Throughput)

View Slide

0
20
40
60
80
100
0 1 2 3 4 5 6 7 8
Accuracy (%)
10 60
High latency
incurred for
good accuracy

View Slide

0
20
40
60
80
100
0 1 2 3 4 5 6 7 8
Accuracy (%)
10 60
High latency
incurred for
good accuracy
Staleness
causes huge
variance and
errors

View Slide

Cellscope Architecture
CellScope
Domain-Speciﬁc MTL
Gradient Boosted Trees
RAN Performance Analyzer
ML Lib
Bearer Level
Trace
Dashboards
Self-Organizing
Networks (SON)
Throughput Drop
Feature
Engineering
PCA-Based Similarity Grouping
Streaming
Hybrid MTL

View Slide

Multi Task Learning (MTL)

View Slide

Jointly learn many tasks by exploiting commonalities and differences

View Slide

ℎ " = $(&' (
, &*
" , … , &,
("))

View Slide

Data Train Model
Task 1
Data Train Model
Task 2
Data Train Model
Task N
…
ℎ " = $(&' (
, &*
" , … , &,
("))

View Slide

Data Train Model
Task 1
Data Train Model
Task 2
Data Train Model
Task N
…
ℎ " = $(&' (
, &*
" , … , &,
("))
Data
Task 1
Data
Task 2
Data
Task N
…
Model
Model
Model
…
Train

View Slide

Data Train Model
Task 1
Data Train Model
Task 2
Data Train Model
Task N
…
ℎ " = $(&' (
, &*
" , … , &,
("))
Data
Task 1
Data
Task 2
Data
Task N
…
Model
Model
Model
…
Train
ℎ " = $./
(&' (
, &*
" , … , &,
("))

View Slide

Data Train Model
Task 1
Data Train Model
Task 2
Data Train Model
Task N
…
ℎ " = $(&' (
, &*
" , … , &,
("))
Data
Task 1
Data
Task 2
Data
Task N
…
Model
Model
Model
…
Train
ℎ " = $./
(&' (
, &*
" , … , &,
("))
Assumes that all tasks are related

View Slide

MTL in Cellscope

View Slide

MTL in Cellscope
Train
Data
Task 1
Data
Task 2
Data
Task N
…
Model
Model
Model
…
ℎ " = $%&
(() *
, (,
" , … , (.
("))

View Slide

MTL in Cellscope
Train
Data
Task 1
Data
Task 2
Data
Task N
…
Model
Model
Model
…
ℎ " = $%&
(() *
, (,
" , … , (.
("))
… …
Train
Data
Task 1
Task 2
Task K
…
Model
…
Group 1
Data
Data
Model
Model
Train
Data
Task 1
Task 2
Task K
…
Model
…
Group N
Data
Data
Model
Model
ℎ " = $0(%&)
(() *
, (,
" , … , (.
("))

View Slide

MTL in Cellscope
Problem: Scalable maintenance of large number of models
Train
Data
Task 1
Data
Task 2
Data
Task N
…
Model
Model
Model
…
ℎ " = $%&
(() *
, (,
" , … , (.
("))
… …
Train
Data
Task 1
Task 2
Task K
…
Model
…
Group 1
Data
Data
Model
Model
Train
Data
Task 1
Task 2
Task K
…
Model
…
Group N
Data
Data
Model
Model
ℎ " = $0(%&)
(() *
, (,
" , … , (.
("))

View Slide

min $ % ℎ ': )*+
, - + /|| 1(': )*+
)||
Hybrid MTL
Model estimation by L1 regularized loss minimization

View Slide

min $ % ℎ ': )*+
, - + /|| 1(': )*+
)||
Hybrid MTL
Prediction error

View Slide

min $ % ℎ ': )*+
, - + /|| 1(': )*+
)||
Hybrid MTL
Per base-station
parameters
Prediction error

View Slide

min $ % ℎ ': )*+
, - + /|| 1(': )*+
)||
Hybrid MTL
Per base-station
parameters
Regularization
parameter
Prediction error

View Slide

min $ % ℎ ': )*+
, - + /|| 1(': )*+
)||
Hybrid MTL
Per base-station
parameters
Regularization
parameter
Prediction error
Decompose parameters into shared common set fc
and base station specific set fs

View Slide

min $ % ℎ ': )*+
, - + /|| 1(': )*+
)||
Hybrid MTL
Per base-station
parameters
Regularization
parameter
Prediction error
$( ∑% ℎ ': )+
, )5
, - + /||1(': )+
)||) + /||1(': )5
)||

View Slide

min $ % ℎ ': )*+
, - + /|| 1(': )*+
)||
Hybrid MTL
Per base-station
parameters
Regularization
parameter
Prediction error
Base-station specific
$( ∑% ℎ ': )+
, )5
, - + /||1(': )+
)||) + /||1(': )5
)||

View Slide

Hybrid MTL
Structure of determines efficient implementation
ℎ : #$
, #&

View Slide

Hybrid MTL
Restrict models to be of form w . x
ℎ : #$
, #&

View Slide

Hybrid MTL
Leverage
ensemble methods
ℎ : #$
, #&

View Slide

Hybrid MTL
Dataset
Leverage
ensemble methods
ℎ : #$
, #&

View Slide

Hybrid MTL
Dataset
Model 2 Model 3 Model 4 Model N
Model 1 …
Leverage
ensemble methods
ℎ : #$
, #&

View Slide

Hybrid MTL
Dataset
Ensemble Model
Model 1 …
Leverage
ensemble methods
ℎ : #$
, #&

View Slide

Hybrid MTL
Dataset
Ensemble Model
Model 1 …
Leverage
ensemble methods
ℎ : #$
, #&
f1
f2
f3
f4
fN

View Slide

Hybrid MTL
Dataset
Ensemble Model
Model 1 …
Leverage
ensemble methods
Gradient Boosted
Trees
ℎ : #$
, #&
f1
f2
f3
f4
fN

View Slide

Hybrid MTL More details in the paper
Dataset
Ensemble Model
Model 1 …
Leverage
ensemble methods
Gradient Boosted
Trees
ℎ : #$
, #&
f1
f2
f3
f4
fN

View Slide

Data Grouping for MTL
Key Idea: Use Principal Component Analysis (PCA) to find normal behavior

View Slide

Project large number of correlated dimensions to a
small set of orthogonal dimensions.

View Slide

…

View Slide

…
n

View Slide

…
…
…
…
…
…
…
…
n
m
Measurement matrix

View Slide

…
…
…
…
…
…
…
…
…
…
…
…
…
…
…
…
…
…
…
…
n
m
n
k
Measurement matrix

View Slide

…
…
…
…
…
…
…
…
…
…
…
…
…
…
…
…
…
…
…
…
n
m
n
k
loadings
Measurement matrix

View Slide

PCA Similarity
Find the similarity between the principal components

View Slide

PCA Similarity
…
…
…
…
…
…
…
…
mA
n
…
…
…
…
…
…
…
…
mB
n

View Slide

PCA Similarity
…
…
…
…
…
…
…
…
mA
n
…
…
…
…
…
…
…
…
mB
n
…
…
…
…
…
…
…
…
…
…
…
…
n
k
…
…
…
…
…
…
…
…
…
…
…
…
n
k

View Slide

PCA Similarity
…
…
…
…
…
…
…
…
mA
n
…
…
…
…
…
…
…
…
mB
n
…
…
…
…
…
…
…
…
…
…
…
…
n
k
…
…
…
…
…
…
…
…
…
…
…
…
n
k
S"#$%%&'()$
= +
,-.
/
+
0-.
1
|30,
− 50,
|

View Slide

PCA Similarity
…
…
…
…
…
…
…
…
mA
n
…
…
…
…
…
…
…
…
mB
n
…
…
…
…
…
…
…
…
…
…
…
…
n
k
…
…
…
…
…
…
…
…
…
…
…
…
n
k
S"#$%%&'()$
= +
,-.
/
+
0-.
1
|30,
− 50,
| × 7809:;1'$(=,?)

View Slide

Implementation & Evaluation
§ Implemented on Apache Spark
§ Extends Mllib and provides a simple API for grouping
§ Evaluated using data from a live RAN
§ Data over several months
§ Models for two metrics: drops and throughput
prediction
§ Also analyzed several issues in the wild

View Slide

Cellscope Reduces Latency
0
20
40
60
80
100
0 1 2 3 4 5 6 7 8
Accuracy (%)
Data Collection Latency (min)
Per Base Station Cellscope
10 60
> 90% accuracy with 3
minutes data (compared to
60 minutes)
3 x

View Slide

Cellscope Improves Accuracy
0
20
40
60
80
100
0 1 2 3 4 5 6 7 8
Accuracy (%)
Data Collection Latency (min)
Per Base Station Cellscope
10 60
Achieves high
accuracy in
small timespans
1.4 x
4 x

View Slide

Real-world Analysis with Cellscope
§ Cellscope can significantly reduce operator efforts
§ Reduces the need for field trials, can build accurate models quickly
§ Up to 2 order of magnitudes (10s of hours → minutes)
§ Cellscope found new issues previously unknown
§ E.g., Grouping revealed high interference base station clusters
§ Cellscope can aid domain expert
§ Can reduce the troubleshooting search space significantly

View Slide

Much more in the paper…
§ Extending the techniques to other domains
§ Straightforward & effective
§ Comparison with strawman grouping techniques
§ Why they’re not sufficient
§ Implementation details of our hybrid MTL & API
§ Extensive evaluation
§ Real-world analysis & findings

View Slide

Summary
§ Mobile data analytics popular
§ Need low-latency decisions on live data
§ Latency-Accuracy Trade-off
§ Not enough data in small timespans, staleness determines bounds on
data collection latencies
§ Intelligent grouping and efficient task formulations
§ Hybrid MTL and PCA based partitioning
http://www.cs.berkeley.edu/~api
[email protected]

View Slide

Mitigating the Latency-Accuracy Trade-off in Mobile Data Analytics Systems

Mitigating the Latency-Accuracy Trade-off in Mobile Data Analytics Systems

Anand Iyer

More Decks by Anand Iyer

Other Decks in Research

Featured

Transcript