Mitigating the Latency-Accuracy Trade-off in Mobile Data Analytics Systems

Transcript

1. Mitigating the Latency-Accuracy Trade- off in Mobile Data Analytics Systems

   Anand Iyer ⋆, Li Erran Li⬩, Mosharaf Chowdhury✢, Ion Stoica ⋆ ⋆UC Berkeley ⬩Fudan University/Pony.ai ✢University of Michigan MobiCom, November 1, 2018

2. § Many emerging domains Mobile Data Analytics Very Popular

3. § Many emerging domains Mobile Data Analytics Very Popular

4. § Many emerging domains Mobile Data Analytics Very Popular

5. § Many emerging domains Common Goal: Understand user/entity behavior Mobile

   Data Analytics Very Popular

6. Mobile Data Analytics

7. Mobile Data Analytics

8. Mobile Data Analytics

9. Mobile Data Analytics

10. Uplink SINR > -11.75 RSRQ > -16.5 RSRQ Available? Success

    Drop Uplink SINR > -5.86 CQI > 5.875 Drop Drop Yes No Yes No No Yes Success No No Yes Yes Success Mobile Data Analytics

11. Uplink SINR > -11.75 RSRQ > -16.5 RSRQ Available? Success

    Drop Uplink SINR > -5.86 CQI > 5.875 Drop Drop Yes No Yes No No Yes Success No No Yes Yes Success Mobile Data Analytics Tasks operate on data ingested in (near) real-time for low-latency decisions

12. Uplink SINR > -11.75 RSRQ > -16.5 RSRQ Available? Success

    Drop Uplink SINR > -5.86 CQI > 5.875 Drop Drop Yes No Yes No No Yes Success No No Yes Yes Success Mobile Data Analytics Tasks operate on data ingested in (near) real-time for low-latency decisions Model/predict per-user/per-entity behavior

13. Latency-Accuracy Trade-off Data collection latency Model Accuracy

14. Latency-Accuracy Trade-off Data collection latency Model Accuracy

15. Latency-Accuracy Trade-off Data collection latency Model Accuracy Statistically insignificant

16. Latency-Accuracy Trade-off Data collection latency Model Accuracy Statistically insignificant

17. Latency-Accuracy Trade-off Data collection latency Model Accuracy Statistically insignificant High

    latency

18. Latency-Accuracy Trade-off Data collection latency Model Accuracy Statistically insignificant High

    latency 1 hour latency for 94% accuracy

19. Latency-Accuracy Trade-off Data collection latency Model Accuracy Statistically insignificant High

    latency 1 hour latency for 94% accuracy

20. Latency-Accuracy Trade-off Data collection latency Model Accuracy Statistically insignificant High

    latency 1 hour latency for 94% accuracy Staleness enforces short interval analyses

21. Latency-Accuracy Trade-off Data collection latency Model Accuracy Statistically insignificant High

    latency 1 hour latency for 94% accuracy Staleness enforces short interval analyses Highest achieved accuracy ~66%

22. Latency-Accuracy Trade-off Data collection latency Model Accuracy Statistically insignificant High

    latency 1 hour latency for 94% accuracy Staleness enforces short interval analyses Highest achieved accuracy ~66% Need to update models frequently

23. Mitigating Latency-Accuracy Trade-off Data collection latency Model Accuracy

24. Mitigating Latency-Accuracy Trade-off Data collection latency Model Accuracy

25. Mitigating Latency-Accuracy Trade-off Data collection latency Model Accuracy Efficient Task

    Formulations

26. Mitigating Latency-Accuracy Trade-off Data collection latency Model Accuracy Efficient Task

    Formulations Intelligent Data Grouping

27. Hybrid Multi-Task Learning Mitigating Latency-Accuracy Trade-off Data collection latency Model

    Accuracy Efficient Task Formulations Intelligent Data Grouping

28. PCA based partitioning Hybrid Multi-Task Learning Mitigating Latency-Accuracy Trade-off Data

    collection latency Model Accuracy Efficient Task Formulations Intelligent Data Grouping

29. Cellular RAN Performance Diagnostics

30. Cellular RAN Performance Diagnostics

31. Goal: Diagnose problems using data collected at base stations Cellular

    RAN Performance Diagnostics

32. Base stations vary widely in data �� ���� ����� �����

    ����� ����� ����� ����� ����� � � � � � �� �� ����������������� ��������������������������������� ���������

33. Base stations vary widely in data �� ���� ����� �����

    ����� ����� ����� ����� ����� � � � � � �� �� ����������������� ��������������������������������� ��������� Many base stations do not collect enough data in small intervals

34. Latency-Accuracy Trade-off in RANs 0 20 40 60 80 100

    0 1 2 3 4 5 6 7 8 Accuracy (%) Data Collection Latency (minutes) Random Forest Lasso Regression 10 60 (Call drops) (Throughput)

35. Latency-Accuracy Trade-off in RANs 0 20 40 60 80 100

    0 1 2 3 4 5 6 7 8 Accuracy (%) Data Collection Latency (minutes) Random Forest Lasso Regression 10 60 High latency incurred for good accuracy (Call drops) (Throughput)

36. Latency-Accuracy Trade-off in RANs 0 20 40 60 80 100

    0 1 2 3 4 5 6 7 8 Accuracy (%) Data Collection Latency (minutes) Random Forest Lasso Regression 10 60 High latency incurred for good accuracy Staleness causes huge variance and errors (Call drops) (Throughput)

37. Cellscope Architecture CellScope Domain-Specific MTL Gradient Boosted Trees RAN Performance

    Analyzer ML Lib Bearer Level Trace Dashboards Self-Organizing Networks (SON) Throughput Drop Feature Engineering PCA-Based Similarity Grouping Streaming Hybrid MTL

38. Cellscope Architecture CellScope Domain-Specific MTL Gradient Boosted Trees RAN Performance

    Analyzer ML Lib Bearer Level Trace Dashboards Self-Organizing Networks (SON) Throughput Drop Feature Engineering PCA-Based Similarity Grouping Streaming Hybrid MTL

39. Cellscope Architecture CellScope Domain-Specific MTL Gradient Boosted Trees RAN Performance

    Analyzer ML Lib Bearer Level Trace Dashboards Self-Organizing Networks (SON) Throughput Drop Feature Engineering PCA-Based Similarity Grouping Streaming Hybrid MTL

40. Cellscope Architecture CellScope Domain-Specific MTL Gradient Boosted Trees RAN Performance

    Analyzer ML Lib Bearer Level Trace Dashboards Self-Organizing Networks (SON) Throughput Drop Feature Engineering PCA-Based Similarity Grouping Streaming Hybrid MTL

41. Multi Task Learning (MTL)

42. Multi Task Learning (MTL) Jointly learn many tasks by exploiting

    commonalities and differences

43. Multi Task Learning (MTL) Jointly learn many tasks by exploiting

    commonalities and differences ℎ " = $(&' ( , &* " , … , &, ("))

44. Multi Task Learning (MTL) Jointly learn many tasks by exploiting

    commonalities and differences Data Train Model Task 1 Data Train Model Task 2 Data Train Model Task N … ℎ " = $(&' ( , &* " , … , &, ("))

45. Multi Task Learning (MTL) Jointly learn many tasks by exploiting

    commonalities and differences Data Train Model Task 1 Data Train Model Task 2 Data Train Model Task N … ℎ " = $(&' ( , &* " , … , &, (")) Data Task 1 Data Task 2 Data Task N … Model Model Model … Train

46. Multi Task Learning (MTL) Jointly learn many tasks by exploiting

    commonalities and differences Data Train Model Task 1 Data Train Model Task 2 Data Train Model Task N … ℎ " = $(&' ( , &* " , … , &, (")) Data Task 1 Data Task 2 Data Task N … Model Model Model … Train ℎ " = $./ (&' ( , &* " , … , &, ("))

47. Multi Task Learning (MTL) Jointly learn many tasks by exploiting

    commonalities and differences Data Train Model Task 1 Data Train Model Task 2 Data Train Model Task N … ℎ " = $(&' ( , &* " , … , &, (")) Data Task 1 Data Task 2 Data Task N … Model Model Model … Train ℎ " = $./ (&' ( , &* " , … , &, (")) Assumes that all tasks are related

48. MTL in Cellscope

49. MTL in Cellscope Train Data Task 1 Data Task 2

    Data Task N … Model Model Model … ℎ " = $%& (() * , (, " , … , (. ("))

50. MTL in Cellscope Train Data Task 1 Data Task 2

    Data Task N … Model Model Model … ℎ " = $%& (() * , (, " , … , (. (")) … … Train Data Task 1 Task 2 Task K … Model … Group 1 Data Data Model Model Train Data Task 1 Task 2 Task K … Model … Group N Data Data Model Model ℎ " = $0(%&) (() * , (, " , … , (. ("))

51. MTL in Cellscope Problem: Scalable maintenance of large number of

    models Train Data Task 1 Data Task 2 Data Task N … Model Model Model … ℎ " = $%& (() * , (, " , … , (. (")) … … Train Data Task 1 Task 2 Task K … Model … Group 1 Data Data Model Model Train Data Task 1 Task 2 Task K … Model … Group N Data Data Model Model ℎ " = $0(%&) (() * , (, " , … , (. ("))

52. min $ % ℎ ': )*+ , - + /||

    1(': )*+ )|| Hybrid MTL Model estimation by L1 regularized loss minimization

53. min $ % ℎ ': )*+ , - + /||

    1(': )*+ )|| Hybrid MTL Model estimation by L1 regularized loss minimization Prediction error

54. min $ % ℎ ': )*+ , - + /||

    1(': )*+ )|| Hybrid MTL Model estimation by L1 regularized loss minimization Per base-station parameters Prediction error

55. min $ % ℎ ': )*+ , - + /||

    1(': )*+ )|| Hybrid MTL Model estimation by L1 regularized loss minimization Per base-station parameters Regularization parameter Prediction error

56. min $ % ℎ ': )*+ , - + /||

    1(': )*+ )|| Hybrid MTL Model estimation by L1 regularized loss minimization Per base-station parameters Regularization parameter Prediction error Decompose parameters into shared common set fc and base station specific set fs

57. min $ % ℎ ': )*+ , - + /||

    1(': )*+ )|| Hybrid MTL Model estimation by L1 regularized loss minimization Per base-station parameters Regularization parameter Prediction error Decompose parameters into shared common set fc and base station specific set fs $( ∑% ℎ ': )+ , )5 , - + /||1(': )+ )||) + /||1(': )5 )||

58. min $ % ℎ ': )*+ , - + /||

    1(': )*+ )|| Hybrid MTL Model estimation by L1 regularized loss minimization Per base-station parameters Regularization parameter Prediction error Decompose parameters into shared common set fc and base station specific set fs Base-station specific $( ∑% ℎ ': )+ , )5 , - + /||1(': )+ )||) + /||1(': )5 )||

59. Hybrid MTL Structure of determines efficient implementation ℎ : #$

    , #&

60. Hybrid MTL Structure of determines efficient implementation Restrict models to

    be of form w . x ℎ : #$ , #&

61. Hybrid MTL Structure of determines efficient implementation Restrict models to

    be of form w . x Leverage ensemble methods ℎ : #$ , #&

62. Hybrid MTL Structure of determines efficient implementation Dataset Restrict models

    to be of form w . x Leverage ensemble methods ℎ : #$ , #&

63. Hybrid MTL Structure of determines efficient implementation Dataset Model 2

    Model 3 Model 4 Model N Model 1 … Restrict models to be of form w . x Leverage ensemble methods ℎ : #$ , #&

64. Hybrid MTL Structure of determines efficient implementation Dataset Ensemble Model

    Model 2 Model 3 Model 4 Model N Model 1 … Restrict models to be of form w . x Leverage ensemble methods ℎ : #$ , #&

65. Hybrid MTL Structure of determines efficient implementation Dataset Ensemble Model

    Model 2 Model 3 Model 4 Model N Model 1 … Restrict models to be of form w . x Leverage ensemble methods ℎ : #$ , #& f1 f2 f3 f4 fN

66. Hybrid MTL Structure of determines efficient implementation Dataset Ensemble Model

    Model 2 Model 3 Model 4 Model N Model 1 … Restrict models to be of form w . x Leverage ensemble methods Gradient Boosted Trees ℎ : #$ , #& f1 f2 f3 f4 fN

67. Hybrid MTL More details in the paper Structure of determines

    efficient implementation Dataset Ensemble Model Model 2 Model 3 Model 4 Model N Model 1 … Restrict models to be of form w . x Leverage ensemble methods Gradient Boosted Trees ℎ : #$ , #& f1 f2 f3 f4 fN

68. Data Grouping for MTL Key Idea: Use Principal Component Analysis

    (PCA) to find normal behavior

69. Data Grouping for MTL Project large number of correlated dimensions

    to a small set of orthogonal dimensions. Key Idea: Use Principal Component Analysis (PCA) to find normal behavior

70. Data Grouping for MTL Project large number of correlated dimensions

    to a small set of orthogonal dimensions. Key Idea: Use Principal Component Analysis (PCA) to find normal behavior …

71. Data Grouping for MTL Project large number of correlated dimensions

    to a small set of orthogonal dimensions. Key Idea: Use Principal Component Analysis (PCA) to find normal behavior … n

72. Data Grouping for MTL Project large number of correlated dimensions

    to a small set of orthogonal dimensions. Key Idea: Use Principal Component Analysis (PCA) to find normal behavior … … … … … … … … n m Measurement matrix

73. Data Grouping for MTL Project large number of correlated dimensions

    to a small set of orthogonal dimensions. Key Idea: Use Principal Component Analysis (PCA) to find normal behavior … … … … … … … … n m Measurement matrix

74. Data Grouping for MTL Project large number of correlated dimensions

    to a small set of orthogonal dimensions. Key Idea: Use Principal Component Analysis (PCA) to find normal behavior … … … … … … … … … … … … … … … … … … … … n m n k Measurement matrix

75. Data Grouping for MTL Project large number of correlated dimensions

    to a small set of orthogonal dimensions. Key Idea: Use Principal Component Analysis (PCA) to find normal behavior … … … … … … … … … … … … … … … … … … … … n m n k Measurement matrix

76. Data Grouping for MTL Project large number of correlated dimensions

    to a small set of orthogonal dimensions. Key Idea: Use Principal Component Analysis (PCA) to find normal behavior … … … … … … … … … … … … … … … … … … … … n m n k loadings Measurement matrix

77. PCA Similarity Find the similarity between the principal components

78. PCA Similarity Find the similarity between the principal components …

    … … … … … … … mA n … … … … … … … … mB n

79. PCA Similarity Find the similarity between the principal components …

    … … … … … … … mA n … … … … … … … … mB n … … … … … … … … … … … … n k … … … … … … … … … … … … n k

80. PCA Similarity Find the similarity between the principal components …

    … … … … … … … mA n … … … … … … … … mB n … … … … … … … … … … … … n k … … … … … … … … … … … … n k

81. PCA Similarity Find the similarity between the principal components …

    … … … … … … … mA n … … … … … … … … mB n … … … … … … … … … … … … n k … … … … … … … … … … … … n k S"#$%%&'()$ = + ,-. / + 0-. 1 |30, − 50, |

82. PCA Similarity Find the similarity between the principal components …

    … … … … … … … mA n … … … … … … … … mB n … … … … … … … … … … … … n k … … … … … … … … … … … … n k S"#$%%&'()$ = + ,-. / + 0-. 1 |30, − 50, | × 7809:;1'$(=,?)

83. Implementation & Evaluation § Implemented on Apache Spark § Extends

    Mllib and provides a simple API for grouping § Evaluated using data from a live RAN § Data over several months § Models for two metrics: drops and throughput prediction § Also analyzed several issues in the wild

84. Cellscope Reduces Latency 0 20 40 60 80 100 0

    1 2 3 4 5 6 7 8 Accuracy (%) Data Collection Latency (min) Per Base Station Cellscope 10 60 > 90% accuracy with 3 minutes data (compared to 60 minutes) 3 x

85. Cellscope Improves Accuracy 0 20 40 60 80 100 0

    1 2 3 4 5 6 7 8 Accuracy (%) Data Collection Latency (min) Per Base Station Cellscope 10 60 Achieves high accuracy in small timespans 1.4 x 4 x

86. Real-world Analysis with Cellscope § Cellscope can significantly reduce operator

    efforts § Reduces the need for field trials, can build accurate models quickly § Up to 2 order of magnitudes (10s of hours → minutes) § Cellscope found new issues previously unknown § E.g., Grouping revealed high interference base station clusters § Cellscope can aid domain expert § Can reduce the troubleshooting search space significantly

87. Much more in the paper… § Extending the techniques to

    other domains § Straightforward & effective § Comparison with strawman grouping techniques § Why they’re not sufficient § Implementation details of our hybrid MTL & API § Extensive evaluation § Real-world analysis & findings

88. Summary § Mobile data analytics popular § Need low-latency decisions

    on live data § Latency-Accuracy Trade-off § Not enough data in small timespans, staleness determines bounds on data collection latencies § Intelligent grouping and efficient task formulations § Hybrid MTL and PCA based partitioning http://www.cs.berkeley.edu/~api api@cs.berkeley.edu