Md Shahriar Iqbal UNICORN: Reasoning about Configurable System Performance through the Lens of Causality Rahul Krishna MA Javidian Baishakhi Ray Pooyan Jamshidi
Each component has a plethora of configuration options 6 Video
Decoder Stream Muxer Primary Detector Object Tracker Secondary Classifier # Configuration Options 55 86 14 44 86 Con fi gurations Possible 2285 Complex interactions between options (intra or inter components) give rise to a combinatorially large con fi guration space Compression
Energy (Joules) Performance varies significantly when systems are deployed with different configurations 7 Latency Energy Consumption 5 10 15 20 25 10 20 30 40 50 >99.99% >99.99% Latency (Seconds) Varies by 5x Varies by 4.5x It is expected to set the system to a con fi guration for which the performance remains optimal or close to the optimal
Energy (Joules) Performance varies significantly when systems are deployed with different configurations 8 Latency Energy Consumption 5 10 15 20 25 10 20 30 40 50 >99.99% >99.99% Latency (Seconds) Varies by 5x Varies by 4.5x It is expected to set the system to a con fi guration for which the performance remains optimal or close to the optimal Reaching desired performance goal is di ffi cult due to sheer size of the con fi guration space and high con fi guration measurement cost
Real world example: Deployment environment change 10 When we are trying to transplant our CUDA source code from TX1 to TX2, it behaved strange. We noticed that TX2 has twice computing-ability as TX1 in GPU, as expectation, we think TX2 will 30% - 40% faster than TX1 at least. Unfortunately, most of our code base spent twice the time as TX1, in other words, TX2 only has 1/2 speed as TX1, mostly. We believe that TX2’s CUDA API runs much slower than TX1 in many cases. The user is transferring the code from one hardware to another The code ran 2x slower on
Real world example: Deployment environment change 11 When we are trying to transplant our CUDA source code from TX1 to TX2, it behaved strange. We noticed that TX2 has twice computing-ability as TX1 in GPU, as expectation, we think TX2 will 30% - 40% faster than TX1 at least. Unfortunately, most of our code base spent twice the time as TX1, in other words, TX2 only has 1/2 speed as TX1, mostly. We believe that TX2’s CUDA API runs much slower than TX1 in many cases. The user is transferring the code from one hardware to another The code ran 2x slower on
the more powerful hardware Incorrect understanding about the performance
What is misconfiguration? 13 Miscon fi gurations happen due to unexpected interactions between con fi guration options in the deployment system stack. The system does not crash but remains operational with degraded performance e.g., high latency, low throughput, high energy consumption. Latency Energy Consumption 5 10 15 20 25 10 20 30 40 50 >99.99% >99.99% Latency (Seconds) Energy (Joules) Miscon fi guration
Performance task: Debugging 14 Performance debugging aims at fi nding the root cause of the miscon fi guration and fi x it. When we are trying to transplant our CUDA source code from TX1 to TX2, it behaved strange. We noticed that TX2 has twice computing-ability as TX1 in GPU, as expectation, we think TX2 will 30% - 40% faster than TX1 at least. Unfortunately, most of our code base spent twice the time as TX1, in other words, TX2 only has 1/2 speed as TX1, mostly. We believe that TX2’s CUDA API runs much slower than TX1 in many cases. The user is transferring the code from one hardware to another The code ran 2x slower on
the more powerful hardware The user expects 30-40%
Energy (Joules) Performance task: Optimization 15 Latency Energy Consumption 5 10 15 20 25 10 20 30 40 50 >99.99% >99.99% Latency (Seconds) Here, the developer aims at fi nding the optimal con fi guration with or without experiencing any miscon fi guration.
Performance debugging tasks take significantly long time, the fixes are typically non-intuitive (changes to seemingly underrated options) 16 June 3 June 4 June 4 June 5 Any suggestions on how to improve my performance? Thanks! Please do the following and let us know if it works 1. Install JetPack 3.0 2. Set nvpmodel=MAX-N 3. Run jetson_clock.sh We have already tried this. We still have high latency. Any other suggestions? TX2 is pascal architecture. Please update your CMakeLists: + set(CUDA_STATIC_RUNTIME OFF) ... + -gencode=arch=compute_62,code=sm_62 The user had several misconfigurations In Software: ✖ Wrong compilation flags ✖ Wrong SDK version In Hardware: ✖ Wrong power mode ✖ Wrong clock/fan settings
18 Performance In fl uence Models number of counters number of splitters latency (ms) 100 150 1 200 250 2 300 Cubic Interpolation Over Finer Grid 2 4 3 6 8 4 10 12 5 14 16 6 18 Bitrate (bits/s) Enable Padding … Cache Misses … Through put (fps) c1 1k 1 … 42m … 7 c2 2k 1 … 32m … 22 … … … … … … … cn 5k 0 … 12m … 25 Observational Data Black-box models Regression Equation Throughput = 5.1 × Bitrate + 2.5 × BatchSize + 12.3 × Bitrate × BatchSize Interactions Options Options This is a representative work, but there are many other works related to using regression models (as well as other statistical models) for building performance models We have selection bias here ;)
increase it Purely statistical models built on this data will be unreliable. This is counter-intuitive Performance Influence Models might be Unreliable
Cache Misses Throughput (FPS) 20 10 0 100k 200k 23 Cache Misses Throughput (FPS) LRU FIFO LIFO MRU 20 10 0 100k 200k Segregating data on Cache Policy indicates that within each group increase of Cache Misses result in a decrease in Throughput. FIFO LIFO MRU LRU Performance Influence Models might be Unreliable
28 Our Key Idea: Building Causal Performance Model instead of Performance Influence Models Expresses the relationships between Con fi guration options System Events Non-functional
Properties Cache Misses Throughput (FPS) 20 10 0 100k 200k interacting variables as a causal graph Direction of
Why Causal Performance Models? To reuse them when the system environment changes 30 Causal models remain relatively stable A partial causal performance
model in Jetson Xavier A partial causal performance
UNICORN: End-to-end Pipeline 34 5- Estimate Causal Queries • What is the root-cause of fault? • How do I fix misconfiguration? • How do I optimize perf.? • How do I understand perf.? Software: DeepStream Middleware: TF, TensorRT Hardware: Nvidia Xavier Configuration: Default Estimate probability of satisfying QoS if BufferSize is set to 6k? 2- Learn Causal Performance Model Initial Perf. Data P(Th > 40/s|do(Buffersize = 6k)) 1- Specify Performance Query QoS : Th > 40/s Observed : Th < 30/s ± 5/s Causal Performance Model System Stack Performance Tasks Performance Fault/Issue 4- Update Causal Performance Model Causal Inference Engine 3- Determine Next Configuration Uses Stage Stages I. Specify Performance Query
Stage-II: Learn Causal Performance Model 40 FPS Energy Branch Misses Cache Misses No of Cycles Bitrate Buffer Size Batch Size Enable Padding Partial Ancestral Graph (PAG) A PAG can have three types of edges between
any nodes X and Y X Y X is a parent of Y X Y A confounder exists between X and Y X Y Not su ffi cient data to recover causal direction X Y X Y or or
Stage-II: Learn Causal Performance Model 41 FPS Energy Branch Misses Cache Misses No of Cycles Bitrate Buffer Size Batch Size Enable Padding Partial Ancestral Graph (PAG) A PAG can have three types of edges between
any nodes X and Y X Y X is a parent of Y X Y A confounder exists between X and Y X Y Not su ffi cient data to recover causal direction X Y X Y or or
Stage-II: Learn Causal Performance Model 42 FPS Energy Branch Misses Cache Misses No of Cycles Bitrate Buffer Size Batch Size Enable Padding Partial Ancestral Graph (PAG) A PAG can have three types of edges between
any nodes X and Y X Y X is a parent of Y X Y A confounder exists between X and Y X Y Not su ffi cient data to recover causal direction X Y X Y or or
45 x In real world case, the causal graphs can be very complex x It may be intractable to reason over the entire graph directly A real world causal graph for a
46 Extracting Causal Paths from the Causal Model Extract paths Always begins with a configuration option Or a system event Always terminates at a performance objective Cache
Ranking Causal Paths from the Causal Model Expected value of Bitrate when we artificially intervene by setting Bitrate to the value b Expected value of Branch Misses when we artificially intervene by setting Bitrate to the value a If this difference is large, then small changes to Bitrate will cause large changes to Branch Misses Average over all permitted values of Bitrate. ACE(BranchMisses, Bitrate) = 1 N ∑ E(BranchMisses|do(Bitrate = b)) − E(BranchMisses|do(Bitrate = a)) 47 Bitrate Branch
Misses FPS • There may be too many causal paths.
• We need to select the most useful ones.
• Compute the Average Causal E ff ect (ACE) of each pair of neighbors in a path.
48 Ranking Causal Paths from the Causal Model ● Average the ACE of all pairs of adjacent nodes in the path ● Rank paths from highest path ACE (PACE) score to the lowest ● Use the top K paths for subsequent analysis Sum over all pairs of nodes in the causal path. PACE (Z, Y) = 1 2 (ACE(Z, X) + ACE(X, Y)) Bitrate Branch
How to reason over a path? 49 To reason, we need to evaluate counterfactual queries that can be formulated using the con fi guration options and performance objectives in a particular path to resolve a particular performance task.
Counterfactual Queries 50 ● Counterfactual inference asks “what if” questions about changes to the misconfigurations We are interested in the scenario where: • We hypothetically have low throughput; Conditioned on the following events: • We hypothetically set the new Bitrate to 10000 • Bitrate was initially set to 6000 • We observed low throughput when Bitrate was set to 6000 • Everything else remains the same Example "Given that my current Bitrate is 6000 and I have low throughput, what is the probability of having low throughput if my Bitrate is increased to 10000"?
Selecting configuration for next intervention Top K paths Enumerate all possible changes Change with the largest ICE Set every configuration option in the path to all permitted values ICE (change) Inferred from observational data. This is very cheap 51 Bitrate Branch
Stage-V: Estimate Causal Queries 54 P(Throughput > 40/s|do(BufferSize = 20000)) Estimate the probability of satisfying QoS given Bu ff erSize=20000 • Use do-calculus to evaluate causal queries • Estimate budget and additional constraints
Results: Scalability 63 Discovery time, query evaluation time and total time do not increase exponentially as the number of con fi guration options and systems events are increased
68 Cache Misses Throughput (FPS) 20 10 0 100k 200k Cache Misses Throughput (FPS) LRU FIFO LIFO MRU 20 10 0 100k 200k Through- put Cache Misses Cache Policy 5- Estimate Causal Queries • What is the root-cause of fault? • How do I fix misconfiguration? • How do I optimize perf.? • How do I understand perf.? Software: DeepStream Middleware: TF, TensorRT Hardware: Nvidia Xavier Configuration: Default Estimate probability of satisfying QoS if BufferSize is set to 6k? 2- Learn Causal Performance Model Initial Perf. Data P(Th > 40/s|do(Buffersize = 6k)) 1- Specify Performance Query QoS : Th > 40/s Observed : Th < 30/s ± 5/s Causal Performance Model System Stack Performance Tasks Performance Fault/Issue 4- Update Causal Performance Model Causal Inference Engine 3- Determine Next Configuration Uses Stage Decoder Muxer Detector Tracker Classi fi er Causal reasoning enables more reliable performance analyses and more transferable performance models
Causal reasoning enables more reliable performance analyses and more transferable performance models 69 Cache Misses Throughput (FPS) 20 10 0 100k 200k Cache Misses Throughput (FPS) LRU FIFO LIFO MRU 20 10 0 100k 200k Through- put Cache Misses Cache Policy 5- Estimate Causal Queries • What is the root-cause of fault? • How do I fix misconfiguration? • How do I optimize perf.? • How do I understand perf.? Software: DeepStream Middleware: TF, TensorRT Hardware: Nvidia Xavier Configuration: Default Estimate probability of satisfying QoS if BufferSize is set to 6k? 2- Learn Causal Performance Model Initial Perf. Data P(Th > 40/s|do(Buffersize = 6k)) 1- Specify Performance Query QoS : Th > 40/s Observed : Th < 30/s ± 5/s Causal Performance Model System Stack Performance Tasks Performance Fault/Issue 4- Update Causal Performance Model Causal Inference Engine 3- Determine Next Configuration Uses Stage Decoder Muxer Detector Tracker Classi fi er
Maintaining performance in a highly configurable system is challenging 72 • The con fi guration space is combinatorially large with 1000's of con fi guration options.
Maintaining performance in a highly configurable system is challenging 73 • The con fi guration space is combinatorially large with 1000's of con fi guration options.
• Con fi guration options from each components interact non-trivially with one another.
Maintaining performance in a highly configurable system is challenging 74 • The con fi guration space is combinatorially large with 1000's of con fi guration options.
• Con fi guration options from each components interact non-trivially with one another.
• Individual component developers have a localized and limited understanding of the performance behavior of these systems.
Maintaining performance in a highly configurable system is challenging 75 • The con fi guration space is combinatorially large with 1000's of con fi guration options.
• Con fi guration options from each components interact non-trivially with one another.
• Individual component developers have a localized and limited understanding of the performance behavior of these systems.
• Each deployment needs to be con fi gured correctly which is prone to miscon fi gurations.
Maintaining performance in a highly configurable system is challenging 76 • The con fi guration space is combinatorially large with 1000's of con fi guration options.
• Con fi guration options from each components interact non-trivially with one another.
• Individual component developers have a localized and limited understanding of the performance behavior of these systems.
• Each deployment needs to be con fi gured correctly every time an environmental changes occur. Incorrect understanding about the performance
83 Performance Influence Model 0 5 10 15 20 25 30 35 40 45 50 Terms (a) Common Terms (Source ! Target) Total Terms (Source) Total Terms (Target) Error (Source) Error (Target) Error (Source ! Target) 0 30 60 90 Regression Models Causal Performance Model 0 5 10 15 20 25 30 35 40 45 50 0 30 60 90 Regression Models MAPE (%) Low error when reused High error when reused Common Predictors are Large Common Predictors are lower in number Causal models can be reliably reused when environmental changes occur. Why Causal Inference? - Accurate across Environments
pjamshidi
carlislerainey
eureco
nkmr_rl
konakalab
hidenorimatsubayashi
wata909
ianozsvald
ykamit
miyayou
florets1
joisino
destraynor
panda_program
jponch
brad_frost
samanthasiow
lauravandoore
jlugia
sachag
robhawkes
roundedbygravity
deanohume