On Debugging the Performance of Configurable Software Systems Miguel Velez (CMU -> Google) Norbert Siegmund (Leipzig) Pooyan Jamshidi (UofSC & Google) Sven Apel (Saarland) Christian Kästner (CMU) Developer Needs and Tailored Tool Support
Developer • Understand how options affect performance • Design, implement, and maintain efficient software Help developers debug the performance of configurable software systems
Identify Information Needs Influencing Options Option Hotspots Cause-Effect Chain Help developers debug the performance of configurable software systems
Identify Information Needs Influencing Options Option Hotspots Cause-Effect Chain Help developers debug the performance of configurable software systems Tailor Ingredients CPU Profiling Program Slicing Performance Modeling Global Local
Information Needs RQ1: What information do developers look for when debugging the performance of con fi gurable systems? RQ2: What is the process that developers follow and the activities that they perform to obtain this information?
Information Needs RQ1: What information do developers look for when debugging the performance of con fi gurable systems? RQ2: What is the process that developers follow and the activities that they perform to obtain this information? Exploratory User Study
Exploratory User Study Influencing Options Option Hotspots Cause-Effect Chain How in fl uencing options are used in the implementation to directly and indirectly affect the performance of option hotspots
Exploratory User Study Influencing Options Option Hotspots Cause-Effect Chain Compared problematic con fi guration to a non-problematic con fi guration
User Study to Identify Information Needs Influencing Options Option Hotspots Cause-Effect Chain Help developers debug the performance of configurable software systems
How in fl uencing options are used in the implementation to directly and indirectly affect the performance of option hotspots Cause-Effect Chain CPU Pro fi ling & Program Slicing
Overview of Ingredients Influencing Options Option Hotspots Cause-Effect Chain Help developers debug the performance of configurable software systems CPU Profiling Program Slicing Modeling Global Local
Evaluation RQ1: To what extent do the designed information providers help developers debug the performance of con fi gurable software systems? Validation User Study Confirmatory User Study
“Within-subject” study Validation User Study 30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Control - Did not use GLIMPS Treatment - Used GLIMPS System: Density Converter
Validation User Study 30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Control - Did not use GLIMPS Treatment - Used GLIMPS “Within-subject” study System: Density Converter
Validation User Study 30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Control - Did not use GLIMPS Treatment - Used GLIMPS “Within-subject” study System: Density Converter
Validation User Study 30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Control - Did not use GLIMPS Treatment - Used GLIMPS “Within-subject” study System: Density Converter
Validation User Study 30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Control - Did not use GLIMPS Treatment - Used GLIMPS SCALE +86.3 AConverter.compress(…) +67.4 “Within-subject” study System: Density Converter
Validation User Study 30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Control - Did not use GLIMPS Treatment - Used GLIMPS SCALE +86.3 AConverter.compress(…) +67.4 “Within-subject” study System: Density Converter Information providers support information needs
30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Confirmatory User Study 55’ 60’ Treatment - Used GLIMPS Control - Did not use GLIMPS System: Berkeley DB Between-subject study
30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Confirmatory User Study 55’ 60’ Treatment - Used GLIMPS Control - Did not use GLIMPS System: Berkeley DB Between-subject study
30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Confirmatory User Study 55’ 60’ Treatment - Used GLIMPS Control - Did not use GLIMPS System: Berkeley DB Between-subject study
30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Confirmatory User Study 55’ 60’ Treatment - Used GLIMPS Control - Did not use GLIMPS System: Berkeley DB Between-subject study
30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Confirmatory User Study 55’ 60’ Treatment - Used GLIMPS Control - Did not use GLIMPS System: Berkeley DB Between-subject study
30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Confirmatory User Study 55’ 60’ Treatment - Used GLIMPS Control - Did not use GLIMPS System: Berkeley DB Between-subject study Information providers help developers debug the performance of complex con fi gurable systems
Influencing Options Option Hotspots Cause-Effect Chain Help developers debug the performance of configurable software systems CPU Profiling Program Slicing Performance Modeling Global Local Tailor and Evaluate Ingredients
96 Overview of Ingredients Tailor and Evaluate Ingredients CONTRIBUTION User Study to Identify Information Needs CONTRIBUTION White-box Performance-In fl uence Modeling CONTRIBUTION
Compositionality Performance-in fl uence models can be built by composing models built independently for smaller regions of code System Performance-In fl uence Model Local Performance-In fl uence Model Region Local Performance-In fl uence Model Local Performance-In fl uence Model Local Performance-In fl uence Model Region Region Region Measure System Decompose Measure Regions Compose 98
Performance-In fl uence Model Local Performance-In fl uence Model Region Local Performance-In fl uence Model Local Performance-In fl uence Model Local Performance-In fl uence Model Region Region Region Measure System Decompose Measure Regions Compose System 99 Compositionality Performance-in fl uence models can be built by composing models built independently for smaller regions of code
Performance-In fl uence Model Local Performance-In fl uence Model Region Local Performance-In fl uence Model Local Performance-In fl uence Model Local Performance-In fl uence Model Region Region Region Measure System Decompose Measure Regions Compose System 100 Compositionality Performance-in fl uence models can be built by composing models built independently for smaller regions of code
Performance-In fl uence Model Local Performance-In fl uence Model Region Local Performance-In fl uence Model Local Performance-In fl uence Model Local Performance-In fl uence Model Region Region Region Measure System Decompose Measure Regions Compose System 101 Compositionality Performance-in fl uence models can be built by composing models built independently for smaller regions of code
Performance-In fl uence Model Local Performance-In fl uence Model Region Local Performance-In fl uence Model Local Performance-In fl uence Model Local Performance-In fl uence Model Region Region Region Measure System Decompose Measure Regions Compose System 102 Compositionality Performance-in fl uence models can be built by composing models built independently for smaller regions of code
ConfigCrusher Comprex Static Taint Analysis Regions: Control-flow statements Dynamic Taint Analysis Regions: Methods 132 Both prototypes ef fi ciently build accurate and interpretable models Dynamic taint analysis scales to larger systems Method-level granularity does not sacri fi ces compression potential
Results: Density Converter 0 Cost (minutes) Error (MAPE) 0 40 20 Comprex x 200 random con fi gurations & Random Forest 5 10 200 random con fi gurations & Stepwise linear regression 200 400 100 300 x x 60 Con fi gCrusher x 135
x + + + Results: Density Converter 0 Cost (minutes) Error (MAPE) 0 40 20 Comprex 200 random con fi gurations & Random Forest 5 10 200 random con fi gurations & Stepwise linear regression 200 400 100 300 60 Con fi gCrusher + Interpretable x Not interpretable 136
x + + + Results: Density Converter 0 Cost (minutes) Error (MAPE) 0 40 20 Comprex 200 random con fi gurations & Random Forest 5 10 200 random con fi gurations & Stepwise linear regression 200 400 100 300 60 Con fi gCrusher + Interpretable x Not interpretable 137 Ef fi ciently build accurate performance- in fl uence models Models are interpretable for debugging purposes
Influencing Options Option Hotspots White-box Modeling Global Local 138 Help developers debug the performance of configurable software systems White-box Performance-In fl uence Modeling CONTRIBUTION
Thesis Statement Tailoring speci fi c white-box analyses to track how con fi guration options in fl uence the performance of code-level structures in con fi gurable software systems helps developers to (1) ef fi ciently build accurate and interpretable global and local performance-in fl uence models (2) more easily inspect, trace, understand, and debug con fi guration-related performance issues. 139
pjamshidi
sh01k
fullflu
javiergs
ooshita
wide_vsix
sgnm
kitachan_black
hicustomer
takehaya
shunsukesaito
masatoto
vanstee
jnunemaker
thoeni
davidbonilla
roundedbygravity
palkan
paulrobertlloyd
marcelosomers
moore
tmm1
reverentgeek
mclloyd