Share
Embed
On Debugging the Performance of Configurable Software Systems: Developer Needs and Tailored Tool Support
Paper: https://arxiv.org/abs/2203.10356
Talk: https://youtu.be/VoegPhYnh84
Code: https://github.com/miguelvelezmj25/icse22-sm/
Pooyan Jamshidi
May 26, 2022
Transcript
139 slides
-
On Debugging the Performance of Configurable Software Systems Miguel Velez
(CMU -> Google) Norbert Siegmund (Leipzig) Pooyan Jamshidi (UofSC & Google) Sven Apel (Saarland) Christian Kästner (CMU) Developer Needs and Tailored Tool Support -
Developer • Design, implement, and maintain efficient software • Debug
surprising performance behaviors -
8 59% - Con fi guration-related performance issues 61% -
Average fi x time of 5 weeks 50% - Con fi guration-related patches -
14 Limited empirical evidence of the usefulness to help debug
the performance of con fi gurable systems -
Developer • Understand how options affect performance • Design, implement,
and maintain efficient software Help developers debug the performance of configurable software systems -
Identify Information Needs Influencing Options Option Hotspots Cause-Effect Chain Help
developers debug the performance of configurable software systems -
Identify Information Needs Influencing Options Option Hotspots Cause-Effect Chain Help
developers debug the performance of configurable software systems Tailor Ingredients CPU Profiling Program Slicing Performance Modeling Global Local -
24 User Study to Identify Information Needs Overview of Ingredients
Tailor and Evaluate Ingredients -
25 Overview of Ingredients Tailor and Evaluate Ingredients User Study
to Identify Information Needs -
Information Needs RQ1: What information do developers look for when
debugging the performance of con fi gurable systems? RQ2: What is the process that developers follow and the activities that they perform to obtain this information? -
Information Needs RQ1: What information do developers look for when
debugging the performance of con fi gurable systems? RQ2: What is the process that developers follow and the activities that they perform to obtain this information? Exploratory User Study -
Exploratory User Study Influencing Options The option or interaction causing
the unexpected performance behavior -
Exploratory User Study Influencing Options Option Hotspots The methods where
options affect the performance of the system -
Exploratory User Study Influencing Options Option Hotspots Cause-Effect Chain How
in fl uencing options are used in the implementation to directly and indirectly affect the performance of option hotspots -
Exploratory User Study Influencing Options Option Hotspots Cause-Effect Chain Compared
problematic con fi guration to a non-problematic con fi guration -
User Study to Identify Information Needs Influencing Options Option Hotspots
Cause-Effect Chain Help developers debug the performance of configurable software systems -
35 Tailor and Evaluate Ingredients User Study to Identify Information
Needs Overview of Ingredients -
The option or interaction causing the unexpected performance behavior Influencing
Options Global Performance-In fl uence Models -
… 23 seconds 28 seconds 20 seconds Con fi gurations
Performance T = 25 + 3· - 5· + 9· · Performance-Influence Models -
… 23 seconds 28 seconds 20 seconds Con fi gurations
Performance T = 25 + 3· - 5· + 9· · Performance-Influence Models -
… 23 seconds 28 seconds 20 seconds Con fi gurations
Performance T = 25 + 3· - 5· + 9· · Performance-Influence Models -
Modeling Global T = 25 + 3· - 5· +
9· · Influencing Options -
The methods where options affect the performance of the system
Option Hotspots -
The methods where options affect the performance of the system
Option Hotspots Local Performance-In fl uence Models -
Modeling Local Tfoo = 5 + 3· + 9· ·
Option Hotspots -
How in fl uencing options are used in the implementation
to directly and indirectly affect the performance of option hotspots Cause-Effect Chain -
How in fl uencing options are used in the implementation
to directly and indirectly affect the performance of option hotspots Cause-Effect Chain CPU Pro fi ling & Program Slicing -
Overview of Ingredients Influencing Options Option Hotspots Cause-Effect Chain Help
developers debug the performance of configurable software systems CPU Profiling Program Slicing Modeling Global Local -
50 User Study to Identify Information Needs Overview of Ingredients
Tailor and Evaluate Ingredients -
Influencing Options Option Hotspots Cause-Effect Chain CPU Profiling Program Slicing
Performance Modeling Global Local -
Influencing Options Option Hotspots Cause-Effect Chain CPU Profiling Program Slicing
Performance Modeling Global Local Information Providers -
Main.main(…) Cursor.put(…) Main.init(…) Database.init(…) def init(Database db) Database.checkForNullParam(db.name, "dbName"); Log.msg(Level.INFO,
"db " + db.name + " open"); ... if(db.replicated) configReplicated(...); ... this.cacheMode = db.cacheMode; this.commit = db.trans ? true : false; this.sync = db.dups ? true : false; if(db.evict) Evictor.init(db.evictorThreads); ... Cause-Effect Chain Program Slicing -
Main.main(…) Cursor.put(…) Main.init(…) Database.init(…) def init(Database db) Database.checkForNullParam(db.name, "dbName"); Log.msg(Level.INFO,
"db " + db.name + " open"); ... if(db.replicated) configReplicated(...); ... this.cacheMode = db.cacheMode; this.commit = db.trans ? true : false; this.sync = db.dups ? true : false; if(db.evict) Evictor.init(db.evictorThreads); ... Cause-Effect Chain Program Slicing -
Main.main(…) Cursor.put(…) Main.init(…) Database.init(…) def init(Database db) Database.checkForNullParam(db.name, "dbName"); Log.msg(Level.INFO,
"db " + db.name + " open"); ... if(db.replicated) configReplicated(...); ... this.cacheMode = db.cacheMode; this.commit = db.trans ? true : false; this.sync = db.dups ? true : false; if(db.evict) Evictor.init(db.evictorThreads); ... Cause-Effect Chain Program Slicing -
Main.main(…) Cursor.put(…) Main.init(…) Database.init(…) def init(Database db) Database.checkForNullParam(db.name, "dbName"); Log.msg(Level.INFO,
"db " + db.name + " open"); ... if(db.replicated) configReplicated(...); ... this.cacheMode = db.cacheMode; this.commit = db.trans ? true : false; this.sync = db.dups ? true : false; if(db.evict) Evictor.init(db.evictorThreads); ... Cause-Effect Chain Program Slicing -
Main.main(…) Cursor.put(…) Main.init(…) Database.init(…) def init(Database db) Database.checkForNullParam(db.name, "dbName"); Log.msg(Level.INFO,
"db " + db.name + " open"); ... if(db.replicated) configReplicated(...); ... this.cacheMode = db.cacheMode; this.commit = db.trans ? true : false; this.sync = db.dups ? true : false; if(db.evict) Evictor.init(db.evictorThreads); ... Cause-Effect Chain Program Slicing -
Main.main(…) Cursor.put(…) Main.init(…) Database.init(…) def init(Database db) Database.checkForNullParam(db.name, "dbName"); Log.msg(Level.INFO,
"db " + db.name + " open"); ... if(db.replicated) configReplicated(...); ... this.cacheMode = db.cacheMode; this.commit = db.trans ? true : false; this.sync = db.dups ? true : false; if(db.evict) Evictor.init(db.evictorThreads); ... Cause-Effect Chain Program Slicing -
Influencing Options Option Hotspots Cause-Effect Chain CPU Profiling Program Slicing
Performance Modeling Global Local Performance Modeling -
Influencing Options Option Hotspots Cause-Effect Chain CPU Profiling Program Slicing
Performance Modeling Global Local Performance Modeling GLIMPS -
Evaluation RQ1: To what extent do the designed information providers
help developers debug the performance of con fi gurable software systems? -
Evaluation RQ1: To what extent do the designed information providers
help developers debug the performance of con fi gurable software systems? Validation User Study Confirmatory User Study -
“Within-subject” study Validation User Study 30’ 35’ 45’ 05’ 10’
20’ 25’ 15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Control - Did not use GLIMPS Treatment - Used GLIMPS System: Density Converter -
Validation User Study 30’ 35’ 45’ 05’ 10’ 20’ 25’
15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Control - Did not use GLIMPS Treatment - Used GLIMPS “Within-subject” study System: Density Converter -
Validation User Study 30’ 35’ 45’ 05’ 10’ 20’ 25’
15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Control - Did not use GLIMPS Treatment - Used GLIMPS “Within-subject” study System: Density Converter -
Validation User Study 30’ 35’ 45’ 05’ 10’ 20’ 25’
15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Control - Did not use GLIMPS Treatment - Used GLIMPS “Within-subject” study System: Density Converter -
Validation User Study 30’ 35’ 45’ 05’ 10’ 20’ 25’
15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Control - Did not use GLIMPS Treatment - Used GLIMPS SCALE +86.3 AConverter.compress(…) +67.4 “Within-subject” study System: Density Converter -
Validation User Study 30’ 35’ 45’ 05’ 10’ 20’ 25’
15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Control - Did not use GLIMPS Treatment - Used GLIMPS SCALE +86.3 AConverter.compress(…) +67.4 “Within-subject” study System: Density Converter Information providers support information needs -
30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’
Influencing Options Option Hotspots Cause-Effect Chain Confirmatory User Study 55’ 60’ Treatment - Used GLIMPS Control - Did not use GLIMPS System: Berkeley DB Between-subject study -
30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’
Influencing Options Option Hotspots Cause-Effect Chain Confirmatory User Study 55’ 60’ Treatment - Used GLIMPS Control - Did not use GLIMPS System: Berkeley DB Between-subject study -
30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’
Influencing Options Option Hotspots Cause-Effect Chain Confirmatory User Study 55’ 60’ Treatment - Used GLIMPS Control - Did not use GLIMPS System: Berkeley DB Between-subject study -
30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’
Influencing Options Option Hotspots Cause-Effect Chain Confirmatory User Study 55’ 60’ Treatment - Used GLIMPS Control - Did not use GLIMPS System: Berkeley DB Between-subject study -
30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’
Influencing Options Option Hotspots Cause-Effect Chain Confirmatory User Study 55’ 60’ Treatment - Used GLIMPS Control - Did not use GLIMPS System: Berkeley DB Between-subject study -
30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’
Influencing Options Option Hotspots Cause-Effect Chain Confirmatory User Study 55’ 60’ Treatment - Used GLIMPS Control - Did not use GLIMPS System: Berkeley DB Between-subject study Information providers help developers debug the performance of complex con fi gurable systems -
Influencing Options Option Hotspots Cause-Effect Chain Help developers debug the
performance of configurable software systems CPU Profiling Program Slicing Performance Modeling Global Local Tailor and Evaluate Ingredients -
On Debugging the Performance of Configurable Software Systems: Developer Needs
and Tailored Tool Support -
96 Overview of Ingredients Tailor and Evaluate Ingredients CONTRIBUTION User
Study to Identify Information Needs CONTRIBUTION White-box Performance-In fl uence Modeling CONTRIBUTION -
Compositionality Performance-in fl uence models can be built by composing
models built independently for smaller regions of code System Performance-In fl uence Model Local Performance-In fl uence Model Region Local Performance-In fl uence Model Local Performance-In fl uence Model Local Performance-In fl uence Model Region Region Region Measure System Decompose Measure Regions Compose 98 -
Performance-In fl uence Model Local Performance-In fl uence Model Region
Local Performance-In fl uence Model Local Performance-In fl uence Model Local Performance-In fl uence Model Region Region Region Measure System Decompose Measure Regions Compose System 99 Compositionality Performance-in fl uence models can be built by composing models built independently for smaller regions of code -
Performance-In fl uence Model Local Performance-In fl uence Model Region
Local Performance-In fl uence Model Local Performance-In fl uence Model Local Performance-In fl uence Model Region Region Region Measure System Decompose Measure Regions Compose System 100 Compositionality Performance-in fl uence models can be built by composing models built independently for smaller regions of code -
Performance-In fl uence Model Local Performance-In fl uence Model Region
Local Performance-In fl uence Model Local Performance-In fl uence Model Local Performance-In fl uence Model Region Region Region Measure System Decompose Measure Regions Compose System 101 Compositionality Performance-in fl uence models can be built by composing models built independently for smaller regions of code -
Performance-In fl uence Model Local Performance-In fl uence Model Region
Local Performance-In fl uence Model Local Performance-In fl uence Model Local Performance-In fl uence Model Region Region Region Measure System Decompose Measure Regions Compose System 102 Compositionality Performance-in fl uence models can be built by composing models built independently for smaller regions of code -
def process(boolean x, boolean y) { if(x) convert(y); ... //
execution time: 5 seconds } def convert(boolean x) { if(x) ... // execution time: 3 seconds else ... // execution time: 2 seconds } 103 -
Time TRUE FALSE Time TRUE TRUE TRUE FALSE def process(boolean
x, boolean y) { if(x) convert(y); ... // execution time: 5 seconds } def convert(boolean x) { if(x) ... // execution time: 3 seconds else ... // execution time: 2 seconds } 104 -
Time TRUE 5 FALSE 0 def process(boolean x, boolean y)
{ if(x) convert(y); ... // execution time: 5 seconds } def convert(boolean x) { if(x) ... // execution time: 3 seconds else ... // execution time: 2 seconds } 105 Time TRUE TRUE TRUE FALSE -
Time TRUE 5 FALSE 0 Tprocess = 5· def process(boolean
x, boolean y) { if(x) convert(y); ... // execution time: 5 seconds } def convert(boolean x) { if(x) ... // execution time: 3 seconds else ... // execution time: 2 seconds } 106 Time TRUE TRUE TRUE FALSE -
Time TRUE 5 FALSE 0 Tprocess = 5· Time TRUE
TRUE 3 TRUE FALSE 2 Tconvert = 2· + 1·. · def process(boolean x, boolean y) { if(x) convert(y); ... // execution time: 5 seconds } def convert(boolean x) { if(x) ... // execution time: 3 seconds else ... // execution time: 2 seconds } 107 -
Tprocess = 5· Tconvert = 2· + 1·. · Compositionality
+ def process(boolean x, boolean y) { if(x) convert(y); ... // execution time: 5 seconds } def convert(boolean x) { if(x) ... // execution time: 3 seconds else ... // execution time: 2 seconds } 108 -
Tprocess = 5· Tconvert = 2· + 1·. · Compositionality
+ def process(boolean x, boolean y) { if(x) convert(y); ... // execution time: 5 seconds } def convert(boolean x) { if(x) ... // execution time: 3 seconds else ... // execution time: 2 seconds } T = 7· + 1· · 109 -
Compositionality System System 27 = 128 con fi gurations 22
+ 24 + 22 + 24 + 22 = 38 con fi gurations 110 -
Compositionality System System 27 = 128 con fi gurations 22
+ 24 + 22 + 24 + 22 = 38 con fi gurations 111 -
Compositionality System System 27 = 128 con fi gurations 22
+ 24 + 22 + 24 + 22 = 38 con fi gurations 112 -
Compression Compression allow us to simultaneously explore paths in multiple
independent regions with a few con fi gurations 114 -
Compression if(a) ... // execution time: 1 second if(b) ...
// execution time: 2 seconds if(c) ... // execution time: 3 seconds 115 -
Compression if(a) ... // execution time: 1 second if(b) ...
// execution time: 2 seconds if(c) ... // execution time: 3 seconds Time TRUE FALSE Time TRUE FALSE Time TRUE FALSE 116 -
FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE FALSE Compression
if(a) ... // execution time: 1 second if(b) ... // execution time: 2 seconds if(c) ... // execution time: 3 seconds Time TRUE FALSE Time TRUE FALSE Time TRUE FALSE 117 -
FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE FALSE if(a)
... // execution time: 1 second if(b) ... // execution time: 2 seconds if(c) ... // execution time: 3 seconds Time TRUE FALSE Time TRUE FALSE Time TRUE FALSE 118 Compression -
FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE FALSE if(a)
... // execution time: 1 second if(b) ... // execution time: 2 seconds if(c) ... // execution time: 3 seconds Time TRUE FALSE 0 Time TRUE FALSE 0 Time TRUE FALSE 0 119 Compression -
FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE FALSE if(a)
... // execution time: 1 second if(b) ... // execution time: 2 seconds if(c) ... // execution time: 3 seconds Time TRUE 1 FALSE 0 Time TRUE 3 FALSE 0 Time TRUE 2 FALSE 0 120 Compression -
System 22 + 24 + 22 + 24 + 22
= 38 con fi gurations System 4 con fi gurations 121 Compression -
System 22 + 24 + 22 + 24 + 22
= 38 con fi gurations System 4 con fi gurations 122 Compression -
System 22 + 24 + 22 + 24 + 22
= 38 con fi gurations System 4 con fi gurations 123 Compression -
System 4 con fi gurations FALSE FALSE FALSE FALSE FALSE
FALSE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE FALSE 124 Compression -
System 4 con fi gurations FALSE FALSE FALSE FALSE FALSE
FALSE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE FALSE 125 Compression -
System 4 con fi gurations FALSE FALSE FALSE FALSE FALSE
FALSE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE FALSE 126 Compression -
def main() { boolean a = ... process(a, b); }
def process(boolean x, boolean y) { if(x) convert(y); ... // execution time: 5 seconds } def convert(boolean x) { if(x) ... // execution time: 3 seconds else ... // execution time: 2 seconds } In fl uencing options 129 -
ConfigCrusher Comprex Static Taint Analysis Regions: Control-flow statements Performance measurement:
Instrumentation Dynamic Taint Analysis Regions: Methods Performance measurement: Off-the-shelf profiler 131 -
ConfigCrusher Comprex Static Taint Analysis Regions: Control-flow statements Dynamic Taint
Analysis Regions: Methods 132 Both prototypes ef fi ciently build accurate and interpretable models Dynamic taint analysis scales to larger systems Method-level granularity does not sacri fi ces compression potential -
134 ConfigCrusher Comprex Evaluate cost, accuracy, interpretability 13 open-source Java
systems Evaluation 50 combinations of Sampling and Machine Learning -
Results: Density Converter 0 Cost (minutes) Error (MAPE) 0 40
20 Comprex x 200 random con fi gurations & Random Forest 5 10 200 random con fi gurations & Stepwise linear regression 200 400 100 300 x x 60 Con fi gCrusher x 135 -
x + + + Results: Density Converter 0 Cost (minutes)
Error (MAPE) 0 40 20 Comprex 200 random con fi gurations & Random Forest 5 10 200 random con fi gurations & Stepwise linear regression 200 400 100 300 60 Con fi gCrusher + Interpretable x Not interpretable 136 -
x + + + Results: Density Converter 0 Cost (minutes)
Error (MAPE) 0 40 20 Comprex 200 random con fi gurations & Random Forest 5 10 200 random con fi gurations & Stepwise linear regression 200 400 100 300 60 Con fi gCrusher + Interpretable x Not interpretable 137 Ef fi ciently build accurate performance- in fl uence models Models are interpretable for debugging purposes -
Influencing Options Option Hotspots White-box Modeling Global Local 138 Help
developers debug the performance of configurable software systems White-box Performance-In fl uence Modeling CONTRIBUTION -
Thesis Statement Tailoring speci fi c white-box analyses to track
how con fi guration options in fl uence the performance of code-level structures in con fi gurable software systems helps developers to (1) ef fi ciently build accurate and interpretable global and local performance-in fl uence models (2) more easily inspect, trace, understand, and debug con fi guration-related performance issues. 139