Upgrade to Pro — share decks privately, control downloads, hide ads and more …

On Debugging the Performance of Configurable So...

On Debugging the Performance of Configurable Software Systems: Developer Needs and Tailored Tool Support

Pooyan Jamshidi

May 26, 2022
Tweet

More Decks by Pooyan Jamshidi

Other Decks in Research

Transcript

  1. On Debugging the Performance of Configurable Software Systems Miguel Velez

    (CMU -> Google) Norbert Siegmund (Leipzig) Pooyan Jamshidi (UofSC & Google) Sven Apel (Saarland) Christian Kästner (CMU) Developer Needs and Tailored Tool Support
  2. 8 59% - Con fi guration-related performance issues 61% -

    Average fi x time of 5 weeks 50% - Con fi guration-related patches
  3. 13

  4. 14 Limited empirical evidence of the usefulness to help debug

    the performance of con fi gurable systems
  5. 15

  6. 16

  7. Developer • Understand how options affect performance • Design, implement,

    and maintain efficient software Help developers debug the performance of configurable software systems
  8. Identify Information Needs Influencing Options Option Hotspots Cause-Effect Chain Help

    developers debug the performance of configurable software systems
  9. Identify Information Needs Influencing Options Option Hotspots Cause-Effect Chain Help

    developers debug the performance of configurable software systems Tailor Ingredients CPU Profiling Program Slicing Performance Modeling Global Local
  10. Information Needs RQ1: What information do developers look for when

    debugging the performance of con fi gurable systems? RQ2: What is the process that developers follow and the activities that they perform to obtain this information?
  11. Information Needs RQ1: What information do developers look for when

    debugging the performance of con fi gurable systems? RQ2: What is the process that developers follow and the activities that they perform to obtain this information? Exploratory User Study
  12. Exploratory User Study Influencing Options Option Hotspots Cause-Effect Chain How

    in fl uencing options are used in the implementation to directly and indirectly affect the performance of option hotspots
  13. Exploratory User Study Influencing Options Option Hotspots Cause-Effect Chain Compared

    problematic con fi guration to a non-problematic con fi guration
  14. User Study to Identify Information Needs Influencing Options Option Hotspots

    Cause-Effect Chain Help developers debug the performance of configurable software systems
  15. … 23 seconds 28 seconds 20 seconds Con fi gurations

    Performance T = 25 + 3· - 5· + 9· · Performance-Influence Models
  16. … 23 seconds 28 seconds 20 seconds Con fi gurations

    Performance T = 25 + 3· - 5· + 9· · Performance-Influence Models
  17. … 23 seconds 28 seconds 20 seconds Con fi gurations

    Performance T = 25 + 3· - 5· + 9· · Performance-Influence Models
  18. Modeling Global T = 25 + 3· - 5· +

    9· · Influencing Options
  19. The methods where options affect the performance of the system

    Option Hotspots Local Performance-In fl uence Models
  20. How in fl uencing options are used in the implementation

    to directly and indirectly affect the performance of option hotspots Cause-Effect Chain
  21. How in fl uencing options are used in the implementation

    to directly and indirectly affect the performance of option hotspots Cause-Effect Chain CPU Pro fi ling & Program Slicing
  22. Overview of Ingredients Influencing Options Option Hotspots Cause-Effect Chain Help

    developers debug the performance of configurable software systems CPU Profiling Program Slicing Modeling Global Local
  23. Influencing Options Option Hotspots Cause-Effect Chain CPU Profiling Program Slicing

    Performance Modeling Global Local Information Providers
  24. Main.main(…) Cursor.put(…) Main.init(…) Database.init(…) def init(Database db) Database.checkForNullParam(db.name, "dbName"); Log.msg(Level.INFO,

    "db " + db.name + " open"); ... if(db.replicated) configReplicated(...); ... this.cacheMode = db.cacheMode; this.commit = db.trans ? true : false; this.sync = db.dups ? true : false; if(db.evict) Evictor.init(db.evictorThreads); ... Cause-Effect Chain Program Slicing
  25. Main.main(…) Cursor.put(…) Main.init(…) Database.init(…) def init(Database db) Database.checkForNullParam(db.name, "dbName"); Log.msg(Level.INFO,

    "db " + db.name + " open"); ... if(db.replicated) configReplicated(...); ... this.cacheMode = db.cacheMode; this.commit = db.trans ? true : false; this.sync = db.dups ? true : false; if(db.evict) Evictor.init(db.evictorThreads); ... Cause-Effect Chain Program Slicing
  26. Main.main(…) Cursor.put(…) Main.init(…) Database.init(…) def init(Database db) Database.checkForNullParam(db.name, "dbName"); Log.msg(Level.INFO,

    "db " + db.name + " open"); ... if(db.replicated) configReplicated(...); ... this.cacheMode = db.cacheMode; this.commit = db.trans ? true : false; this.sync = db.dups ? true : false; if(db.evict) Evictor.init(db.evictorThreads); ... Cause-Effect Chain Program Slicing
  27. Main.main(…) Cursor.put(…) Main.init(…) Database.init(…) def init(Database db) Database.checkForNullParam(db.name, "dbName"); Log.msg(Level.INFO,

    "db " + db.name + " open"); ... if(db.replicated) configReplicated(...); ... this.cacheMode = db.cacheMode; this.commit = db.trans ? true : false; this.sync = db.dups ? true : false; if(db.evict) Evictor.init(db.evictorThreads); ... Cause-Effect Chain Program Slicing
  28. Main.main(…) Cursor.put(…) Main.init(…) Database.init(…) def init(Database db) Database.checkForNullParam(db.name, "dbName"); Log.msg(Level.INFO,

    "db " + db.name + " open"); ... if(db.replicated) configReplicated(...); ... this.cacheMode = db.cacheMode; this.commit = db.trans ? true : false; this.sync = db.dups ? true : false; if(db.evict) Evictor.init(db.evictorThreads); ... Cause-Effect Chain Program Slicing
  29. Main.main(…) Cursor.put(…) Main.init(…) Database.init(…) def init(Database db) Database.checkForNullParam(db.name, "dbName"); Log.msg(Level.INFO,

    "db " + db.name + " open"); ... if(db.replicated) configReplicated(...); ... this.cacheMode = db.cacheMode; this.commit = db.trans ? true : false; this.sync = db.dups ? true : false; if(db.evict) Evictor.init(db.evictorThreads); ... Cause-Effect Chain Program Slicing
  30. Influencing Options Option Hotspots Cause-Effect Chain CPU Profiling Program Slicing

    Performance Modeling Global Local Performance Modeling
  31. Influencing Options Option Hotspots Cause-Effect Chain CPU Profiling Program Slicing

    Performance Modeling Global Local Performance Modeling GLIMPS
  32. Evaluation RQ1: To what extent do the designed information providers

    help developers debug the performance of con fi gurable software systems?
  33. Evaluation RQ1: To what extent do the designed information providers

    help developers debug the performance of con fi gurable software systems? Validation User Study Confirmatory User Study
  34. “Within-subject” study Validation User Study 30’ 35’ 45’ 05’ 10’

    20’ 25’ 15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Control - Did not use GLIMPS Treatment - Used GLIMPS System: Density Converter
  35. Validation User Study 30’ 35’ 45’ 05’ 10’ 20’ 25’

    15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Control - Did not use GLIMPS Treatment - Used GLIMPS “Within-subject” study System: Density Converter
  36. Validation User Study 30’ 35’ 45’ 05’ 10’ 20’ 25’

    15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Control - Did not use GLIMPS Treatment - Used GLIMPS “Within-subject” study System: Density Converter
  37. Validation User Study 30’ 35’ 45’ 05’ 10’ 20’ 25’

    15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Control - Did not use GLIMPS Treatment - Used GLIMPS “Within-subject” study System: Density Converter
  38. Validation User Study 30’ 35’ 45’ 05’ 10’ 20’ 25’

    15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Control - Did not use GLIMPS Treatment - Used GLIMPS SCALE +86.3 AConverter.compress(…) +67.4 “Within-subject” study System: Density Converter
  39. Validation User Study 30’ 35’ 45’ 05’ 10’ 20’ 25’

    15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Control - Did not use GLIMPS Treatment - Used GLIMPS SCALE +86.3 AConverter.compress(…) +67.4 “Within-subject” study System: Density Converter Information providers support information needs
  40. 30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’

    Influencing Options Option Hotspots Cause-Effect Chain Confirmatory User Study 55’ 60’ Treatment - Used GLIMPS Control - Did not use GLIMPS System: Berkeley DB Between-subject study
  41. 30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’

    Influencing Options Option Hotspots Cause-Effect Chain Confirmatory User Study 55’ 60’ Treatment - Used GLIMPS Control - Did not use GLIMPS System: Berkeley DB Between-subject study
  42. 30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’

    Influencing Options Option Hotspots Cause-Effect Chain Confirmatory User Study 55’ 60’ Treatment - Used GLIMPS Control - Did not use GLIMPS System: Berkeley DB Between-subject study
  43. 30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’

    Influencing Options Option Hotspots Cause-Effect Chain Confirmatory User Study 55’ 60’ Treatment - Used GLIMPS Control - Did not use GLIMPS System: Berkeley DB Between-subject study
  44. 30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’

    Influencing Options Option Hotspots Cause-Effect Chain Confirmatory User Study 55’ 60’ Treatment - Used GLIMPS Control - Did not use GLIMPS System: Berkeley DB Between-subject study
  45. 30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’

    Influencing Options Option Hotspots Cause-Effect Chain Confirmatory User Study 55’ 60’ Treatment - Used GLIMPS Control - Did not use GLIMPS System: Berkeley DB Between-subject study Information providers help developers debug the performance of complex con fi gurable systems
  46. Influencing Options Option Hotspots Cause-Effect Chain Help developers debug the

    performance of configurable software systems CPU Profiling Program Slicing Performance Modeling Global Local Tailor and Evaluate Ingredients
  47. 96 Overview of Ingredients Tailor and Evaluate Ingredients CONTRIBUTION User

    Study to Identify Information Needs CONTRIBUTION White-box Performance-In fl uence Modeling CONTRIBUTION
  48. Compositionality Performance-in fl uence models can be built by composing

    models built independently for smaller regions of code System Performance-In fl uence Model Local Performance-In fl uence Model Region Local Performance-In fl uence Model Local Performance-In fl uence Model Local Performance-In fl uence Model Region Region Region Measure System Decompose Measure Regions Compose 98
  49. Performance-In fl uence Model Local Performance-In fl uence Model Region

    Local Performance-In fl uence Model Local Performance-In fl uence Model Local Performance-In fl uence Model Region Region Region Measure System Decompose Measure Regions Compose System 99 Compositionality Performance-in fl uence models can be built by composing models built independently for smaller regions of code
  50. Performance-In fl uence Model Local Performance-In fl uence Model Region

    Local Performance-In fl uence Model Local Performance-In fl uence Model Local Performance-In fl uence Model Region Region Region Measure System Decompose Measure Regions Compose System 100 Compositionality Performance-in fl uence models can be built by composing models built independently for smaller regions of code
  51. Performance-In fl uence Model Local Performance-In fl uence Model Region

    Local Performance-In fl uence Model Local Performance-In fl uence Model Local Performance-In fl uence Model Region Region Region Measure System Decompose Measure Regions Compose System 101 Compositionality Performance-in fl uence models can be built by composing models built independently for smaller regions of code
  52. Performance-In fl uence Model Local Performance-In fl uence Model Region

    Local Performance-In fl uence Model Local Performance-In fl uence Model Local Performance-In fl uence Model Region Region Region Measure System Decompose Measure Regions Compose System 102 Compositionality Performance-in fl uence models can be built by composing models built independently for smaller regions of code
  53. def process(boolean x, boolean y) { if(x) convert(y); ... //

    execution time: 5 seconds } def convert(boolean x) { if(x) ... // execution time: 3 seconds else ... // execution time: 2 seconds } 103
  54. Time TRUE FALSE Time TRUE TRUE TRUE FALSE def process(boolean

    x, boolean y) { if(x) convert(y); ... // execution time: 5 seconds } def convert(boolean x) { if(x) ... // execution time: 3 seconds else ... // execution time: 2 seconds } 104
  55. Time TRUE 5 FALSE 0 def process(boolean x, boolean y)

    { if(x) convert(y); ... // execution time: 5 seconds } def convert(boolean x) { if(x) ... // execution time: 3 seconds else ... // execution time: 2 seconds } 105 Time TRUE TRUE TRUE FALSE
  56. Time TRUE 5 FALSE 0 Tprocess = 5· def process(boolean

    x, boolean y) { if(x) convert(y); ... // execution time: 5 seconds } def convert(boolean x) { if(x) ... // execution time: 3 seconds else ... // execution time: 2 seconds } 106 Time TRUE TRUE TRUE FALSE
  57. Time TRUE 5 FALSE 0 Tprocess = 5· Time TRUE

    TRUE 3 TRUE FALSE 2 Tconvert = 2· + 1·. · def process(boolean x, boolean y) { if(x) convert(y); ... // execution time: 5 seconds } def convert(boolean x) { if(x) ... // execution time: 3 seconds else ... // execution time: 2 seconds } 107
  58. Tprocess = 5· Tconvert = 2· + 1·. · Compositionality

    + def process(boolean x, boolean y) { if(x) convert(y); ... // execution time: 5 seconds } def convert(boolean x) { if(x) ... // execution time: 3 seconds else ... // execution time: 2 seconds } 108
  59. Tprocess = 5· Tconvert = 2· + 1·. · Compositionality

    + def process(boolean x, boolean y) { if(x) convert(y); ... // execution time: 5 seconds } def convert(boolean x) { if(x) ... // execution time: 3 seconds else ... // execution time: 2 seconds } T = 7· + 1· · 109
  60. Compositionality System System 27 = 128 con fi gurations 22

    + 24 + 22 + 24 + 22 = 38 con fi gurations 110
  61. Compositionality System System 27 = 128 con fi gurations 22

    + 24 + 22 + 24 + 22 = 38 con fi gurations 111
  62. Compositionality System System 27 = 128 con fi gurations 22

    + 24 + 22 + 24 + 22 = 38 con fi gurations 112
  63. Compression Compression allow us to simultaneously explore paths in multiple

    independent regions with a few con fi gurations 114
  64. Compression if(a) ... // execution time: 1 second if(b) ...

    // execution time: 2 seconds if(c) ... // execution time: 3 seconds 115
  65. Compression if(a) ... // execution time: 1 second if(b) ...

    // execution time: 2 seconds if(c) ... // execution time: 3 seconds Time TRUE FALSE Time TRUE FALSE Time TRUE FALSE 116
  66. FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE FALSE Compression

    if(a) ... // execution time: 1 second if(b) ... // execution time: 2 seconds if(c) ... // execution time: 3 seconds Time TRUE FALSE Time TRUE FALSE Time TRUE FALSE 117
  67. FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE FALSE if(a)

    ... // execution time: 1 second if(b) ... // execution time: 2 seconds if(c) ... // execution time: 3 seconds Time TRUE FALSE Time TRUE FALSE Time TRUE FALSE 118 Compression
  68. FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE FALSE if(a)

    ... // execution time: 1 second if(b) ... // execution time: 2 seconds if(c) ... // execution time: 3 seconds Time TRUE FALSE 0 Time TRUE FALSE 0 Time TRUE FALSE 0 119 Compression
  69. FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE FALSE if(a)

    ... // execution time: 1 second if(b) ... // execution time: 2 seconds if(c) ... // execution time: 3 seconds Time TRUE 1 FALSE 0 Time TRUE 3 FALSE 0 Time TRUE 2 FALSE 0 120 Compression
  70. System 22 + 24 + 22 + 24 + 22

    = 38 con fi gurations System 4 con fi gurations 121 Compression
  71. System 22 + 24 + 22 + 24 + 22

    = 38 con fi gurations System 4 con fi gurations 122 Compression
  72. System 22 + 24 + 22 + 24 + 22

    = 38 con fi gurations System 4 con fi gurations 123 Compression
  73. System 4 con fi gurations FALSE FALSE FALSE FALSE FALSE

    FALSE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE FALSE 124 Compression
  74. System 4 con fi gurations FALSE FALSE FALSE FALSE FALSE

    FALSE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE FALSE 125 Compression
  75. System 4 con fi gurations FALSE FALSE FALSE FALSE FALSE

    FALSE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE FALSE 126 Compression
  76. def main() { boolean a = ... process(a, b); }

    def process(boolean x, boolean y) { if(x) convert(y); ... // execution time: 5 seconds } def convert(boolean x) { if(x) ... // execution time: 3 seconds else ... // execution time: 2 seconds } In fl uencing options 129
  77. ConfigCrusher Comprex Static Taint Analysis Regions: Control-flow statements Performance measurement:

    Instrumentation Dynamic Taint Analysis Regions: Methods Performance measurement: Off-the-shelf profiler 131
  78. ConfigCrusher Comprex Static Taint Analysis Regions: Control-flow statements Dynamic Taint

    Analysis Regions: Methods 132 Both prototypes ef fi ciently build accurate and interpretable models Dynamic taint analysis scales to larger systems Method-level granularity does not sacri fi ces compression potential
  79. 134 ConfigCrusher Comprex Evaluate cost, accuracy, interpretability 13 open-source Java

    systems Evaluation 50 combinations of Sampling and Machine Learning
  80. Results: Density Converter 0 Cost (minutes) Error (MAPE) 0 40

    20 Comprex x 200 random con fi gurations & Random Forest 5 10 200 random con fi gurations & Stepwise linear regression 200 400 100 300 x x 60 Con fi gCrusher x 135
  81. x + + + Results: Density Converter 0 Cost (minutes)

    Error (MAPE) 0 40 20 Comprex 200 random con fi gurations & Random Forest 5 10 200 random con fi gurations & Stepwise linear regression 200 400 100 300 60 Con fi gCrusher + Interpretable x Not interpretable 136
  82. x + + + Results: Density Converter 0 Cost (minutes)

    Error (MAPE) 0 40 20 Comprex 200 random con fi gurations & Random Forest 5 10 200 random con fi gurations & Stepwise linear regression 200 400 100 300 60 Con fi gCrusher + Interpretable x Not interpretable 137 Ef fi ciently build accurate performance- in fl uence models Models are interpretable for debugging purposes
  83. Influencing Options Option Hotspots White-box Modeling Global Local 138 Help

    developers debug the performance of configurable software systems White-box Performance-In fl uence Modeling CONTRIBUTION
  84. Thesis Statement Tailoring speci fi c white-box analyses to track

    how con fi guration options in fl uence the performance of code-level structures in con fi gurable software systems helps developers to (1) ef fi ciently build accurate and interpretable global and local performance-in fl uence models (2) more easily inspect, trace, understand, and debug con fi guration-related performance issues. 139