Chef: Prototyping Symbolic Execution Engines for Interpreted Languages

Chef: Prototyping Symbolic Execution Engines for Interpreted Languages

66b2d5393e5e1a030eb52832855d9cbb?s=128

Stefan Bucur

March 03, 2014
Tweet

Transcript

  1. CHEF Prototyping Symbolic Execution Engines for Interpreted Languages School of

    Computer and Communication Sciences EPFL, Switzerland Stefan Bucur, Johannes Kinder, George Candea
  2. Automated Software Testing

  3. Automated Software Testing Symbolic execution

  4. Automated Software Testing Symbolic execution is used for bug finding,

    increasing coverage, debugging
  5. Automated Software Testing Symbolic execution is used for bug finding,

    increasing coverage, debugging and applied on device drivers, system utilities, file parsers, distributed systems,
  6. Automated Software Testing Symbolic execution is used for bug finding,

    increasing coverage, debugging and applied on device drivers, system utilities, file parsers, distributed systems, interpreted programs Today
  7. Symbolic Execution int foo(int x) { if (x > 10)

    { return 2*x; } x = x + 1; if (x > 5) { return 3*x; } else { return x; } }
  8. Symbolic Execution int foo(int x) { if (x > 10)

    { return 2*x; } x = x + 1; if (x > 5) { return 3*x; } else { return x; } } x⟵λ
  9. λ>10 λ≤10 Symbolic Execution int foo(int x) { if (x

    > 10) { return 2*x; } x = x + 1; if (x > 5) { return 3*x; } else { return x; } } x⟵λ
  10. λ>10 λ≤10 Symbolic Execution int foo(int x) { if (x

    > 10) { return 2*x; } x = x + 1; if (x > 5) { return 3*x; } else { return x; } } ret 2λ x⟵λ x⟵λ+1
  11. λ+1>5 λ>10 λ≤10 Symbolic Execution int foo(int x) { if

    (x > 10) { return 2*x; } x = x + 1; if (x > 5) { return 3*x; } else { return x; } } ret 2λ ret λ+1 λ+1≤5 x⟵λ x⟵λ+1 ret 3(λ+1)
  12. Test case: λ = 6 (λ≤10)∧(λ+1>5) λ+1>5 λ>10 λ≤10 Symbolic

    Execution int foo(int x) { if (x > 10) { return 2*x; } x = x + 1; if (x > 5) { return 3*x; } else { return x; } } ret 2λ ret λ+1 λ+1≤5 Test cases for bug finding and statement coverage x⟵λ x⟵λ+1 ret 3(λ+1)
  13. Programming Languages Symbolic Execution Engines

  14. Programming Languages Symbolic Execution Engines Scala Java C# C C++

  15. Programming Languages Symbolic Execution Engines Java Bytecode LLVM x86 ARM

    Scala Java C# C C++ Compiled
  16. Programming Languages Symbolic Execution Engines Java Bytecode LLVM x86 ARM

    Scala Java C# C C++ Compiled BitBlaze KLEE JPF SAGE S2E
  17. Programming Languages Symbolic Execution Engines Java Bytecode LLVM x86 ARM

    Scala Java C# C C++ Python Ruby Lua JavaScript Bash Perl Compiled BitBlaze KLEE JPF SAGE S2E
  18. Programming Languages Symbolic Execution Engines Java Bytecode LLVM x86 ARM

    Scala Java C# C C++ Python Ruby Lua JavaScript Bash Perl Compiled BitBlaze KLEE JPF SAGE S2E ?
  19. def parse_file(file_name): with open(file_name, "r") as f: data = f.read()

    return json.loads(data, encoding="utf-8") Interpreted Languages
  20. def parse_file(file_name): with open(file_name, "r") as f: data = f.read()

    return json.loads(data, encoding="utf-8") Interpreted Languages Complex semantics Complete File Read
  21. def parse_file(file_name): with open(file_name, "r") as f: data = f.read()

    return json.loads(data, encoding="utf-8") Interpreted Languages Complex semantics + Ambiguity in specifications Complete File Read Incomplete Specification
  22. def parse_file(file_name): with open(file_name, "r") as f: data = f.read()

    return json.loads(data, encoding="utf-8") Interpreted Languages Complex semantics + Ambiguity in specifications + Evolving language Since Python 2.5 Complete File Read Incomplete Specification
  23. def parse_file(file_name): with open(file_name, "r") as f: data = f.read()

    return json.loads(data, encoding="utf-8") Interpreted Languages Complex semantics + Ambiguity in specifications + Evolving language + Large standard library Since Python 2.5 Complete File Read Incomplete Specification
  24. def parse_file(file_name): with open(file_name, "r") as f: data = f.read()

    return json.loads(data, encoding="utf-8") Interpreted Languages Complex semantics + Ambiguity in specifications + Evolving language + Large standard library + Widespread native methods Since Python 2.5 Complete File Read Incomplete Specification
  25. def parse_file(file_name): with open(file_name, "r") as f: data = f.read()

    return json.loads(data, encoding="utf-8") Interpreted Languages Complex semantics + Ambiguity in specifications + Evolving language + Large standard library + Widespread native methods Since Python 2.5 Complete File Read Incomplete Specification Significant Engineering Effort
  26. “Consequently, if you were coming from Mars and tried to

    re-implement Python from this document alone, you might have to guess things and in fact you would probably end up implementing quite a different language.” - The Python Language Reference
  27. How can we efficiently obtain a correct symbolic execution engine?

  28. Key idea: Use the language interpreter as executable specification

  29. Symbolic Execution Engine for Language X CHEF Key idea: Use

    the language interpreter as executable specification Language X Interpreter
  30. Symbolic Execution Engine for Language X CHEF Program + Symbolic

    Tests Test Cases Key idea: Use the language interpreter as executable specification Language X Interpreter
  31. Chef Overview • Built on top of the S2E symbolic

    execution engine for x86 • Relies on lightweight interpreter instrumentation + optimizations • Prototyped engines for Python and Lua in 5 + 3 person-days
  32. Talk Outline 1. Challenges 2. Class-Uniform Path Analysis (CUPA) 3.

    Interpreter Recipe 4. Uses and Evaluation
  33. Talk Outline 1. Challenges 2. Class-Uniform Path Analysis (CUPA) 3.

    Interpreter Recipe 4. Uses and Evaluation
  34. Testing Interpreted Programs Naive approach: Run interpreter in a stock

    symbolic execution engine
  35. Testing Interpreted Programs def validateEmail(email): pos = email.find("@") if pos

    < 1: raise InvalidEmailError() if email.rfind(".") < pos: raise InvalidEmailError() Naive approach: Run interpreter in a stock symbolic execution engine
  36. Testing Interpreted Programs def validateEmail(email): pos = email.find("@") if pos

    < 1: raise InvalidEmailError() if email.rfind(".") < pos: raise InvalidEmailError() Naive approach: Run interpreter in a stock symbolic execution engine Python Interpreter ./python program.py
  37. Testing Interpreted Programs def validateEmail(email): pos = email.find("@") if pos

    < 1: raise InvalidEmailError() if email.rfind(".") < pos: raise InvalidEmailError() x86 Symbolic Execution Engine (S2E) Naive approach: Run interpreter in a stock symbolic execution engine Python Interpreter ./python program.py
  38. pos = email.find("@") Naive approach: Run interpreter in a stock

    symbolic execution engine
  39. Py_LOCAL_INLINE(Py_ssize_t) fastsearch(const STRINGLIB_CHAR* s, Py_ssize_t n, const STRINGLIB_CHAR* p, Py_ssize_t

    m, Py_ssize_t maxcount, int mode) { unsigned long mask; Py_ssize_t skip, count = 0; Py_ssize_t i, j, mlast, w; w = n - m; if (w < 0 || (mode == FAST_COUNT && maxcount == 0)) return -1; /* look for special cases */ if (m <= 1) { pos = email.find("@") Naive approach: Run interpreter in a stock symbolic execution engine
  40. STRINGLIB_BLOOM_ADD(mask, p[i]); if (p[i] == p[0]) skip = i -

    1; } for (i = w; i >= 0; i--) { if (s[i] == p[0]) { /* candidate match */ for (j = mlast; j > 0; j--) if (s[i+j] != p[j]) break; if (j == 0) /* got a match! */ return i; /* miss: check if previous character is part of if (i > 0 && !STRINGLIB_BLOOM(mask, s[i-1])) i = i - m; else i = i - skip; } else { /* skip: check if previous character is part of if (i > 0 && !STRINGLIB_BLOOM(mask, s[i-1])) i = i - m; } } } if (mode != FAST_COUNT) pos = email.find("@") Naive approach: Run interpreter in a stock symbolic execution engine
  41. STRINGLIB_BLOOM_ADD(mask, p[i]); if (p[i] == p[0]) skip = i -

    1; } for (i = w; i >= 0; i--) { if (s[i] == p[0]) { /* candidate match */ for (j = mlast; j > 0; j--) if (s[i+j] != p[j]) break; if (j == 0) /* got a match! */ return i; /* miss: check if previous character is part of if (i > 0 && !STRINGLIB_BLOOM(mask, s[i-1])) i = i - m; else i = i - skip; } else { /* skip: check if previous character is part of if (i > 0 && !STRINGLIB_BLOOM(mask, s[i-1])) i = i - m; } } } if (mode != FAST_COUNT) pos = email.find("@") Naive approach: Run interpreter in a stock symbolic execution engine Path Explosion
  42. STRINGLIB_BLOOM_ADD(mask, p[i]); if (p[i] == p[0]) skip = i -

    1; } for (i = w; i >= 0; i--) { if (s[i] == p[0]) { /* candidate match */ for (j = mlast; j > 0; j--) if (s[i+j] != p[j]) break; if (j == 0) /* got a match! */ return i; /* miss: check if previous character is part of if (i > 0 && !STRINGLIB_BLOOM(mask, s[i-1])) i = i - m; else i = i - skip; } else { /* skip: check if previous character is part of if (i > 0 && !STRINGLIB_BLOOM(mask, s[i-1])) i = i - m; } } } if (mode != FAST_COUNT) pos = email.find("@") Gets lost in the details of the implementation Naive approach: Run interpreter in a stock symbolic execution engine Path Explosion
  43. High-level Execution Paths def validateEmail(email): pos = email.find("@") if pos

    < 1: raise InvalidEmailError() if email.rfind(".") < pos: raise InvalidEmailError() High-level execution tree
  44. High-level Execution Paths def validateEmail(email): pos = email.find("@") if pos

    < 1: raise InvalidEmailError() if email.rfind(".") < pos: raise InvalidEmailError() High-level execution tree Low-level (x86) execution tree
  45. High-level Execution Paths def validateEmail(email): pos = email.find("@") if pos

    < 1: raise InvalidEmailError() if email.rfind(".") < pos: raise InvalidEmailError() High-level execution tree Low-level (x86) execution tree
  46. High-level Execution Paths def validateEmail(email): pos = email.find("@") if pos

    < 1: raise InvalidEmailError() if email.rfind(".") < pos: raise InvalidEmailError() High-level execution tree Low-level (x86) execution tree
  47. High-level Execution Paths def validateEmail(email): pos = email.find("@") if pos

    < 1: raise InvalidEmailError() if email.rfind(".") < pos: raise InvalidEmailError() HL/LL path ratio is low due to path explosion 3 HL paths 10 LL paths High-level execution tree Low-level (x86) execution tree
  48. Goal: Prioritize the low-level paths that maximize the HL/LL path

    ratio.
  49. High-level Execution Paths Alternative approach: Select states at high-level branches

  50. High-level Execution Paths Fork (low-level) Divergence (high-level) Alternative approach: Select

    states at high-level branches
  51. High-level Execution Paths Fork (low-level) Divergence (high-level) High-level fork points

    are unpredictable Alternative approach: Select states at high-level branches
  52. Talk Outline 1. Challenges 2. Class-Uniform Path Analysis (CUPA) 3.

    Interpreter Recipe 4. Uses and Evaluation
  53. Talk Outline 1. Challenges 2. Class-Uniform Path Analysis (CUPA) 3.

    Interpreter Recipe 4. Uses and Evaluation
  54. Reducing Path Explosion Program

  55. Reducing Path Explosion Fork points clustered in hot spots Program

    Low High Selection probability
  56. Reducing Path Explosion Fork points clustered in hot spots Program

    “Fork bomb” (e.g., input dependent loop) Low High Selection probability Global DFS / BFS / randomized strategy
  57. Reducing Path Explosion Fork points clustered in hot spots Clusters

    grow bigger 㱺 Slower overall progress Program “Fork bomb” (e.g., input dependent loop) Low High Selection probability Global DFS / BFS / randomized strategy
  58. Reducing Path Explosion Fork points clustered in hot spots Clusters

    grow bigger 㱺 Slower overall progress Program “Fork bomb” (e.g., input dependent loop) Low High Selection probability Reduced state diversity Global DFS / BFS / randomized strategy
  59. Reducing Path Explosion Program Idea: Partition the state space into

    groups
  60. Reducing Path Explosion Program Idea: Partition the state space into

    groups
  61. Reducing Path Explosion Program Idea: Partition the state space into

    groups Select group Select state from group
  62. Reducing Path Explosion Program Idea: Partition the state space into

    groups Select group Select state from group Faster progress across all groups
  63. Reducing Path Explosion Program Idea: Partition the state space into

    groups Select group Select state from group Faster progress across all groups Increased state diversity
  64. Class-Uniform Path Analysis

  65. Class-Uniform Path Analysis States arranged in a class hierarchy Class

    1 Class 2
  66. Class-Uniform Path Analysis States arranged in a class hierarchy Class

    1 Class 2
  67. Partitioning High-level Paths

  68. Partitioning High-level Paths High-level Instruction (Bytecode Instruction)

  69. Partitioning High-level Paths High-level Instruction (Bytecode Instruction)

  70. Partitioning High-level Paths High-level Instruction (Bytecode Instruction) High-level Program Counter

  71. Partitioning High-level Paths High-level Instruction (Bytecode Instruction) Low-level x86 PC

    High-level Program Counter
  72. Partitioning High-level Paths High-level Instruction (Bytecode Instruction) Low-level x86 PC

    High-level Program Counter 1st CUPA Class
  73. Partitioning High-level Paths High-level Instruction (Bytecode Instruction) Low-level x86 PC

    High-level Program Counter 1st CUPA Class 2nd CUPA Class Reconstruct high-level execution tree
  74. CUPA Classes 1. High-level PC • Uniform HL instruction exploration

    • Obtained via instrumentation 2. x86 PC • Uniform native method exploration • Approximated as the PC of fork point Coverage-optimized CUPA in the paper
  75. Talk Outline 1. Challenges 2. Class-Uniform Path Analysis (CUPA) 3.

    Interpreter Recipe 4. Uses and Evaluation
  76. Talk Outline 1. Challenges 2. Class-Uniform Path Analysis (CUPA) 3.

    Interpreter Recipe 4. Uses and Evaluation
  77. switch (opcode) { case LOAD: ... case STORE: ... case

    CALL_FUNCTION: ... ... } hlpc++; } Interpreter Loop Instrumentation while (true) { fetch_instr(hlpc, &opcode, &params);
  78. switch (opcode) { case LOAD: ... case STORE: ... case

    CALL_FUNCTION: ... ... } hlpc++; } Interpreter Loop Instrumentation while (true) { fetch_instr(hlpc, &opcode, &params); Reconstruct high-level execution tree and CFG chef_log_hlpc(hlpc, opcode);
  79. Interpreter Optimizations static long string_hash(PyStringObject *a) { #ifdef SYMBEX_HASHES return

    0; #else register Py_ssize_t len; register unsigned char *p; register long x; len = Py_SIZE(a); p = (unsigned char *) a->ob_sval; x = _Py_HashSecret.prefix; x ^= *p << 7; while (--len >= 0) x = (1000003*x) ^ *p++; x ^= Py_SIZE(a); x ^= _Py_HashSecret.suffix; if (x == -1) x = -2; return x; #endif } Hash neutralization
  80. Interpreter Optimizations • Simple changes to interpreter source static long

    string_hash(PyStringObject *a) { #ifdef SYMBEX_HASHES return 0; #else register Py_ssize_t len; register unsigned char *p; register long x; len = Py_SIZE(a); p = (unsigned char *) a->ob_sval; x = _Py_HashSecret.prefix; x ^= *p << 7; while (--len >= 0) x = (1000003*x) ^ *p++; x ^= Py_SIZE(a); x ^= _Py_HashSecret.suffix; if (x == -1) x = -2; return x; #endif } Hash neutralization
  81. Interpreter Optimizations • Simple changes to interpreter source • “Anti-optimizations”

    in linear performance... static long string_hash(PyStringObject *a) { #ifdef SYMBEX_HASHES return 0; #else register Py_ssize_t len; register unsigned char *p; register long x; len = Py_SIZE(a); p = (unsigned char *) a->ob_sval; x = _Py_HashSecret.prefix; x ^= *p << 7; while (--len >= 0) x = (1000003*x) ^ *p++; x ^= Py_SIZE(a); x ^= _Py_HashSecret.suffix; if (x == -1) x = -2; return x; #endif } Hash neutralization
  82. Interpreter Optimizations • Simple changes to interpreter source • “Anti-optimizations”

    in linear performance... • ... but exponential gains in symbolic mode static long string_hash(PyStringObject *a) { #ifdef SYMBEX_HASHES return 0; #else register Py_ssize_t len; register unsigned char *p; register long x; len = Py_SIZE(a); p = (unsigned char *) a->ob_sval; x = _Py_HashSecret.prefix; x ^= *p << 7; while (--len >= 0) x = (1000003*x) ^ *p++; x ^= Py_SIZE(a); x ^= _Py_HashSecret.suffix; if (x == -1) x = -2; return x; #endif } Hash neutralization
  83. Program + Symbolic Tests Symbolic Execution Engine for Language X

    Chef Summary Language X Interpreter (+instrumentation) CHEF API HL Tree Reconstr. CUPA State Selection S2E x86 Symbolic Execution CHEF
  84. Program + Symbolic Tests Symbolic Execution Engine for Language X

    Chef Summary Language X Interpreter (+instrumentation) CHEF API HL Tree Reconstr. CUPA State Selection S2E x86 Symbolic Execution CHEF
  85. Talk Outline 1. Challenges 2. Class-Uniform Path Analysis (CUPA) 3.

    Interpreter Recipe 4. Uses and Evaluation
  86. Talk Outline 1. Challenges 2. Class-Uniform Path Analysis (CUPA) 3.

    Interpreter Recipe 4. Uses and Evaluation
  87. Chef-Prototyped Engines Python 5 person-days 321 LoC Lua 3 person-days

    277 LoC
  88. Chef-Prototyped Engines Python 5 person-days 321 LoC Lua 3 person-days

    277 LoC
  89. Evaluation Questions How does a Chef-obtained engine... • ... work

    for test case generation? • ... benefit from CUPA and optimizations? • ... compare to a dedicated implementation?
  90. Using a Chef Engine class ArgparseTest(SymbolicTest): def setUp(self): self.argparse =

    import_module("argparse") def runTest(self): parser = self.argparse.ArgumentParser() arg_name = self.getSymString(size=3) arg_value = self.getSymString(size=3) parser.add_argument(arg_name) args = parser.parse_args([arg_value])
  91. Using a Chef Engine class ArgparseTest(SymbolicTest): def setUp(self): self.argparse =

    import_module("argparse") def runTest(self): parser = self.argparse.ArgumentParser() arg_name = self.getSymString(size=3) arg_value = self.getSymString(size=3) parser.add_argument(arg_name) args = parser.parse_args([arg_value])
  92. Using a Chef Engine class ArgparseTest(SymbolicTest): def setUp(self): self.argparse =

    import_module("argparse") def runTest(self): parser = self.argparse.ArgumentParser() arg_name = self.getSymString(size=3) arg_value = self.getSymString(size=3) parser.add_argument(arg_name) args = parser.parse_args([arg_value])
  93. Using a Chef Engine class ArgparseTest(SymbolicTest): def setUp(self): self.argparse =

    import_module("argparse") def runTest(self): parser = self.argparse.ArgumentParser() arg_name = self.getSymString(size=3) arg_value = self.getSymString(size=3) parser.add_argument(arg_name) args = parser.parse_args([arg_value])
  94. Using a Chef Engine class ArgparseTest(SymbolicTest): def setUp(self): self.argparse =

    import_module("argparse") def runTest(self): parser = self.argparse.ArgumentParser() arg_name = self.getSymString(size=3) arg_value = self.getSymString(size=3) parser.add_argument(arg_name) args = parser.parse_args([arg_value]) CHEF Symbolic Test Library Program
  95. Testing Python Packages xlrd simplejson argparse HTMLParser ConfigParser unicodecsv 6

    Popular Packages 10.9K lines of Python code 30 min. / package > 7,000 tests generated 4 undocumented exceptions found
  96. Testing Python Packages xlrd simplejson argparse HTMLParser ConfigParser unicodecsv 6

    Popular Packages 10.9K lines of Python code 30 min. / package > 7,000 tests generated 4 undocumented exceptions found High bug finding potential for dynamic languages
  97. xlrd simplejson argparse HTMLParser ConfigParser unicodecsv Efficiency Package 0.1 1

    10 100 1000 10000 Path Ratio (P / PBaseline) CUPA + Optimizations Baseline
  98. xlrd simplejson argparse HTMLParser ConfigParser unicodecsv Efficiency Package 0.1 1

    10 100 1000 10000 Path Ratio (P / PBaseline) CUPA + Optimizations Optimizations Only CUPA Only Baseline
  99. Comparison to Dedicated Engine • Symbolic execution engine of NICE

    [1] • Targets OpenFlow applications in Python • Case Study: Switch MAC learning algorithm [1] M. Canini, D. Venzano, P. Peresini, D. Kostic, and J. Rexford. “A NICE way to test OpenFlow applications.” NSDI 2012.
  100. Overhead 1 10 100 1000 1 2 3 4 5

    6 7 8 9 10 Size of Symbolic Input [# of Ethernet frames] CHEF Overhead TCHEF /TNICE
  101. Overhead 1 10 100 1000 1 2 3 4 5

    6 7 8 9 10 Size of Symbolic Input [# of Ethernet frames] CHEF Overhead TCHEF /TNICE >100×
  102. Overhead 1 10 100 1000 1 2 3 4 5

    6 7 8 9 10 Size of Symbolic Input [# of Ethernet frames] CHEF Overhead TCHEF /TNICE >100× 5× O ne-tim e Initialization
  103. Overhead 1 10 100 1000 1 2 3 4 5

    6 7 8 9 10 Size of Symbolic Input [# of Ethernet frames] CHEF Overhead TCHEF /TNICE >100× 5× 40× O ne-tim e Initialization x86 Reasoning Overhead (Instructions + Constraints)
  104. Chef Engine as Reference Implementation Chef-Python Reference Paths

  105. Test Cases Chef Engine as Reference Implementation NICE Dedicated Engine

    Chef-Python Reference Paths
  106. Test Cases Chef Engine as Reference Implementation NICE Dedicated Engine

    Chef-Python Reference Paths
  107. Test Cases Chef Engine as Reference Implementation NICE Dedicated Engine

    Missing Paths Duplicate Paths Chef-Python Reference Paths Chef engine’s correctness outweighs performance penalty
  108. Conclusions http://dslab.epfl.ch/proj/chef CHEF Program + Symbolic Tests Language X Interpreter

    Test Cases