Cloud9: Parallel Symbolic Execution for Automated Real-World Software Testing

Transcript

1. Parallel Symbolic Execution for Automated Real-World Software Testing Stefan Bucur, Vlad Ureche, Cristian Zam r, George Candea Cloud9 School of Computer and Communication Sciences
2. Automated Techniques Automated Software Testing 2 λ Symbolic Execution Model Checking Industrial SW Testing Manual Testing Static Analysis Fuzzing Scalability Applicability Usability
3. Cloud9 - The Big Picture • Parallel symbolic execution • Linear scalability on commodity clusters • Full symbolic POSIX support • Applicable on real-world systems • Platform for writing test cases • Easy-to-use platform API 3
4. Automated Systems Testing 4 [*] C. Cadar, D. Dunbar, D. Engler, “KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs”, OSDI 2008 • Promising for systems testing: KLEE [*] • High-coverage test cases • Found new bugs • ... But applied only on small programs λ Symbolic Execution
5. 5 Memcached GNU Coreutils Apache
6. void proc_pkt(packet_t* pkt) { if (pkt->magic != 0xC9) { err(pkt); return; } if (pkt->cmd == GET) { ... } else if ... ... } Symbolic Execution in a Nutshell [C9 A0 ... ] 6
7. void proc_pkt(packet_t* pkt) { if (pkt->magic != 0xC9) { err(pkt); return; } if (pkt->cmd == GET) { ... } else if ... ... } Symbolic Execution in a Nutshell [C9 A0 ... ] 6
8. pkt->magic != 0xC9 void proc_pkt(packet_t* pkt) { if (pkt->magic != 0xC9) { err(pkt); return; } if (pkt->cmd == GET) { ... } else if ... ... } Symbolic Execution in a Nutshell [C9 A0 ... ] 6
9. pkt->cmd == GET pkt->magic != 0xC9 void proc_pkt(packet_t* pkt) { if (pkt->magic != 0xC9) { err(pkt); return; } if (pkt->cmd == GET) { ... } else if ... ... } Symbolic Execution in a Nutshell [C9 A0 ... ] 6
10. pkt->cmd == GET pkt->magic != 0xC9 void proc_pkt(packet_t* pkt) { if (pkt->magic != 0xC9) { err(pkt); return; } if (pkt->cmd == GET) { ... } else if ... ... } Symbolic Execution in a Nutshell [C9 A0 ... ] 6
11. void proc_pkt(packet_t* pkt) { if (pkt->magic != 0xC9) { err(pkt); return; } if (pkt->cmd == GET) { ... } else if ... ... } Symbolic Execution in a Nutshell 7 λ
12. λ.magic == 0xC9 λ.magic != 0xC9 void proc_pkt(packet_t* pkt) { if (pkt->magic != 0xC9) { err(pkt); return; } if (pkt->cmd == GET) { ... } else if ... ... } Symbolic Execution in a Nutshell 7 λ
13. λ.cmd == GET λ.cmd != GET λ.magic == 0xC9 λ.magic != 0xC9 void proc_pkt(packet_t* pkt) { if (pkt->magic != 0xC9) { err(pkt); return; } if (pkt->cmd == GET) { ... } else if ... ... } Symbolic Execution in a Nutshell 7 λ
14. λ.cmd == GET λ.cmd != GET λ.magic == 0xC9 λ.magic != 0xC9 void proc_pkt(packet_t* pkt) { if (pkt->magic != 0xC9) { err(pkt); return; } if (pkt->cmd == GET) { ... } else if ... ... } Symbolic Execution in a Nutshell 7 ∼2 paths λ program size
15. 8 CPU Bottleneck Memory Exhaustion
16. W1 W2 W3 Parallel Tree Exploration 8
17. W1 W2 W3 Parallel Tree Exploration 8 Key research problem: Scalable parallel exploration
18. Linear Solution to Exponential Problem 9 Program Size Time to Test
19. Linear Solution to Exponential Problem 9 Program Size Time to Test Testing target 1 worker
20. Linear Solution to Exponential Problem 9 Program Size Time to Test Testing target Bring testing time down to practical values 1 worker 2 workers 4 workers 8 workers
21. Throw Hardware at the Problem 10
22. Scalability Challenges Tree structure not known a priori ? ? ? ? ? ? ? ? ? ? 11
23. Scalability Challenges Static Allocation 12
24. Scalability Challenges 12
25. Scalability Challenges Anticipate Allocation 13
26. Scalability Challenges 13
27. Outline • Scalable Parallel Symbolic Execution • POSIX Environment Model • Evaluation 14
28. Cloud9 Architecture 15 Global Symbolic Tree
29. Cloud9 Architecture 15 W1’s Local Tree W2’s Local Tree W3’s Local Tree Each worker runs a local sequential symbolic execution engine (KLEE)
30. Cloud9 Architecture 16 Candidate nodes Fence nodes • Candidate nodes are selected for exploration • Fence nodes bound the local tree
31. Load Balancing LB W1 W2 W3 17 Hybrid distributed system: centralized reports, P2P work transfer
32. Load Balancing LB W1 W2 W3 17 Hybrid distributed system: centralized reports, P2P work transfer
33. Load Balancing LB W1 W2 W3 17 Hybrid distributed system: centralized reports, P2P work transfer
34. Work Transfer W1 18 Candidate Fence
35. Work Transfer W1 W2 18 Candidate Fence
36. Work Transfer W1 W2 Virtual 18 Candidate Fence
37. Work Transfer W1 W2 Virtual 18 Candidate Fence
38. Work Transfer W1 W2 Materialized 18 Candidate Fence
39. Work Transfer W1 W2 18 Exploration disjointness + completeness Candidate Fence
40. 1 1 1 1 1 0 0 0 0 0 0 0 0 Path-based Encoding 19 • Nodes are encoded as paths in tree • Compact binary representation • Two paths can share common pre x • Small encoding size • For a tree of 2100 leaves, a path ts in <128 bits (16 bytes)
41. Load Balancing in Practice 20 LB stops after 1 min LB stops after 4 min Continuous load balancing Work done [% of total instructions] Time [minutes] 0 10 20 30 40 50 60 70 80 90 100 0 2 4 6 8 10 Load balancing necessary to ensure scalability
42. Outline • Scalable Parallel Symbolic Execution • POSIX Environment Model • Evaluation 21
43. Calls into the Environment 22 if (fork() == 0) { ... if ((res = recv(sock, buff, size, 0)) > 0) { pthread_mutex_lock(&mutex); memcpy(gBuff, buff, res); pthread_mutex_unlock(&mutex); } ... } else { ... pid_t pid = wait(&stat); ... }
44. fork() Program Under Test Environment (C Library / OS) Environment Model 23 Cannot directly execute symbolically
45. fork() Program Under Test Environment (C Library / OS) Environment Model 23 Model Code Symbolic Execution Engine Equivalent functionality Executable symbolically
46. Starting Point 24 Symbolic Execution Engine Network Stubs Files POSIX Single-threaded isolated nodes Single-threaded utilities
47. POSIX Environment Model 25 Symbolic Execution Engine Network TCP/UDP/UNIX Files Pipes Threads pthread_* Processes POSIX M essage passing Servers and clients M ulti-threaded program s Distributed system s Signals Asynchronous events, IPC Single-threaded utilities
48. Key Changes in Symbolic Execution Multithreading and Scheduling • Deterministic or symbolic scheduling • Non-preemptive execution model Address Space Isolation • Copy on Write (CoW) between processes • CoW domains for memory sharing 26
49. Symbolic Engine System Calls • Symbolic engine support needed for threads/processes 1. Thread/process lifecycle 2. Synchronization 3. Shared memory 27 Symbolic Engine System Calls thread_create thread_terminate process_fork process_terminate get_context thread_preempt thread_sleep thread_notify get_wait_list make_shared 1 2 3
50. Outline • Scalable Parallel Symbolic Execution • POSIX Environment Model • Evaluation 28
51. Testing Real-World Software 29 Memcached GNU Coreutils Apache
52. Time to Reach Target Coverage 30 printf Faster time-to-cover, higher coverage values 60% coverage 70% coverage 80% coverage 90% coverage 0 10 20 30 40 50 60 1 4 8 24 48 Time to achieve target coverage [minutes] Number of workers
53. Increase in Code Coverage 0 10 20 30 40 50 0 10 20 30 40 50 60 70 80 90 Additional code covered [ % of program LOC ] Index of tested Coreutil (sorted by additional coverage) 31 Coreutils suite (12 workers, 10 min.) Consistent code coverage increase
54. Exhaustive Exploration 32 0 1 2 3 4 5 6 2 4 6 12 24 48 Time to complete exhaustive test [hours] Number of workers Scalability of exhaustive path exploration memcached (7.4×104 paths)
55. Instruction Throughput 33 0.0e+00 2.0e+09 4.0e+09 6.0e+09 8.0e+09 1.0e+10 1.2e+10 1.4e+10 1.6e+10 1.8e+10 1 4 6 12 24 48 Useful work done [ # of instructions ] Number of workers 4 minutes 6 minutes 8 minutes 10 minutes memcached Linear scalability with number of workers
56. Execute the “whole world” symbolically Symbolic State Experimental Setup 34 Client Process memcached/ Apache/ lighttpd TCP Stream Symbolic cmd. Srv. response
57. Symbolic Test Cases • Easy-to-use API for developers to write symbolic test cases • Basic symbolic memory support • POSIX extensions for environment control • Network conditions, fault injection, symbolic scheduler 35
58. Symbolic Test Cases 36 Testing HTTP header extension make_symbolic(hdrData); // Append symbolic header to request strcat(req, “X-NewExtension: “); strcat(req, hdrData); // Enable fault injection on socket ioctl(ssock, SIO_FAULT_INJ, RD | WR); // Symbolic stream fragmentation ioctl(ssock, SIO_PKT_FRAGMENT, RD);
59. Conclusions • Parallel symbolic execution • Linear scalability on commodity clusters • Full POSIX environment model • Real-world systems testing • Use cases • Increasing coverage • Exhaustive path exploration • Bug patch veri cation 37