EXTENT-2016: Industry Practices of Advanced Program Analysis

EXTENT-2016: Software Testing & Trading Technology Trends
22 June, 2016, 10 Paternoster Square, London

Industry Practices of Advanced Program Analysis
Alexey Khoroshilov, Snr. Researcher, ISP RAS

Would like to know more?
Visit our website: extentconf.com
Follow us:
https://www.linkedin.com/company/exactpro-systems-llc?trk=biz-companies-cym
https://twitter.com/exactpro
#extent2016
#exactpro

Transcript

1. Institute for System Programming of the Russian Academy of Sciences Industry practices of advanced program analysis Alexey Khoroshilov khoroshilov@ispras.ru ExTENT-2016 London, 22 June 2016
2. The Goal of Verification to make sure there is no any bugs in any possible execution of the target software • with minimal cost Verification Fresh Software Verified Software
3. The Goal of Verification to achieve maximal assurance that there is no any bugs in any possible execution of the target software • within reasonable cost Verification Fresh Software Verified Software
4. Testing Test Test Suite 1 kind of bugs all kinds of bugs in all executions in 1 execution
5. Inductive Reasoning • Conclusions are supported by its premises • If premises are true, it would be unlikely impossible for the conclusions to be false • Example: • Every time I’ve walked by that dog, he hasn’t tried to bite me. • So, the next time I walk by that dog he won’t try to bite me.
6. Inductive Reasoning • Conclusions are supported by its premises • If premises are true, it would be unlikely impossible for the conclusions to be false • Example: • The component behaves correctly on all test data. • So, the component always behaves correctly.
7. Inductive Reasoning on Program Correctness • The component behaves correctly on all test data • Tests are quite representative => • The component always behaves correctly
8. Deductive Reasoning • Conclusions certainly follow from its premises • If premises are true, it is impossible for the conclusions to be false • Example: • All men are mortal. • Socrates is a man. • Therefore, Socrates is mortal.
9. Deductive Reasoning on Program Correctness Target Component Requirements Review
10. Deductive Reasoning on Program Correctness Target Component Requirements
11. Deductive Reasoning on Program Correctness • If assumptions are held • regarding compiler&linker • regarding environment behaviour • regarding input data • The component always behaves correctly
12. Deductive Reasoning on Program Correctness • If assumptions are held • regarding compiler&linker • regarding environment behaviour • regarding input data • The component always behaves correctly (i.e. terminates and its output satisfies to postcondition)
13. Deductive Verification Historical Perspective 1947 1970 • Lecture of Alan Turing to London Mathematical Society • Methods of Floyd/Hoare
14. START: ( y 1 , y 2 )  ( 0, x 1 ) y 2  x 2 F T ( y 1 , y 2 )  ( y 1 +1, y 2 - x 2 ) HALT: ( z 1 , z 2 )  ( y 1 , y 2 ) A: Precondition (x 1  0)  (x 2  0) С: Postcondition (x 1 = z 1 x 2 + z 2 )  (0  z 2 < x 2 ) B: Invariant (x 1 = y 1 x 2 + y 2 ) ∧ (y 2  0) Deductive Verification Prove program correctness by induction: Let precondition to be held in A we pass A->B => Invariant is held in B Let invariant to be held in B we pass B-T->B => Invariant is held in B Let invariant to be held in B we pass B-F->C => Postcondition is held in С Let precondition to be held in A we pass A->C => Postcondition is held in С =>
15. Deductive Verification Historical Perspective 1947 1970 2000 • Lecture of Alan Turing to London Mathematical Society • Methods of Floyd/Hoare • Deductive verifcation tools for Ada, C, Java, С# • SunRise, ESC/Java, Frama-C, LOOP, Boogie/VCC
18. Deductive Verification Historical Perspective 1947 1970 2000 2010 • Lecture of Alan Turing to London Mathematical Society • Methods of Floyd/Hoare • Deductive verifcation tools for Ada, C, Java, С# • SunRise, ESC/Java, Frama-C, LOOP, Boogie/VCC • Application in real-life projects for small-size components • Nuclear power (UK, France) • Avionics (Airbus, NASA, UK Air Traffic Control) • Components of operating systems (seL4, Verisoft, Verisoft-XT)
19. Industry Applications • UK Air Traffic Management System • 250 KLOC of logical lines of code (in Ada) • proof type safety, few functional correctness code • 153K VCs, of which 98.76% are proven automatically (*) Angela Wallenburg “Safe and Secure Programming Using Spark”
20. Years Tools Target code Scope Size Verisoft 2004-2008 Isabelle designed for verification hw/kernel/ compiler/ libraries/apps 10 kLOC (kernel) L4.verified seL4 2004-2009 Isabelle designed for verification, performance oriented microkernel security model (no MMU) 7.5 kLOC (without asm and boot) Verisoft-XT small-hv 2007-2013 VCC designed for verification separation property only 2.5 kLOC Verisoft-XT Hyper-V 2007-2013 VCC industrial separation property only 100 kLOC Verisoft-XT PikeOS 2007-2013 VCC industrial, simplicity for performance some system calls 10 KLOC OS Deductive Verification
21. Linux Verification Center founded in 2005 • OLVER Program • Linux Standard Base Infrastructure Program • Linux Driver Verification Program • Linux File System Verification Program • Linux Deductive Verification Program
22. Toolset for Event-B models verification Model of security requirements Formalized security model Formalized low-level security model Manual Automated verification Security requirements Security model AstraVer Toolchain (*) for deductive verification of C programs (based on Frama-C – Jessie – Why3) Pre-/post-conditions of LSM operations Security arcitecture LSM source code Model of security requirements Mathematical notation LSM Implementation implements Specificatiion of library functions Linux kernel AstraVer Project AstraLinux (*) The research on deductive verification tools development was carried out with funding from the Ministry of Education and Science of Russia (the project unique identifier is RFMEFI60414X0051
23. Deductive Verification Status • Reasonable Tool Support • Ada, C, C#, Java • Functional specification as comments, even natively supported in Ada-2012 • Dedicated languages: Boogie, Why3 • Manual efforts still significant • up to 10x of development efforts • highly skilled team required
24. Testing and Deductive Verification 1 kind bugs all kinds of bugs in all executions Deductive 1. Proof of complete correctness under some assumptions 2. Very labour intensive and time consuming in 1 execution Test Test Suite
25. Testing Deductive Verification Kind of bugs almost all almost all Executions under analysis small almost all Development cost linear huge Execution cost hw small (target hw) hw small Result analysis cost small big Maintenance cost small to big huge Testing and Deductive Verification
26. Deductive Verification Status (2) • reasonable to apply only for really important code
27. Static Analysis • is a commodity • mature market of static analysis tools
28. SVACE by ISPRAS • Static analysis of C/C++/Java code, Linux/Windows • 150+ kinds of defects • Buffer overflows, NULL-pointer dereferences • Memory management, tainted input • Concurrency issues • Lightweight analysis of semantic patterns • Eclipse plugin or WebUI
29. Testing Static Analysis Deductive Verification Kind of bugs almost all safety only almost all Executions under analysis small almost all almost all Development cost linear 0 to small huge Execution cost hw small (target hw) hw small hw small Result analysis cost small medium (false alarms) big Maintenance cost small to big small huge Testing and Program Analysis
30. Testing Static Analysis Deductive Verification Kind of bugs almost all safety only almost all Executions under analysis small big almost all almost all Development cost linear 0 to small huge Execution cost hw small (target hw) hw small hw small Result analysis cost small medium (false alarms) big Maintenance cost small to big small huge Testing and Program Analysis
31. Testing and Program Analysis 1 kind bugs all kinds of bugs in all executions Deductive in 1 execution Test Test Suite SVACE 1. Static analysis 2. Quickly finds potential bugs 3. No any guarantee
32. R ... Model Checking R 2 R 1 R 0 [Clarke/Emerson, Sifakis 1981] • Iterative fixpoint post computation
33. Software Model Checking entry point error location • Reachability problem
34. Error Location? int f(int y) { struct urb *x; x = usb_alloc_urb(0,GFP_KERNEL); ... usb_free_urb(x); return y; }
35. Error Location? int f(int y) { struct urb *x; x = usb_alloc_urb(0,GFP_KERNEL); // allocate new URB ... usb_free_urb(x); // deallocate URB: assert(x is NULL or previously allocated URB) return y; } … // after module exit: assert( all allocated URBs are deallocated)
36. Instrumentation int f(int y) { struct urb *x; x = usb_alloc_urb(0,GFP_KERNEL); ... usb_free_urb(x); return y; } set URBS = empty; int f(int y) { struct urb *x; x = usb_alloc_urb(); add(URBS, urb); ... assert(contains(URBS, x)); usb_free_urb(x); remove(URBS, urb); return y; } … // after module exit assert(is_empty(URBS));
37. Software Model Checking entry point error location • Reachability problem
38. Bounded Model Checking • finite unfolding of transition relation b 1 a 2 a 2 b 2 b 2 b 2 b 2 b 1 a 2 a 2 b 2 b 2 b 2 b 2 r
39. Counter-Example Guided Abstraction Refinement • Detailed model of a program is huge • Detailed model of a program is not needed to check a particular property • Detailed model of a concrete path in a program is suitable for analysis => • Build a model that is just enough to check the particular property
40. SV-COMP-2013 Competition Results Drivers Drivers Bit Bit operations operations Pointers Pointers CEGAR CEGAR BMC BMC Shape- Shape- analysis analysis
41. SVCOMP'14 Results
42. SVCOMP'15 Results BLAST CPAchecker 15 tools in 2014 22 tools in 2015 35 tools in 2016
43. Industry Applications
44. Linux Verification Center founded in 2005 • OLVER Program • Linux Standard Base Infrastructure Program • Linux Driver Verification Program • Linux File System Verification Program • Linux Deductive Verification Program
45. Linux Driver Verification http://linuxtesting.org/ldv
46. Bugs Found http://linuxtesting.org/results/ldv >230 patches already applied
47. SVACE vs. LDV SVACE LDV-CPAchecker Time of analysis 2 hrs 111 hrs (4.5 days) Warnings 35 328 True bugs 8 103 True positive rate 23% 31% • Target code: • Linux kernel 3.17-rc1, allmodconfig, x86-64 • 3 223 modules, 33 373 source files • Target bugs: • double free, memory leaks There is no a single common bugs!
48. Testing and Program Analysis 1 kind bugs all kinds of bugs in all executions Deductive in 1 execution Test Test Suite SVACE LDV 1. Investigates all possible paths 2. Is able to prove absence of bugs of particular kind 3. Domain-specific (framework) adaptation (environment model, library model) 4. Applicability limited to medium-sized components (up to 50 KLoC) 5. Requires nonzero hardware resources - time: 15 minutes per rule per module - memory: 15 Gb
49. Testing Static Analysis Software Model Checking Deductive Verification Kind of bugs almost all safety only safety only almost all Executions under analysis small big almost all almost all Development cost linear 0 to small medium huge Execution cost hw small (target hw) hw small hw big hw small Result analysis cost small medium (false alarms) medium (false alarms) big Maintenance cost small to big small small huge Testing and Program Analysis
50. Conclusions • There is no silver bullet • The key is in a competent combination of available techniques
51. Conclusions (2) • Two advanced program analysis techniques: • Deductive verification • + proof of complete correctness under some assumptions • – significant manual efforts • – highly skilled team required • => only for really important code • Software model checking • + investigates almost all possible paths • + complimentary to static analysis • – per framework/domian adaptation required • – limited to medium-sized components (up to 50 KLoC)
52. Institute for System Programming of the Russian Academy of Sciences Thank you! Alexey Khoroshilov khoroshilov@ispras.ru http://linuxtesting.org/ Morris Kline. “Mathematics: The Loss of Certainty” Oxford Press, 1980