ParaFuzz: Fuzzing Multicore OCaml programs “KC” Sivaramakrishna n joint work wit h Sumit Padhiyar and Adharsh Kamath 

• Adds native support for concurrency and parallelism to OCaml Multicore OCaml 

• Adds native support for concurrency and parallelism to OCaml Multicore OCaml Overlapped execution A B A C B Time 

• Adds native support for concurrency and parallelism to OCaml Multicore OCaml Overlapped execution A B A C B Time Simultaneous execution A B C Time 

• Adds native support for concurrency and parallelism to OCaml Multicore OCaml Overlapped execution A B A C B Time Simultaneous execution A B C Time Effect Handlers 

• Adds native support for concurrency and parallelism to OCaml Multicore OCaml Overlapped execution A B A C B Time Simultaneous execution A B C Time Effect Handlers Domains 

• Adds native support for concurrency and parallelism to OCaml Multicore OCaml Overlapped execution A B A C B Time Simultaneous execution A B C Time Effect Handlers Domains using Testing 

Testing Parallel Programs 

Testing Parallel Programs • The assertion can fail for a particular input and scheduling combination 

Testing Parallel Programs • The assertion can fail for a particular input and scheduling combination 10 

Testing Parallel Programs • The assertion can fail for a particular input and scheduling combination happens before 10 

Testing Parallel Programs • The assertion can fail for a particular input and scheduling combination happens before happens before 10 

Testing Parallel Programs • The assertion can fail for a particular input and scheduling combination • Logic bugs are more than just detecting data race s ✦ No data races here happens before happens before 10 

Testing Parallel Programs • The assertion can fail for a particular input and scheduling combination • Logic bugs are more than just detecting data race s ✦ No data races here • Goal — An effective and pragmatic testing technique for the working OCaml programmer. happens before happens before 10 

Existing solutions • Testin g ✦ Stress testing — run the program over and over again and hope that the assertion is triggere d ✦ Random testing — generate random inputs, and perturb the OS scheduler (somehow) to trigger bugs 

Existing solutions • Testin g ✦ Stress testing — run the program over and over again and hope that the assertion is triggere d ✦ Random testing — generate random inputs, and perturb the OS scheduler (somehow) to trigger bugs • Model checking — SPIN, TLC model checker s ✦ Strong guarantees, but not practical with limited time budge t ✦ Often works on a model of the program and not directly on the source code 

Existing solutions • Testin g ✦ Stress testing — run the program over and over again and hope that the assertion is triggere d ✦ Random testing — generate random inputs, and perturb the OS scheduler (somehow) to trigger bugs • Model checking — SPIN, TLC model checker s ✦ Strong guarantees, but not practical with limited time budge t ✦ Often works on a model of the program and not directly on the source code • Formal veri fi catio n ✦ Requires expert knowledge and lots of time and effort 

Goal + Approach • Build upon effective pragmatic testing technique s ✦ Ignore concurrency for the moment — only input non-determinism 

Goal + Approach • Build upon effective pragmatic testing technique s ✦ Ignore concurrency for the moment — only input non-determinism • Property-based testin g ✦ Use a generator to generate random inputs to test a functio n ✦ Quick-check 

Goal + Approach • Build upon effective pragmatic testing technique s ✦ Ignore concurrency for the moment — only input non-determinism • Property-based testin g ✦ Use a generator to generate random inputs to test a functio n ✦ Quick-check • Fuzzin g ✦ Generate random inputs to crash a progra m ✦ AFL — Extremely effective grey-box (coverage-guided) fuzzer 

Goal + Approach • Build upon effective pragmatic testing technique s ✦ Ignore concurrency for the moment — only input non-determinism • Property-based testin g ✦ Use a generator to generate random inputs to test a functio n ✦ Quick-check • Fuzzin g ✦ Generate random inputs to crash a progra m ✦ AFL — Extremely effective grey-box (coverage-guided) fuzzer • Crowbar = Fuzzing + QuickChec k ✦ Coverage-guided property-fuzzin g ✦ https://github.com/stedolan/crowbar 

ParaFuzz Parafuzz = Crowbar (grey-box fuzzing + property-based testing) + Parallelism 

ParaFuzz • How to control parallel thread scheduling ? ✦ OS controls thread scheduling in parallel program s ✦ Need to force a buggy schedule Parafuzz = Crowbar (grey-box fuzzing + property-based testing) + Parallelism 

Effect handlers • Mock the parallelism API using effect handler s • Effect Handler s ✦ Modular and composable basis of non-local control- fl ow mechanism s ✤ Exceptions, async/await, lightweight threads, co-routines, etc . ✦ Structured programming with delimited continuations 

Effect Handlers + Fuzzing • Simulate parallel thread scheduler using effect handler s ✦ OS thread scheduler → user-level thread schedule r ✦ Retain control over the scheduling decisions 

Effect Handlers + Fuzzing • Simulate parallel thread scheduler using effect handler s ✦ OS thread scheduler → user-level thread schedule r ✦ Retain control over the scheduling decisions • Fuzzing the schedule r ✦ Yield at every synchronisation point ✤ Synchronisation point — context-switch leads to non-determinis m ✦ Use AFL to pick next thread to run from ready queue 

Effect Handlers + Fuzzing • Simulate parallel thread scheduler using effect handler s ✦ OS thread scheduler → user-level thread schedule r ✦ Retain control over the scheduling decisions • Fuzzing the schedule r ✦ Yield at every synchronisation point ✤ Synchronisation point — context-switch leads to non-determinis m ✦ Use AFL to pick next thread to run from ready queue • Synchronisation point s ✦ Domain (spawn, join), Atomic (get, set, CAS), Mutex (lock, unlock), Condition variable (wait, notify, broadcast) 

ParaFuzz • System layout 

ParaFuzz • System layout • Advantage s ✦ Drop-in replacement for Multicore OCaml — separate compiler switc h ✦ No false positive s ✦ Deterministic record-and-replay 

Evaluation 

Evaluation 

Evaluation Effectivenes s fraction of runs that found the bug Ef fi cienc y Mean-time to failure 

Evaluation Effectivenes s fraction of runs that found the bug Ef fi cienc y Mean-time to failure 

Evaluation Effectivenes s fraction of runs that found the bug Ef fi cienc y Mean-time to failure 

Future work: Data races 

Future work: Data races • ParaFuzz currently assumes that the programs are data-race-free (DRF) ✦ DRF programs in OCaml have SC semantics 

Future work: Data races • ParaFuzz currently assumes that the programs are data-race-free (DRF) ✦ DRF programs in OCaml have SC semantics • OCaml memory model (PLDI’18) also has a simple operational model for racy program s ✦ Racy reads may return one of a subset of writes performed to a non- atomic location 

Future work: Data races • ParaFuzz currently assumes that the programs are data-race-free (DRF) ✦ DRF programs in OCaml have SC semantics • OCaml memory model (PLDI’18) also has a simple operational model for racy program s ✦ Racy reads may return one of a subset of writes performed to a non- atomic location • Extend ParaFuzz to racy program s ✦ Use AFL to pick the value that a read should retur n ✦ Force a yield at non-atomic reads and writes 

Future work: Data races • ParaFuzz currently assumes that the programs are data-race-free (DRF) ✦ DRF programs in OCaml have SC semantics • OCaml memory model (PLDI’18) also has a simple operational model for racy program s ✦ Racy reads may return one of a subset of writes performed to a non- atomic location • Extend ParaFuzz to racy program s ✦ Use AFL to pick the value that a read should retur n ✦ Force a yield at non-atomic reads and writes • Can we make it fast enough for pragmatic use? 

Summary • ParaFuzz ✦ property-based fuzzing tool for Parallel OCaml program s ✦ Easy to use — drop-in replacement for Multicore OCaml program s ✦ Effective and ef fi cient at fi nding concurrency bug s • Future work — Detecting bugs under data races https://github.com/ocaml-multicore/parafuzz