Bug-Free Concurrency in Rust
Tokyo Rust Meetup
Motoyuki Kimura
1
Slide 2
Slide 2 text
● Work at Cyberagent, Inc.
● Rust Developer 🦀
● Used to use Scala
● A member of tokio-rs
Hi, I’m Motoyuki(@mox692) 👋
2
Slide 3
Slide 3 text
Question: How many of you know loom?
3
Slide 4
Slide 4 text
4
Slide 5
Slide 5 text
5
Slide 6
Slide 6 text
Goal
● Introduction to the loom library
● Understand the inner workings
○ High level idea of the model checking
○ What the Dynamic Partial Order Reduction is
○ How loom handle memory order
6
Slide 7
Slide 7 text
Multi-threaded Programs are hard
7
Slide 8
Slide 8 text
8
Slide 9
Slide 9 text
9
Slide 10
Slide 10 text
Why so hard?
● Non-determinism of thread scheduling
● There are so many cases we have to care about
10
Slide 11
Slide 11 text
Who can help us?
11
Slide 12
Slide 12 text
12
Slide 13
Slide 13 text
13
Slide 14
Slide 14 text
14
Slide 15
Slide 15 text
What about Rust?
● Rust’s type system can prevent certain kinds of
multi-threaded problems
● But NOT all problems
15
Slide 16
Slide 16 text
Good way to find subtle bugs?
16
Slide 17
Slide 17 text
17
Slide 18
Slide 18 text
Exhaustive way
18
Slide 19
Slide 19 text
Much effort / research is going on around this area
● Theorem Proving
○ Coq, Agda, Lean
● Model Checking
○ TLA+
○ Loom 🦀
19
Slide 20
Slide 20 text
20
Slide 21
Slide 21 text
● Loom explores all possible permutations of thread
scheduling
● It uses DPOR, one of the model checking techniques
● Used in many places in the tokio codebase 🗼
21
Slide 22
Slide 22 text
Let’s see examples! (Demo time)
22
Slide 23
Slide 23 text
23
#[test]
fn test_buggy_concurrent_logic() {
let v1 = Arc::new(AtomicUsize::new(0));
let v2 = v1.clone();
thread::spawn(move || v1.store(1, Relaxed));
let v = v2.load(Relaxed);
assert_eq!(1, v);
}
Slide 24
Slide 24 text
24
#[test]
fn test_concurrent_logic_loom() {
loom::model(|| {
let v1 = Arc::new(AtomicUsize::new(0));
let v2 = v1.clone();
thread::spawn(move || v1.store(1, Relaxed));
let v = v2.load(Relaxed);
assert_eq!(0, v);
});
}
Slide 25
Slide 25 text
Questions so far?
25
Slide 26
Slide 26 text
We’re all happy!
26
Slide 27
Slide 27 text
… Really??
27
Slide 28
Slide 28 text
We’re Engineers, so … Let’s dig in!
28
Slide 29
Slide 29 text
29
Slide 30
Slide 30 text
30
Slide 31
Slide 31 text
Don't worry, I'm not expert either 🙂
31
Slide 32
Slide 32 text
How to model program states?
● Once away from Rust
● How to explore all program states? 🤔
32
Slide 33
Slide 33 text
State and Transition
● Program has State S
i
● t
i
is a Transition that changes the state from S
i
to S
i+1
a = 0 a = 1 a = 2
S
0
S
1
S
2
t
0
t
1
・・・
33
Slide 34
Slide 34 text
34
thread::spawn(move || v1.store(1, Relaxed));
let v = v2.load(Relaxed);
assert_eq!(1, v);
Simple example again
Slide 35
Slide 35 text
v1.store
S
0
35
v2.load
Slide 36
Slide 36 text
v1.store
S
0
36
v2.load assert_eq() v1.store
v2.load
x = 1 P = 1
x = 1
y = 2
x = 1
P = 1
P = 1
x = 1
P = 1
a = 1
x = 1
y = 2
z = 3
x = 1
y = 2
P = 1
x = 1
P = 1
y = 2
x = 1
P = 1
a = 1
P = 1
x = 1
y = 2
P = 1
x = 1
a = 1
P = 1
a = 1
x = 1
P = 1
a = 1
b = 2
41
Slide 42
Slide 42 text
x = 1 P = 1
x = 1
y = 2
x = 1
P = 1
P = 1
x = 1
P = 1
a = 1
x = 1
y = 2
z = 3
x = 1
y = 2
P = 1
x = 1
P = 1
y = 2
x = 1
P = 1
a = 1
P = 1
x = 1
y = 2
P = 1
x = 1
a = 1
P = 1
a = 1
x = 1
P = 1
a = 1
b = 2
42
Slide 43
Slide 43 text
Partial Order Reduction
● Usually, program has several semantically identical
transitions.
● Try to reduce those identical transitions by introducing the
dependency concept.
43
Slide 44
Slide 44 text
Dependency
● Independent Operation
○ If reordering the two operations does not change the
final state of the program, they are independent
○ e.g. read from the thread local variables
● Dependent Operation
○ e.g. acquire a lock from the Mutex
44
println!(“thread 1”)
S
0
49
assert_eq() println!(“thread 1”) v1.store
println!(“thread 2”)
println!(“thread 2”)
・・・
Interleaving between “independent operations”
Slide 50
Slide 50 text
println!(“thread 1”)
S
0
50
assert_eq() println!(“thread 1”) v1.store
println!(“thread 2”)
println!(“thread 2”)
・・・
Based on the POR notion, we can reduce a bunch of execution paths
Slide 51
Slide 51 text
Dynamic Partial Order Reduction?
51
Slide 52
Slide 52 text
Dynamic Partial Order Reduction
● Historically, POR was a static analysis-based method
● DPOR checks the state space at runtime
● Using backtracking + DFS, search for all permutations of
threads
52
Slide 53
Slide 53 text
S
0
S
1
thread1
1st Execution
dependent access
53
Slide 54
Slide 54 text
S
0
S
1
S
2
S
6
thread1
thread1 thread2
1st Execution
dependent access
54
Slide 55
Slide 55 text
S
0
S
1
S
2
S
6
thread1
thread1 thread2
1st Execution
Backtracking Point List (Stack)
[S
1
]
save as a backtrack point
55
Slide 56
Slide 56 text
S
0
S
1
S
2
S
6
thread1
thread1 thread2
S
3
1st Execution
Backtracking Point List (Stack)
dependent access again!!
[S
1
]
56
Slide 57
Slide 57 text
S
0
S
1
S
2
S
6
thread1
thread1 thread2
S
3
S
4
S
5
thread1 thread3
1st Execution
[S
3
, S
1
]
Backtracking Point List (Stack)
save as another backtrack point
push
57
Slide 58
Slide 58 text
S
0
S
1
S
2
S
6
thread1
thread1 thread2
S
3
S
4
S
5
thread1 thread3
1st Execution
[S
3
, S
1
]
Backtracking Point List (Stack)
End
push
58
Slide 59
Slide 59 text
S
0
S
1
S
2
S
6
thread1
thread1 thread2
S
3
2nd Execution
Backtracking Point List (Stack)
[S
1
]
S
3
pop
59
Slide 60
Slide 60 text
S
0
S
1
S
2
S
6
thread1
thread1 thread2
S
3
S
4
S
5
thread1 thread3
2nd Execution
[S
1
]
Backtracking Point List (Stack)
S
3
pop
60
Slide 61
Slide 61 text
S
0
S
1
thread1
3rd Execution
[S
1
]
Backtracking Point List (Stack)
61
Slide 62
Slide 62 text
S
0
S
1
S
2
S
6
thread1
thread1 thread2
S
3
S
4
S
5
thread1 thread3
3rd Execution
[]
Backtracking Point List (Stack)
S
1
pop
62
Slide 63
Slide 63 text
Dynamic Partial Order Reduction
● If we find a state that needs to be explored, add it as a
backtracking point at runtime
● Repeat the program until all backtrack points are
explored
63
Slide 64
Slide 64 text
Let’s Feel DPOR 😆 (Demo time)
64
Slide 65
Slide 65 text
Okay, DPOR sounds cool.
… But, what about memory model?
65
Slide 66
Slide 66 text
66
Slide 67
Slide 67 text
Memory Ordering
● The rules how memory operations are executed and
observed in concurrent systems
● DPOR only takes into account a sequential execution
model
67
Slide 68
Slide 68 text
68
thread::spawn(move || v1.store(1, Relaxed));
let v = v2.load(Relaxed);
assert_eq!(1, v);
Slide 69
Slide 69 text
69
thread::spawn(move || v1.store(1, Relaxed));
let v = v2.load(Relaxed);
assert_eq!(1, v);
➡
Slide 70
Slide 70 text
70
thread::spawn(move || v1.store(1, Relaxed));
let v = v2.load(Relaxed);
assert_eq!(1, v);
// -> 1 or 0
➡
Slide 71
Slide 71 text
71
thread::spawn(move || v1.store(1, Relaxed));
let v = v2.load(Relaxed);
assert_eq!(1, v); Even after the store,
it's possible to see 0
↓
Algorithm should explore all
possible load’s result
➡ // -> 1 or 0
Slide 72
Slide 72 text
S
0
S
1
thread1
Backtracking Point List (Stack)
[]
72
Read involving memory ordering
Slide 73
Slide 73 text
S
0
S
1
S
2
S
3
thread1
load result1
Backtracking Point List (Stack)
[S
1
]
save as a backtrack point
73
load result2
Read involving memory orders
Slide 74
Slide 74 text
● To handle the memory model, we particularly pay attention
to read operation
● In addition to DPOR's thread execution backtrack points,
load result selection based on memory models is also
added to the backtracking points
74
Slide 75
Slide 75 text
75
Slide 76
Slide 76 text
Want to know more?
76
Slide 77
Slide 77 text
● Recap traditional methods (POR, etc.) and introduce DPOR
77
Slide 78
Slide 78 text
● Based on previous DPOR paper, discuss how to handle
C/C++ memory model
● Implementation available (GitHub)
78
Slide 79
Slide 79 text
Are you getting interested in loom?
79
Slide 80
Slide 80 text
Need help with
● Implementation clean up
● Unimplemented types
● PR reviews
● Multi-threading support
80