Dl-Check: dynamic potential deadlock detection tool for Java programs Nikita Koval1,2 Dmitry Tsitelov2 Roman Elizarov2 1ITMO University, Computer Technology Department, St. Petersburg, Russia [email protected] 2Devexperts, LLC, St. Petersburg, Russia, [email protected] TMPA Conference, March 2017

Table of contents 1. Introduction 2. Algorithm 3. Implementation 4. Evaluation 5. Conclusion Slide 2/65. Copyright © 2017. Devexperts LLC. All rights reserved.

INTRODUCTION

Multithreaded programming problems • Data race • Starvation • Livelock • Deadlock - Communication deadlock - Resource deadlock Slide 4/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Resource deadlock void transfer(Account from, Account to, int amount) { from.lock() to.lock() // Do transfer unlock(from, to) } transfer(a, b, …): 1 a.lock() 4 b.lock() transfer(b, a, …): 2 b.lock() 3 a.lock() DEADLOCK! Slide 5/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Analysis approaches • Static analysis - Can guarantee deadlock-freedom - Many false positives • Model checking - Can guarantee deadlock-freedom - Requires a lot of computational resources • Dynamic analysis - Depends on the execution - Produces few false positives - Applicable for large programs Slide 6/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Dynamic analysis • Collecting execution traces - Analyzes collected traces off-line - e.g. JCarder, MulticoreSDK • Detecting immediately when a potential deadlock happens - Has all context information, such as stack traces - e.g. VisualThreads Slide 7/65. Copyright © 2017. Devexperts LLC. All rights reserved.

ALGORITHM

“Acquired before” relation • Lock is acquired before lock ( → ) if at some point lock is acquired under lock in the same thread Thread 1 Thread 2 1 a.lock() 2 b.lock() 3 c.lock() 4 d.lock() → → → → → → Slide 9/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Lock hierarchy • Lock hierarchy is a partial order on locks, where → ⇒ < < < < < < < () Thread 1 Thread 2 1 a.lock() 2 b.lock() 3 c.lock() 4 d.lock() Slide 10/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Potential deadlock • Lock hierarchy is a primary method to avoid deadlocks • Potential deadlock is a lock hierarchy violation Thread 1 Thread 2 1 a.lock() 2 b.lock() 3 b.lock() 4 a.lock() → → < < () Slide 11/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Lock-order graph (1) • Every vertex is associated with a lock • Edge , means that → • Cycles refer to potential deadlocks Thread 1 Thread 2 1 a.lock() 2 b.lock() 3 unlock(a, b) 4 b.lock() 5 a.lock() 6 unlock(a, b) Slide 12/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Lock-order graph (2) Thread 1 Thread 2 1 a.lock() 2 b.lock() 3 unlock(a, b) 4 b.lock() 5 a.lock() 6 unlock(a, b) • Every vertex is associated with a lock • Edge , means that → • Cycles refer to potential deadlocks Slide 13/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Lock-order graph (3) Thread 1 Thread 2 1 a.lock() 2 b.lock() 3 unlock(a, b) 4 b.lock() 5 a.lock() 6 unlock(a, b) • Every vertex is associated with a lock • Edge , means that → • Cycles refer to potential deadlocks Slide 14/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Lock-order graph (4) Thread 1 Thread 2 1 a.lock() 2 b.lock() 3 unlock(a, b) 4 b.lock() 5 a.lock() 6 unlock(a, b) • Every vertex is associated with a lock • Edge , means that → • Cycles refer to potential deadlocks Slide 15/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Minimization principle: problem • Two cycles may refer to the same potential deadlock Thread 1 Thread 2 1 a.lock() 2 c.lock() 3 b.lock() 4 unlock(a,b,c) 5 b.lock() 6 a.lock() 7 unlock(a,b) Slide 16/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Minimization principle: problem (1) • Two cycles may refer to the same potential deadlock Thread 1 Thread 2 1 a.lock() 2 c.lock() 3 b.lock() 4 unlock(a,b,c) 5 b.lock() 6 a.lock() 7 unlock(a,b) Slide 17/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Minimization principle: problem (2) • Two cycles may refer to the same potential deadlock Thread 1 Thread 2 1 a.lock() 2 c.lock() 3 b.lock() 4 unlock(a,b,c) 5 b.lock() 6 a.lock() 7 unlock(a,b) Slide 18/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Minimization principle: problem (3) • Two cycles may refer to the same potential deadlock Thread 1 Thread 2 1 a.lock() 2 c.lock() 3 b.lock() 4 unlock(a,b,c) 5 b.lock() 6 a.lock() 7 unlock(a,b) Slide 19/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Minimization principle: problem (4) • Two cycles may refer to the same potential deadlock Thread 1 Thread 2 1 a.lock() 2 c.lock() 3 b.lock() 4 unlock(a,b,c) 5 b.lock() 6 a.lock() 7 unlock(a,b) Slide 20/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Minimization principle: problem (5) • Two cycles may refer to the same potential deadlock Thread 1 Thread 2 1 a.lock() 2 c.lock() 3 b.lock() 4 unlock(a,b,c) 5 b.lock() 6 a.lock() 7 unlock(a,b) Slide 21/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Minimization principle: problem (6) • Two cycles may refer to the same potential deadlock Thread 1 Thread 2 1 a.lock() 2 c.lock() 3 b.lock() 4 unlock(a,b,c) 5 b.lock() 6 a.lock() 7 unlock(a,b) Slide 22/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Minimization principle: rule • The shorter cycle is more useful • Only one shortest cycle is to be produced Slide 23/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Algorithm layout • Capture lock acquire and release operations • After lock acquisition: - Add lock to the multiset of locks held by the current thread - Add new edges to the lock-order graph - Report new cycles • Before lock release: - Remove lock from the multiset of held locks Slide 24/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Lock acquisition and release capturing Thread 1 Thread 2 4 a.lock() 5 c.lock() 7 b.lock() 8 a.unlock() 9 c.unlock() 10 b.unlock() 1 b.lock() 2 a.lock() 3 a.unlock() 6 b.unlock() Thread #1 Thread #2 Slide 25/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Lock acquisition and release capturing (1) Thread #1 Thread #2 b Thread 1 Thread 2 4 a.lock() 5 c.lock() 7 b.lock() 8 a.unlock() 9 c.unlock() 10 b.unlock() 1 b.lock() 2 a.lock() 3 a.unlock() 6 b.unlock() Slide 26/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Lock acquisition and release capturing (2) Thread #1 Thread #2 b a Thread 1 Thread 2 4 a.lock() 5 c.lock() 7 b.lock() 8 a.unlock() 9 c.unlock() 10 b.unlock() 1 b.lock() 2 a.lock() 3 a.unlock() 6 b.unlock() Slide 27/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Lock acquisition and release capturing (3) Thread #1 Thread #2 b a Thread 1 Thread 2 4 a.lock() 5 c.lock() 7 b.lock() 8 a.unlock() 9 c.unlock() 10 b.unlock() 1 b.lock() 2 a.lock() 3 a.unlock() 6 b.unlock() Slide 28/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Lock acquisition and release capturing (4) Thread #1 Thread #2 b a Thread 1 Thread 2 4 a.lock() 5 c.lock() 7 b.lock() 8 a.unlock() 9 c.unlock() 10 b.unlock() 1 b.lock() 2 a.lock() 3 a.unlock() 6 b.unlock() Slide 29/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Lock acquisition and release capturing (5) Thread #1 Thread #2 b a c Thread 1 Thread 2 4 a.lock() 5 c.lock() 7 b.lock() 8 a.unlock() 9 c.unlock() 10 b.unlock() 1 b.lock() 2 a.lock() 3 a.unlock() 6 b.unlock() Slide 30/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Lock acquisition and release capturing (6) Thread #1 Thread #2 a c b Thread 1 Thread 2 4 a.lock() 5 c.lock() 7 b.lock() 8 a.unlock() 9 c.unlock() 10 b.unlock() 1 b.lock() 2 a.lock() 3 a.unlock() 6 b.unlock() Slide 31/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Lock acquisition and release capturing (7) Thread #1 Thread #2 a c b Thread 1 Thread 2 4 a.lock() 5 c.lock() 7 b.lock() 8 a.unlock() 9 c.unlock() 10 b.unlock() 1 b.lock() 2 a.lock() 3 a.unlock() 6 b.unlock() Slide 32/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Lock acquisition and release capturing (8) Thread #1 Thread #2 a c b Thread 1 Thread 2 4 a.lock() 5 c.lock() 7 b.lock() 8 a.unlock() 9 c.unlock() 10 b.unlock() 1 b.lock() 2 a.lock() 3 a.unlock() 6 b.unlock() Slide 33/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Lock acquisition and release capturing (9) Thread #2 Thread #1 c b Thread 1 Thread 2 4 a.lock() 5 c.lock() 7 b.lock() 8 a.unlock() 9 c.unlock() 10 b.unlock() 1 b.lock() 2 a.lock() 3 a.unlock() 6 b.unlock() Slide 34/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Lock acquisition and release capturing (10) Thread #1 Thread #2 b Thread 1 Thread 2 4 a.lock() 5 c.lock() 7 b.lock() 8 a.unlock() 9 c.unlock() 10 b.unlock() 1 b.lock() 2 a.lock() 3 a.unlock() 6 b.unlock() Slide 35/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Multiset of held locks • Common lock usage patterns synchronized(o) { // Do something } synchronized void f() { // Do something } l.lock() try { // Do something } finally { l.unlock() } • Locks are acquired and released in LIFO (stack) order Slide 36/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Multiset of held locks • Stack with possibility of removing from the middle • Commonly works in 1 • Works in () at worst - – number of locks acquired by the current thread Slide 37/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Topological order • Incremental topological order maintenance - MNR algorithm suggested by Marchetti-Spaccamela et al. • The acyclic part of the lock-order graph is to be kept • Adding (, ) leads to topological order violation ⇒ the shortest path ↝ relates to the cycle Slide 38/65. Copyright © 2017. Devexperts LLC. All rights reserved.

New nodes processing • Typical pattern in the lock-order graph x a b c d e f g h • One long-lived lock at the center and many hundreds of short-lived locks around it Slide 39/65. Copyright © 2017. Devexperts LLC. All rights reserved.

New nodes processing • Lock-free buffer for new nodes • Initialize topological order value - if buffer is full - when outgoing edge should be added New lock nodes buffer Lock-order graph Slide 40/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Complexity • On lock acquire - If acquisition does not produce new cycles • Typical case • ( + || + ||) - New cycle detected • ⋅ + • On lock release - Removes the lock from the lock multiset • 1 typically • at worst Slide 41/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Limitations: cycle minimization void transfer(Account from, Account to, int amount) { from.lock() to.lock() // Do transfer unlock(from, to) } 1 transfer(a, c, …) 2 transfer(c, b, …) 3 transfer(b, a, …) 4 transfer(a, b, …) Slide 42/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Limitations: cycle minimization (1) void transfer(Account from, Account to, int amount) { from.lock() to.lock() // Do transfer unlock(from, to) } 1 transfer(a, c, …) 2 transfer(c, b, …) 3 transfer(b, a, …) 4 transfer(a, b, …) Slide 43/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Limitations: cycle minimization (2) void transfer(Account from, Account to, int amount) { from.lock() to.lock() // Do transfer unlock(from, to) } 1 transfer(a, c, …) 2 transfer(c, b, …) 3 transfer(b, a, …) 4 transfer(a, b, …) Slide 44/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Limitations: cycle minimization (3) void transfer(Account from, Account to, int amount) { from.lock() to.lock() // Do transfer unlock(from, to) } 1 transfer(a, c, …) 2 transfer(c, b, …) 3 transfer(b, a, …) 4 transfer(a, b, …) Slide 45/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Limitations: cycle minimization (4) void transfer(Account from, Account to, int amount) { from.lock() to.lock() // Do transfer unlock(from, to) } 1 transfer(a, c, …) 2 transfer(c, b, …) 3 transfer(b, a, …) 4 transfer(a, b, …) Slide 46/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Limitation: cycle minimization (3) • Edge (, ) has formed a cycle • Minimize cycle , … , ⇔ minimize path ↝ - Can be achieved via BFS • Usually cycle length is 2 or 3 • Current implementation skips this part Slide 47/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Limitations: independent cycles • Logically independent cycles: - , , - , • Independent cycle is ignored due to minimization principle • One useful cycle is better J Slide 48/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Limitations: single threaded and guarded cycles • Single threaded cycle Lock a, b 1 a.lock() 2 b.lock() 3 unlock(a, b) 4 b.lock() 5 a.lock() 6 unlock(a, b) Slide 49/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Limitations: single threaded and guarded cycles • Guarded cycle Thread 1 Thread 2 1 g.lock() 2 a.lock() 3 b.lock() 4 unlock(a, b) 5 g.unlock() 6 g.lock() 7 b.lock() 8 a.lock() 9 unlock(a, b) 10 g.unlock() Slide 50/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Limitations: single threaded and guarded cycles • They are still lock hierarchy violation • Such potential deadlock may become possible after code refactoring Slide 51/65. Copyright © 2017. Devexperts LLC. All rights reserved.

IMPLEMENTATION

Dl-Check • Implemented as java agent - java -javaagent:dlcheck.jar -jar your_app.jar - Can be used with Junit: -Ddlcheck.fail=true • Inserts analysis code into classes at run-time Slide 53/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Dl-Check output example ========================== !!! Potential deadlock !!! ========================== ### Cycle in lock graph: ### Lock ReentrantLock@75915908 was acquired at: com/intellij/ide/util/treeView/AbstractTreeUi.acquireLock(AbstractTreeUi.java:2464) com/intellij/ide/util/treeView/AbstractTreeUi.attemptLock(AbstractTreeUi.java:2460) Lock ProjectViewTreeUpdater@46e87eab was acquired at: com/intellij/ide/util/treeView/AbstractTreeUpdater.reset(AbstractTreeUpdater.java:415) com/intellij/ide/util/treeView/AbstractTreeUpdater.getUpdateCount(AbstractTreeUpdater.java:344) com/intellij/ide/util/treeView/AbstractTreeUpdater.performUpdate(AbstractTreeUpdater.java:242) com/intellij/ide/util/treeView/AbstractTreeUpdater.isEnqueuedToUpdate(AbstractTreeUpdater.java:380) com/intellij/ide/util/treeView/AbstractTreeUpdater.cancelAllRequests(AbstractTreeUpdater.java:326) ### Current lock stack: ### Lock Object@5cbe2bca was acquired at: com/intellij/openapi/application/impl/LaterInvocator$FlushQueue.a(LaterInvocator.java:320) Lock ReentrantLock@75915908 was acquired at: com/intellij/ide/util/treeView/AbstractTreeUi.acquireLock(AbstractTreeUi.java:2464) com/intellij/ide/util/treeView/AbstractTreeUi.attemptLock(AbstractTreeUi.java:2460) ### Current stacktrace: ### com.intellij.ide.util.treeView.AbstractTreeUpdater.cancelAllRequests(AbstractTreeUpdater.java:326) com.intellij.ide.util.treeView.AbstractTreeUi.releaseNow(AbstractTreeUi.java:457) com.intellij.ide.util.treeView.AbstractTreeUi.access$1400(AbstractTreeUi.java:66) com.intellij.ide.util.treeView.AbstractTreeUi$12.perform(AbstractTreeUi.java:438) ... Slide 54/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Byte-code instrumentation • Captures lock acquire and release operations: - After MONITORENTER and before MONITOREXIT instructions - At the beginning and the end of synchronized method - java.util.concurrent.locks.Lock • After lock method • After successful tryLock method invocation • Before unlock method Slide 55/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Scalability • MNR algorithm is NOT concurrent • Read-write lock - Read lock for read-only operations and current node modification - Write lock for topological order modification - SWMR (single-writer multi-reader) data structures Slide 56/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Memory management • Java has a Garbage Collector • Old lock instances and nodes should be collected by GC • WeakReference is used for internal data structures Slide 57/65. Copyright © 2017. Devexperts LLC. All rights reserved.

EVALUATION

Benchmarks • SpecJVM2008: Apache Derby (4 and 40 threads) • DaCapo: Apache Lucene • DaCapo: Banking application • Fine-Grained (10 threads, 100 and 10’000 locks) - Like transfer(from, to, amount) from examples • … Slide 59/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Memory usage impact 1 1,07 1,06 1 1,61 1,34 0,61 1,22 1 1 0,74 1,12 1,35 1 0 0,2 0,4 0,6 0,8 1 1,2 1,4 1,6 1,8 2 Derby 1 Derby 2 Lucene Bank FG 1 FG 2 INCREASE FACTOR Dl-Check Jcarder MulticoreSDK Slide 60/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Performance impact 1,53 1,76 1,03 2,6 4,29 18,05 9,7 128,2 1,25 2,6 7,3 120,71 40,33 3,25 1 2 4 8 16 32 64 128 Derby 1 Derby 2 Lucene Bank FG 1 FG 2 SLOWDOWN FACTOR Dl-Check Jcarder MulticoreSDK Slide 61/65. Copyright © 2017. Devexperts LLC. All rights reserved.

CONCLUSION

Summary • Algorithm to detect potential deadlocks at run-time • Dl-Check tool has been implemented • https://github.com/Devexperts/dlcheck • Future work: - Lock grouping feature - Contracts to describe lock acquisition rules Slide 63/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Thank you for your attention! Slide 64/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Dl-Check: dynamic potential deadlock detection tool for Java programs Nikita Koval1,2 Dmitry Tsitelov2 Roman Elizarov2 1ITMO University, Computer Technology Department, St. Petersburg, Russia [email protected] 2Devexperts, LLC, St. Petersburg, Russia, [email protected] TMPA Conference, March 2017

Topological order: extra • Can be maintained in a single iteration in case no cycles are formed • Already acquired locks are ordered already • Edge from the rightest node should be used Slide 66/65. Copyright © 2017. Devexperts LLC. All rights reserved. ... 1 a.lock() 2 b.lock() 3 c.lock() ... Topological order

Topological order: example (0) void transfer(Account from, Account to, int amount) { from.lock() to.lock() // Do transfer unlock(from, to) } 1 transfer(d, b) 2 transfer(d, c) 3 transfer(c, d) 4 transfer(e, a) 5 transfer(b, a) Slide 67/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Topological order: example (1) void transfer(Account from, Account to, int amount) { from.lock() to.lock() // Do transfer unlock(from, to) } 1 transfer(d, b) 2 transfer(d, c) 3 transfer(c, d) 4 transfer(e, a) 5 transfer(b, a) Affected region Slide 68/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Topological order: example (2) void transfer(Account from, Account to, int amount) { from.lock() to.lock() // Do transfer unlock(from, to) } 1 transfer(d, b) 2 transfer(d, c) 3 transfer(c, d) 4 transfer(e, a) 5 transfer(b, a) Affected region Slide 69/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Topological order: example (3) void transfer(Account from, Account to, int amount) { from.lock() to.lock() // Do transfer unlock(from, to) } 1 transfer(d, b) 2 transfer(d, c) 3 transfer(c, d) 4 transfer(e, a) 5 transfer(b, a) Affected region Slide 70/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Topological order: example (4) void transfer(Account from, Account to, int amount) { from.lock() to.lock() // Do transfer unlock(from, to) } 1 transfer(d, b) 2 transfer(d, c) 3 transfer(c, d) 4 transfer(e, a) 5 transfer(b, a) Affected region Slide 71/65. Copyright © 2017. Devexperts LLC. All rights reserved.

Topological order: example (4) void transfer(Account from, Account to, int amount) { from.lock() to.lock() // Do transfer unlock(from, to) } 1 transfer(d, b) 2 transfer(d, c) 3 transfer(c, d) 4 transfer(e, a) 5 transfer(b, a) Affected region Slide 72/65. Copyright © 2017. Devexperts LLC. All rights reserved.