Data Flow Analysis in IntelliJ IDEA

IntelliJ IDEA users are very familiar with the warnings that appear in Java code, like 'Condition is always true', 'Method invocation might produce NullPointerException', and so on. These warnings are powered by an abstract interpretation and data flow analysis engine. In this talk, we will delve into analysis internals and learn how your code looks from the point of view of the IDE, what kind of errors the IDE can detect, and what kind of problems are too difficult for it. We will see why false-positive warnings appear sometimes. Hopefully, this talk will help you understand better how the “IDE brain” works.

Transcript

1. Data Flow Analysis in IntelliJ IDEA __ How the IDE Perceives Your Code Tagir Valeev
2. 2014-2015: Contributed to FindBugs static analyzer Since 2016: In JetBrains, Java team, data-flow analysis 2
4. void visitMethodCall(Call call) { Method m = call.resolve(); if (m.equals(STREAM_FOR_EACH)) { Expression qualifier = call.getQualifier(); if (qualifier instanceof Call) { Method m1 = ((Call) qualifier).resolve(); if (m1.equals(COLLECTION_STREAM)) { reportWarning(call, "stream().forEach() can be replaced with forEach()"); } } } } 4
6. Constant conditions & exceptions warnings 6
7. Condition is always true/false 7
8. idx < 4 -> true idx >= 4 -> exception 8
9. > 5 5 if statement if body … 9
11. > 3 2 - * 12 * % / 6 19 4 Math.abs -3 11
12. > 3 2 - * 12 * % / 6 19 4 Math.abs -3 =36 =3 =3 =3 =9 =27 =true 12
13. 1. PUSH 12 2. PUSH 3 3. BINOP * 4. PUSH 19 5. PUSH 6 6. BINOP / 7. PUSH 4 8. BINOP % 9. PUSH -3 10.CALL Math.abs 11.BINOP * 12.BINOP – 13.PUSH 2 14.BINOP > 13
14. 1. PUSH 12 12 2. PUSH 3 12 3 3. BINOP * 36 4. PUSH 19 36 19 5. PUSH 6 36 19 6 6. BINOP / 36 3 7. PUSH 4 36 3 4 8. BINOP % 36 3 9. PUSH -3 36 3 -3 10.CALL Math.abs 36 3 3 11.BINOP * 36 9 12.BINOP – 27 13.PUSH 2 27 2 14.BINOP > true 14
15. 1. PUSH x 2. PUSH 5 3. BINOP % 4. PUSH y 5. PUSH 10 6. BINOP % 7. PUSH 20 8. BINOP + 9. BINOP > 15
16. 1. PUSH x ? 2. PUSH 5 ? 5 3. BINOP % 4. PUSH y 5. PUSH 10 6. BINOP % 7. PUSH 20 8. BINOP + 9. BINOP > 16
17. 1. PUSH x ? 2. PUSH 5 ? 5 3. BINOP % [-4..4] 4. PUSH y 5. PUSH 10 6. BINOP % 7. PUSH 20 8. BINOP + 9. BINOP > 17
18. 1. PUSH x ? 2. PUSH 5 ? 5 3. BINOP % [-4..4] 4. PUSH y [-4..4] ? 5. PUSH 10 [-4..4] ? 10 6. BINOP % [-4..4] [-9..9] 7. PUSH 20 [-4..4] [-9..9] 20 8. BINOP + [-4..4] [11..29] 9. BINOP > false 18
19. 1. PUSH x ? 2. PUSH 2 ? 2 3. BINOP * [even] 4. PUSH y [even] ? 5. PUSH 4 [even] ? 4 6. BINOP * [even] [even] 7. PUSH 1 [even] [even] 1 8. BINOP + [even] [odd] 9. BINOP == false 19
20. Ctrl+Shift+P (Expression type) Ctrl+Shift+P twice (Advanced expression type) 20
22. 1. PUSH x 2. PUSH 3 3. ASSIGN 4. POP int x = 3 22
23. 1. PUSH x 2. PUSH 3 3. ASSIGN 4. POP 5. PUSH y 6. PUSH x 7. PUSH x 8. BINOP * 9. ASSIGN 10.POP 11.PUSH z 12.PUSH x 13.PUSH 2 14.BINOP * 15.ASSIGN 16.POP 17.PUSH x 18.PUSH z 19.BINOP + 20.PUSH y 21.BINOP > int x = 3 int y = x * x int z = x * 2 x + z > y 23
24. 1. PUSH x x 2. PUSH 3 x 3 3. ASSIGN x ; x=3 4. POP ; x=3 5. PUSH y 6. PUSH x 7. PUSH x 8. BINOP * 9. ASSIGN 10.POP 11.PUSH z 12.PUSH x 13.PUSH 2 14.BINOP * 15.ASSIGN 16.POP 17.PUSH x 18.PUSH z 19.BINOP + 20.PUSH y 21.BINOP > 24
25. 1. PUSH x x 2. PUSH 3 x 3 3. ASSIGN x ; x=3 4. POP ; x=3 5. PUSH y y ; x=3 6. PUSH x y x ; x=3 7. PUSH x y x x ; x=3 8. BINOP * y 9 ; x=3, y=9 9. ASSIGN y ; x=3, y=9 10.POP ; x=3, y=9 11.PUSH z z ; x=3, y=9 12.PUSH x z x ; x=3, y=9 13.PUSH 2 z x 2 ; x=3, y=9 14.BINOP * z 6 ; x=3, y=9 15.ASSIGN z ; x=3, y=9, z=6 16.POP ; x=3, y=9, z=6 17.PUSH x x ; x=3, y=9, z=6 18.PUSH z x z ; x=3, y=9, z=6 19.BINOP + 9 ; x=3, y=9, z=6 20.PUSH y 9 y ; x=3, y=9, z=6 21.BINOP > false ; x=3, y=9, z=6 25
26. 1. PUSH x 2. PUSH 0 3. BINOP < 4. IF_FALSE 10 5. PUSH x 6. PUSH 1 7. BINOP > 8. IF_FALSE 10 9. … <if body> 10.END 26
27. 1. PUSH x x 2. PUSH 0 x 0 3. BINOP < true ; x < 0 false; x >= 0 4. IF_FALSE 10 5. PUSH x 6. PUSH 1 7. BINOP > 8. IF_FALSE 10 9. … <if body> 10.END 27
28. 1. PUSH x x 2. PUSH 0 x 0 3. BINOP < true ; x < 0 false; x >= 0 4. IF_FALSE 10 ; x < 0 5. PUSH x 6. PUSH 1 7. BINOP > 8. IF_FALSE 10 9. … <if body> 10.END ; x >= 0 28
29. 1. PUSH x x 2. PUSH 0 x 0 3. BINOP < true ; x < 0 false; x >= 0 4. IF_FALSE 10 ; x < 0 5. PUSH x x ; x < 0 6. PUSH 1 x 1 ; x < 0 7. BINOP > false; x < 0 // always false! 8. IF_FALSE 10 9. … <if body> 10.END ; x < 0 ; x >= 0 29
30. void test(int a, int b, int c, int d, int e, int f, int g, int h) { if (a > 0) System.out.println("a > 0"); // a>0; a<=0 if (b > 0) System.out.println("b > 0"); // a>0, b>0; a>0, b<=0; a<=0, b>0; a<=0, b<=0 if (c > 0) System.out.println("c > 0"); // 8 states if (d > 0) System.out.println("d > 0"); // 16 states if (e > 0) System.out.println("e > 0"); // 32 states if (f > 0) System.out.println("f > 0"); // 64 states if (g > 0) System.out.println("g > 0"); // 128 states if (h > 0) System.out.println("h > 0"); // 256 states } 30
31. <empty stack> ; x < 0, y < 0 <empty stack> ; x > 0, y < 0 <empty stack> ; x != 0, y < 0 join 31
33. int a = x > 0 ? 1 : 0; // State#1: x > 0, a = 1 // State#2: x <= 0, a = 0 int b = y > 0 ? 1 : 0; // State#1: x > 0, a = 1, y > 0, b = 1 // State#2: x > 0, a = 1, y <= 0, b = 0 // State#3: x <= 0, a = 0, y > 0, b = 1 // State#4: x <= 0, a = 0, y <= 0, b = 0 if (a + b == 2) { // State#1: x > 0, a = 1, y > 0, b = 1 if (x < 0) { ... } } 33
35. void printNumbers(int length) { for (int i = 0; i < length; ++i) { System.out.println(i); } } 1. PUSH i 2. PUSH 0 3. ASSIGN 4. POP 5. PUSH i 6. PUSH length 7. BINOP < 8. IF_FALSE 19 9. PUSH System.out 10.PUSH i 11.CALL println 12.PUSH i 13.PUSH i 14.PUSH 1 15.BINOP + 16.ASSIGN 17.POP 18.GOTO 5 19.END int i = 0 i < length ++i 35
36. void printNumbers(int length) { for (int i = 0; i < length; ++i) { System.out.println(i); } } Iteration#1: i = 0; length > 0; Iteration#2: i = 0; length > 0 | i =1; length > 1 Iteration#3: i = 0; length > 0 | i =1; length > 1 | i =2; length > 2 Iteration#4: i = 0; length > 0 | i =1; length > 1 | i =2; length > 2 | i = 3; length > 3 … 36
37. void printNumbers(int length) { for (int i = 0; i < length; ++i) { System.out.println(i); } } Iteration#1: i = 0; length > 0; Iteration#2: i = [0..1]; length > 0 Iteration#3: i = [0..2]; length > 0 Iteration#4: i = [0..3]; length > 0 … 37
38. Type constraints _ 38
39. Type constraints _ 39
40. Type constraints _ 40
41. Nullness _ 41
42. Nullness _ 42
43. Nullness kinds: ✓ not-null ✓ unknown ✓ nullable ✓ null 43
44. A B A B A B A ∪ B join A ∩ B meet Algebraic lattice 44
45. Nullness domain Not-null Null Nullable Unknown ??? 45
50. https://stackoverflow.com/questions/4963300/which-notnull-java-annotation-should-i-use 50
52. Tracking relations _ 52
53. Tracking relations _ 53
54. Tracking relations _ 54
55. Tracking relations _ 55
56. Tracking relations _ 56
57. Fields _ 57
58. Fields _ 58
60. Pure methods _ 60
62. Purity inference _ 62
63. External annotations _ 63
65. External annotations _ 65
66. Locality tracking _ 66
67. Array elements _ 67
68. Array elements _ void test(int[] array, int i) { if (array[0] == array[1]) { array[i] = 10; int diff = array[0] - array[1]; System.out.println(diff); } } 68
69. Array elements _ void test(int[] array, int i) { if (array[0] == array[1] && i >= 2) { array[i] = 10; int diff = array[0] - array[1]; System.out.println(diff); } } 69
70. Array elements _ void test(int[] array, int i, int j) { if (array[i] == array[j]) { int diff = array[i] - array[j]; System.out.println(diff); } } 70
71. public final class Point { private final int x, y; Point(int x, int y) { this.x = x; this.y = y; } public int getX() { return x; } public int getY() { return y; } } Getters _ 71
72. ✓ Array length ✓ String length ✓ Collection size (mutable) ✓ Map size (mutable) ✓ Optional value ✓ Boxed value Special fields _ 72
73. Hardcoded contracts _ String.isEmpty(): this.length == 0 -> true; else -> false 73
74. Hardcoded contracts _ String.charAt(int idx): idx >= this.length -> fail; idx < 0 -> fail 74
75. Hardcoded contracts _ String.startsWith(String str): this.length < str.length -> false 75
76. Hardcoded contracts _ String.startsWith(String str): this.length < str.length -> false 76
77. Method handlers _ 77
78. Method handlers _ java.lang.String: contains, indexOf, startsWith, endsWith, lastIndexOf, length, trim, substring, equals, equalsIgnoreCase, charAt, codePointAt, compareTo, replace, toCharArray, valueOf java.lang.Math: abs, sqrt, min, max java.lang.Integer/java.lang.Long: toString, toBinaryString, toHexString, toOctalString, toUnsignedString, compare, compareUnsigned java.lang.Collections: singleton, singletonList, singletonMap, emptyList, emptySet, emptyMap java.util.Arrays: asList, copyOf 78
79. Inliners _ 79
80. Inliners _ 80
81. Method ranges _ 81
82. Method ranges _ 82
83. Mutability _ 83
84. Mutability _ java.util.Collections: java.util.List: 84
85. Common dataflow _ 85
86. Common dataflow _ 86
87. Common dataflow _ 87
88. Common dataflow _ 88
89. Dataflow in quick-fixes _ 89
90. Dataflow in quick-fixes _ 90
91. Dataflow in completion _ 91
92. Dataflow in completion _ 92
93. Dataflow in debugger _ 93
94. Dataflow in debugger _ 94
95. Conclusion _ ✓ Data flow analysis performs abstract interpretation of your code ✓ It tracks constant values, ranges, oddity, types, nullability, mutability, variable equality and so on ✓ It knows about behavior of many library methods ✓ See how data flow analysis works using advanced expression type feature (Ctrl+Shift+P several times) ✓ Check inferred annotations to understand the interprocedural analyser reasoning ✓ Control its behavior by explicit or external annotations such as @Contract, @Nullable, @Range, @UnmodifiableView, etc. ✓ Data flow analysis is everywhere: inspections, quick-fixes, completion, debugger, advanced expression type, refactorings, “dataflow to here”. 95
96. Conclusion _ ✓ Data flow analysis is your friend! ✓ Submit issues at https://youtrack.jetbrains.com/ 96
97. Thanks! Tagir Valeev tagir.valeev@jetbrains.com @tagir_valeev