Data Flow Analysis in IntelliJ IDEA

IntelliJ IDEA users are very familiar with the warnings that appear in Java code, like 'Condition is always true', 'Method invocation might produce NullPointerException', and so on. These warnings are powered by an abstract interpretation and data flow analysis engine. In this talk, we will delve into analysis internals and learn how your code looks from the point of view of the IDE, what kind of errors the IDE can detect, and what kind of problems are too difficult for it. We will see why false-positive warnings appear sometimes. Hopefully, this talk will help you understand better how the “IDE brain” works.

Transcript

1. Data Flow Analysis in IntelliJ IDEA
   __
   How the IDE Perceives Your Code
   Tagir Valeev
   

   View Slide

2. 2014-2015: Contributed to FindBugs static analyzer
   Since 2016: In JetBrains, Java team, data-flow analysis
   2
   

   View Slide

3. View Slide

4. void visitMethodCall(Call call) {
   Method m = call.resolve();
   if (m.equals(STREAM_FOR_EACH)) {
   Expression qualifier = call.getQualifier();
   if (qualifier instanceof Call) {
   Method m1 = ((Call) qualifier).resolve();
   if (m1.equals(COLLECTION_STREAM)) {
   reportWarning(call, "stream().forEach() can be replaced with forEach()");
   }
   }
   }
   }
   4
   

   View Slide

5. View Slide

6. Constant conditions & exceptions warnings
   6
   

   View Slide

7. Condition is always true/false
   7
   

   View Slide

8. idx < 4 -> true
   idx >= 4 -> exception
   8
   

   View Slide

9. >
   5 5
   if statement
   if body
   …
   9
   

   View Slide

10. View Slide

11. >
    3
    2
    -
    *
    12
    *
    %
    /
    6
    19
    4
    Math.abs
    -3
    11
    

    View Slide

12. >
    3
    2
    -
    *
    12
    *
    %
    /
    6
    19
    4
    Math.abs
    -3
    =36
    =3
    =3 =3
    =9
    =27
    =true
    12
    

    View Slide

13. 1. PUSH 12
    2. PUSH 3
    3. BINOP *
    4. PUSH 19
    5. PUSH 6
    6. BINOP /
    7. PUSH 4
    8. BINOP %
    9. PUSH -3
    10.CALL Math.abs
    11.BINOP *
    12.BINOP –
    13.PUSH 2
    14.BINOP >
    13
    

    View Slide

14. 1. PUSH 12 12
    2. PUSH 3 12 3
    3. BINOP * 36
    4. PUSH 19 36 19
    5. PUSH 6 36 19 6
    6. BINOP / 36 3
    7. PUSH 4 36 3 4
    8. BINOP % 36 3
    9. PUSH -3 36 3 -3
    10.CALL Math.abs 36 3 3
    11.BINOP * 36 9
    12.BINOP – 27
    13.PUSH 2 27 2
    14.BINOP > true
    14
    

    View Slide

15. 1. PUSH x
    2. PUSH 5
    3. BINOP %
    4. PUSH y
    5. PUSH 10
    6. BINOP %
    7. PUSH 20
    8. BINOP +
    9. BINOP >
    15
    

    View Slide

16. 1. PUSH x ?
    2. PUSH 5 ? 5
    3. BINOP %
    4. PUSH y
    5. PUSH 10
    6. BINOP %
    7. PUSH 20
    8. BINOP +
    9. BINOP >
    16
    

    View Slide

17. 1. PUSH x ?
    2. PUSH 5 ? 5
    3. BINOP % [-4..4]
    4. PUSH y
    5. PUSH 10
    6. BINOP %
    7. PUSH 20
    8. BINOP +
    9. BINOP >
    17
    

    View Slide

18. 1. PUSH x ?
    2. PUSH 5 ? 5
    3. BINOP % [-4..4]
    4. PUSH y [-4..4] ?
    5. PUSH 10 [-4..4] ? 10
    6. BINOP % [-4..4] [-9..9]
    7. PUSH 20 [-4..4] [-9..9] 20
    8. BINOP + [-4..4] [11..29]
    9. BINOP > false
    18
    

    View Slide

19. 1. PUSH x ?
    2. PUSH 2 ? 2
    3. BINOP * [even]
    4. PUSH y [even] ?
    5. PUSH 4 [even] ? 4
    6. BINOP * [even] [even]
    7. PUSH 1 [even] [even] 1
    8. BINOP + [even] [odd]
    9. BINOP == false
    19
    

    View Slide

20. Ctrl+Shift+P (Expression type)
    Ctrl+Shift+P twice (Advanced expression type)
    20
    

    View Slide

21. View Slide

22. 1. PUSH x
    2. PUSH 3
    3. ASSIGN
    4. POP
    int x = 3
    22
    

    View Slide

23. 1. PUSH x
    2. PUSH 3
    3. ASSIGN
    4. POP
    5. PUSH y
    6. PUSH x
    7. PUSH x
    8. BINOP *
    9. ASSIGN
    10.POP
    11.PUSH z
    12.PUSH x
    13.PUSH 2
    14.BINOP *
    15.ASSIGN
    16.POP
    17.PUSH x
    18.PUSH z
    19.BINOP +
    20.PUSH y
    21.BINOP >
    int x = 3
    int y = x * x
    int z = x * 2
    x + z > y
    23
    

    View Slide

24. 1. PUSH x x
    2. PUSH 3 x 3
    3. ASSIGN x ; x=3
    4. POP ; x=3
    5. PUSH y
    6. PUSH x
    7. PUSH x
    8. BINOP *
    9. ASSIGN
    10.POP
    11.PUSH z
    12.PUSH x
    13.PUSH 2
    14.BINOP *
    15.ASSIGN
    16.POP
    17.PUSH x
    18.PUSH z
    19.BINOP +
    20.PUSH y
    21.BINOP >
    24
    

    View Slide

25. 1. PUSH x x
    2. PUSH 3 x 3
    3. ASSIGN x ; x=3
    4. POP ; x=3
    5. PUSH y y ; x=3
    6. PUSH x y x ; x=3
    7. PUSH x y x x ; x=3
    8. BINOP * y 9 ; x=3, y=9
    9. ASSIGN y ; x=3, y=9
    10.POP ; x=3, y=9
    11.PUSH z z ; x=3, y=9
    12.PUSH x z x ; x=3, y=9
    13.PUSH 2 z x 2 ; x=3, y=9
    14.BINOP * z 6 ; x=3, y=9
    15.ASSIGN z ; x=3, y=9, z=6
    16.POP ; x=3, y=9, z=6
    17.PUSH x x ; x=3, y=9, z=6
    18.PUSH z x z ; x=3, y=9, z=6
    19.BINOP + 9 ; x=3, y=9, z=6
    20.PUSH y 9 y ; x=3, y=9, z=6
    21.BINOP > false ; x=3, y=9, z=6
    25
    

    View Slide

26. 1. PUSH x
    2. PUSH 0
    3. BINOP <
    4. IF_FALSE 10
    5. PUSH x
    6. PUSH 1
    7. BINOP >
    8. IF_FALSE 10
    9. …
    10.END
    26
    

    View Slide

27. 1. PUSH x x
    2. PUSH 0 x 0
    3. BINOP < true ; x < 0 false; x >= 0
    4. IF_FALSE 10
    5. PUSH x
    6. PUSH 1
    7. BINOP >
    8. IF_FALSE 10
    9. …
    10.END
    27
    

    View Slide

28. 1. PUSH x x
    2. PUSH 0 x 0
    3. BINOP < true ; x < 0 false; x >= 0
    4. IF_FALSE 10 ; x < 0
    5. PUSH x
    6. PUSH 1
    7. BINOP >
    8. IF_FALSE 10
    9. …
    10.END ; x >= 0
    28
    

    View Slide

29. 1. PUSH x x
    2. PUSH 0 x 0
    3. BINOP < true ; x < 0 false; x >= 0
    4. IF_FALSE 10 ; x < 0
    5. PUSH x x ; x < 0
    6. PUSH 1 x 1 ; x < 0
    7. BINOP > false; x < 0 // always false!
    8. IF_FALSE 10
    9. …
    10.END ; x < 0 ; x >= 0
    29
    

    View Slide

30. void test(int a, int b, int c, int d, int e, int f, int g, int h) {
    if (a > 0) System.out.println("a > 0");
    // a>0; a<=0
    if (b > 0) System.out.println("b > 0");
    // a>0, b>0; a>0, b<=0; a<=0, b>0; a<=0, b<=0
    if (c > 0) System.out.println("c > 0");
    // 8 states
    if (d > 0) System.out.println("d > 0");
    // 16 states
    if (e > 0) System.out.println("e > 0");
    // 32 states
    if (f > 0) System.out.println("f > 0");
    // 64 states
    if (g > 0) System.out.println("g > 0");
    // 128 states
    if (h > 0) System.out.println("h > 0");
    // 256 states
    }
    30
    

    View Slide

31. ; x < 0, y < 0 ; x > 0, y < 0
    ; x != 0, y < 0
    join
    31
    

    View Slide

32. View Slide

33. int a = x > 0 ? 1 : 0;
    // State#1: x > 0, a = 1
    // State#2: x <= 0, a = 0
    int b = y > 0 ? 1 : 0;
    // State#1: x > 0, a = 1, y > 0, b = 1
    // State#2: x > 0, a = 1, y <= 0, b = 0
    // State#3: x <= 0, a = 0, y > 0, b = 1
    // State#4: x <= 0, a = 0, y <= 0, b = 0
    if (a + b == 2) {
    // State#1: x > 0, a = 1, y > 0, b = 1
    if (x < 0) {
    ...
    }
    }
    33
    

    View Slide

34. View Slide

35. void printNumbers(int length) {
    for (int i = 0; i < length; ++i) {
    System.out.println(i);
    }
    }
    1. PUSH i
    2. PUSH 0
    3. ASSIGN
    4. POP
    5. PUSH i
    6. PUSH length
    7. BINOP <
    8. IF_FALSE 19
    9. PUSH System.out
    10.PUSH i
    11.CALL println
    12.PUSH i
    13.PUSH i
    14.PUSH 1
    15.BINOP +
    16.ASSIGN
    17.POP
    18.GOTO 5
    19.END
    int i = 0
    i < length
    ++i
    35
    

    View Slide

36. void printNumbers(int length) {
    for (int i = 0; i < length; ++i) {
    System.out.println(i);
    }
    }
    Iteration#1: i = 0; length > 0;
    Iteration#2: i = 0; length > 0 | i =1; length > 1
    Iteration#3: i = 0; length > 0 | i =1; length > 1 | i =2; length > 2
    Iteration#4: i = 0; length > 0 | i =1; length > 1 | i =2; length > 2 | i = 3; length > 3
    …
    36
    

    View Slide

37. void printNumbers(int length) {
    for (int i = 0; i < length; ++i) {
    System.out.println(i);
    }
    }
    Iteration#1: i = 0; length > 0;
    Iteration#2: i = [0..1]; length > 0
    Iteration#3: i = [0..2]; length > 0
    Iteration#4: i = [0..3]; length > 0
    …
    37
    

    View Slide

38. Type constraints
    _
    38
    

    View Slide

39. Type constraints
    _
    39
    

    View Slide

40. Type constraints
    _
    40
    

    View Slide

41. Nullness
    _
    41
    

    View Slide

42. Nullness
    _
    42
    

    View Slide

43. Nullness kinds:
    ✓ not-null
    ✓ unknown
    ✓ nullable
    ✓ null
    43
    

    View Slide

44. A B
    A B
    A B
    A ∪ B
    join
    A ∩ B
    meet
    Algebraic lattice
    44
    

    View Slide

45. Nullness domain
    Not-null Null
    Nullable
    Unknown
    ???
    45
    

    View Slide

46. View Slide

47. View Slide

48. View Slide

49. View Slide

50. https://stackoverflow.com/questions/4963300/which-notnull-java-annotation-should-i-use 50
    

    View Slide

51. View Slide

52. Tracking relations
    _
    52
    

    View Slide

53. Tracking relations
    _
    53
    

    View Slide

54. Tracking relations
    _
    54
    

    View Slide

55. Tracking relations
    _
    55
    

    View Slide

56. Tracking relations
    _
    56
    

    View Slide

57. Fields
    _
    57
    

    View Slide

58. Fields
    _
    58
    

    View Slide

59. View Slide

60. Pure methods
    _
    60
    

    View Slide

61. View Slide

62. Purity inference
    _
    62
    

    View Slide

63. External annotations
    _
    63
    

    View Slide

64. View Slide

65. External annotations
    _
    65
    

    View Slide

66. Locality tracking
    _
    66
    

    View Slide

67. Array elements
    _
    67
    

    View Slide

68. Array elements
    _
    void test(int[] array, int i) {
    if (array[0] == array[1]) {
    array[i] = 10;
    int diff = array[0] - array[1];
    System.out.println(diff);
    }
    }
    68
    

    View Slide

69. Array elements
    _
    void test(int[] array, int i) {
    if (array[0] == array[1] && i >= 2) {
    array[i] = 10;
    int diff = array[0] - array[1];
    System.out.println(diff);
    }
    }
    69
    

    View Slide

70. Array elements
    _
    void test(int[] array, int i, int j) {
    if (array[i] == array[j]) {
    int diff = array[i] - array[j];
    System.out.println(diff);
    }
    }
    70
    

    View Slide

71. public final class Point {
    private final int x, y;
    Point(int x, int y) {
    this.x = x;
    this.y = y;
    }
    public int getX() {
    return x;
    }
    public int getY() {
    return y;
    }
    }
    Getters
    _
    71
    

    View Slide

72. ✓ Array length
    ✓ String length
    ✓ Collection size (mutable)
    ✓ Map size (mutable)
    ✓ Optional value
    ✓ Boxed value
    Special fields
    _
    72
    

    View Slide

73. Hardcoded contracts
    _
    String.isEmpty():
    this.length == 0 -> true; else -> false
    73
    

    View Slide

74. Hardcoded contracts
    _
    String.charAt(int idx):
    idx >= this.length -> fail;
    idx < 0 -> fail
    74
    

    View Slide

75. Hardcoded contracts
    _
    String.startsWith(String str):
    this.length < str.length -> false
    75
    

    View Slide

76. Hardcoded contracts
    _
    String.startsWith(String str):
    this.length < str.length -> false
    76
    

    View Slide

77. Method handlers
    _
    77
    

    View Slide

78. Method handlers
    _
    java.lang.String:
    contains, indexOf, startsWith, endsWith, lastIndexOf, length, trim, substring, equals, equalsIgnoreCase, charAt,
    codePointAt, compareTo, replace, toCharArray, valueOf
    java.lang.Math:
    abs, sqrt, min, max
    java.lang.Integer/java.lang.Long:
    toString, toBinaryString, toHexString, toOctalString, toUnsignedString, compare, compareUnsigned
    java.lang.Collections:
    singleton, singletonList, singletonMap, emptyList, emptySet, emptyMap
    java.util.Arrays:
    asList, copyOf
    78
    

    View Slide

79. Inliners
    _
    79
    

    View Slide

80. Inliners
    _
    80
    

    View Slide

81. Method ranges
    _
    81
    

    View Slide

82. Method ranges
    _
    82
    

    View Slide

83. Mutability
    _
    83
    

    View Slide

84. Mutability
    _
    java.util.Collections:
    java.util.List:
    84
    

    View Slide

85. Common dataflow
    _
    85
    

    View Slide

86. Common dataflow
    _
    86
    

    View Slide

87. Common dataflow
    _
    87
    

    View Slide

88. Common dataflow
    _
    88
    

    View Slide

89. Dataflow in quick-fixes
    _
    89
    

    View Slide

90. Dataflow in quick-fixes
    _
    90
    

    View Slide

91. Dataflow in completion
    _
    91
    

    View Slide

92. Dataflow in completion
    _
    92
    

    View Slide

93. Dataflow in debugger
    _
    93
    

    View Slide

94. Dataflow in debugger
    _
    94
    

    View Slide

95. Conclusion
    _
    ✓ Data flow analysis performs abstract interpretation of your code
    ✓ It tracks constant values, ranges, oddity, types, nullability, mutability,
    variable equality and so on
    ✓ It knows about behavior of many library methods
    ✓ See how data flow analysis works using advanced expression type feature
    (Ctrl+Shift+P several times)
    ✓ Check inferred annotations to understand the interprocedural analyser
    reasoning
    ✓ Control its behavior by explicit or external annotations such as
    @Contract, @Nullable, @Range, @UnmodifiableView, etc.
    ✓ Data flow analysis is everywhere: inspections, quick-fixes, completion, debugger,
    advanced expression type, refactorings, “dataflow to here”.
    95
    

    View Slide

96. Conclusion
    _
    ✓ Data flow analysis is your friend!
    ✓ Submit issues at https://youtrack.jetbrains.com/
    96
    

    View Slide

97. Thanks!
    Tagir Valeev
    tagir.valeev@jetbrains.com
    @tagir_valeev
    

    View Slide