Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Data Flow Analysis in IntelliJ IDEA

Tagir Valeev
February 26, 2021

Data Flow Analysis in IntelliJ IDEA

IntelliJ IDEA users are very familiar with the warnings that appear in Java code, like 'Condition is always true', 'Method invocation might produce NullPointerException', and so on. These warnings are powered by an abstract interpretation and data flow analysis engine. In this talk, we will delve into analysis internals and learn how your code looks from the point of view of the IDE, what kind of errors the IDE can detect, and what kind of problems are too difficult for it. We will see why false-positive warnings appear sometimes. Hopefully, this talk will help you understand better how the “IDE brain” works.

Tagir Valeev

February 26, 2021
Tweet

More Decks by Tagir Valeev

Other Decks in Programming

Transcript

  1. Data Flow Analysis in IntelliJ IDEA
    __
    How the IDE Perceives Your Code
    Tagir Valeev

    View Slide

  2. 2014-2015: Contributed to FindBugs static analyzer
    Since 2016: In JetBrains, Java team, data-flow analysis
    2

    View Slide

  3. 3

    View Slide

  4. void visitMethodCall(Call call) {
    Method m = call.resolve();
    if (m.equals(STREAM_FOR_EACH)) {
    Expression qualifier = call.getQualifier();
    if (qualifier instanceof Call) {
    Method m1 = ((Call) qualifier).resolve();
    if (m1.equals(COLLECTION_STREAM)) {
    reportWarning(call, "stream().forEach() can be replaced with forEach()");
    }
    }
    }
    }
    4

    View Slide

  5. 5

    View Slide

  6. Constant conditions & exceptions warnings
    6

    View Slide

  7. Condition is always true/false
    7

    View Slide

  8. idx < 4 -> true
    idx >= 4 -> exception
    8

    View Slide

  9. >
    5 5
    if statement
    if body

    9

    View Slide

  10. 10

    View Slide

  11. >
    3
    2
    -
    *
    12
    *
    %
    /
    6
    19
    4
    Math.abs
    -3
    11

    View Slide

  12. >
    3
    2
    -
    *
    12
    *
    %
    /
    6
    19
    4
    Math.abs
    -3
    =36
    =3
    =3 =3
    =9
    =27
    =true
    12

    View Slide

  13. 1. PUSH 12
    2. PUSH 3
    3. BINOP *
    4. PUSH 19
    5. PUSH 6
    6. BINOP /
    7. PUSH 4
    8. BINOP %
    9. PUSH -3
    10.CALL Math.abs
    11.BINOP *
    12.BINOP –
    13.PUSH 2
    14.BINOP >
    13

    View Slide

  14. 1. PUSH 12 12
    2. PUSH 3 12 3
    3. BINOP * 36
    4. PUSH 19 36 19
    5. PUSH 6 36 19 6
    6. BINOP / 36 3
    7. PUSH 4 36 3 4
    8. BINOP % 36 3
    9. PUSH -3 36 3 -3
    10.CALL Math.abs 36 3 3
    11.BINOP * 36 9
    12.BINOP – 27
    13.PUSH 2 27 2
    14.BINOP > true
    14

    View Slide

  15. 1. PUSH x
    2. PUSH 5
    3. BINOP %
    4. PUSH y
    5. PUSH 10
    6. BINOP %
    7. PUSH 20
    8. BINOP +
    9. BINOP >
    15

    View Slide

  16. 1. PUSH x ?
    2. PUSH 5 ? 5
    3. BINOP %
    4. PUSH y
    5. PUSH 10
    6. BINOP %
    7. PUSH 20
    8. BINOP +
    9. BINOP >
    16

    View Slide

  17. 1. PUSH x ?
    2. PUSH 5 ? 5
    3. BINOP % [-4..4]
    4. PUSH y
    5. PUSH 10
    6. BINOP %
    7. PUSH 20
    8. BINOP +
    9. BINOP >
    17

    View Slide

  18. 1. PUSH x ?
    2. PUSH 5 ? 5
    3. BINOP % [-4..4]
    4. PUSH y [-4..4] ?
    5. PUSH 10 [-4..4] ? 10
    6. BINOP % [-4..4] [-9..9]
    7. PUSH 20 [-4..4] [-9..9] 20
    8. BINOP + [-4..4] [11..29]
    9. BINOP > false
    18

    View Slide

  19. 1. PUSH x ?
    2. PUSH 2 ? 2
    3. BINOP * [even]
    4. PUSH y [even] ?
    5. PUSH 4 [even] ? 4
    6. BINOP * [even] [even]
    7. PUSH 1 [even] [even] 1
    8. BINOP + [even] [odd]
    9. BINOP == false
    19

    View Slide

  20. Ctrl+Shift+P (Expression type)
    Ctrl+Shift+P twice (Advanced expression type)
    20

    View Slide

  21. 21

    View Slide

  22. 1. PUSH x
    2. PUSH 3
    3. ASSIGN
    4. POP
    int x = 3
    22

    View Slide

  23. 1. PUSH x
    2. PUSH 3
    3. ASSIGN
    4. POP
    5. PUSH y
    6. PUSH x
    7. PUSH x
    8. BINOP *
    9. ASSIGN
    10.POP
    11.PUSH z
    12.PUSH x
    13.PUSH 2
    14.BINOP *
    15.ASSIGN
    16.POP
    17.PUSH x
    18.PUSH z
    19.BINOP +
    20.PUSH y
    21.BINOP >
    int x = 3
    int y = x * x
    int z = x * 2
    x + z > y
    23

    View Slide

  24. 1. PUSH x x
    2. PUSH 3 x 3
    3. ASSIGN x ; x=3
    4. POP ; x=3
    5. PUSH y
    6. PUSH x
    7. PUSH x
    8. BINOP *
    9. ASSIGN
    10.POP
    11.PUSH z
    12.PUSH x
    13.PUSH 2
    14.BINOP *
    15.ASSIGN
    16.POP
    17.PUSH x
    18.PUSH z
    19.BINOP +
    20.PUSH y
    21.BINOP >
    24

    View Slide

  25. 1. PUSH x x
    2. PUSH 3 x 3
    3. ASSIGN x ; x=3
    4. POP ; x=3
    5. PUSH y y ; x=3
    6. PUSH x y x ; x=3
    7. PUSH x y x x ; x=3
    8. BINOP * y 9 ; x=3, y=9
    9. ASSIGN y ; x=3, y=9
    10.POP ; x=3, y=9
    11.PUSH z z ; x=3, y=9
    12.PUSH x z x ; x=3, y=9
    13.PUSH 2 z x 2 ; x=3, y=9
    14.BINOP * z 6 ; x=3, y=9
    15.ASSIGN z ; x=3, y=9, z=6
    16.POP ; x=3, y=9, z=6
    17.PUSH x x ; x=3, y=9, z=6
    18.PUSH z x z ; x=3, y=9, z=6
    19.BINOP + 9 ; x=3, y=9, z=6
    20.PUSH y 9 y ; x=3, y=9, z=6
    21.BINOP > false ; x=3, y=9, z=6
    25

    View Slide

  26. 1. PUSH x
    2. PUSH 0
    3. BINOP <
    4. IF_FALSE 10
    5. PUSH x
    6. PUSH 1
    7. BINOP >
    8. IF_FALSE 10
    9. …
    10.END
    26

    View Slide

  27. 1. PUSH x x
    2. PUSH 0 x 0
    3. BINOP < true ; x < 0 false; x >= 0
    4. IF_FALSE 10
    5. PUSH x
    6. PUSH 1
    7. BINOP >
    8. IF_FALSE 10
    9. …
    10.END
    27

    View Slide

  28. 1. PUSH x x
    2. PUSH 0 x 0
    3. BINOP < true ; x < 0 false; x >= 0
    4. IF_FALSE 10 ; x < 0
    5. PUSH x
    6. PUSH 1
    7. BINOP >
    8. IF_FALSE 10
    9. …
    10.END ; x >= 0
    28

    View Slide

  29. 1. PUSH x x
    2. PUSH 0 x 0
    3. BINOP < true ; x < 0 false; x >= 0
    4. IF_FALSE 10 ; x < 0
    5. PUSH x x ; x < 0
    6. PUSH 1 x 1 ; x < 0
    7. BINOP > false; x < 0 // always false!
    8. IF_FALSE 10
    9. …
    10.END ; x < 0 ; x >= 0
    29

    View Slide

  30. void test(int a, int b, int c, int d, int e, int f, int g, int h) {
    if (a > 0) System.out.println("a > 0");
    // a>0; a<=0
    if (b > 0) System.out.println("b > 0");
    // a>0, b>0; a>0, b<=0; a<=0, b>0; a<=0, b<=0
    if (c > 0) System.out.println("c > 0");
    // 8 states
    if (d > 0) System.out.println("d > 0");
    // 16 states
    if (e > 0) System.out.println("e > 0");
    // 32 states
    if (f > 0) System.out.println("f > 0");
    // 64 states
    if (g > 0) System.out.println("g > 0");
    // 128 states
    if (h > 0) System.out.println("h > 0");
    // 256 states
    }
    30

    View Slide

  31. ; x < 0, y < 0 ; x > 0, y < 0
    ; x != 0, y < 0
    join
    31

    View Slide

  32. 32

    View Slide

  33. int a = x > 0 ? 1 : 0;
    // State#1: x > 0, a = 1
    // State#2: x <= 0, a = 0
    int b = y > 0 ? 1 : 0;
    // State#1: x > 0, a = 1, y > 0, b = 1
    // State#2: x > 0, a = 1, y <= 0, b = 0
    // State#3: x <= 0, a = 0, y > 0, b = 1
    // State#4: x <= 0, a = 0, y <= 0, b = 0
    if (a + b == 2) {
    // State#1: x > 0, a = 1, y > 0, b = 1
    if (x < 0) {
    ...
    }
    }
    33

    View Slide

  34. 34

    View Slide

  35. void printNumbers(int length) {
    for (int i = 0; i < length; ++i) {
    System.out.println(i);
    }
    }
    1. PUSH i
    2. PUSH 0
    3. ASSIGN
    4. POP
    5. PUSH i
    6. PUSH length
    7. BINOP <
    8. IF_FALSE 19
    9. PUSH System.out
    10.PUSH i
    11.CALL println
    12.PUSH i
    13.PUSH i
    14.PUSH 1
    15.BINOP +
    16.ASSIGN
    17.POP
    18.GOTO 5
    19.END
    int i = 0
    i < length
    ++i
    35

    View Slide

  36. void printNumbers(int length) {
    for (int i = 0; i < length; ++i) {
    System.out.println(i);
    }
    }
    Iteration#1: i = 0; length > 0;
    Iteration#2: i = 0; length > 0 | i =1; length > 1
    Iteration#3: i = 0; length > 0 | i =1; length > 1 | i =2; length > 2
    Iteration#4: i = 0; length > 0 | i =1; length > 1 | i =2; length > 2 | i = 3; length > 3

    36

    View Slide

  37. void printNumbers(int length) {
    for (int i = 0; i < length; ++i) {
    System.out.println(i);
    }
    }
    Iteration#1: i = 0; length > 0;
    Iteration#2: i = [0..1]; length > 0
    Iteration#3: i = [0..2]; length > 0
    Iteration#4: i = [0..3]; length > 0

    37

    View Slide

  38. Type constraints
    _
    38

    View Slide

  39. Type constraints
    _
    39

    View Slide

  40. Type constraints
    _
    40

    View Slide

  41. Nullness
    _
    41

    View Slide

  42. Nullness
    _
    42

    View Slide

  43. Nullness kinds:
    ✓ not-null
    ✓ unknown
    ✓ nullable
    ✓ null
    43

    View Slide

  44. A B
    A B
    A B
    A ∪ B
    join
    A ∩ B
    meet
    Algebraic lattice
    44

    View Slide

  45. Nullness domain
    Not-null Null
    Nullable
    Unknown
    ???
    45

    View Slide

  46. 46

    View Slide

  47. 47

    View Slide

  48. 48

    View Slide

  49. 49

    View Slide

  50. https://stackoverflow.com/questions/4963300/which-notnull-java-annotation-should-i-use 50

    View Slide

  51. 51

    View Slide

  52. Tracking relations
    _
    52

    View Slide

  53. Tracking relations
    _
    53

    View Slide

  54. Tracking relations
    _
    54

    View Slide

  55. Tracking relations
    _
    55

    View Slide

  56. Tracking relations
    _
    56

    View Slide

  57. Fields
    _
    57

    View Slide

  58. Fields
    _
    58

    View Slide

  59. 59

    View Slide

  60. Pure methods
    _
    60

    View Slide

  61. 61

    View Slide

  62. Purity inference
    _
    62

    View Slide

  63. External annotations
    _
    63

    View Slide

  64. 64

    View Slide

  65. External annotations
    _
    65

    View Slide

  66. Locality tracking
    _
    66

    View Slide

  67. Array elements
    _
    67

    View Slide

  68. Array elements
    _
    void test(int[] array, int i) {
    if (array[0] == array[1]) {
    array[i] = 10;
    int diff = array[0] - array[1];
    System.out.println(diff);
    }
    }
    68

    View Slide

  69. Array elements
    _
    void test(int[] array, int i) {
    if (array[0] == array[1] && i >= 2) {
    array[i] = 10;
    int diff = array[0] - array[1];
    System.out.println(diff);
    }
    }
    69

    View Slide

  70. Array elements
    _
    void test(int[] array, int i, int j) {
    if (array[i] == array[j]) {
    int diff = array[i] - array[j];
    System.out.println(diff);
    }
    }
    70

    View Slide

  71. public final class Point {
    private final int x, y;
    Point(int x, int y) {
    this.x = x;
    this.y = y;
    }
    public int getX() {
    return x;
    }
    public int getY() {
    return y;
    }
    }
    Getters
    _
    71

    View Slide

  72. ✓ Array length
    ✓ String length
    ✓ Collection size (mutable)
    ✓ Map size (mutable)
    ✓ Optional value
    ✓ Boxed value
    Special fields
    _
    72

    View Slide

  73. Hardcoded contracts
    _
    String.isEmpty():
    this.length == 0 -> true; else -> false
    73

    View Slide

  74. Hardcoded contracts
    _
    String.charAt(int idx):
    idx >= this.length -> fail;
    idx < 0 -> fail
    74

    View Slide

  75. Hardcoded contracts
    _
    String.startsWith(String str):
    this.length < str.length -> false
    75

    View Slide

  76. Hardcoded contracts
    _
    String.startsWith(String str):
    this.length < str.length -> false
    76

    View Slide

  77. Method handlers
    _
    77

    View Slide

  78. Method handlers
    _
    java.lang.String:
    contains, indexOf, startsWith, endsWith, lastIndexOf, length, trim, substring, equals, equalsIgnoreCase, charAt,
    codePointAt, compareTo, replace, toCharArray, valueOf
    java.lang.Math:
    abs, sqrt, min, max
    java.lang.Integer/java.lang.Long:
    toString, toBinaryString, toHexString, toOctalString, toUnsignedString, compare, compareUnsigned
    java.lang.Collections:
    singleton, singletonList, singletonMap, emptyList, emptySet, emptyMap
    java.util.Arrays:
    asList, copyOf
    78

    View Slide

  79. Inliners
    _
    79

    View Slide

  80. Inliners
    _
    80

    View Slide

  81. Method ranges
    _
    81

    View Slide

  82. Method ranges
    _
    82

    View Slide

  83. Mutability
    _
    83

    View Slide

  84. Mutability
    _
    java.util.Collections:
    java.util.List:
    84

    View Slide

  85. Common dataflow
    _
    85

    View Slide

  86. Common dataflow
    _
    86

    View Slide

  87. Common dataflow
    _
    87

    View Slide

  88. Common dataflow
    _
    88

    View Slide

  89. Dataflow in quick-fixes
    _
    89

    View Slide

  90. Dataflow in quick-fixes
    _
    90

    View Slide

  91. Dataflow in completion
    _
    91

    View Slide

  92. Dataflow in completion
    _
    92

    View Slide

  93. Dataflow in debugger
    _
    93

    View Slide

  94. Dataflow in debugger
    _
    94

    View Slide

  95. Conclusion
    _
    ✓ Data flow analysis performs abstract interpretation of your code
    ✓ It tracks constant values, ranges, oddity, types, nullability, mutability,
    variable equality and so on
    ✓ It knows about behavior of many library methods
    ✓ See how data flow analysis works using advanced expression type feature
    (Ctrl+Shift+P several times)
    ✓ Check inferred annotations to understand the interprocedural analyser
    reasoning
    ✓ Control its behavior by explicit or external annotations such as
    @Contract, @Nullable, @Range, @UnmodifiableView, etc.
    ✓ Data flow analysis is everywhere: inspections, quick-fixes, completion, debugger,
    advanced expression type, refactorings, “dataflow to here”.
    95

    View Slide

  96. Conclusion
    _
    ✓ Data flow analysis is your friend!
    ✓ Submit issues at https://youtrack.jetbrains.com/
    96

    View Slide

  97. Thanks!
    Tagir Valeev
    [email protected]
    @tagir_valeev

    View Slide