Troubleshooting Apache Cassandra

Transcript

1. AARON MORTON THE LAST PICKLE TROUBLESHOOTING APACHE CASSANDRA : OBSERVATION

   AND CONTROL

2. AARON MORTON @AARONMORTON CEO, THE LAST PICKLE APACHE CASSANDRA COMMITTER,

   PMC MEMBER

3. PROBLEMS LET'S TALK ABOUT

4. 4 OPTIMISING OR TROUBLESHOOTING 4

5. 5 OPTIMISE THROUGHPUT. MORE REQUESTS PER SECOND. 5

6. 6 OPTIMISE LATENCY. FASTER REQUESTS. 6

7. 7 OPTIMISE COST. LOWER COST PER REQUEST. 7

8. 8 TROUBLESHOOT AVAILABILITY. REQUESTS NOT STARTING OR NOT COMPLETING. 8

9. 9 TROUBLESHOOT CONSISTENCY. REQUESTS NOT RETURNING EXPECTED DATA. 9

10. 10 HOW DO YOU SOLVE PROBLEMS? 10

11. 11 ANTI METHOD: STREETLIGHT 11

12. 12 ANTI METHOD: RANDOM CHANGE 12

13. 13 ANTI METHOD: BLAME SOMEONE ELSE 13

14. 14 OODA LOOP: OBSERVE, ORIENTATE, DECIDE, ACT. 14

15. 15 15

16. OBSERVE

17. 17 “WHAT CAN I SEE?” 17

18. 18 A QUICK WORD ABOUT TIME. 18

19. 19 19

20. 20 GENERALLY CARE ABOUT MILLISECONDS AND MICROSECONDS. 20

21. 21 21

22. 22 FOR HUMANS TIME FLOWS IN ONE DIRECTION… 22

23. 23 … FOR COMPUTERS IT STOPS, JUMPS BACK, JUMPS FORWARD,

    AND IS DIFFERENT ON EVERY MACHINE. 23

24. 24 (BACK TO) “WHAT CAN I SEE?” 24

25. 25 LOGS ARE VERY IMPORTANT, TAKE GOOD CARE OF THEM.

    25

26. 26 OBSERVING WITH NODETOOL CFSTATS 26

27. 27 KEYSPACE: TEST_CLUSTER READ COUNT: 294681 READ LATENCY: 0.11929593017534215 MS.

    WRITE COUNT: 0 WRITE LATENCY: NAN MS. PENDING FLUSHES: 0 TABLE: MY_TABLE SSTABLE COUNT: 2 SPACE USED (LIVE): 1669941 SPACE USED (TOTAL): 1669941 SPACE USED BY SNAPSHOTS (TOTAL): 0 OFF HEAP MEMORY USED (TOTAL): 57384 SSTABLE COMPRESSION RATIO: 0.40934065577372514 NUMBER OF KEYS (ESTIMATE): 39683 MEMTABLE CELL COUNT: 0 MEMTABLE DATA SIZE: 0 MEMTABLE OFF HEAP MEMORY USED: 0 MEMTABLE SWITCH COUNT: 0 27 LOCAL READ COUNT: 294681 LOCAL READ LATENCY: 0.114 MS LOCAL WRITE COUNT: 0 LOCAL WRITE LATENCY: NAN MS PENDING FLUSHES: 0 BLOOM FILTER FALSE POSITIVES: 1143 BLOOM FILTER FALSE RATIO: 0.00467 BLOOM FILTER SPACE USED: 51896 BLOOM FILTER OFF HEAP MEMORY USED: 51880 INDEX SUMMARY OFF HEAP MEMORY USED: 5184 COMPRESSION METADATA OFF HEAP MEMORY USED: 320 COMPACTED PARTITION MINIMUM BYTES: 18 COMPACTED PARTITION MAXIMUM BYTES: 72 COMPACTED PARTITION MEAN BYTES: 68 AVERAGE LIVE CELLS PER SLICE (LAST FIVE MINUTES): 1.0 MAXIMUM LIVE CELLS PER SLICE (LAST FIVE MINUTES): 1 AVERAGE TOMBSTONES PER SLICE (LAST FIVE MINUTES): 1.0 MAXIMUM TOMBSTONES PER SLICE (LAST FIVE MINUTES): 1

28. 28 NODETOOL CFHISTOGRAMS 28

29. 29 TEST_CLUSTER/MY_TABLE HISTOGRAMS PERCENTILE SSTABLES WRITE LATENCY READ LATENCY PARTITION

    SIZE CELL COUNT (MICROS) (MICROS) (BYTES) 50% 1.00 0.00 105.78 72 2 75% 1.00 0.00 126.93 72 2 95% 1.00 0.00 152.32 72 2 98% 1.00 0.00 182.79 72 2 99% 1.00 0.00 219.34 72 2 MIN 1.00 0.00 9.89 18 0 MAX 1.00 0.00 10090.81 72 2 29

30. 30 NODETOOL TPSTATS 30

31. 31 POOL NAME ACTIVE PENDING COMPLETED BLOCKED ALL TIME BLOCKED

    MUTATIONSTAGE 0 0 10462 0 0 READSTAGE 0 0 339046 0 0 REQUESTRESPONSESTAGE 0 0 313949 0 0 READREPAIRSTAGE 0 0 48074 0 0 COUNTERMUTATIONSTAGE 0 0 0 0 0 HINTEDHANDOFF 0 0 30 0 0 MISCSTAGE 0 0 0 0 0 COMPACTIONEXECUTOR 0 0 7544 0 0 MEMTABLERECLAIMMEMORY 0 0 27 0 0 PENDINGRANGECALCULATOR 0 0 18 0 0 GOSSIPSTAGE 0 0 47521 0 0 MIGRATIONSTAGE 0 0 0 0 0 MEMTABLEPOSTFLUSH 0 0 306 0 0 VALIDATIONEXECUTOR 0 0 0 0 0 SAMPLER 0 0 0 0 0 MEMTABLEFLUSHWRITER 0 0 27 0 0 INTERNALRESPONSESTAGE 0 0 0 0 0 ANTIENTROPYSTAGE 0 0 0 0 0 CACHECLEANUPEXECUTOR 0 0 0 0 0 NATIVE-TRANSPORT-REQUESTS 0 0 28108 0 0 RPC-THREAD 204 204 0 0 0 31 MESSAGE TYPE DROPPED READ 0 RANGE_SLICE 0 _TRACE 0 MUTATION 0 COUNTER_MUTATION 0 REQUEST_RESPONSE 0 PAGED_RANGE 0 READ_REPAIR 0

32. 32 OBSERVING WITH IOSTATS 32

33. 33 AVG-CPU: %USER %NICE %SYSTEM %IOWAIT %STEAL %IDLE 4.34 0.78

    1.35 3.86 0.19 89.47 DEVICE: RRQM/S WRQM/S R/S W/S RKB/S WKB/S AVGRQ-SZ AVGQU-SZ AWAIT R_AWAIT W_AWAIT SVCTM %UTIL XVDG 0.07 0.27 1269.19 7.05 6172.58 749.07 10.85 0.21 0.16 0.15 3.30 0.24 30.17 33

34. 34 OBSERVING WITH NETSTATS 34

35. 35 ACTIVE INTERNET CONNECTIONS PROTO RECV-Q SEND-Q LOCAL ADDRESS FOREIGN

    ADDRESS (STATE) TCP4 0 0 172.20.10.2.54779 138.213.186.35.B.HTTPS ESTABLISHED TCP4 0 0 172.20.10.2.54778 192.168.1.104.SUN-SR-H SYN_SENT TCP4 0 31 172.20.10.2.54718 XX-FBCDN-SHV-01-.HTTPS CLOSING 35

36. 36 36

37. 37 METRICS ARE BETTER! 37

38. 38 METRICS OVERVIEW WIKI.APACHE.ORG/CASSANDRA/ METRICS 38

39. 39 39

40. 40 CLUSTER THROUGHPUT 40

41. 41 .O.A.C.M.CLIENTREQUEST. WRITE.LATENCY.1MINUTERATE READ.LATENCY.1MINUTERATE 41

42. 42 LOCAL TABLE THROUGHPUT 42

43. 43 .O.A.C.M.COLUMNFAMILY. KEYSPACE.TABLE.WRITELATENCY.1MINUTERATE KEYSPACE.TABLE.READLATENCY.1MINUTERATE 43

44. 44 CLUSTER LATENCY 44

45. 45 .O.A.C.M.CLIENTREQUEST. WRITE.LATENCY.95PERCENTILE READ.LATENCY.95PERCENTILE 45

46. 46 LOCAL TABLE LATENCY 46

47. 47 .O.A.C.M.COLUMNFAMILY. KEYSPACE.TABLE.WRITELATENCY.95PERCENTILE KEYSPACE.TABLE.READLATENCY.95PERCENTILE 47

48. 48 READ PATH 48

49. 49 .O.A.C.M.COLUMNFAMILY.KEYSPACE.TABLE. LIVESCANNEDHISTOGRAM.95PERCENTILE TOMBSTONESCANNEDHISTOGRAM.95PERCENTILE SSTABLESPERREADHISTOGRAM.95PERCENTILE 49

50. 50 YAY: THERE IS A LOT OF METRICS. BOO: THERE

    IS A LOT OF METRICS. 50

51. 51 TLP DASHBOARDS FOR DATA DOG AND GRAPHANA. 51

52. TLP - OVERVIEW

53. TLP - WRITE PATH

54. TLP - READ PATH

55. TLP - SSTABLE MANAGEMENT

56. 56 SEE DATA DOG AND TLP WEBSITE OVER THE NEXT

    WEEK TO GET ACCESS. 56

57. ORIENT

58. 58 “BECAUSE OF ‘X’ I THINK ‘Y’ IS HAPPENING” 58

59. 59 59

60. 60 A QUICK WORD ABOUT WHAT CASSANDRA DOES. 60

61. 61 SEND BYTES FROM THE CLIENT… 61

62. 62 …TO THE CO-ORDINATOR… 62

63. 63 … COPY BYTES, SEND THEM TO REPLICAS… 63

64. 64 … WRITE BYTES FROM MEMORY TO DISK. 64

65. 65 “COPY BYTES FROM MEMORY ON ONE MACHINE TO ANOTHER

    AND WRITE TO DISK.” 65

66. 66 AND THE REVERSE FOR READS. 66

67. 67 67 CLIENTS API DYNAMO DATABASE DISK CLIENTS API DYNAMO

    DATABASE DISK Node 1 Node 2

68. 68 (BACK TO) “BECAUSE OF ‘X’ I THINK ‘Y’ IS

    HAPPENING” 68

69. 69 REVIEW SUB SYSTEMS… 69

70. 70 WHAT ARE THE CLIENTS SEEING? 70

71. 71 WHAT IS THE CLUSTER LEVEL THROUGHPUT, LATENCY, ERRORS ETC?

    71

72. 72 WHAT VIEW OF THE CLUSTER DO THE NODES HAVE?

    72

73. 73 HOW IS THE STORAGE ENGINE PERFORMING? 73

74. 74 WHAT IS THE OS AND HARDWARE TELLING CASSANDRA? 74

75. 75 HOW IS THE NETWORK BEHAVING? 75

76. 76 WHAT DECISIONS IS THE CO- ORDINATOR TAKING FOR WRITE

    & READ RESULT? 76

77. 77 (AND ALWAYS) IS THE JVM BEHAVING? 77

78. 78 IS COMPACTION KEEPING UP? 78

79. 79 IS REPAIR RUNNING? 79

80. 80 IS HINTED HANDOFF OR READ REPAIR RUNNING? 80

81. 81 WE CAN ASK LOTS OF QUESTIONS, BUT WE NEED

    SOME STRUCTURE. 81

82. 82 WORK FROM THE CLIENT TO THE DISK. 82

83. 83 BUILD A MODEL OF WHAT EACH SYSTEM IS DOING

    BASED ON EVIDENCE… 83

84. 84 …USE EVIDENCE TO EXPLAIN WHY, DO NOT GUESS. 84

85. DECIDE

86. 86 “BECAUSE ‘Y’ IS HAPPENING WE SHOULD…” 86

87. 87 A QUICK WORD ABOUT HOW CASSANDRA DOES WHAT IT

    DOES. 87

88. 88 CASSANDRA MUST FOLLOW THE CODE. 88

89. 89 OBSERVING CONFIGURATION, METRICS, LOGS, AND TOOLING ALLOWS YOU TO

    UNDERSTAND WHAT CODE IS DOING… 89

90. 90 AND PREDICT WHAT WILL HAPPEN WHEN THEY CHANGE. 90

91. 91 THE BEST GUIDE TO UNDERSTANDING WHAT IS HAPPENING IS

    THE CODE. 91

92. 92 92

93. 93 (BACK TO) “BECAUSE ‘Y’ IS HAPPENING WE SHOULD…” 93

94. 94 RETURN TO OBSERVATION OR TAKE ACTION ‘Z’? 94

95. 95 MAKE MORE OBSERVATIONS WHEN UNDERSTANDING IS UNCLEAR. 95

96. 96 TAKE ACTIONS TO VALIDATE MENTAL MODEL OR FIX PROBLEM.

    96

97. ACT

98. 98 “BECAUSE OF ‘X’ I THINK ‘Y’ IS HAPPENING AND

    SO WILL DO ‘Z’.” 98

99. 99 TEST ASSUMPTIONS BEFORE MAKING CHANGES. 99

100. 100 DOCUMENT STATE OF CLUSTER BEFORE STARTING. 100

101. 101 RECORD WHEN THE CHANGES WERE MADE. 101

102. 102 TEST PREDICTIONS AFTER ACTIONS. LOOP IF NEEDED. 102

103. LOOP

104. 104 104

105. 105 CHECK PREDICTIONS MADE WHEN DECIDING ON THE ACTION. 105

106. 106 RETURN TO OBSERVATIONS AND RESTART LOOP 106

107. 107 MY OODA LOOP… 107

108. 108 1. START WITH GUESS, THEN BISECT THE PROBLEM. 108

109. 109 2. BUILD A MENTAL MODEL THAT HAS TESTABLE PREDICTIONS

     BASED ON OBSERVATIONS. 109

110. 110 3. LOOK FOR EVIDENCE OF PREDICTIONS. 110

111. 111 4. MAKE CHANGES BASED ON EVIDENCE (OR LACK OFF).

     111

112. 112 5. GOTO 1. 112

113. QUESTIONS?