Running Consul at Scale—Journey from RFC to Production

We had many VMs in AWS - were ingesting millions of metrics per second - and were having pain around service discovery and quick configuration changes. This is the story of how we integrated Consul into our environment, what it helped us with, mistakes we made and some tips for successful implementation in your own environment. 10 months later, our growing cluster was using Consul to facilitate 60 second cluster-wide configuration changes and make service discovery simpler and more flexible.

This presentation was given at SREcon16 in Santa Clara, California on April 8, 2016.

More information and a link to a recording of the talk is available here:

https://blog.froese.org/2016/04/08/srecon-running-consul-at-scale/

Transcript

1. C O N S U L @ S R E C O N
   R U N N I N G C O N S U L @ S C A L E
   J O U R N E Y F R O M R F C T O P R O D U C T I O N
   

   View Slide

2. D A R R O N F R O E S E
   D A R R O N @ F R O E S E . O R G - @ D A R R O N
   

   View Slide

3. W H E R E W E R E W E ?
   W I N T E R 2 0 1 4 @ D A TA D O G
   

   View Slide

4. L AT E 2 0 1 4
   • 4 year old codebase.
   • Cutting apart our monolith.
   • Rapid growth across the board.
   • Having config management and service discovery
   pain.
   

   View Slide

5. O L D P R A C T I C E S W E R E F R AY I N G
   W E C O U L D N ’ T D O I T T H E S A M E WA Y A N Y M O R E .
   H T T P : / / J A S O N W I L D E R . C O M / B L O G / 2 0 1 4 / 0 2 / 0 4 / S E R V I C E - D I S C O V E RY- I N - T H E - C L O U D /
   

   View Slide

6. S E R V I C E D I S C O V E RY
   WA S A H Y B R I D
   • Chef searches. 30 minutes to update.
   • Large numbers of manually managed IP addresses.
   • There was nothing really wrong with it - but it was
   getting harder to manage.
   

   View Slide

7. D I S T R I B U T E D S Y S T E M S
   “ M O S Y S T E M S . M O P R O B L E M S . ” - T H E N O T O R I O U S B . I . G .
   

   View Slide

8. M I D 2 0 1 4
   B A C K I N G S T O R E F O R D O C K E R C O N TA I N E R S
   

   View Slide

9. O V E R A L L P L A N
   N O V E M B E R 2 0 1 4
   

   View Slide

10. W H AT I S C O N S U L ?
    

    View Slide

11. R A F T C O N S E N S U S
    H T T P : / / T H E S E C R E T L I V E S O F D A TA . C O M / R A F T /
    

    View Slide

12. C A N I T H E L P D ATA D O G ?
    W E W E R E N ’ T S U R E .
    

    View Slide

13. S TA G I N G
    • ~100 nodes in total.
    • 3 x m3.medium server nodes

    4GB of RAM - 3 ECU - 1 cpu core - SSD drives.
    

    View Slide

14. P H A S E 1 P L A N
    • Initial deploy
    • Small amount of services.
    • Minimal KV usage
    • How will it act?
    • Consul 0.4.1.
    

    View Slide

15. B E F O R E P R O D
    “ M O N I T O R F I R S T ”
    H T T P S : / / B L O G . F R O E S E . O R G / P R E S E N TA T I O N S /
    

    View Slide

16. S H I P I T
    I T ’ S P R O B A B LY F I N E
    

    View Slide

17. D E P L O Y E D T O P R O D
    L A T E D E C E M B E R 2 0 1 4 .
    

    View Slide

18. Y O L O T O T H E M A X
    T H I S WA S N O T
    • Not in the critical path.
    • An outage with Consul could NOT take us down.
    • Our decision to actually depend on Consul would
    come later - when it had proven itself.
    

    View Slide

19. P R O D
    • 5 x m3.large server nodes

    7.5GB of RAM - 6.5 ECU

    2 cpu cores - SSD drives.
    • Rapidly required us to spin up 2 more server nodes
    - it wasn’t stable at 3.
    

    View Slide

20. I T S TA B I L I Z E D
    A N D A L L WA S W E L L
    

    View Slide

21. D ATA D O G S E R V I C E
    O N E O F T H E F I R S T T H I N G S W E A D D E D .
    

    View Slide

22. D ATA D O G S E R V I C E
    

    View Slide

23. D ATA D O G S E R V I C E
    O N E O F T H E F I R S T T H I N G S W E A D D E D .
    

    View Slide

24. C O N S U L E X E C
    T O Y E D W I T H A N D D I S A B L E D
    

    View Slide

25. G I T 2 C O N S U L
    S T R O N G LY C O N S I S T E N T 

    K E Y VA L U E S T O R E A VA I L A B L E 

    O N L O C A L H O S T W I T H A N H T T P Q U E RY.
    H T T P S : / / G I T H U B . C O M / C I M P R E S S - M C P / G I T 2 C O N S U L
    

    View Slide

26. C O N S U L - C O N F I G
    G I T 2 C O N S U L
    

    View Slide

27. G I T 2 C O N S U L + C O N S U L - C O N F I G
    H O W I T W O R K S
    

    View Slide

28. M O R E A N D M O R E U S E
    U P A N D T O T H E R I G H T
    

    View Slide

29. L E A D E R S H I P T R A N S I T I O N S
    P R E T T Y C O M M O N - M O S T LY H A R M L E S S ( I N L O W D O S E S )
    

    View Slide

30. M AY 2 0 1 5
    N O D E S + +
    

    View Slide

31. S E R V I C E R E G I S T R AT I O N
    W E ’ R E G E T T I N G S E R I O U S N O W
    

    View Slide

32. S E R V I C E D I S C O V E RY
    C U R L / H T T P L O O K U P
    

    View Slide

33. S E R V I C E D I S C O V E RY
    D N S L O O K U P
    

    View Slide

34. W O U L D I T F L A P ?
    I N A N D O U T O F T H E S E R V I C E C A TA L O G
    

    View Slide

35. N O . I T D I D N O T F L A P
    I N A N D O U T O F T H E S E R V I C E C A TA L O G
    

    View Slide

36. U S I N G D N S
    W O R R I E D A B O U T S P E E D
    

    View Slide

37. D N S M A S Q
    F R O N T E D C O N S U L’ S D N S R E S O LV E R
    

    View Slide

38. C O N S U L _ D N S _ B A C K U P
    ( T H E H O S T S F I L E )
    C O N S U L - T E M P L A T E
    

    View Slide

39. N O T S U C C E S S F U L
    E V E N I N S TA G I N G
    

    View Slide

40. U S E T H E K V S T O R E T O
    D I S T R I B U T E .
    B U I LT O N T H E S E R V E R N O D E S
    

    View Slide

41. I T W O R K S R E A L LY W E L L
    

    View Slide

42. W I T H O U T R AT E L I M I T I N G
    I T WA S A B I T H A I RY
    

    View Slide

43. N O L E A D E R S H I P
    T R A N S I T I O N S
    N O N E A T A L L
    

    View Slide

44. T H E V E RY N E X T D AY
    “ L E T ’ S C L E A N T H I S U P ”
    

    View Slide

45. R E A D - P R E S S U R E
    C A U S I N G L E A D E R S H I P T R A N S I T I O N S
    

    View Slide

46. C O N S U L I S N E W
    T H E E D G E WA S A L I T T L E B L O O D Y
    T H E R E WA S V E RY L I T T L E
    R E A L W O R L D I N F O R M A T I O N A B O U T I T.
    

    View Slide

47. L O T S O F S M A L L K E Y S
    D O N ’ T D O T H I S - W I T H C O N S U L 0 . 5 . X
    

    View Slide

48. O N C E W E U N D E R S T O O D
    W E U P S I Z E D O U R V M S
    

    View Slide

49. T H I N G S Q U I E T E D D O W N
    I N S TA L L E D L A R G E R S E R V E R N O D E S
    

    View Slide

50. H O W D I D I T W O R K ?
    T H E D N S M A S Q T H I N G …
    

    View Slide

51. D N S M A S Q H O N O R S T H E
    C O N S U L T T L
    A S K S O N C E E V E RY 1 0 S E C O N D S
    

    View Slide

52. I T R E S P O N D S Q U I C K LY
    W E L L
    

    View Slide

53. C L U S T E R W I D E S TAT S
    D N S M A S Q A N D T H E M A G I C A L H O S T S F I L E
    H T T P S : / / G I T H U B . C O M / D A R R O N / G O S H E
    

    View Slide

54. D U R I N G A FA I L O V E R
    

    View Slide

55. P E O P L E W E R E A B I T
    S C A R E D
    W E R E N ’ T S U R E I F
    W E W E R E G O I N G T O C O N T I N U E
    

    View Slide

56. N O D E S W E R E G O I N G
    D E A F
    T H E Y D R O P P E D O U T O F T H E C A TA L O G
    

    View Slide

57. B U T I C O U L D F I N A L LY
    D U P L I C AT E T H E
    P R O B L E M
    I WA S V I S I T I N G T H E
    O F F I C E & H E A R D S O M E G R U M B L I N G
    

    View Slide

58. H A S H I C O R P L E N T A H A N D
    H U G E T H A N K S T O J A M E S A N D A R M O N F O R A L L T H E I R H E L P !
    

    View Slide

59. 2 D E A D L O C K S F I X E D
    B U T T H E K E Y WA S Q U A K E R E L A T E D .
    

    View Slide

60. A N D A L L WA S R I G H T
    F O R T H E M O S T PA R T
    

    View Slide

61. O C T O B E R 2 0 1 5
    N O D E S + + + - M O S T LY S TA B L E - B U T C A U T I O U S
    

    View Slide

62. C O N S U L - C O N F I G
    H E L P E D A S W E G R E W
    M A D E R E T I R I N G &
    S WA P P I N G S O M E S E R V I C E S E A S I E R
    

    View Slide

63. S TA R T E D U S I N G E V E N T S
    L I K E C O N S U L E X E C B U T W I T H S E C U R I T Y + +
    

    View Slide

64. C O N S U L L O C K
    R U N 1 O F N P R O C E S S E S - W I T H H O T S PA R E S
    R U N N I N G
    WA I T I N G
    WA I T I N G
    

    View Slide

65. C O N S U L L O C K
    R U N 1 O F N P R O C E S S E S - W I T H H O T S PA R E S
    C R A S H E S
    R U N N I N G
    WA I T I N G
    X
    

    View Slide

66. C O N S U L L O C K
    R U N 1 O F N P R O C E S S E S - W I T H H O T S PA R E S
    

    View Slide

67. W H E N T H E L E A D E R S H I P
    T R A N S I T I O N S G R E W
    S U P E R S I Z E D O U R S E R V E R S
    

    View Slide

68. C 3 . 2 X L A R G E D I D T H E T R I C K
    A T 1 0 0 0 + N O D E S - I T ’ S L I K E W E ( A L M O S T ) T U R N E D T H E M O F F.
    

    View Slide

69. 2 S M A L L O U TA G E S
    L A S T Y E A R .
    O N E F O R 3 M I N U T E S B E C A U S E O F A PA C K A G I N G P R O B L E M
    O N E F O R A N H O U R T H A T WA S D O C U M E N TA T I O N A N D
    “ B R O A D C A S T I N P U T T O A L L PA N E S ” R E L A T E D
    

    View Slide

70. W H AT S H O U L D Y O U K N O W ?
    W H A T H A V E W E L E A R N E D ?
    

    View Slide

71. C O N S U L I S A W E S O M E
    I T ’ S Y O U R D A TA C E N T E R ’ S B A C K B O N E
    

    View Slide

72. M O N I T O R I N G I S E S S E N T I A L
    J U S T D O I T
    

    View Slide

73. U P G R A D E T O 0 . 6 . X
    T O N S O F F I X E S A N D U P G R A D E S
    

    View Slide

74. U S E S L E S S M E M O RY
    TA S T E S G R E A T, L E S S F I L L I N G !
    

    View Slide

75. C O N S U L L O V E S C P U
    F E E D I T A L L T H E C P U S
    

    View Slide

76. S O M E E X A M P L E S I Z I N G
    • m3.large ~300 nodes
    • c3.xlarge ~500 nodes
    • c3.2xlarge ~800 nodes
    • As always YMMV.
    • 0.6 is more efficient - might be able to use smaller
    nodes.
    

    View Slide

77. E M B R A C E FA I L U R E
    B U I L D F O R I T - A D D R E T R I E S - B A C K O F F - C I R C U I T B R E A K E R S
    M A K E S Y O U R W H O L E S Y S T E M M O R E R E S I L I E N T
    

    View Slide

78. WAT C H Y O U R R E A D
    V E L O C I T Y
    D O N ’ T D D O S Y O U R S E L F
    

    View Slide

79. U S E F E W E R A N D
    L A R G E R K E Y S
    R A T H E R T H A N L O T S O F S M A L L K E Y S
    E S P E C I A L LY I F Y O U ’ R E R E A D I N G
    A L O T O F T H E M A T O N C E
    

    View Slide

80. L O C K D O W N PA R T S O F
    T H E K V S T O R E
    F E E D I N D A TA F R O M T H E O U T S I D E
    A C L S A R E Y O U R F R I E N D
    

    View Slide

81. C O N S U L WAT C H E S
    A R E P O W E R F U L
    M A K E S U R E T H E Y O N LY F I R E W H E N Y O U WA N T T H E M T O
    H T T P S : / / G I T H U B . C O M / D A R R O N / S I F T E R
    

    View Slide

82. I F O U T P U T I S N ’ T
    U N I Q U E
    D O N ’ T B U I L D C O N F I G O N E V E RY N O D E
    U S E T H E K V S T O R E T O M O V E T H O S E F I L E S A R O U N D
    

    View Slide

83. O N E M O R E T H I N G
    T H A T ’ S M Y L A S T T I P F O R T O D A Y B U T W E H A V E
    

    View Slide

84. K V E X P R E S S
    U S E T H E K V S T O R E T O T R A N S P O R T
    C O N F I G U R A T I O N F I L E S
    I N & O U T
    B O T H D I R E C T I O N S
    

    View Slide

85. M A I N F E AT U R E S
    • 10MB Go binary
    • Uploads and downloads files under 512KB
    • Emits Dogstatsd metrics and Datadog Events
    • Files sent == files delivered
    • Doesn’t re-upload or re-deliver
    • Very safe
    • Runs commands after delivery
    

    View Slide

86. I T ’ S S U P E R FA S T
    < 5 0 0 M S T O D E L I V E R A F I L E T O 1 0 0 0 N O D E S
    

    View Slide

87. S P E E D I N A N D O U T
    

    View Slide

88. C A N P O S T D I F F S
    W H E N F I L E S U P D A T E
    

    View Slide

89. M E T R I C S F O R A L L
    M E A S U R E A L L T H E T H I N G S
    

    View Slide

90. H T T P S : / / G I T H U B . C O M /
    D ATA D O G / K V E X P R E S S
    

    View Slide

91. S T O P.
    D E M O T I M E .
    

    View Slide

92. Q U E S T I O N S ?
    

    View Slide

93. T H A N K S !
    D A R R O N @ F R O E S E . O R G
    @ D A R R O N
    G I T H U B . C O M / D A R R O N
    C O N S U L @ S R E C O N
    R U N N I N G C O N S U L @ S C A L E
    J O U R N E Y F R O M R F C T O P R O D U C T I O N
    

    View Slide