How to ride a whale

Step by step functionnal coreos cluster

Transcript

1. HOW TO RIDE A WHALE Creating a functionnal CoreOS cluster
2. THE SERVICE Based on systemd Unit or template files with basic variables
3. A MONGODB DATABASE units template : mongodb@.service [ U n i t ] D e s c r i p t i o n = M o n g o D B A f t e r = d o c k e r . s e r v i c e [ S e r v i c e ] E x e c S t a r t P r e = / u s r / b i n / d o c k e r p u l l m o n g o : l a t e s t E x e c S t a r t P r e = - / u s r / b i n / d o c k e r r m - f - v % p . % i E x e c S t a r t = / u s r / b i n / d o c k e r r u n - - r m - - n a m e % p . % i - p 2 7 0 1 7 : 2 7 0 1 7 \ - - v o l u m e = / h o m e / d a t a / m o n g o / l a t e s t : / d a t a / d b - - c p u - s h a r e s = 4 - m = " 3 0 g " m o n g o : l a t e s t E x e c S t a r t P o s t = / b i n / e t c d c t l s e t / s k y d n s / f r / m y c l o u d / % p / % i ' { " h o s t " : " % H " , " p o r t " E x e c S t a r t P o s t = / b i n / e t c d c t l s e t / d a t a b a s e s / % p / % i ' { " h o s t " : " % H " , " p o r t " : 2 7 0 1 7 E x e c S t o p = / b i n / e t c d c t l r m / s k y d n s / f r / m y c l o u d / % p / % i E x e c S t o p = / b i n / e t c d c t l r m / d a t a b a s e s / % p . % i E x e c S t o p = - / u s r / b i n / d o c k e r s t o p % p . % i R e s t a r t = a l w a y s
4. FLEET Remote control Basic scheduler Make services survive to machine crash
5. Simple command line f l e e t c t l s u b m i t m o n g o d b @ . s e r v i c e f l e e t c t l s t a r t m o n g o d b @ x 1 f l e e t c t l s t a r t m o n g o d b @ x 2 f l e e t c t l s t a r t m o n g o d b @ x 3 f l e e t c t l l i s t - u n i t s f l e e t c t l j o u r n a l - f m o n g o d b @ x 1 c o r e o s : f l e e t : p u b l i c - i p : $ p u b l i c _ i p v 4 m e t a d a t a : d i s k = s s d , h o s t e r = o v h , l o c a t i o n = F R
6. ETCD Cluster registry Raft consensus implementation Use etcd2 http://thesecretlivesofdata.com/raft/
7. Access anywhere # U s e i t v i a c o m m a n d l i n e > e t c d c t l s e t / s k y d n s / f r / m y c l o u d / m o n g o d b / x 1 { " h o s t " : " c o r e - 1 " , " p o r t " : 2 7 0 1 7 # O r H T T P > c u r l - L h t t p : / / 1 2 7 . 0 . 0 . 1 : 4 0 0 1 / v 2 / k e y s / s k y d n s / f r / m y c l o u d / m o n g o d b / x 1 { " h o s t " : " c o r e - 1 " , " p o r t " : 2 7 0 1 7 } c o r e o s : e t c d : # g e n e r a t e a n e w t o k e n f o r e a c h u n i q u e c l u s t e r f r o m h t t p s : / / d i s c o v e r y . e t c d . i o d i s c o v e r y : h t t p s : / / d i s c o v e r y . e t c d . i o / { t o k e n } a d d r : $ p u b l i c _ i p v 4 : 4 0 0 1 p e e r - a d d r : $ p u b l i c _ i p v 4 : 7 0 0 1 Or etcd-browser for human beings
9. SKYDNS Nameserver (@95%) Store and read data from etcd (or consul) Simplest DNS to administrate ever
10. d i g * . m o n g o d b . m y c l o u d . f r S R V @ l o c a l h o s t [ U n i t ] D e s c r i p t i o n = S k y D N S A f t e r = d o c k e r . s e r v i c e [ S e r v i c e ] E x e c S t a r t P r e = - / u s r / b i n / d o c k e r r m - f s k y d n s E x e c S t a r t = / u s r / b i n / d o c k e r r u n - - r m - - n a m e s k y d n s \ - e E T C D _ M A C H I N E S = h t t p : / / 1 2 7 . 0 . 0 . 1 : 4 0 0 1 \ - e S K Y D N S _ A D D R = 0 . 0 . 0 . 0 : 5 3 \ - e S K Y D N S _ D O M A I N = m y c l o u d . f r \ - e S K Y D N S _ N A M E S E R V E R S = 8 . 8 . 8 . 8 : 5 3 , 8 . 8 . 4 . 4 : 5 3 \ - - n e t = h o s t \ s k y n e t s e r v i c e s / s k y d n s : l a t e s t E x e c S t o p = - / u s r / b i n / d o c k e r s t o p s k y d n s R e s t a r t = a l w a y s
11. CONFD Read etcd and write config files Basic go template check, and reload services
12. [ m o n g o d b ] p o r t = 2 7 0 1 7 { { r a n g e g e t s " / d a t a b a s e s / m o n g o d b / * " } } { { $ s e r v e r : = j s o n . V a l u e s e r v e r s [ ] = { { $ s e r v e r . h o s t } } { { e n d } } [ U n i t ] D e s c r i p t i o n = C o n f d W a n t s = e t c d . s e r v i c e [ S e r v i c e ] E x e c S t a r t P r e = - / u s r / b i n / m k d i r - p / o p t / b i n E x e c S t a r t P r e = - / u s r / b i n / w g e t - N - P / o p t / b i n h t t p s : / / g i t h u b . c o m / k e l s e y h i g h t o w e r E x e c S t a r t P r e = - / b i n / r m / o p t / b i n / c o n f d E x e c S t a r t P r e = - / b i n / m v / o p t / b i n / c o n f d - 0 . 9 . 0 - l i n u x - a m d 6 4 / o p t / b i n / c o n f d E x e c S t a r t P r e = - / b i n / c h m o d + x / o p t / b i n / c o n f d E x e c S t a r t = / o p t / b i n / c o n f d - b a c k e n d e t c d - n o d e 1 2 7 . 0 . 0 . 1 : 4 0 0 1 - i n t e r v a l = 1 0 E x e c S t o p = - / b i n / k i l l $ ( p i d o f c o n f d ) R e s t a r t = a l w a y s
13. HAPROXY TCP proxy, load balancing, health check and mitigation since 1996
15. h t t p : / / m y c l o u d . f r : 1 0 0 0 e t c d c t l s e t / s e r v i c e s / m y w o r d p r e s s . f r / s c h e m e h t t p e t c d c t l s e t / s e r v i c e s / m y w o r d p r e s s . f r / h o s t s / 1 1 . 2 . 3 . 4 : 8 0 e t c d c t l s e t / s e r v i c e s / m y w o r d p r e s s . f r / h o s t s / 1 1 . 2 . 3 . 5 : 8 0 [ U n i t ] D e s c r i p t i o n = H A p r o x y l o a d b a l a n c e r A f t e r = d o c k e r . s e r v i c e [ S e r v i c e ] E x e c S t a r t P r e = - / u s r / b i n / d o c k e r r m - f b a l a n c e r E x e c S t a r t = / u s r / b i n / d o c k e r r u n - - r m - - n a m e h a p r o x y \ - e E T C D _ N O D E = 1 2 7 . 0 . 0 . 1 : 4 0 0 1 - - v o l u m e / e t c / c e r t s : / e t c / c e r t s / \ - - n e t h o s t \ c s t p d k / h a p r o x y - c o n f d E x e c R e l o a d = / u s r / b i n / d o c k e r e x e c b a l a n c e r s e r v i c e h a p r o x y r e l o a d E x e c S t o p = / u s r / b i n / d o c k e r s t o p b a l a n c e r R e s t a r t = a l w a y s
16. SYNCTHING Torrent based private cloud Neat web interface
17. [ U n i t ] D e s c r i p t i o n = S y n c t h i n g [ S e r v i c e ] E x e c S t a r t P r e = - / u s r / b i n / d o c k e r r m - f - v s y n c t h i n g E x e c S t a r t P r e = - / u s r / b i n / d o c k e r p u l l i s t e p a n o v / s y n c t h i n g : l a t e s t E x e c S t a r t = / u s r / b i n / d o c k e r r u n - - r m - - n a m e s y n c t h i n g \ - p 9 0 8 0 : 8 0 8 0 - p 2 2 0 0 0 : 2 2 0 0 0 - p 2 1 0 2 5 : 2 1 0 2 5 / u d p \ - v / e t c / s y n c t h i n g : / h o m e / s y n c t h i n g / . c o n f i g / s y n c t h i n g \ - v / h o m e / d a t a / s y n c : / h o m e / s y n c t h i n g / S y n c \ i s t e p a n o v / s y n c t h i n g E x e c S t o p = - / u s r / b i n / d o c k e r s t o p s y n c t h i n g R e s t a r t = a l w a y s
18. DATADOG plug everything in easily StatsD Graphite on vitamins Monitoring and Alerting Or cadvisor+heapster+fluentd or prometheus
19. [ U n i t ] D e s c r i p t i o n = D a t a d o g m o n i t o r i n g [ S e r v i c e ] E x e c S t a r t P r e = - / u s r / b i n / d o c k e r r m - f d a t a d o g E x e c S t a r t P r e = - / u s r / b i n / d o c k e r p u l l d a t a d o g / d o c k e r - d d - a g e n t E x e c S t a r t = / u s r / b i n / d o c k e r r u n - - r m - - n a m e d a t a d o g \ - v / e t c / d a t a d o g / c o n f . d : / e t c / d d - a g e n t / c o n f . d \ - v / v a r / r u n / d o c k e r . s o c k : / v a r / r u n / d o c k e r . s o c k \ - v / p r o c / m o u n t s : / h o s t / p r o c / m o u n t s : r o \ - v / s y s / f s / c g r o u p / : / h o s t / s y s / f s / c g r o u p : r o \ - e A P I _ K E Y = G E T - I T - O N - D A T A D O G \ - - n e t = h o s t \ d a t a d o g / d o c k e r - d d - a g e n t E x e c S t o p = - / u s r / b i n / d o c k e r s t o p d a t a d o g R e s t a r t = a l w a y s
20. CLOUD INIT BareMetal (ovh, online) Virtual machines (Ganeti, Openstack) Cloud providers (AWS, DO, GC ...)
21. # c l o u d - c o n f i g c o r e o s : u p d a t e : r e b o o t - s t r a t e g y : b e s t - e f f o r t g r o u p : s t a b l e f l e e t : p u b l i c - i p : $ p u b l i c _ i p v 4 u n i t s : - n a m e : h a p r o x y . s e r v i c e c o m m a n d : s t a r t c o n t e n t : | [ U n i t ] . . . . w r i t e _ f i l e s : - p a t h : / e t c / r e s o l v . c o n f o w n e r : r o o t : r o o t c o n t e n t : |
22. QUESTIONS ? g i t c l o n e h t t p s : / / g i t h u b . c o m / V i n c e v e v e / a c h a b c d a c h a b v a g r a n t u p https://coreos.com/blog/managing-coreos-with-ansible/ http://www.freedesktop.org/software/systemd/man/systemd.service.html https://github.com/kelseyhightower/confd/blob/master/docs/templates.md http://cbonte.github.io/haproxy-dconv/configuration-1.5.html https://www.digitalocean.com/community/tutorials/how-to-set-up-a-coreos-cluster-on- digitalocean