Graceful Upgrade for Go App

Go Conference 2015 Winter

Transcript

1. 2015/12/6 Graceful Upgrade for Go App http://localhost:3999/main.slide#1 1/17 Graceful Upgrade for Go App Go Conference 2015 Winter Yoshiki Nakagawa Infrastructure Engineer, pairs Division eureka, Inc.
2. 2015/12/6 Graceful Upgrade for Go App http://localhost:3999/main.slide#1 2/17 Agenda Deploy Go App without Downtime Graceful Upgrade Goji running under Einhorn facebookgo/grace facebookgo/httpdown combined with Einhorn Summary
3. 2015/12/6 Graceful Upgrade for Go App http://localhost:3999/main.slide#1 3/17 Deploy Go App without Downtime ▶ Blue/Green Deployment e.g. swap Auto Scaling Groups Zero Downtime :) But this is not Today's theme ...
4. 2015/12/6 Graceful Upgrade for Go App http://localhost:3999/main.slide#1 4/17 Deploy Go App without Downtime ▶ Hot Deployment Today's theme ! Upgrade app without dropping any requests :) Gopher by Renée French (http://www.reneefrench.com)
5. 2015/12/6 Graceful Upgrade for Go App http://localhost:3999/main.slide#1 5/17 Graceful Upgrade Other requirements Daemonize Go Server (using daemontools, supervisor, etc)
6. 2015/12/6 Graceful Upgrade for Go App http://localhost:3999/main.slide#1 6/17 Good example : Goji running under Einhorn
7. 2015/12/6 Graceful Upgrade for Go App http://localhost:3999/main.slide#1 7/17 Goji the minimalistic web framework that values composability and simplicity ▪ Features ... Automatic support for Einhorn (https://github.com/stripe/einhorn) , systemd, and more Graceful shutdown, and zero-downtime graceful reload when combined with Einhorn (https://github.com/stripe/einhorn) # h t t p s : / / g i t h u b . c o m / z e n a z n / g o j i / b l o b / m a s t e r / b i n d / b i n d . g o I f a n o p t i o n i s n o t e x p l i c i t l y p a s s e d , t h e i m p l e m e n t a t i o n w i l l a u t o m a t i c a l l y s e l e c t b e t w e e n u s i n g " e i n h o r n @ 0 " , " f d @ 3 " , a n d " : 8 0 0 0 " , d e p e n d i n g o n w h e t h e r E i n h o r n o r s y s t e m d ( o r n e i t h e r ) i s d e t e c t e d . What is Einhorn (https://github.com/stripe/einhorn) ? > tips. Einhorn is the German word for Unicorn. the language-independent shared socket manager
8. 2015/12/6 Graceful Upgrade for Go App http://localhost:3999/main.slide#1 8/17 Goji running under Einhorn Without changing application code ! ▶ Daemonize # s u p e r v i s o r d . c o n f [ p r o g r a m : g o j i ] c o m m a n d = / u s r / l o c a l / b i n / e i n h o r n - c g o j i - b 1 2 7 . 0 . 0 . 1 : 1 2 3 4 - m m a n u a l e x a m p l e - b i n d e i n h o r n @ 0 Einhorn option What kinds of ACK to expect from workers. - m m a n u a l = > p r o c e s s w i l l s p e a k t o c o m m a n d s o c k e t w h e n r e a d y Goji option A convenient way to bind to sockets. - b i n d e i n h o r n @ 0 = > b i n d i n g t o E i n h o r n s o c k e t s ( i n c l u d i n g E i n h o r n A C K s u p p o r t )
9. 2015/12/6 Graceful Upgrade for Go App http://localhost:3999/main.slide#1 9/17 [DEMO] Goji running under Einhorn # # # S t a r t s e r v e r # # # [ M A S T E R 5 7 ] I N F O : L a u n c h i n g 1 n e w w o r k e r s [ M A S T E R 5 7 ] I N F O : = = = > L a u n c h e d 7 7 ( i n d e x : 0 ) 2 0 1 5 / 1 2 / 0 5 2 2 : 2 6 : 0 8 S t a r t i n g G o j i o n 1 2 7 . 0 . 0 . 1 : 1 2 3 4 2 0 1 5 / 1 2 / 0 5 2 2 : 2 6 : 0 8 b i n d : A C K i n g t o e i n h o r n . . . # # # G r a c e f u l U p g r a d e # # # $ / u s r / l o c a l / b i n / e i n h o r n s h - c g o j i - e u p g r a d e [ M A S T E R 5 7 ] I N F O : R e l o a d i n g e i n h o r n m a s t e r . . R e l o a d n e w b i n a r y ( M A S T E R p i d : 5 7 ) [ M A S T E R 5 7 ] I N F O : S t a r t i n g s m o o t h u p g r a d e f r o m v e r s i o n 0 . . . [ M A S T E R 5 7 ] I N F O : = = = > L a u n c h e d 9 3 ( i n d e x : 1 ) 2 0 1 5 / 1 2 / 0 5 2 2 : 4 2 : 1 9 . 5 9 8 6 1 0 S t a r t i n g G o j i o n 1 2 7 . 0 . 0 . 1 : 1 2 3 4 N e w w o r k e r ( p i d : 9 3 ) 2 0 1 5 / 1 2 / 0 5 2 2 : 4 2 : 1 9 . 5 9 8 6 2 5 b i n d : A C K i n g t o e i n h o r n [ M A S T E R 5 7 ] I N F O : R e c e i v e d a m a n u a l A C K f r o m 9 3 [ M A S T E R 5 7 ] I N F O : U p t o 1 / 1 m a n u a l A C K s [ M A S T E R 5 7 ] I N F O : K i l l i n g o f f 1 o l d w o r k e r s . [ M A S T E R 5 7 ] I N F O : S e n d i n g U S R 2 t o [ 7 7 ] 2 0 1 5 / 1 2 / 0 5 2 2 : 4 2 : 1 9 G o j i r e c e i v e d s i g n a l , g r a c e f u l l y s t o p p i n g G r a c e f u l S h u t d o w n ( p i d : 7 7 ) 2 0 1 5 / 1 2 / 0 5 2 2 : 4 2 : 1 9 G o j i s t o p p e d [ M A S T E R 5 7 ] I N F O : = = = > E x i t e d w o r k e r 7 7 Good :)
10. 2015/12/6 Graceful Upgrade for Go App http://localhost:3999/main.slide#1 10/17 facebookgo/grace
11. 2015/12/6 Graceful Upgrade for Go App http://localhost:3999/main.slide#1 11/17 facebookgo/grace Package grace (https://github.com/facebookgo/grace) provides a library that makes it easy to build socket based servers that can be gracefully terminated & restarted (that is, without dropping any connections). $ $ G O P A T H / b i n / g r a c e d e m o 2 0 1 5 / 1 2 / 0 6 0 8 : 2 9 : 4 8 S e r v i n g [ : : ] : 4 8 5 6 7 w i t h p i d 2 4 0 0 . . . $ s u d o l s o f - i | g r e p g r a c e d e m o g r a c e d e m o 2 4 0 0 y y o s h i k i 4 1 4 u I P v 6 0 x f 8 6 a 6 7 e f b 2 9 d 4 4 5 0 t 0 T C P * : 4 8 5 6 7 ( L I S T E N ) # m a s t e r p i d : 2 4 0 0 # t r i g g e r a g r a c e f u l s e r v e r r e s t a r t $ k i l l - U S R 2 2 4 0 0 $ s u d o l s o f - i | g r e p g r a c e d e m o g r a c e d e m o 2 4 6 4 y y o s h i k i 4 1 4 u I P v 6 0 x f 8 6 a 6 7 e f b 2 9 d 4 4 5 0 t 0 T C P * : 4 8 5 6 7 ( L I S T E N ) # m a s t e r p i d : 2 4 6 4 Master pid is changed. Not suitable for background daemon processes :(
12. 2015/12/6 Graceful Upgrade for Go App http://localhost:3999/main.slide#1 12/17 facebookgo/httpdown combined with Einhorn
13. 2015/12/6 Graceful Upgrade for Go App http://localhost:3999/main.slide#1 13/17 facebookgo/httpdown combined with Einhorn Package httpdown (https://github.com/facebookgo/httpdown) provides a library that makes it easy to build a HTTP server that can be shutdown gracefully (that is, without dropping any connections). grace (https://github.com/facebookgo/grace) package uses this package underneath. ▶ Need Implements facebookgo/httpdown 1. Communicates with the einhorn master from workers (e.g. send ACK). # U s e p a c k a g e g o - e i n h o r n i m p o r t g i t h u b . c o m / s t r i p e / g o - e i n h o r n 2. Be able to receive SIGUSR2 signal (from the einhorn master and then graceful shutdown). # A d d S I G U S R 2 s i g n a l s i g n a l . N o t i f y ( . . . , s y s c a l l . S I G U S R 2 )
14. 2015/12/6 Graceful Upgrade for Go App http://localhost:3999/main.slide#1 14/17 [DEMO] facebookgo/httpdown combined with Einhorn # # # S t a r t s e r v e r # # # $ e i n h o r n - b 1 2 7 . 0 . 0 . 1 : 8 0 8 0 - m m a n u a l - c h t t p d o w n - e i n h o r n h t t p d o w n _ e x a m p l e [ M A S T E R 8 9 ] I N F O : B i n d i n g t o 1 2 7 . 0 . 0 . 1 : 8 0 8 0 w i t h f l a g s [ ] [ M A S T E R 8 9 ] I N F O : L a u n c h i n g 1 n e w w o r k e r s [ M A S T E R 8 9 ] I N F O : = = = > L a u n c h e d 9 3 ( i n d e x : 0 ) . . . # # # G r a c e f u l U p g r a d e # # # $ e i n h o r n s h - c h t t p d o w n - e i n h o r n - e u p g r a d e [ M A S T E R 8 9 ] I N F O : R e l o a d i n g e i n h o r n m a s t e r . . R e l o a d n e w b i n a r y ( M A S T E R p i d : 8 9 ) [ M A S T E R 8 9 ] I N F O : S t a r t i n g s m o o t h u p g r a d e f r o m v e r s i o n 0 . . . [ M A S T E R 8 9 ] I N F O : = = = > L a u n c h e d 9 9 ( i n d e x : 1 ) 2 0 1 5 / 1 2 / 0 6 0 9 : 5 4 : 2 3 s e r v i n g [ : : ] : 8 0 8 0 w i t h p i d 9 9 N e w w o r k e r ( p i d : 9 9 ) [ M A S T E R 8 9 ] I N F O : R e c e i v e d a m a n u a l A C K f r o m 9 9 [ M A S T E R 8 9 ] I N F O : U p t o 1 / 1 m a n u a l A C K s [ M A S T E R 8 9 ] I N F O : K i l l i n g o f f 1 o l d w o r k e r s . [ M A S T E R 8 9 ] I N F O : S e n d i n g U S R 2 t o [ 9 3 ] 2 0 1 5 / 1 2 / 0 6 0 9 : 5 6 : 5 7 s i g n a l r e c e i v e d : u s e r d e f i n e d s i g n a l 2 G r a c e f u l S h u t d o w n ( p i d : 9 3 ) 2 0 1 5 / 1 2 / 0 6 0 9 : 5 6 : 5 7 e x i t i n g [ M A S T E R 8 9 ] I N F O : = = = > E x i t e d w o r k e r 9 3 Good :)
15. 2015/12/6 Graceful Upgrade for Go App http://localhost:3999/main.slide#1 15/17 Summary ▶ Graceful Updates with Einhorn Requirements (too much..) Graceful Shutdown (e.g. facebookgo/httpdown) Communicates with the einhorn master from workers (e.g. stripe/go-einhorn) Receive SIGUSR2 signal (from the einhorn master and then graceful shutdown). ▶ Other Server::Starter lestrrat/go-server-starter (https://github.com/lestrrat/go-server-starter) facebookgo/grace (https://github.com/facebookgo/grace) more....
16. 2015/12/6 Graceful Upgrade for Go App http://localhost:3999/main.slide#1 16/17 Thank you Yoshiki Nakagawa Infrastructure Engineer, pairs Division eureka, Inc. https://github.com/yyoshiki41 (https://github.com/yyoshiki41) (#ZgotmplZ) esola.co (http://esola.co)
17. 2015/12/6 Graceful Upgrade for Go App http://localhost:3999/main.slide#1 17/17