Distributed Systems

Lightning Talk on Distributed Systems at Gocon 2014 Autumn (Tokyo, Japan)

Source of the presentation at http://github.com/imkira/gocon/tree/master/2014

Transcript

1. Distributed Systems Lightning Talk @ gocon2014-autumn 30 November 2014 Mario Freitas CyberAgent, Inc.
2. github.com/imkira/gocon
3. About me Full-Stack Engineer Japan ♥ (> 6 years) Golang ♥ (> 4 months) Distributed Systems ♥ (> 1 years)
6. consul.io http://consul.io By HashiCorp (Vagrant, Packer, Serf, Terraform) go get github.com/armon/consul-api
9. Discovering Services f u n c l i s t N e w ( c l i e n t * c o n s u l a p i . C l i e n t , o p t s * c o n s u l a p i . Q u e r y O p t i o n s ) { e n t r i e s , m e t a , e r r : = c l i e n t . H e a l t h ( ) . S e r v i c e ( " c a n i h a z c l u s t e r s " , " " , f a l s e , o p t s ) i f e r r ! = n i l { r e t u r n } i f o p t s . W a i t I n d e x ! = m e t a . L a s t I n d e x { o p t s . W a i t I n d e x = m e t a . L a s t I n d e x f m t . P r i n t f ( " % d s e r v i c e s \ n " , l e n ( e n t r i e s ) ) f o r _ , e : = r a n g e e n t r i e s { p r i n t S e r v i c e ( e ) } } } f u n c p r i n t S e r v i c e ( e * c o n s u l a p i . S e r v i c e E n t r y ) { f o r _ , c : = r a n g e e . C h e c k s { i f c . S e r v i c e N a m e = = " c a n i h a z c l u s t e r s " { f m t . P r i n t f ( " % s % s : % d = % s \ n " , e . S e r v i c e . I D , e . N o d e . A d d r e s s , e . S e r v i c e . P o r t , c . S t a t u s ) r e t u r n } } } Run
10. Detecting Failures f u n c n o d e H e a l t h y ( e * c o n s u l a p i . S e r v i c e E n t r y ) b o o l { f o r _ , c : = r a n g e e . C h e c k s { i f c . S t a t u s ! = " p a s s i n g " { r e t u r n f a l s e } } r e t u r n l e n ( e . C h e c k s ) > 0 } f u n c s e r v i c e H e a l t h y ( e * c o n s u l a p i . S e r v i c e E n t r y , n a m e s t r i n g ) b o o l { f o u n d : = f a l s e f o r _ , c : = r a n g e e . C h e c k s { i f c . S e r v i c e N a m e = = n a m e { f o u n d = t r u e i f c . S t a t u s ! = " p a s s i n g " { r e t u r n f a l s e } } } r e t u r n f o u n d }
11. Configuring Services f u n c g e t ( c l i e n t * c o n s u l a p i . C l i e n t ) ( [ ] b y t e , e r r o r ) { p a i r , _ , e r r : = c l i e n t . K V ( ) . G e t ( " c o n f i g " , n i l ) i f e r r ! = n i l | | p a i r = = n i l { r e t u r n n i l , e r r } r e t u r n p a i r . V a l u e , n i l } f u n c p u t ( c l i e n t * c o n s u l a p i . C l i e n t , b [ ] b y t e ) e r r o r { p a i r : = & c o n s u l a p i . K V P a i r { K e y : " c o n f i g " , V a l u e : b } _ , e r r : = c l i e n t . K V ( ) . P u t ( p a i r , n i l ) r e t u r n e r r } Run
12. Coordinating (eg. Distributed Locks, Transactions) f u n c r u n T r a n s a c t i o n ( c l i e n t * c o n s u l a p i . C l i e n t , n o d e I D s t r i n g ) { s e : = & c o n s u l a p i . S e s s i o n E n t r y { N a m e : " k e y L o c k " , C h e c k s : [ ] s t r i n g { " s e r f H e a l t h " , " s e r v i c e : " + n o d e I D } , } s e s s i o n I D , _ , e r r : = c l i e n t . S e s s i o n ( ) . C r e a t e ( s e , n i l ) i f e r r ! = n i l { r e t u r n } d e f e r c l i e n t . S e s s i o n ( ) . D e s t r o y ( s e s s i o n I D , n i l ) f o r { p : = & c o n s u l a p i . K V P a i r { K e y : " l o c k e d K e y " , V a l u e : n i l , S e s s i o n : s e s s i o n I D } i f o k , _ , e r r : = c l i e n t . K V ( ) . A c q u i r e ( p , n i l ) ; e r r ! = n i l | | ! o k { t i m e . S l e e p ( 1 * t i m e . M i l l i s e c o n d ) c o n t i n u e } d e f e r c l i e n t . K V ( ) . R e l e a s e ( p , n i l ) s l o w T r a n s a c t i o n ( n o d e I D ) r e t u r n } } Run
13. Leader Election Similar to the transaction example: Leader acquires lock Leader does something Leader releases lock Watch key-value for changes (leaders and followers!): Leader steps down Leader fails (including false positives)
14. Summary consul io: Simple but very powerful! Makes distributed systems development easy! Very fun to work with! Unfortunately, it is still a bit "edge" :'( But future looks promising ;)
15. Thank you Mario Freitas CyberAgent, Inc. imkira@gmail.com (mailto:imkira@gmail.com) http://github.com/imkira (http://github.com/imkira) present http://github.com/imkira/gocon (http://github.com/imkira/gocon)