MODELLING INFRASTRUCTURE WITH INVENTORY DATA HOW DOES ANSIBLE MODEL ITS INVENTORY AND HOW CAN WE MODEL OUR INFRASTRUCTURE?

$ WHOAMI {'name': 'Serge van Ginderachter'} {'url': 'http://ginsys.eu'} {'email': '[email protected]'} {'IM': '[email protected]'} {'twitter': '@svg'} {'IRC': 'svg'} {'Github': 'sergevanginderachter'}

$WORK INFRASTRUCTURE "CONSULTANT" Serge van Ginderachter Started in M$ shops in Small Business Environments ( at Belgian scale: 5-150 employees ) Followed the Jedi path and turned to the bright side of life after 2005 Ansible since +/- 18 months No experience with other cfgmgmt tools First upstream commit c o m m i t d a 9 2 c e 7 9 6 b 4 8 e c 8 0 e 3 e a d 1 c f e 9 b c b c 7 1 f 5 f c e 8 0 5 A u t h o r S e r g e v a n G i n d e r a c h t e r D a t e W e d O c t 1 0 1 9 : 3 8 : 3 0 2 0 1 2 + 0 2 0 0 f i x m i s s i n g - - l i m i t i n d o c s s i t e e x a m p l e s

CURRENT MAJOR PROJECT Flemish Government, one of the complex parts of Belgium http://www.milieuinfo.be: about environmental things Open Source island within the Flemish Government Enterprise upcoming stuff: "cloud" built on Ceph storage and Cloudstack Ansible since Summer 2012

ANSIBLE INVENTORY How can this thing be implemented? ~ / s r c / a n s i b l e / l i b / a n s i b l e $ l s i n v e n t o r y / d i r . p y e x p a n d _ h o s t s . p y g r o u p . p y h o s t . p y i n i . p y _ _ i n i t _ _ . p y s c r i p t . p y v a r s _ p l u g i n s

ANSIBLE.INVENTORY 9 6 e l i f o s . p a t h . e x i s t s ( h o s t _ l i s t ) : 9 7 i f o s . p a t h . i s d i r ( h o s t _ l i s t ) : 9 8 # E n s u r e b a s e d i r i s i n s i d e t h e d i r e c t o r y 9 9 s e l f . h o s t _ l i s t = o s . p a t h . j o i n ( s e l f . h o s t _ l i s t , " " ) 1 0 0 s e l f . p a r s e r = I n v e n t o r y D i r e c t o r y ( f i l e n a m e = h o s t _ l i s t ) 1 0 1 s e l f . g r o u p s = s e l f . p a r s e r . g r o u p s . v a l u e s ( ) 1 0 2 e l i f u t i l s . i s _ e x e c u t a b l e ( h o s t _ l i s t ) : 1 0 3 s e l f . p a r s e r = I n v e n t o r y S c r i p t ( f i l e n a m e = h o s t _ l i s t ) 1 0 4 s e l f . g r o u p s = s e l f . p a r s e r . g r o u p s . v a l u e s ( ) 1 0 5 e l s e : 1 0 6 s e l f . p a r s e r = I n v e n t o r y P a r s e r ( f i l e n a m e = h o s t _ l i s t ) 1 0 7 s e l f . g r o u p s = s e l f . p a r s e r . g r o u p s . v a l u e s ( ) 1 0 8 1 0 9 u t i l s . p l u g i n s . v a r s _ l o a d e r . a d d _ d i r e c t o r y ( s e l f . b a s e d i r ( ) , w i t h _ s u b d i r = T r u e )

ANSIBLE.INVENTORY.DIR 5 8 i f o s . p a t h . i s d i r ( f u l l p a t h ) : 5 9 p a r s e r = I n v e n t o r y D i r e c t o r y ( f i l e n a m e = f u l l p a t h ) 6 0 e l i f u t i l s . i s _ e x e c u t a b l e ( f u l l p a t h ) : 6 1 p a r s e r = I n v e n t o r y S c r i p t ( f i l e n a m e = f u l l p a t h ) 6 2 e l s e : 6 3 p a r s e r = I n v e n t o r y P a r s e r ( f i l e n a m e = f u l l p a t h ) 6 4 s e l f . p a r s e r s . a p p e n d ( p a r s e r )

~/SRC/ANSIBLE/EXAMPLES/HOSTS # T h i s i s t h e d e f a u l t a n s i b l e ' h o s t s ' f i l e . # U n g r o u p e d h o s t s , s p e c i f y b e f o r e a n y g r o u p h e a d e r s . g r e e n . e x a m p l e . c o m b l u e . e x a m p l e . c o m 1 9 2 . 1 6 8 . 1 0 0 . 1 1 9 2 . 1 6 8 . 1 0 0 . 1 0 # A c o l l e c t i o n o f h o s t s b e l o n g i n g t o t h e ' w e b s e r v e r s ' g r o u p [ w e b s e r v e r s ] a l p h a . e x a m p l e . o r g b e t a . e x a m p l e . o r g 1 9 2 . 1 6 8 . 1 . 1 0 0 1 9 2 . 1 6 8 . 1 . 1 1 0 # A c o l l e c t i o n o f d a t a b a s e s e r v e r s i n t h e ' d b s e r v e r s ' g r o u p [ d b s e r v e r s ] d b 0 1 . i n t r a n e t . m y d o m a i n . n e t d b 0 2 . i n t r a n e t . m y d o m a i n . n e t 1 0 . 2 5 . 1 . 5 6 1 0 . 2 5 . 1 . 5 7

DYNAMIC INVENTORY SCRIPTS { " d a t a b a s e s " : { " h o s t s " : [ " h o s t 1 . e x a m p l e . c o m " , " h o s t 2 . e x a m p l e . c o m " ] , " v a r s " : { " a " : t r u e } } , " w e b s e r v e r s " : [ " h o s t 2 . e x a m p l e . c o m " , " h o s t 3 . e x a m p l e . c o m " ] , " a t l a n t a " : { " h o s t s " : [ " h o s t 1 . e x a m p l e . c o m " , " h o s t 4 . e x a m p l e . c o m " , " h o s t 5 . e x a m p l e . c o m " ] , " v a r s " : { " b " : f a l s e } , " c h i l d r e n " : [ " m a r i e t t a " , " 5 p o i n t s " ] , } , " m a r i e t t a " : [ " h o s t 6 . e x a m p l e . c o m " ] , " 5 p o i n t s " : [ " h o s t 7 . e x a m p l e . c o m " ] " _ m e t a " : { " h o s t v a r s " : { " m o o c o w . e x a m p l e . c o m " : { " a s d f " : 1 2 3 4 } , " l l a m a . e x a m p l e . c o m " : { " a s d f " : 5 6 7 8 } , } } }

THE ANSIBLE INVENTORY IS NOT A TREE! g o o g l e g o o g l e / g c a l e n d a r g o o g l e / g c a l e n d a r / b a c k e n d g o o g l e / g c a l e n d a r / b a c k e n d / s t o r a g e 1 g o o g l e / g c a l e n d a r / b a c k e n d / s t o r a g e 2 g o o g l e / g c a l e n d a r / b a c k e n d / s t o r a g e 3 g o o g l e / g c a l e n d a r / f r o n t e n d g o o g l e / g c a l e n d a r / f r o n t e n d / w e b 1 g o o g l e / g c a l e n d a r / f r o n t e n d / w e b 2 g o o g l e / g c a l e n d a r / f r o n t e n d / w e b 3 g o o g l e / g d r i v e g o o g l e / g d r i v e / b a c k e n d g o o g l e / g d r i v e / b a c k e n d / s t o r a g e 1 g o o g l e / g d r i v e / b a c k e n d / s t o r a g e 2 g o o g l e / g d r i v e / b a c k e n d / s t o r a g e 3 g o o g l e / g d r i v e / f r o n t e n d g o o g l e / g d r i v e / f r o n t e n d / w e b 1 g o o g l e / g d r i v e / f r o n t e n d / w e b 2 g o o g l e / g d r i v e / f r o n t e n d / w e b 3 g o o g l e / g m a i l g o o g l e / g m a i l / b a c k e n d g o o g l e / g m a i l / b a c k e n d / s t o r a g e 1 g o o g l e / g m a i l / b a c k e n d / s t o r a g e 2 g o o g l e / g m a i l / b a c k e n d / s t o r a g e 3 g o o g l e / g m a i l / f r o n t e n d g o o g l e / g m a i l / f r o n t e n d / w e b 1 g o o g l e / g m a i l / f r o n t e n d / w e b 2 g o o g l e / g m a i l / f r o n t e n d / w e b 3

NODES CAN LIVE IN DIFFERENT GROUPS ON DIFFERENT LEVELS g o o g l e / n g i n x g o o g l e / n g i n x / w e b 1 g o o g l e / n g i n x / w e b 2 g o o g l e / n g i n x / w e b 3 g o o g l e / t o m c a t g o o g l e / t o m c a t / s t o r a g e 1 g o o g l e / t o m c a t / s t o r a g e 2 g o o g l e / t o m c a t / s t o r a g e 3

INTERNAL ANSIBLE INVENTORY MODEL {'web1': {'groups': ['all', 'google', 'gcalendar', 'gdrive', 'gmail', 'frontend']}} {'storage2': {'groups': ['all', 'google', 'gcalendar', 'gdrive', 'gmail', 'backend']}}

INTERNAL ANSIBLE INVENTORY MODEL - 2 {'web1': {'groups': [{'groupname': 'all', 'depth': 0}, {'groupname': 'google', 'depth': 1}, {'groupname': 'gcalendar', 'depth': 2}, {'groupname': 'gdrive', 'depth': 2}, {'groupname': 'gmail', 'depth': 2}, {'groupname': 'frontend', 'depth': 3}]}} {'storage2': {'groups': [{'groupname': 'all', 'depth': 0}, {'groupname': 'google', 'depth': 1}, {'groupname': 'gcalendar', 'depth': 2}, {'groupname': 'gdrive', 'depth': 2}, {'groupname': 'gmail', 'depth': 2}, {'groupname': 'frontend', 'depth': 3}]}}

HOW ABOUT INVENTORY VARIABLE PRECEDENCE? Ansible docs are very succinct on how inventory variables precede each other Because As all things Ansible, KEEP IT SIMPLE, they say. There is only one Empire State Building. One Mona Lisa, etc. Figure out where to define a variable, and do not make it complicated. Remember: Child groups override parent groups, and hosts always override their groups.

SO ANSIBLE INVENTORY IS NOT A TREE, BUT VARIABLES PRECEDE IN CHILD GROUPS? Child? Parent? Tree? 3 5 d e f a d d _ c h i l d _ g r o u p ( s e l f , g r o u p ) : 3 6 3 7 i f s e l f = = g r o u p : 3 8 r a i s e E x c e p t i o n ( " c a n n o t a d d g r o u p t o i t s e l f " ) 3 9 4 0 # d o n o t a d d i f i t i s a l r e a d y t h e r e 4 1 i f n o t g r o u p i n s e l f . c h i l d _ g r o u p s : 4 2 s e l f . c h i l d _ g r o u p s . a p p e n d ( g r o u p ) 4 3 g r o u p . d e p t h = m a x ( [ s e l f . d e p t h + 1 , g r o u p . d e p t h ] ) 4 4 g r o u p . p a r e n t _ g r o u p s . a p p e n d ( s e l f ) 4 5 s e l f . c l e a r _ h o s t s _ c a c h e ( ) When we encounter a child group, then "depth" gets a level deeper Deeper == more child is more precedence That *is* some kind of a tree, no?

THE PROJECT mostly a Java shop, very standardized {'75% of all virtual machines are tomcat hosts': 'ONE ROLE'} every tomcat app is a two node "cluster" behind a loadbalancer; per app one or more healthchecks per cluster typically 1 application (context), sometimes more 1 big functional application typically == several application clusters applications are part of a project, a project is part of an organisation every application has three instances in each environment ['development', 'testing', 'production'] some applications communicate with other applications through the loadbalancer

INVENTORY ORGANISATION a l l i n v e n t o r y | _ o r g a n i s a t i o n 1 | _ p r o j e c t 1 | _ a p p l i c a t i o n 1 | _ d e v | _ n o d e 1 | _ n o d e 2 | _ t e s t | _ . . | _ p r o d | _ . . | _ a p p l i c a t i o n 2 | _ . . | _ p r o j e c t 2 | _ . . | _ o r g a n i s a t i o n 1 | _ . .

OTHER GROUP TREES (WE HAVE DIFFERENT TREE'S!) | _ d e v e l o p m e n t | _ o r g a n i s a t i o n 1 - d e v | _ a p p l i c a t i o n 1 - d e v | _ t e s t i n g | _ p r o d u c t i o n OR ALSO t o m c a t | _ a p p l i c a t i o n 1 | _ a p p l i c a t i o n 2 < s o m e _ o t h e r _ s e r v e r _ r o l e _ b e s i d e s _ t o m c a t > | _ a p p l i c a t i o n 7 | _ a p p l i c a t i o n 9 < / s o m e _ o t h e r _ s e r v e r _ r o l e _ b e s i d e s _ t o m c a t > lowest groups with nodes are at different levels in different tree's

EXAMPLE: A SET OF REVERSE PROXIES: a l l ( 0 ) . i n f r a ( 1 ) . . r p ( 2 ) . . . r p - v o n e t - o e ( 3 ) . . . r p - v o n e t - o n ( 3 ) . . . r p - v o n e t - p r ( 3 ) . . . r p - i n n e t - o e ( 3 ) . . . r p - i n n e t - o n ( 3 ) . . . r p - i n n e t - p r ( 3 ) . . r p - v o n e t ( 2 ) . . . r p - v o n e t - o n ( 3 ) . . . r p - v o n e t - o e ( 3 ) . . . r p - v o n e t - p r ( 3 ) . . r p - i n n e t ( 2 ) . . . r p - i n n e t - o n ( 3 ) . . . r p - i n n e t - o e ( 3 ) . . . r p - i n n e t - p r ( 3 )

SOMETIMES DEPTH IS WRONG! . u n i s o n ( 1 ) . . g e o s e r v e r - d o v p u b ( 2 ) . . g e o s e r v e r - d o v p u b - o e ( 2 ) . . g e o s e r v e r - d o v p u b - o n ( 2 ) . . g e o s e r v e r - d o v p u b - p r ( 2 ) [geoserver-dovpub] is a tomcat application, but is *also* member of the [unison] group (to which a unison role maps) child groups are environment specific subgroups, but they get the the same depth as their parent

PLAYBOOK REMINDER: ANSIBLE LOOPS => WITH_ITEMS Please "deploy some apps" app1 app2 t a s k : - c o m m a n d : a 2 e n s i t e { { i t e m } } w i t h _ i t e m s : a p p l i c a t i o n s l i s t applicationslist is defined in the inventory a p p l i c a t i o n s l i s t : - a p p l i c a t i o n 1 - a p p l i c a t i o n 2 more loops? nested loops? with_nested, with_subelements, ..

WITHIN AN APPLICATION SERVER we have several apps and each app can have one or more healthchecks n o d e | _ s u b a p p 1 | _ h e a l t h c h e c k 1 | _ h e a l t h c h e c k 2 | _ s u b a p p 1 . . . . . .

DATA MODELLING SO FAR: NODES, POOLS AND HEALTHCHECKS p u b l i s h e d a p p s : - n a m e : " w e b " t y p e : " { { d e f a u l t _ a p p t y p e } } " p o r t : 8 0 8 0 l b p o r t : " { { d e f a u l t _ l b p o r t } } " m o n i t o r t y p e : " { { d e f a u l t _ m o n i t o r t y p e } } " q u o r u m : 0 m o n i t o r s : - n a m e : " { { d e f a u l t _ m o n i t o r _ a p p n a m e } } " t y p e : h t t p g e t _ p a t h : " { { d e f a u l t _ g e t _ p a t h } } " p r o t o c o l : " { { d e f a u l t _ p r o t o c o l } } " g e t _ e x t r a : " { { d e f a u l t _ g e t _ e x t r a } } " r e c e i v e : " { { d e f a u l t _ r e c e i v e _ s t r i n g } } "

DATA MODELLING SO FAR: NODES, POOLS AND HEALTHCHECKS - CONTINUED ( . . . p u b l i s h e d a p p s : ) - n a m e : t c p t y p e : t c p p o r t : 1 2 3 4 l b p o r t : 6 0 1 2 3 4 m o n i t o r t y p e : " { { d e f a u l t _ m o n i t o r t y p e } } " q u o r u m : 0 m o n i t o r s : - n a m e : " t c p " t y p e : t c p _ h a l f _ o p e n s e n d : " " r e c e i v e : " "

CREATE THE CLUSTER APPLICATION MONITORS (HEALTHCHECKS) snippet of a task: m o d u l e : . . . n a m e : > { { i t e m . 0 . m o n i t o r n a m e | d e f a u l t ( ' M O N - ' ~ i t e m . 0 . t y p e | d e f a u l t ( d e f a u l t _ a p p t y p e ) | u p p e r ~ ' - ' ~ a p p ~ ( ' - ' ~ i t e m . 0 . n a m e | d e f a u l t ( ' w e b ' ) ) | r e p l a c e ( ' - w e b ' , ' ' ) ~ ( ' - ' ~ i t e m . 1 . n a m e | d e f a u l t ( ' w e b ' ) ) | r e p l a c e ( ' - w e b ' , ' ' ) ~ ' - ' ~ z u i l ) } } s e n d : > { { ' G E T ' ~ i t e m . 1 . g e t _ p a t h ~ ' ' ~ i t e m . 1 . p r o t o c o l | d e f a u l t ( d e f a u l t _ p r o t o c o l ) ~ i t e m . 1 . g e t _ e x t r a | d e f a u l t ( ' ' ) ~ ' \ \ r \ \ n \ \ r \ \ n ' } } r e c e i v e : " { { i t e m . 1 . r e c e i v e | d e f a u l t ( d e f a u l t _ r e c e i v e _ s t r i n g ) } } " p o r t : " { { i t e m . 1 . p o r t | d e f a u l t ( 0 ) } } " w h e n : i t e m . 1 . t y p e i s d e f i n e d a n d i t e m . 1 . t y p e = = ' h t t p ' w i t h _ s u b e l e m e n t s : - p u b l i s h e d a p p s - m o n i t o r s I will need this monitor name again later, but I can't easily definite and keep it in a variable, as it comes from a loop construct

CREATE THE CLUSTER SERVER POOLS task snippet where we define the monitors (healthchecks) to be applied to the pool m o n i t o r s : > { % f o r m o n i n i t e m . m o n i t o r s % } { { ' / C o m m o n / ' ~ i t e m . m o n i t o r n a m e | d e f a u l t ( ' M O N - ' ~ i t e m . t y p e | d e f a u l t ( d e f a u l t _ a p p t y p e ) | u p p e r ~ ' - ' ~ a p p ~ ( ' - ' ~ i t e m . n a m e | d e f a u l t ( ' w e b ' ) ) | r e p l a c e ( ' - w e b ' , ' ' ) | ~ ( ' - ' ~ m o n . n a m e | d e f a u l t ( ' w e b ' ) ) | r e p l a c e ( ' - w e b ' , ' ' ) | ~ ' - ' ~ z u i l ) } } { % i f n o t l o o p . l a s t % } , { % e n d i f % } { % e n d f o r % } w i t h _ i t e m s : p u b l i s h e d a p p s sometimes looping within a jinja2 template gives more power, and more possibilities (and more bugs)

MORE LOOPS, MORE DEEPLY NESTED? TODO: I need to define proxies based on previous pools Each application needs access to a set of other applications. == An applications needs the definition of a list of other applictions to get access to Then I need to loop though that list, and then the list of publishedapps for every application... This can possibly go cross-environment...(eg we don't have separate smtp servers per environment, so dev nodes need access to smtp-production) Maybe the list of applications can differ per environment? ... THIS IS GETTING TOO COMPLEX... Or should we NOT automate all things?

MANAGING INVENTORY DATA updating software versions of applications in different stacks, in different environments update default java version: for all dev servers? Keep deployed version for each application in each environment seperately Nice to have = an inventory tool that keeps track of this let's you inherit defaults, but then remember them for particular setup

Q/A ?

THANKS! https://github.com/sergevanginderachter/revealjs- presentations/tree/cfgmgmtcampeu-ansible- inventory