INFRASTRUCTURE IN CODE STRIVING FOR BETTER ABSTRACTIONS AND AUTOMATION @pyr

SHORT BIO Pierre-Yves Ritschard CTO @ exoscale - The leading swiss public cloud provider Open Source Developer - riemann, collectd, pallet, openbsd Architect of several cloud platforms - paper.li Recovering Operations Engineer

Cloud History 101 Shedding the cognitive load Let cloudstack help you

CLOUD HISTORY 101 Getting rid of the physical world

Three decades of infrastructure Where we are today What went wrong

THREE DECADES OF INFRASTRUCTURE

In the 90's we had switches, routers and servers and they all had funny names

In the 2000's we had switches, routers ,hypervisors and virtual machines Tangentially, configuration management became a thing

Now we have commoditized infrastructure In common speak we have moved from machines to instances

WHERE WE ARE TODAY

Programmable provisioning

Programmable decomissioning

Ubiquitous IAAS

Much much simpler capacity planning

Negligible provision times

So, that's it ?

« Welp, looks like we done fixed infrastructure once and for all » - no one, ever

WHAT WENT WRONG

Credit: Jochen Smolka, Lund University

We still apply a big ball of mud approach to infrastructure instances are still black boxes full of mutable state

As a corollary, we keep a mapping of services to instances Although admittedly, configuration management helps

We insist on carrying over concepts from the physical world c r e a t e I P F o r w a r d i n g R u l e ... are we crazy ?

snowflakes !

SHEDDING THE COGNITIVE LOAD Embracing simplicity

Better abstractions Better automation

BETTER ABSTRACTIONS

Complexity is the enemy of scalability

Complexity is the enemy of security

We need to change our trust model No more NAT No more Address association No more volumes

Simpler abstractions Security groups Snapshots

BETTER AUTOMATION

Stop treating instances as the base unit of reasoning mitigates the risk of ending with a big ball of mud

Nodes are part of homogeneous groups (clusters)

Nodes should avoid configuration drift Strive for immutable infrastructure

Integrate IAAS features in automation by storing hints for configuration management when provisioning instances

Use your IAAS as a low level config registry let clusters discover themselves

LET CLOUDSTACK HELP YOU A short guide to better infrastructure

Abstractions Tooling Library support

ABSTRACTIONS

Keypairs you probably use them already

User Data c l o u d - i n i t is a great tool

Basic networking suffices for almost all use cases More scalable routing Great firewalling abilities through security groups

Tags Arbitrary metadata on instances

Projects, Instance Groups More heavy weight abstractions

TOOLING

Puppet Resource manipulation support cloud-init integration

Chef knife support for bootstrapping instances cloud-init integration

Cloudmonkey Great interaction tool, actively maintained

LIBRARY SUPPORT

Name your language Java: jclouds Ruby: fog, cloudstack_ruby_client Python: libcloud Clojure: pallet

Pallet: a small introduction Configuration management, provisionning, command and control Same tool space than chef and knife, but a library Built on top of apache Jclouds Clojure

Pallet: simple abstractions Node specifications Server Specifications Groups

Node specifications ( d e f u b u n t u - n o d e ( n o d e - s p e c : n e t w o r k { : i n b o u n d - p o r t s [ 2 2 , 8 0 , 4 4 3 ] } , : i m a g e { : o s - f a m i l y : u b u n t u , : o s - v e r s i o n - m a t c h e s " 1 2 . 0 4 " } , : h a r d w a r e { : m i n - c o r e s 1 , : m i n - d i s k 1 0 , : m i n - r a m 5 1 2 } ) )

Server specifications ( d e f w e b - s e r v e r ( s e r v e r - s p e c : p h a s e s { : c o n f i g u r e ( p l a n - f n ( p a c k a g e " n g i n x " ) ( p a c k a g e " m y - w e b - a p p " ) ) } ) ) ( d e f l b - s e r v e r ( s e r v e r - s p e c : p h a s e s { : c o n f i g u r e ( p l a n - f n ( p a c k a g e " h a p r o x y " ) ( h a p r o x y / a d d - b a c k e n d " w e b " : s e r v e r s ( n o d e s - w i t h - r o l e : w e b ) ) ) } ) )

Groups ( d e f w e b ( g r o u p - s p e c " w e b " : r o l e s [ : w e b ] : e x t e n d s [ b a s e - s e r v e r w e b - s e r v e r ] : n o d e - s p e c u b u n t u - n o d e ) ) ( d e f l b ( g r o u p - s p e c " l b " : r o l e s [ : l b ] : e x t e n d s [ b a s e - s e r v e r l b - s e r v e r ] : n o d e - s p e c u b u n t u - n o d e ) )

Provision from the CLI l e i n p a l l e t c o n v e r g e l b 1 w e b 4 Or your Code ( c o n v e r g e { l b 1 , w e b 4 } )

Pallet embraces the cloudstack API ( { : k e y " p a l l e t - g r o u p " , : r e s o u r c e i d " 3 b d 6 d 1 c d - e 8 f 8 - 4 4 5 f - 8 1 8 0 - 6 4 4 9 4 6 f 1 2 e d d " , : r e s o u r c e t y p e " U s e r V M " , : v a l u e " w e b " } , { : k e y " p a l l e t - s t a t e " , : r e s o u r c e i d " 3 b d 6 d 1 c d - e 8 f 8 - 4 4 5 f - 8 1 8 0 - 6 4 4 9 4 6 f 1 2 e d d " , : r e s o u r c e t y p e { : b o o t s t r a p p e d t r u e } , : v a l u e " w e b " } , { : k e y " p a l l e t - g r o u p " , : r e s o u r c e i d " 3 b d 6 d 1 c d - e 8 f 8 - 4 4 5 f - 8 1 8 0 - 6 4 4 9 4 6 f 1 2 e d d " , : r e s o u r c e t y p e " U s e r V M " , : v a l u e { : o s - v e r s i o n " 1 2 . 0 2 " , : o s - f a m i l y : u b u n t u , : i m a g e - i d " a 1 7 b 4 0 d 6 - 8 3 e 4 - 4 f 2 a - 9 e f 0 - d c e 6 a f 5 7 5 f f f a " } } )

PARTING WORDS Where to go from here

From infrastructure as code to infrastructure as data next step: reactive infrastructure

More power to the controller From IAAS to COAAS

Resources https://palletops.com https://github.com/exoscale/pallet-exoscale-demo

THANK YOU ! QUESTIONS ? @pyr