Cloudstack design decisions

Slide 1

Slide 1 text

CLOUDSTACK DESIGN DECISIONS CLOUD OPERATIONS AT SCALE

Slide 2

Slide 2 text

SHORT BIO Pierre-Yves Ritschard CTO @ exoscale - The safe home for your cloud applications Open Source Developer - pithos, cyanite, riemann, collectd, openbsd Architect of several cloud platforms - paper.li Recovering Operations Engineer

Slide 3

Slide 3 text

Simple and efficient cloud hosting platform Full compatibility with automation tools Hosted in a safe jurisdiction

Slide 4

Slide 4 text

CLOUD BUILDING BLOCKS service infrastructure software people

Slide 5

Slide 5 text

SERVICE SIMPLICITY AND SCALABILITY Cloudstack based Basic networking Local storage KVM hypervisor: SmartOS inspired

Slide 6

Slide 6 text

CLOUDSTACK Great extensibility, easy to plug into.

Slide 7

Slide 7 text

BASIC NETWORKING One IP per VM. Security groups are hypervisor controlled layer 2 firewall rules. Provides all the flexibility of a traditionnal firewall, completely API controlled.

Slide 8

Slide 8 text

LOCAL STORAGE Fast I/O, persistent disks.

Slide 9

Slide 9 text

KVM HYPERVISOR Best in class hypervisor. Diskless and netboot approach. Avoids resource waste, facilitates upgrades.

Slide 10

Slide 10 text

INFRASTRUCTURE THE GOOD CITIZEN CONTRACT Configuration management Visibility Build factory Remote execution

Slide 11

Slide 11 text

THE GOOD CITIZEN CONTRACT new machines have roles role defines converged configuration as sum of components each component has an expected normal state and reports it no local intervention needed

Slide 12

Slide 12 text

CONFIGURATION MANAGEMENT code is a great way to define infrastructure ensures homogeneity ability to iterate fast great source of change tracking avoids fear of change

Slide 13

Slide 13 text

OVER 3000 COMMITS

Slide 14

Slide 14 text

CONFIGURATION MANAGEMENT: PUPPET battle tested tool simple declarative DSL to express configuration fits our component approach well

Slide 15

Slide 15 text

VISIBILITY FROM THE MAP TO THE TERRITORY logs metrics alerts

Slide 16

Slide 16 text

WHY FOCUS ON VISIBILITY distributed systems with lots of moving parts, high node volatility

Slide 17

Slide 17 text

LOGS all application and system logs sent over the wire logstash disects and extracts metadata elasticsearch indexes for easy retrieval simple correlation

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

METRICS all application and system metrics sent over the wire by collectd graphite's carbon aggregates and produces appropriate roll- ups if it moves, graph it. if it doesn't, graph it if it starts moving.

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

ALERTS unbounded stream of log and metric data passive approach bodes well with node volatility riemann takes decisions based on stream content ability to extract meaningful information

Slide 22

Slide 22 text

BUILD FACTORY continuous integration package repositories

Slide 23

Slide 23 text

CONTINUOUS INTEGRATION over 60 build jobs ties into our code hosting platform handled by jenkins

Slide 24

Slide 24 text

PACKAGE REPOSITORIES generates valid and signed Debian repositories ensures fast upgrades simplifies configuration management

Slide 25

Slide 25 text

REMOTE EXECUTION a simple pubsub system recurrent commands stored as scenarios command line, HTTP and IRC interaction

Slide 26

Slide 26 text

A SIMPLE PUBSUB SYSTEM each node runs an agent responsible for carrying out commands. commands are sent to groups of nodes (by predicates such as role).

Slide 27

Slide 27 text

RECURRENT COMMANDS STORED AS SCENARIOS intricate workflows can be expressed through a simple DSL

Slide 28

Slide 28 text

COMMAND LINE, HTTP AND IRC INTERACTION most of our production environment can be controlled through our chatroom

Slide 29

Slide 29 text

SOFTWARE FILLING IN THE GAPS Customer management Real-time metering and billing Integrated console A few other things

Slide 30

Slide 30 text

CUSTOMER MANAGEMENT Keeping track of our users Support services (ticket management, coupons, emails)

Slide 31

Slide 31 text

REAL-TIME METERING AND BILLING can't be tied to a cloudstack only solution cloudstack emits useful data ties into our customer management

Slide 32

Slide 32 text

INTEGRATED CONSOLE integrated experience across our services hides complexity and cloudstack specifics exposes exoscale specific features

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

A FEW OTHER THINGS pithos cyanite fleet collectd add-ons

Slide 35

Slide 35 text

PEOPLE EFFICIENT WORK. QUIET NIGHTS Small SRE team Avoiding deploy anxiety

Slide 36

Slide 36 text

SMALL SRE TEAM Our platform must be simple to operate, additional moving parts must provide business value or help operations

Slide 37

Slide 37 text

AVOIDING DEPLOY ANXIETY Our software and infrastructure helps ensure we have good tools to ensure quiet nights and easily caught errors

Slide 38

Slide 38 text

LOOKING BACK Cloudstack is a solid foundation for a IAAS platform There's a bit more to it than just installing cloudstack Building a sustainable and scalable platform on top of cloudstack is possible

Slide 39

Slide 39 text

QUESTIONS ?