Tweet
Tweet

Slide 1

Slide 1 text

Extending CI/CD in Operations Using Chef & LXC @RanjibDey

Slide 2

Slide 2 text

About Me @RanjibDey

Slide 3

Slide 3 text

@RanjibDey Part 1 : The Case for CI/CD in Operations

Slide 4

Slide 4 text

Challenges in Operations @RanjibDey 1.Integrating disparate systems 2.Maintaining legacy components 3.The burden of tactical works

Slide 5

Slide 5 text

CI and CD @RanjibDey 1.Layered test suites invoked on every commit 2.Every commit that passes test gets deployed 3.Deployment stage can be semi-automatic

Slide 6

Slide 6 text

CI and CD @RanjibDey 1. Running tests reduces failure 2. Automated deployment reduces human error. 3. Reduce build test deploy cycle time → →

Slide 7

Slide 7 text

But CI/CD In Operations Is Hard @RanjibDey 1.Lack of domain knowledge 2.Maturity of tooling, interoperability issues 3.Diversity of system components 4.Longer build/test times

Slide 8

Slide 8 text

@RanjibDey Part 2 : Tools of the Trade

Slide 9

Slide 9 text

GoCD – Fan In & Fan Out @RanjibDey 1. An OpenSource CI system focused on pipelines 2.Run tests on dependency changes 3. Run tests of dependent projects 4. Allows extensive parallelization http://www.go.cd/

Slide 10

Slide 10 text

LXC – System containers for the win @RanjibDey 1. Full system containers (cron, init, syslog) 2. Usernamespace, seccomp, apparmor 3. Liblxc allows first class python/ruby binding 4. No filesystem layers. Isolated container rootfs 5. Minimal containment system.

Slide 11

Slide 11 text

Ruby LXC @RanjibDey Ruby bindings for liblxc, allows container operations from arbitrary ruby code. require 'lxc' c = LXC::Container.new('foo') c.create('ubuntu') c.start c.stop c.destroy https://github.com/lxc/ruby-lxc

Slide 12

Slide 12 text

Chef – A system automation framework @RanjibDey 1. Represent infrastructure with vanilla Ruby. 2. A configuration management system 3. Baked in metadata service 4. Baked in host discovery 5. Subsystem for building CLI tools (knife)

Slide 13

Slide 13 text

Chef-LXC @RanjibDey Chef integration for LXC, allows creating containers From chef cookbooks. lxc "web" do template "ubuntu" action [:create, :start] end https://github.com/ranjib/chef-lxc

Slide 14

Slide 14 text

Chef-LXC @RanjibDey Allows executing chef resources inside containers without installing chef inside them. lxc "web" do template "ubuntu" recipe do package "apache2" service "apache2" do action [:start, :enable] end end action [:create, :start] end

Slide 15

Slide 15 text

@RanjibDey Part 3 : A CI/CD Pipeline for Operations

Slide 16

Slide 16 text

Standard CI pipeline @RanjibDey 1.Unit tests for fast feedback 2.Functional tests for feature validation 3.Integration tests for end to end testing

Slide 17

Slide 17 text

Unit tests @RanjibDey directory "/var/go/.ssh" do owner 'go' group 'go' mode 0700 end it 'creates the go user ssh directory' do expect(runner).to create_directory('/var/go/.ssh').with( owner: 'go', group: 'go', mode: 0700 ) end https://github.com/sethvargo/chefspec

Slide 18

Slide 18 text

Unit tests @RanjibDey pagerduty/infra[master]$rake spec Finished in 1 minute 15.16 seconds 5629 examples, 0 failures, 4 pending

Slide 19

Slide 19 text

Functional tests @RanjibDey it 'creates a standalone zk node' do ct = helper.container('pd-zk') out = ct.execute do h= 'localhost' p = 2181 telnet = Net::Telnet::new('Host' => h, 'Port' => p) state = telnet.cmd('String' => 'ruok') telnet.close state end expect(out).to eq('imok') end

Slide 20

Slide 20 text

Integration Tests @RanjibDey

Slide 21

Slide 21 text

CI Pipeline Stages for Operations @RanjibDey

Slide 22

Slide 22 text

Courtesy artifacts @RanjibDey Chef cookbooks used for building production infrastructure, are distributed as debian packages that can be consumed by ancillary projects, like building developer infrastructure

Slide 23

Slide 23 text

@RanjibDey Part 3 : Case studies

Slide 24

Slide 24 text

Blender @RanjibDey 1. A modular remote command execution framework 2. Pluggable host discovery (Chef, Serf etc) 3. Pluggable command dispatch mechanism (ssh, serf)

Slide 25

Slide 25 text

Blender :: Integration Test Suite @RanjibDey Blender-Core Blender-Serf Blender-Chef Blender-Zk Blender Integration Tests https://github.com/pagerduty/blender-integration

Slide 26

Slide 26 text

Blender :: Integration Test Suite @RanjibDey

Slide 27

Slide 27 text

GoatOS @RanjibDey 1. An integration testing framework for Chef cookbooks 2. Build chef omnibus installers for every commit 3. Tests community cookbooks against new installers

Slide 28

Slide 28 text

GoatOS :: Stages @RanjibDey Chef • Unit tests • Functional test • Build Gem Chef-LXC • Functional Tests GoatOS-Spec • Test cookbooks (mysql, sudo, nginx etc) Build Omnibus • Debian package https://github.com/GoatOS

Slide 29

Slide 29 text

GoatOS :: GoCD Pipeline View @RanjibDey

Slide 30

Slide 30 text

GoatOS :: Chef build on Raspberry Pi @RanjibDey Chef omnibus installers for Raspberry Pi are created via GoatOS

Slide 31

Slide 31 text

Learnings @RanjibDey 1. Co-develop CI gears with automation suite 2. Embrace standard software engineering principles 3. Avoid ad hoc scripts. 4. Upstream everything

Slide 32

Slide 32 text

@RanjibDey Thank You