Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Extending CI/CD in Operations Using Chef & LXC

Extending CI/CD in Operations Using Chef & LXC

Chef is a popular configuration management system used for system automation, that promotes the idea of infrastructure as code. In this presentation I'll walk through how we are using LXC to perform integration testing of our entire infrastructure automation code base, and then re-using the build/test artifacts to compose developer workstation. I'll be covering a) current state of chef-lxc integration and how we are using it, b) how our build-ci infrastructure was extended to build and test our chef/automation code base using LXC, and c) how the test artifacts are served via traditional debain repos to build developer workstations (i.e. the LXC rootfs packaging gears)

Ranjib Dey

August 20, 2015
Tweet

More Decks by Ranjib Dey

Other Decks in Programming

Transcript

  1. Extending CI/CD in Operations Using
    Chef & LXC
    @RanjibDey

    View Slide

  2. About Me
    @RanjibDey

    View Slide

  3. @RanjibDey
    Part 1 : The Case for CI/CD in Operations

    View Slide

  4. Challenges in Operations
    @RanjibDey
    1.Integrating disparate systems
    2.Maintaining legacy components
    3.The burden of tactical works

    View Slide

  5. CI and CD
    @RanjibDey
    1.Layered test suites invoked on every commit
    2.Every commit that passes test gets deployed
    3.Deployment stage can be semi-automatic

    View Slide

  6. CI and CD
    @RanjibDey
    1. Running tests reduces failure
    2. Automated deployment reduces human error.
    3. Reduce build test deploy cycle time
    → →

    View Slide

  7. But CI/CD In Operations Is Hard
    @RanjibDey
    1.Lack of domain knowledge
    2.Maturity of tooling, interoperability issues
    3.Diversity of system components
    4.Longer build/test times

    View Slide

  8. @RanjibDey
    Part 2 : Tools of the Trade

    View Slide

  9. GoCD – Fan In & Fan Out
    @RanjibDey
    1. An OpenSource CI system focused on pipelines
    2.Run tests on dependency changes
    3. Run tests of dependent projects
    4. Allows extensive parallelization
    http://www.go.cd/

    View Slide

  10. LXC – System containers for the win
    @RanjibDey
    1. Full system containers (cron, init, syslog)
    2. Usernamespace, seccomp, apparmor
    3. Liblxc allows first class python/ruby binding
    4. No filesystem layers. Isolated container rootfs
    5. Minimal containment system.

    View Slide

  11. Ruby LXC
    @RanjibDey
    Ruby bindings for liblxc, allows container operations
    from arbitrary ruby code.
    require 'lxc'
    c = LXC::Container.new('foo')
    c.create('ubuntu')
    c.start
    c.stop
    c.destroy
    https://github.com/lxc/ruby-lxc

    View Slide

  12. Chef – A system automation framework
    @RanjibDey
    1. Represent infrastructure with vanilla Ruby.
    2. A configuration management system
    3. Baked in metadata service
    4. Baked in host discovery
    5. Subsystem for building CLI tools (knife)

    View Slide

  13. Chef-LXC
    @RanjibDey
    Chef integration for LXC, allows creating containers
    From chef cookbooks.
    lxc "web" do
    template "ubuntu"
    action [:create, :start]
    end
    https://github.com/ranjib/chef-lxc

    View Slide

  14. Chef-LXC
    @RanjibDey
    Allows executing chef resources inside containers
    without installing chef inside them.
    lxc "web" do
    template "ubuntu"
    recipe do
    package "apache2"
    service "apache2" do
    action [:start, :enable]
    end
    end
    action [:create, :start]
    end

    View Slide

  15. @RanjibDey
    Part 3 : A CI/CD Pipeline for Operations

    View Slide

  16. Standard CI pipeline
    @RanjibDey
    1.Unit tests for fast feedback
    2.Functional tests for feature validation
    3.Integration tests for end to end testing

    View Slide

  17. Unit tests
    @RanjibDey
    directory "/var/go/.ssh" do
    owner 'go'
    group 'go'
    mode 0700
    end
    it 'creates the go user ssh directory' do
    expect(runner).to create_directory('/var/go/.ssh').with(
    owner: 'go',
    group: 'go',
    mode: 0700
    )
    end
    https://github.com/sethvargo/chefspec

    View Slide

  18. Unit tests
    @RanjibDey
    pagerduty/infra[master]$rake spec
    Finished in 1 minute 15.16 seconds
    5629 examples, 0 failures, 4 pending

    View Slide

  19. Functional tests
    @RanjibDey
    it 'creates a standalone zk node' do
    ct = helper.container('pd-zk')
    out = ct.execute do
    h= 'localhost'
    p = 2181
    telnet = Net::Telnet::new('Host' => h, 'Port' => p)
    state = telnet.cmd('String' => 'ruok')
    telnet.close
    state
    end
    expect(out).to eq('imok')
    end

    View Slide

  20. Integration Tests
    @RanjibDey

    View Slide

  21. CI Pipeline Stages for Operations
    @RanjibDey

    View Slide

  22. Courtesy artifacts
    @RanjibDey
    Chef cookbooks used for building
    production infrastructure, are distributed
    as debian packages that can be consumed
    by ancillary projects, like building
    developer infrastructure

    View Slide

  23. @RanjibDey
    Part 3 : Case studies

    View Slide

  24. Blender
    @RanjibDey
    1. A modular remote command execution framework
    2. Pluggable host discovery (Chef, Serf etc)
    3. Pluggable command dispatch mechanism (ssh, serf)

    View Slide

  25. Blender :: Integration Test Suite
    @RanjibDey
    Blender-Core
    Blender-Serf
    Blender-Chef
    Blender-Zk
    Blender
    Integration Tests
    https://github.com/pagerduty/blender-integration

    View Slide

  26. Blender :: Integration Test Suite
    @RanjibDey

    View Slide

  27. GoatOS
    @RanjibDey
    1. An integration testing framework for Chef cookbooks
    2. Build chef omnibus installers for every commit
    3. Tests community cookbooks against new installers

    View Slide

  28. GoatOS :: Stages
    @RanjibDey
    Chef

    Unit tests

    Functional test

    Build Gem Chef-LXC

    Functional Tests
    GoatOS-Spec

    Test cookbooks
    (mysql, sudo, nginx etc)
    Build Omnibus

    Debian package
    https://github.com/GoatOS

    View Slide

  29. GoatOS :: GoCD Pipeline View
    @RanjibDey

    View Slide

  30. GoatOS :: Chef build on Raspberry Pi
    @RanjibDey
    Chef omnibus installers for Raspberry Pi are created
    via GoatOS

    View Slide

  31. Learnings
    @RanjibDey
    1. Co-develop CI gears with automation suite
    2. Embrace standard software engineering principles
    3. Avoid ad hoc scripts.
    4. Upstream everything

    View Slide

  32. @RanjibDey
    Thank You

    View Slide