Extending CI/CD in Operations Using Chef & LXC

Chef is a popular configuration management system used for system automation, that promotes the idea of infrastructure as code. In this presentation I'll walk through how we are using LXC to perform integration testing of our entire infrastructure automation code base, and then re-using the build/test artifacts to compose developer workstation. I'll be covering a) current state of chef-lxc integration and how we are using it, b) how our build-ci infrastructure was extended to build and test our chef/automation code base using LXC, and c) how the test artifacts are served via traditional debain repos to build developer workstations (i.e. the LXC rootfs packaging gears)

Transcript

1. Extending CI/CD in Operations Using
   Chef & LXC
   @RanjibDey
   

   View Slide

2. About Me
   @RanjibDey
   

   View Slide

3. @RanjibDey
   Part 1 : The Case for CI/CD in Operations
   

   View Slide

4. Challenges in Operations
   @RanjibDey
   1.Integrating disparate systems
   2.Maintaining legacy components
   3.The burden of tactical works
   

   View Slide

5. CI and CD
   @RanjibDey
   1.Layered test suites invoked on every commit
   2.Every commit that passes test gets deployed
   3.Deployment stage can be semi-automatic
   

   View Slide

6. CI and CD
   @RanjibDey
   1. Running tests reduces failure
   2. Automated deployment reduces human error.
   3. Reduce build test deploy cycle time
   → →
   

   View Slide

7. But CI/CD In Operations Is Hard
   @RanjibDey
   1.Lack of domain knowledge
   2.Maturity of tooling, interoperability issues
   3.Diversity of system components
   4.Longer build/test times
   

   View Slide

8. @RanjibDey
   Part 2 : Tools of the Trade
   

   View Slide

9. GoCD – Fan In & Fan Out
   @RanjibDey
   1. An OpenSource CI system focused on pipelines
   2.Run tests on dependency changes
   3. Run tests of dependent projects
   4. Allows extensive parallelization
   http://www.go.cd/
   

   View Slide

10. LXC – System containers for the win
    @RanjibDey
    1. Full system containers (cron, init, syslog)
    2. Usernamespace, seccomp, apparmor
    3. Liblxc allows first class python/ruby binding
    4. No filesystem layers. Isolated container rootfs
    5. Minimal containment system.
    

    View Slide

11. Ruby LXC
    @RanjibDey
    Ruby bindings for liblxc, allows container operations
    from arbitrary ruby code.
    require 'lxc'
    c = LXC::Container.new('foo')
    c.create('ubuntu')
    c.start
    c.stop
    c.destroy
    https://github.com/lxc/ruby-lxc
    

    View Slide

12. Chef – A system automation framework
    @RanjibDey
    1. Represent infrastructure with vanilla Ruby.
    2. A configuration management system
    3. Baked in metadata service
    4. Baked in host discovery
    5. Subsystem for building CLI tools (knife)
    

    View Slide

13. Chef-LXC
    @RanjibDey
    Chef integration for LXC, allows creating containers
    From chef cookbooks.
    lxc "web" do
    template "ubuntu"
    action [:create, :start]
    end
    https://github.com/ranjib/chef-lxc
    

    View Slide

14. Chef-LXC
    @RanjibDey
    Allows executing chef resources inside containers
    without installing chef inside them.
    lxc "web" do
    template "ubuntu"
    recipe do
    package "apache2"
    service "apache2" do
    action [:start, :enable]
    end
    end
    action [:create, :start]
    end
    

    View Slide

15. @RanjibDey
    Part 3 : A CI/CD Pipeline for Operations
    

    View Slide

16. Standard CI pipeline
    @RanjibDey
    1.Unit tests for fast feedback
    2.Functional tests for feature validation
    3.Integration tests for end to end testing
    

    View Slide

17. Unit tests
    @RanjibDey
    directory "/var/go/.ssh" do
    owner 'go'
    group 'go'
    mode 0700
    end
    it 'creates the go user ssh directory' do
    expect(runner).to create_directory('/var/go/.ssh').with(
    owner: 'go',
    group: 'go',
    mode: 0700
    )
    end
    https://github.com/sethvargo/chefspec
    

    View Slide

18. Unit tests
    @RanjibDey
    pagerduty/infra[master]$rake spec
    Finished in 1 minute 15.16 seconds
    5629 examples, 0 failures, 4 pending
    

    View Slide

19. Functional tests
    @RanjibDey
    it 'creates a standalone zk node' do
    ct = helper.container('pd-zk')
    out = ct.execute do
    h= 'localhost'
    p = 2181
    telnet = Net::Telnet::new('Host' => h, 'Port' => p)
    state = telnet.cmd('String' => 'ruok')
    telnet.close
    state
    end
    expect(out).to eq('imok')
    end
    

    View Slide

20. Integration Tests
    @RanjibDey
    

    View Slide

21. CI Pipeline Stages for Operations
    @RanjibDey
    

    View Slide

22. Courtesy artifacts
    @RanjibDey
    Chef cookbooks used for building
    production infrastructure, are distributed
    as debian packages that can be consumed
    by ancillary projects, like building
    developer infrastructure
    

    View Slide

23. @RanjibDey
    Part 3 : Case studies
    

    View Slide

24. Blender
    @RanjibDey
    1. A modular remote command execution framework
    2. Pluggable host discovery (Chef, Serf etc)
    3. Pluggable command dispatch mechanism (ssh, serf)
    

    View Slide

25. Blender :: Integration Test Suite
    @RanjibDey
    Blender-Core
    Blender-Serf
    Blender-Chef
    Blender-Zk
    Blender
    Integration Tests
    https://github.com/pagerduty/blender-integration
    

    View Slide

26. Blender :: Integration Test Suite
    @RanjibDey
    

    View Slide

27. GoatOS
    @RanjibDey
    1. An integration testing framework for Chef cookbooks
    2. Build chef omnibus installers for every commit
    3. Tests community cookbooks against new installers
    

    View Slide

28. GoatOS :: Stages
    @RanjibDey
    Chef
    •
    Unit tests
    •
    Functional test
    •
    Build Gem Chef-LXC
    •
    Functional Tests
    GoatOS-Spec
    •
    Test cookbooks
    (mysql, sudo, nginx etc)
    Build Omnibus
    •
    Debian package
    https://github.com/GoatOS
    

    View Slide

29. GoatOS :: GoCD Pipeline View
    @RanjibDey
    

    View Slide

30. GoatOS :: Chef build on Raspberry Pi
    @RanjibDey
    Chef omnibus installers for Raspberry Pi are created
    via GoatOS
    

    View Slide

31. Learnings
    @RanjibDey
    1. Co-develop CI gears with automation suite
    2. Embrace standard software engineering principles
    3. Avoid ad hoc scripts.
    4. Upstream everything
    

    View Slide

32. @RanjibDey
    Thank You
    

    View Slide