Continuous Integration before and after Chef

How we built our Abiquo continuous integration and continuous deployment pipelines to scale using jclouds and Chef.

Transcript

1. CI before and after Chef Chef Users London Meetup
2. Disclaimer
3. Xavier Fernandez CTO and proud founder at Abiquo Ignasi Barrera Senior Engineer at Abiquo Apache jclouds committer & PMC @IgnasiBarrera @xfernandez
4. • About abiquo • Why CI is important in abiquo • Infrastructure software needs • Our CI/CD pipelines • jclouds-chef • Lessons learned Agenda
5. abiquo - anycloud.abiquo.com • Founded 2006 • Venture Capital Funded – $24.5m • 25 employees, with 16 in Engineering ◦ 7 in UK ◦ 18 in Spain • Locations ◦ Headquartered in Heathrow, UK ◦ R&D in Barcelona, Catalonia, Spain • + 3000 customers around the world
6. Why CI is important in abiquo • We introduced Agile Programming (something like scrum) from the beginning. • TDD and other test techniques are good to ensure the quality of a new feature but… CI helps preserving quality over time
7. Nexenta NetApp Generic NFS Generic iSCSI Infrastructure software LVM Monolithic Distributed Multi-datacenter ESX Hyper-V KVM XenServer OracleVM Amazon Rackspace HP Google DigitalOcean ElasticHosts CloudSigma DEPLOYMENTS HYPERVISORS PUBLIC CLOUDS STORAGE
8. Nexenta NetApp Generic NFS Generic iSCSI Infrastructure software LVM Monolithic Distributed Multi-datacenter ESX Hyper-V KVM XenServer OracleVM Amazon Rackspace HP Google DigitalOcean ElasticHosts CloudSigma DEPLOYMENTS HYPERVISORS PUBLIC CLOUDS STORAGE HOW MANY COMBINATIONS?
9. • Too many infrastructure combinations to have static staging / testing environments • Too many resources required • Continuous evolution of third party products to be aligned with the evolution of your product Infrastructure software
10. CI/CD pipelines needs • Customizable • Scalable • Extensible • User friendly • End-to-end Jenkins provides everything we need
11. Continuous integration Provision Jenkins Slaves Commit Integration tests Build
12. Continuous deployment CI Create OS packages & build repo Provision staging environment Functional + UI tests ISO
13. Configurable pipelines
14. jclouds-chef • Chef is not only for ruby fans! • Complete Chef API for Java • Open Source & Enterprise Chef • Chef Solo • knife bootstrap
15. Bootstrap example ChefContext ctx = ContextBuilder.newBuilder("chef") .endpoint(“http://abiquo-chef.bcn.abiquo.com”) .credentials("provisioner", readFile (“provisioner.pem”)) .buildView(ChefContext.class); ChefService chef = ctx.getChefService(); BootstrapConfig bootstrapConfig = BootstrapConfig.builder() .runList(new RunListBuilder().addRole("abiquo-nightly").build()); .attributes(new JsonBall(readFile (“attributes.json”)))); .environment(“Staging”); .build(); chef.updateBootstrapConfigForGroup(“staging”, bootstrapConfig); Statement bootstrap = chef.createBootstrapScriptForGroup(“staging”); compute.runScriptOnNodesMatching(inGroup (“staging”), bootstrap); attributes.json { "set_fqdn": "{{hostname}}", "abiquo": { "license": "{{license}}", "yum": { "nightly-repo": "http://staging.bcn.abiquo.com/{{version}}/rpm" } } } Dynamic run lists Dynamic environments Dynamic attributes Reusable & persistent configuration
16. Lessons learned • Infrastructure as code • Test everything • Use the right tools • Think big
17. Thank you! Q&A