A One stop solution for Puppet and Openstack - PuppetConf 13'

1. A one stop solution for Puppet and Openstack Daniel Lobato Garcia daniel.lobato.garcia@cern,ch @eLobatoss
3. What is CERN Between Geneva and the Jura mountains, straddling the Swiss- French border Mission: learn what is the universe made of and how does it work? 3
4. Fundamental questions in physics Why do particles have mass? What is 96% of the universe made of? Why isn’t there anti-matter in the universe? What was the state of matter after the Big Bang? 4
5. 8/12/2013 Document reference 5
6. 8/12/2013 Document reference 6
7. 8/12/2013 Document reference 7
8. 8/12/2013 Document reference 8
9. Current status • 270 Openstack hypervisors • 2900 virtual machines • 300 users • 14 Puppet masters • 6 Foreman backend nodes • Some production services migrating to our cloud – early birds 9
10. Goals • Ramp up to 15K hypervisors – 150-200K vms in 2015 • Multi-site (Hungary) 10
11. 8/12/2013 Document reference 11
12. 8/12/2013 Document reference 12
13. Why? • Unnecessary homebrew stack of tools • Shift to cloud standards with minimal customizations • High turnover – can’t teach new tools 13
14. Why? • Symbiotic relationship with the community 14
15. Openstack? • Modular IaaS free open source project • APIs ~compatible with those of Amazon 15
16. Openstack Nova (compute) Cloud fabric controller 16
17. Openstack Keystone (Identity) RBAC Integrated with LDAP Multiple auth* methods 17
18. Openstack Glance (Images) Discovery, registration, delivery of images 18
19. Openstack Horizon (Dashboard) 19
20. Modules • Puppet definitions for every use case you can imagine. • Dynamic environments • Hadoop node • Openstack hypervisor • … you name it 20
21. 21 Workflow..?
22. Modules and Git • Manifests and hieradata are version controlled 22
23. 23 Git workflow Puppet masters
24. 24 Easy cherry pick
25. 25 Git workflow
26. 26 Git workflow Jens ‘Puppetfiles’ Separate repositories Makes environments from YAML config files Puppet master rsync
27. 27 upstream.yaml --- default: production notifications: ai@cern.ch overrides: modules: foreman: upstream puppet: upstream ‘Puppetfiles’
28. Foreman • Lifecycle management tool for VMs and physical servers • External Node Classifier – tells the puppet master what a node should look like 28
31. Power operations & Foreman 8/12/2013 Document reference 31 Foreman Proxy Physical box IPMI Physical box IPMI Physical box IPMI VM VM VM Openstack Nova API
32. Openstack VM creation 8/12/2013 Document reference 32
33. Openstack VM creation 8/12/2013 Document reference 33
34. Openstack VM creation 8/12/2013 Document reference 34
35. Scalability experiences • Split up services • Puppet – critical vs non critical 35 12 backend nodes Batch 4 backend nodes Interactive
36. Scalability experiences • Foreman – split into different services 36 ENC Reports processing UI/API Load balancer 9443 – UI/API 9444 – Reports 9445 – ENC …
37. Scalability experiences • Autoscale via alarms (Heat) • Define situations (i.e: load threshold..) • Spin up VMs as needed 37
38. Scalability guidelines 38
39. github.com / cernops 39