MadridDevops September 2014: "From chef09 to chef11, one approach to devops"

@MadridDevops September 2014: "From chef09 to chef11, one approach to devops"

Madrid, September 25, 2014

http://madrid.devops.es

An excuse to talk about devops, chef (and puppet as comparison), its community and a migration project I have been involved with.

Transcript

1. From chef09 to chef11 my approach to a real devops task Antonio Peña @apenav for @MadridDevops September 2014 http://about.me/apenav
2. http://madrid.devops.es
3. Motivation • feedback • sharing experience • learning • talking about devops
6. Impostor Syndrome
7. Three Parts • Devops • Chef • Migration
8. About DevOps
9. Software is broken or will be
10. Devops (I)
11. Devops (II)
12. Devops (III)
13. So little time, so much to talk about
14. Big Business. DevOps Myths (I)
15. Big Business. DevOps Myths (II)
16. My approach to a configuration management project Antonio Peña @apenav for @MadridDevops September 2014 http://about.me/apenav
17. Best tool to make config mgmt? • cfengine • puppet • chef • salt • ansible -- Business decision-making. Make your choice -- Strengths/weakness. Tend to converge.
18. About Chef
21. What is Chef? • a company ◦ Opscode Inc. ◦ getchef.com ◦ @chef • a product (with paid support) ◦ packages, not recommended install from gemfiles • an open source project • a community (open source but not only)
22. Beginners, experts and middle class
23. Berkshelf.I
24. Berkshelf.II
25. “Spanish” Way Master Chef RTVE
26. Howto devops in Spain? (I) • e.g. Demo in a Big Company
27. Howto devops in Spain? (II) • startup • why not? • sandbox
28. Howto devops in Spain? (III) • consulting
29. Working in a company as a Developer. Daily tasks
30. Initial problem Problem to be solved (in aprox. 2 weeks): 1. install chef-server 11 in a centos6 behind a VPN 2. install 2 web frontends + 2 backends a. (redis s/m + mysql m/s)
31. Real problem 1. refactor/migrate/write from scratch about 20 old-fashioned (2009) but very reliable (daily used, very difficult to extend for an outsider) ubuntu ruby18-webapp-related recipes to newer ones 2. proof of concept to migrate chef09 company’ s repository to anything better
32. From scratch http://devopsreactions.tumblr.com/post/97637875636/what-happens-when-i-am-allowed-to-rewrite-code-from
33. Methodology or the like “Big bang” approach, but iterative: • use chef11 community cookbooks • almost no direct migration from chef09 ones
34. Criteria to use community cookbooks? https://github.com/opscode-cookbooks/mysql When in doubt use popular opscode community cookbooks. So many content to follow!
35. About Migration
36. To the kitchen
37. Let’s cook! 1. Hosting Environment 2. Cookbook Development 3. Monolithic cookbook approach 4. Everything is a cookbook 5. Production Cookbook Deployment 6. Production Cookbook Development 7. Opscode packages 8. Developers’ isolated ruby environment 9. Final Steps 10. Conclusions
38. Chef Server Internals
39. 1.- Hosting Environment (I) Open Source chef-server in a centos6. Modify and test undocumented chef-server config, with some “little” external problems: - shared server behind a VPN - unavailable ports: reassign 80/443 to 81/8443 - 8000 not opened in firewall (so no reports will be available)
40. Asking for opening ports in firewall http://devopsreactions.tumblr.com/post/41094252078/asking-the-security-team-for-a-firewall-exception
41. 2.- Cookbook Development (I) Opscode free(*) hosting • Great help because Multiproject! • http://learn.getchef.com • Berkshelf helped a lot with dependencies (**) • Good practices freezing versions (*) not open-source (limited to 10 nodes) (**) not so easy when not opscode server
42. 2.- Cookbook Development (II) Chef-DK: • ruby binaries in specific /opt folders • not needed rvm/rbenv/ruby source code/... • own path & pre-installed useful gems and tools
43. 2.- Cookbook Development (III) • http://jtimberman.housepub.org/blog/2014/04/30/chefdk- and-ruby/ • Berkshelf 3.0. • The Test Kitchen integration testing framework. • ChefSpec, for unit testing cookbooks. • Foodcritic, static code analysis on cookbooks. • All of the Chef tools you're already familiar with: Chef Client, Knife, Ohai and Chef Zero.
44. 2.- Cookbook Development (IV) Better practices? Any? • No time for tests nor TDD (to learn about) • Not enough RAM in laptop nor in AWS micro instances for making testing/CI with Vagrant • Foodcritic and more, but later • I hope to use them ASAP: just because I like it. The same with Puppet.
45. 2.- Cookbook Development (V) First stages of development • make risks, quick tests, quickwin, try and try • think in advance, virtualhosts library • ugly code you know it will be easy to change • the community code is better than yours • extend, not create from the ground • parametrize cookbooks: redis
46. 3.- Monolithic cookbook approach Thanks a lot to Mathias Lafeldt @mlafeldt “monolithic cookbook” idea, easy to refactor later (see Puppet Module Structure Redux). • http://mlafeldt.github.io/practicing-ruby- cookbook • https://github.com/elm-city- craftworks/practicing-ruby-cookbook
47. 4.- Everything is a cookbook • https://tomduffield.com/everything-as-a- cookbook-chefconf2014/ • https://speakerdeck.com/tduffield/everything- as-a-cookbook-1
48. Cool guys don’t look back to explosions?
49. 5.-ProductionCookbookDevelopment • A new beginning. Almost from scratch. Develop new recipes and cookbooks against a new chef organization in opscode free hosting. • install, fix, check-apply and repeat in frontend staging instances (Poor’s man plan–do–check–adjust) https://en.wikipedia.org/wiki/PDCA
50. 6.-ProductionCookbookDeployment • chef site cookbook install “cookbook” (new git branch in ~/chef-repo/cookbook/) • chef site cookbook install chef-client
51. 7.- Opscode packages (I) • WARNING: chefdk.rpm installs binaries under /usr/bin and gems in its own gemsdir • REMOVE any puppet or ruby preexistent package or binary, no interferences please! • BEWARE: never use binaries without explicit PATH e.g. /opt/chef/bin/chef-client
52. 7.- Opscode packages (II) NO RVM nor system ruby allowed • chef-server.rpm • chef.rpm • chefdk.rpm every opscode rpm package installs one or more ruby binaries in its own path
53. 8.- Developers isolated ruby environment (I) • Developers need multiple rubies installed (1.9.3 & 2.0.X) • unprivileged users will install its own gems, without interfering with chef ruby-binaries SOLUTION: compile explicit versions from source code, rewrite PATH in user environment and install “bundler gem” as root.
54. 8.- Developers isolated ruby environment (II) Opscode chef.rpm package provides 2.1 ruby • install passenger-apache library+binary compiled and installed as a gem under chef. rpm gemlib path (a community recipe is in charge of it) • compatible to both 2.0.x and 1.9.x user rubies
55. 8.- Developers isolated ruby environment(III) Explicit ruby version (1.9.3 or 2.0.X) in appropriate virtualhost apache files (from templates) NOTE: Foodcritic tool helped a lot when looking for errors (e.g. specially with chef templates)
56. 9. Final Steps (I) After just 2 weeks, it almost worked ok. Traditional approach to development: logrotate and monit forked recipes with “999” suffix added to “semver” in metadata.rb • not perfect • Explicit installation of dependencies when in Opensource Opscode server
57. 9. Final Steps (II) Pareto principle! 80/20 Rule. • Two more weeks with “fringes” • create new organization in opscode free account • chef-solo-search for “local” databags • BIG CRISIS: no chef-server available -> chef-solo deployment
58. Angry chef?
59. 9. Final Steps (III) Create git repo (first commit in 2 weeks) • install explicit dependencies • “chef cookbook site install” • only needed “ancient” (fork and modify) approach in two cookbooks: monit and passenger
60. 9. Final Steps (IV) “In extremis”: Refactor the “monolithic” cookbook Monolithic cookbook refactored as 3 cookbooks Three layers of attribute+template files: • general cookbook (i.e. language, servers,...) ◦ company cookbook (i.e. final customer specific) ▪ project cookbook (i.e. virtualhost config)
61. 9. Final Steps (V) • only two “old fashioned” forked recipes: monit and passenger • new recipes forces a different way to use databags
62. 10. Conclusions (I) To FIX: • move logrotate recipes in railssites2 to another new bootstrapping-node-sysadmin cookbook • not exactly applied but inspired in "everything is a cookbook"
63. 10. Conclusions (II) • You need to have “Luck” ◦ no way to configure staging chef-server (only working through 127.0.0.1 and VPN) ◦ no way to configure ubuntu from opscode deb packages • You need big motivation to success
64. 10. Conclusions (III) To be improved: • ubuntu/RHEL cookbooks • tests • mysql m/s cookbook • monit recipe • opensource chef-server hosting
66. THANKS FOR YOUR PATIENCE