Infrastructure Automation with Opsworks Chef

An infrastructure solution presentation to Ruby Developer Meetup

Transcript

1. Infrastructure Automation with Opsworks Chef … IMRAN AHMED e : imran@crichq.com t : reallyimran
2. imran_ahmed.inspect • Work at crichq.com • Sysadmin since 2001 • Not a Programmer • Not a Ruby Developer • Best tool for the Job = Bash + PHP
3. imran_ahmed.inspect • Work at crichq.com • Sysadmin since 2001 • Not a Programmer • Not a Ruby Developer • Best tool for the Job = Bash + PHP • But now Ruby ?!! more...
4. What Is CricHQ? CricHQ is a technology platform built for cricket benefiting stakeholders including: • Administrators • Players • Fans • Coaches • Governing Bodies
5. Social Network Aggregate cricket content on personalised news feeds Scoring Capture and share scorecards and elite level data Administration Platform Create, manage, publish and publicise competitions and leagues How It Works
6. Modern Web Apps Requirements • Reliable • Scalable • Operations Challenges : ◦ Configuration Management ▪ Configuration Drift ◦ Centralized Management ◦ Automation of processes ▪ deployment ▪ retirement ▪ scaling ▪ security ▪ user management ▪ data management
7. What is Opsworks? “ AWS OpsWorks is an application management service that makes it easy to deploy and operate applications of all shapes and sizes.” -- AWS “ Free Hosted Chef server” -- Me Success Stories : • CricHQ • Artsy • Intuit • Many more...
8. Why Opsworks • Unified Control of Components • Scaling ◦ Load Based ◦ Time Based • Automation of Everything • Infrastructure as Code ◦ Versionable ◦ Testable ◦ Cloneable ◦ Shareable • EBS / ElasticIP / ELB Resource Management • User / Firewall Permission • Free Chef
9. Why Opsworks For Us ? • Capistrano Issues ◦ Now : No puppet Infrastructure to host ◦ Now : Zero Downtime Deployment Architecture • Light ◦ 1 Sysadmin for more than 150 resources, could easily handle MORE • It’s Ruby! ◦ based on Chef ( Ruby DSL ) ◦ First incarnation created by : Daniel Zollinger
10. Typical Cloud App Firewall Load Balancer
11. Opsworks Architecture Firewall Load Balancer Security ELB App Layer Layer Groups or RDS Stack
12. Scenerio : • Got a brand new rails application ◦ deploy it Opsworks Architecture
13. Opsworks Architecture
14. Opsworks Architecture
15. Opsworks Architecture
16. Opsworks Architecture ruby 1.9.3 / 2.0 / 2.1 supported
17. Opsworks Architecture
18. Opsworks Architecture
19. Opsworks Architecture
20. Opsworks Architecture
21. Opsworks Architecture
22. Opsworks Architecture
23. Opsworks Architecture
24. Opsworks Architecture
25. Opsworks Architecture cookbook/ssh_users/recipes/default.rb group 'opsworks' ## chef DSL existing_ssh_users = load_existing_ssh_users ## pure ruby ... existing_ssh_users.each do |id, name| unless node[:ssh_users][id] teardown_user(name) end end node[:ssh_users].each_key do |id| ….
26. Opsworks Architecture cookbook/ssh_users/libraries/user.rb module OpsWorks module User def load_existing_ssh_users return {} unless node[:opsworks_gid] existing_ssh_users = {} (node[:passwd] || node[:etc][:passwd]).each do |username, entry| if entry[:gid] == node[:opsworks_gid] existing_ssh_users[entry[:uid].to_s] = username end end …
27. Opsworks Architecture Useful Resources: package “mysql” do action :install end file , directory, bash, ruby , python , template , service , cron , user
28. Opsworks Architecture Solution : Implement Memcached Dalli Cache Stores
29. Opsworks Architecture
30. Opsworks Architecture Gemfile gem ‘dalli’ config/environments/development.rb config.cache_store = :dalli_store, 'cache-1.example. com', 'cache-2.example.com', { :namespace => NAME_OF_RAILS_APP, :expires_in => 1.day, :compress => true }
31. Opsworks Architecture solution : config/dalli.yml + via initializer defaults: &defaults compress: true development: options: <<: *defaults namespace: CRICHQ_DEVELOPMENT servers: - server-1:11211 - server-2:11211
32. Opsworks Architecture Possible Solutions : • Environment Variables ◦ No built in solution. Community cookbooks available • Generate Files From node.attributes ◦ cookbook attributes ◦ infrastructure key value ◦ custom JSON ▪ deployments ▪ stack settings ▪ commands
33. Opsworks Architecture Hooks Available : Capistrano Based Instance Lifecycle Based app_folder/deploy/before_restart.rb cookbooks/ssh_users/recipes/default.rb cookbooks/unicorn/recipes/rails.rb
34. Opsworks Architecture wellrailed_demo_application_cookbooks └── dalli ├── attributes │ └── dalli.rb ## key values ├── recipes │ └── dalli.rb ## instructions └── templates └── default └── dalli.erb ## config file template ├── redis ... └── resque ...
35. Opsworks Architecture appfolder/deploy/before_restart.rb memcservers = search(:node, "role:memcached") template "#{current_release}/config/dalli.yml" do source 'dalli.erb' cookbook "dalli" owner deploy_user group deploy_group mode 00660 variables { :servers => memcservers , :env => deploy_node[:deploy][app_name][:rails_env] } action :create end
36. What have we automated ? • Continuous Integration ◦ Automated Deployment ◦ Release Log Generation ◦ Release Email Notifications ◦ JIRA automation • Feature Branch Stack Creation
37. Questions? e : imran@crichq.com t : reallyimran