Keep Ops Happy: Designing For Production

Talk I did for Mountain West Ruby Conf 2013 about code patterns and snippets I've used to help with deploying and managing applications in production.

Transcript

1. KEEP OPS HAPPY! Designing for Production Collective Idea Jason Roelofs @jasonroelofs
3. DOCUMENT ALL THE THINGS
4. Documentation What does it do? Three questions all documentation should answer How do I get it running on my machine? What does it need?
5. DEPLOY EARLY DEPLOY OFTEN
6. CONFIGURATION
7. class RedisService HOST = "redis.host" USERNAME = "user" PASSWORD = "Password1!" end Configuration class RedisService USERNAME = "user" if Rails.env.production? HOST = "..." PASSWORD = “Password1!" elsif Rails.env.staging? HOST = "..." PASSWORD = "Password2!" else ... end end
8. Configuration config = YAML.load_file( Rails.root.join("config", "redis.yml") )[Rails.env] config["host"] # 1.2.3.4 # config/redis.yml production: host: 1.2.3.4 username: user password: Password1!
9. ENV["REDIS_URL"] Figaro.env.redis_url Figaro https://github.com/laserlemon/figaro Configuration # config/application.yml production: REDIS_HOST: 1.2.3.4 REDIS_HOST: username REDIS_HOST: Password!3
10. RAILS.ENV
11. def domain if Rails.env.development? "site.dev" elsif Rails.env.staging? "staging.site.com" elsif Rails.env.qa? "qa.site.com" elsif Rails.env.production? "site.com" end end Rails.env def domain Figaro.env.site_domain # ENV[‘SITE_DOMAIN’] end
12. if %w(development test).include?(Rails.env) # Do some stuff in dev else # Do something else in production end Rails.env I test my code in production!
13. THREAD SAFETY
14. Thread Safety Share Nothing!
15. before_filter :set_context def set_context if logged_in? Thread.current[:user_id] = current_user.id end end Thread Safety: Thread.current around_filter :set_context def set_context if logged_in? Thread.current[:user_id] = current_user.id end yield Thread.current[:user_id] = nil end
16. Thread Safety: Thread.current before_filter :set_context def set_context # Always clear at the beginning of a request Thread.current[:user_id] = nil if logged_in? Thread.current[:user_id] = current_user.id end end
17. SECURITY
18. Security: Attributes class User < ActiveRecord::Base attr_accessible :name, :login, :email end class UserController < ActionController::Base def create @user = User.create(params[:user]) end end This approach considered deprecated.
19. Security: Attributes class UserController < ActionController::Base def create @user = User.create(user_params) end def user_params params. require(:user). permit(:name, :login, :email) end end Strong Parameters https://github.com/rails/strong_parameters
20. Security: Passwords MD5, SHA, SHA256, SHA512 1 GPU* MD5 SHA SHA256 SHA512 426 million / s 85 million / s 65 million / s 13 million / s GPU cluster MD5 SHA SHA256 SHA512 BILLIONS PER SECOND * http://www.insidepro.com/eng/egb.shtml
21. Security: Passwords USE BCRYPT! (or scrypt) Generating 100 hashes at increasing costs: Rehearsal ------------------------------------------------------- Cost of 5 0.250000 0.000000 0.250000 ( 0.249723) Cost of 10 7.740000 0.010000 7.750000 ( 7.879849) Cost of 15 247.510000 0.460000 247.970000 (255.346897) -------------------------------------------- total: 255.970000sec user system total real Cost of 5 0.250000 0.000000 0.250000 ( 0.272549) Cost of 10 7.750000 0.030000 7.780000 ( 8.442511) Cost of 15 247.530000 0.480000 248.010000 (254.815985) http://yorickpeterse.com/articles/use-bcrypt-fool/
22. Security: Serialization Don’t! My first rule of serialization: Only use base types of the language
23. session[:current_car] = current_car Security: Serialization Coupling data with implementation class Car < AR::Base end class Vehicle < AR::Base end Class gets renamed
24. class User < ActiveRecord::Base has_many :followers end Background.enqueue(DoStuffJob, @user) Overflowed a MySQL text field. Instead, send the minimum required information. Security: Serialization Background.enqueue(DoStuffJob, @user.id) Death by Eager Loading Recursion
25. Security: Other Situations Cross Site Scripting (XSS) IFrame Click Jacking Cross-Site Request Forgery (CSRF) Path Traversal Attacks IP Spoofing Session / Cookie Hijacking Grey Hat Security Researchers
26. Security: Other Situations
27. # config.ru use Rack::Protection run RailsApplication Rack-Protection https://github.com/rkh/rack-protection Security: Other Situations
28. Security: Automated Scripts Added a Harmony Security Page in light of Rails issues. We’ve been DoS’d repeatedly since then.
29. Rack-Attack https://github.com/kickstarter/rack-attack Security: Automated Scripts Rack::Attack.throttle( "post-spam", :limit => 20, :period => 60.seconds ) do |request| if request.post? && request.path =~ %r{^/comments} request.env["action_dispatch.remote_ip"] end end ActiveSupport::Notifications.subscribe("rack.attack") do |*args| Stats.increment("request.throttled") end config.middleware.insert_after( ActionDispatch::RemoteIp, Rack::Attack )
30. Security: Automated Scripts Now happily throttling over-eager scripts!
31. Security: Automated Scripts Honey Pots for fail-fast <input type='text' name='color_of_sky' style='display: none; visiblity: hidden'/> if params[:color_of_sky].present? # out out damn bot! end
32. QUICK TIPS
33. Quick Tips: File Storage Use Amazon’s S3, Rackspace Cloud, etc Don’t store content on the file system. It’s constantly getting cheaper and they scale very well.
34. Quick Tips: Scheduling Jobs Don’t schedule jobs between 1:00 am and 3:00 am Cause these times either don’t exist, or happen twice! DST Starts 1:59:59 am 3:00:00 am DST Ends 1:59:59 am 1:00:00 am
35. Quick Tips: Billing Use Stripe, Braintree, Spreedly, etc Don’t write your own billing engine! Money is by far the worst thing I’ve had to deal with.
36. SERVICES
37. •Pingdom - Uptime Monitoring •NewRelic - Performance Monitoring •Instrumental - Stats Gathering and Graphing •Loggly - Centralized Logging Aggregation •Papertrail - Centralized Logging Aggregation Services
38. Services Dead Man’s Snitch rake some_daily_task && curl https://nosnch.in/abcd1234
39. Jason Roelofs collectiveidea.com jasonroelofs.com github.com/jasonroelofs @jasonroelofs