Zero Downtime Deploys with Rails

http://mattduncan.org/talks/zero-downtime-deploys/

Every deploy introduces the risk of downtime because of changes which are not backwards compatible. At Yammer, we deploy our Rails codebase to hundreds of servers many times a week. In this talk, I’ll discuss many of the strategies we use to mitigate downtime during deploys. This includes how we handle database changes as well as background workers and external services.

Transcript

1. Zero Downtime Deploys Made Easy
2. Matt Duncan RAILS TEAM at Yammer
4. There is no silver bullet.
5. This is not just a problem with Rails.
6. Database Async Workers External Services
7. Database
8. I want to be able to make users admins.
9. change_table(:users) do |t| t.column :admin, :boolean, :null => false, :default => false end
10. rake db:migrate
11. CRITICAL
12. SELECT * FROM pg_locks WHERE granted = true AND mode = 'AccessExclusiveLock';
13. Migrations are transactional.
15. Faster Database
16. change_table(:users) do |t| t.column :admin, :boolean, :default => false end
17. task(:init_admin) do User.find_each(:admin => nil) do |user| user.update_attribute(:admin, false) end end
18. rake db:migrate
19. rake init_admin
20. change_table(:users) do |t| t.column :admin, :boolean, :null => false, :default => false end
21. rake db:migrate
22. UP
23. Whoa.
24. Long Database Locks
25. Non-null constraints Indexes
26. CREATE INDEX CONCURRENTLY
27. Out of Sync Schema
28. PGError: ERROR: column "admin" does not exist
29. INSERT INTO users (email, ..., admin)
30. Renaming Removing Columns
31. Writing to 'admin' column Stop writing to 'admin' column Remove 'admin' column
32. Writing to 'admin' column Stop writing to 'admin' column Remove 'admin' column
33. change_table(:users) do |t| t.column :admin, :boolean, :null => true end
34. class User def self.columns super.reject do |column| column.name == ‘admin’ end end
35. Writing to 'admin' column Stop writing to 'admin' column Remove 'admin' column
36. remove_column :users, :admin
37. Writing to 'admin' column Stop writing to 'admin' column Remove 'admin' column Cleanup
38. NoSQL doesn’t solve this.
39. Asynchronous Workers
40. Queues are not empty.
41. External Services
42. Version Them.
43. API Version 1 Your Application
44. API Version 1 Your Application API Version 2
45. API Version 1 Your Application API Version 2
46. Switches
47. API Version 1 Your Application API Version 2
48. Don’t forget to deploy your services the same way.
49. Lessons Learned
50. Anything can go wrong.
51. Not everything is worth the effort.
52. Make deploys simple and fast.
53. Super awesome stuff
55. Separate migrations from deploys.
57. Roll out services gradually.
58. https://github.com/jrallison/rollout_ui
59. Questions?
60. How can frameworks make this easier? How can databases?
61. Thank You! eng.yammer.com is hiring!