The Unavoidable Big Bang

Talk about launching GOV.UK and working in a large organisation. Presented at RAMP in Budapest.

Transcript

1. GDS Gareth Rushgrove The Unavoidable Big Bang and life in Government
2. Who (Who is this person?) GDS Gareth Rushgrove
3. GDS Gareth Rushgrove Gareth Rushgrove Technical Architect Government Digital Service @garethr
4. GDS Gareth Rushgrove Agencies Startups Big companies Myself Worked for
5. GDS Gareth Rushgrove
6. GDS Gareth Rushgrove Last code I wrote
7. GDS Gareth Rushgrove
8. GDS Gareth Rushgrove
9. GDS Gareth Rushgrove
10. GDS Gareth Rushgrove
11. What (What will I get from this talk?) GDS Gareth Rushgrove
12. GDS Gareth Rushgrove Reasons to avoid a big bang release 1
13. GDS Gareth Rushgrove How to make a big bang release work if you have to 2
14. GDS Gareth Rushgrove How to work in a large organisation 3
15. GDS Gareth Rushgrove And a desire to work in the public sector (maybe) 4
16. Background (Government and GOV.UK) GDS Gareth Rushgrove
17. GDS Gareth Rushgrove October 2010
18. GDS Gareth Rushgrove June 2011
19. GDS Gareth Rushgrove January 2012
20. GDS (Government Digital Service) GDS Gareth Rushgrove
21. GDS Gareth Rushgrove
22. GDS Gareth Rushgrove October 2012
23. GDS Gareth Rushgrove Tools as well as content
24. GDS Gareth Rushgrove Award winning
25. GDS Gareth Rushgrove http://www.flickr.com/photos/psd/9099796942
26. By the numbers (The size of the problem) GDS Gareth Rushgrove
27. 38 GDS Gareth Rushgrove Weeks since launch
28. 2 GDS Gareth Rushgrove Sites closed on day one
29. GDS Gareth Rushgrove 59 Sites closed since
30. 222 GDS Gareth Rushgrove Subdomains closed
31. GDS Gareth Rushgrove 139 Million visits since launch
32. 300 GDS Gareth Rushgrove Thousand redirects
33. GDS Gareth Rushgrove 10-100 Members of the team
34. GDS Gareth Rushgrove http://www.flickr.com/photos/psd/8756580339
35. Don’t do this at home (Unless you have to) GDS Gareth Rushgrove
36. GDS Gareth Rushgrove Find alternatives
37. GDS Gareth Rushgrove Start small?
38. GDS Gareth Rushgrove Replace one site at a time?
39. GDS Gareth Rushgrove Percentage of visitors?
40. GDS Gareth Rushgrove
41. GDS Gareth Rushgrove http://www.flickr.com/photos/psd/9116635297
42. GDS Gareth Rushgrove Existing contracts with associated dates
43. GDS Gareth Rushgrove Public commitments
44. GDS Gareth Rushgrove http://www.flickr.com/photos/psd/9120523574
45. GDS Gareth Rushgrove http://www.flickr.com/photos/psd/9104280608
46. Do it in public (Running an alpha and beta) GDS Gareth Rushgrove
47. GDS Gareth Rushgrove http://www.flickr.com/photos/psd/9122642253
48. GDS Gareth Rushgrove
49. GDS Gareth Rushgrove
50. GDS Gareth Rushgrove Not just the service
51. GDS Gareth Rushgrove Blogging
52. GDS Gareth Rushgrove Code
53. GDS Gareth Rushgrove Publishing tools
54. GDS Gareth Rushgrove Frontend applications
55. GDS Gareth Rushgrove Developer environment
56. GDS Gareth Rushgrove Handy utilities
57. GDS Gareth Rushgrove Project backlogs
58. GDS Gareth Rushgrove Performance data
59. GDS Gareth Rushgrove
60. GDS Gareth Rushgrove
61. GDS Gareth Rushgrove But didn’t being open cause problems?
62. GDS Gareth Rushgrove No This slide stolen from @tomskitomski
63. Make change easy (Technology AND process) GDS Gareth Rushgrove
64. GDS Gareth Rushgrove One click deploy
65. GDS Gareth Rushgrove Single place to deploy
66. GDS Gareth Rushgrove Not just applications
67. GDS Gareth Rushgrove Configuration management
68. GDS Gareth Rushgrove
69. GDS Gareth Rushgrove package { 'apache2': ensure => latest, } service { 'apache2': ensure => running, provider => upstart, require => Package['apache2'] }
70. class govuk::apps::calendars( $port = 3011 ) { govuk::app { 'calendars': app_type => 'rack', port => $port, health_check_path => ‘/bank-holidays’, } } GDS Gareth Rushgrove Higher level constructs
71. GDS Gareth Rushgrove Infrastructure not just configuration
72. GDS Gareth Rushgrove Provisioning APIs
73. GDS Gareth Rushgrove Fog Libcloud VCloud AWS Used at different times
74. require 'rubygems' require 'nat' nat do snat :interface => "Client Data", :original => { :ip => "10.0.0.0/xx" }, :translated => { :ip => "xx.xx.xx.xx" }, :desc => "Outbound internet traffic" dnat :interface => "Client Data", :original => { :ip => "xx.xx.xx.xx", :port => 22 }, :translated => { :ip => "10.0.0.xx", :port => 22 }, :desc => "jumpbox-1 SSH" dnat :interface => "Client Data", :original => { :ip => "xx.xx.xx.xx", :port => 80 },, :translated => { :ip => "10.0.0.xx", :port => 80 }, :desc => "jenkins, logging, monitoring HTTP" GDS Gareth Rushgrove Network in code
75. require 'rubygems' require 'firewall' firewall do # internal rules rule "ssh access to jumpbox1" do source :ip => "Any" destination :ip => "xx.xx.xx.xx", :port => 22 end rule "http to backend applications" do source :ip => "Any" destination :ip => "xx.xx.xx.xx", :port => 80 end rule "https to backend applications" do GDS Gareth Rushgrove Firewalls in code
76. GDS Gareth Rushgrove Requires rolling your own code at times
77. GDS Gareth Rushgrove Developers want Visibility of deploys
78. GDS Gareth Rushgrove Organisation want Auditability of deploys
79. GDS Gareth Rushgrove App showing deploys
80. GDS Gareth Rushgrove Embrace process discussions
81. GDS Gareth Rushgrove Change control
82. GDS Gareth Rushgrove Risk management
83. GDS Gareth Rushgrove Share language
84. GDS Gareth Rushgrove Optimise only as far as needed
85. Make change small (Lots of small changes) GDS Gareth Rushgrove
86. GDS Gareth Rushgrove Change one thing at once
87. GDS Gareth Rushgrove http://www.flickr.com/photos/fatty/9158066939 We use a physical token
88. GDS Gareth Rushgrove
89. GDS Gareth Rushgrove Regular releases reduce risk
90. GDS Gareth Rushgrove Back to GOV.UK
91. GDS Gareth Rushgrove
92. GDS Gareth Rushgrove We changed less software on the day of launch than probably any day before or since
93. GDS Gareth Rushgrove DNS?
94. GDS Gareth Rushgrove Content Delivery Network
95. GDS Gareth Rushgrove http://www.flickr.com/photos/psd/8040082207
96. GDS Gareth Rushgrove Load testing
97. GDS Gareth Rushgrove http://www.flickr.com/photos/psd/8040667413 Replay traffic
98. GDS Gareth Rushgrove Defend yourself from load tests
99. http { proxy_intercept_errors on; limit_req_zone $binary_remote_addr zone=rat limit_conn_zone $binary_remote_addr zone=co } Rate/connection limiting in nginx GDS Gareth Rushgrove
100. GDS Gareth Rushgrove Web application firewall
101. Embrace failures (but stop them affecting people) GDS Gareth Rushgrove
102. GDS Gareth Rushgrove Serve stale content
103. sub vcl_recv { if (req.backend.healthy) { set req.grace = 30s; } else { set req.grace = 24h; } } sub vcl_fetch { set beresp.grace = 24h; } Grace mode in Varnish GDS Gareth Rushgrove
104. Except when you can’t GDS Gareth Rushgrove
105. GDS Gareth Rushgrove Partial page failures
106. GDS Gareth Rushgrove
107. GDS Gareth Rushgrove Making change easy makes fixing problems fast
108. GDS Gareth Rushgrove Critical issues fixed in hours or minutes
109. GDS Gareth Rushgrove Measure everything
110. GDS Gareth Rushgrove
111. GDS Gareth Rushgrove Long tail of failures
112. GDS Gareth Rushgrove Allow for investigation
113. GDS Gareth Rushgrove Make monitoring easy
114. @normal Scenario: check quick answers load When I visit "/vat-rates" Then I should see "VAT rates" GDS Gareth Rushgrove Run as smoke tests and monitoring
115. @high Scenario: check quick answers load When I visit "/vat-rates" Then I should see "VAT rates" GDS Gareth Rushgrove Change who responds to issue
116. Inside Government (or any large organisation) GDS Gareth Rushgrove
117. GDS Gareth Rushgrove Involve technology colleagues
118. GDS Gareth Rushgrove Involve security and auditing colleagues
119. GDS Gareth Rushgrove Involve procurement colleagues
120. GDS Gareth Rushgrove (Procurement means rules for buying things)
121. GDS Gareth Rushgrove Demonstrate capability quickly
122. Conclusions (What about my project?) GDS Gareth Rushgrove
123. GDS Gareth Rushgrove Own your own problems 1
124. GDS Gareth Rushgrove Avoid hard problems when possible 2
125. GDS Gareth Rushgrove Make things open 3
126. GDS Gareth Rushgrove Work on business processes AND technology 4
127. Find out more (A public service announcement) GDS Gareth Rushgrove
128. GDS Gareth Rushgrove
129. GDS Gareth Rushgrove Government Service Design Manual
130. GDS Gareth Rushgrove Even contains a definition of devops
131. Questions? (and thanks for listening) GDS Gareth Rushgrove
132. GDS Gareth Rushgrove Gareth Rushgrove Technical Architect Government Digital Service @garethr