The Unavoidable Big Bang

The Unavoidable Big Bang

Talk about launching GOV.UK and working in a large organisation. Presented at RAMP in Budapest.

98234c645fe8c935edc0fec0186d28b8?s=128

Gareth Rushgrove

July 11, 2013
Tweet

Transcript

  1. GDS Gareth Rushgrove The Unavoidable Big Bang and life in

    Government
  2. Who (Who is this person?) GDS Gareth Rushgrove

  3. GDS Gareth Rushgrove Gareth Rushgrove Technical Architect Government Digital Service

    @garethr
  4. GDS Gareth Rushgrove Agencies Startups Big companies Myself Worked for

  5. GDS Gareth Rushgrove

  6. GDS Gareth Rushgrove Last code I wrote

  7. GDS Gareth Rushgrove

  8. GDS Gareth Rushgrove

  9. GDS Gareth Rushgrove

  10. GDS Gareth Rushgrove

  11. What (What will I get from this talk?) GDS Gareth

    Rushgrove
  12. GDS Gareth Rushgrove Reasons to avoid a big bang release

    1
  13. GDS Gareth Rushgrove How to make a big bang release

    work if you have to 2
  14. GDS Gareth Rushgrove How to work in a large organisation

    3
  15. GDS Gareth Rushgrove And a desire to work in the

    public sector (maybe) 4
  16. Background (Government and GOV.UK) GDS Gareth Rushgrove

  17. GDS Gareth Rushgrove October 2010

  18. GDS Gareth Rushgrove June 2011

  19. GDS Gareth Rushgrove January 2012

  20. GDS (Government Digital Service) GDS Gareth Rushgrove

  21. GDS Gareth Rushgrove

  22. GDS Gareth Rushgrove October 2012

  23. GDS Gareth Rushgrove Tools as well as content

  24. GDS Gareth Rushgrove Award winning

  25. GDS Gareth Rushgrove http://www.flickr.com/photos/psd/9099796942

  26. By the numbers (The size of the problem) GDS Gareth

    Rushgrove
  27. 38 GDS Gareth Rushgrove Weeks since launch

  28. 2 GDS Gareth Rushgrove Sites closed on day one

  29. GDS Gareth Rushgrove 59 Sites closed since

  30. 222 GDS Gareth Rushgrove Subdomains closed

  31. GDS Gareth Rushgrove 139 Million visits since launch

  32. 300 GDS Gareth Rushgrove Thousand redirects

  33. GDS Gareth Rushgrove 10-100 Members of the team

  34. GDS Gareth Rushgrove http://www.flickr.com/photos/psd/8756580339

  35. Don’t do this at home (Unless you have to) GDS

    Gareth Rushgrove
  36. GDS Gareth Rushgrove Find alternatives

  37. GDS Gareth Rushgrove Start small?

  38. GDS Gareth Rushgrove Replace one site at a time?

  39. GDS Gareth Rushgrove Percentage of visitors?

  40. GDS Gareth Rushgrove

  41. GDS Gareth Rushgrove http://www.flickr.com/photos/psd/9116635297

  42. GDS Gareth Rushgrove Existing contracts with associated dates

  43. GDS Gareth Rushgrove Public commitments

  44. GDS Gareth Rushgrove http://www.flickr.com/photos/psd/9120523574

  45. GDS Gareth Rushgrove http://www.flickr.com/photos/psd/9104280608

  46. Do it in public (Running an alpha and beta) GDS

    Gareth Rushgrove
  47. GDS Gareth Rushgrove http://www.flickr.com/photos/psd/9122642253

  48. GDS Gareth Rushgrove

  49. GDS Gareth Rushgrove

  50. GDS Gareth Rushgrove Not just the service

  51. GDS Gareth Rushgrove Blogging

  52. GDS Gareth Rushgrove Code

  53. GDS Gareth Rushgrove Publishing tools

  54. GDS Gareth Rushgrove Frontend applications

  55. GDS Gareth Rushgrove Developer environment

  56. GDS Gareth Rushgrove Handy utilities

  57. GDS Gareth Rushgrove Project backlogs

  58. GDS Gareth Rushgrove Performance data

  59. GDS Gareth Rushgrove

  60. GDS Gareth Rushgrove

  61. GDS Gareth Rushgrove But didn’t being open cause problems?

  62. GDS Gareth Rushgrove No This slide stolen from @tomskitomski

  63. Make change easy (Technology AND process) GDS Gareth Rushgrove

  64. GDS Gareth Rushgrove One click deploy

  65. GDS Gareth Rushgrove Single place to deploy

  66. GDS Gareth Rushgrove Not just applications

  67. GDS Gareth Rushgrove Configuration management

  68. GDS Gareth Rushgrove

  69. GDS Gareth Rushgrove package { 'apache2': ensure => latest, }

    service { 'apache2': ensure => running, provider => upstart, require => Package['apache2'] }
  70. class govuk::apps::calendars( $port = 3011 ) { govuk::app { 'calendars':

    app_type => 'rack', port => $port, health_check_path => ‘/bank-holidays’, } } GDS Gareth Rushgrove Higher level constructs
  71. GDS Gareth Rushgrove Infrastructure not just configuration

  72. GDS Gareth Rushgrove Provisioning APIs

  73. GDS Gareth Rushgrove Fog Libcloud VCloud AWS Used at different

    times
  74. require 'rubygems' require 'nat' nat do snat :interface => "Client

    Data", :original => { :ip => "10.0.0.0/xx" }, :translated => { :ip => "xx.xx.xx.xx" }, :desc => "Outbound internet traffic" dnat :interface => "Client Data", :original => { :ip => "xx.xx.xx.xx", :port => 22 }, :translated => { :ip => "10.0.0.xx", :port => 22 }, :desc => "jumpbox-1 SSH" dnat :interface => "Client Data", :original => { :ip => "xx.xx.xx.xx", :port => 80 },, :translated => { :ip => "10.0.0.xx", :port => 80 }, :desc => "jenkins, logging, monitoring HTTP" GDS Gareth Rushgrove Network in code
  75. require 'rubygems' require 'firewall' firewall do # internal rules rule

    "ssh access to jumpbox1" do source :ip => "Any" destination :ip => "xx.xx.xx.xx", :port => 22 end rule "http to backend applications" do source :ip => "Any" destination :ip => "xx.xx.xx.xx", :port => 80 end rule "https to backend applications" do GDS Gareth Rushgrove Firewalls in code
  76. GDS Gareth Rushgrove Requires rolling your own code at times

  77. GDS Gareth Rushgrove Developers want Visibility of deploys

  78. GDS Gareth Rushgrove Organisation want Auditability of deploys

  79. GDS Gareth Rushgrove App showing deploys

  80. GDS Gareth Rushgrove Embrace process discussions

  81. GDS Gareth Rushgrove Change control

  82. GDS Gareth Rushgrove Risk management

  83. GDS Gareth Rushgrove Share language

  84. GDS Gareth Rushgrove Optimise only as far as needed

  85. Make change small (Lots of small changes) GDS Gareth Rushgrove

  86. GDS Gareth Rushgrove Change one thing at once

  87. GDS Gareth Rushgrove http://www.flickr.com/photos/fatty/9158066939 We use a physical token

  88. GDS Gareth Rushgrove

  89. GDS Gareth Rushgrove Regular releases reduce risk

  90. GDS Gareth Rushgrove Back to GOV.UK

  91. GDS Gareth Rushgrove

  92. GDS Gareth Rushgrove We changed less software on the day

    of launch than probably any day before or since
  93. GDS Gareth Rushgrove DNS?

  94. GDS Gareth Rushgrove Content Delivery Network

  95. GDS Gareth Rushgrove http://www.flickr.com/photos/psd/8040082207

  96. GDS Gareth Rushgrove Load testing

  97. GDS Gareth Rushgrove http://www.flickr.com/photos/psd/8040667413 Replay traffic

  98. GDS Gareth Rushgrove Defend yourself from load tests

  99. http { proxy_intercept_errors on; limit_req_zone $binary_remote_addr zone=rat limit_conn_zone $binary_remote_addr zone=co

    } Rate/connection limiting in nginx GDS Gareth Rushgrove
  100. GDS Gareth Rushgrove Web application firewall

  101. Embrace failures (but stop them affecting people) GDS Gareth Rushgrove

  102. GDS Gareth Rushgrove Serve stale content

  103. sub vcl_recv { if (req.backend.healthy) { set req.grace = 30s;

    } else { set req.grace = 24h; } } sub vcl_fetch { set beresp.grace = 24h; } Grace mode in Varnish GDS Gareth Rushgrove
  104. Except when you can’t GDS Gareth Rushgrove

  105. GDS Gareth Rushgrove Partial page failures

  106. GDS Gareth Rushgrove

  107. GDS Gareth Rushgrove Making change easy makes fixing problems fast

  108. GDS Gareth Rushgrove Critical issues fixed in hours or minutes

  109. GDS Gareth Rushgrove Measure everything

  110. GDS Gareth Rushgrove

  111. GDS Gareth Rushgrove Long tail of failures

  112. GDS Gareth Rushgrove Allow for investigation

  113. GDS Gareth Rushgrove Make monitoring easy

  114. @normal Scenario: check quick answers load When I visit "/vat-rates"

    Then I should see "VAT rates" GDS Gareth Rushgrove Run as smoke tests and monitoring
  115. @high Scenario: check quick answers load When I visit "/vat-rates"

    Then I should see "VAT rates" GDS Gareth Rushgrove Change who responds to issue
  116. Inside Government (or any large organisation) GDS Gareth Rushgrove

  117. GDS Gareth Rushgrove Involve technology colleagues

  118. GDS Gareth Rushgrove Involve security and auditing colleagues

  119. GDS Gareth Rushgrove Involve procurement colleagues

  120. GDS Gareth Rushgrove (Procurement means rules for buying things)

  121. GDS Gareth Rushgrove Demonstrate capability quickly

  122. Conclusions (What about my project?) GDS Gareth Rushgrove

  123. GDS Gareth Rushgrove Own your own problems 1

  124. GDS Gareth Rushgrove Avoid hard problems when possible 2

  125. GDS Gareth Rushgrove Make things open 3

  126. GDS Gareth Rushgrove Work on business processes AND technology 4

  127. Find out more (A public service announcement) GDS Gareth Rushgrove

  128. GDS Gareth Rushgrove

  129. GDS Gareth Rushgrove Government Service Design Manual

  130. GDS Gareth Rushgrove Even contains a definition of devops

  131. Questions? (and thanks for listening) GDS Gareth Rushgrove

  132. GDS Gareth Rushgrove Gareth Rushgrove Technical Architect Government Digital Service

    @garethr