The Unavoidable Big Bang

Transcript

1. 1.

   GDS Gareth Rushgrove The Unavoidable Big Bang and life in

   Government
2. 2.

   Who (Who is this person?) GDS Gareth Rushgrove

3. 3.

   GDS Gareth Rushgrove Gareth Rushgrove Technical Architect Government Digital Service

   @garethr
4. 4.

   GDS Gareth Rushgrove Agencies Startups Big companies Myself Worked for

5. 5.

   GDS Gareth Rushgrove

6. 6.

   GDS Gareth Rushgrove Last code I wrote

7. 7.

   GDS Gareth Rushgrove

8. 8.

   GDS Gareth Rushgrove

9. 9.

   GDS Gareth Rushgrove

10. 10.

    GDS Gareth Rushgrove

11. 11.

    What (What will I get from this talk?) GDS Gareth

    Rushgrove
12. 12.

    GDS Gareth Rushgrove Reasons to avoid a big bang release

    1
13. 13.

    GDS Gareth Rushgrove How to make a big bang release

    work if you have to 2
14. 14.

    GDS Gareth Rushgrove How to work in a large organisation

    3
15. 15.

    GDS Gareth Rushgrove And a desire to work in the

    public sector (maybe) 4
16. 16.

    Background (Government and GOV.UK) GDS Gareth Rushgrove

17. 17.

    GDS Gareth Rushgrove October 2010

18. 18.

    GDS Gareth Rushgrove June 2011

19. 19.

    GDS Gareth Rushgrove January 2012

20. 20.

    GDS (Government Digital Service) GDS Gareth Rushgrove

21. 21.

    GDS Gareth Rushgrove

22. 22.

    GDS Gareth Rushgrove October 2012

23. 23.

    GDS Gareth Rushgrove Tools as well as content

24. 24.

    GDS Gareth Rushgrove Award winning

25. 25.

    GDS Gareth Rushgrove http://www.flickr.com/photos/psd/9099796942

26. 26.

    By the numbers (The size of the problem) GDS Gareth

    Rushgrove
27. 27.

    38 GDS Gareth Rushgrove Weeks since launch

28. 28.

    2 GDS Gareth Rushgrove Sites closed on day one

29. 29.

    GDS Gareth Rushgrove 59 Sites closed since

30. 30.

    222 GDS Gareth Rushgrove Subdomains closed

31. 31.

    GDS Gareth Rushgrove 139 Million visits since launch

32. 32.

    300 GDS Gareth Rushgrove Thousand redirects

33. 33.

    GDS Gareth Rushgrove 10-100 Members of the team

34. 34.

    GDS Gareth Rushgrove http://www.flickr.com/photos/psd/8756580339

35. 35.

    Don’t do this at home (Unless you have to) GDS

    Gareth Rushgrove
36. 36.

    GDS Gareth Rushgrove Find alternatives

37. 37.

    GDS Gareth Rushgrove Start small?

38. 38.

    GDS Gareth Rushgrove Replace one site at a time?

39. 39.

    GDS Gareth Rushgrove Percentage of visitors?

40. 40.

    GDS Gareth Rushgrove

41. 41.

    GDS Gareth Rushgrove http://www.flickr.com/photos/psd/9116635297

42. 42.

    GDS Gareth Rushgrove Existing contracts with associated dates

43. 43.

    GDS Gareth Rushgrove Public commitments

44. 44.

    GDS Gareth Rushgrove http://www.flickr.com/photos/psd/9120523574

45. 45.

    GDS Gareth Rushgrove http://www.flickr.com/photos/psd/9104280608

46. 46.

    Do it in public (Running an alpha and beta) GDS

    Gareth Rushgrove
47. 47.

    GDS Gareth Rushgrove http://www.flickr.com/photos/psd/9122642253

48. 48.

    GDS Gareth Rushgrove

49. 49.

    GDS Gareth Rushgrove

50. 50.

    GDS Gareth Rushgrove Not just the service

51. 51.

    GDS Gareth Rushgrove Blogging

52. 52.

    GDS Gareth Rushgrove Code

53. 53.

    GDS Gareth Rushgrove Publishing tools

54. 54.

    GDS Gareth Rushgrove Frontend applications

55. 55.

    GDS Gareth Rushgrove Developer environment

56. 56.

    GDS Gareth Rushgrove Handy utilities

57. 57.

    GDS Gareth Rushgrove Project backlogs

58. 58.

    GDS Gareth Rushgrove Performance data

59. 59.

    GDS Gareth Rushgrove

60. 60.

    GDS Gareth Rushgrove

61. 61.

    GDS Gareth Rushgrove But didn’t being open cause problems?

62. 62.

    GDS Gareth Rushgrove No This slide stolen from @tomskitomski

63. 63.

    Make change easy (Technology AND process) GDS Gareth Rushgrove

64. 64.

    GDS Gareth Rushgrove One click deploy

65. 65.

    GDS Gareth Rushgrove Single place to deploy

66. 66.

    GDS Gareth Rushgrove Not just applications

67. 67.

    GDS Gareth Rushgrove Configuration management

68. 68.

    GDS Gareth Rushgrove

69. 69.

    GDS Gareth Rushgrove package { 'apache2': ensure => latest, }

    service { 'apache2': ensure => running, provider => upstart, require => Package['apache2'] }
70. 70.

    class govuk::apps::calendars( $port = 3011 ) { govuk::app { 'calendars':

    app_type => 'rack', port => $port, health_check_path => ‘/bank-holidays’, } } GDS Gareth Rushgrove Higher level constructs
71. 71.

    GDS Gareth Rushgrove Infrastructure not just configuration

72. 72.

    GDS Gareth Rushgrove Provisioning APIs

73. 73.

    GDS Gareth Rushgrove Fog Libcloud VCloud AWS Used at different

    times
74. 74.

    require 'rubygems' require 'nat' nat do snat :interface => "Client

    Data", :original => { :ip => "10.0.0.0/xx" }, :translated => { :ip => "xx.xx.xx.xx" }, :desc => "Outbound internet traffic" dnat :interface => "Client Data", :original => { :ip => "xx.xx.xx.xx", :port => 22 }, :translated => { :ip => "10.0.0.xx", :port => 22 }, :desc => "jumpbox-1 SSH" dnat :interface => "Client Data", :original => { :ip => "xx.xx.xx.xx", :port => 80 },, :translated => { :ip => "10.0.0.xx", :port => 80 }, :desc => "jenkins, logging, monitoring HTTP" GDS Gareth Rushgrove Network in code
75. 75.

    require 'rubygems' require 'firewall' firewall do # internal rules rule

    "ssh access to jumpbox1" do source :ip => "Any" destination :ip => "xx.xx.xx.xx", :port => 22 end rule "http to backend applications" do source :ip => "Any" destination :ip => "xx.xx.xx.xx", :port => 80 end rule "https to backend applications" do GDS Gareth Rushgrove Firewalls in code
76. 76.

    GDS Gareth Rushgrove Requires rolling your own code at times

77. 77.

    GDS Gareth Rushgrove Developers want Visibility of deploys

78. 78.

    GDS Gareth Rushgrove Organisation want Auditability of deploys

79. 79.

    GDS Gareth Rushgrove App showing deploys

80. 80.

    GDS Gareth Rushgrove Embrace process discussions

81. 81.

    GDS Gareth Rushgrove Change control

82. 82.

    GDS Gareth Rushgrove Risk management

83. 83.

    GDS Gareth Rushgrove Share language

84. 84.

    GDS Gareth Rushgrove Optimise only as far as needed

85. 85.

    Make change small (Lots of small changes) GDS Gareth Rushgrove

86. 86.

    GDS Gareth Rushgrove Change one thing at once

87. 87.

    GDS Gareth Rushgrove http://www.flickr.com/photos/fatty/9158066939 We use a physical token

88. 88.

    GDS Gareth Rushgrove

89. 89.

    GDS Gareth Rushgrove Regular releases reduce risk

90. 90.

    GDS Gareth Rushgrove Back to GOV.UK

91. 91.

    GDS Gareth Rushgrove

92. 92.

    GDS Gareth Rushgrove We changed less software on the day

    of launch than probably any day before or since
93. 93.

    GDS Gareth Rushgrove DNS?

94. 94.

    GDS Gareth Rushgrove Content Delivery Network

95. 95.

    GDS Gareth Rushgrove http://www.flickr.com/photos/psd/8040082207

96. 96.

    GDS Gareth Rushgrove Load testing

97. 97.

    GDS Gareth Rushgrove http://www.flickr.com/photos/psd/8040667413 Replay traffic

98. 98.

    GDS Gareth Rushgrove Defend yourself from load tests

99. 99.

    http { proxy_intercept_errors on; limit_req_zone $binary_remote_addr zone=rat limit_conn_zone $binary_remote_addr zone=co

    } Rate/connection limiting in nginx GDS Gareth Rushgrove
100. 100.

     GDS Gareth Rushgrove Web application firewall

101. 101.

     Embrace failures (but stop them affecting people) GDS Gareth Rushgrove

102. 102.

     GDS Gareth Rushgrove Serve stale content

103. 103.

     sub vcl_recv { if (req.backend.healthy) { set req.grace = 30s;

     } else { set req.grace = 24h; } } sub vcl_fetch { set beresp.grace = 24h; } Grace mode in Varnish GDS Gareth Rushgrove
104. 104.

     Except when you can’t GDS Gareth Rushgrove

105. 105.

     GDS Gareth Rushgrove Partial page failures

106. 106.

     GDS Gareth Rushgrove

107. 107.

     GDS Gareth Rushgrove Making change easy makes fixing problems fast

108. 108.

     GDS Gareth Rushgrove Critical issues fixed in hours or minutes

109. 109.

     GDS Gareth Rushgrove Measure everything

110. 110.

     GDS Gareth Rushgrove

111. 111.

     GDS Gareth Rushgrove Long tail of failures

112. 112.

     GDS Gareth Rushgrove Allow for investigation

113. 113.

     GDS Gareth Rushgrove Make monitoring easy

114. 114.

     @normal Scenario: check quick answers load When I visit "/vat-rates"

     Then I should see "VAT rates" GDS Gareth Rushgrove Run as smoke tests and monitoring
115. 115.

     @high Scenario: check quick answers load When I visit "/vat-rates"

     Then I should see "VAT rates" GDS Gareth Rushgrove Change who responds to issue
116. 116.

     Inside Government (or any large organisation) GDS Gareth Rushgrove

117. 117.

     GDS Gareth Rushgrove Involve technology colleagues

118. 118.

     GDS Gareth Rushgrove Involve security and auditing colleagues

119. 119.

     GDS Gareth Rushgrove Involve procurement colleagues

120. 120.

     GDS Gareth Rushgrove (Procurement means rules for buying things)

121. 121.

     GDS Gareth Rushgrove Demonstrate capability quickly

122. 122.

     Conclusions (What about my project?) GDS Gareth Rushgrove

123. 123.

     GDS Gareth Rushgrove Own your own problems 1

124. 124.

     GDS Gareth Rushgrove Avoid hard problems when possible 2

125. 125.

     GDS Gareth Rushgrove Make things open 3

126. 126.

     GDS Gareth Rushgrove Work on business processes AND technology 4

127. 127.

     Find out more (A public service announcement) GDS Gareth Rushgrove

128. 128.

     GDS Gareth Rushgrove

129. 129.

     GDS Gareth Rushgrove Government Service Design Manual

130. 130.

     GDS Gareth Rushgrove Even contains a definition of devops

131. 131.

     Questions? (and thanks for listening) GDS Gareth Rushgrove

132. 132.

     GDS Gareth Rushgrove Gareth Rushgrove Technical Architect Government Digital Service

     @garethr