The Unavoidable Big Bang

Talk about launching GOV.UK and working in a large organisation. Presented at RAMP in Budapest.

Transcript

1. GDS
   Gareth Rushgrove
   The
   Unavoidable
   Big Bang
   and life in Government
   

   View Slide

2. Who
   (Who is this person?)
   GDS
   Gareth Rushgrove
   

   View Slide

3. GDS
   Gareth Rushgrove
   Gareth Rushgrove
   Technical Architect
   Government Digital Service
   @garethr
   

   View Slide

4. GDS
   Gareth Rushgrove
   Agencies
   Startups
   Big companies
   Myself
   Worked for
   

   View Slide

5. GDS
   Gareth Rushgrove
   

   View Slide

6. GDS
   Gareth Rushgrove
   Last code
   I wrote
   

   View Slide

7. GDS
   Gareth Rushgrove
   

   View Slide

8. GDS
   Gareth Rushgrove
   

   View Slide

9. GDS
   Gareth Rushgrove
   

   View Slide

10. GDS
    Gareth Rushgrove
    

    View Slide

11. What
    (What will I get from this talk?)
    GDS
    Gareth Rushgrove
    

    View Slide

12. GDS
    Gareth Rushgrove
    Reasons to avoid a
    big bang release
    1
    

    View Slide

13. GDS
    Gareth Rushgrove
    How to make a big
    bang release work if
    you have to
    2
    

    View Slide

14. GDS
    Gareth Rushgrove
    How to work in a
    large organisation
    3
    

    View Slide

15. GDS
    Gareth Rushgrove
    And a desire to work
    in the public sector
    (maybe)
    4
    

    View Slide

16. Background
    (Government and GOV.UK)
    GDS
    Gareth Rushgrove
    

    View Slide

17. GDS
    Gareth Rushgrove
    October 2010
    

    View Slide

18. GDS
    Gareth Rushgrove
    June 2011
    

    View Slide

19. GDS
    Gareth Rushgrove
    January 2012
    

    View Slide

20. GDS
    (Government Digital Service)
    GDS
    Gareth Rushgrove
    

    View Slide

21. GDS
    Gareth Rushgrove
    

    View Slide

22. GDS
    Gareth Rushgrove
    October 2012
    

    View Slide

23. GDS
    Gareth Rushgrove
    Tools as well
    as content
    

    View Slide

24. GDS
    Gareth Rushgrove
    Award
    winning
    

    View Slide

25. GDS
    Gareth Rushgrove
    http://www.flickr.com/photos/psd/9099796942
    

    View Slide

26. By the numbers
    (The size of the problem)
    GDS
    Gareth Rushgrove
    

    View Slide

27. 38
    GDS
    Gareth Rushgrove
    Weeks since
    launch
    

    View Slide

28. 2
    GDS
    Gareth Rushgrove
    Sites closed
    on day one
    

    View Slide

29. GDS
    Gareth Rushgrove
    59
    Sites closed
    since
    

    View Slide

30. 222
    GDS
    Gareth Rushgrove
    Subdomains
    closed
    

    View Slide

31. GDS
    Gareth Rushgrove
    139
    Million visits
    since launch
    

    View Slide

32. 300
    GDS
    Gareth Rushgrove
    Thousand
    redirects
    

    View Slide

33. GDS
    Gareth Rushgrove
    10-100
    Members of
    the team
    

    View Slide

34. GDS
    Gareth Rushgrove
    http://www.flickr.com/photos/psd/8756580339
    

    View Slide

35. Don’t do this at home
    (Unless you have to)
    GDS
    Gareth Rushgrove
    

    View Slide

36. GDS
    Gareth Rushgrove
    Find alternatives
    

    View Slide

37. GDS
    Gareth Rushgrove
    Start small?
    

    View Slide

38. GDS
    Gareth Rushgrove
    Replace one site at a
    time?
    

    View Slide

39. GDS
    Gareth Rushgrove
    Percentage of
    visitors?
    

    View Slide

40. GDS
    Gareth Rushgrove
    

    View Slide

41. GDS
    Gareth Rushgrove
    http://www.flickr.com/photos/psd/9116635297
    

    View Slide

42. GDS
    Gareth Rushgrove
    Existing contracts
    with associated dates
    

    View Slide

43. GDS
    Gareth Rushgrove
    Public commitments
    

    View Slide

44. GDS
    Gareth Rushgrove
    http://www.flickr.com/photos/psd/9120523574
    

    View Slide

45. GDS
    Gareth Rushgrove
    http://www.flickr.com/photos/psd/9104280608
    

    View Slide

46. Do it in public
    (Running an alpha and beta)
    GDS
    Gareth Rushgrove
    

    View Slide

47. GDS
    Gareth Rushgrove
    http://www.flickr.com/photos/psd/9122642253
    

    View Slide

48. GDS
    Gareth Rushgrove
    

    View Slide

49. GDS
    Gareth Rushgrove
    

    View Slide

50. GDS
    Gareth Rushgrove
    Not just the service
    

    View Slide

51. GDS
    Gareth Rushgrove
    Blogging
    

    View Slide

52. GDS
    Gareth Rushgrove
    Code
    

    View Slide

53. GDS
    Gareth Rushgrove
    Publishing
    tools
    

    View Slide

54. GDS
    Gareth Rushgrove
    Frontend
    applications
    

    View Slide

55. GDS
    Gareth Rushgrove
    Developer
    environment
    

    View Slide

56. GDS
    Gareth Rushgrove
    Handy
    utilities
    

    View Slide

57. GDS
    Gareth Rushgrove
    Project
    backlogs
    

    View Slide

58. GDS
    Gareth Rushgrove
    Performance
    data
    

    View Slide

59. GDS
    Gareth Rushgrove
    

    View Slide

60. GDS
    Gareth Rushgrove
    

    View Slide

61. GDS
    Gareth Rushgrove
    But didn’t being open
    cause problems?
    

    View Slide

62. GDS
    Gareth Rushgrove
    No
    This slide stolen from @tomskitomski
    

    View Slide

63. Make change easy
    (Technology AND process)
    GDS
    Gareth Rushgrove
    

    View Slide

64. GDS
    Gareth Rushgrove
    One click deploy
    

    View Slide

65. GDS
    Gareth Rushgrove
    Single place
    to deploy
    

    View Slide

66. GDS
    Gareth Rushgrove
    Not just applications
    

    View Slide

67. GDS
    Gareth Rushgrove
    Configuration
    management
    

    View Slide

68. GDS
    Gareth Rushgrove
    

    View Slide

69. GDS
    Gareth Rushgrove
    package { 'apache2':
    ensure => latest,
    }
    service { 'apache2':
    ensure => running,
    provider => upstart,
    require => Package['apache2']
    }
    

    View Slide

70. class govuk::apps::calendars( $port = 3011 ) {
    govuk::app { 'calendars':
    app_type => 'rack',
    port => $port,
    health_check_path => ‘/bank-holidays’,
    }
    }
    GDS
    Gareth Rushgrove
    Higher level
    constructs
    

    View Slide

71. GDS
    Gareth Rushgrove
    Infrastructure not just
    configuration
    

    View Slide

72. GDS
    Gareth Rushgrove
    Provisioning APIs
    

    View Slide

73. GDS
    Gareth Rushgrove
    Fog
    Libcloud
    VCloud
    AWS
    Used at
    different times
    

    View Slide

74. require 'rubygems'
    require 'nat'
    nat do
    snat :interface => "Client Data",
    :original => { :ip => "10.0.0.0/xx" },
    :translated => { :ip => "xx.xx.xx.xx" },
    :desc => "Outbound internet traffic"
    dnat :interface => "Client Data",
    :original => { :ip => "xx.xx.xx.xx", :port => 22 },
    :translated => { :ip => "10.0.0.xx", :port => 22 },
    :desc => "jumpbox-1 SSH"
    dnat :interface => "Client Data",
    :original => { :ip => "xx.xx.xx.xx", :port => 80 },,
    :translated => { :ip => "10.0.0.xx", :port => 80 },
    :desc => "jenkins, logging, monitoring HTTP"
    GDS
    Gareth Rushgrove
    Network in
    code
    

    View Slide

75. require 'rubygems'
    require 'firewall'
    firewall do
    # internal rules
    rule "ssh access to jumpbox1" do
    source :ip => "Any"
    destination :ip => "xx.xx.xx.xx", :port => 22
    end
    rule "http to backend applications" do
    source :ip => "Any"
    destination :ip => "xx.xx.xx.xx", :port => 80
    end
    rule "https to backend applications" do
    GDS
    Gareth Rushgrove
    Firewalls in
    code
    

    View Slide

76. GDS
    Gareth Rushgrove
    Requires rolling your
    own code at times
    

    View Slide

77. GDS
    Gareth Rushgrove
    Developers want
    Visibility of deploys
    

    View Slide

78. GDS
    Gareth Rushgrove
    Organisation want
    Auditability of deploys
    

    View Slide

79. GDS
    Gareth Rushgrove
    App showing
    deploys
    

    View Slide

80. GDS
    Gareth Rushgrove
    Embrace process
    discussions
    

    View Slide

81. GDS
    Gareth Rushgrove
    Change control
    

    View Slide

82. GDS
    Gareth Rushgrove
    Risk management
    

    View Slide

83. GDS
    Gareth Rushgrove
    Share
    language
    

    View Slide

84. GDS
    Gareth Rushgrove
    Optimise only as far
    as needed
    

    View Slide

85. Make change small
    (Lots of small changes)
    GDS
    Gareth Rushgrove
    

    View Slide

86. GDS
    Gareth Rushgrove
    Change one thing at
    once
    

    View Slide

87. GDS
    Gareth Rushgrove
    http://www.flickr.com/photos/fatty/9158066939
    We use a
    physical token
    

    View Slide

88. GDS
    Gareth Rushgrove
    

    View Slide

89. GDS
    Gareth Rushgrove
    Regular
    releases
    reduce risk
    

    View Slide

90. GDS
    Gareth Rushgrove
    Back to
    GOV.UK
    

    View Slide

91. GDS
    Gareth Rushgrove
    

    View Slide

92. GDS
    Gareth Rushgrove
    We changed less
    software on the day
    of launch than
    probably any day
    before or since
    

    View Slide

93. GDS
    Gareth Rushgrove
    DNS?
    

    View Slide

94. GDS
    Gareth Rushgrove
    Content Delivery
    Network
    

    View Slide

95. GDS
    Gareth Rushgrove
    http://www.flickr.com/photos/psd/8040082207
    

    View Slide

96. GDS
    Gareth Rushgrove
    Load testing
    

    View Slide

97. GDS
    Gareth Rushgrove
    http://www.flickr.com/photos/psd/8040667413
    Replay
    traffic
    

    View Slide

98. GDS
    Gareth Rushgrove
    Defend yourself from
    load tests
    

    View Slide

99. http {
    proxy_intercept_errors on;
    limit_req_zone $binary_remote_addr zone=rat
    limit_conn_zone $binary_remote_addr zone=co
    }
    Rate/connection
    limiting in nginx
    GDS
    Gareth Rushgrove
    

    View Slide

100. GDS
     Gareth Rushgrove
     Web application
     firewall
     

     View Slide

101. Embrace failures
     (but stop them affecting people)
     GDS
     Gareth Rushgrove
     

     View Slide

102. GDS
     Gareth Rushgrove
     Serve stale content
     

     View Slide

103. sub vcl_recv {
     if (req.backend.healthy) {
     set req.grace = 30s;
     } else {
     set req.grace = 24h;
     }
     }
     sub vcl_fetch {
     set beresp.grace = 24h;
     } Grace mode
     in Varnish
     GDS
     Gareth Rushgrove
     

     View Slide

104. Except when you
     can’t
     GDS
     Gareth Rushgrove
     

     View Slide

105. GDS
     Gareth Rushgrove
     Partial page
     failures
     

     View Slide

106. GDS
     Gareth Rushgrove
     

     View Slide

107. GDS
     Gareth Rushgrove
     Making change easy
     makes fixing
     problems fast
     

     View Slide

108. GDS
     Gareth Rushgrove
     Critical issues
     fixed in hours
     or minutes
     

     View Slide

109. GDS
     Gareth Rushgrove
     Measure everything
     

     View Slide

110. GDS
     Gareth Rushgrove
     

     View Slide

111. GDS
     Gareth Rushgrove
     Long tail of
     failures
     

     View Slide

112. GDS
     Gareth Rushgrove
     Allow for
     investigation
     

     View Slide

113. GDS
     Gareth Rushgrove
     Make monitoring easy
     

     View Slide

114. @normal
     Scenario: check quick answers load
     When I visit "/vat-rates"
     Then I should see "VAT rates"
     GDS
     Gareth Rushgrove
     Run as smoke tests
     and monitoring
     

     View Slide

115. @high
     Scenario: check quick answers load
     When I visit "/vat-rates"
     Then I should see "VAT rates"
     GDS
     Gareth Rushgrove
     Change who
     responds to issue
     

     View Slide

116. Inside Government
     (or any large organisation)
     GDS
     Gareth Rushgrove
     

     View Slide

117. GDS
     Gareth Rushgrove
     Involve technology
     colleagues
     

     View Slide

118. GDS
     Gareth Rushgrove
     Involve security and
     auditing colleagues
     

     View Slide

119. GDS
     Gareth Rushgrove
     Involve procurement
     colleagues
     

     View Slide

120. GDS
     Gareth Rushgrove
     (Procurement means
     rules for buying
     things)
     

     View Slide

121. GDS
     Gareth Rushgrove
     Demonstrate
     capability quickly
     

     View Slide

122. Conclusions
     (What about my project?)
     GDS
     Gareth Rushgrove
     

     View Slide

123. GDS
     Gareth Rushgrove
     Own your own
     problems
     1
     

     View Slide

124. GDS
     Gareth Rushgrove
     Avoid hard problems
     when possible
     2
     

     View Slide

125. GDS
     Gareth Rushgrove
     Make things open
     3
     

     View Slide

126. GDS
     Gareth Rushgrove
     Work on business
     processes AND
     technology
     4
     

     View Slide

127. Find out more
     (A public service announcement)
     GDS
     Gareth Rushgrove
     

     View Slide

128. GDS
     Gareth Rushgrove
     

     View Slide

129. GDS
     Gareth Rushgrove
     Government Service Design Manual
     

     View Slide

130. GDS
     Gareth Rushgrove
     Even contains
     a definition of
     devops
     

     View Slide

131. Questions?
     (and thanks for listening)
     GDS
     Gareth Rushgrove
     

     View Slide

132. GDS
     Gareth Rushgrove
     Gareth Rushgrove
     Technical Architect
     Government Digital Service
     @garethr
     

     View Slide