Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Unavoidable Big Bang

The Unavoidable Big Bang

Talk about launching GOV.UK and working in a large organisation. Presented at RAMP in Budapest.

Gareth Rushgrove

July 11, 2013
Tweet

More Decks by Gareth Rushgrove

Other Decks in Technology

Transcript

  1. GDS
    Gareth Rushgrove
    The
    Unavoidable
    Big Bang
    and life in Government

    View Slide

  2. Who
    (Who is this person?)
    GDS
    Gareth Rushgrove

    View Slide

  3. GDS
    Gareth Rushgrove
    Gareth Rushgrove
    Technical Architect
    Government Digital Service
    @garethr

    View Slide

  4. GDS
    Gareth Rushgrove
    Agencies
    Startups
    Big companies
    Myself
    Worked for

    View Slide

  5. GDS
    Gareth Rushgrove

    View Slide

  6. GDS
    Gareth Rushgrove
    Last code
    I wrote

    View Slide

  7. GDS
    Gareth Rushgrove

    View Slide

  8. GDS
    Gareth Rushgrove

    View Slide

  9. GDS
    Gareth Rushgrove

    View Slide

  10. GDS
    Gareth Rushgrove

    View Slide

  11. What
    (What will I get from this talk?)
    GDS
    Gareth Rushgrove

    View Slide

  12. GDS
    Gareth Rushgrove
    Reasons to avoid a
    big bang release
    1

    View Slide

  13. GDS
    Gareth Rushgrove
    How to make a big
    bang release work if
    you have to
    2

    View Slide

  14. GDS
    Gareth Rushgrove
    How to work in a
    large organisation
    3

    View Slide

  15. GDS
    Gareth Rushgrove
    And a desire to work
    in the public sector
    (maybe)
    4

    View Slide

  16. Background
    (Government and GOV.UK)
    GDS
    Gareth Rushgrove

    View Slide

  17. GDS
    Gareth Rushgrove
    October 2010

    View Slide

  18. GDS
    Gareth Rushgrove
    June 2011

    View Slide

  19. GDS
    Gareth Rushgrove
    January 2012

    View Slide

  20. GDS
    (Government Digital Service)
    GDS
    Gareth Rushgrove

    View Slide

  21. GDS
    Gareth Rushgrove

    View Slide

  22. GDS
    Gareth Rushgrove
    October 2012

    View Slide

  23. GDS
    Gareth Rushgrove
    Tools as well
    as content

    View Slide

  24. GDS
    Gareth Rushgrove
    Award
    winning

    View Slide

  25. GDS
    Gareth Rushgrove
    http://www.flickr.com/photos/psd/9099796942

    View Slide

  26. By the numbers
    (The size of the problem)
    GDS
    Gareth Rushgrove

    View Slide

  27. 38
    GDS
    Gareth Rushgrove
    Weeks since
    launch

    View Slide

  28. 2
    GDS
    Gareth Rushgrove
    Sites closed
    on day one

    View Slide

  29. GDS
    Gareth Rushgrove
    59
    Sites closed
    since

    View Slide

  30. 222
    GDS
    Gareth Rushgrove
    Subdomains
    closed

    View Slide

  31. GDS
    Gareth Rushgrove
    139
    Million visits
    since launch

    View Slide

  32. 300
    GDS
    Gareth Rushgrove
    Thousand
    redirects

    View Slide

  33. GDS
    Gareth Rushgrove
    10-100
    Members of
    the team

    View Slide

  34. GDS
    Gareth Rushgrove
    http://www.flickr.com/photos/psd/8756580339

    View Slide

  35. Don’t do this at home
    (Unless you have to)
    GDS
    Gareth Rushgrove

    View Slide

  36. GDS
    Gareth Rushgrove
    Find alternatives

    View Slide

  37. GDS
    Gareth Rushgrove
    Start small?

    View Slide

  38. GDS
    Gareth Rushgrove
    Replace one site at a
    time?

    View Slide

  39. GDS
    Gareth Rushgrove
    Percentage of
    visitors?

    View Slide

  40. GDS
    Gareth Rushgrove

    View Slide

  41. GDS
    Gareth Rushgrove
    http://www.flickr.com/photos/psd/9116635297

    View Slide

  42. GDS
    Gareth Rushgrove
    Existing contracts
    with associated dates

    View Slide

  43. GDS
    Gareth Rushgrove
    Public commitments

    View Slide

  44. GDS
    Gareth Rushgrove
    http://www.flickr.com/photos/psd/9120523574

    View Slide

  45. GDS
    Gareth Rushgrove
    http://www.flickr.com/photos/psd/9104280608

    View Slide

  46. Do it in public
    (Running an alpha and beta)
    GDS
    Gareth Rushgrove

    View Slide

  47. GDS
    Gareth Rushgrove
    http://www.flickr.com/photos/psd/9122642253

    View Slide

  48. GDS
    Gareth Rushgrove

    View Slide

  49. GDS
    Gareth Rushgrove

    View Slide

  50. GDS
    Gareth Rushgrove
    Not just the service

    View Slide

  51. GDS
    Gareth Rushgrove
    Blogging

    View Slide

  52. GDS
    Gareth Rushgrove
    Code

    View Slide

  53. GDS
    Gareth Rushgrove
    Publishing
    tools

    View Slide

  54. GDS
    Gareth Rushgrove
    Frontend
    applications

    View Slide

  55. GDS
    Gareth Rushgrove
    Developer
    environment

    View Slide

  56. GDS
    Gareth Rushgrove
    Handy
    utilities

    View Slide

  57. GDS
    Gareth Rushgrove
    Project
    backlogs

    View Slide

  58. GDS
    Gareth Rushgrove
    Performance
    data

    View Slide

  59. GDS
    Gareth Rushgrove

    View Slide

  60. GDS
    Gareth Rushgrove

    View Slide

  61. GDS
    Gareth Rushgrove
    But didn’t being open
    cause problems?

    View Slide

  62. GDS
    Gareth Rushgrove
    No
    This slide stolen from @tomskitomski

    View Slide

  63. Make change easy
    (Technology AND process)
    GDS
    Gareth Rushgrove

    View Slide

  64. GDS
    Gareth Rushgrove
    One click deploy

    View Slide

  65. GDS
    Gareth Rushgrove
    Single place
    to deploy

    View Slide

  66. GDS
    Gareth Rushgrove
    Not just applications

    View Slide

  67. GDS
    Gareth Rushgrove
    Configuration
    management

    View Slide

  68. GDS
    Gareth Rushgrove

    View Slide

  69. GDS
    Gareth Rushgrove
    package { 'apache2':
    ensure => latest,
    }
    service { 'apache2':
    ensure => running,
    provider => upstart,
    require => Package['apache2']
    }

    View Slide

  70. class govuk::apps::calendars( $port = 3011 ) {
    govuk::app { 'calendars':
    app_type => 'rack',
    port => $port,
    health_check_path => ‘/bank-holidays’,
    }
    }
    GDS
    Gareth Rushgrove
    Higher level
    constructs

    View Slide

  71. GDS
    Gareth Rushgrove
    Infrastructure not just
    configuration

    View Slide

  72. GDS
    Gareth Rushgrove
    Provisioning APIs

    View Slide

  73. GDS
    Gareth Rushgrove
    Fog
    Libcloud
    VCloud
    AWS
    Used at
    different times

    View Slide

  74. require 'rubygems'
    require 'nat'
    nat do
    snat :interface => "Client Data",
    :original => { :ip => "10.0.0.0/xx" },
    :translated => { :ip => "xx.xx.xx.xx" },
    :desc => "Outbound internet traffic"
    dnat :interface => "Client Data",
    :original => { :ip => "xx.xx.xx.xx", :port => 22 },
    :translated => { :ip => "10.0.0.xx", :port => 22 },
    :desc => "jumpbox-1 SSH"
    dnat :interface => "Client Data",
    :original => { :ip => "xx.xx.xx.xx", :port => 80 },,
    :translated => { :ip => "10.0.0.xx", :port => 80 },
    :desc => "jenkins, logging, monitoring HTTP"
    GDS
    Gareth Rushgrove
    Network in
    code

    View Slide

  75. require 'rubygems'
    require 'firewall'
    firewall do
    # internal rules
    rule "ssh access to jumpbox1" do
    source :ip => "Any"
    destination :ip => "xx.xx.xx.xx", :port => 22
    end
    rule "http to backend applications" do
    source :ip => "Any"
    destination :ip => "xx.xx.xx.xx", :port => 80
    end
    rule "https to backend applications" do
    GDS
    Gareth Rushgrove
    Firewalls in
    code

    View Slide

  76. GDS
    Gareth Rushgrove
    Requires rolling your
    own code at times

    View Slide

  77. GDS
    Gareth Rushgrove
    Developers want
    Visibility of deploys

    View Slide

  78. GDS
    Gareth Rushgrove
    Organisation want
    Auditability of deploys

    View Slide

  79. GDS
    Gareth Rushgrove
    App showing
    deploys

    View Slide

  80. GDS
    Gareth Rushgrove
    Embrace process
    discussions

    View Slide

  81. GDS
    Gareth Rushgrove
    Change control

    View Slide

  82. GDS
    Gareth Rushgrove
    Risk management

    View Slide

  83. GDS
    Gareth Rushgrove
    Share
    language

    View Slide

  84. GDS
    Gareth Rushgrove
    Optimise only as far
    as needed

    View Slide

  85. Make change small
    (Lots of small changes)
    GDS
    Gareth Rushgrove

    View Slide

  86. GDS
    Gareth Rushgrove
    Change one thing at
    once

    View Slide

  87. GDS
    Gareth Rushgrove
    http://www.flickr.com/photos/fatty/9158066939
    We use a
    physical token

    View Slide

  88. GDS
    Gareth Rushgrove

    View Slide

  89. GDS
    Gareth Rushgrove
    Regular
    releases
    reduce risk

    View Slide

  90. GDS
    Gareth Rushgrove
    Back to
    GOV.UK

    View Slide

  91. GDS
    Gareth Rushgrove

    View Slide

  92. GDS
    Gareth Rushgrove
    We changed less
    software on the day
    of launch than
    probably any day
    before or since

    View Slide

  93. GDS
    Gareth Rushgrove
    DNS?

    View Slide

  94. GDS
    Gareth Rushgrove
    Content Delivery
    Network

    View Slide

  95. GDS
    Gareth Rushgrove
    http://www.flickr.com/photos/psd/8040082207

    View Slide

  96. GDS
    Gareth Rushgrove
    Load testing

    View Slide

  97. GDS
    Gareth Rushgrove
    http://www.flickr.com/photos/psd/8040667413
    Replay
    traffic

    View Slide

  98. GDS
    Gareth Rushgrove
    Defend yourself from
    load tests

    View Slide

  99. http {
    proxy_intercept_errors on;
    limit_req_zone $binary_remote_addr zone=rat
    limit_conn_zone $binary_remote_addr zone=co
    }
    Rate/connection
    limiting in nginx
    GDS
    Gareth Rushgrove

    View Slide

  100. GDS
    Gareth Rushgrove
    Web application
    firewall

    View Slide

  101. Embrace failures
    (but stop them affecting people)
    GDS
    Gareth Rushgrove

    View Slide

  102. GDS
    Gareth Rushgrove
    Serve stale content

    View Slide

  103. sub vcl_recv {
    if (req.backend.healthy) {
    set req.grace = 30s;
    } else {
    set req.grace = 24h;
    }
    }
    sub vcl_fetch {
    set beresp.grace = 24h;
    } Grace mode
    in Varnish
    GDS
    Gareth Rushgrove

    View Slide

  104. Except when you
    can’t
    GDS
    Gareth Rushgrove

    View Slide

  105. GDS
    Gareth Rushgrove
    Partial page
    failures

    View Slide

  106. GDS
    Gareth Rushgrove

    View Slide

  107. GDS
    Gareth Rushgrove
    Making change easy
    makes fixing
    problems fast

    View Slide

  108. GDS
    Gareth Rushgrove
    Critical issues
    fixed in hours
    or minutes

    View Slide

  109. GDS
    Gareth Rushgrove
    Measure everything

    View Slide

  110. GDS
    Gareth Rushgrove

    View Slide

  111. GDS
    Gareth Rushgrove
    Long tail of
    failures

    View Slide

  112. GDS
    Gareth Rushgrove
    Allow for
    investigation

    View Slide

  113. GDS
    Gareth Rushgrove
    Make monitoring easy

    View Slide

  114. @normal
    Scenario: check quick answers load
    When I visit "/vat-rates"
    Then I should see "VAT rates"
    GDS
    Gareth Rushgrove
    Run as smoke tests
    and monitoring

    View Slide

  115. @high
    Scenario: check quick answers load
    When I visit "/vat-rates"
    Then I should see "VAT rates"
    GDS
    Gareth Rushgrove
    Change who
    responds to issue

    View Slide

  116. Inside Government
    (or any large organisation)
    GDS
    Gareth Rushgrove

    View Slide

  117. GDS
    Gareth Rushgrove
    Involve technology
    colleagues

    View Slide

  118. GDS
    Gareth Rushgrove
    Involve security and
    auditing colleagues

    View Slide

  119. GDS
    Gareth Rushgrove
    Involve procurement
    colleagues

    View Slide

  120. GDS
    Gareth Rushgrove
    (Procurement means
    rules for buying
    things)

    View Slide

  121. GDS
    Gareth Rushgrove
    Demonstrate
    capability quickly

    View Slide

  122. Conclusions
    (What about my project?)
    GDS
    Gareth Rushgrove

    View Slide

  123. GDS
    Gareth Rushgrove
    Own your own
    problems
    1

    View Slide

  124. GDS
    Gareth Rushgrove
    Avoid hard problems
    when possible
    2

    View Slide

  125. GDS
    Gareth Rushgrove
    Make things open
    3

    View Slide

  126. GDS
    Gareth Rushgrove
    Work on business
    processes AND
    technology
    4

    View Slide

  127. Find out more
    (A public service announcement)
    GDS
    Gareth Rushgrove

    View Slide

  128. GDS
    Gareth Rushgrove

    View Slide

  129. GDS
    Gareth Rushgrove
    Government Service Design Manual

    View Slide

  130. GDS
    Gareth Rushgrove
    Even contains
    a definition of
    devops

    View Slide

  131. Questions?
    (and thanks for listening)
    GDS
    Gareth Rushgrove

    View Slide

  132. GDS
    Gareth Rushgrove
    Gareth Rushgrove
    Technical Architect
    Government Digital Service
    @garethr

    View Slide