The Road to Continuous Deployment: a case study (IPC Berlin 2017)

The Road to Continuous Deployment: a case study (IPC Berlin 2017)

It’s a situation many of us are familiar with: a large legacy, monolithic application, limited or no tests, slow and manual release process, low velocity, no confidence… A lot of refactoring is required, but management keeps pushing for new features. How to proceed? Using examples and lessons learned from a real-world case, I’ll show you how to replace a legacy application with a modern service-oriented architecture and build a continuous integration and deployment pipeline to deliver value from the first sprint. On the way, we’ll take a look at the process, automated testing, monitoring, master/trunk based development and various tips and best practices.

2f4800411154a8c66dde489448a044d2?s=128

Michiel Rook

May 31, 2017
Tweet

Transcript

  1. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY MICHIEL

    ROOK @michieltcs
  2. ▸ Developer, consultant, trainer, speaker ▸ @michieltcs

  3. None
  4. @michieltcs

  5. @michieltcs Large aging monolith

  6. @michieltcs Large aging monolith Generates income

  7. @michieltcs Large aging monolith Slow & complex Generates income

  8. @michieltcs Large aging monolith Slow & complex Generates income Technical

    debt
  9. FRONTEND MYSQL DB BACKEND LOAD BALANCERS / VARNISH JOB SITE

    JOB SITE JOB SITE FRONTEND FRONTEND FRONTEND BACKEND BACKEND BACKEND MEMCACHE FTP EXT. SERVICES SOLR @michieltcs
  10. @michieltcs

  11. @michieltcs Manual releases

  12. @michieltcs Manual releases Fragile tests

  13. @michieltcs Manual releases Frequent issues Fragile tests

  14. @michieltcs Manual releases Frequent issues Fragile tests Low velocity

  15. GOALS @michieltcs

  16. GOALS REDUCE
 ISSUES 1 @michieltcs

  17. GOALS REDUCE
 ISSUES REDUCE
 LEAD TIME 1 2 @michieltcs

  18. GOALS REDUCE
 ISSUES REDUCE
 LEAD TIME INCREASE PRODUCTIVITY 1 2

    3 @michieltcs
  19. GOALS REDUCE
 ISSUES REDUCE
 LEAD TIME INCREASE PRODUCTIVITY INCREASE MOTIVATION

    1 2 3 4 @michieltcs
  20. REFACTOR? REBUILD?

  21. APPROACH ▸ API first ▸ Services per domain object (job,

    jobseeker, ...) ▸ Migrate individual pages @michieltcs
  22. ORIGINAL MONOLITH DB @michieltcs

  23. ORIGINAL MONOLITH PROXY SERVICE ORIGINAL MONOLITH DB DB DB @michieltcs

  24. ORIGINAL MONOLITH PROXY SERVICE ORIGINAL MONOLITH ORIGINAL MONOLITH SERVICE SERVICE

    SERVICE PROXY DB DB DB DB DB DB @michieltcs
  25. ARCHITECTURE FRONTENDS ARE SERVICES @michieltcs

  26. ARCHITECTURE FRONTENDS ARE SERVICES SERVICES BEHIND LOAD BALANCERS @michieltcs

  27. ARCHITECTURE FRONTENDS ARE SERVICES SERVICES BEHIND LOAD BALANCERS ACCESS LEGACY

    DB'S @michieltcs
  28. ARCHITECTURE FRONTENDS ARE SERVICES SERVICES BEHIND LOAD BALANCERS ACCESS LEGACY

    DB'S SERVICES IN CONTAINERS @michieltcs
  29. PROCESS

  30. CONTINUOUS EVERYTHING

  31. CD?

  32. DEV BUILD / TEST CONTINUOUS INTEGRATION @michieltcs

  33. DEV BUILD / TEST ACCEPTANCE PRODUCTION CONTINUOUS DELIVERY @michieltcs

  34. DEV BUILD / TEST STAGING / ACCEPTANCE PRODUCTION CONTINUOUS DEPLOYMENT

    @michieltcs
  35. EVERY COMMIT GOES TO PRODUCTION

  36. IF IT HURTS DO IT MORE OFTEN

  37. SMALL STEPS @michieltcs

  38. SMALL STEPS EARLY FEEDBACK @michieltcs

  39. SMALL STEPS EARLY FEEDBACK REDUCE TIME TO RECOVER @michieltcs

  40. SMALL STEPS EXPERIMENTS! EARLY FEEDBACK REDUCE TIME TO RECOVER @michieltcs

  41. ONLY COMMIT TO MASTER

  42. NO BRANCHES

  43. NO BRANCHES REALLY.

  44. None
  45. None
  46. BRANCHES ARE CHEAP?

  47. DELAYING INTEGRATION

  48. CONFLICTS

  49. FEATURE TOGGLES

  50. DECOUPLE DEPLOYMENTS FROM RELEASES

  51. @michieltcs

  52. FEATURE TOGGLE DEBT

  53. CODE REVIEWS?

  54. PAIR PROGRAMMING

  55. CONTINUOUS INLINE CODE REVIEW

  56. BOY SCOUT RULE

  57. GATES

  58. 100% CODE COVFEFE *

  59. 100% CODE COVERAGE*

  60. DEVOPS

  61. DEVSECOPS

  62. DEVSECQAOPS

  63. DEVSECQABIZOPS

  64. DEVSECQABIZNETOPS

  65. DEVSECQABIZSYSNETOPS

  66. DEVTESTSECQABIZSYSNETOPS

  67. DEV*OPS

  68. TEAM BUILDS IT

  69. TEAM RUNS IT

  70. TEAM IMPROVES IT

  71. TEAM IS RESPONSIBLE

  72. BUILD PIPELINE

  73. AUTOMATE REPEATABLE THINGS

  74. CONTINUOUS TESTING

  75. CONFIDENCE

  76. DEFENSE IN DEPTH UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE
 TESTS UI

    TESTS @michieltcs
  77. UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE UI TESTS public function testJobCannotBeFound()

    {
 $jobRepository = $this->prophesize(JobRepository::class);
 $jobRepository->getById(EXPECTED_JOB_ID)
 ->shouldBeCalled()
 ->willReturn(false);
 
 $jobService = new JobService($jobRepository->reveal());
 
 $this->assertFalse($jobService->getById(EXPECTED_JOB_ID));
 } DEFENSE IN DEPTH @michieltcs
  78. UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE
 TESTS UI TESTS public function

    testFindJob() {
 $expectedJob = $this->loadFixture('active_job.yml');
 $actualJob = $this->repository->getById($expectedJob->getId());
 
 self::assertInstanceOf(Job::class, $actualJob);
 self::assertEquals($expectedJob->getId(), $actualJob->getId());
 } DEFENSE IN DEPTH @michieltcs
  79. DEFENSE IN DEPTH UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE
 TESTS UI

    TESTS Scenario: Link to related job
 Given a job exists
 And there are related jobs available
 When that job is viewed
 Then a list of related jobs is shown
 And each related job links to the detail page of the related job @michieltcs
  80. DEFENSE IN DEPTH UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE
 TESTS UI

    TESTS @michieltcs
  81. UNIT TESTS INTEGRATION TESTS ACCEPTANCE TESTS UI TESTS SMOKE
 TESTS

    Cost Speed Exploratory
 testing Monitoring @michieltcs
  82. GOOD ENOUGH

  83. PIPELINE SPEED = KEY

  84. PIPELINE AS CODE node {
 stage('Run tests') {
 sh "phpunit"


    sh "behat"
 }
 
 stage('Build docker image') {
 sh "docker build -t jobservice:${env.BUILD_NUMBER} ."
 sh "docker push jobservice:${env.BUILD_NUMBER}"
 }
 
 stage('Deploy staging') {
 sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER}
 -i staging deploy.yml"
 }
 
 stage('Deploy production') {
 sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER}
 -i prod deploy.yml"
 }
 } @michieltcs
  85. PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS

    / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER @michieltcs
  86. PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS

    / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker pull @michieltcs
  87. PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS

    / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker run @michieltcs
  88. PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS

    / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER wait_for: port=8080 delay=5 timeout=15 @michieltcs
  89. PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS

    / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER uri:
 url: http://localhost:8080/health
 status_code: 200
 timeout: 30 @michieltcs
  90. PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS

    / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER template: src=haproxy.cfg.j2
 dest=/etc/haproxy/haproxy.cfg service: name=haproxy state=reloaded @michieltcs
  91. PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS

    / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER template: src=haproxy.cfg.j2
 dest=/etc/haproxy/haproxy.cfg service: name=haproxy state=reloaded @michieltcs
  92. PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS

    / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker stop docker rm @michieltcs
  93. @michieltcs

  94. FEEDBACK!

  95. RESULTS

  96. Team acceptance @michieltcs

  97. Team acceptance New technology @michieltcs

  98. Team acceptance New technology Docker stability @michieltcs

  99. Team acceptance New technology Docker stability Pipeline stability @michieltcs

  100. Feature toggle cap @michieltcs

  101. Feature toggle cap Business alignment @michieltcs

  102. Feature toggle cap Business alignment Focus on replacing legacy @michieltcs

  103. Build per service
 < 10 min. 1 @michieltcs

  104. Build per service
 < 10 min. 1 50+ deploys per

    day 2 @michieltcs
  105. Build per service
 < 10 min. 1 50+ deploys per

    day Reduced number of issues 2 3 @michieltcs
  106. Build per service
 < 10 min. 1 50+ deploys per

    day Reduced number of issues Improved page load times 2 3 4 @michieltcs
  107. Improved metrics & audience statistics 5 @michieltcs

  108. Improved metrics & audience statistics 5 Learning new technology 6

    @michieltcs
  109. Improved metrics & audience statistics 5 Learning new technology Increased

    confidence, velocity & fun 6 7 @michieltcs
  110. LITERATURE https://trunkbaseddevelopment.com/

  111. THANK YOU! @michieltcs / mrook@fourscouts.nl www.michielrook.nl