The Road to Continuous Deployment (DevOpsPro Vilnius 2017)

The Road to Continuous Deployment (DevOpsPro Vilnius 2017)

It’s a situation many of us are familiar with: a large legacy, monolithic application, limited or no tests, slow & manual release process, low velocity, no confidence… A lot of refactoring is required, but management keeps pushing for new features.

How to proceed? Using examples and lessons learned from a real-world case, I’ll show you how to replace a legacy application with a modern service-oriented architecture and build a continuous integration and deployment pipeline to deliver value from the first sprint. On the way, we'll take a look at the process, automated testing, monitoring, master/trunk based development and various tips and best practices.

2f4800411154a8c66dde489448a044d2?s=128

Michiel Rook

April 06, 2017
Tweet

Transcript

  1. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY MICHIEL

    ROOK @michieltcs
  2. ▸ Developer, consultant, trainer, speaker ▸ @michieltcs

  3. None
  4. @michieltcs

  5. @michieltcs Large aging monolith

  6. @michieltcs Large aging monolith Generates income

  7. @michieltcs Large aging monolith Slow & complex Generates income

  8. @michieltcs Large aging monolith Slow & complex Generates income Technical

    debt
  9. FRONTEND MYSQL DB BACKEND LOAD BALANCERS / VARNISH ITBANEN INTERMEDIAIR

    NATIONALEVACATUREBANK FRONTEND FRONTEND FRONTEND BACKEND BACKEND BACKEND MEMCACHE FTP EXT. SERVICES SOLR @michieltcs
  10. @michieltcs

  11. @michieltcs Manual releases

  12. @michieltcs Manual releases Fragile tests

  13. @michieltcs Manual releases Frequent issues Fragile tests

  14. @michieltcs Manual releases Frequent issues Fragile tests Low velocity

  15. GOALS @michieltcs

  16. GOALS @michieltcs REDUCE
 ISSUES 1

  17. GOALS @michieltcs REDUCE
 ISSUES REDUCE
 LEAD TIME 1 2

  18. GOALS @michieltcs REDUCE
 ISSUES REDUCE
 LEAD TIME INCREASE PRODUCTIVITY 1

    2 3
  19. GOALS @michieltcs REDUCE
 ISSUES REDUCE
 LEAD TIME INCREASE PRODUCTIVITY INCREASE

    MOTIVATION 1 2 3 4
  20. REFACTOR? REBUILD?

  21. APPROACH ▸ API first ▸ Services per domain object (job,

    jobseeker, ...) ▸ Migrate individual pages @michieltcs
  22. ORIGINAL MONOLITH DB @michieltcs

  23. ORIGINAL MONOLITH PROXY SERVICE ORIGINAL MONOLITH DB DB DB @michieltcs

  24. ORIGINAL MONOLITH PROXY SERVICE ORIGINAL MONOLITH ORIGINAL MONOLITH SERVICE SERVICE

    SERVICE PROXY DB DB DB DB DB DB @michieltcs
  25. PROXY @michieltcs RewriteEngine On # Serve feature from new service

    for internal network
 RewriteCond expr "%{HTTP:X-FORWARDED-FOR} -ipmatch '192.168.0.0/24'"
 RewriteRule ^/feature/(.*)$ ${NEW_SERVICE_URL}/$1 [P,L] # Proxy everything else to legacy application
 RewriteRule ^/(.*) ${LEGACY_URL}/$1 [P]
 ProxyPassReverse / ${LEGACY_URL}/
  26. ARCHITECTURE @michieltcs FRONTENDS ARE SERVICES

  27. ARCHITECTURE @michieltcs FRONTENDS ARE SERVICES SERVICES BEHIND LOAD BALANCERS

  28. ARCHITECTURE @michieltcs FRONTENDS ARE SERVICES SERVICES BEHIND LOAD BALANCERS ACCESS

    LEGACY DB'S
  29. ARCHITECTURE @michieltcs FRONTENDS ARE SERVICES SERVICES BEHIND LOAD BALANCERS ACCESS

    LEGACY DB'S SERVICES IN CONTAINERS
  30. LEGACY ELASTIC SEARCH DB JOB SERVICE RMQ ITBANEN INTERMEDIAIR NATIONALEVACATUREBANK

    MONGO DB ITBANEN JOBSEEKER SERVICE NVB INTERMEDIAIR @michieltcs
  31. PROCESS

  32. CONTINUOUS EVERYTHING

  33. CD?

  34. DEV BUILD / TEST CONTINUOUS INTEGRATION @michieltcs

  35. DEV BUILD / TEST ACCEPTANCE PRODUCTION CONTINUOUS DELIVERY @michieltcs

  36. DEV BUILD / TEST STAGING / ACCEPTANCE PRODUCTION CONTINUOUS DEPLOYMENT

    @michieltcs
  37. @michieltcs SMALL STEPS

  38. @michieltcs SMALL STEPS EARLY FEEDBACK

  39. @michieltcs SMALL STEPS EARLY FEEDBACK REDUCE TIME TO RECOVER

  40. @michieltcs SMALL STEPS EXPERIMENTS! EARLY FEEDBACK REDUCE TIME TO RECOVER

  41. @michieltcs

  42. EVERY COMMIT GOES TO PRODUCTION

  43. ONLY COMMIT TO MASTER

  44. NO BRANCHES

  45. NO BRANCHES REALLY.

  46. None
  47. None
  48. PAIR PROGRAMMING

  49. FOCUS ON VALUE

  50. FEATURE TOGGLES, A/B TESTS @michieltcs

  51. BOY SCOUT RULE

  52. GATES

  53. 100% CODE COVERAGE*

  54. DEVOPS

  55. DEVSECOPS

  56. DEVSECQAOPS

  57. DEVSECQABIZOPS

  58. DEVSECQABIZNETOPS

  59. DEVSECQABIZSYSNETOPS

  60. DEVTESTSECQABIZSYSNETOPS

  61. DEV*OPS

  62. WORKED FINE IN DEV OPS PROBLEM NOW

  63. MONITORING

  64. None
  65. None
  66. BUILD
 PIPELINE

  67. AUTOMATE REPEATABLE THINGS

  68. CONTINUOUS TESTING

  69. DEFENSE IN DEPTH UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE
 TESTS UI

    TESTS @michieltcs
  70. DEFENSE IN DEPTH UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE UI TESTS

    @Test
 public void jobCannotBeFound() {
 when(jobRepository.getById(EXPECTED_JOB_ID))
 .thenReturn(null);
 
 JobService jobService = new JobService(jobRepository);
 
 assertNull(jobService.getById(EXPECTED_JOB_ID));
 verify(jobRepository).getById(EXPECTED_JOB_ID);
 } @michieltcs
  71. DEFENSE IN DEPTH UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE
 TESTS UI

    TESTS @Test
 public void shouldFindJob() {
 expectedJob = loadFixture('active_job.yml');
 actualJob = repository.getById(expectedJob.getId());
 
 assertThat(actualJob, isA(Job.class));
 assertEquals(expectedJob.getId(), actualJob.getId());
 } @michieltcs
  72. DEFENSE IN DEPTH UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE
 TESTS UI

    TESTS Scenario: Link to related job
 Given a job exists
 And there are related jobs available
 When that job is viewed
 Then a list of related jobs is shown
 And each related job links to the detail page of the related job @michieltcs
  73. DEFENSE IN DEPTH UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE
 TESTS UI

    TESTS @michieltcs
  74. DEFENSE IN DEPTH TESTER? UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE
 TESTS

    UI @michieltcs
  75. None
  76. CONTINUOUS TESTING UNIT TESTS INTEGRATION TESTS ACCEPTANCE TESTS UI TESTS

    SMOKE
 TESTS Cost Speed Exploratory
 testing Monitoring @michieltcs
  77. PIPELINE AS CODE node {
 stage('Run tests') {
 sh "phpunit"


    sh "behat"
 }
 
 stage('Build docker image') {
 sh "docker build -t jobservice:${env.BUILD_NUMBER} ."
 sh "docker push jobservice:${env.BUILD_NUMBER}"
 }
 
 stage('Deploy staging') {
 sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER}
 -i staging deploy.yml"
 }
 
 stage('Deploy production') {
 sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER}
 -i prod deploy.yml"
 }
 } @michieltcs
  78. DEPLOYING: ROLLING UPDATE PULL IMAGE START NEW CONTAINER WAIT FOR

    PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER @michieltcs
  79. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker pull @michieltcs
  80. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker run @michieltcs
  81. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER wait_for: port=8080 delay=5 timeout=15 @michieltcs
  82. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER uri:
 url: http://localhost:8080/health
 status_code: 200
 timeout: 30 @michieltcs
  83. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER template: src=haproxy.cfg.j2
 dest=/etc/haproxy/haproxy.cfg service: name=haproxy state=reloaded @michieltcs
  84. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER template: src=haproxy.cfg.j2
 dest=/etc/haproxy/haproxy.cfg service: name=haproxy state=reloaded @michieltcs
  85. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker stop docker rm @michieltcs
  86. BUILD PIPELINE @michieltcs

  87. FEEDBACK!

  88. RESULTS

  89. Build per service
 < 10 min. 1 @michieltcs

  90. Build per service
 < 10 min. 1 50+ deploys per

    day 2 @michieltcs
  91. Build per service
 < 10 min. 1 50+ deploys per

    day Reduced number of issues 2 3 @michieltcs
  92. Build per service
 < 10 min. 1 50+ deploys per

    day Reduced number of issues Improved page load times 2 3 4 @michieltcs
  93. Improved metrics & audience statistics 5 @michieltcs

  94. Improved metrics & audience statistics 5 Learning new technology 6

    @michieltcs
  95. Improved metrics & audience statistics 5 Learning new technology Increased

    confidence, velocity & fun 6 7 @michieltcs
  96. Team acceptance @michieltcs

  97. Team acceptance New technology @michieltcs

  98. Team acceptance New technology Docker stability @michieltcs

  99. Team acceptance New technology Docker stability Pipeline stability @michieltcs

  100. Feature toggle cap @michieltcs

  101. Feature toggle cap Business alignment @michieltcs

  102. Feature toggle cap Business alignment Focus on replacing legacy @michieltcs

  103. LITERATURE https://trunkbaseddevelopment.com/

  104. Reduce cycle times. React faster to the market and test

    product ideas. Make things! We make Agile, DevOps and Continuous Delivery accessible! Hands-on, results- oriented approach to get you where you need to be. Modern infrastructure and pipelines in minutes.
  105. THANK YOU! @michieltcs / michiel@make.io www.michielrook.nl