The Road to Continuous Deployment (PHP UK Conference 2017)

2f4800411154a8c66dde489448a044d2?s=47 Michiel Rook
February 16, 2017

The Road to Continuous Deployment (PHP UK Conference 2017)

It's a situation many of us are familiar with: a large legacy, monolithic application, limited or no tests, slow & manual release process, low velocity, no confidence...

A lot of refactoring is required, but management keeps pushing for new features. How to proceed?

Using examples and lessons learned from a real-world case, I'll show you how to replace a legacy application with a modern service-oriented architecture and build a continuous integration and deployment pipeline to deliver value from the first sprint. On the way, we’ll take a look at the process, automated testing, monitoring, master/trunk based development and various tips and best practices.

2f4800411154a8c66dde489448a044d2?s=128

Michiel Rook

February 16, 2017
Tweet

Transcript

  1. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY MICHIEL

    ROOK @michieltcs
  2. ▸ Java, PHP & Scala developer ▸ Consultant, trainer, speaker

    ▸ make.io ▸ Dutch Web Alliance ▸ @michieltcs
  3. TODAY ▸ Background ▸ The approach ▸ Process / standards

    ▸ Build pipelines ▸ Results & lessons learned @michieltcs
  4. None
  5. THE SYSTEM - SAN DIEGO ▸ ... or the Big

    Ball Of Mud ▸ Large legacy monolith ▸ Generates significant income ▸ Slow & complex ▸ Technical debt @michieltcs
  6. SAN DIEGO FRONTEND MYSQL DB SAN DIEGO BACKEND LOAD BALANCERS

    / VARNISH ITBANEN INTERMEDIAIR NATIONALEVACATUREBANK SAN DIEGO FRONTEND SAN DIEGO FRONTEND SAN DIEGO FRONTEND SAN DIEGO BACKEND SAN DIEGO BACKEND SAN DIEGO BACKEND MEMCACHE FTP EXT. SERVICES SOLR @michieltcs
  7. THE SYSTEM - SAN DIEGO ▸ Infrequent, manual releases ▸

    Fragile tests ▸ Low velocity ▸ Frequent outages / bugs / issues ▸ Frustrated team ▸ Low confidence modifying existing code @michieltcs
  8. GOALS ▸ Reduce issues ▸ Reduce lead & cycle time

    ▸ Increase productivity ▸ Increase motivation @michieltcs
  9. REFACTOR? REBUILD?

  10. APPROACH ▸ Strangler pattern ▸ API first ▸ Services per

    domain object (job, jobseeker, ...) ▸ Migrate individual pages @michieltcs
  11. ORIGINAL MONOLITH PROXY SERVICE ORIGINAL MONOLITH ORIGINAL MONOLITH SERVICE SERVICE

    SERVICE PROXY DB DB DB DB DB DB @michieltcs
  12. PROXY @michieltcs RewriteEngine On # Serve feature from new service

    for internal network
 RewriteCond expr "%{HTTP:X-FORWARDED-FOR} -ipmatch '192.168.0.0/24'"
 RewriteRule ^/feature/(.*)$ ${NEW_SERVICE_URL}/$1 [P,L] # Proxy everything else to legacy application
 RewriteRule ^/(.*) ${LEGACY_URL}/$1 [P]
 ProxyPassReverse / ${LEGACY_URL}/
  13. APPROACH ▸ Services behind load balancers ▸ Access legacy db’s

    ▸ Continuous deployment ▸ Docker containers ▸ Frontends are services @michieltcs
  14. SAN DIEGO ELASTIC SEARCH LEGACY DB JOB SERVICE RMQ ITBANEN

    INTERMEDIAIR NATIONALEVACATUREBANK MONGO DB ITBANEN JOBSEEKER SERVICE NVB INTERMEDIAIR @michieltcs
  15. PROCESS

  16. CONTINUOUS EVERYTHING

  17. CD?

  18. DEV BUILD / TEST CONTINUOUS INTEGRATION @michieltcs

  19. DEV BUILD / TEST ACCEPTANCE PRODUCTION CONTINUOUS DELIVERY @michieltcs

  20. DEV BUILD / TEST STAGING / ACCEPTANCE PRODUCTION CONTINUOUS DEPLOYMENT

    @michieltcs
  21. WHY CONTINUOUS DEPLOYMENT ▸ Small steps ▸ Early feedback ▸

    Reduce time to recover ▸ Experiments! @michieltcs
  22. SUCCESSFUL TEAMS HAVE @michieltcs

  23. TDD

  24. BDD

  25. EVERY COMMIT GOES TO PRODUCTION

  26. ONLY COMMIT TO MASTER

  27. NO BRANCHES

  28. NO BRANCHES REALLY.

  29. None
  30. None
  31. PAIR PROGRAMMING

  32. FOCUS ON VALUE

  33. FEATURE TOGGLES, A/B TESTS @michieltcs

  34. BOY SCOUT RULE

  35. QUALITY GATES

  36. 100% CODE COVERAGE*

  37. DEVOPS

  38. DEVSECOPS

  39. DEVSECQAOPS

  40. DEVSECQABIZOPS

  41. DEVSECQABIZNETOPS

  42. DEVSECQABIZSYSNETOPS

  43. DEVTESTSECQABIZSYSNETOPS

  44. DEV*OPS

  45. WORKED FINE IN DEV OPS PROBLEM NOW

  46. MONITORING

  47. None
  48. None
  49. BUILD
 PIPELINE

  50. AUTOMATE REPEATABLE THINGS

  51. ▸ Builds ▸ Testing ▸ Deployments ▸ Orchestration ▸ Config

    management @michieltcs AUTOMATION
  52. CONTINUOUS TESTING

  53. DEFENSE IN DEPTH UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE
 TESTS UI

    TESTS @michieltcs
  54. DEFENSE IN DEPTH UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE UI TESTS

    public function testJobCannotBeFound() {
 $jobRepository = $this->prophesize(JobRepository::class);
 $jobRepository->getById(EXPECTED_JOB_ID)
 ->shouldBeCalled()
 ->willReturn(false);
 
 $jobService = new JobService($jobRepository->reveal());
 
 $this->assertFalse($jobService->getById(EXPECTED_JOB_ID));
 } @michieltcs
  55. DEFENSE IN DEPTH UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE
 TESTS UI

    TESTS public function testFindJob() {
 $expectedJob = $this->loadFixture('active_job.yml');
 $actualJob = $this->repository->getById($expectedJob->getId());
 
 self::assertInstanceOf(Job::class, $actualJob);
 self::assertEquals($expectedJob->getId(), $actualJob->getId());
 } @michieltcs
  56. DEFENSE IN DEPTH UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE
 TESTS UI

    TESTS Scenario: Link to related job
 Given a job exists
 And there are related jobs available
 When that job is viewed
 Then a list of related jobs is shown
 And each related job links to the detail page of the related job @michieltcs
  57. DEFENSE IN DEPTH UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE
 TESTS UI

    TESTS @michieltcs
  58. DEFENSE IN DEPTH TESTER? UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE
 TESTS

    UI @michieltcs
  59. None
  60. CONTINUOUS TESTING UNIT TESTS INTEGRATION TESTS ACCEPTANCE TESTS UI TESTS

    SMOKE
 TESTS Cost Speed Exploratory
 testing Monitoring @michieltcs
  61. PIPELINE AS CODE node {
 stage('Run tests') {
 sh "phpunit"


    sh "behat"
 }
 
 stage('Build docker image') {
 sh "docker build -t jobservice:${env.BUILD_NUMBER} ."
 sh "docker push jobservice:${env.BUILD_NUMBER}"
 }
 
 stage('Deploy staging') {
 sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER}
 -i staging deploy.yml"
 }
 
 stage('Deploy production') {
 sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER}
 -i prod deploy.yml"
 }
 } @michieltcs
  62. DOCKERFILE FROM php:7.0-apache ADD vhost.conf /etc/apache2/sites-available/000-default.conf ADD . /var/www/html @michieltcs

  63. DEPLOYING: ROLLING UPDATE PULL IMAGE START NEW CONTAINER WAIT FOR

    PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER @michieltcs
  64. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker pull @michieltcs
  65. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker run @michieltcs
  66. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER wait_for: port=8080 delay=5 timeout=15 @michieltcs
  67. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER uri:
 url: http://localhost:8080/_health
 status_code: 200
 timeout: 30 @michieltcs
  68. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER template: src=haproxy.cfg.j2
 dest=/etc/haproxy/haproxy.cfg service: name=haproxy state=reloaded @michieltcs
  69. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER template: src=haproxy.cfg.j2
 dest=/etc/haproxy/haproxy.cfg service: name=haproxy state=reloaded @michieltcs
  70. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker stop docker rm @michieltcs
  71. BUILD PIPELINE @michieltcs

  72. FEEDBACK!

  73. RESULTS

  74. RESULTS ▸ Total build time per service < 10 minutes

    ▸ 50+ deploys per day ▸ Reduced issues ▸ Significantly improved page load times @michieltcs
  75. RESULTS ▸ Improved audience stats (time on page, pages per

    session, session duration, seo ranking, etc) ▸ Experimented with new tech/stacks (angular, jvm, event sourcing) ▸ Increased confidence, velocity & fun! @michieltcs
  76. LESSONS LEARNED ▸ Team acceptance ▸ Change is hard ▸

    Experience with new tech ▸ Docker orchestration & stability @michieltcs
  77. LESSONS LEARNED ▸ Stability of build pipelines: JavaScript testing ▸

    Business alignment ▸ Feature toggles ▸ Not enough focus on replacing legacy application @michieltcs
  78. LITERATURE https://trunkbaseddevelopment.com/

  79. Reduce cycle times. React faster to the market and test

    product ideas. Make things! We make Agile, DevOps and Continuous Delivery accessible! Hands-on, results- oriented approach to get you where you need to be. Modern infrastructure and pipelines in minutes.
  80. THANK YOU! @michieltcs / michiel@make.io www.michielrook.nl https://joind.in/talk/7687b