The Road to Continuous Deployment (PHP UK Conference 2017)

2f4800411154a8c66dde489448a044d2?s=47 Michiel Rook
February 16, 2017

The Road to Continuous Deployment (PHP UK Conference 2017)

It's a situation many of us are familiar with: a large legacy, monolithic application, limited or no tests, slow & manual release process, low velocity, no confidence...

A lot of refactoring is required, but management keeps pushing for new features. How to proceed?

Using examples and lessons learned from a real-world case, I'll show you how to replace a legacy application with a modern service-oriented architecture and build a continuous integration and deployment pipeline to deliver value from the first sprint. On the way, we’ll take a look at the process, automated testing, monitoring, master/trunk based development and various tips and best practices.

2f4800411154a8c66dde489448a044d2?s=128

Michiel Rook

February 16, 2017
Tweet

Transcript

  1. 2.

    ▸ Java, PHP & Scala developer ▸ Consultant, trainer, speaker

    ▸ make.io ▸ Dutch Web Alliance ▸ @michieltcs
  2. 3.

    TODAY ▸ Background ▸ The approach ▸ Process / standards

    ▸ Build pipelines ▸ Results & lessons learned @michieltcs
  3. 4.
  4. 5.

    THE SYSTEM - SAN DIEGO ▸ ... or the Big

    Ball Of Mud ▸ Large legacy monolith ▸ Generates significant income ▸ Slow & complex ▸ Technical debt @michieltcs
  5. 6.

    SAN DIEGO FRONTEND MYSQL DB SAN DIEGO BACKEND LOAD BALANCERS

    / VARNISH ITBANEN INTERMEDIAIR NATIONALEVACATUREBANK SAN DIEGO FRONTEND SAN DIEGO FRONTEND SAN DIEGO FRONTEND SAN DIEGO BACKEND SAN DIEGO BACKEND SAN DIEGO BACKEND MEMCACHE FTP EXT. SERVICES SOLR @michieltcs
  6. 7.

    THE SYSTEM - SAN DIEGO ▸ Infrequent, manual releases ▸

    Fragile tests ▸ Low velocity ▸ Frequent outages / bugs / issues ▸ Frustrated team ▸ Low confidence modifying existing code @michieltcs
  7. 8.

    GOALS ▸ Reduce issues ▸ Reduce lead & cycle time

    ▸ Increase productivity ▸ Increase motivation @michieltcs
  8. 10.

    APPROACH ▸ Strangler pattern ▸ API first ▸ Services per

    domain object (job, jobseeker, ...) ▸ Migrate individual pages @michieltcs
  9. 12.

    PROXY @michieltcs RewriteEngine On # Serve feature from new service

    for internal network
 RewriteCond expr "%{HTTP:X-FORWARDED-FOR} -ipmatch '192.168.0.0/24'"
 RewriteRule ^/feature/(.*)$ ${NEW_SERVICE_URL}/$1 [P,L] # Proxy everything else to legacy application
 RewriteRule ^/(.*) ${LEGACY_URL}/$1 [P]
 ProxyPassReverse / ${LEGACY_URL}/
  10. 13.

    APPROACH ▸ Services behind load balancers ▸ Access legacy db’s

    ▸ Continuous deployment ▸ Docker containers ▸ Frontends are services @michieltcs
  11. 14.

    SAN DIEGO ELASTIC SEARCH LEGACY DB JOB SERVICE RMQ ITBANEN

    INTERMEDIAIR NATIONALEVACATUREBANK MONGO DB ITBANEN JOBSEEKER SERVICE NVB INTERMEDIAIR @michieltcs
  12. 15.
  13. 17.

    CD?

  14. 21.

    WHY CONTINUOUS DEPLOYMENT ▸ Small steps ▸ Early feedback ▸

    Reduce time to recover ▸ Experiments! @michieltcs
  15. 23.

    TDD

  16. 24.

    BDD

  17. 29.
  18. 30.
  19. 37.
  20. 38.
  21. 44.
  22. 47.
  23. 48.
  24. 54.

    DEFENSE IN DEPTH UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE UI TESTS

    public function testJobCannotBeFound() {
 $jobRepository = $this->prophesize(JobRepository::class);
 $jobRepository->getById(EXPECTED_JOB_ID)
 ->shouldBeCalled()
 ->willReturn(false);
 
 $jobService = new JobService($jobRepository->reveal());
 
 $this->assertFalse($jobService->getById(EXPECTED_JOB_ID));
 } @michieltcs
  25. 55.

    DEFENSE IN DEPTH UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE
 TESTS UI

    TESTS public function testFindJob() {
 $expectedJob = $this->loadFixture('active_job.yml');
 $actualJob = $this->repository->getById($expectedJob->getId());
 
 self::assertInstanceOf(Job::class, $actualJob);
 self::assertEquals($expectedJob->getId(), $actualJob->getId());
 } @michieltcs
  26. 56.

    DEFENSE IN DEPTH UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE
 TESTS UI

    TESTS Scenario: Link to related job
 Given a job exists
 And there are related jobs available
 When that job is viewed
 Then a list of related jobs is shown
 And each related job links to the detail page of the related job @michieltcs
  27. 59.
  28. 60.

    CONTINUOUS TESTING UNIT TESTS INTEGRATION TESTS ACCEPTANCE TESTS UI TESTS

    SMOKE
 TESTS Cost Speed Exploratory
 testing Monitoring @michieltcs
  29. 61.

    PIPELINE AS CODE node {
 stage('Run tests') {
 sh "phpunit"


    sh "behat"
 }
 
 stage('Build docker image') {
 sh "docker build -t jobservice:${env.BUILD_NUMBER} ."
 sh "docker push jobservice:${env.BUILD_NUMBER}"
 }
 
 stage('Deploy staging') {
 sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER}
 -i staging deploy.yml"
 }
 
 stage('Deploy production') {
 sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER}
 -i prod deploy.yml"
 }
 } @michieltcs
  30. 63.

    DEPLOYING: ROLLING UPDATE PULL IMAGE START NEW CONTAINER WAIT FOR

    PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER @michieltcs
  31. 64.

    DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker pull @michieltcs
  32. 65.

    DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker run @michieltcs
  33. 66.

    DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER wait_for: port=8080 delay=5 timeout=15 @michieltcs
  34. 67.

    DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER uri:
 url: http://localhost:8080/_health
 status_code: 200
 timeout: 30 @michieltcs
  35. 68.

    DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER template: src=haproxy.cfg.j2
 dest=/etc/haproxy/haproxy.cfg service: name=haproxy state=reloaded @michieltcs
  36. 69.

    DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER template: src=haproxy.cfg.j2
 dest=/etc/haproxy/haproxy.cfg service: name=haproxy state=reloaded @michieltcs
  37. 70.

    DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker stop docker rm @michieltcs
  38. 72.
  39. 73.
  40. 74.

    RESULTS ▸ Total build time per service < 10 minutes

    ▸ 50+ deploys per day ▸ Reduced issues ▸ Significantly improved page load times @michieltcs
  41. 75.

    RESULTS ▸ Improved audience stats (time on page, pages per

    session, session duration, seo ranking, etc) ▸ Experimented with new tech/stacks (angular, jvm, event sourcing) ▸ Increased confidence, velocity & fun! @michieltcs
  42. 76.

    LESSONS LEARNED ▸ Team acceptance ▸ Change is hard ▸

    Experience with new tech ▸ Docker orchestration & stability @michieltcs
  43. 77.

    LESSONS LEARNED ▸ Stability of build pipelines: JavaScript testing ▸

    Business alignment ▸ Feature toggles ▸ Not enough focus on replacing legacy application @michieltcs
  44. 79.

    Reduce cycle times. React faster to the market and test

    product ideas. Make things! We make Agile, DevOps and Continuous Delivery accessible! Hands-on, results- oriented approach to get you where you need to be. Modern infrastructure and pipelines in minutes.