The Road to Continuous Deployment (DigitalOcean Amsterdam Meetup Feb 2017)

The Road to Continuous Deployment (DigitalOcean Amsterdam Meetup Feb 2017)

As presented at the DigitalOcean Amsterdam Meetup: The February Edition

It's a situation many of us are familiar with: a large legacy, monolithic application, limited or no tests, slow & manual release process, low velocity, no confidence... A lot of refactoring is required, but management keeps pushing for new features. How to proceed? Using examples and lessons learned from a real-world case, I'll show you how to replace a legacy application with a modern service-oriented architecture and build a continuous integration and deployment pipeline to deliver value from the first sprint. On the way, we’ll take a look at the process, automated testing, monitoring, master/trunk based development and various tips and best practices.

2f4800411154a8c66dde489448a044d2?s=128

Michiel Rook

February 24, 2017
Tweet

Transcript

  1. 3.
  2. 4.

    THE SYSTEM - SAN DIEGO ▸ Large aging monolith ▸

    Generates significant income ▸ Slow & complex ▸ Technical debt @michieltcs
  3. 5.

    SAN DIEGO FRONTEND MYSQL DB SAN DIEGO BACKEND LOAD BALANCERS

    / VARNISH ITBANEN INTERMEDIAIR NATIONALEVACATUREBANK SAN DIEGO FRONTEND SAN DIEGO FRONTEND SAN DIEGO FRONTEND SAN DIEGO BACKEND SAN DIEGO BACKEND SAN DIEGO BACKEND MEMCACHE FTP EXT. SERVICES SOLR @michieltcs
  4. 6.

    THE SYSTEM - SAN DIEGO ▸ Manual releases ▸ Fragile

    tests ▸ Low velocity ▸ Frequent outages & issues ▸ Frustrated team @michieltcs
  5. 9.

    APPROACH ▸ API first ▸ Services per domain object (job,

    jobseeker, ...) ▸ Migrate individual pages @michieltcs
  6. 11.

    PROXY @michieltcs RewriteEngine On # Serve feature from new service

    for internal network
 RewriteCond expr "%{HTTP:X-FORWARDED-FOR} -ipmatch '192.168.0.0/24'"
 RewriteRule ^/feature/(.*)$ ${NEW_SERVICE_URL}/$1 [P,L] # Proxy everything else to legacy application
 RewriteRule ^/(.*) ${LEGACY_URL}/$1 [P]
 ProxyPassReverse / ${LEGACY_URL}/
  7. 12.

    SAN DIEGO ELASTIC SEARCH LEGACY DB JOB SERVICE RMQ ITBANEN

    INTERMEDIAIR NATIONALEVACATUREBANK MONGO DB ITBANEN JOBSEEKER SERVICE NVB INTERMEDIAIR @michieltcs
  8. 13.
  9. 15.

    CD?

  10. 24.
  11. 25.
  12. 30.
  13. 32.
  14. 33.
  15. 39.
  16. 42.
  17. 43.
  18. 48.

    DEFENSE IN DEPTH UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE UI TESTS

    public function testJobCannotBeFound() {
 $jobRepository = $this->prophesize(JobRepository::class);
 $jobRepository->getById(EXPECTED_JOB_ID)
 ->shouldBeCalled()
 ->willReturn(false);
 
 $jobService = new JobService($jobRepository->reveal());
 
 $this->assertFalse($jobService->getById(EXPECTED_JOB_ID));
 } @michieltcs
  19. 49.

    DEFENSE IN DEPTH UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE
 TESTS UI

    TESTS public function testFindJob() {
 $expectedJob = $this->loadFixture('active_job.yml');
 $actualJob = $this->repository->getById($expectedJob->getId());
 
 self::assertInstanceOf(Job::class, $actualJob);
 self::assertEquals($expectedJob->getId(), $actualJob->getId());
 } @michieltcs
  20. 50.

    DEFENSE IN DEPTH UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE
 TESTS UI

    TESTS Scenario: Link to related job
 Given a job exists
 And there are related jobs available
 When that job is viewed
 Then a list of related jobs is shown
 And each related job links to the detail page of the related job @michieltcs
  21. 53.
  22. 54.

    CONTINUOUS TESTING UNIT TESTS INTEGRATION TESTS ACCEPTANCE TESTS UI TESTS

    SMOKE
 TESTS Cost Speed Exploratory
 testing Monitoring @michieltcs
  23. 55.

    PIPELINE AS CODE node {
 stage('Run tests') {
 sh "phpunit"


    sh "behat"
 }
 
 stage('Build docker image') {
 sh "docker build -t jobservice:${env.BUILD_NUMBER} ."
 sh "docker push jobservice:${env.BUILD_NUMBER}"
 }
 
 stage('Deploy staging') {
 sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER}
 -i staging deploy.yml"
 }
 
 stage('Deploy production') {
 sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER}
 -i prod deploy.yml"
 }
 } @michieltcs
  24. 56.

    DEPLOYING: ROLLING UPDATE PULL IMAGE START NEW CONTAINER WAIT FOR

    PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER @michieltcs
  25. 57.

    DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker pull @michieltcs
  26. 58.

    DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker run @michieltcs
  27. 59.

    DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER wait_for: port=8080 delay=5 timeout=15 @michieltcs
  28. 60.

    DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER uri:
 url: http://localhost:8080/_health
 status_code: 200
 timeout: 30 @michieltcs
  29. 61.

    DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER template: src=haproxy.cfg.j2
 dest=/etc/haproxy/haproxy.cfg service: name=haproxy state=reloaded @michieltcs
  30. 62.

    DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER template: src=haproxy.cfg.j2
 dest=/etc/haproxy/haproxy.cfg service: name=haproxy state=reloaded @michieltcs
  31. 63.

    DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker stop docker rm @michieltcs
  32. 65.
  33. 66.
  34. 67.

    RESULTS ▸ Total build time per service < 10 minutes

    ▸ 50+ deploys per day ▸ Reduced issues ▸ Significantly improved page load times @michieltcs
  35. 68.

    RESULTS ▸ Improved audience stats (time on page, pages per

    session, session duration, seo ranking, etc) ▸ Experimented with new tech/stacks (angular, jvm, event sourcing) ▸ Increased confidence, velocity & fun! @michieltcs
  36. 69.

    LESSONS LEARNED ▸ Team acceptance ▸ Change is hard ▸

    Experience with new tech ▸ Docker orchestration & stability @michieltcs
  37. 70.

    LESSONS LEARNED ▸ Stability of build pipelines: JavaScript testing ▸

    Cap feature toggles ▸ Business alignment ▸ Not enough focus on replacing legacy application @michieltcs
  38. 72.

    Reduce cycle times. React faster to the market and test

    product ideas. Make things! We make Agile, DevOps and Continuous Delivery accessible! Hands-on, results- oriented approach to get you where you need to be. Modern infrastructure and pipelines in minutes.