The Road to Continuous Deployment (DevOpsCon München 2016)

2f4800411154a8c66dde489448a044d2?s=47 Michiel Rook
December 06, 2016

The Road to Continuous Deployment (DevOpsCon München 2016)

It’s a situation many of us are familiar with: a large legacy application, limited or no tests, slow and manual release process, low velocity, no confidence … Oh, and management wants new features, fast. But how to proceed? Using examples and lessons learned from a real-world case, I’ll show you how to strangle the legacy application with a modern service architecture and build a continuous deployment pipeline to deliver value from the first sprint. On the way, we take a look at testing strategies and various (possibly controversial!) tips and best practices.

2f4800411154a8c66dde489448a044d2?s=128

Michiel Rook

December 06, 2016
Tweet

Transcript

  1. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY MICHIEL

    ROOK @michieltcs
  2. ▸ Java, PHP & Scala developer ▸ Consultant, trainer, speaker

    ▸ make.io ▸ Dutch Web Alliance ▸ @michieltcs
  3. TODAY ▸ Background ▸ The approach ▸ Process / standards

    ▸ Build pipelines ▸ Results & lessons learned @michieltcs
  4. None
  5. THE SYSTEM - SAN DIEGO ▸ ... or the Big

    Ball Of Mud ▸ Large legacy monolith ▸ Generates significant income ▸ Slow & complex ▸ Technical debt @michieltcs
  6. SAN DIEGO FRONTEND MYSQL DB SAN DIEGO BACKEND LOAD BALANCERS

    / VARNISH ITBANEN INTERMEDIAIR NATIONALEVACATUREBANK SAN DIEGO FRONTEND SAN DIEGO FRONTEND SAN DIEGO FRONTEND SAN DIEGO BACKEND SAN DIEGO BACKEND SAN DIEGO BACKEND MEMCACHE FTP EXT. SERVICES SOLR @michieltcs
  7. THE SYSTEM - SAN DIEGO ▸ Infrequent, manual releases ▸

    Fragile tests ▸ Low velocity ▸ Frequent outages / bugs / issues ▸ Frustrated team ▸ Low confidence modifying existing code @michieltcs
  8. GOALS ▸ Reduce issues ▸ Reduce cycle time ▸ Increase

    productivity ▸ Increase motivation @michieltcs
  9. REFACTOR? REBUILD?

  10. APPROACH ▸ Strangler pattern ▸ API first ▸ Services per

    domain object (job, jobseeker, ...) ▸ Migrate individual pages @michieltcs
  11. ORIGINAL MONOLITH PROXY SERVICE ORIGINAL MONOLITH ORIGINAL MONOLITH SERVICE SERVICE

    SERVICE PROXY DB DB DB DB DB DB @michieltcs
  12. PROXY @michieltcs RewriteEngine On # Serve feature from new service

    for internal network
 RewriteCond expr "%{HTTP:X-FORWARDED-FOR} -ipmatch '192.168.0.0/24'"
 RewriteRule ^/feature/(.*)$ ${NEW_SERVICE_URL}/$1 [P,L] # Proxy everything else to legacy application
 RewriteRule ^/(.*) ${LEGACY_URL}/$1 [P]
 ProxyPassReverse / ${LEGACY_URL}/
  13. APPROACH ▸ Services behind load balancers ▸ Access legacy db’s

    ▸ Continuous deployment ▸ Docker containers ▸ Frontends are services @michieltcs
  14. SAN DIEGO ELASTIC SEARCH LEGACY DB JOB SERVICE RMQ ITBANEN

    INTERMEDIAIR NATIONALEVACATUREBANK MONGO DB ITBANEN JOBSEEKER SERVICE NVB INTERMEDIAIR @michieltcs
  15. PROCESS

  16. STARTING OFF ▸ Scrum, 1 week sprints ▸ TDD /

    BDD ▸ Definition of Done ▸ Team mindset ▸ Focus on value @michieltcs
  17. CONTINUOUS EVERYTHING

  18. DEV BUILD / TEST CONTINUOUS INTEGRATION @michieltcs

  19. DEV BUILD / TEST ACCEPTANCE PRODUCTION CONTINUOUS DELIVERY @michieltcs

  20. DEV BUILD / TEST STAGING PRODUCTION CONTINUOUS DEPLOYMENT @michieltcs

  21. WHY CONTINUOUS DEPLOYMENT ▸ Small steps ▸ Early feedback ▸

    Reduce cycle time ▸ Reduce risk ▸ Experiments! @michieltcs
  22. SUCCESSFUL TEAMS HAVE @michieltcs

  23. ONLY COMMIT TO MASTER

  24. NO BRANCHES

  25. NO BRANCHES REALLY.

  26. PAIR PROGRAMMING

  27. BOY SCOUT RULE

  28. QUALITY GATES

  29. 100% CODE COVERAGE*

  30. FEATURE TOGGLES, A/B TESTS @michieltcs

  31. DEVOPS

  32. DASHBOARDS

  33. BUILD
 PIPELINE

  34. AUTOMATE REPEATABLE THINGS

  35. AUTOMATION ▸ Builds ▸ Testing ▸ Deployments ▸ Orchestration ▸

    Config management @michieltcs
  36. EVERY COMMIT GOES TO PRODUCTION

  37. CONTINUOUS TESTING

  38. DEFENSE IN DEPTH UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE
 TESTS UI

    TESTS @michieltcs
  39. DEFENSE IN DEPTH UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE UI TESTS

    public function testJobCannotBeFound() {
 $jobRepository = $this->prophesize(JobRepository::class);
 $jobRepository->getById(EXPECTED_JOB_ID)
 ->shouldBeCalled()
 ->willReturn(false);
 
 $jobService = new JobService($jobRepository->reveal());
 
 $this->assertFalse($jobService->getById(EXPECTED_JOB_ID));
 } @michieltcs
  40. DEFENSE IN DEPTH UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE
 TESTS UI

    TESTS public function testFindJob() {
 $expectedJob = $this->loadFixture('active_job.yml');
 $actualJob = $this->repository->getById($expectedJob->getId());
 
 self::assertInstanceOf(Job::class, $actualJob);
 self::assertEquals($expectedJob->getId(), $actualJob->getId());
 } @michieltcs
  41. DEFENSE IN DEPTH UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE
 TESTS UI

    TESTS Scenario: Link to related job
 Given a job exists
 And there are related jobs available
 When that job is viewed
 Then a list of related jobs is shown
 And each related job links to the detail page of the related job @michieltcs
  42. DEFENSE IN DEPTH UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE
 TESTS UI

    TESTS @michieltcs
  43. DEFENSE IN DEPTH MANUAL TESTING? UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE


    TESTS UI @michieltcs
  44. CONTINUOUS TESTING UNIT TESTS INTEGRATION TESTS ACCEPTANCE TESTS UI TESTS

    SMOKE
 TESTS Cost Speed Exploratory
 testing Monitoring @michieltcs
  45. PIPELINE AS CODE node {
 stage 'Run tests'
 sh "phpunit"


    sh "behat"
 
 stage 'Build docker image'
 sh "phing build"
 sh "docker build -t jobservice:${env.BUILD_NUMBER} ."
 sh "docker push jobservice:${env.BUILD_NUMBER}"
 
 stage 'Deploy acceptance'
 sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER}
 -i acc deploy.yml"
 
 stage 'Deploy production'
 sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER}
 -i prod deploy.yml"
 } @michieltcs
  46. DOCKERFILE FROM php:7.0-apache ADD vhost.conf /etc/apache2/sites-available/000-default.conf ADD . /var/www/html @michieltcs

  47. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER @michieltcs
  48. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker pull @michieltcs
  49. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker run @michieltcs
  50. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER wait_for: port=8080 delay=5 timeout=15 @michieltcs
  51. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER uri:
 url: http://localhost:8080/_health
 status_code: 200
 timeout: 30 @michieltcs
  52. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER template: src=haproxy.cfg.j2
 dest=/etc/haproxy/haproxy.cfg service: name=haproxy state=reloaded @michieltcs
  53. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER template: src=haproxy.cfg.j2
 dest=/etc/haproxy/haproxy.cfg service: name=haproxy state=reloaded @michieltcs
  54. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE

    TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker stop docker rm @michieltcs
  55. BUILD PIPELINE @michieltcs

  56. FEEDBACK!

  57. RESULTS

  58. RESULTS ▸ Total build time per service < 10 minutes

    ▸ Significantly improved page load times ▸ Improved audience stats (time on page, pages per session, session duration, traffic, seo ranking, etc) ▸ Increased confidence and velocity ▸ Experimented with new tech/stacks (angular, jvm, event sourcing) ▸ More fun @michieltcs
  59. LESSONS LEARNED ▸ Team acceptance ▸ Change is hard ▸

    Overhead of weekly sprint; requires discipline ▸ Docker orchestration ▸ Issues with traffic between Amazon <-> on-premise datacenter ▸ Javascript testing @michieltcs
  60. LESSONS LEARNED ▸ Experience with new tech ▸ Stability of

    build pipelines ▸ Business alignment ▸ Feature toggles ▸ Not enough focus on replacing legacy application @michieltcs
  61. LITERATURE

  62. Turn-key Continuous Deployment Zero downtime deployments Modern, autoscaling infrastructure with

    built-in monitoring Pipeline in five minutes
  63. THANK YOU! @michieltcs / michiel@make.io www.michielrook.nl