Continuous Deployment 2.0: Playtime is Over

8fc45f4725efe8e8bc8d6c1f92224b65?s=47 Michelangelo
September 12, 2019

Continuous Deployment 2.0: Playtime is Over

Continuous Deployment, TDD and Continuous Integration is nothing new anymore and many of you are already veterans deploying multiple times each day code into production. But you’re still suffering from cutting corners and doing something quick on production when time is pressing, violating the processes you all worked hard on to get approved.

In this talk we take this very concept to the next level and show you how we used unconventional ways to deploy faster with less issues and regained control over a 10+ year old legacy application.

8fc45f4725efe8e8bc8d6c1f92224b65?s=128

Michelangelo

September 12, 2019
Tweet

Transcript

  1. CONTINUOUS DEPLOYMENT 2.0 PLAYTIME IS OVER

  2. in it PROFESSIONAL PHP SERVICES ‣ CEO and co-founder of

    in2it ‣ Community leader 
 (PHP, Testing, DevOps, Azure, Security) ‣ OSS contributor ‣ Conference Speaker ‣ Coach at CoderDojo MICHELANGELO VAN DAM
  3. MICHIEL ROOK OUR CD WAS INSPIRED BY Continuous Deployment 2.0

    www.in2it.be - @in2itvof in it
  4. MICHIEL’S TALK WAS TO SAY THE LEAST… QUITE CONTROVERSIAL ▸

    deploy to master, no branches ▸ pair programming, quick code reviews ▸ 100% code coverage ▸ strangler pattern to replace 10+ year old legacy app Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  5. THAT GOT US THINKING… Continuous Deployment 2.0 www.in2it.be - @in2itvof

    in it
  6. LEGACY APPLICATIONS COMMON FOR ALL CLIENTS: Continuous Deployment 2.0 www.in2it.be

    - @in2itvof in it
  7. IN2IT PROCESSES? ? ? ? ? ? ? ? ?

    Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  8. REFACTORING ADDING FEATURES REWRITE FROM SCRATCH TESTING IS HARD ONLY

    NEW FEATURES TESTED LOTS OF WORK Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  9. Continuous Deployment 2.0 www.in2it.be - @in2itvof in it

  10. IN2IT PROCESSES ONLY MASTER PAIR PROGRAMMING 100% CODE COVERAGE STRANGLER

    PATTERN Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  11. IN2IT PROCESSES ONLY MASTER ❌ PAIR PROGRAMMING 100% CODE COVERAGE

    STRANGLER PATTERN Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  12. IN2IT PROCESSES ONLY MASTER ❌ PAIR PROGRAMMING ✅ 100% CODE

    COVERAGE STRANGLER PATTERN Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  13. IN2IT PROCESSES ONLY MASTER ❌ PAIR PROGRAMMING ✅ 100% CODE

    COVERAGE ✅ STRANGLER PATTERN Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  14. IN2IT PROCESSES ONLY MASTER ❌ PAIR PROGRAMMING ✅ 100% CODE

    COVERAGE ✅ STRANGLER PATTERN ❌ Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  15. LET’S SEE WHAT WE’RE MISSING DO WE NEED TO COMMIT

    TO MASTER? ▸ All PR’s are going into master branch ▸ Every developer has their own GIT repo ▸ Branching for features ▸ Branching for bug fixes ▸ Deployment Master ▸ Integrates one or more dev branches ▸ Pushes it onto master Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  16. DEVELOPERS WORK LOCALLY ON A PRIVATE GIT REPO OUR GIT

    WORKFLOW DEVELOPER PRIVATE REPO DEVELOPER PRIVATE REPO Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  17. ONCE DONE, THEY PUSH TO THEIR PUBLIC REPOS OUR GIT

    WORKFLOW DEVELOPER PRIVATE REPO DEVELOPER PRIVATE REPO DEVELOPER PUBLIC REPO DEVELOPER PUBLIC REPO Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  18. AND SEND A PULL REQUEST (PR) OUR GIT WORKFLOW DEVELOPER

    PRIVATE REPO DEVELOPER PRIVATE REPO DEVELOPER PUBLIC REPO DEVELOPER PUBLIC REPO TEAM REPO RELEASE BRANCH Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  19. OUR RELEASE MANAGER MERGES THE CHANGES IN OUR GIT WORKFLOW

    DEVELOPER PRIVATE REPO DEVELOPER PRIVATE REPO DEVELOPER PUBLIC REPO DEVELOPER PUBLIC REPO TEAM REPO MASTER BRANCH RELEASE MANAGER Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  20. CI KICKS IN AND DEPLOYS TO TEST/STAGING/PROD OUR GIT WORKFLOW

    DEVELOPER PRIVATE REPO DEVELOPER PRIVATE REPO DEVELOPER PUBLIC REPO DEVELOPER PUBLIC REPO TEAM REPO MASTER BRANCH RELEASE MANAGER CI TEST/STAGING/PROD Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  21. OUR CONCLUSION COMMITTING STRAIGHT TO MASTER ▸ Will make the

    role of “Release Master” obsolete ▸ Good for small teams ▸ Speeds up the deployment process ▸ In our case ▸ We work with distributed teams (3-5 people in each team) ▸ Will only work when all features are live from the start (feature flags) Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  22. LET’S SEE WHAT WE’RE MISSING… STRANGLER PATTERN ▸ Is a

    clean way to replace legacy code with clean, distributed and optimised code. ▸ Requires domain knowledge (especially for 10+ years of evolution of an app) ▸ Creates duplicate functionality during transitions Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  23. INCREMENTALLY MIGRATE A LEGACY SYSTEM BY GRADUALLY REPLACING SPECIFIC PIECES

    OF FUNCTIONALITY WITH NEW APPLICATIONS AND SERVICES. AS FEATURES FROM THE LEGACY SYSTEM ARE REPLACED, THE NEW SYSTEM EVENTUALLY REPLACES ALL OF THE OLD SYSTEM'S FEATURES, STRANGLING THE OLD SYSTEM AND ALLOWING YOU TO DECOMMISSION IT. Microsoft Cloud Architecture Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  24. LEGACY APPLICATION NEW REPLACEMENT STRANGLER FAÇADE Continuous Deployment 2.0 www.in2it.be

    - @in2itvof in it
  25. NEW REPLACEMENT STRANGLER FAÇADE LEGACY APPLICATION Continuous Deployment 2.0 www.in2it.be

    - @in2itvof in it
  26. OUR CONCLUSION STRANGLING LEGACY CODE ▸ Will take time to

    replace old with new code ▸ Until ready, duplicated functional will exist ▸ In our case ▸ We need to replace legacy code with better solutions ▸ We can live with duplicated functionality Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  27. IMPROVING LEGACY THE PATH TO Continuous Deployment 2.0 www.in2it.be -

    @in2itvof in it
  28. PEAR LIBS PECL EXTS FRAMEWORK X FRAMEWORK Y CUSTOM LIBS

    BUSINESS LOGIC Monolithic Code Monster Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  29. WE NEED A BETTER SOLUTION! Continuous Deployment 2.0 www.in2it.be -

    @in2itvof in it
  30. PEAR LIBS PECL EXTS FRAMEWORK X FRAMEWORK Y CUSTOM LIBS

    BUSINESS LOGIC Monolithic Code Monster FUNCTIONAL MICROSERVICE FUNCTIONAL MICROSERVICE FUNCTIONAL MICROSERVICE FUNCTIONAL MICROSERVICE FUNCTIONAL MICROSERVICE FUNCTIONAL MICROSERVICE FUNCTIONAL MICROSERVICE FUNCTIONAL MICROSERVICE FUNCTIONAL MICROSERVICE FUNCTIONAL MICROSERVICE Functional Microservices Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  31. I CAN HEAR YOU ASK: WHY MICROSERVICES? ▸ Not tied

    to a single technology ▸ A service per functionality makes it easier to upgrade without impacting the whole application ▸ Improved security on application, network and infrastructure ▸ Easy to scale and extend ▸ Better resilience against failing services Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  32. LET’S LOOK IF IT IS A SOLUTION FOR US PROS

    AND CONS Pro Con Not tied to a single technology Lots of different skillsets Easy to upgrade/fix or add features Lost in forest of services Improved security Requires lots of monitoring Better scalability Expensive* Improved resilience for failure (*) The cost lies in the higher number of “instances” and faster network required to run these services Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  33. MICROSERVICES & CONTINUOUS DEPLOYMENT Continuous Deployment 2.0 www.in2it.be - @in2itvof

    in it
  34. GOALS OF CONTINUOUS DEPLOYMENT? Continuous Deployment 2.0 www.in2it.be - @in2itvof

    in it
  35. CONTINUOUS DEPLOYMENT CAN BE THOUGHT OF AS AN EXTENSION OF

    CONTINUOUS INTEGRATION, AIMING AT MINIMIZING LEAD TIME, THE TIME ELAPSED BETWEEN DEVELOPMENT WRITING ONE NEW LINE OF CODE AND THIS NEW CODE BEING USED BY LIVE USERS, IN PRODUCTION. Agile Alliance - agilealliance.com Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  36. DOING THINGS MORE THAN ONCE LEAD UP TO… AUTOMATE DEPLOYMENT

    PROCESSES ▸ A series of actions taken before changes are put in production ▸ Chained together with fail switch to create a pipeline ▸ Provides full reports on each step of the delivery process ▸ Can be optimized over time ▸ Is repeatable Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  37. COMPUTERS ARE GREAT AT REPETITIVE TASKS! MINIMIZE LEAD TIME ▸

    Through automation, actions are executed faster in parallel ▸ Any failures will be reported immediately ▸ Any successes will be notified through ▸ E-mail ▸ Wiki ▸ Slack ▸ … Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  38. STRESS, PANIC, SICKNESS, … ALL HAVE AN IMPACT ON YOUR

    CONCENTRATION PREVENTING HUMAN ERROR! ▸ Automation removes human errors ▸ by stress or a “bad” day ▸ Optimized over time ▸ Repeatable for all stages Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  39. PROGRAMMING AGREEMENTS Continuous Deployment 2.0 www.in2it.be - @in2itvof in it

  40. COMMIT SMALL, COMMIT OFTEN Credits to @CalEvans THE PROGRAMMER’S MANTRA

    Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  41. PERFECT WORLD WOULD MAKE THIS POSSIBLE OPTIMAL COMMIT ▸ creating/updating

    class (no body) ▸ creating/changing class method (no body) ▸ creating/changing functionality in method ▸ creating/changing config (no body) ▸ … Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  42. ON THIS PLANET WE ARE SATISFIED WITH THIS REALISTIC COMMIT

    ▸ complete a functionality ▸ complete a logic operation ▸ complete a configuration change ▸ fix a bug or a vulnerability Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  43. SMALL AND OFTEN COMMITS BALANCE PROS AND CONS Pro Con

    Incremental small improvements Many commits for one task Errors can be reverted quickly Requires change of attitude Easy to review Small impact on production code Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  44. AIM FOR 100% CODE COVERAGE Michiel Rook - @michielcts LET’S

    GO FOR IT! Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  45. SMALL CHANGES REQUIRE MINIMAL OVERHEAD COMMIT SMALL WITH 100% COVERAGE

    ▸ small code change ▸ small test covering all lines ▸ make use of @covers annotation ▸ Run PHPUnit with “--strict-coverage” Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  46. EXAMPLE OF USING THE @COVERS ANNOTATION /** * Authentication throws

    exception for too short password * * @param string $password * * @covers \LoginForm\Auth\Service\AuthenticationService::__construct * @covers \LoginForm\Auth\Service\AuthenticationService::authenticate * @dataProvider shortPasswordProvider * @expectedException \InvalidArgumentException */ public function testAuthenticationThrowsExceptionForTooShortPassword(string $password) { $this->validator->expects($this->once()) ->method('isValid') ->willReturn(false); $authService = new AuthenticationService( $this->validator, $this->accountModel, $this->accountEntity, $this->twoFactorServiceMock ); $authService->authenticate('username', $password, false); $this->fail('Authentication service should throw an exception for too short password input'); } Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  47. EXAMPLE OF USING THE @COVERS ANNOTATION /** * Authentication throws

    exception for too short password * * @param string $password * * @covers \LoginForm\Auth\Service\AuthenticationService::__construct * @covers \LoginForm\Auth\Service\AuthenticationService::authenticate * @dataProvider shortPasswordProvider * @expectedException \InvalidArgumentException */ public function testAuthenticationThrowsExceptionForTooShortPassword(string $password) { $this->validator->expects($this->once()) ->method('isValid') ->willReturn(false); $authService = new AuthenticationService( $this->validator, $this->accountModel, $this->accountEntity, $this->twoFactorServiceMock ); $authService->authenticate('username', $password, false); $this->fail('Authentication service should throw an exception for too short password input'); } Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  48. SHOULD WE ADOPT 100% CODE COVERAGE? PROS AND CONS Pro

    Con Higher quality of code A huge effort Every line of code is accountable Requires change of attitude Huge time/money gain in long tail Time intensive Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  49. COVER MOST CRITICAL PART OF AN APPLICATION 100% WITH TESTS,

    EVEN IF IT’S ONLY 1% OF TOTAL COVERAGE. Michelangelo van Dam TWEETABLE QUOTE Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  50. THE PIPELINE Continuous Deployment 2.0 www.in2it.be - @in2itvof in it

  51. A COMMON CI/CD SETUP Continuous Deployment 2.0 www.in2it.be - @in2itvof

    in it
  52. WE CAN DO BETTER! Continuous Deployment 2.0 www.in2it.be - @in2itvof

    in it
  53. THE BEGINNING OF A BEAUTIFUL STORY CODE ASSEMBLY STEP ▸

    Check out SCM ▸ Optionally install submodules ▸ Composer packages Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  54. ARE WE GOOD TO GO? CODE QUALITY STEP ▸ Version

    checks (are we on latest version?) ▸ Vulnerability checks (CVE’s)* ▸ Unit testing ▸ Static Analysis (*) See SensioLabs for composer package validation service Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  55. LET’S PREPARE EVERYTHING LIKE A TRUE CHEF PROVISIONING STEP ▸

    Create a new environment ▸ Package the code and assets ▸ Sign the package with GPG ▸ Deploy the code ▸ Perform DB migrations ▸ Warm-up caches Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  56. TEST ALL THE THINGS, AND ADD SOME… RELEASE QUALITY STEP

    ▸ Integration testing ▸ Acceptance testing ▸ End-to-end testing ▸ Regression testing ▸ Performance testing ▸ Security testing ▸ Resilience testing ▸ Accessibility testing ▸ UX Testing ▸ Certificate Testing ▸ … Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  57. TIME TO SHIP IT!!! DEPLOYMENT STEP ▸ Stop crons, workers

    and daemons ▸ Phase out old system with new ▸ Put new system in load balancer ▸ Direct portion of traffic to new node ▸ Validate behaviour of new node ▸ Remove old system from load balancer ▸ Validate deployment ▸ Start crons, workers and daemons Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  58. KNOWING IS GOOD, KNOWING EVERYTHING IS BETTER - DAVE EGGARS

    “THE CIRCLE” REPORTING STEP ▸ Generate release notes ▸ Generate user documentation ▸ Notify training dept. of changes ▸ Generate release report/stats Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  59. Continuous Deployment 2.0 www.in2it.be - @in2itvof in it

  60. Continuous Deployment 2.0 www.in2it.be - @in2itvof in it

  61. WITH PIPELINES YOU HAVE… ENDLESS POSSIBILITIES TO DEPLOY Continuous Deployment

    2.0 www.in2it.be - @in2itvof in it
  62. AND WHEN SOMETHING DOES GOES WRONG… THE PIPELINE WILL STOP…

    Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  63. WE ROLL FORWARD WHEN FAILURE HAPPENS Continuous Deployment 2.0 www.in2it.be

    - @in2itvof in it
  64. LEAN DEVELOPMENT & DEVOPS Continuous Deployment 2.0 www.in2it.be - @in2itvof

    in it
  65. AGILE LEAN KANBAN Continuous Deployment 2.0 www.in2it.be - @in2itvof in

    it
  66. THE 3 WAY PHILOSOPHY Continuous Deployment 2.0 www.in2it.be - @in2itvof

    in it
  67. HOW IS WORK FLOWING FROM LEFT TO RIGHT? THE FIRST

    WAY: SYSTEM THINKING ▸ Encapsulates performance of the whole system ▸ How does “work” flows through the system? ▸ What are the constrains in that flow? ▸ Some key values: ▸ Never passing a known defect downstream ▸ Never allowing local optimisation to create global degradation ▸ Always seeking to increase flow ▸ Always seeking to achieve profound understanding of the system Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  68. HOW IS WORK FLOWING FROM LEFT TO RIGHT? THE FIRST

    WAY: SYSTEM THINKING ▸ Encapsulates performance of the whole system ▸ How does “work” flows through the system? ▸ What are the constrains in that flow? ▸ Some key values: ▸ Never passing a known defect downstream ▸ Never allowing local optimisation to create global degradation ▸ Always seeking to increase flow ▸ Always seeking to achieve profound understanding of the system Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  69. HOW IS FEEDBACK MOVING FROM RIGHT TO LEFT? THE SECOND

    WAY: AMPLIFY FEEDBACK LOOPS ▸ Shorten and amplify feedback loops ▸ Necessary corrections can be continually made ▸ More telemetry, more insight ▸ system data: CPU, memory, disk space ▸ app data: active users, load time, errors ▸ other: query count, items in shopping cart ▸ Make it available to EVERYONE!!! Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  70. WHAT SHOULD YOU DO WITH THE KNOWLEDGE THE THIRD WAY:

    CONTINUOUS LEARNING ▸ Continual experimentation, taking risks and learning from failure ▸ Understanding that repetition and practice is the prerequisite to mastery ▸ Learn and experiment as a hive, not as individual so global knowledge is shared and preserved Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  71. YOUR TURN Continuous Deployment 2.0 www.in2it.be - @in2itvof in it

  72. TEST IT ALL! CONTINUOUS DEPLOYMENT Continuous Deployment 2.0 www.in2it.be -

    @in2itvof in it
  73. SMALL REMINDER BY DESIGN ▸ Quality ▸ Security ▸ Privacy

    ▸ Resilience ▸ Accessibility Continuous Deployment 2.0 www.in2it.be - @in2itvof in it
  74. SET AGREEMENTS AS A TEAM IN NEED OF STRUCTURE? Continuous

    Deployment 2.0 www.in2it.be - @in2itvof in it
  75. WITH PIPELINES SUCCEED EVERY TIME Continuous Deployment 2.0 www.in2it.be -

    @in2itvof in it
  76. IMPROVE THE FEEDBACK LOOP KNOWLEDGE IS KEY! Continuous Deployment 2.0

    www.in2it.be - @in2itvof in it
  77. RESOURCE LIST Continuous Deployment 2.0 www.in2it.be - @in2itvof in it

  78. SOME TOOLS WE USE CONTINUOUS INTEGRATION Continuous Deployment 2.0 www.in2it.be

    - @in2itvof in it
  79. SOME TOOLS WE USE TESTING Continuous Deployment 2.0 www.in2it.be -

    @in2itvof in it
  80. SOME TOOLS WE USE DEPLOYMENT Envoyer Continuous Deployment 2.0 www.in2it.be

    - @in2itvof in it
  81. RESOURCES THAT INSPIRED ME… RECOMMENDED FOR YOU Continuous Deployment 2.0

    www.in2it.be - @in2itvof in it
  82. Continuous Deployment 2.0 www.in2it.be - @in2itvof in it

  83. QUESTIONS? ARE THERE ANY Continuous Deployment 2.0 www.in2it.be - @in2itvof

    in it