Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Real World App Deployment Processes @ Bright People

Real World App Deployment Processes @ Bright People

CI and CD of both massive legacy and greenfield applications in a Windows world. Automated build and deployment process using TeamCity and Octopus Deploy.

Josh Michielsen

May 17, 2017
Tweet

More Decks by Josh Michielsen

Other Decks in Programming

Transcript

  1. Continuous Integration & Deployment Presented by: Josh Michielsen DevOps Systems

    Administrator Bright People Technologies
  2. Product Overview 2 Legacy Applications › Workforce logistics › Workforce

    mobilisation › Planning and rostering › Reporting and forecasting › Workforce planning › Workforce validation › Verification and compliance › Self-service model Overview › Web based; SaaS › Focus on the resources industry › VB.NET/C#.NET › .NET Framework › ASP.NET Webforms, MVC, & Web API › MSTest; NUnit › SQL Server › 2+ Million LoC combined
  3. Product Overview (2) 3 Cited Tech › Greenfield SaaS –

    https://cited.com.au › Targeted to all individuals and organisations › Workforce verification and compliance › E.g. National police checks; Visa checks; etc. › C#.NET › .NET Framework › ASP.NET MVC & Web API. Can be deployed independently. › NUnit › EventStore & MySQL
  4. Hotfix Deployment Pipeline 4 Legacy Applications Git - Assembla ›

    Automatically generate merge requests › Mandatory code reviews › Ticket/card tracking CI - TeamCity › Git hook, triggered after merge › Build via MSBuild › Test via NUnit › Generate .nupkg with OctoPack › Generate release notes (PoSH) › Push to NuGet feed › Build server config/environment specified in DSC as much as possible. CD - Octopus Deploy QA › Deploys and “installs” .nupkg files › Run scripts, configure IIS, import certificates, etc. › Repeatable single click deployments through environments › Enforces environment lifecycles, version requirements, etc. CI UAT Prod Demo › Alternating 2 week release cycle › All commits are made to master › Card-based testing › Branched for release at sprint milestone › Full regression testing › Pet-style infrastructure › Validate: Over 115 deployment steps, 12-18 mins › Logistics: 25 steps, 4-6 mins Master Channel Release Channel Details
  5. Deployment Pipeline 5 Cited Git - Assembla › Automatically generate

    merge requests › Mandatory code reviews › Ticket/card tracking CI - TeamCity › Git hook, triggered before and after merge. › Build via MSBuild › Test via NUnit › Generate .nupkg with OctoPack › Generate release notes (PoSH) › Push to NuGet feed CD - Octopus Deploy QA › Deploys and “installs” .nupkg files › Run scripts, configure IIS, import certificates, etc. › Repeatable single click deployments through environments › Enforces environment lifecycles, version requirements, etc. CI (auto) UAT Prod Demo › Almost identical workflow as legacy teams › Multiple (no limit) releases per day to production (release early, release often). › All commits trigger CI build › No branching. Merged builds deployed automatically to CI › Some manual testing, much shorter than legacy › Infrastructure deployment and configuration automated (CloudFormation, cloud-init, and PowerShell DSC (deployed via Octopus) › Deployment time: 1-2 minutes Details
  6. General Information 6 Who “owns” the deployment process? › DevOps

    (mostly me) invests the most time into management › Collaboration with the lead developers › We actively seek ways to improve the process, especially in the areas of speed and reliability Improvements in the last 12 months? › Migrated build and deployment infrastructure to cloud (AWS) › More proactive in updating CI/CD platforms › Dedicated resource (me), ‘DevOps’ process and implementation was started by a dev. When he left the business saw the need for a more permanent ‘DevOps’ engineer (with larger ops background). Further improvements in the next 12 months? › Heavier investment in PowerShell DSC › Octopus Deploy (via API) and TeamCity configuration (via their Kotlin DSL) as code and in VCS. › Really – just as much configuration as code as possible. › Provides visibility when changes are made › Allows for PRs/merge request and code reviews. › Provides repeatability and consistency › Devs have even greater control of environments and config, while having less access to the machines and admin consoles– massive improvement for security › Terraform, packer, and vault are all under consideration at the moment › Legacy: Better base images; Sub 10 min deployments for Validate (physics permitting) › Cited: Lambda (API), Elastic Beanstalk or Docker (MVC) Best/worst decisions made? › Best: Octopus Deploy › Worst: Devs with local admin to prod machines
  7. Thanks! jmichielsen@brighttechnologies.com.au

  8. Prod deployments (Cited) and downtime? › “Rolling” deployments - only

    1 web server updated a time › Connections are drained from IIS before deployment All commits to master – dealing with “half baked features” › All developers are required to use feature toggles › Toggles are often controlled via Octopus variables › E.g. Large features are generally tested in UAT, therefore a featureX.IsEnabled variable would exist twice, once as a catch-all across environments (value: false), and again with a scope of only UAT (value: true) › In rare cases devs will branch and can manually trigger builds (+ tests) against that branch. Only master branches can be deployed though Questions Received How are the devs handling the move to ‘DevOps’? › Process started by a dev, definitely helped to sell it › Devs want to commit code and not have to worry about the how it gets delivered Removing developer access to machines & consoles – priority and feasibility? › DevOps and config management story has greatly improved in the Windows world, so feasibility › Greater investment into PowerShell DSC (think Puppet DSL, or Chef’s Ruby-based DSL) is a high priority, making the change much more feasible › Removing admin access to prod is very high priority