BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY #GHC16 2016 Creating a Continuous Integration/Deployment System Melissa Kam Sabeen Syed
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Workshop Objectives • Understand the concept of Continuous Integration and Continuous Deployment • Understand the benefits of CI/CD • Become familiar with Jenkins, a commonly used CI/CD tool, and how a CI/CD pipeline can be implemented in Jenkins
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY What is CI/CD? • Automation of testing, building, and deploying code • Continuous Integration — Testing and creating a deployable build • Continuous Deployment — Deploying the build • Continuous Deployment vs. Delivery — Delivery requires a manual step for controlling when to deploy code • Some commonly used open source CI/CD systems include Jenkins, Travis CI, and Buildbot
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Example of a Manual Process • Going from source code to a tangible result — A developer writes code and adds it to a source code repository — Another developer downloads the new code and runs unit tests against it — A test engineer downloads the code, runs the application locally on their computer, and then runs functional tests against it
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Example of a Manual Process • Going from source code to a tangible result — An operations engineer downloads the code and compiles it into a build artifact, like an executable or a binary — An operations engineer deploys the build artifact to the production environment — A test engineer runs functional tests against the production environment
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Why CI/CD? • Minimal to no manual intervention required • Faster release cycles • Provides a repeatable and more reliable testing and deployment process • Improved code quality from constant automated testing • Provides central location for anyone to view the process • Allows anyone to initiate the process
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Example CI/CD Workflow Unit Tests Functional Tests Create Build Tag Build with Release Version Deploy Build to Production Test Production
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Workshop Environments • Pokestarter — Sample Python application that randomly returns one of three Pokemon — https://github.com/melissa-kam/pokemonGHC —
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Workshop Environments • Jenkins — <ip-address>:8080 • Test Environment — <ip-address>:5050 • Production Environment — <ip-address>:5000
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Jenkins and Jenkins Job • Jenkins is a system and user interface for automating various tasks — e.g., downloading source code, running shell commands, executing code • A Jenkins “job” is a grouping of these tasks and consists of: — Source Code Management — Build Triggers — Build Environment — Build Steps — Post-build Actions • A “build” in this case is a single execution of a Jenkins job, not necessarily the same as a “release build”
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Unit Testing • Testing a small, isolated unit of code, such as a single method • Tests without having the code running • Often written by the developer of the code rather than a separate testing developer • Pokestarter example: — Testing that calling the get_random_pokemon method returns a random Pokemon
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY “Unit Tests” Job • Click on the “Unit Tests” job in Jenkins under the “CI” tab • Click “Configure” to view the configuration of the job • Source Code Management — GitHub repository with sample application code • Build — Create a Python virtual environment (virtualenv) for running tests — Install Python packages required for testing — Use a test runner (nosetests) to run the unit tests with the option of creating a report file with the tests results • Post Build Action — Consume the created test results file and present it in a more readable fashion
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY “Unit Tests” Job • Click “Build Now” to start the job • Click “#1” that appears under the Build History • Click “Console Output” to see the output of all the steps specified in the job • After the tests are complete, click “Test Result.” This may require refreshing the page you’re currently on.
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY “Unit Tests” Job • Click “tests.unit.test_starters” and “TestPokemon” to drill down to the individual test results • These are the tests in the file tests/unit/test_starters.py and in the class TestPokemon
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Functional Testing • Testing the end to end functionality of a feature against a running system • Similar or identical to integration testing • More rigorous than unit testing • Because it tests against a running system, requires some sort of test environment • Pokestarter example: — Testing that a request to the API returns a 200 status code and the expected body response in JSON format
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY “Functional Tests” Job • Click on the “Functional Tests” job in Jenkins under the “CI” tab • Click “Configure” • Source Code Management — GitHub repository with sample application code • Build — Deploy latest GitHub code to the test environment — Create a Python virtual environment for running tests — Install Python packages required for testing — Use a test runner (nosetests) to run the functional tests with the option of creating a report file with the tests results • Post Build Action — Consume the created test results file and present it in a more readable fashion
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY “Functional Tests” Job • Click “Build Now” to start the job • Click “#1” that appears under the Build History • Click “Console Output” to see the output of all the steps specified in the job • Refresh the test environment (<ip>:5050) to see that changes have been deployed • After the tests are complete, click “Test Result.” This may require refreshing the page you’re currently on.
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Creating a Release Build • Creating and packaging a deployable artifact • The artifact is a “release build” of the current code that can be deployed to an environment at any time • Can also involve uploading the build artifact to a file system that contains all the releasable artifacts • The build is given a “tag,” which is the name or version — For builds that are being created often, can use current date as a temporary tag • Pokestarter example: — Copying all the needed files into a single archive file (like a .tar or .zip file)
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY “Build Release Artifact” Job • Click “Build Release Artifact” under the “CI” tab and “Configure” • Parameters — Has a parameter for specifying the version name • Source Code Management — GitHub repository with sample application code • Build — Copies the needed files to run the Python sample application and archives them in a .tar file with the specified tag as the file name — “Uploads” the files to a release_artifacts folder • Click “Build with Parameters,” input any value for RELEASE_TAG, and click “Build” • Click “#1” > “Console Output” to see the output of all the steps specified in the job
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Continuous Integration • Unit testing, functional testing, and building all run in serial • Each step of the process must pass before it is logical to move onto the next one — If unit tests fail, most likely functional tests will fail — If functional tests fail, the code is not ready to be deployed and should not be built • At the end of the CI process, there is a tested build artifact that is ready to be deployed either automatically or manually Unit Tests Functional Tests Create Build
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY “Continuous Integration Pipeline” Job • Pipeline/Build Flow jobs orchestrate how to run different jobs • Rather than manually triggering each job, the pipeline job specifies the order and runs them all automatically • If one job fails, the pipeline exits without proceeding by default • Click the “Continuous Integration Pipeline” job > “Configure” and scroll down to “Pipeline” — The first block of code gets the current date and time, which will be used as the tag for the release artifact — The next specifies the different stages of the pipeline and which jobs to build
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY “Continuous Integration Pipeline” Job • Click “Build Now” to start the pipeline job • Each stage of this pipeline triggers the building of the same Unit Tests, Functional Tests, and Build Release Artifact jobs • If you go to any of these jobs, there will be #1 and #2 now in the build history
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Continuous Deployment Tag Build with Release Version Deploy Build to Production Test Production
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Tagging a Build with a Release Version • Because artifacts are continuously being built, their tags might not always be in an easy to consume format • This is a completely optional step if the original tag is suitable for a release tag • Pokestarter example: — The builds created by the CI pipeline are tagged with the date and time they were built, but we want the build to be called v1.0.0.
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY “Tag Release Version” Job • Click on “Tag Release Version” under the “CD” tab > “Configure” • Parameters — Has a parameter for specifying the original version name and the desired version name • Source Code Management — GitHub repository with sample application code • Build — Copies the original .tar file and renames it with the new tag
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY “Tag Release Version” Job • Refer back to the “Build Release Artifact” job under the CI tab and click on the latest job in the job history (#2) • Click “Parameters” to see what the tag was for this build • Copy this tag • Navigate back to “Tag Release Version” under the “CD” tab • Click “Build with Parameters” and use the copied tag as the ORIGINAL_TAG and v1.0.0 as the RELEASE_TAG • Click “Build” and view the console, though there won’t be much in the console logs for this job
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Deploying a Build to Production • Uploading the specified release artifact to the production environment • Reconfiguring and restarting the application if needed • Executing database migrations if needed • Can be automated using tools like Chef, Ansible, and Puppet
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY “Deploy to Production” Job • Click “Deploy to Production” under “CD” and then “Configure” • Parameters — Has a parameter for specifying the version to deploy • Source Code Management — Only has the GitHub code because there is a deploy script in the repo — The code being deployed has already been built • Runs the deploy_to_production script with the specified version • Click “Build Now” and then “#1” > “Console Output” to see the output of all the steps specified in the job • Refresh the production environment (:5000) to see the changes
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Testing Production • Functional tests validating the new code running on production • Usually a subset of the functional test suite with the essential “smoke” tests • The code has already been tested in the CI process, so it doesn’t necessarily need the full suite to run again • Focus on validating the deployment process had no errors or on anything that might be different between the test and production environments • An additional CD step can be to revert to the previous code version if the production tests fail
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY “Production Smoke Tests” Job • Click on “Production Smoke Tests” under the “CD” tab and then “Configure” • Source Code Management — GitHub repository with sample application code • Build — Create a Python virtual environment for testing — Install Python packages required for testing — Use a test runner to run the functional tests with the option of creating a report file with the tests results — Note: NO additional deployment to production • Post Build Action — Consume the created test results file and present it in a more readable fashion
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Continuous Deployment • Tags the build with a release version, deploys it to production, and then tests production • Tagging step is optional if the build is already tagged properly • Reverting back to a previous version if the smoke tests fail is recommended, but not included in this workshop example Tag Build with Release Version Deploy Build to Production Test Production
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY “Continuous Deployment Pipeline” Job • Very similar to the “Continuous Integration Pipeline” • Click the “Continuous Deployment Pipeline” job > “Configure” • Parameters — ORIGINAL_TAG and RELEASE_TAG • Pipeline — Specifies the jobs to run and passes them the needed parameters • Click “Build with Parameters” and refer to the “Build Release Artifact” again for the ORIGINAL_TAG. • If you go to any of the CD jobs, there will be #1 and #2 now in the build history
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Continuous Integration and Deployment • Combines both CI and CD in a single workflow • Fully automated release cycle • Other possible CI/CD steps: — Performance testing — Security testing — Deploying to additional environments, like development or staging — Reverting production if smoke tests fail — Sending emails on build completion or failure
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Continuous Integration and Deployment • Can have jobs run in parallel where applicable • Can run parts of the CI/CD pipeline when needed • Can trigger the CI/CD pipeline or parts of the pipeline automatically — Run unit and functional tests when there is a code change proposal (e.g., GitHub pull request) — Run the CI pipeline nightly so there is always a build with the latest code to be deployed the next day — Run the entire CI/CD pipeline when there is the master version of the code is updated so that production always has the latest version
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY “Continuous Integration and Deployment Pipeline” Job • Click on “Continuous Integration and Deployment Pipeline” under the “Complete CI CD” tab and then “Configure” • Parameters — RELEASE_TAG • Click “Build with Parameters,” input a release version, and click “Build” • Watch the entire CI/CD pipeline in action!
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Local Setup • Jenkins — Open source program for CI/CD automation • Docker — Program for deploying application inside of containers — Will run Jenkins and workshop demo environments • GitHub — Git source control hosting • Ngrok — Tool for tunneling to localhost — Used for local testing only
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Local Setup • Install Docker Toolbox for Windows/Mac — https://www.docker.com/products/docker-toolbox • Install Docker Engine and Docker Compose for Linux — https://docs.docker.com/engine/installation/ — https://docs.docker.com/compose/install/ • Download and unzip ngrok — https://ngrok.com/ • Navigate to GitHub repository — https://github.com/melissa-kam/pokemonGHC — Fork repository if you have a GitHub account
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Windows/Mac • Open the Docker CLI in Kitematic • Clone repository or download the docker-compose.yml file from GitHub • Navigate in the Docker CLI to where the docker-compose.yml is located and run: — docker-compose up
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Linux • Clone repository or download the docker-compose.yml file from GitHub • Navigate to where the docker-compose.yml is located and run: — docker-compose up • Jenkins: http://localhost:8080 • Test Environment: http://localhost:5050 • Production Environment: http://localhost:5000
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY GitHub Pull Request Integration • Jenkins can automatically trigger a job when a pull request is opened in GitHub and report the result of the job back to the pull request when it is finished. • In GitHub, go to Settings -> Personal access tokens • Click Generate new token • Give the token a description and click the checkbox for repo:status • Click Generate token and copy the created token
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY GitHub Pull Request Integration • In Jenkins, go to Manage Jenkins -> Configure System • Scroll down to GitHub Pull Request Builder • Click the Add button by Credentials • Select Secret text as Kind • Paste your access token into Secret and give it a name in the ID • Click Add and then select your created credential
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY GitHub Pull Request Webhooks • Run ngrok to connect your local Jenkins to the internet: — ./ngrok http <jenkins-ip>:8080 • Navigate to your forked repository on GitHub and go to Settings -> Webhooks • Copy your ngrok URL into the Payload URL and add /ghprbhook/ to the end — E.g., http://cd1e819f.ngrok.io/ghprbhook/ • Select Let me select individual events and check Pull request, Pull request review comment, and Issue comment. • In Jenkins, navigate to the Pull Request CI tab. For pull request jobs, under Build Triggers, check Use github hooks for build triggering • Now when a PR is opened, GitHub will alert Jenkins using the information specified in the webhook
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY GitHub Push Webhooks • Jenkins can automatically trigger jobs when code has been pushed to GitHub • Create a second webhook with /github-webhook/ at the end — E.g., http://cd1e819f.ngrok.io/github-webhook/ • Select Just the push event • In Jenkins, navigate to the Pull Request CI tab. For the Trigger Pipeline job, check Build when a change is pushed to GitHub.
BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Thank you Feedback? Download at http://bit.ly/ghc16app or search GHC 16 in the app store Rate and review the session on our mobile app
mkam
hiyanger
emiki
tsumita
oracle4engineer
isaoshimizu
eijikominami
lycorp_recruit_jp
cyberagentdevelopers
chou500
sudoakiy
miu_crescent
hatefulcrawdad
soudai
hannesfritz
paulrobertlloyd
iamctodd
jacobian
matthewcrist
trishagee
paigeccino
eileencodes
chrislema
eitanlees
