Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

Building a Serverless Continuous Integration an...

Building a Serverless Continuous Integration and Delivery Pipeline

Continuous integration and delivery (CI/CD) systems are part and parcel of today’s software delivery pipelines. Today, there are two choices for a CI/CD system: you either pay for a service or host your own. However, most of the time, we end up waiting for resources to free up or paying for under-utilized resources.

Anubhav Mishra explains how to use serverless computing to create a cost-effective and reliable CI/CD pipeline. Anubhav offers an overview of serverless computing projects that focus on solving the CI/CD problem and shows you where to get started. Anubhav then walks you through creating a CI/CD pipeline for a modern containerized application from scratch using serverless platforms like Amazon Lambda and Google Cloud Functions. Along the way, Anubhav also shows how this CI/CD system can be integrated with the tools developers use daily like GitHub and Slack to make the software delivery experience even better, using philosophies like ChatOps.

O'Reilly OSCON page: https://conferences.oreilly.com/oscon/oscon-or/public/schedule/detail/67648

Anubhav Mishra

July 18, 2018
Tweet

More Decks by Anubhav Mishra

Other Decks in Technology

Transcript

  1. @anubhavm  CI/CD Code stored in VCS (Git, SVN) Using

    automation systems (Jenkins, CircleCI, TravisCI) Unit tests and integration tests Deploy to dev, staging, and production Canary deployments $ go test
  2. @anubhavm  CI/CD Developer $ git push Build Test Deploy

    (Production) Trigger Webhook Compute (Servers)
  3. @anubhavm  Drawbacks Cost of Infrastructure (on-prem or hosted) Cost

    of Idle Time Long Term Maintenance Reduced Agility Longer Delivery Times
  4. @anubhavm  “Serverless” Google Cloud defines Serverless “Serverless is a

    new paradigm of computing that abstracts away the complexity associated with managing servers for mobile and API backends, ETL, data processing jobs, databases, and more.” AWS defines Serverless “Serverless computing allows you to build and run applications and services without thinking about servers. Serverless applications don't require you to provision, scale, and manage any servers.” Microsoft (Azure) defines Serverless “Serverless computing is the abstraction of servers, infrastructure, and operating systems.”
  5. @anubhavm  “Serverless” Google Cloud defines Serverless “Serverless is a

    new paradigm of computing that abstracts away the complexity associated with managing servers for mobile and API backends, ETL, data processing jobs, databases, and more.” AWS defines Serverless “Serverless computing allows you to build and run applications and services without thinking about servers. Serverless applications don't require you to provision, scale, and manage any servers.” Microsoft (Azure) defines Serverless “Serverless computing is the abstraction of servers, infrastructure, and operating systems.”
  6. @anubhavm  “Serverless” “Serverless is a technology pattern that provides

    services and concepts to minimize operational overhead that comes with managing servers. It is a powerful abstraction when used can result in an increased focus on business value.” - Anubhav Mishra OSCON 2018, Portland
  7. @anubhavm  “Serverless” “Serverless is a technology pattern that provides

    services and concepts to minimize operational overhead that comes with managing servers. It is a powerful abstraction when used can result in an increased focus on business value.” - Anubhav Mishra OSCON 2018, Portland FaaS = Functions as a Service BaaS = Backend as a Service
  8. @anubhavm  Benefits / Advantages Reduced Operational/Infrastructure Cost Pay for

    What You Use Faster Development Cycles Increased Elasticity / Scaling
  9. @anubhavm  Benefits / Advantages Reduced Operational/Infrastructure Cost Pay for

    What You Use Faster Development Cycles Increased Elasticity / Scaling
  10. @anubhavm  No Nodes/Servers to Manage Launch 10,000+ Builds/Containers in

    Seconds No Cost for Idle Time Cost Summary (1 vCPU, 2GB memory): • With Jenkins on Fargate: 100 builds * 5 mins = $0.633/month • With Jenkins on EC2 Instances: ~ $50/month Advantages of Using CaaS for Jenkins
  11. @anubhavm  Cold Starts - Slower Boot Times for Clients

    Large Container Images (~1G) No Root Access Ephemeral Storage (Default) Drawbacks of Using CaaS for Jenkins
  12. @anubhavm  [1] Released ~2015-2016 (Blog Post) 3000+ Github Stars

    Written in Node.js Supports AWS Lambda OSS “Serverless” CI/CD Projects LambCI Serverless continuous integration [1] Michael Hart’s blog post on LambCI - a serverless build system https://medium.com/@hichaelmart/lambci-4c3e29d6599b
  13. @anubhavm  1000+ Concurrent Builds No Maintenance of Servers No

    Cost for Idle Time Flexible Billing (100 Concurrent Builds of 10 secs vs 10 Individual 5 Minute Builds) Cost Summary (1GB Memory): • With AWS Lambda Free Tier: 1000 builds * 5 mins = $0.00/month • With AWS Lambda Paid: 1000 builds * 5 mins = $5.00/month Advantages of using FaaS for CI/CD
  14. @anubhavm  Limited Runtime Support (Node.js, Python, Golang) Limited Support

    for Plugins No Root Access ~5 Minutes Maximum Build Time Drawbacks of Using FaaS for CI/CD