Building a Serverless Continuous Integration and Delivery Pipeline

Building a Serverless Continuous Integration and Delivery Pipeline

Continuous integration and delivery (CI/CD) systems are part and parcel of today’s software delivery pipelines. Today, there are two choices for a CI/CD system: you either pay for a service or host your own. However, most of the time, we end up waiting for resources to free up or paying for under-utilized resources.

Anubhav Mishra explains how to use serverless computing to create a cost-effective and reliable CI/CD pipeline. Anubhav offers an overview of serverless computing projects that focus on solving the CI/CD problem and shows you where to get started. Anubhav then walks you through creating a CI/CD pipeline for a modern containerized application from scratch using serverless platforms like Amazon Lambda and Google Cloud Functions. Along the way, Anubhav also shows how this CI/CD system can be integrated with the tools developers use daily like GitHub and Slack to make the software delivery experience even better, using philosophies like ChatOps.

O'Reilly OSCON page: https://conferences.oreilly.com/oscon/oscon-or/public/schedule/detail/67648

26896287bc831a13e768cea7efe29632?s=128

Anubhav Mishra

July 18, 2018
Tweet

Transcript

  1. Building a Serverless CI/CD Pipeline Photo by Ricardo Gomez Angel

    on Unsplash @anubhavm 
  2. Building a Serverless Continuous Integration and Delivery Pipeline Photo by

    Ricardo Gomez Angel on Unsplash @anubhavm 
  3. @anubhavm Anubhav Mishra Developer Advocate, HashiCorp

  4. @anubhavm Anubhav Mishra Developer Advocate, HashiCorp Atlan&s

  5. @anubhavm Anubhav Mishra Developer Advocate, HashiCorp Atlan&s

  6. @anubhavm Anubhav Mishra Developer Advocate, HashiCorp Atlan&s Gopher Artwork by

    Ashley McNamara
  7. @anubhavm Anubhav Mishra Developer Advocate, HashiCorp Atlan&s Gopher Artwork by

    Ashley McNamara
  8. None
  9. Building a CI/CD Pipeline for Serverless Applications Photo by Ricardo

    Gomez Angel on Unsplash @anubhavm 
  10. Building a CI/CD Pipeline for Serverless Applications Photo by Ricardo

    Gomez Angel on Unsplash @anubhavm 
  11. @anubhavm  Agenda Software Delivery using CI/CD “Serverless” - The

    Term “Serverless” for CI/CD Live Demos!
  12. Software Delivery Pipelines Photo by Simon Rae on Unsplash

  13. @anubhavm  Software Delivery Pipeline

  14. @anubhavm  Software Delivery Pipeline Source/Code

  15. @anubhavm  Software Delivery Pipeline Source/Code Build

  16. @anubhavm  Software Delivery Pipeline Source/Code Build Test

  17. @anubhavm  Software Delivery Pipeline Source/Code Build Test Deploy (Production)

  18. @anubhavm  Software Delivery Pipeline Source/Code Build Test Deploy (Production)

    Continuous Integration (CI)
  19. @anubhavm  Software Delivery Pipeline Source/Code Build Test Deploy (Production)

    Continuous Integration (CI) Continuous Delivery (CD)
  20. @anubhavm  Software Delivery Pipeline Source/Code Build Test Deploy (Production)

    Continuous Integration (CI) Continuous Delivery (CD)
  21. @anubhavm  Software Delivery Pipeline Source/Code Build Test Deploy (Production)

    Continuous Integration (CI) Continuous Delivery (CD)
  22. @anubhavm  CI/CD Code stored in VCS (Git, SVN) Using

    automation systems (Jenkins, CircleCI, TravisCI) Unit tests and integration tests Deploy to dev, staging, and production Canary deployments $ go test
  23. @anubhavm  CI/CD Build Test Deploy (Production) Developer $ git

    push Trigger Webhook
  24. @anubhavm  CI/CD Build Test Deploy (Production) Developer $ git

    push Trigger Webhook Compute (Servers)
  25. @anubhavm  CI/CD Build Test Deploy (Production) Developer $ git

    push Trigger Webhook Compute (Servers)
  26. @anubhavm  CI/CD Developer $ git push Build Test Deploy

    (Production) Trigger Webhook Compute (Servers)
  27. @anubhavm  CI/CD Developer $ git push Build Test Deploy

    (Production) Trigger Webhook
  28. @anubhavm  CI/CD Developer $ git push Trigger Webhook

  29. @anubhavm  CI/CD Developer $ git push Trigger Webhook

  30. @anubhavm  CI/CD Developer $ git push Trigger Webhook

  31. @anubhavm  CI/CD Developer $ git push Trigger Webhook Master

  32. @anubhavm  CI/CD Developer $ git push Trigger Webhook Master

    Slaves Clients
  33. @anubhavm  CI/CD Developer $ git push Trigger Webhook Master

    Slaves Clients
  34. @anubhavm  CI/CD Developer $ git push Trigger Webhook Master

    Slaves Clients
  35. @anubhavm  CI/CD Developer $ git push Trigger Webhook Master

    Slaves Clients
  36. @anubhavm  CI/CD Developer $ git push Trigger Webhook Master

    Slaves Clients
  37. @anubhavm  CI/CD Developer $ git push Trigger Webhook Master

    Slaves Clients
  38. @anubhavm  CI/CD Developer $ git push Trigger Webhook Master

    Slaves Clients
  39. @anubhavm 

  40. @anubhavm  Drawbacks Cost of Infrastructure (on-prem or hosted) Cost

    of Idle Time Long Term Maintenance Reduced Agility Longer Delivery Times
  41. @anubhavm  CI/CD Build Test Deploy (Production) Developer $ git

    push Trigger Webhook Compute (Servers)
  42. @anubhavm  CI/CD Developer $ git push Trigger Webhook

  43. “Serverless” - The Term Photo by Yu Kato on Unsplash

  44. @anubhavm 

  45. @anubhavm 

  46. @anubhavm  https://twitter.com/anubhavm/status/1016454479548506112

  47. @anubhavm  “Serverless” Google Cloud defines Serverless “Serverless is a

    new paradigm of computing that abstracts away the complexity associated with managing servers for mobile and API backends, ETL, data processing jobs, databases, and more.” AWS defines Serverless “Serverless computing allows you to build and run applications and services without thinking about servers. Serverless applications don't require you to provision, scale, and manage any servers.” Microsoft (Azure) defines Serverless “Serverless computing is the abstraction of servers, infrastructure, and operating systems.”
  48. @anubhavm  “Serverless” Google Cloud defines Serverless “Serverless is a

    new paradigm of computing that abstracts away the complexity associated with managing servers for mobile and API backends, ETL, data processing jobs, databases, and more.” AWS defines Serverless “Serverless computing allows you to build and run applications and services without thinking about servers. Serverless applications don't require you to provision, scale, and manage any servers.” Microsoft (Azure) defines Serverless “Serverless computing is the abstraction of servers, infrastructure, and operating systems.”
  49. @anubhavm 

  50. @anubhavm  “Serverless” “Serverless is a technology pattern that provides

    services and concepts to minimize operational overhead that comes with managing servers. It is a powerful abstraction when used can result in an increased focus on business value.” - Anubhav Mishra OSCON 2018, Portland
  51. @anubhavm  “Serverless” “Serverless is a technology pattern that provides

    services and concepts to minimize operational overhead that comes with managing servers. It is a powerful abstraction when used can result in an increased focus on business value.” - Anubhav Mishra OSCON 2018, Portland FaaS = Functions as a Service BaaS = Backend as a Service
  52. @anubhavm 

  53. @anubhavm 

  54. @anubhavm 

  55. @anubhavm 

  56. @anubhavm  Benefits / Advantages Reduced Operational/Infrastructure Cost Pay for

    What You Use Faster Development Cycles Increased Elasticity / Scaling
  57. @anubhavm  Benefits / Advantages Reduced Operational/Infrastructure Cost Pay for

    What You Use Faster Development Cycles Increased Elasticity / Scaling
  58. @anubhavm  “Serverless” CI/CD Developer Trigger Webhook $ git push

  59. @anubhavm  “Serverless” CI/CD Developer Trigger Webhook $ git push

  60. @anubhavm  “Serverless” CI/CD Developer Trigger Webhook $ git push

  61. @anubhavm  “Serverless” CI/CD Developer Trigger Webhook $ git push

  62. @anubhavm  Let’s build stuff

  63. @anubhavm  Let’s build stuff

  64. @anubhavm  Live Demos

  65. @anubhavm  Live Demos

  66. @anubhavm  Extending Jenkins using Serverless Build Test Deploy (Production)

    $ git push Trigger Webhook
  67. @anubhavm  Extending Jenkins using Serverless $ git push Trigger

    Webhook
  68. @anubhavm  Extending Jenkins using Serverless $ git push Trigger

    Webhook Master Slaves Clients
  69. @anubhavm  No Nodes/Servers to Manage Launch 10,000+ Builds/Containers in

    Seconds No Cost for Idle Time Cost Summary (1 vCPU, 2GB memory): • With Jenkins on Fargate: 100 builds * 5 mins = $0.633/month • With Jenkins on EC2 Instances: ~ $50/month Advantages of Using CaaS for Jenkins
  70. @anubhavm  Cold Starts - Slower Boot Times for Clients

    Large Container Images (~1G) No Root Access Ephemeral Storage (Default) Drawbacks of Using CaaS for Jenkins
  71. @anubhavm  [1] Released ~2015-2016 (Blog Post) 3000+ Github Stars

    Written in Node.js Supports AWS Lambda OSS “Serverless” CI/CD Projects LambCI Serverless continuous integration [1] Michael Hart’s blog post on LambCI - a serverless build system https://medium.com/@hichaelmart/lambci-4c3e29d6599b
  72. @anubhavm  Continuous Integration using LambCI $ git push Trigger

    Webhook LambCI +
  73. @anubhavm  1000+ Concurrent Builds No Maintenance of Servers No

    Cost for Idle Time Flexible Billing (100 Concurrent Builds of 10 secs vs 10 Individual 5 Minute Builds) Cost Summary (1GB Memory): • With AWS Lambda Free Tier: 1000 builds * 5 mins = $0.00/month • With AWS Lambda Paid: 1000 builds * 5 mins = $5.00/month Advantages of using FaaS for CI/CD
  74. @anubhavm  Limited Runtime Support (Node.js, Python, Golang) Limited Support

    for Plugins No Root Access ~5 Minutes Maximum Build Time Drawbacks of Using FaaS for CI/CD
  75. @anubhavm  https://twitter.com/kelseyhightower/status/902525728725405697?lang=en

  76. The Future Photo by Pierre Châtel-Innocenti on Unsplash

  77. www.hashicorp.com FOR EVERYONE, EVERYWHERE Thank You! I have stickers! Ask

    me anything. @anubhavm Anubhav Mishra