$30 off During Our Annual Pro Sale. View Details »

Building a Serverless Continuous Integration and Delivery Pipeline

Building a Serverless Continuous Integration and Delivery Pipeline

Continuous integration and delivery (CI/CD) systems are part and parcel of today’s software delivery pipelines. Today, there are two choices for a CI/CD system: you either pay for a service or host your own. However, most of the time, we end up waiting for resources to free up or paying for under-utilized resources.

Anubhav Mishra explains how to use serverless computing to create a cost-effective and reliable CI/CD pipeline. Anubhav offers an overview of serverless computing projects that focus on solving the CI/CD problem and shows you where to get started. Anubhav then walks you through creating a CI/CD pipeline for a modern containerized application from scratch using serverless platforms like Amazon Lambda and Google Cloud Functions. Along the way, Anubhav also shows how this CI/CD system can be integrated with the tools developers use daily like GitHub and Slack to make the software delivery experience even better, using philosophies like ChatOps.

O'Reilly OSCON page: https://conferences.oreilly.com/oscon/oscon-or/public/schedule/detail/67648

Anubhav Mishra

July 18, 2018
Tweet

More Decks by Anubhav Mishra

Other Decks in Technology

Transcript

  1. Building a Serverless CI/CD Pipeline
    Photo by Ricardo Gomez Angel on Unsplash
    @anubhavm

    View Slide

  2. Building a Serverless Continuous
    Integration and Delivery Pipeline
    Photo by Ricardo Gomez Angel on Unsplash
    @anubhavm

    View Slide

  3. @anubhavm
    Anubhav Mishra
    Developer Advocate, HashiCorp

    View Slide

  4. @anubhavm
    Anubhav Mishra
    Developer Advocate, HashiCorp
    Atlan&s

    View Slide

  5. @anubhavm
    Anubhav Mishra
    Developer Advocate, HashiCorp
    Atlan&s

    View Slide

  6. @anubhavm
    Anubhav Mishra
    Developer Advocate, HashiCorp
    Atlan&s
    Gopher Artwork by Ashley McNamara

    View Slide

  7. @anubhavm
    Anubhav Mishra
    Developer Advocate, HashiCorp
    Atlan&s
    Gopher Artwork by Ashley McNamara

    View Slide

  8. View Slide

  9. Building a CI/CD Pipeline for Serverless
    Applications
    Photo by Ricardo Gomez Angel on Unsplash
    @anubhavm

    View Slide

  10. Building a CI/CD Pipeline for Serverless
    Applications
    Photo by Ricardo Gomez Angel on Unsplash
    @anubhavm

    View Slide

  11. @anubhavm

    Agenda
    Software Delivery using CI/CD
    “Serverless” - The Term
    “Serverless” for CI/CD
    Live Demos!

    View Slide

  12. Software Delivery Pipelines
    Photo by Simon Rae on Unsplash

    View Slide

  13. @anubhavm

    Software Delivery Pipeline

    View Slide

  14. @anubhavm

    Software Delivery Pipeline
    Source/Code

    View Slide

  15. @anubhavm

    Software Delivery Pipeline
    Source/Code Build

    View Slide

  16. @anubhavm

    Software Delivery Pipeline
    Source/Code Build Test

    View Slide

  17. @anubhavm

    Software Delivery Pipeline
    Source/Code Build Test
    Deploy
    (Production)

    View Slide

  18. @anubhavm

    Software Delivery Pipeline
    Source/Code Build Test
    Deploy
    (Production)
    Continuous Integration (CI)

    View Slide

  19. @anubhavm

    Software Delivery Pipeline
    Source/Code Build Test
    Deploy
    (Production)
    Continuous Integration (CI)
    Continuous Delivery (CD)

    View Slide

  20. @anubhavm

    Software Delivery Pipeline
    Source/Code Build Test
    Deploy
    (Production)
    Continuous Integration (CI)
    Continuous Delivery (CD)

    View Slide

  21. @anubhavm

    Software Delivery Pipeline
    Source/Code Build Test
    Deploy
    (Production)
    Continuous Integration (CI)
    Continuous Delivery (CD)

    View Slide

  22. @anubhavm

    CI/CD
    Code stored in VCS (Git, SVN)
    Using automation systems (Jenkins, CircleCI,
    TravisCI)
    Unit tests and integration tests
    Deploy to dev, staging, and production
    Canary deployments
    $ go test

    View Slide

  23. @anubhavm

    CI/CD
    Build Test
    Deploy
    (Production)
    Developer
    $ git push
    Trigger Webhook

    View Slide

  24. @anubhavm

    CI/CD
    Build Test
    Deploy
    (Production)
    Developer
    $ git push
    Trigger Webhook
    Compute (Servers)

    View Slide

  25. @anubhavm

    CI/CD
    Build Test
    Deploy
    (Production)
    Developer
    $ git push
    Trigger Webhook
    Compute (Servers)

    View Slide

  26. @anubhavm

    CI/CD
    Developer
    $ git push
    Build Test
    Deploy
    (Production)
    Trigger Webhook
    Compute (Servers)

    View Slide

  27. @anubhavm

    CI/CD
    Developer
    $ git push
    Build Test
    Deploy
    (Production)
    Trigger Webhook

    View Slide

  28. @anubhavm

    CI/CD
    Developer
    $ git push
    Trigger Webhook

    View Slide

  29. @anubhavm

    CI/CD
    Developer
    $ git push
    Trigger Webhook

    View Slide

  30. @anubhavm

    CI/CD
    Developer
    $ git push
    Trigger Webhook

    View Slide

  31. @anubhavm

    CI/CD
    Developer
    $ git push
    Trigger Webhook
    Master

    View Slide

  32. @anubhavm

    CI/CD
    Developer
    $ git push
    Trigger Webhook
    Master
    Slaves Clients

    View Slide

  33. @anubhavm

    CI/CD
    Developer
    $ git push
    Trigger Webhook
    Master
    Slaves Clients

    View Slide

  34. @anubhavm

    CI/CD
    Developer
    $ git push
    Trigger Webhook
    Master
    Slaves Clients

    View Slide

  35. @anubhavm

    CI/CD
    Developer
    $ git push
    Trigger Webhook
    Master
    Slaves Clients

    View Slide

  36. @anubhavm

    CI/CD
    Developer
    $ git push
    Trigger Webhook
    Master
    Slaves Clients

    View Slide

  37. @anubhavm

    CI/CD
    Developer
    $ git push
    Trigger Webhook
    Master
    Slaves Clients

    View Slide

  38. @anubhavm

    CI/CD
    Developer
    $ git push
    Trigger Webhook
    Master
    Slaves Clients

    View Slide

  39. @anubhavm

    View Slide

  40. @anubhavm

    Drawbacks
    Cost of Infrastructure (on-prem or hosted)
    Cost of Idle Time
    Long Term Maintenance
    Reduced Agility
    Longer Delivery Times

    View Slide

  41. @anubhavm

    CI/CD
    Build Test
    Deploy
    (Production)
    Developer
    $ git push
    Trigger Webhook
    Compute (Servers)

    View Slide

  42. @anubhavm

    CI/CD
    Developer
    $ git push
    Trigger Webhook

    View Slide

  43. “Serverless” - The Term
    Photo by Yu Kato on Unsplash

    View Slide

  44. @anubhavm

    View Slide

  45. @anubhavm

    View Slide

  46. @anubhavm

    https://twitter.com/anubhavm/status/1016454479548506112

    View Slide

  47. @anubhavm

    “Serverless”
    Google Cloud defines Serverless
    “Serverless is a new paradigm of computing that abstracts away the complexity
    associated with managing servers for mobile and API backends, ETL, data
    processing jobs, databases, and more.”
    AWS defines Serverless
    “Serverless computing allows you to build and run applications and services without
    thinking about servers. Serverless applications don't require you to provision, scale,
    and manage any servers.”
    Microsoft (Azure) defines Serverless
    “Serverless computing is the abstraction of servers, infrastructure, and operating
    systems.”

    View Slide

  48. @anubhavm

    “Serverless”
    Google Cloud defines Serverless
    “Serverless is a new paradigm of computing that abstracts away the complexity
    associated with managing servers for mobile and API backends, ETL, data
    processing jobs, databases, and more.”
    AWS defines Serverless
    “Serverless computing allows you to build and run applications and services without
    thinking about servers. Serverless applications don't require you to provision, scale,
    and manage any servers.”
    Microsoft (Azure) defines Serverless
    “Serverless computing is the abstraction of servers, infrastructure, and operating
    systems.”

    View Slide

  49. @anubhavm

    View Slide

  50. @anubhavm

    “Serverless”
    “Serverless is a technology pattern that provides
    services and concepts to minimize operational overhead
    that comes with managing servers. It is a powerful
    abstraction when used can result in an increased focus
    on business value.” - Anubhav Mishra
    OSCON 2018, Portland

    View Slide

  51. @anubhavm

    “Serverless”
    “Serverless is a technology pattern that provides
    services and concepts to minimize operational overhead
    that comes with managing servers. It is a powerful
    abstraction when used can result in an increased focus
    on business value.” - Anubhav Mishra
    OSCON 2018, Portland
    FaaS = Functions as a Service
    BaaS = Backend as a Service

    View Slide

  52. @anubhavm

    View Slide

  53. @anubhavm

    View Slide

  54. @anubhavm

    View Slide

  55. @anubhavm

    View Slide

  56. @anubhavm

    Benefits / Advantages
    Reduced Operational/Infrastructure Cost
    Pay for What You Use
    Faster Development Cycles
    Increased Elasticity / Scaling

    View Slide

  57. @anubhavm

    Benefits / Advantages
    Reduced Operational/Infrastructure Cost
    Pay for What You Use
    Faster Development Cycles
    Increased Elasticity / Scaling

    View Slide

  58. @anubhavm

    “Serverless” CI/CD
    Developer
    Trigger Webhook
    $ git push

    View Slide

  59. @anubhavm

    “Serverless” CI/CD
    Developer
    Trigger Webhook
    $ git push

    View Slide

  60. @anubhavm

    “Serverless” CI/CD
    Developer
    Trigger Webhook
    $ git push

    View Slide

  61. @anubhavm

    “Serverless” CI/CD
    Developer
    Trigger Webhook
    $ git push

    View Slide

  62. @anubhavm

    Let’s build stuff

    View Slide

  63. @anubhavm

    Let’s build stuff

    View Slide

  64. @anubhavm

    Live Demos

    View Slide

  65. @anubhavm

    Live Demos

    View Slide

  66. @anubhavm

    Extending Jenkins using Serverless
    Build Test
    Deploy
    (Production)
    $ git push
    Trigger Webhook

    View Slide

  67. @anubhavm

    Extending Jenkins using Serverless
    $ git push
    Trigger Webhook

    View Slide

  68. @anubhavm

    Extending Jenkins using Serverless
    $ git push
    Trigger Webhook
    Master
    Slaves Clients

    View Slide

  69. @anubhavm

    No Nodes/Servers to Manage
    Launch 10,000+ Builds/Containers in Seconds
    No Cost for Idle Time
    Cost Summary (1 vCPU, 2GB memory):
    • With Jenkins on Fargate: 100 builds * 5 mins = $0.633/month
    • With Jenkins on EC2 Instances: ~ $50/month
    Advantages of Using CaaS for Jenkins

    View Slide

  70. @anubhavm

    Cold Starts - Slower Boot Times for Clients
    Large Container Images (~1G)
    No Root Access
    Ephemeral Storage (Default)
    Drawbacks of Using CaaS for Jenkins

    View Slide

  71. @anubhavm

    [1] Released ~2015-2016 (Blog Post)
    3000+ Github Stars
    Written in Node.js
    Supports AWS Lambda
    OSS “Serverless” CI/CD Projects
    LambCI
    Serverless continuous integration
    [1] Michael Hart’s blog post on LambCI - a serverless build system https://medium.com/@hichaelmart/lambci-4c3e29d6599b

    View Slide

  72. @anubhavm

    Continuous Integration using LambCI
    $ git push
    Trigger Webhook
    LambCI
    +

    View Slide

  73. @anubhavm

    1000+ Concurrent Builds
    No Maintenance of Servers
    No Cost for Idle Time
    Flexible Billing (100 Concurrent Builds of 10 secs vs 10 Individual 5 Minute
    Builds)
    Cost Summary (1GB Memory):
    • With AWS Lambda Free Tier: 1000 builds * 5 mins = $0.00/month
    • With AWS Lambda Paid: 1000 builds * 5 mins = $5.00/month
    Advantages of using FaaS for CI/CD

    View Slide

  74. @anubhavm

    Limited Runtime Support (Node.js, Python, Golang)
    Limited Support for Plugins
    No Root Access
    ~5 Minutes Maximum Build Time
    Drawbacks of Using FaaS for CI/CD

    View Slide

  75. @anubhavm

    https://twitter.com/kelseyhightower/status/902525728725405697?lang=en

    View Slide

  76. The Future
    Photo by Pierre Châtel-Innocenti on Unsplash

    View Slide

  77. www.hashicorp.com
    FOR EVERYONE, EVERYWHERE
    Thank You!
    I have stickers!
    Ask me anything. @anubhavm
    Anubhav Mishra

    View Slide