Building a Serverless Continuous Integration and Delivery Pipeline

Continuous integration and delivery (CI/CD) systems are part and parcel of today’s software delivery pipelines. Today, there are two choices for a CI/CD system: you either pay for a service or host your own. However, most of the time, we end up waiting for resources to free up or paying for under-utilized resources.

Anubhav Mishra explains how to use serverless computing to create a cost-effective and reliable CI/CD pipeline. Anubhav offers an overview of serverless computing projects that focus on solving the CI/CD problem and shows you where to get started. Anubhav then walks you through creating a CI/CD pipeline for a modern containerized application from scratch using serverless platforms like Amazon Lambda and Google Cloud Functions. Along the way, Anubhav also shows how this CI/CD system can be integrated with the tools developers use daily like GitHub and Slack to make the software delivery experience even better, using philosophies like ChatOps.

O'Reilly OSCON page: https://conferences.oreilly.com/oscon/oscon-or/public/schedule/detail/67648

Transcript

1. Building a Serverless CI/CD Pipeline
   Photo by Ricardo Gomez Angel on Unsplash
   @anubhavm
   
   

   View Slide

2. Building a Serverless Continuous
   Integration and Delivery Pipeline
   Photo by Ricardo Gomez Angel on Unsplash
   @anubhavm
   
   

   View Slide

3. @anubhavm
   Anubhav Mishra
   Developer Advocate, HashiCorp
   

   View Slide

4. @anubhavm
   Anubhav Mishra
   Developer Advocate, HashiCorp
   Atlan&s
   

   View Slide

5. @anubhavm
   Anubhav Mishra
   Developer Advocate, HashiCorp
   Atlan&s
   

   View Slide

6. @anubhavm
   Anubhav Mishra
   Developer Advocate, HashiCorp
   Atlan&s
   Gopher Artwork by Ashley McNamara
   

   View Slide

7. @anubhavm
   Anubhav Mishra
   Developer Advocate, HashiCorp
   Atlan&s
   Gopher Artwork by Ashley McNamara
   

   View Slide

8. View Slide

9. Building a CI/CD Pipeline for Serverless
   Applications
   Photo by Ricardo Gomez Angel on Unsplash
   @anubhavm
   
   

   View Slide

10. Building a CI/CD Pipeline for Serverless
    Applications
    Photo by Ricardo Gomez Angel on Unsplash
    @anubhavm
    
    

    View Slide

11. @anubhavm
    
    Agenda
    Software Delivery using CI/CD
    “Serverless” - The Term
    “Serverless” for CI/CD
    Live Demos!
    

    View Slide

12. Software Delivery Pipelines
    Photo by Simon Rae on Unsplash
    

    View Slide

13. @anubhavm
    
    Software Delivery Pipeline
    

    View Slide

14. @anubhavm
    
    Software Delivery Pipeline
    Source/Code
    

    View Slide

15. @anubhavm
    
    Software Delivery Pipeline
    Source/Code Build
    

    View Slide

16. @anubhavm
    
    Software Delivery Pipeline
    Source/Code Build Test
    

    View Slide

17. @anubhavm
    
    Software Delivery Pipeline
    Source/Code Build Test
    Deploy
    (Production)
    

    View Slide

18. @anubhavm
    
    Software Delivery Pipeline
    Source/Code Build Test
    Deploy
    (Production)
    Continuous Integration (CI)
    

    View Slide

19. @anubhavm
    
    Software Delivery Pipeline
    Source/Code Build Test
    Deploy
    (Production)
    Continuous Integration (CI)
    Continuous Delivery (CD)
    

    View Slide

20. @anubhavm
    
    Software Delivery Pipeline
    Source/Code Build Test
    Deploy
    (Production)
    Continuous Integration (CI)
    Continuous Delivery (CD)
    

    View Slide

21. @anubhavm
    
    Software Delivery Pipeline
    Source/Code Build Test
    Deploy
    (Production)
    Continuous Integration (CI)
    Continuous Delivery (CD)
    

    View Slide

22. @anubhavm
    
    CI/CD
    Code stored in VCS (Git, SVN)
    Using automation systems (Jenkins, CircleCI,
    TravisCI)
    Unit tests and integration tests
    Deploy to dev, staging, and production
    Canary deployments
    $ go test
    

    View Slide

23. @anubhavm
    
    CI/CD
    Build Test
    Deploy
    (Production)
    Developer
    $ git push
    Trigger Webhook
    

    View Slide

24. @anubhavm
    
    CI/CD
    Build Test
    Deploy
    (Production)
    Developer
    $ git push
    Trigger Webhook
    Compute (Servers)
    

    View Slide

25. @anubhavm
    
    CI/CD
    Build Test
    Deploy
    (Production)
    Developer
    $ git push
    Trigger Webhook
    Compute (Servers)
    

    View Slide

26. @anubhavm
    
    CI/CD
    Developer
    $ git push
    Build Test
    Deploy
    (Production)
    Trigger Webhook
    Compute (Servers)
    

    View Slide

27. @anubhavm
    
    CI/CD
    Developer
    $ git push
    Build Test
    Deploy
    (Production)
    Trigger Webhook
    

    View Slide

28. @anubhavm
    
    CI/CD
    Developer
    $ git push
    Trigger Webhook
    

    View Slide

29. @anubhavm
    
    CI/CD
    Developer
    $ git push
    Trigger Webhook
    

    View Slide

30. @anubhavm
    
    CI/CD
    Developer
    $ git push
    Trigger Webhook
    

    View Slide

31. @anubhavm
    
    CI/CD
    Developer
    $ git push
    Trigger Webhook
    Master
    

    View Slide

32. @anubhavm
    
    CI/CD
    Developer
    $ git push
    Trigger Webhook
    Master
    Slaves Clients
    

    View Slide

33. @anubhavm
    
    CI/CD
    Developer
    $ git push
    Trigger Webhook
    Master
    Slaves Clients
    

    View Slide

34. @anubhavm
    
    CI/CD
    Developer
    $ git push
    Trigger Webhook
    Master
    Slaves Clients
    

    View Slide

35. @anubhavm
    
    CI/CD
    Developer
    $ git push
    Trigger Webhook
    Master
    Slaves Clients
    

    View Slide

36. @anubhavm
    
    CI/CD
    Developer
    $ git push
    Trigger Webhook
    Master
    Slaves Clients
    

    View Slide

37. @anubhavm
    
    CI/CD
    Developer
    $ git push
    Trigger Webhook
    Master
    Slaves Clients
    

    View Slide

38. @anubhavm
    
    CI/CD
    Developer
    $ git push
    Trigger Webhook
    Master
    Slaves Clients
    

    View Slide

39. @anubhavm
    
    

    View Slide

40. @anubhavm
    
    Drawbacks
    Cost of Infrastructure (on-prem or hosted)
    Cost of Idle Time
    Long Term Maintenance
    Reduced Agility
    Longer Delivery Times
    

    View Slide

41. @anubhavm
    
    CI/CD
    Build Test
    Deploy
    (Production)
    Developer
    $ git push
    Trigger Webhook
    Compute (Servers)
    

    View Slide

42. @anubhavm
    
    CI/CD
    Developer
    $ git push
    Trigger Webhook
    

    View Slide

43. “Serverless” - The Term
    Photo by Yu Kato on Unsplash
    

    View Slide

44. @anubhavm
    
    

    View Slide

45. @anubhavm
    
    

    View Slide

46. @anubhavm
    
    https://twitter.com/anubhavm/status/1016454479548506112
    

    View Slide

47. @anubhavm
    
    “Serverless”
    Google Cloud defines Serverless
    “Serverless is a new paradigm of computing that abstracts away the complexity
    associated with managing servers for mobile and API backends, ETL, data
    processing jobs, databases, and more.”
    AWS defines Serverless
    “Serverless computing allows you to build and run applications and services without
    thinking about servers. Serverless applications don't require you to provision, scale,
    and manage any servers.”
    Microsoft (Azure) defines Serverless
    “Serverless computing is the abstraction of servers, infrastructure, and operating
    systems.”
    

    View Slide

48. @anubhavm
    
    “Serverless”
    Google Cloud defines Serverless
    “Serverless is a new paradigm of computing that abstracts away the complexity
    associated with managing servers for mobile and API backends, ETL, data
    processing jobs, databases, and more.”
    AWS defines Serverless
    “Serverless computing allows you to build and run applications and services without
    thinking about servers. Serverless applications don't require you to provision, scale,
    and manage any servers.”
    Microsoft (Azure) defines Serverless
    “Serverless computing is the abstraction of servers, infrastructure, and operating
    systems.”
    

    View Slide

49. @anubhavm
    
    

    View Slide

50. @anubhavm
    
    “Serverless”
    “Serverless is a technology pattern that provides
    services and concepts to minimize operational overhead
    that comes with managing servers. It is a powerful
    abstraction when used can result in an increased focus
    on business value.” - Anubhav Mishra
    OSCON 2018, Portland
    

    View Slide

51. @anubhavm
    
    “Serverless”
    “Serverless is a technology pattern that provides
    services and concepts to minimize operational overhead
    that comes with managing servers. It is a powerful
    abstraction when used can result in an increased focus
    on business value.” - Anubhav Mishra
    OSCON 2018, Portland
    FaaS = Functions as a Service
    BaaS = Backend as a Service
    

    View Slide

52. @anubhavm
    
    

    View Slide

53. @anubhavm
    
    

    View Slide

54. @anubhavm
    
    

    View Slide

55. @anubhavm
    
    

    View Slide

56. @anubhavm
    
    Benefits / Advantages
    Reduced Operational/Infrastructure Cost
    Pay for What You Use
    Faster Development Cycles
    Increased Elasticity / Scaling
    

    View Slide

57. @anubhavm
    
    Benefits / Advantages
    Reduced Operational/Infrastructure Cost
    Pay for What You Use
    Faster Development Cycles
    Increased Elasticity / Scaling
    

    View Slide

58. @anubhavm
    
    “Serverless” CI/CD
    Developer
    Trigger Webhook
    $ git push
    

    View Slide

59. @anubhavm
    
    “Serverless” CI/CD
    Developer
    Trigger Webhook
    $ git push
    

    View Slide

60. @anubhavm
    
    “Serverless” CI/CD
    Developer
    Trigger Webhook
    $ git push
    

    View Slide

61. @anubhavm
    
    “Serverless” CI/CD
    Developer
    Trigger Webhook
    $ git push
    

    View Slide

62. @anubhavm
    
    Let’s build stuff
    

    View Slide

63. @anubhavm
    
    Let’s build stuff
    

    View Slide

64. @anubhavm
    
    Live Demos
    

    View Slide

65. @anubhavm
    
    Live Demos
    

    View Slide

66. @anubhavm
    
    Extending Jenkins using Serverless
    Build Test
    Deploy
    (Production)
    $ git push
    Trigger Webhook
    

    View Slide

67. @anubhavm
    
    Extending Jenkins using Serverless
    $ git push
    Trigger Webhook
    

    View Slide

68. @anubhavm
    
    Extending Jenkins using Serverless
    $ git push
    Trigger Webhook
    Master
    Slaves Clients
    

    View Slide

69. @anubhavm
    
    No Nodes/Servers to Manage
    Launch 10,000+ Builds/Containers in Seconds
    No Cost for Idle Time
    Cost Summary (1 vCPU, 2GB memory):
    • With Jenkins on Fargate: 100 builds * 5 mins = $0.633/month
    • With Jenkins on EC2 Instances: ~ $50/month
    Advantages of Using CaaS for Jenkins
    

    View Slide

70. @anubhavm
    
    Cold Starts - Slower Boot Times for Clients
    Large Container Images (~1G)
    No Root Access
    Ephemeral Storage (Default)
    Drawbacks of Using CaaS for Jenkins
    

    View Slide

71. @anubhavm
    
    [1] Released ~2015-2016 (Blog Post)
    3000+ Github Stars
    Written in Node.js
    Supports AWS Lambda
    OSS “Serverless” CI/CD Projects
    LambCI
    Serverless continuous integration
    [1] Michael Hart’s blog post on LambCI - a serverless build system https://medium.com/@hichaelmart/lambci-4c3e29d6599b
    

    View Slide

72. @anubhavm
    
    Continuous Integration using LambCI
    $ git push
    Trigger Webhook
    LambCI
    +
    

    View Slide

73. @anubhavm
    
    1000+ Concurrent Builds
    No Maintenance of Servers
    No Cost for Idle Time
    Flexible Billing (100 Concurrent Builds of 10 secs vs 10 Individual 5 Minute
    Builds)
    Cost Summary (1GB Memory):
    • With AWS Lambda Free Tier: 1000 builds * 5 mins = $0.00/month
    • With AWS Lambda Paid: 1000 builds * 5 mins = $5.00/month
    Advantages of using FaaS for CI/CD
    

    View Slide

74. @anubhavm
    
    Limited Runtime Support (Node.js, Python, Golang)
    Limited Support for Plugins
    No Root Access
    ~5 Minutes Maximum Build Time
    Drawbacks of Using FaaS for CI/CD
    

    View Slide

75. @anubhavm
    
    https://twitter.com/kelseyhightower/status/902525728725405697?lang=en
    

    View Slide

76. The Future
    Photo by Pierre Châtel-Innocenti on Unsplash
    

    View Slide

77. www.hashicorp.com
    FOR EVERYONE, EVERYWHERE
    Thank You!
    I have stickers!
    Ask me anything. @anubhavm
    Anubhav Mishra
    

    View Slide