Building a Serverless Continuous Integration and Delivery Pipeline

Continuous integration and delivery (CI/CD) systems are part and parcel of today’s software delivery pipelines. Today, there are two choices for a CI/CD system: you either pay for a service or host your own. However, most of the time, we end up waiting for resources to free up or paying for under-utilized resources.

Anubhav Mishra explains how to use serverless computing to create a cost-effective and reliable CI/CD pipeline. Anubhav offers an overview of serverless computing projects that focus on solving the CI/CD problem and shows you where to get started. Anubhav then walks you through creating a CI/CD pipeline for a modern containerized application from scratch using serverless platforms like Amazon Lambda and Google Cloud Functions. Along the way, Anubhav also shows how this CI/CD system can be integrated with the tools developers use daily like GitHub and Slack to make the software delivery experience even better, using philosophies like ChatOps.

O'Reilly OSCON page: https://conferences.oreilly.com/oscon/oscon-or/public/schedule/detail/67648

Transcript

1. Building a Serverless CI/CD Pipeline Photo by Ricardo Gomez Angel on Unsplash @anubhavm 
2. Building a Serverless Continuous Integration and Delivery Pipeline Photo by Ricardo Gomez Angel on Unsplash @anubhavm 
3. @anubhavm Anubhav Mishra Developer Advocate, HashiCorp
4. @anubhavm Anubhav Mishra Developer Advocate, HashiCorp Atlan&s
5. @anubhavm Anubhav Mishra Developer Advocate, HashiCorp Atlan&s
6. @anubhavm Anubhav Mishra Developer Advocate, HashiCorp Atlan&s Gopher Artwork by Ashley McNamara
7. @anubhavm Anubhav Mishra Developer Advocate, HashiCorp Atlan&s Gopher Artwork by Ashley McNamara
9. Building a CI/CD Pipeline for Serverless Applications Photo by Ricardo Gomez Angel on Unsplash @anubhavm 
10. Building a CI/CD Pipeline for Serverless Applications Photo by Ricardo Gomez Angel on Unsplash @anubhavm 
11. @anubhavm  Agenda Software Delivery using CI/CD “Serverless” - The Term “Serverless” for CI/CD Live Demos!
12. Software Delivery Pipelines Photo by Simon Rae on Unsplash
13. @anubhavm  Software Delivery Pipeline
14. @anubhavm  Software Delivery Pipeline Source/Code
15. @anubhavm  Software Delivery Pipeline Source/Code Build
16. @anubhavm  Software Delivery Pipeline Source/Code Build Test
17. @anubhavm  Software Delivery Pipeline Source/Code Build Test Deploy (Production)
18. @anubhavm  Software Delivery Pipeline Source/Code Build Test Deploy (Production) Continuous Integration (CI)
19. @anubhavm  Software Delivery Pipeline Source/Code Build Test Deploy (Production) Continuous Integration (CI) Continuous Delivery (CD)
20. @anubhavm  Software Delivery Pipeline Source/Code Build Test Deploy (Production) Continuous Integration (CI) Continuous Delivery (CD)
21. @anubhavm  Software Delivery Pipeline Source/Code Build Test Deploy (Production) Continuous Integration (CI) Continuous Delivery (CD)
22. @anubhavm  CI/CD Code stored in VCS (Git, SVN) Using automation systems (Jenkins, CircleCI, TravisCI) Unit tests and integration tests Deploy to dev, staging, and production Canary deployments $ go test
23. @anubhavm  CI/CD Build Test Deploy (Production) Developer $ git push Trigger Webhook
24. @anubhavm  CI/CD Build Test Deploy (Production) Developer $ git push Trigger Webhook Compute (Servers)
25. @anubhavm  CI/CD Build Test Deploy (Production) Developer $ git push Trigger Webhook Compute (Servers)
26. @anubhavm  CI/CD Developer $ git push Build Test Deploy (Production) Trigger Webhook Compute (Servers)
27. @anubhavm  CI/CD Developer $ git push Build Test Deploy (Production) Trigger Webhook
28. @anubhavm  CI/CD Developer $ git push Trigger Webhook
29. @anubhavm  CI/CD Developer $ git push Trigger Webhook
30. @anubhavm  CI/CD Developer $ git push Trigger Webhook
31. @anubhavm  CI/CD Developer $ git push Trigger Webhook Master
32. @anubhavm  CI/CD Developer $ git push Trigger Webhook Master Slaves Clients
33. @anubhavm  CI/CD Developer $ git push Trigger Webhook Master Slaves Clients
34. @anubhavm  CI/CD Developer $ git push Trigger Webhook Master Slaves Clients
35. @anubhavm  CI/CD Developer $ git push Trigger Webhook Master Slaves Clients
36. @anubhavm  CI/CD Developer $ git push Trigger Webhook Master Slaves Clients
37. @anubhavm  CI/CD Developer $ git push Trigger Webhook Master Slaves Clients
38. @anubhavm  CI/CD Developer $ git push Trigger Webhook Master Slaves Clients
39. @anubhavm 
40. @anubhavm  Drawbacks Cost of Infrastructure (on-prem or hosted) Cost of Idle Time Long Term Maintenance Reduced Agility Longer Delivery Times
41. @anubhavm  CI/CD Build Test Deploy (Production) Developer $ git push Trigger Webhook Compute (Servers)
42. @anubhavm  CI/CD Developer $ git push Trigger Webhook
43. “Serverless” - The Term Photo by Yu Kato on Unsplash
44. @anubhavm 
45. @anubhavm 
46. @anubhavm  https://twitter.com/anubhavm/status/1016454479548506112
47. @anubhavm  “Serverless” Google Cloud defines Serverless “Serverless is a new paradigm of computing that abstracts away the complexity associated with managing servers for mobile and API backends, ETL, data processing jobs, databases, and more.” AWS defines Serverless “Serverless computing allows you to build and run applications and services without thinking about servers. Serverless applications don't require you to provision, scale, and manage any servers.” Microsoft (Azure) defines Serverless “Serverless computing is the abstraction of servers, infrastructure, and operating systems.”
48. @anubhavm  “Serverless” Google Cloud defines Serverless “Serverless is a new paradigm of computing that abstracts away the complexity associated with managing servers for mobile and API backends, ETL, data processing jobs, databases, and more.” AWS defines Serverless “Serverless computing allows you to build and run applications and services without thinking about servers. Serverless applications don't require you to provision, scale, and manage any servers.” Microsoft (Azure) defines Serverless “Serverless computing is the abstraction of servers, infrastructure, and operating systems.”
49. @anubhavm 
50. @anubhavm  “Serverless” “Serverless is a technology pattern that provides services and concepts to minimize operational overhead that comes with managing servers. It is a powerful abstraction when used can result in an increased focus on business value.” - Anubhav Mishra OSCON 2018, Portland
51. @anubhavm  “Serverless” “Serverless is a technology pattern that provides services and concepts to minimize operational overhead that comes with managing servers. It is a powerful abstraction when used can result in an increased focus on business value.” - Anubhav Mishra OSCON 2018, Portland FaaS = Functions as a Service BaaS = Backend as a Service
52. @anubhavm 
53. @anubhavm 
54. @anubhavm 
55. @anubhavm 
56. @anubhavm  Benefits / Advantages Reduced Operational/Infrastructure Cost Pay for What You Use Faster Development Cycles Increased Elasticity / Scaling
57. @anubhavm  Benefits / Advantages Reduced Operational/Infrastructure Cost Pay for What You Use Faster Development Cycles Increased Elasticity / Scaling
58. @anubhavm  “Serverless” CI/CD Developer Trigger Webhook $ git push
59. @anubhavm  “Serverless” CI/CD Developer Trigger Webhook $ git push
60. @anubhavm  “Serverless” CI/CD Developer Trigger Webhook $ git push
61. @anubhavm  “Serverless” CI/CD Developer Trigger Webhook $ git push
62. @anubhavm  Let’s build stuff
63. @anubhavm  Let’s build stuff
64. @anubhavm  Live Demos
65. @anubhavm  Live Demos
66. @anubhavm  Extending Jenkins using Serverless Build Test Deploy (Production) $ git push Trigger Webhook
67. @anubhavm  Extending Jenkins using Serverless $ git push Trigger Webhook
68. @anubhavm  Extending Jenkins using Serverless $ git push Trigger Webhook Master Slaves Clients
69. @anubhavm  No Nodes/Servers to Manage Launch 10,000+ Builds/Containers in Seconds No Cost for Idle Time Cost Summary (1 vCPU, 2GB memory): • With Jenkins on Fargate: 100 builds * 5 mins = $0.633/month • With Jenkins on EC2 Instances: ~ $50/month Advantages of Using CaaS for Jenkins
70. @anubhavm  Cold Starts - Slower Boot Times for Clients Large Container Images (~1G) No Root Access Ephemeral Storage (Default) Drawbacks of Using CaaS for Jenkins
71. @anubhavm  [1] Released ~2015-2016 (Blog Post) 3000+ Github Stars Written in Node.js Supports AWS Lambda OSS “Serverless” CI/CD Projects LambCI Serverless continuous integration [1] Michael Hart’s blog post on LambCI - a serverless build system https://medium.com/@hichaelmart/lambci-4c3e29d6599b
72. @anubhavm  Continuous Integration using LambCI $ git push Trigger Webhook LambCI +
73. @anubhavm  1000+ Concurrent Builds No Maintenance of Servers No Cost for Idle Time Flexible Billing (100 Concurrent Builds of 10 secs vs 10 Individual 5 Minute Builds) Cost Summary (1GB Memory): • With AWS Lambda Free Tier: 1000 builds * 5 mins = $0.00/month • With AWS Lambda Paid: 1000 builds * 5 mins = $5.00/month Advantages of using FaaS for CI/CD
74. @anubhavm  Limited Runtime Support (Node.js, Python, Golang) Limited Support for Plugins No Root Access ~5 Minutes Maximum Build Time Drawbacks of Using FaaS for CI/CD
75. @anubhavm  https://twitter.com/kelseyhightower/status/902525728725405697?lang=en
76. The Future Photo by Pierre Châtel-Innocenti on Unsplash
77. www.hashicorp.com FOR EVERYONE, EVERYWHERE Thank You! I have stickers! Ask me anything. @anubhavm Anubhav Mishra