Bob: A composable and inclusive CI/CD Platform

Fully built in Clojure and embodying the UNIX philosophy, almost Emacs like external extensibility and simplicity of Clojure, Bob tries to be least in our way and tries to be inclusive of our needs and is a CI/CD platform than just a tool, enabling us to build the CI we want, not be hammering away at an existing stubborn thing and more importantly making it much more approachable and inclusive to newbies.

Transcript

1. Bob A composable and inclusive CI/CD platform
2. Hello, world! • @lispyclouds • (❤ :FOSS :Diversity :Sustainable-Living :Anarcho-communism) • (constantly [:Clojure :Infrastructure]) • Backend and Infrastructure tooling development, Consulting @thoughtworks • Compilers, Language(Human and Machine) Nerd, VMs, High Performance • https://twitter.com/lispyclouds • https://github.com/lispyclouds
3. Image courtesy: https://github.com/tallesl/Rich-Hickey-fanclub
5. Too much is done at the core!
6. • The UI. • Pipeline orchestration logic. • VCS control. • Tight coupling to popular VCS platforms. • Opinionated branching control. • The build plan definition as a Domain Specific Language. • Master-Worker configuration. • Build state like artifacts stored inline in non scalable ways.
7. One tool does it all. Easy right?
8. The traditional notion of plugins locks you into a tight coupling with the CI along with issues like vulnerabilities and upgrades. Addressing them outside the core isn’t simple.
9. Badly implemented plugins cause issues at the core and are the bottlenecks of scale.
10. We absolutely have to use the same tech to write plugins in which the CI was built in in the traditional plugin architecture.
11. An UI which was designed to address certain problems in mind is delivered by default hence pretty much providing pre-defined views of the CI infra and to control it. Multi- Modal interfaces or accessing on them can be tricky.
12. Adding features and updates are unreliable and involve downtimes/restarts. Eg: features like adding proper pipeline support is arduous.
13. The opinions of the CI flows up and downstream.
14. Narcissistic Design @stuarthalloway
15. Simple made Easy.
16. Break it down to build it up!
17. Design, Composition and Performance @richhickey
18. Tools.deps over Boot over Leiningen
19. Pick and choose libs over Frameworks
20. Plain SQL over ORMs
21. Plain Data over Complex code hierarchy
22. It’s all Data!© Guess what the © here stands for?
23. Image credits: BBC UK
24. One CI to rule them all?
25. Nah, one CI to build them all!
26. Bob is a CI Platform. https://bob-cd.github.io/bob/
27. Container Pipeline Artifact Resource
28. Resource Providers Artifact Stores External UI External Behaviour REST REST REST REST Docker clj-docker-client Pipeline Artifact Resource Clojure on JVM
29. Bob • FOSS with GNU AGPL v3+. • Exposes its entire vocabulary as a REST API caring only about Execution. • Highly concurrent. • Masterless cluster based scaling with a shared PostgreSQL DB for build state and events. • Has no opinions of an UI, or an opinionated build plan language like YAML and even build planning and scheduling.
30. FROM clojure:boot as builder WORKDIR /opt COPY . . RUN boot show "--deps RUN boot build FROM docker:dind WORKDIR /opt RUN apk add -u wget RUN wget "https:"//cdn.azul.com/zulu/bin/zulu12.3.11-ca-jdk12.0.2-linux_musl_x64.tar.gz" RUN tar -zxvf *.tar.gz RUN rm *.tar.gz RUN mv zulu* jdk COPY "--from=builder /opt/target/bob-standalone.jar . COPY bob-entrypoint.sh /opt RUN chmod +x /opt/bob-entrypoint.sh ENTRYPOINT ["/opt/bob-entrypoint.sh"]
31. Pipeline • Pre-Provisioned build environment with Docker. • Consists of Steps. • Steps are a composition of: (λ resources) "=> artifacts. • Optional environment variables.
32. { "image": "busybox:musl", "vars": { "env": "test" }, "steps": [ { "cmd": "echo hello > test.txt" }, { "cmd": "cat test.txt", "produces_artifact": { "name": "afile", "path": "test.txt", "store": "local" } }, { "needs_resource": "my-source", "cmd": "ls", "produces_artifact": { "name": "license-file", "path": "LICENSE", "store": "local" } } ], "resources": [ { "name": "my-source", "type": "external", "provider": "git-provider", "params": { "repo": "https://github.com/bob-cd/bob", "branch": "master" } } ] }
33. Image1 Container1 Cmd1 Commit Image2 Cmd2 Container2 Commit … [Resource] [Artifact] Passed | Failed [Resource] [Artifact]
34. (reduce execute image steps)
35. Resource • An abstraction over Source Code, Raw files, pipeline outputs etc. • Provided via a Resource Provider which is a web server reachable by Bob. • Can be re-used multiple times in a single build.
36. Resource Provider /bob_resource GET TAR And anything
37. Artifact • The result of a step execution over a resource. • Must be pushed out of Bob as CIs should not be storing them. • Artifacts can be produced during any step of the pipeline. • Stored by an Artifact Store which is a web server reachable from Bob. • The Artifact Store is an abstraction over how to store/publish the artifact. Eg. S3, Docker Hub, Bintray etc.
38. Artifact Store /bob_artifact GET, POST, DELETE And anything
39. Bob Resource Provider Artifact Store Humans and Machines Putting it together
40. Some pre-built opinionated tooling
41. Wendy: The CLI and the external build orchestrator. Native compiled Clojure with GraalVM.
42. Resource-Git: The simple resource provider for public Github repos. In Python 3.
43. Artifact-Local: The simple S3 like storage backed by file-system. In Python 3.
44. can-we-build-it
45. Going beyond just CI/CD • Bob is generic enough to orchestrate any task. • Can be used for any scheduling/orchestration jobs like CD for Machine Learning, data processing, scheduled batch jobs etc. • Just Execute.
46. Other projects in the same spirit • Concourse CI • The UNIX tooling and philosophy • tools.deps • aws-api • Emacs
47. Everything should be made as simple as possible, but no simpler. - Albert Einstein
48. Thank You! Rahul De @lispyclouds Destroy all classes! •https://github.com/bob-cd/bob •https://github.com/bob-cd/wendy •https://github.com/bob-cd/resource-git •https://github.com/bob-cd/artifact-local •https://github.com/lispyclouds/clj-docker-client •https://github.com/lispyclouds •https://twitter.com/lispyclouds