Fireballs, Icebats and 1,000,000 plugins: A story of continuous delivery

Walks through Atlassian's journey towards continuous delivery to OnDemand, covering how we replaced maven and a bunch of shell scripts with Puppet + hiera.

Transcript

2. Fireballs, Icebats and 1,000,000 Plugins PuppetCamp Sydney - February 2014
3. Matt Moor Build Engineering @ Atlassian @imprecise_matt
4. Continuous Delivery (a refresher) Develop => Build => Test => Release!
5. Continuous Delivery (a refresher) Develop => Build => Test => Release! smaller change delta more feedback faster feedback automated upgrade smaller magnitude of failure
6. Our happy path 0 200 400 600 800 2011 2012 2013 2014 2015 Deployments/year quarterly fortnightly weekly daily many/day
7. A little bit of history…
8. A little bit of history… 0 10000 20000 30000 40000 October 2011 June 2012 January 2013 June 2013 December 2013 40,000 32,000 24,000 15,000 500 Approximate number of OnDemand instances
9. A little bit of history… Atlassian Application Application Core Application Bundled Plugins plugin1 plugin2 plugin3 plugin4 pluginN
10. A little bit of history… Atlassian Application Application Core Application Bundled Plugins plugin1 plugin2 plugin3 plugin4 pluginN Alice Bob Charlie Dave Ed Frank Gerry
11. A little bit of history… bash$ cat jira/pom.xml … <dependency> <groupId>com.atlassian.jira</groupId> <artifactId>jira-api</artifactId> <version>${project.version}</version> </dependency> …
12. A little bit of history… Atlassian Application Application Core Application Bundled Plugins plugin1 plugin2 plugin3 plugin4 pluginN Atlassian Application Application Core Application Bundled Plugins plugin1 plugin2 plugin3 plugin4 pluginN Atlassian Application Application Core Application Bundled Plugins plugin1 plugin2 plugin3 plugin4 pluginN
13. Enter the fireball
14. Enter the fireball bash$ cat fireball/pom.xml … <dependency> <GroupId>${jira.group.id}</GroupId> <artifactId>${jira.war.artifact.id}</artifactId> <version>${jira.war.version}</version> … </dependency> …
15. Launching the fireball OpenVZ Container Postgres Apache Confluence JIRA Bamboo Crowd Customer Instance NFS
16. Launching the fireball Datacenter Rack Rack Rack Rack Rack Rack Rack Rack Rack Rack Rack Rack Rack Rack Rack Rack Rack Rack Rack Rack Rack
17. Launching the fireball bash$ sudo /path/to/new.version/install.sh INSTALLED_SERVICES=“postgres jira bamboo confluence crowd apache” for service in ${INSTALLED_SERVICES}; do /path/to/new.version/$service/install.sh done … OR http://www.flickr.com/photos/64577732@N00/1414883135 http://www.flickr.com/photos/44124473769@N01/292642699
18. Launching the fireball Right?
19. A little bit of history… jira/install.sh postgres/install.sh confluence/ install.sh bamboo/ install.sh crowd/install.sh * (over) simplified model
20. A little bit of history… jira/install.sh postgres/install.sh confluence/ install.sh bamboo/ install.sh crowd/install.sh * (over) simplified model http://www.flickr.com/photos/64577732@N00/1414883135
21. A little bit of history… http://www.flickr.com/photos/7337467@N04/3051244623
22. Our happy path 0 200 400 600 800 2011 2012 2013 2014 2015 Deployments/year Progress quarterly fortnightly weekly daily many/day You are here
23. A little bit of history… Atlassian Application Application Core Application Bundled Plugins plugin1 plugin2 plugin3 plugin4 pluginN Alice Bob Charlie Dave Ed Frank Gerry
24. Enter the icebat Fire + Ball Ice + Bat
25. Enter the icebat j2ee-app jira postgres crowd confluence bamboo apache j2ee-plugin vendor plugin postgres 5.4 6.1 5.0 2.7.5 1.1 hiera - versions.yaml
26. Enter the icebat Development 5.4m3 6.2-snapshot 5.1-m1 2.7.5-m6 1.2.1-snapshot hiera - versions.yaml Dogfooding Production 5.4m2.1 6.1m7 5.1 2.7.5-m5 1.2 hiera - versions.yaml 5.4m2 6.1m6 5.0-m1 2.7.5-m2 1.1-m1 hiera - versions.yaml
27. Local overrides global configuration datacenter configuration rack configuration local configuration http://www.flickr.com/photos/streamishmc/4936051/
28. Hard Labour Jason ‘Mechanical Turk’ Birch
29. The Great Leap Forward
30. The Great Leap Forward Applications
31. The Great Leap Forward Plugins
32. The Great Leap Forward Environments
33. The Great Leap Forward Versions
34. The great leap forward “client” facter environment licensed products puppet apply manifesto
35. Our happy path 0 200 400 600 800 2011 2012 2013 2014-01 2014-12 2015 Deployments/year Progress quarterly fortnightly weekly many/week many/day You are here twice daily
36. Next Steps • Onboard everything into Manifesto, such that development teams control their own destinies • Automated acceptance testing • Fault tolerance for manifesto • No more fireballs • ? Dark feature (feature flags) control within Manifesto/icebat
37. Special thanks to… Ryan Thomas Jason Birch
38. Thanks! Dev Ops Devs who can Op Ops who can Dev We’re hiring! atlassian.com/jobs (or speak to me)