Maxistentialism (pipeline patterns)

Max Griffiths & Max Lincoln both work for ThoughtWorks and have been noticing a distinct lack of upfront thinking about how your software delivery pipeline can affect culture, feedback, and governance. This presentation first aired at DevOps Delaware meetup, followed by DevOpsDays Detroit 2015 and aims to document the patterns they've seen in their combined 20+ years Dev and Ops experience. They specifically address how your pipeline is likely to scale in 3 different directions: more stages between commit and production; more products/services/applications; and then concurrently developing different versions of the same application.

Transcript

1. MAXISTENTIALISM (pipeline patterns) Max Griffiths & Max Lincoln 1
2. 2 MAX GRIFFITHS @_maxamg ! • DevOps thing for 10+ years ! • Work for ThoughtWorks in San Francisco ! • Currently tackling [seemingly] simple problems at scale in large financial organisations
 
 …in Delaware
3. 3 MAX LINCOLN @devopsy ! ! • Also DevOps thing for 10+ years ! • Work for ThoughtWorks in New York ! • Lots of OpenSource
 
 …and Swagger
4. PROBLEM STATEMENT http://il9.picdn.net/shutterstock/videos/5203178/thumb/1.jpg
6. SO MANY THINGS WE COULD TALK ABOUT…
7. SO MANY THINGS WE COULD TALK ABOUT… https://1.bp.blogspot.com/-kmv_Txf_hfo/TmURSvoWiVI/AAAAAAAAAI4/HNdn0fjuq74/s1600/VSM+Example+3.gif
8. SO MANY THINGS WE COULD TALK ABOUT…
9. SO MANY THINGS WE COULD TALK ABOUT…
10. SO MANY THINGS WE COULD TALK ABOUT…
11. SO MANY THINGS WE COULD TALK ABOUT…
12. SO MANY THINGS WE COULD TALK ABOUT…
13. ANYONE READ ANY OF THESE?
14. ANYONE READ ANY OF THESE? https://www.thoughtworks.com/radar
16. BUT WE’RE GOING TO FOCUS ON A FEW OBJECTIVES:
17. BUT WE’RE GOING TO FOCUS ON A FEW OBJECTIVES: ◆ culture ◆ feedback ◆ governance
18. GOOD PIPELINES:
19. GOOD PIPELINES:
20. GOOD PIPELINES: -Incremental quality assurance
21. GOOD PIPELINES: -Incremental quality assurance -Fast feedback to the right people
22. GOOD PIPELINES: -Incremental quality assurance -Fast feedback to the right people -Visibility and shared understanding of the progress of a release candidate
23. GOOD PIPELINES: -Incremental quality assurance -Fast feedback to the right people -Visibility and shared understanding of the progress of a release candidate -A consistent and auditable software delivery process
25. AND SCALING CHALLENGES:
26. PIPELINE BUILDING BLOCKS
27. 11 THE GOAL Build Acceptance Test Regression Test Developers Frequent Commits Showcase CFR Tests Partner Preview Production V1 V2 V3 V4 Automatically Triggered Manually Triggered
28. 12 Material (Version Control) Git
29. 13 Job Git Build
30. 13 Job Git Build Build Build
31. 14 Stage Git Build Build Lint
32. 14 Stage Git Build Build Build Lint Lint
33. Build Build Lint 15 Pipeline Git Build Publish Upload
34. 16 Artifact Repo Git Build Build Lint Build Publish Upload Artifacts
35. 16 Artifact Repo Git Build Build Lint Build Publish Upload Artifacts Publish Upload
36. 17 Git Build Build Artifacts Publish
37. 18 Build Build Artifacts Publish Acceptance Deploy Test Exploratory Deploy
38. 18 Build Build Artifacts Publish Acceptance Deploy Test Exploratory Deploy
39. 19 Build Build Publish Acceptance Deploy Test Exploratory Deploy Prod Deploy
40. X-AXIS: SCALING MORE STAGES
41. 21 HOW SIMPLE CAN YOU GET?
42. Prod Source 21 HOW SIMPLE CAN YOU GET?
43. Source Prod HOW SIMPLE CAN YOU GET? Test Build
44. Source Prod HOW SIMPLE CAN YOU GET? Test Build
45. Source Prod HOW SIMPLE CAN YOU GET? Test Build ◆ feedback
46. 25 Source Prod MOAR TESTING Test Build QA ◆ feedback
47. Source Prod HMM… Test Build QA Perf Sec ◆ culture ◆ feedback ◆ governance
48. Source Prod HMM… Test Build QA Perf Sec ◆ culture ◆ governance
49. Source Prod Test Build QA Perf Sec
50. Source Prod Test Build QA Perf Sec PARALLELISM
51. Example
52. Source Prod Build Test Int QA E.g.
53. Source Prod Build Test Int QA 85% 15% 15% E.g.
54. Source Prod Build Test Int QA 85% 15% 15% 2-6hrs E.g.
55. Source Prod Build Test Int QA 85% 15% 15% 2-6hrs E.g.
56. Source Prod Build Test Int QA 0.5hr
57. Source Prod Build Test Int QA Sec Perf
58. Source Prod Build Test Int QA Sec Perf ◆ feedback ◆ governance
59. X-AXIS: PATTERNS
60. SIMPLE LINEAR
61. SIMPLE PARALLEL ◆ feedback
62. LAYERED Gold Silver Bronze ◆ culture
63. LAYERED w/ ARTIFACT PROMOTION Gold Silver Bronze ◆ governance ◆ culture
64. RELEASE CHANNELS Gold Silver Bronze Beta Users Other Teams ◆ feedback
65. X-AXIS: SCALING MORE STAGES
66. Y-AXIS: SCALING MORE PROJECTS X-AXIS: SCALING MORE STAGES
67. Catalog Prod Test Build Price Prod Test Build Deals Prod Test Build Web Prod Test Build
68. Web Prod Test Build Catalog Prod Test Build Price Prod Test Build Deals Prod Test Build Catalog Web Price Deals
69. Web Prod Test Build Catalog Prod Test Build Price Prod Test Build Deals Prod Test Build Catalog Web Price Deals
70. Sec Perf Web Prod Test Build Catalog Build Price Build Deals Build
71. Test Sec Perf Web Prod Build Catalog Build Price Build Deals Build
72. Test Sec Perf Web Prod Build Catalog Build Price Build Deals Build v8 v4 v9 v3 v8 v4 v9 v3
73. Test Sec Perf Web Prod Build Catalog Build Price Build Deals Build v8 v4 v9 v3 v8 v4 v9 v3 ??
74. Test Sec Perf Web Prod Build Catalog Build Price Build Deals Build v8 v4 v9 v3 v8 v4 v9 v3 ?? ??
75. Test Sec Perf Web Prod Build Catalog Build Price Build Deals Build v8 v4 v9 v3 v8 v4 v9 v3 ?? ?? ?? ??
76. PATTERNS: Y-AXIS PATTERNS: X-AXIS
77. PROPAGATING (VERSION) LABELS v2 v2 v2 v2 ◆ governance ◆ culture
78. VCS LABELING r123 r123 r123 r123 r123 ◆ feedback
79. JOINT LABELING v4 v7 v4-v7 ◆ feedback
80. AGGREGATE LABELING v1 v9 v4 v2 Group v1
81. AGGREGATE ARTIFACTS v1 v9 v4 v2 Group v1 References ◆ governance
82. INDEPENDENT PIPELINES
83. SHARED LAST MILE
84. Y-AXIS: SCALING MORE PROJECTS X-AXIS: SCALING MORE STAGES
85. Z-AXIS: CONCURRENT RELEASES Y-AXIS: SCALING MORE PROJECTS X-AXIS: SCALING MORE STAGES
86. Web Prod Test Build Web Prod Test Build Web Prod Test Build
87. Web Prod Test Build Web Prod Test Build Web Prod Test Build
88. Web Next Test Build Web Beta Test Build Web Prod Test Build
89. Web Next Test Build Web Beta Test Build Web Prod Test Build v1.0.0 v1.0.0 v1.0.0 v1.0.0
90. Web Next Test Build Web Beta Test Build Web Prod Test Build v1.1.0 v1.1.0 v1.1.0 v1.1.0 v1.0.0 v1.0.0 v1.0.0 v1.0.0
91. Web Next Test Build Web Beta Test Build Web Prod Test Build v2.0.0 v2.0.0 v2.0.0 v2.0.0 v1.1.0 v1.1.0 v1.1.0 v1.1.0 v1.0.0 v1.0.0 v1.0.0 v1.0.0
92. Web Next Test Build Web Beta Test Build Web Prod Test Build
93. Web Next Test Build Web Beta Test Build Web Prod Test Build
94. Web Next Test Build Web Beta Test Build Web Prod Test Build
95. Web Next Test Build Web Beta Test Build Web Prod Test Build
96. PATTERNS: Y-AXIS PATTERNS: Z-AXIS PATTERNS: X-AXIS
97. NOT THIS! http://paulhammant.com/blog/branch_by_abstraction.html
98. BETTER http://paulhammant.com/blog/branch_by_abstraction.html
99. CLONE BY BRANCH v700 v699 v701 trunk v1.0.0 v1.0.0 v1.0.0 v1.0.0 R1 v2.0.67 v2.0.69 v2.0.67 v2.0.67 R2
100. LONG LIVED trunk Release
101. PRE MERGE VALIDATE trunk patch1 patch2
102. PRE-COMMIT PIPELINES? ! • Developer workstations
103. ! - Same base image - Connected pipeline PRE-COMMIT PIPELINES? ! • Developer workstations
104. IMAGE REPO OS ISO -RHEL -ubuntu -whatever
105. IMAGE REPO Prod Web Test Build
106. IMAGE REPO Workstation Web Test Build
107. IMAGE REPO Workstation .iso Web Test Build
108. IMAGE REPO Packer Workstation .iso Web Test Build
109. IMAGE REPO Packer Workstation .ami .vbox .iso Web Test Build
110. IMAGE REPO Packer Workstation .ami .vbox .iso Web Test Build
111. IMAGE REPO Packer Workstation .ami .vbox .iso Web Test Build
112. THATS IT…
113. THATS IT… …OR IS IT

114. THATS IT… …OR IS IT
 • The problem is ongoing. Continuously improve
115. THATS IT… …OR IS IT
 • The problem is ongoing. Continuously improve • Continue the discussion: github.com/ThoughtWorks/PipelinePatterns
117. REMEMBER:
 • Pipelines aren’t just a product of your software delivery process, they’re an opportunity to improve feedback, culture, and governance
118. REMEMBER:
 • Pipelines aren’t just a product of your software delivery process, they’re an opportunity to improve feedback, culture, and governance
119. THANKS! ! ! Max Griffiths (@_maxamg) Max Lincoln (@devopsy)