Rise of the Machines - Automate your Development

When we talk about automation in software development, we immediately think of automated builds and deployments. We may also be using scripts to help make our daily work easier. But this is really just the beginning of the rise of the machines.
I show you how leading developers in our industry are using open source and commercial tools for automating much more. They've got "robots" for monitoring production servers, updating issues, supporting customers, reviewing code, setting up laptops, doing development reporting, conducting customer feedback -- even automating daily standups. In what instances is it useful to automate? In what cases does it not make sense? Automation prevents us from having to do the same thing twice, helps us to work better together, reduces workflow errors and frees up time to write production code. Plus, as it turns out, spending time on automation is fun! Don't be afraid of robots in software development, embrace them! Even if I save you just half an hour a week, this talk will be a beneficial investment of your time.

Transcript

1. Rise of the machines Automate your development @svenpet
2. Disclaimer Scripts Chef, Puppet, etc better build pipelines
3. Disclaimer Ideas through examples
4. Rise of the machines
5. ROBOTS are coming to our day to day lives
10. Rise of the machines IN Software Development
11. Sven Peters Atlassian Evangelist @svenpet svenpet.com
12. Automation in Software Development my personal story build scripts (make) 1996 JUnit 2001 2004 ant -> later maven 2005 Eclipse (code generation) Hudson 2006 2009 Puppet 1998 GUI builder 2013 Docker
13. Build & Tests Deployments Small scripts What are we automating right now?
14. What are we automating right now? Code Reviews Stand Ups Ops Support Merging Engineering Health
15. Bots everywhere Coding bots Ops bots Service bots Doc bots Report bots Test bots
16. Welcome… or I’m new
17. Add ons Postgres Scala IntelliJ Git JDK Happy installing & setup… for the next 8 hours
25. Agent Charlie Meet
26. Download Install Configure Agent Charlie Meet
27. Download Install Configure Agent Charlie Meet
28. Puppet, Shell & Ruby Works with OSX & Ubuntu Atlassian Plugin SDK Bitbucket SSH Keys Git Mercurial HipChat HomeBrew IntelliJ IDEA JDK Maven PostgreSQL Stash Keys and much more…
29. Without Agent Charlie With Agent Charlie
30. Coding bots Automate your Coding phase
31. 27’’ is not enough
32. Chat Email Repo manager IDE Command line Issue Tracker
33. Issue tracker workflow TODO IN PROGRESS IN REVIEW DONE
34. BRANCH CODE REFACTOR PULL MERGE Developer workflow
35. BRANCH CODE REFACTOR PULL MERGE What you should do TODO IN PROGRESS IN REVIEW DONE
36. BRANCH CODE REFACTOR PULL MERGE What you do TODO IN PROGRESS IN REVIEW DONE
37. Keep the team updated!
39. Smart commits mention issue_key and #trigger the issue tracker Github JIRA YouTrack Trello Bitbucket
40. Control the issue by commits TODO IN PROGRESS IN REVIEW DONE JRA-566 #start-Progress JRA-566 #start-review fixes #566
41. No tool switching Robot is just receiving commands
42. TODO IN PROGRESS IN REVIEW DONE branch commit Controlled by action pull create review request merge close review
44. Our workflow TODO IN PROGRESS IN REVIEW DONE branch pull merge
45. branch pull merge master JRA-554-bugfix
46. branch pull merge master commits code directly to the master branch!
47. branch pull merge master
48. branch pull merge master JRA-554-bugfix
49. branch pull merge master JRA-554-bugfix
50. bad surprise sh*t happens, but can be avoided
51. bad surprise sh*t doesn’t happen that often no
52. branch pull merge don’t branch from a red build master JRA-554-bugfix
53. branch pull merge master JRA-554-bugfix invite the best reviewers
55. Recent comitters or author
56. auto suggest reviewer
57. branch pull merge minimum 2 approvals master JRA-554-bugfix
58. branch pull merge unapprove reviewers when code changes master JRA-554-bugfix
59. The need to update older versions
60. master Version 1.1 Version 1.0 Support older versions
61. master Version 1.1 Version 1.0 JRA-234-hotfix Support older versions
62. master Version 1.1 Version 1.0 JRA-234-bugfix Support older versions boring!
63. Huhu I can do it!
64. master Version 1.1 Version 1.0 JRA-234-bugfix auto merges
65. Test bots SMARTER Test Automation
66. &:hover { text-decoration: none; color: @invertedBodyTextColor background-color: #366ca6; cursor: pointer; }
67. &:hover { text-decoration: none; color: @invertedBodyTextColor background-color: #366ca6; cursor: pointer; } Static code analysis can catch this! Findbugs - Checkstyle - PMD
68. Listen to what Freud tells you about source code your
69. Freud Bot &:hover { text-decoration: none; color: @invertedBodyTextColor background-color: #366ca6; cursor: pointer; } You should use a variable instead for . I suggest that you use a variable of similar color: #366CA6 @globalHeaderPrimaryActionFallbackColor @globalHeaderTopColor @primaryLinkColor
70. Freud Bot for Pull Requests
71. Freud Bot less review work makes suggestions
72. We are trying hard to write awesome code
73. package com.miguelcatalan.materialsearchview; import android.app.Activity; import android.widget.ListView; import android.widget.RelativeLayout; import android.widget.TextView; import java.lang.reflect.Field; import java.util.List; /** * @author Miguel Catalan Bañuls */ public class MaterialSearchView extends FrameLayout implements Filter.FilterListener { private MenuItem mMenuItem; private boolean mIsSearchOpen = false; public MaterialSearchView(Context context) { this(context, null); } public MaterialSearchView(Context context, AttributeSet attrs) { this(context, attrs, 0); @Override public void onTextChanged(CharSequence s, int start, int before, int count) { mUserQuery = s; startFilter(s); MaterialSearchView.this.onTextChanged(s); } } LOC: 32 MyAwesomeClass
74. One month later
75. package com.miguelcatalan.materialsearchview; import android.app.Activity; import android.content.Context; import android.content.Intent; import android.content.pm.PackageManager; import android.content.pm.ResolveInfo; import android.content.res.TypedArray; import android.graphics.Rect; import android.graphics.drawable.Drawable; import android.os.Build; import android.os.Parcel; import android.os.Parcelable; import android.speech.RecognizerIntent; import android.text.Editable; import android.text.TextUtils; import android.text.TextWatcher; import android.util.AttributeSet; import android.util.Log; import android.view.KeyEvent; import android.view.LayoutInflater; import android.view.MenuItem; import android.view.View; import android.view.inputmethod.InputMethodManager; import android.widget.AdapterView; import android.widget.EditText; import android.widget.Filter; import android.widget.Filterable; import android.widget.FrameLayout; import android.widget.ImageButton; import android.widget.ListAdapter; import android.widget.ListView; import android.widget.RelativeLayout; import android.widget.TextView; import com.miguelcatalan.materialsearchview.utils.AnimationUtil; …one month later LOC: counting…. MyAwesomeClass
76. mSearchSrcTextView.setText(null); mSearchSrcTextView.requestFocus(); if (animate) { AnimationUtil.fadeInView(mSearchLayout, AnimationUtil.ANIMATION_DURATION_MEDIUM, new AnimationUtil.AnimationListener() { @Override public boolean onAnimationStart(View view) { return false; } @Override public boolean onAnimationEnd(View view) { if (mSearchViewListener != null) { mSearchViewListener.onSearchViewShown(); } return false; } @Override public boolean onAnimationCancel(View view) { return false; } }); } else { mSearchLayout.setVisibility(VISIBLE); if (mSearchViewListener != null) { mSearchViewListener.onSearchViewShown(); } } mIsSearchOpen = true; } /** * Close search view. */ public void closeSearch() { …one month later LOC: 986 MyAwesomeClass
77. Code over time 86% 20 1045 27 test coverage deprecated methods old unit tests (JUnit 3) compiler warnings
78. Code over time 83% test coverage 23 deprecated methods 1136 old unit tests (JUnit 3) 36 compiler warnings
79. Code over time 75% test coverage 32 deprecated methods 1289 old unit tests (JUnit 3) 45 compiler warnings
80. Developers are trying to solve problems Developers don’t write bad code on purpose
81. Calling Dr. Code measure constantly the engineering health
82. measure constantly the engineering health Calling Dr. Code
83. jira.stats.tests.junit.3.count No more JUnit 3 tests! JUnit 3 JUnit 4
84. jira.stats.bundled.jars.in.plugins Inspect all the things jira.stats.tests.junit.3.count
85. Hall of Shame! You’ll enter the JS not minified API compatibility :( New JUnit3 test Caught by Dr Code?
86. Hall of Shame! You’ll enter the Wallboard - so everyone sees it!
87. Hall of Fame! You’ll enter the Fixed a matrix?
88. public abstract class Task { public final void executeWith(Callback callback) { execute(); if (callback != null) { callback.call(); } } public abstract void execute(); } public abstract class Task { public final void executeWith(Callback callback) { execute(); if (callback != null) { callback.call(20); } } public abstract void execute(); } Old version New version
89. public abstract class Task { public final void executeWith(Callback callback) { execute(); if (callback != null) { callback.call(); } } public abstract void execute(); } public abstract class Task { public final void executeWith(Callback callback) { execute(); if (callback != null) { callback.call( 20); } } public abstract void execute(); } Old version New version
90. Old version New version
91. Old version New version Hard to spot?
92. Image Compare Bot Compare images (on build) Alert on differences (build warning) Easy reporting (build report)
93. Our functional tests ran for h urs so we ran them parallel
94. Test block 1 Test block 2 Test block 3 Test block 4 Test block 5 statically sized test blocks waiting for the longest test block
95. Hallelujah Functional Test Balancer
96. T1 T 2 T 3 T4 T5 T6 T7 T8 T9 T10 T11 T 12 T13 T14 Hallelujah
97. build status
98. build status
99. build status
100. Flaky Test Detector
101. Flaky Test Detector failing test rerun test successful test test in quarantaine continue build create issue
102. 1888 flaky tests detected 817 quarantined tests fixed
103. Flaky Test Detector no log file searching forces fixing flaky tests no manual reruns no more ‘this one again’ moments
104. OPS bots Automated Operations
106. Ops now problem
107. ops Slow feedback loop dev prob
108. Faster feedback loop dev prob
109. Faster feedback loop knows which team -> service responsibility
110. Faster feedback loop But that’s a lot of configuration!
111. Faster feedback loop add team in deployment descriptor add connection to pager duty
112. Own your code end-to-end
113. Prepared for the release? WAIT… Release Notes?
114. New Feature Improvement Bugfix Do you know which features really ship? Release Branch
115. Merged JRA-345 Commit messages + Issue information = Release notes Release Notes Bot
117. Label in GitHub Issues Label in GitHub Issues
120. New features Improvements Fixed bugs
121. Issue Description as Release Notes Human readable issue descriptions Think about customer value No surprises
122. Service bots Automate your Service
123. Support Developer scanning log files ask for log file system knowledge
124. pause work
125. Support Developer scanning log files ask for log file system knowledge
126. Support Hercules scanning log files ask for log file system knowledge
127. Teaching the robot
128. Hercules get’s smarter suggest read update
129. {Well f*%k me THE ROBOT was right! }
130. Your boss has a really important task for your team
131. Everyone is working from home today
133. by
135. With chat you can reach everyone immediately
136. All these questions! How do I download the Sprint report? How do I set up a build for my feature branch? What Java versions do we support?
137. !ask I’ll look it up for you Self service for chat
138. !ask the knowledge base first!
139. REPORT bots Manual Reporting sucks
140. Stand Up Automating s ?
141. too long setup VC calls timezones remind people Stand Up Problems
143. Never miss a stand up Music - Start at 09:29:00 Crontab!
144. Video - Open at 09:30 No video set up time
145. Stand up report bots colocated distributed different timezones
146. Status update
147. Team update
148. Stand up bot
149. Exception Build failure New issue Deployment Chat Bots everywhere
150. ΩΩΩΩ Builds
151. ΩΩΩΩ Deployments
152. ΩΩΩΩ Exceptions
153. Chat Bots everywhere Fast feedback loops Team knowledge Start discussions
154. The build failed New deployment
155. The build failed New deployment
156. The build failed New deployment
157. New important issue New deployment
158. New important issue New deployment
159. New important issue New deployment Get the big picture
160. Get the big picture
161. Every team has one or more http://atlasboard.bitbucket.org/
162. Every team has one or more http://atlasboard.bitbucket.org/ see what’s going on point directly
163. All kind of boards Advocates Developer Support
164. Bots everywhere Coding bots Ops bots Service bots Doc bots Report bots Test bots
165. Reinvent the wheel? Lots of unmaintained projects Writing automation code Very specific problem
166. No. 1 Problem
167. 10% for better engineering health work time Technical debt
168. 10% for better engineering health work time Technical debt
170. is serious fun! Automation Go - Ruby - PHP - Python Languages Puppet - Chef - Ansible - Docker Technologies Fast feedback Success Your Problems Motivation
171. From scripts on a developer hard disc
172. to professional automation
173. Source Control Add code to
174. Code Reviews Do for every change
175. Tests Write
176. Deployment Scripts Have
177. Keep your automated systems under control
178. Keep your automated systems under control Lots of different places
179. Instances for automation services
180. cost centre ow ner service We use labels for better overview
181. Micros <our own PaaS> Docker Java Go Python
182. micros service:create <service-name> Service descriptor Owner PagerDuty Cost Centre Micros Simple to create & register
183. Were are we heading?
184. Better customer insides better conclusions collecting more data Were are we heading?
185. Faster dev feedback loops less time updating & searching integrated tools Were are we heading?
186. Better Quality deeper code analysis smarter tests Were are we heading?
187. Building software Parts can be automated craft is a Creating customer value is an art Hard to automate
188. Rise of the machines …Skynet became self aware
190. @svenpet svenpet.com/talks I’ll be back… find slides here
191. Photo Credits “VW Golf TDI Clean Diesel was 2010 8983" by Mariordo Mario Roberto Duran Ortiz - Own work. Licensed under CC BY-SA 3.0 via Commons - https:// commons.wikimedia.org/wiki/ Backyard Series by syauqee mohamad "http:// www.flickr.com/photos/7901820@N07/6945260735" Tesla Model S by Chrishmt0423 from flickr Burnt & Abandoned Computer "http://www.flickr.com/ photos/99649389@N02/13262802714"