The Accidental Sys Admin

Slides from my talk at Agile Cambridge 2016

I never set out to be a sysadmin but after 15 years in technology management it’s the hat I put on most often. And one I encourage people to wear on a daily basis.
Without a shared understanding of what it takes to run a software system in production, delivery falters and failure demand rises.

The way I’ve gone about fostering this shared understanding is to start by establishing a culture of trust. One manifestation of this is a permissive approach to root and admin permissions with constraints only added for what they enable.

I’ve also done away with staging. For me Staging had become a ‘Pot Roast’ - something you have just because you’ve always had one in the past. As an experiment at Honeycomb we did without and haven’t looked back. It makes our deployment pipeline simpler and quicker, discourages batching of features and saves us money.

I encourage a ‘Do It Yourself’ attitude amongst the people I work with. If the bottleneck or impediment is somewhere in the stack that you don’t feel at home in but is where the need presents itself, a willingness to learn skills outside of your domain and generalise can get things moving far quicker than handing off to a specialist.

There are limits and risks to this. A shallow knowledge of something can lead to mistakes. A while back I naively created an AWS VPC with a public CIDR block. This mistake was not discovered for quite a while and then took quite a bit of effort to put right.

There is a tension between the speed and agility of generalisation and the robustness and focus of specialisation. When I do introduce specialisation I look for the people involved, those providing the specialist capability and those with the need it addresses to collaborate to establish the interface between them.

To ensure interfaces such as this work, that resources are balanced against demand, you need instrumentation. I urge developers to make Instrumentation and Monitoring first class citizens and to get them into the code from the get go.

All of these practices are there to encourage an engagement with the system as a whole wether you work as a generalist across a number of disciplines or as a specialist focused on specific concerns.

Transcript

1. The Accidental Sys Admin Chris Young – CTO Honeycomb TV
2. $ whoami
3. •  Bedroom Coder, *nkerer •  Technology manager •  Returned to coding •  CTO
4. Who are you? Manager Developer Operations
5. When I say SysAdmin…
   A system administrator, or sysadmin, is a person who is responsible for the upkeep, configuration, and reliable operation of computer systems; especially multi-user computers, such as servers. The system administrator seeks to ensure that the uptime, performance, resources, and security of the computers he or she manages meet the needs of the users, without exceeding the budget. https://en.wikipedia.org/wiki/System_administrator
6. Why Accidental?
8. Unix computer systems administrator
10. A Program Brooks – The Mythical Man Month 1975
11. Brooks – The Mythical Man Month 1975 A Program A Programming Product (Generaliza*on, Tes*ng, Documenta*on, Maintenance) x3
12. Brooks – The Mythical Man Month 1975 A Program A Programming System A Programming Product (Generaliza*on, Tes*ng, Documenta*on, Maintenance) (Interfaces, Systems Integra*on) x3 x3
13. Brooks – The Mythical Man Month 1975 A Program A Programming System A Programming Product (Generaliza*on, Tes*ng, Documenta*on, Maintenance) (Interfaces, Systems Integra*on) A Programming Systems Product x3 x3
14. My (Current) Practices
15. 1. Permissive Root Access dev$ sudo mount WARNING: Improper use of the sudo command could lead to data loss or the deletion of important system files. Please double- check your typing when using sudo. Type "man sudo" for more information. To proceed, enter your password, or type Ctrl-C to abort. Password: dev is not in the sudoers file. This incident will be reported.
17. 2. No staging
18. 2. No staging
19. 3. Do it yourself… Purpose AWS Features HTML / CSS / JavaScript Rails/Node/Go Ansible Docker Strategy VMs The Real
20. 3. Do it yourself… Purpose AWS Features HTML / CSS / JavaScript Rails/Node/Go Ansible Docker Strategy VMs The Real
21. 3. Do it yourself… Purpose AWS Features HTML / CSS / JavaScript Rails/Node/Go Ansible Docker Strategy VMs The Real
22. …up to a point
30. “Are you rated to create an AWS VPC Young?”
31. 4. Agree Interfaces
33. John Seddon - Systems Thinking in the Public Sector - 2008
34. Instrument! Instrument! Instrument!
38. 1.  Permissive Root Access 2.  No Staging 3.  Do it Yourself…up to a point 4.  Agree Interfaces 5.  Instrument!
39. Thank You! chris.young@honeycomb.tv @worldofchris @honeycomb_tv