Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Accidental Sys Admin

1797307564c2566c3e006aac2f44458f?s=47 Chris Young
September 29, 2016

The Accidental Sys Admin

Slides from my talk at Agile Cambridge 2016

I never set out to be a sysadmin but after 15 years in technology management it’s the hat I put on most often. And one I encourage people to wear on a daily basis.
Without a shared understanding of what it takes to run a software system in production, delivery falters and failure demand rises.

The way I’ve gone about fostering this shared understanding is to start by establishing a culture of trust. One manifestation of this is a permissive approach to root and admin permissions with constraints only added for what they enable.

I’ve also done away with staging. For me Staging had become a ‘Pot Roast’ - something you have just because you’ve always had one in the past. As an experiment at Honeycomb we did without and haven’t looked back. It makes our deployment pipeline simpler and quicker, discourages batching of features and saves us money.

I encourage a ‘Do It Yourself’ attitude amongst the people I work with. If the bottleneck or impediment is somewhere in the stack that you don’t feel at home in but is where the need presents itself, a willingness to learn skills outside of your domain and generalise can get things moving far quicker than handing off to a specialist.

There are limits and risks to this. A shallow knowledge of something can lead to mistakes. A while back I naively created an AWS VPC with a public CIDR block. This mistake was not discovered for quite a while and then took quite a bit of effort to put right.

There is a tension between the speed and agility of generalisation and the robustness and focus of specialisation. When I do introduce specialisation I look for the people involved, those providing the specialist capability and those with the need it addresses to collaborate to establish the interface between them.

To ensure interfaces such as this work, that resources are balanced against demand, you need instrumentation. I urge developers to make Instrumentation and Monitoring first class citizens and to get them into the code from the get go.

All of these practices are there to encourage an engagement with the system as a whole wether you work as a generalist across a number of disciplines or as a specialist focused on specific concerns.

1797307564c2566c3e006aac2f44458f?s=128

Chris Young

September 29, 2016
Tweet

More Decks by Chris Young

Other Decks in Programming

Transcript

  1. The Accidental Sys Admin Chris Young – CTO Honeycomb TV

  2. $ whoami

  3. •  Bedroom Coder, *nkerer •  Technology manager •  Returned to

    coding •  CTO
  4. Who are you? Manager Developer Operations

  5. When I say SysAdmin… A system administrator, or sysadmin, is

    a person who is responsible for the upkeep, configuration, and reliable operation of computer systems; especially multi-user computers, such as servers. The system administrator seeks to ensure that the uptime, performance, resources, and security of the computers he or she manages meet the needs of the users, without exceeding the budget. https://en.wikipedia.org/wiki/System_administrator
  6. Why Accidental?

  7. None
  8. Unix computer systems administrator

  9. None
  10. A Program Brooks – The Mythical Man Month 1975

  11. Brooks – The Mythical Man Month 1975 A Program A

    Programming Product (Generaliza*on, Tes*ng, Documenta*on, Maintenance) x3
  12. Brooks – The Mythical Man Month 1975 A Program A

    Programming System A Programming Product (Generaliza*on, Tes*ng, Documenta*on, Maintenance) (Interfaces, Systems Integra*on) x3 x3
  13. Brooks – The Mythical Man Month 1975 A Program A

    Programming System A Programming Product (Generaliza*on, Tes*ng, Documenta*on, Maintenance) (Interfaces, Systems Integra*on) A Programming Systems Product x3 x3
  14. My (Current) Practices

  15. 1. Permissive Root Access dev$ sudo mount WARNING: Improper use

    of the sudo command could lead to data loss or the deletion of important system files. Please double- check your typing when using sudo. Type "man sudo" for more information. To proceed, enter your password, or type Ctrl-C to abort. Password: dev is not in the sudoers file. This incident will be reported.
  16. None
  17. 2. No staging

  18. 2. No staging

  19. 3. Do it yourself… Purpose AWS Features HTML / CSS

    / JavaScript Rails/Node/Go Ansible Docker Strategy VMs The Real
  20. 3. Do it yourself… Purpose AWS Features HTML / CSS

    / JavaScript Rails/Node/Go Ansible Docker Strategy VMs The Real
  21. 3. Do it yourself… Purpose AWS Features HTML / CSS

    / JavaScript Rails/Node/Go Ansible Docker Strategy VMs The Real
  22. …up to a point

  23. None
  24. None
  25. None
  26. None
  27. None
  28. None
  29. None
  30. “Are you rated to create an AWS VPC Young?”

  31. 4. Agree Interfaces

  32. None
  33. John Seddon - Systems Thinking in the Public Sector -

    2008
  34. Instrument! Instrument! Instrument!

  35. None
  36. None
  37. None
  38. 1.  Permissive Root Access 2.  No Staging 3.  Do it

    Yourself…up to a point 4.  Agree Interfaces 5.  Instrument!
  39. Thank You! chris.young@honeycomb.tv @worldofchris @honeycomb_tv