Small Scale Engineering - Velocity NY '18

Scaling to hundreds of hosts and serving thousands of requests per second is interesting and hard. But what about when everything is smaller? How can a tiny team—or even one systems person working alone—make sure small but vital systems stay reliable? Are SRE practices still relevant?

We explain why small scale engineering is just as challenging as large-scale engineering and offer ideas on how to survive technical debt, poor communication, and other everyday challenges.

Transcript

1. effie mouzeli Velocity NY ‘18 manjiki
2. About jijiki/effie @freenode effie@wikimedia.org manjiki
3. Who cares? • Solo systems engineers • Single “sysadm*” teams • Small orgs • Startups • Juniors! * ops/SRE/systems/DevOps(?) 3
4. Do we care? • Most startups are small scale • Small scale is everywhere • Everything was small scale once • We rely on products of small orgs 4
5. Life at small scale • Intimate • Time to grow up • Make decisions • Take responsibility • Be resourceful and creative • A.M.A. Engineering 5
6. Life at small scale • Stressful • Lonely • Fighting many battles at once • No safety nets • Always on call • A.M.A. Engineering • Invited late to the party manjiki 6
7. How we got here? • Lack of processes and experience • Adoption of immature tech • Re-inventing the wheel and cargo cult • Frequent change of requirements • Overestimation 7
8. How we got here? Startups • Short term planning • A lot of duct tape • Early adopters of fancy tech • No documentation, processes, boundaries • No Backups :) 8
9. How we got here? Small Organisations • More duct tape • Lack of original information • Waiting for a hero or heroes • Communication issues • Untouchable systems 9
10. What’s our role • Set standards • Create processes • Identify and solve problems • Bridge the gap between production and developers • Design and advise for the future 10
11. Challenges • New environment • Unknown infrastructure • A lot of debt • Bad habits • Finding information • What’s coming? 11 manjiki
12. The 5 stages of technical debt • Denial • Anger • Bargaining • Depression • Acceptance 12
13. It’s all about Control • Ask the right questions • Understand your product • Write down everything • Task dashboard • Know your current limits • Choose software wisely • Learn to let go • Be consistent and create rules 13
14. The big picture • Provisioning and automations • Environments • Monitoring & Alerting • Metrics & Logging • Update management • Viable backups • Security • Legacy systems 14
15. The Dev team is YOUR team • Build relationships and trust • Be approachable • Don’t prioritise alone • Stay in touch • Share what you know and train • Break bad habits • Don’t be arrogant 15
16. Killing the sysadmn (or the team) • Lack of recognition • Overworking • Burnout • Toxicity • Martyrs and Messiahs 16 manjiki
17. Creating a culture • Guard your processes • Junior engineers • Adopt good practices • To ERR is human • Following the Golden Rule* can go a long way * https://en.wikipedia.org/wiki/Golden_Rule 17
18. TL;DR 18
19. On a final note • Working in small orgs is not a waste of time • Small scale is hard • Good foundations will make scaling easier • Always say “we” • Embrace failure • Remember to breathe • No systems are perfect, just manageable 19
20. Thank you! jijiki/effie @freenode effie@wikimedia.org manjiki