Are you setting an appropiate TTL on your technical skills

Transcript

1. Are you setting an appropriate TTL on your technical skills
2. Antisthenes - 400(ish) BC “The most useful piece of learning for the uses of life is to unlearn what is untrue.” Or has become untrue. (cue ominous music)
3. Hi, I’m Ramin Career Highlights • Operations for seven startups • Telco, ISP, webhosting, video streaming, SaaS • Windows office automation consulting • Consumer and enterprise web apps • 20 years (almost) and 13 jobs • Currently an SRE at Twitter
4. Same problems in each Some variations, but themes the same • It’s always been done that way • We don’t run $common_tech, it doesn’t work • Cargo-culting of configs, systems, processes that are inappropriate for the workloads
5. Why?
6. The Human Brain We are pattern finding maniacs • We do it constantly. • We do it unconsciously. • That last one might be a problem.
7. Related to Sysadmin how? We rarely start from scratch • Rely on experience first • Match current circumstances against past experiences • If that fails, try a bunch of things • Finally if all else fails, start from the beginning and work logically towards a solution
8. Related to Sysadmin how? Mechanized thinking • Aka the Einstellung Effect • Predisposition to solve a problem in a specific manner based on recent successes • Human brain's ways of finding an appropriate solution/behavior as efficiently as possible, but not the most efficient solution
9. Related to Sysadmin how? The changing ecosystem we inhabit • HDDs -> SSDs • 1 -> 24+ cores • Ethernet -> 100GE • C10K -> C10M (concurrent connections) • I heard all talks are incomplete without Docker
10. Heuristics - definition “Any approach to problem solving, learning, or discovery that employs a practical method not guaranteed to be optimal or perfect, but sufficient for the immediate goals.” -- Wikipedia
11. Why Heuristics Why are we relying on heuristics? • Every profession uses them • Takes less time to use • Takes less information to use • Takes less experience to use
12. Good Heuristics Can be timeless or app specific • Always check your backups • When troubleshooting test ONE THING AT A TIME. • The asset deploy pipeline for the webapp is brittle, check it first • n+1
13. Expired (Bad?) Heuristics Like every myth, a bit of truth • It’s always the network • Building from src gets you more performance • RAID 5 is a good default • Add your own here
14. Bounds of Rationality “when individuals make decisions, their rationality is limited by the available information, the tractability of the decision problem, the cognitive limitations of their minds, and the time available to make the decision” --wikipedia
15. A little less formal Bounds of our Rationality • Available data • Understanding of the problem • Cognitive ability • Time to process all the above
16. In hindsight, it’s obvious Bounds of Rationality grows over time • Monday morning quarterbacks • Bad postmortems • Accept your earlier work as part of the process
17. Let’s start tying it together Heuristics are created within the bounds of our rationality • Incomplete data • May not completely understand the problem • Need to solve it now • Looking for a pattern
18. Heuristics -> Superstitions Be suspicious of what you know • Changing ecosystem • Incomplete information when heuristic created
19. Heuristic v2, v3, v4 ... Heuristics need to be updated • As our Bounds of Rationality increases we can create new heuristics to replace old ones. • We can update current heuristics to be more accurate
20. Cognitive Behavior Therapy “Mainstream CBT helps individuals replace ‘maladaptive... coping skills, cognitions, emotions and behaviors with more adaptive ones’, by challenging an individual's way of thinking and the way that they react to certain habits or behaviors” --wikipedia
21. Cherry picking cognition For our purposes we are concerned with cognitions rather than behavior • We want to apply this to heuristics • Behavior therapy is a huge field
22. A general example We feel responsible for parents divorce • Caused a fight between our parents • Shortly thereafter parents divorced • We caused the divorce
23. A general example (reality) At 30yrs old we can see it differently • We were seven years old • Left toy at supermarket, had to go back • Caused a fight between our parents • This fight was actually one of dozens • Other stress factors in the relationship also recalled like moving, new job, death in the family, etc.
24. A sysadmin example We caused the outage to last longer • One of the DB slaves stopped replicating • Didn’t notice the alert on that slave • Web app using bad slave couldn’t serve API traffic • Distracted by drop off in serving out of cache • Even after fixed, site was slow for a few hours
25. A sysadmin example (+1day) Some problems in the system • Release went out • All DB slaves experienced replication lag. It wasn’t till they started to catch up that it was possible to notice one was wedged • Replication alert lost in the general alert noise • Site performance was worse than expected
26. Heuristics v1 (+1day) These might look familiar • Releases are dangerous • Replication lag is dangerous • Read all monitoring emails during troubleshooting • Throw hardware at the problem
27. Borrowing from CPT Write down what you think you know • We often act or know intuitively which is hard to reason about or explain • The act of writing it down gives you something concrete • “Use your words”
28. A sysadmin example (+1wk) So so many problems in the system • Schema update locked busiest table for 8 minutes • All DB slaves experienced replication lag. It wasn’t till they started to catch up that it was possible to notice one was wedged • The TTL on hot data was 5 minutes which expired during the schema update. • API could not serve the high traffic event w/ cold cache
29. Heuristics v7 (+1wk) Some refinements • Schema changes lock tables, avoid on production sized data. • Compare req/rate of API proxy cache vs API in cases where we have a cold cache • Throwing hardware at the problem near impossible at the request rates we have on this system
30. Borrowing again from CBT Context matters, avoid • Overgeneralizations • Polarization (“it’s black or white”) • Filtering, magnifying the negative
31. Context, more is better Better heuristics have more context • Releases are dangerous • Release with schema changes are dangerous • Releases w/schema changes on busy tables are dangerous. • Releases w/schema changes on large busy tables can affect the system's ability to serve traffic at production levels.
32. Is this Heuristic done? No, no, nope, nada • Online schema changes more widespread • Will work on most tables • Wait, “most tables”?
33. Borrowing more from CBT You need a second person (at least) • Keeps you honest with yourself • Keeps you from taking too much responsibility • Asks the questions you didn’t think to ask • “use your words” • Post mortems, done well, should look like this
34. Reprocssing How would I solve X problem today? • Our design principles are heuristics too • Caution yourself, it works
35. Reprocssing Was X as successful as it appeared? • Retrospectives • Switching jobs makes this a bit harder • Keep in touch with coworkers
36. tl;dr Go forth and use heuristics well • Heuristics require upkeep • Bounds of Rationality • Be aware of the mental pitfalls • Reprocess • With collaboration
37. Questions? Thank you for attending • @Ramin_DK • github.com/rkhatibi/
38. Appendix 1. https://medium.com/@garywiz/five-things-old-programmers-need-to-remember-e78caf0b0973#.361en9sfi 2. https://en.wikipedia.org/wiki/Einstellung_effect 3. http://randymatusky.com/2015/02/25/the-hidden-learning-potential-behind-the-einstellung-effect/ 4. http://bigthink.com/endless-innovation/humans-are-the-worlds-best-pattern-recognition-machines-but-for-how-long 5. http://www.kegel.com/c10k.html 6. http://www.amazon.com/Drift-into-Failure-Sidney-Dekker/dp/1409422216 7. http://www.amazon.com/Thinking-Systems-Primer-Meadows-Donella-ebook/dp/B005VSRFEA/ 8. http://io9.gizmodo.com/5746904/how-pigeons-get-to-be-superstitious 9. http://psychcentral.com/lib/15-common-cognitive-distortions/ 10. https://en.wikipedia.org/wiki/Rubber_duck_debugging 11.