cloudisland.nz/@coderanger Kubernetes and Python SRE/Platform for Geomagical Labs, part of IKEA We do CV/AR for the home PyCon AU 2026 – Noah Kantrowitz – @[email protected] 2 Hi there, I'm Noah Kantrowitz. I'm an SRE at Geomagical Labs. We do computer vision and augmented reality stuff for IKEA. But I'm not here to talk about that.
I will make mistakes again This is okay This must be okay PyCon AU 2026 – Noah Kantrowitz – @[email protected] 3 I am a squishy human. I have made mistakes in the past. I am certain to make mistakes again in the future. I like to make plans for the future, and if those plans are going to survive contact with reality, I need to make sure they account for future mistakes.
AU 2026 – Noah Kantrowitz – @[email protected] 4 This is a talk about building resilient systems. That can mean a lot of things. We had a lovely track on Thursday dedicated to technical resilience, building systems that can survive all manner of computer failures. And yesterday we had the security track, all about how to be resilient in the face of bad people out to do bad things. I want to talk about a different kind of resilience, about humans and teams. At time of writing, every team I've been on has been staffed by humans working together to build a thing, and so those humans make mistakes.
parts This is not (just) about "AI" PyCon AU 2026 – Noah Kantrowitz – @[email protected] 5 One of the major advantages to humans teaming up on a challenge is that we can design systems that are better than any one person. But as the software engineering landscape is reshaping itself yet again, the challenges we need to build for are shifting too. Elephant in the room, AI tools are a big change for a lot of teams, mine included. And in large part this talk is a scream into the void from deep in my soul about the stressors that AI is creating. But this isn't really a talk about AI and the drive for teams to work faster and faster stretches back far before that, probably forever. This is about humans and the patterns in how humans collaborate, for better and for worse.
3. What Is The Human Cost? PyCon AU 2026 – Noah Kantrowitz – @[email protected] 6 Our roadmap. First we'll talk about how humans can fail at things. Then strategies and techniques for improving those. And finally how these struggles can affect people and what to watch out for.
lenses PyCon AU 2026 – Noah Kantrowitz – @[email protected] 7 While every human is unique, the ways our minds break down are consistent in the broad strokes. I'm going to list off the most important common failure modes but these are lenses to view the world through. The goal here is not to be able to label every possible bad time but to build a framework we can plan around.
The magic seven words "You are already doing the right thing" PyCon AU 2026 – Noah Kantrowitz – @[email protected] 8 Confirmation bias is a big topic but we have to dive in somewhere. We like to think of ourselves as objective observers of the world around us. But of course we also all know that isn't true, we pick and choose on many layers and we are always primed to be more accepting of information that confirms our current understandings and assumptions. Seven of the most powerful words in any situation are "you are already doing the right thing".
good at this do the right thing I must be doing the right thing PyCon AU 2026 – Noah Kantrowitz – @[email protected] 9 To phrase it as a syllogism, confirmation bias often roots in our ego, our natural desire to see ourselves as capable and powerful. Then we add the industry-wide belief that smart people do good things. And we end up with the logical conclusion that whatever we are doing must be correct.
"It's standard practice" Social Cohesion and Conformity "I don't want to be one of those reviewers" "We can always fix it later" PyCon AU 2026 – Noah Kantrowitz – @[email protected] 10 Confirmation bias can also manifest as deference to authority, over-prioritizing things seen as accepted wisdom or hierarchical thinking. In the same way as we all try to center our own skills and agency, we can also be biased towards fitting in with our teams, giving the benefit of the doubt. Both forms highlight how these biases aren't always in the wrong, they are cognitive shortcuts and we all use them because they very often save time or brain cycles in complex situations. But they also create repeating failure modes when those shortcuts produce bad output.
– Noah Kantrowitz – @[email protected] 11 Moving on to a few classics, inattentional blindness is not quite a bias but functions similarly to one. It's another mental shortcut, eliding data that was not deemed important at the time. But when we think back on those memories, it feels like we have the full picture. Until someone points out the man in the gorilla suit that you completely missed. As applied to tech, the same thing can happen where we remember observing a situation, maybe a requirements meeting or a code review session, but can miss huge amounts and not notice.
bug" PyCon AU 2026 – Noah Kantrowitz – @[email protected] 12 Recency bias is assigning outsized importance to temporally nearby events, even when claiming to be using a more objective measurement. How often is this week's quote low priority ticket going to usurp last week's high priority?
every evening" x += 1 x += 2 x =+ 3 PyCon AU 2026 – Noah Kantrowitz – @[email protected] 13 Similar to inattentional blindness, repetition blindness is the tendency of humans to become desensitized to repeated stimuli. Reading code dials this up to 11, as it's much more repetitive than normal prose.
bad science PyCon AU 2026 – Noah Kantrowitz – @[email protected] 14 And then we get to the big one. Overconfidence bias pervades our industry. Software people have egos, especially the ones who look like me. We, collectively, like to assume that we are better than average. That we've built "the best team". And that through our excellence, we can beat any odds. But that's not how averages work. Asking engineers to do time estimates is such a joke I don't even need a punch line, and yet we all keep doing it anyway. How many of you have shown someone clear evidence that a project is on the wrong track, only to be told "we just need to double down and work harder"? How many have heard that a lot more often in the last year or two? [pause] And a quick aside before I move on, dunning kruger is bad science, please stop quoting it when you mean generalized overconfidence.
– @[email protected] 15 Enough problems, let's talk solutions. We don't have to do this alone and we have a few thousand years of experience to learn from.
when not in the moment Following when stressed PyCon AU 2026 – Noah Kantrowitz – @[email protected] 16 If you take one thing away from this talk, make more checklists. In days long past I did a lot of SCUBA diving and there is a saying hammered into every fresh student, plan your dive and dive your plan. When you know you are going into a stressful situation that requires a lot of judgement, you want to make sure you aren't burning that energy on parts which don't absolutely need it. So you plan out ahead of time, making a set of steps that require a lot less mental effort to go through.
AU 2026 – Noah Kantrowitz – @[email protected] 17 Checklists can help with a lot of different situations, but the three most common I make are "the steps that a code reviewer should take", "the steps for a code deployment", and "the steps for initial outage response".
2. Repetitive tasks 3. High stress, usually time sensitive 4. High impact of errors PyCon AU 2026 – Noah Kantrowitz – @[email protected] 18 More generally we can define leading indicators that a checklist would help. First it's a task that happens a lot. Second, it's very repetitive. Going back to our biases, we know about repetition blindness so let's plan for it. Third, the task will be done in a high stress environment, often it's time sensitive so there is inherent stress from the pressure to move quickly. And fourth, making a mistake will have a large impact, especially when mistakes will be hard to reverse. The more of these you have, the more a checklist will help.
– @[email protected] 19 For a checklist to work, the most important thing is that it has to function as a checklist on a literal level. A human needs to be able to follow the steps, one at a time and in the order presented. There should not be a large number of steps which are expected to be skipped, and absolutely no steps that are impossible. Don't write a checklist with "never write a bug" as a step. If a checklist isn't followable, people will ignore it.
2026 – Noah Kantrowitz – @[email protected] 20 Then you need to practice them. I said before that checklists are a good fit for high-pressure situations. No one wants to learn a new thing while prod is burning, go through them ahead of time so you have the muscle memory as much as possible. Do pairing sessions, do disaster simulations, whatever you need to make sure everyone on the team can do all the steps.
Be Blameless PyCon AU 2026 – Noah Kantrowitz – @[email protected] 21 And finally if someone follows a checklist and gets a bad result, that is the fault of the system, not of the person. I think more or less everything should be blameless, but it's worth underscoring here. When building a team process, we have to assume good faith and that means everyone will be taking whatever action they think is the best one to do. If they pick wrong, that's not a personal failing, it's a sign they didn't have enough information to make a good choice. Checklists fill that in, "here is the information you need (or where to find it) and here is the correct next action".
PyCon AU 2026 – Noah Kantrowitz – @[email protected] 22 As we continue down the path of reducing the impact of human error, a strong tool is to rely on objective measurements. Any place we can use objective numbers is one less place we have to decide in the moment what data to look at.
2026 – Noah Kantrowitz – @[email protected] 24 Except that's not actually true. While tests are a great step towards objective measurement, how do we know we are testing the right things? Assert true is a valid and complete unit test, it will pass, but we have learned nothing about the state of our system.
PyCon AU 2026 – Noah Kantrowitz – @[email protected] 25 A lot of you probably already said to yourself "well that's why we check line coverage during testing". And that can indeed help, we can see a report of which lines or expressions were hit and from that build a model of "are we testing everything we need to test?". Deciding what we need to test creates a new subjective layer though, what do we need to test? Some teams decide to solve this by mandating 100% coverage, every line must be hit in a test. I find this leads to incredibly brittle systems, the commitment to objectivity has become a number chasing mini-game rather than a guiding force.
can self-host PyCon AU 2026 – Noah Kantrowitz – @[email protected] 26 Making a direct recommendation, I like to focus on patch coverage in the context of a single merge request. There's a bunch of tools for this, codecov by Sentry is great and not too complex to self-host.
isn't covered" PyCon AU 2026 – Noah Kantrowitz – @[email protected] 27 If I can give you one vibe check for tests and code coverage, "you didn't write enough tests" is a less helpful mood and "have you considered how to test this line" is more helpful.
PyCon AU 2026 – Noah Kantrowitz – @[email protected] 28 Static typing is another mostlyobjective tool we can lean on. It's only as useful as the annotations make it, but if used thoroughly it can catch many errors that are frustratingly difficult for a human to notice. Let the machine do what it does best.
drink They order -1 They order None They order "one" They order [1, 2, 3] PyCon AU 2026 – Noah Kantrowitz – @[email protected] 29 A classic programmer joke is a QA engineer walks into a bar, they order one, ten, zero, one million, negative one, and so on. Static typing solves this by shrinking the possible input space to only the types allowed. We don't have to write a million unit tests to characterize the behavior of every function when we pass in unexpected types, because the type checker makes sure that can never happen. Python is still early in its static typing journey, for example we can't actually specify "positive integer" as a type yet, but every little bit helps.
it's Black PyCon AU 2026 – Noah Kantrowitz – @[email protected] 30 Code linters and formatters provide not just objectivity in code styling, but also automation. If I never have another argument about quote types or spaces around operators it will still be too soon. This continues our running pattern of off-loading the tasks the machines can do well and let the squishy humans focus on the rest.
– Noah Kantrowitz – @[email protected] 31 Consistency isn't just for fun, it's a key piece towards making a codebase maintainable by a team rather than matching the bugbears of a single author.
blank=True) # vs. CharField(blank=True, null=True) PyCon AU 2026 – Noah Kantrowitz – @[email protected] 32 Always be on the lookout for cultural rules you can promote into a plugin somewhere. As with all of these, not every check is easy to do this for, but when you can shift mental load from humans to machines it's often a good trade. And for the record, null goes before blank.
AI will not save you PyCon AU 2026 – Noah Kantrowitz – @[email protected] 33 Even after we've automated everything we can apply objective analysis to, we're still going to need some human judgement to decide if we've written the right thing or not. Almost always there is going to be a first human who can provide some judgement, the person who wrote the change. If you do pair programming maybe you've got two pairs of eyes at that phase. But industry standard workflows have a dedicated review step where one or more other people look at the change and provide feedback on correctness and completeness.
"Does this SQL query look correct before I run it?" "What does rm -rf * mean?" PyCon AU 2026 – Noah Kantrowitz – @[email protected] 34 Code review is the most common place we check each others' work, but the same principles apply in many cases. Many of you flew here and heard some variant of "arm doors and cross check" shortly before takeoff, every industry where safety is critical has some form of multi person review. I'm very glad my code isn't as important as an airplane door seal but I can still learn from their solutions.
consensus Specialist consultation PyCon AU 2026 – Noah Kantrowitz – @[email protected] 35 To know how to build a good review process for code specifically we need to look at what we want to get out of it. I usually divide things into four pillars. The most obvious and the place most teams stop at is finding and fixing errors in the code before it goes further towards users. But no less important, team members can learn a lot from reviewing code in a system they haven't worked in as much or in a tech stack they aren't familiar with. They might not find as many bugs at first, but never underestimate the value of a "I don't understand this, could you explain it to me?" in finding obvious-inretrospect problems. Code review can also serve as a place to discuss architectural consistency and pull in subject matter experts, though doing so late in the process can lead to frustration as work has to be changed or discarded.
consuming Tiring PyCon AU 2026 – Noah Kantrowitz – @[email protected] 36 I said the main reason most teams do code review is to reduce defects and sadly it's kind of terrible for that. Every one of us has reviewed a change that later on was found to have some kind of airquotes obvious problem, but it sailed right through the process. It also creates friction points in a team as people can feel their reviews aren't happening quickly enough, or that they are otherwise having their agency reduced for no benefit. And it's draining, we'll talk more about the mental burden of review in a moment but it's real and hard to work around.
self-contained problems • "You used the wrong variable name" • "This library isn't thread safe" • "We have a utility function for this already" Macro - Looking at overall structure • "How does this handle authentication?" • "What will the performance of this look like under load?" • "This doesn't match our usual pattern for this kind of API" PyCon AU 2026 – Noah Kantrowitz – @[email protected] 37 Getting on my soapbox, I think part of the problem is there's two very different tasks we lump together as code review. Micro review is examining single lines or a small chunk on its own. You might have some context from knowing how the project is generally laid out, but overall it's as low-context as possible. This is where you get a lot of your usual nit picking, bikeshedding, and best-practice-y reviews. In general micro reviews can work okay to find narrowly scoped issues, but a lot gets missed if the problem spans more than a dozen lines. Micro reviews are also more often things which can be converted to linter rules, but not always. Macro review is the big picture. Looking at the diff as a whole and answering large scale questions. This is where we are looking for overall archiectural elegance or extensibility for the future. I don't think macro code reviews work very well.
AU 2026 – Noah Kantrowitz – @[email protected] 38 So we have solutions for our nits, things like not following a function naming convention. Between objective tools and micro code reviews we can often catch those defects. And maybe we can sometimes catch poorly laid out code, but often those problems are only obvious in retrospect. And we have pretty much no answer for missing things, like noticing that a diff completely forgot to include permissions checks. Checklists can help but a checklist that enumerates every possible thing a reviewer should look for would be too long to be helpful, so we have to just roll the dice a lot of the time.
on "vibes" as you go 2. Use the diff to reconstruct the mental state of the author 3. Imagine what your mental state would be when doing the same work 4. Compare that extrapolated mental state to your assumptions 5. If there are mismatches, highlight them for further reading PyCon AU 2026 – Noah Kantrowitz – @[email protected] 39 And to get on an even soapier box, the hot take that made me want to write this talk: I think we are collectively deluding ourselves into thinking that macro-scale code reviews work, that we are simply The Best Engineers and so we can see all the problems. I think instinctively most overwhelmed engineers adopt a process of not really reviewing the code itself, but reviewing the mental state that the author had at the time of writing, with the code just as a window into that mental state.
this? PyCon AU 2026 – Noah Kantrowitz – @[email protected] 40 To explain what I mean by that, imagine someone on your team writes a new API feature that has 3 very similar endpoints. You probably check the first one in great detail, the second less so, and the third you just skim. This works okay with humans because if they did one correctly, there is a good chance their mental state was correct and that correlates strongly with doing the other two correctly as well. This is not a perfect heuristic, but it does work okay in a lot of situations.
not have a "mental" Or any states thereof PyCon AU 2026 – Noah Kantrowitz – @[email protected] 41 I know I said this wasn't really a talk about AI but I do want to highlight one of my biggest concerns about heavily automated code workflows, this kind of vibe based review becomes dangerous. Piles of linear algebra do not have a mental, let alone a mental state. Because of the stochastic nature, they are equally likely to produce incorrect code anywhere along the process so the first view being perfect tells you very little about the other two. This means reviewing AI-generated code is much much harder than the equivalent volume from a human, the same shortcuts no longer apply.
Cross-training, education, discussion Rubber ducks can still find bugs Just don't assume you'll find all the problems PyCon AU 2026 – Noah Kantrowitz – @[email protected] 42 Before I sound too negative, code review is good and you should keep doing it. My problems with review practices are when you put too much faith in them, and thus too much pressure on people to code review even harder.
they do the wrong thing" PyCon AU 2026 – Noah Kantrowitz – @[email protected] 43 If someone comes to you with a new tool, maybe it's an AI agent, maybe it's boilerplate generator, whatever, they come to you and say "these new tools are fine because we'll notice if they do the wrong thing and fix it", they are proposing a very dangerous game. As we've discussed, humans can fail in a lot of ways and even our best efforts to work around these often boil down to "never ever make a mistake" and that just doesn't work.
@[email protected] 44 Hypervigilance is a defense mechanism, a state of constant awareness of everything around you that could go wrong. I find myself more and more being locked in this kind of mode at work, and I think I'm not the only one. As we are all expected to move faster and faster, it very often feels like the expectations on me are exactly that "never make a mistake" plan, the plan we know doesn't work. Hypervigilance leads directly to anxiety and exhaustion which only tightens the spiral downwards. And at the bottom is burnout.
and How to Get Up Again Tackling Burnout and Strategies for Self Care Jackson Fairchild – PyCon AU 2016 PyCon AU 2026 – Noah Kantrowitz – @[email protected] 45 I am not a mental health professional but many have written or spoken at length about burnout and how it plagues our industry like few others. I would like to highlight a talk from PyCon AU 2016 as a great place to start if any of these words about hypervigilance are landing extra hard for you. Just know you are not alone, and this isn't your fault. We've somehow herded our entire industry on to a path that pushes every possible stress button in all of us.
if you are a D&D fan PyCon AU 2026 – Noah Kantrowitz – @[email protected] 46 Computers have RAM, you can only load so many programs into memory at once. Cognitive load is similar, each person can only have so many background tasks before it becomes overwhelming. The limits vary with the person and the situation, it's not a perfect metaphor, but map out where you want to invest your cognitive load and think about when it's too much and you need to move a cognitive task to a peer.
all function names consistent Have I forgotten any code reviews PyCon AU 2026 – Noah Kantrowitz – @[email protected] 47 Background cognitive loops are mental tasks that require constant or near constant awareness. Things like ensuring all code related to TLS certs is safe. Or social tasks like making sure that you haven't dropped an important code review and someone is now waiting on you. The more senior you get in your career, the more of these tasks you are expected to juggle simultaneously. But they don't get much easier over time.
actually helps) Shift to colleagues PyCon AU 2026 – Noah Kantrowitz – @[email protected] 48 Being overloaded leads to increased error rates and general misery. Looking back at the first half of this talk, many of these loops can be moved into support tools. Don't make people remember PRs, have a Slack bot post them once or twice a day. Boom, cognitive load moved over to a machine. Many of these are more difficult to automate though. And even though I've said unkind things about AI, if you think carefully about the balance of stressors and find that a Claude agent handling a low-priority task gets you net more bandwidth to devote to the most important things for now, that's great, do it. And also talk to your team, they are here to support you as much as you support them and shifting tasks around to match the ebb and flow of availability can help a lot.
cognitive biases Supplement human decision making with checklists Look for objective, deterministic guardrails Build review processes that assume failures Cognitive offload, but use your powers only for good Fix processes, not people PyCon AU 2026 – Noah Kantrowitz – @[email protected] 49 Phew, that was a lot. What did we cover today? We went through some common failure modes of human brains. Then how to supplement squishy humans with checklists. We talked about objective support tools and how to fit them into a development workflow, and how to set up review workflows that are resilient to human errors. And finally we talked about cognitive load, and how to shift load before it shifts you.
– @[email protected] As a note to leave on, treat yourself and your team with kindness and compassion. Because you can be bad at things, and that should be okay. 50
of technical resilience but not this talk • Human resilience, building systems that can survive human fallibility • Also not a talk about blue team, malicious humans are a different kind of threat • We all make mistakes, how can we build systems which are better than any one us individually • Sections: • How we fail • Where can we improve • What is the human cost Ways humans are fallible • Confirmation bias • Selective perception PyCon AU 2026 – Noah Kantrowitz – @[email protected] 52
SiteGround - Reliable hosting with speed, security, and support you can count on.
coderanger
nikkei_engineer_recruiting
riyaamemiya
lehupa
masahirokawahara
cscengineer
sansantech
olivia_0707
hiroyaterui
layerx
zozotech
kishima
yokawasa
gr2m
uxyall
tmiket
popppiees
signer
tammyeverts
katarinadahlin
smashingmag
mfonobong
lauravandoore
eileencodes
oripsolob
![Unit Tests! PyCon AU 2026 – Noah Kantrowitz – @[email protected]](https://files.speakerdeck.com/presentations/2b8172cd2f6e48aaafcaa20da8192e3a/slide_22.jpg){kind=link}
![Add Your Own [project.entry-points."flake8.extension"] X = "flake8_example:ExamplePlugin" Have opinions! CharField(null=True,](https://files.speakerdeck.com/presentations/2b8172cd2f6e48aaafcaa20da8192e3a/slide_31.jpg){kind=link}
![Questions? PyCon AU 2026 – Noah Kantrowitz – @[email protected] 51](https://files.speakerdeck.com/presentations/2b8172cd2f6e48aaafcaa20da8192e3a/slide_50.jpg){kind=link}
