Interviewing tactics for a post-LLM world

Presentation given at TechLead 11th of June, Amsterdam

Transcript

1. Dünya Kırkalı Interviewing tactics for a post-LLM world
2. Interviewing like it’s 2022
3. Interviewing like it’s 2022 Five stages of grief • Denial • Anger • Bargaining • Depression • Acceptance
4. Five stages of grief Denial • Ban AI • Remove take-home • Live-only obsession Photo by Saif71.com on Unsplash
5. Five stages of grief Anger • Hostile process • Gotchas • Purity tests Photo by Nsey Benajah on Unsplash
6. Five stages of grief Bargaining • Awkward constraints • Inconsistent rules • Unclear expectations Photo by charlesdeluvio on Unsplash
7. Five stages of grief Depression • Endless process changes • Low confidence • Stage bloat Photo by charlesdeluvio on Unsplash
8. Five stages of grief Acceptance • Clear process • Encouraged LLM usage • Focus on reasoning over output Photo by Darius Bashar on Unsplash
9. 🌍 • Team Leader at Remote.com • Author of the • Engineering Manager’s Compass • Blogger on • Incremental Forgetting • ❤ Hiring Dünya Kırkalı
10. Why?
11. Laszlo Bock “Great people are a company's most important asset. Hiring is the most important thing you do”
12. Let’s take a step back
13. ACME Corp. • What will they be doing • Who will they work with • Which KPIs • The level of support • The level of autonomy AI Native Engineer
14. Let’s take a step back What & How • Specific skill • Specific personality • Generalist vs Specialist • Past experiences • Future potential • Gap in team • Take home • Open vs closed questions • N stages • Leetcode • Brain teasers • IQ Tests
15. KSAO
16. Knowledge Skills Abilities Other
17. Knowledge Examples • LLM Capabilities • Strengths: drafting, transformation, summarisation • Weaknesses: hallucinations, non-determinism, stale knowledge • AI Risk/Governance • Privacy, prompt injection, intellectual property
18. Skills Examples • Task Decomposition • Break ambiguous work into steps • Design prompts • Build skills • Verification of AI Output • Fact-check • Design tests
19. Abilities Examples • Curiosity • Experiment • Learning Agility • Adapt to new models/tools • Update way-of-working as the ecosystem changes
20. Other Examples • Intellectual humility • Comfortably saying “I don’t know” • Correcting model/user output • Ownership with judgment • Escalate high-risk decisions • Don’t over automate
21. Let’s revamp
22. Let’s revamp Go deep Choose one thing you’ve built and walk me through it • It’s not the results, it’s how we got there • What did they learn from the experience • What would they do differently • Generalist or Specialist • Critical thinking • Domain understanding
23. Let’s revamp Reading code Help your team mate understand what this large code base does • Ambiguity • Legacy • Documentation • LLM Usage • Context Management
24. Let’s revamp Reviewing Code Review this change request and explain what feedback you leave • Focus • Feedback • Communication • Values
25. Let’s revamp Writing an RFC Write on RFC on this topic and present it to us • Systems Thinking • Structured Reasoning • Trade-Off Analysis • Communication Clarity • Collaboration Mindset
26. Let’s revamp Systems Design Design a [ x ] system and explain to us what you’ve come up with • Have them design a system at home • Invite over for a deep-dive • Tradeoffs • Chaos • Communication • Feedback
27. Tips
28. Tips 3 Good vs 10 Mediocre • Quality over quantity • Depth over Breadth Photo by Brett Jordan on Unsplash
29. Tips Not Live • Every second is precious • What if the systems are down • Ask async - review sync Photo by Lucas Alexander on Unsplash
30. Tips Encourage LLM usage • Take home • Share sessions • Process > Output • Tradeoffs Photo by Harrison Broadbent on Unsplash
31. Tips A day at the races • Have them work along your engineers • Have them contribute to OSS • Pay them • Have them join lunch • Collaboration is essential Photo by Gene Devine on Unsplash
32. Tips Extremists • Dangerous • Present dilemma • Tradeoffs • Beliefs Photo by Web Donut on Unsplash
33. Tips Scorecards • Standardize • Close • Disclose at the end • Don’t discuss, rather update Photo by Rong on Unsplash
34. Tips Test your stages • Measure • Time • Accuracy • Try out with engineers • Try out with non-engineers • Using an LLM • A/B Test Photo by Akram Huseyn on Unsplash
35. Tips 10x • Whatever you think you’ll ask, 10x it • Issues • Users • LoC • Pull Requests • Code Complexity • Number of Services, Photo by Miguel Ángel Padriñán Alba on Unsplash
36. Recap
37. Recap • Write down the job description • Define KSAO • Prepare questions • Test internally • Run experiments • Measure success
38. Closing
39. Closing Acknowledgements • 🔥 Idea by LeadDev • 💡 Inspiration by Christopher Redmond • 💡 Inspiration by Dan Cook • 🫶 Support by Maxim Schepelin
40. https://schepelin.com/go/book
41. Closing Resources • Scalar Interviews by Dunya Kirkali • How to combat bias in the hiring process by Maxim Schepelin • Software engineering at the tipping point by Adam Bender • Work Rules! by Laszlo Bock • Make Something Wonderful by Steve Jobs • Who: The A Method for Hiring by Geoff Smart and Randy Street
42. @dunyakirkali Questions