Understanding GenAI: Translating MechInterp Findings for Everyday LLM Users

Transcript

1. Understanding GenAI: Translating MechInterp Findings for Everyday LLM Users Coco Zhang • 11 December 2025 Supervisor: Melissa Webster, Lecturer, MIT Sloan
2. 2 Coco Zhang Wellesley College ’28 CS and Math Major Linguistics Concentration ABOUT ME
3. AGENDA 3 ➢ Context ➢ Research Question ➢ Methodology ➢ Key Findings ➢ Implication & Future Directions ➢ Conclusion ➢ Acknowledgement & Bibliography
4. Mechanistic Interpretability Opening the Black Box of GenAI
5. 5 LLMs and the human brain both use neural networks of simple units that can’t be directly edited A neuron in human brain: A Multi-head Attention Network for LLMs:
6. Generative AI systems are grown more than they are built — their internal mechanisms are “emergent” rather than directly designed. — Christopher Olah, Interpretability Research Lead at Anthropic
7. “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big ?” — Bender, et al. (2021) Probabilistic Imitator 7 We Know Little about LLMs Internal Working “Large Language Models as General Pattern Machines” —Mirchandani, et al. (2023) Pattern Recognizer
8. 8 Mechanistic Interpretability Works Like Brain Imaging to Understand GenAI’s “Thinking” Brain Imaging from FMRI scan MechInterp Methods Diagram: whether the model’s internal work faithful to the natural language results we get.
9. AGENDA 9 ➢ Context ➢ Research Question ➢ Methodology ➢ Key Findings ➢ Implication & Future Directions ➢ Conclusion ➢ Acknowledgement & Bibliography
10. How can insights from mechanistic interpretability research on large language models be translated into high-level mental models on how LLM works, so non-expert users can use LLMs more effectively and better evaluate their outputs?
11. AGENDA 11 ➢ Context ➢ Research Question ➢ Methodology ➢ Key Findings ➢ Implication & Future Directions ➢ Conclusion ➢ Acknowledgement & Bibliography
12. 12 Primary Research: Combined expert interviews with insights from engaged GenAI users. Secondary Research with Literature Review: Read blogs and papers on the progress of Mechanistic Interpretability research. Top two sources: • Neel Nanda’s guides • Anthropic’s Circuit Analysis Thread GenAI Assistance in Research (Early-stage use) • Gather and cluster developments in mechanistic interpretability • Critique and tighten my research question
13. AGENDA 13 ➢ Context ➢ Research Question ➢ Methodology ➢ Key Findings ➢ Implication & Future Directions ➢ Conclusion ➢ Acknowledgement & Bibliography
14. 14 Finding 1: LLMs shows planning-like behaviors when generating output Anthropic Research on Claude 3.5 Haiku Task: Writing a poem We can plan the rhyming word at the end of each line and fill in the rest. Figure: High-level overview of how Haiku plans candidate ending words of the next line. (Lindsey, et al.)
15. “Explaining Interp[retability] findings involves abstractions. Human biases come in when evaluating whether [the abstraction] makes sense.” — MechInterp Researcher
16. “The problem is what [researchers] might miss out on because our priors are too strong.” — Head of Product at MechInterp Start-up
17. 17 What are some Planning-like Behaviours in LLMs: • LLMs generate content non- linearly, beyond the word-by-word model. • Tree-of-Thoughts: Models can explore multiple candidate solution paths in parallel like a search tree. • LLMs can decompose goals such as “make breakfast” into action sequences, then fill in content. Limitations: • Observations are task-specific, and less prominent on longer prompts and reasonings. • Difficulty to generalise to model behaviours • Findings focus on targeted subset in the neural network, which miss out on other contributions. Finding 1: LLMs shows planning-like behaviors when generating output
18. 18 Research on Llama3-8B How do we perform simple math? Mathematical Intuition Q: What is 105*979? Thinking: Anything multiplied by 5 result in a number ending with 5 or 0. 7 is an odd number so the result ends with 5. Finding 2: Large Language Models use Rules-of- thumb for Arithmetic 1.LLMs just memorize examples. 2.LLMs learned a full general-purpose arithmetic algorithm. Naïve Hypotheses : Subtraction problem 226 − 68 should lead to results between 150 and 180. In Practice:
19. 19 Methodology: • Looks for a subset of the model (a circuit) that handles arithmetic logic and examine its functionality with causal analysis • Tracks how the group of rule- of-thumbs can be developed through training Finding 2: Large Language Models use Rules-of- thumb for Arithmetic Limitations for Application: • Generalization: Do these heuristics scale to large numbers or multi-step problems? • Safety / reliability for high-stakes uses (finance or engineering) • Training design: Could we encourage more algorithm-like behaviours, or is heuristic patchwork the better in large neural nets? Research on Llama3-8B
20. 20 Finding 3: Large Language Models may have Introspective Awareness What is introspection? Ability to inspect and reason about one’s own thoughts Why did that bother me so much? Thinking Model’s reasoning activity has discrepancy with its internal working. Anthropic Research on Claude Opus 4 and 4.1
21. “Something we keep in mind is whether models are aware they’re being studied.” — MechInterp Researcher
22. 22 Finding 3: Large Language Models may have Introspective Awareness Anthropic Research on Claude Opus 4 and 4.1 Limitations: • Can we generalise this? Opus 4.1 and 4 display the strongest introspective mechanism but other models shows weaker ones. • Imitation or True Introspection? Models are trained on data that include demonstrations of introspection Methodology: • Inspection: whether the model can “read” and even “control” their internal representations • Injects thoughts into a model, eg. “bread” • Tests whether models can distinguish injected thoughts and prompt inputs
23. AGENDA 23 ➢ Context ➢ Research Question ➢ Methodology ➢ Key Findings ➢ User Implications ➢ Future Directions ➢ Conclusion ➢ Acknowledgement & Bibliography
24. “[Working with GenAI] is not a technology problem but a human responsibility.” — Rajiv, Managing Director of MIT Sloan STS
25. 25 Implications for understanding and working with LLMs Calibrate task- specific trust boundaries Check the model’s knowledge scope at the start Verify the chain- of-thought of reasoning models Breakdown multi-step tasks to help models’ planning Critically combine model suggestion with human review Expect LLMs to apply rules-of- thumb and check accordingly
26. AGENDA 26 ➢ Context ➢ Research Question ➢ Methodology ➢ Key Findings ➢ User Implications ➢ Future Directions ➢ Conclusion ➢ Acknowledgement & Bibliography
27. Technical Research: • Scalability: Develop methods for submodule-level understanding • Causality: Understand causal relations between circuits as ground truth • Availability: Build tools that can make interpretability analysis available for users through API Non-technical Research: • Reasoning Style: Experiment how users develop trust with different GenAI reasoning styles • Explainability: Build and improve metaphors that help non-technical users understand GenAI 27 Future Research Directions
28. AGENDA 28 ➢ Context ➢ Research Question ➢ Methodology ➢ Key Findings ➢ Implication & Future Directions ➢ Conclusion ➢ Acknowledgement & Bibliography
29. 29 LLMs May Plan Ahead LLMs Use Rules-of-Thumbs MechInterp helps researchers open the Black Box of LLMs and informs GenAI users when and how to use GenAI effectively. LLMs Show Proto-Introspective Behaviours
30. AGENDA 30 ➢ Context ➢ Research Question ➢ Methodology ➢ Key Findings ➢ Implication & Future Directions ➢ Conclusion ➢ Acknowledgement & Bibliography
31. 31 Acknowledgements • Professor Max Tegmark: MIT Mechanistic Interpretability Conference 2023 • Multimodal Automated Interpretability Agent (MAIA), MIT CSAIL • Neel Nanda’s detailed MechInterp guide • Dario Amodei’s Blog on ‘the Urgency of Interpretability” • Myra Deng and her team’s work at Goodfire.ai • My advisor, Melissa • My teammates, Deeksha, Emery, and Taylor • STS at MIT Sloan and LTS at Wellesley College • My roommate Diana T. and friends Natalie N., Sophia S., and Helen Y.
32. 32 Recommended Resources Discussions and Forums: • Neel Nanda’s MechInterp guides. URL: https://www.neelnanda.io/mechanistic- interpretability • Interpretability (ML & AI) Section on AI Alignment Forum. URL: https://www.alignmentforum.org/w/interpretability-ml-and-ai Key Breakthroughs in MechInterp • Trenton Bricken*, Adly Templeton, et al. “Towards Monosemanticity: Decomposing Language Models With Dictionary Learning” In Anthropic: Transformer Circuits Thread, 2023. URL: https://transformer- circuits.pub/2023/monosemantic-features/index.html • Adly Templeton, Tom Conerly, et al. “Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet” In Anthropic: Transformer Circuits Thread, 2024. URL: https://transformer-circuits.pub/2024/scaling- monosemanticity/
33. 33 Bibliography 1. Emily M. Bender, Timnit Gebru, et al. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? . In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT ‘21), 2021. URL: https://doi.org/10.1145/3442188.3445922 2. Jack Lindsey, Wes Gurnee, et al. On the Biology of a Large Language Model. In Anthropic: Transformer Circuits Thread, 2025. URL: https://transformer- circuits.pub/2025/attribution-graphs/biology.html#dives-poems 3. Yaniv Nikankin, Anja Reusch, Aaron Mueller, and Yonatan Belinkov. Arithmetic without Algorithms: Language Models Solve Math with a Bag of Heuristics. On International Conference on Learning Representations, 2025. URL: https://arxiv.org/abs/2410.21272 4. Jack Lindsey, Emergent Introspective Awareness in Large Language Models. In Anthropic: Transformer Circuits Thread, 2025. URL: https://transformer- circuits.pub/2025/introspection/index.html
34. Understanding GenAI: Translating MechInterp into Mental Models for Everyday LLM Users Coco Zhang • 11 December 2025 Wellesley College ‘28 Supervisor: Melissa Webster, Lecture, MIT Sloan