Abstractly reasoning – failing with an LLM (next steps for ARC AGI) PyDataLondon 2024-08 lightning talk @IanOzsvald – ianozsvald.com

Can LLMs reason? ARC AGI Abstract JSON “initial → target” Tried “don’t code, just reason” Llama3 70B pretty smart Llama3 8B writes code pretty well, sometimes Abstraction & Reasoning Challenge By [ian]@ianozsvald[.com] Ian Ozsvald

30% solutions pretty good! By [ian]@ianozsvald[.com] Ian Ozsvald It counts! Comments! Reasonable numpy! Correct substitution!

Convincing weirdness By [ian]@ianozsvald[.com] Ian Ozsvald

Big issue – it gets stuck on the same ideas Get LLM to read lots of failed model outputs, summarise, then maybe I could ask it to make new strategies? Notes → NotANumber.email newsletter Next steps By [ian]@ianozsvald[.com] Ian Ozsvald