Possibilities of AI research conducted by AI SHIRO TAKAGI, Independent Researcher, @WCCI2024

[Aschenbrenner 2024] 2

[Aschenbrenner 2024] Automated AI research will trigger intelligence explosion!! 32

Self-improvement!! AI Research Acadmic Research Paper Novel architecture Novel algorithm ... AI Research High throughput rapid research execution without rest Countress copies Goal Significant impact 4

LLMs & LLM Agents ... Reasoning Planning Thinking Tool Use Scholarly Document Processing Coding Computer Operation ... LLMs automate basic operations necessary for research e.g. [Hou et al, 2023] e.g. [Huang et al, 2023, Huang et al, 2024, Mialon et al, 2023, ...] e.g. [Zhao et al, 2023, Wang 2023, ...] 5

Rescent works

Research Question, Research Problem, Hypothesis, Research Idea, ... Literature Based Discovery! 7

Research Question, Research Problem, Hypothesis [Baek et al, 2024] [Wang et al. 2023] [Gu & Krenn, 2024] Literature Based Discovery 8

Given detailed data and model information, it suggests data processing, model architecture, hyperparameters, and training log predictions!! [Zhang et al. 2023] 9

Task Output e.g. train models analyze data discover loss func write a paper ... Research Task Autonomatic Execution 10

[Lu et al, 2024] Instead of selecting from pre- prepared options, the LLM proposes a loss function algorithm code based solely on its own knowledge! 11

[Viswanathan et al, 2023] [Yang et al, 2024] Task description in text! Automatically fetch datasets and pre- trained models suitable for task completion, and automatically train the models! 12

Humans provide the tasks to be completed, the criteria for evaluating task performance, and the template files and workspace e.g. Kaggle tasks, BabyLM task (efficient LM), CLRS (Algorithm Prediction) Given these, the agent autonomously plans, thinks, and executes tasks using tools e.g. file editing, file reading, code execution, ... [Qian et al, 2023] 13

ChemCrow Coscientist [Bran et al. 2023] [Boiko et al. 2023] Generate a protocol to control a machine for the behavior required for an experiment 14

Observe a behavior of a neural network 1. Define the scope of the interpretation 2. Iteratively prune the model, removing unnecessary components 3. Automatically perform the 3rd step of mechanistic interpretability research! (Automatic detection of the circuit in the neural network) [Conmy et al, 2023] 15

Automatically break down, and perform the data science tasks!! [Hong et al. 2024] 16

[Ifargan et al. 2024] Automatically perform data analysis, and even generate a research paper! 17

Good research! Research Paper Little novelty ... Sound experiment! 18 Evaluation

LLMs can generate somewhat useful reviews! [Liang et al, 2023] Automatically review the research paper with LLMs! 19

challenges

Automating academic research: Producing new knowledge for a society Uncovering unknowns: questioning, hypothesizing, and veryfing [Takagi, 2023] Automating research tasks: data analysis, idea generation, literature survey ... 21

Objective Solution Implementation Experiment Plan Solution Idea Experiment Result Research Paper Experiment Implementation Research Problem End-to-end full automation？ 22

??? Complicated/Concrete Idea Simple/Abstract Idea Brain-inspired AI AI model Inspired by visual information processing ... ??? Papers Mathematical Model ??? Code Implementation ??? ??? [Fukushima 1980] 23

[Chollet 2019] Explicit and generally applicable universal rule beyond just patterns ??? ??? ??? 24

Feedback Propose Method Design Experiment Run Experiment Propose Method Design Experiment Run Experiment Feedback 25

Academic Literature Processing 　 ❌ survey, systematic review, critical reading, fining desired literature, ... Planning 　 ❌ long-horizon planning, feasibility in mind, detailed and self-contained plan, ... Thinking ❌ systematic thought, logical reasoning, spontaneous thought, critical thought, ... Mathematical Operation 　 ❌ mathmatical proof, theorem proposition, mathematical modeling, ... Engineering 　 ❌ developing product-ready software, repo-level task, perfect coding & debugging. debugging..... Computer Operation (Behaviour) 　 ❌ perfect computer operation, human-level browser operation, ... ... 26

Condluding remarks

I believe that the development of AI systems capable of conducting AI research is poised to become one of the most critical areas of focus in the coming years. While the potential is immense, significant challenges remain in realizing this concept. I am eager to connect with anyone passionate about this cutting-edge field - if this topic piques your interest, reach out to me! contact: [email protected] Join us and let's pioneer AI conducting AI research! 28

Wataru Kumagai @ OMRON SINIC X [email protected] 29

30

31 Thank you!

Aschenbrenner 2024 Situational Awareness - The Decade Ahead Zhao et al, 2023, A Survey of Large Language Models Wang et al, 2023, A Survey on Large Language Model based Autonomous Agents Huang et al, 2023, Towards Reasoning in Large Language Models: A Survey Mialon et al, 2023, Augmented Language Models: a Survey Hou et al, 2023, Large Language Models for Software Engineering: A Systematic Literature Review Huang 2t al, 2024, Understanding the planning of LLM agents: A survey Baek et al, 2024, ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models Wang et al, 2023, SCIMON : Scientific Inspiration Machines Optimized for Novelty Gu & Krenn, 2024, Generation and human-expert evaluation of interesting research ideas using knowledge graphs and large language model Zhang et al, 2023, AutoML-GPT: Automatic Machine Learning with GPT Lu et al, 2024, Discovering Preference Optimization Algorithms with and for Large Language Models Viswanathan et al, 2023, Prompt2Model: Generating Deployable Models from Natural Language Instructions Yang et al, 2024, AutoMMLab: Automatically Generating Deployable Models from Language Instructions for Computer Vision Tasks Qian et al, 2023, MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation Bran et al. 2023, Augmenting large language models with chemistry tools Boiko et al. 2023, Autonomous chemical research with large language models Con,y et al, 2023, Towards Automated Circuit Discovery for Mechanistic Interpretability Hong et al, 2024, Data Interpreter: An LLM Agent For Data Science Ilfargan et al, 2024, Autonomous LLM-Driven Research from Data to Human-Verifiable Research Papers Liang et al, 2023, Can large language models provide useful feedback on research papers? A large-scale empirical analysis Takagi, 2023, Speculative Exploration on the Concept of Artificial Agents Conducting Autonomous Research Fukushima, 1980, Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position Chollet, 2019, On the Measure of Intelligence 31