Integrating LLMs into Database Systems Education

Large Language Models (LLMs) have sparked a drastic improvement in the ways computers can understand, process, and generate language. As LLM-based offerings become mainstream, we explore the incorporation of such LLMs into introductory or undergraduate database systems education. Students and instructors are both faced with the calculator dilemma: while the use of LLM-based tools may “solve” tasks such as assignments and exams, do they impede or accelerate the learning itself? We review deficiencies of using existing off-the-shelf tools for learning, and further articulate the differentiated needs of database systems students as opposed to trained data practitioners. Building on our exploration, we outline a vision that integrates LLMs into database education in a principled manner, keeping pedagogical best practices in mind. If implemented correctly, we posit that LLMs can drastically amplify the impact of existing instruction, minimizing costs and barriers towards learning database systems fundamentals.

Full paper at https://arnab.org/files/Nandi_DataEd24_Integrating_LLMs_into_Database_Systems_Education.pdf

Transcript

1. 1 Integrating LLMs into Database Systems Education Kishore Prakash, Shashwat Rao, Rayan Hamza, Jack Lukich, Vatsal Chaudhari, Arnab Nandi
2. LLM-based services are taking over everything 4
3. LLM-based services are taking over education 5
4. LLMs taking over education 6 • Initial Reaction: ban immediately! • “New Calculator”… “Plagiarism” • Detect and penalize • Understandable: Assignments and Exams • Synthesis and Essay Questions • Multiple Choice Questions: B+ • Unsupervised / Take-homes?
5. “Banning ChatGPT” is not an option 7 • Too late: Pervasive use, variants • Readying students for an AI-enabled future • Onus is on educators to discover how to integrate LLMs into educational infrastructure
6. Where does an LLM fit into the education landscape? 8
7. Class Roles: Where does an LLM fit in? 9 • Instructor • Teaching Assistant • Textbook • Teaching Tools / Software / Autograder • Tutor
8. Intuition behind “Tutor” 10 with infinite resources, what would we give every student? a personal tutor who assists the student in their learning journey
9. Our Vision: DB Tutor 11 • Provide the students with an LLM-powered chat-based interface that prioritizes personalized learning • Leverage opportunities that are unique to database systems • Building such a system will take some thought and iteration
10. Why LLMs are not the best fit 12 • LLMs are designed and trained to get to the right answer as quickly and efficiently as possible • Getting to the right answer without explanations can impede learning`
11. DB Tutor: Challenges 13 • Bias in Responses • Students’ over-reliance, critical thinking • Cheating and Misuse • Data Privacy and Security • Sensitivity to prompting
12. Challenge: Bias in Responses 14 • LLMs have an inherent bias issue • Training data bias • Recency bias • Demographics bias • Use in learning: amplified effects • Fix training data, or model output
13. Challenge: Over-Reliance, Critical Thinking 15 • High convenience = pervasive use • Long-term dependency • Loss of independent skills • Impedes deeper understanding • Loss of critical thinking (especially ability to notice LLM errors)
14. Challenge: Cheating and Misuse 16 • “Super Tool” for Misuse • Easy to generate human-sounding content • Essay questions, multiple choice • Are take-home assignments still an option? • Detection is an arms-race • Previous Disruptions • Web search, Wikipedia, Calculators
15. Envisioned System Architecture 17 LLM INFRASTRUCTURE (So0ware and data we will set up) ! Course Materials Syllabus, Slides, Tests " LLM Llama v2 or GPT4 via API Virtual Tutor Portal (What the student interacts with) # Learning Outcomes Report $ Chatbot % Database SQLite DBMS Virtual Tutor Engine (So9ware we will build in this research ac;vity) & Data Analysis Engine ' Prompt Engineering
16. Elements of a DB Tutor 18 • Can we go beyond “ChatGPT for Database Education?” • What are some gaps we can fill?
17. Elements of a DB Tutor 19 • Implicit Query Execution • Data Personalization • Learning Outcomes Report • Visual Step Throughs • Pop Quizzes
18. Implicit Query Execution: NL 2 SQL 20 • LLMs hallucinate; let’s pipe all generated code against a runtime (Google Bard) • DBTutor: Before queries are shared with student, execute it against a sandboxed DB • Generate Synthetic Data and Schema • Use results (or errors) to improve query and explanations • Prompt: “What are some possible errors to anticipate with this query?” SQLite Prompt ⚡ SQL Annotated SQL Result Student
19. Data Personalization 21 • Students are more engaged when examples are personalized • Use LLMs to generate sample data that they can relate to Travis Kelsey (American Football) Queries Taylor Swift (Music) Queries
20. Learning Outcomes Report 22 Case Studies and Applications Entity-Relationship (ER) Model ER-to-Relational Model Relational Algebra Relational Calculus Functional Dependencies and Normalization SQL Object Relational Databases Embedded SQL Graphical User Interfaces Indexing and Query Optimization XML Active Databases Concurrency and Transaction Management ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ Keep track and share what the student is learning; rewrite prompts to highlight gaps or assume knowledge
21. Takeaways • Standard LLMs are not designed for education and pose several challenges • Many unique integration opportunities in database systems education • LLM-powered “DB Tutor” that prioritizes student learning 23
22. 24 Thank you