Lessons learned building a GenAI powered app

Transcript

1. Proprietary + Confidential Marc Cohen Developer Advocate at Google mco@google.com Lessons learned building a GenAI powered app Mete Atamel Developer Advocate at Google @meteatamel atamel.dev speakerdeck.com/meteatamel
2. Proprietary + Confidential Before GenAI GenAI arrives Architecture After GenAI Lessons Learned 01 02 03 04 05 Agenda
3. Before GenAI
4. Every great invention started out as someone’s weekend project. This is gonna be huge!
5. August 2016
6. Proprietary + Confidential Demo: Initial app with Open Trivia DB https://opentdb.com/api_config.php
7. Initial problems • Limited list of topics • Limited questions and answers • Limited format: multiple choice with 4 answers • English only • No images • Expanding quiz content is difficult
8. GenAI arrives
9. March 2023 Is it possible to have a more dynamic quiz app with infinite content using GenAI?
10. (pronounced like mosaic)
11. Proprietary + Confidential Demo: GenAI powered app
12. Architecture
14. Flutter for client
15. Cloud Run hosts ui and api servers
16. Cloud Firestore as backend
17. Five Key Data Structures 1. admins 2. generators 3. quizzes 4. sessions 5. results
18. Four Key Personas 1. admin 2. creator 3. host 4. player
19. Data Access Model
20. API
21. Vertex AI on Google Cloud for LLMs
22. Quiz Generators Name Type Format OpenTrivia static multiple choice Palm genAI multiple choice (possible: free-form) Gemini (pro, ultra) genAI multiple choice (possible: free-form)
25. Image Generator Name Type Description ImageGen (v1, v2) genAI Uses ImageGen model to generate images for quizzes
27. After GenAI
28. • Limited Unlimited list of topics • Limited Unlimited questions and answers • Limited Unlimited format • English only Any language • No Unlimited images • Expanding quiz content is difficult easy with GenAI Revisit: Initial problems
29. • Learning curve with GenAI • Inconsistent or no outputs from LLMs • Slow LLM calls • Hallucinations • Hard to check the accuracy and quality of LLM outputs • Fast changing landscape (models, APIs, libraries, etc.) New problems with GenAI
30. Lessons Learned
31. 🎓 General Surprisingly easy to do hard things with GenAI • Quiz/image generation with a single API call Hard to do things well and consistently • Good results require prompt engineering • You will get inconsistent outputs • Hard to measure the output quality
32. 🎓 General Accept uncertainty of LLMs • Same prompt, same model ⇒ different output • Same prompt, same model gets updated ⇒ different output • Same prompt, different model ⇒ different output
33. 🎓 General Free upgrades with new/updated models • Palm ⇒ Gemini-Pro: better quizzes • Gemini-Pro ⇒ Gemini-Ultra: even better quizzes • Imagen v1 ⇒ v2: better images • No or little code changes
34. 🎓 General Do you even need an LLM? • In grading free-form ⇒ LLM vs. TheFuzz library • Image of the app ⇒ ImageGen vs. good old photo editor • Sometimes you don't need an expensive LLM call
35. 🎓 Prompting Be specific and clear with prompts More detailed prompts != better results Manage prompts like code • Version prompts for safe iteration • Prompt + output parsers go hand-in-hand
36. 🎓 Coding with LLMs Code defensively • LLM call can fail => Retry and keep the user informed • LLM can give you malformed JSON ⇒ Can you still parse JSON somehow? • LLM can return empty results ⇒ Can you live with no quizzes or no image? • LLM can be too cautious ⇒ Do you need to change safety settings?
37. 🎓 Coding with LLMs Pin model versions • gemini-1.0-pro refers to the latest and can change to gemini-1.0-pro@001, gemini-1.0-pro@002, … • Use a specific version such as gemini-1.0-pro@001
38. 🎓 Coding with LLMs Consider using a higher level library like LangChain • You can use Gemini from Google AI Studio and Vertex AI but each has different libraries • In Vertex AI, libraries for PaLM and Gemini are different • Other non-Google models have their own libraries • LangChain can help to abstract all of this away
39. 🎓 Coding with LLMs Good old software engineering tricks • Minimize LLM calls by batching prompts • Use parallel calls (eg. quiz and image generation runs in parallel) • Cache common responses
40. Unit/functional tests are as important as ever • Easy to check existence or format • Is this a quiz with 5 questions and 4 answers? • Is the image generated or not? 🎓 Testing and Validation
41. Testing quality and accuracy is more difficult • Is the quiz actually on the topic of history? • Is the answer actually correct? • Is the generated image appropriate for the quiz? (still an open question) Need a way to measure LLM outputs • Automate it, and use as a benchmark to work towards 🎓 Testing and Validation
42. Use LLM to evaluate LLM outputs 🎓 Testing and Validation
43. How do you know if the validator works? • Use OpenTrivia as corpus of accurate quizzes • See how validator performs against OpenTrivia 🎓 Testing and Validation
44. Every multiple choice quiz can be decomposed into four assertions, of the form: Q: question A: answer For example…Who was the first US president? A. Thomas Jefferson B. Alexander Hamilton C. George Washington D. Bill Clinton can be decomposed into these four assertions: • Q: Who was the first US president? A: Thomas Jefferson is False • Q: Who was the first US president? A: Alexander Hamilton is False • Q: Who was the first US president? A: George Washington is True • Q: Who was the first US president? A: Bill Clinton is False
45. Evaluation “In one (and only one) word, are the following assertions true or false?” Q: Who was the first US president? A: Thomas Jefferson Q: Who was the first US president? A: Alexander Hamilton Q: Who was the first US president? A: George Washington Q: Who was the first US president? A: Bill Clinton LLM: False False True False
46. 🎓 Testing and Validation PaLM initially got around 80% accuracy Gemini Ultra got 91% accuracy
47. 🎓 Testing and Validation Ultimately, you need grounding for more accuracy (eg. grounding with Google Search)
48. Is it possible to have a more dynamic and richer quiz app with the help of GenAI? 7 years  7 weeks  7 years to 7 weeks
49. Thank you Marc Cohen Developer Advocate at Google mco@google.com Mete Atamel Developer Advocate at Google @meteatamel atamel.dev speakerdeck.com/meteatamel