try! Swift Tokyo 2024 - Duolingo Max Roleplay

The Roleplay feature of Duolingo Max (by Xingyu Wang)

Transcript

1. Transforming Language Learning with Generative AI try! Swift Tokyo Xingyu Wang March 22, 2024 A Deep Dive into Duolingo’s AI Tutor Feature - Roleplay
4. 88 million MAUs (Monthly Active Users)
5. About me • Full-stack (iOS) engineer at Duolingo since August 2021. First job out of school! • I enjoy meditation, going to the gym, and reading • Learning Japanese and French • Speak Chinese and English • First time speaking at a conference!
6. https://openai.com/customer-stories/duolingo
7. Conversation Helpful Phrases Speech recognition Feedback
8. Duolingo Max - Roleplay
9. Architecture
10. iOS Roleplay backend AI Features backend Roleplay State = chat history Model parameters; 100+ prompts; Requests teaching content from other services OpenAI Completions ARCHITECTUR E
11. Stateless protocol iOS Roleplay backend Chat history Character: … User: … Character: … User: … Chat history Character: … User: … Character: … User: … Character: … ARCHITECTUR E Roleplay State
12. Stateless protocol: Why? • Stateless protocol makes Roleplay backend simple • We don’t allow users to resume a conversation once they left it • The payload object transmitted between iOS and backend, Roleplay State, has a reasonable size ◦ Caveat: media content ARCHITECTUR E
13. Chat interface: MVVM ARCHITECTUR E
14. Challenge #1: Building a Chat Interface
15. ➜ Custom message animations ➜ Working with stateless API: Translate Roleplay State updates to chat updates Chat that can… CHALLENGE #1: Building a Chat Interface
16. ➜ Chat history: UICollectionView ➜ Cell types: top narration, character message, user message, loading ➜ Animations: fade-in, message insertion ◆ Subclass UICollectionViewFlowLayout ◆ Custom UICollectionViewLayoutAttributes ◆ Override initialLayoutAttributesForAppearingItem Chat Interface: UI CHALLENGE #1: Building a Chat Interface
17. Roleplay State UICollectionView updates Backend CHALLENGE #1: Building a Chat Interface
18. Character: What do you want to drink? User: I’d like a coffee. Character: Do you want to pay in cash or card? UICollectionView (in Roleplay View) Roleplay Backend Roleplay VM Roleplay State Character: What do you want to drink? User: I’d like a coffee. Character: Do you want to pay in cash or card? Roleplay Message Processor User: I want to pay by cash. ··· User: I want to pay by cash. Character: Great! Enjoy your coffee! Delete: loading Append: new message Character: Great! Enjoy your coffee!
19. Takeaway: Separation of Concerns • Big application: 25K+ lines for Roleplay • Stateless API puts message handling logic on the client ◦ VM + state manager and message processor • Views/view updates + Complex business logic + Networking • Keeping separation of concerns in mind will help you iterate faster and reuse components CHALLENGE #1: Building a Chat Interface
20. Challenge #2: Latency Optimization on Helpful Phrases
21. CHALLENGE #2: LATENCY OPTIMIZATION Helpful Phrases
22. Generate character’s response Generate helpful phrases
23. Solution: Prompt and model optimization ➜ Use the right GPT model: GPT-3.5? GPT-4? Fine-tuning GPT-3.5? ➜ Decrease the number of output tokens ➜ Utilize cached input: front-load the repeated part of the prompt ◆ Put conversation history to the end CHALLENGE #2: LATENCY OPTIMIZATION
24. iOS Solution! Async generation Send character’s response to the user Client kicks off fetch-helpful-phrase s request Text-to-speech of the character plays User taps on input bar (helpful phrases about to surface) Display phrases to user Cancels request right before displaying. Show default phrases: “I want”, “I have”, “I can”, etc. User thinking how to respond Ready to show: “I want to eat”, “the bouillabaisse”, “please”, and “the” 1-2 seconds A few seconds CHALLENGE #2: LATENCY OPTIMIZATION
25. final class RoleplayRepository { /// Variable to store the fetch task private var fetchHelpfulPhrasesTask: Task<[RoleplayHelpfulPhrase]?, Never>? /// Cancel the fetch request func cancelAsyncHelpfulPhrasesFetch() { guard fetchHelpfulPhrasesTask != nil else { return } // Cancelling the fetch of Helpful Phrases fetchHelpfulPhrasesTask?.cancel() fetchHelpfulPhrasesTask = nil } } iOS Solution in code CHALLENGE #2: LATENCY OPTIMIZATION
26. extension RoleplayRepository { /// Fetch the helpful phrases, while playing the audio of the character’s message func fetchAsyncHelpfulPhrases(roleplayState: RoleplayState) async throws -> [RoleplayHelpfulPhrase]? { fetchHelpfulPhrasesTask = Task { @MainActor in let helpfulPhrases = try? await dataSource.getHelpfulPhrases(roleplayState: roleplayState) guard fetchHelpfulPhrasesTask?.isCancelled == false else { fetchHelpfulPhrasesTask = nil return nil } fetchHelpfulPhrasesTask = nil return helpfulPhrases } return await fetchHelpfulPhrasesTask?.value } } iOS Solution in code CHALLENGE #2: LATENCY OPTIMIZATION
27. ➜ GPT-4 optimization techniques are essential ➜ Think of creative UX/iOS solutions! ◆ Parallelize requests as much as possible ◆ Cancel a request if taking too long + provide default options Learnings CHALLENGE #2: LATENCY OPTIMIZATION
28. Developing AI Applications on iOS: TOP TAKEAWAYS
29. ➜ Handle OpenAI outages ➜ Reduce latency ➜ Update to the latest models 1. Backend Expertise TOP TAKEAWAYS
30. ➜ Make sure GPT will follow the prompt ➜ Monitor questionable and problematic content ➜ Tie the GPT feature to the existing features ◆ Feed existing content to the prompt ◆ Make the new feature coherent 2. Prompt Engineering TOP TAKEAWAYS
31. ➜ Latest GPT models ◆ Reduce cost ◆ Improve the quality of completions ➜ Product iterations ◆ Clear and flexible design patterns in code 3. Fast iteration TOP TAKEAWAYS
32. Unlock possibilities
33. thank you
34. questions?