Transforming Language Learning with Generative AI try! Swift Tokyo Xingyu Wang March 22, 2024 A Deep Dive into Duolingo’s AI Tutor Feature - Roleplay

No content

No content

88 million MAUs (Monthly Active Users)

About me ● Full-stack (iOS) engineer at Duolingo since August 2021. First job out of school! ● I enjoy meditation, going to the gym, and reading ● Learning Japanese and French ● Speak Chinese and English ● First time speaking at a conference!

https://openai.com/customer-stories/duolingo

Conversation Helpful Phrases Speech recognition Feedback

Duolingo Max - Roleplay

Architecture

iOS Roleplay backend AI Features backend Roleplay State = chat history Model parameters; 100+ prompts; Requests teaching content from other services OpenAI Completions ARCHITECTUR E

Stateless protocol iOS Roleplay backend Chat history Character: … User: … Character: … User: … Chat history Character: … User: … Character: … User: … Character: … ARCHITECTUR E Roleplay State

Stateless protocol: Why? ● Stateless protocol makes Roleplay backend simple ● We don’t allow users to resume a conversation once they left it ● The payload object transmitted between iOS and backend, Roleplay State, has a reasonable size ○ Caveat: media content ARCHITECTUR E

Chat interface: MVVM ARCHITECTUR E

Challenge #1: Building a Chat Interface

➜ Custom message animations ➜ Working with stateless API: Translate Roleplay State updates to chat updates Chat that can… CHALLENGE #1: Building a Chat Interface

➜ Chat history: UICollectionView ➜ Cell types: top narration, character message, user message, loading ➜ Animations: fade-in, message insertion ◆ Subclass UICollectionViewFlowLayout ◆ Custom UICollectionViewLayoutAttributes ◆ Override initialLayoutAttributesForAppearingItem Chat Interface: UI CHALLENGE #1: Building a Chat Interface

Roleplay State UICollectionView updates Backend CHALLENGE #1: Building a Chat Interface

Character: What do you want to drink? User: I’d like a coffee. Character: Do you want to pay in cash or card? UICollectionView (in Roleplay View) Roleplay Backend Roleplay VM Roleplay State Character: What do you want to drink? User: I’d like a coffee. Character: Do you want to pay in cash or card? Roleplay Message Processor User: I want to pay by cash. ··· User: I want to pay by cash. Character: Great! Enjoy your coffee! Delete: loading Append: new message Character: Great! Enjoy your coffee!

Takeaway: Separation of Concerns ● Big application: 25K+ lines for Roleplay ● Stateless API puts message handling logic on the client ○ VM + state manager and message processor ● Views/view updates + Complex business logic + Networking ● Keeping separation of concerns in mind will help you iterate faster and reuse components CHALLENGE #1: Building a Chat Interface

Challenge #2: Latency Optimization on Helpful Phrases

CHALLENGE #2: LATENCY OPTIMIZATION Helpful Phrases

Generate character’s response Generate helpful phrases

Solution: Prompt and model optimization ➜ Use the right GPT model: GPT-3.5? GPT-4? Fine-tuning GPT-3.5? ➜ Decrease the number of output tokens ➜ Utilize cached input: front-load the repeated part of the prompt ◆ Put conversation history to the end CHALLENGE #2: LATENCY OPTIMIZATION

iOS Solution! Async generation Send character’s response to the user Client kicks off fetch-helpful-phrase s request Text-to-speech of the character plays User taps on input bar (helpful phrases about to surface) Display phrases to user Cancels request right before displaying. Show default phrases: “I want”, “I have”, “I can”, etc. User thinking how to respond Ready to show: “I want to eat”, “the bouillabaisse”, “please”, and “the” 1-2 seconds A few seconds CHALLENGE #2: LATENCY OPTIMIZATION

final class RoleplayRepository { /// Variable to store the fetch task private var fetchHelpfulPhrasesTask: Task<[RoleplayHelpfulPhrase]?, Never>? /// Cancel the fetch request func cancelAsyncHelpfulPhrasesFetch() { guard fetchHelpfulPhrasesTask != nil else { return } // Cancelling the fetch of Helpful Phrases fetchHelpfulPhrasesTask?.cancel() fetchHelpfulPhrasesTask = nil } } iOS Solution in code CHALLENGE #2: LATENCY OPTIMIZATION

extension RoleplayRepository { /// Fetch the helpful phrases, while playing the audio of the character’s message func fetchAsyncHelpfulPhrases(roleplayState: RoleplayState) async throws -> [RoleplayHelpfulPhrase]? { fetchHelpfulPhrasesTask = Task { @MainActor in let helpfulPhrases = try? await dataSource.getHelpfulPhrases(roleplayState: roleplayState) guard fetchHelpfulPhrasesTask?.isCancelled == false else { fetchHelpfulPhrasesTask = nil return nil } fetchHelpfulPhrasesTask = nil return helpfulPhrases } return await fetchHelpfulPhrasesTask?.value } } iOS Solution in code CHALLENGE #2: LATENCY OPTIMIZATION

➜ GPT-4 optimization techniques are essential ➜ Think of creative UX/iOS solutions! ◆ Parallelize requests as much as possible ◆ Cancel a request if taking too long + provide default options Learnings CHALLENGE #2: LATENCY OPTIMIZATION

Developing AI Applications on iOS: TOP TAKEAWAYS

➜ Handle OpenAI outages ➜ Reduce latency ➜ Update to the latest models 1. Backend Expertise TOP TAKEAWAYS

➜ Make sure GPT will follow the prompt ➜ Monitor questionable and problematic content ➜ Tie the GPT feature to the existing features ◆ Feed existing content to the prompt ◆ Make the new feature coherent 2. Prompt Engineering TOP TAKEAWAYS

➜ Latest GPT models ◆ Reduce cost ◆ Improve the quality of completions ➜ Product iterations ◆ Clear and flexible design patterns in code 3. Fast iteration TOP TAKEAWAYS

Unlock possibilities

thank you

questions?