Gemini in Android Studio I/O extended 2024

인천/송도 IO extended 2024에서 위 주제로 발표한 자료 입니다.
https://festa.io/events/5477

Transcript

1. # Android Gemini In Android Studio ߓ೙઱ | GDG Korea Android
3. Gemini API on Android • Gemini Models & AICore • Gemini Nano use cases on Android • Integration & SDKs Gemini in Android Studio • Code Suggestions • App Quality Insights • Multimodal inputs (Preview) Demo Gemini in Android Studioܳ ഝਊ೧ ࢲ Android Application ѐߊೞӝ ೠ ׀ী ࠁӝ Agenda
4. Gemini API on Android Android •Gemini Models & AICore •Gemini Nano use cases on Android •Integration & SDKs
5. Gemini Models On-device ੘সਸ ୊ܻ ೞחؘ ੓যࢲ о੢ ബҗ ੸ੋ ݽ؛ ׮নೠ ੘সী Ѧ୛ ࢎਊ ೞӝ જ਷ ୭࢚੄ ݽ؛ Gemini Nano Gemini Pro Ҋب۽ ࠂ੟ೠ ੘সਸ ୊ܻ ೞӝ ਤೠ о੢ ъ۱ೠ ݽ؛ Gemini Ultra
6. Gemini Models •Android 14+੄ AI Core ۽ ੸ਊ •Google AI Edge SDKܳ ా೧ API ੽Ӕ •Pixel 8 Pro৬ Samsung S24 दܻૉীࢲ ࢎਊ оמ •ੋఠ֔ োѾ হ੉ب ੘ز •ѐੋ ੿ࠁ ࠁഐ •ߔূ٘ ੋ೐ۄ ҙܻ ೙ਃࢿ ઁѢ On-device ੘সਸ ୊ܻ ೞחؘ ੓যࢲ о੢ ബҗ ੸ੋ ݽ؛ Gemini Nano
7. Gemini Nano ׮নೠ హਵ۽ ݫद૑ ਽׹ೞחؘ ب਑ • Magic Rewrite • Suggested Text ਺ࢿ ֣਺ ӝ۾੄ झ݃౟ ਃড ࢤࢿ • Highlight main topics of voice transcripts(English only) • Powered via Gemini Nano on eligible devices • Available offline + data processed on-device Messages Recorder ఃࠁ٘ীࢲ ޙ੗ ݫद૑ী ؀ೠ झ݃౟ ਽׹ ઁউ GBoard
8. AI Core Android on-device AI under the hood, I/O 24 Client app AI Edge SDK LoRA Gemini Nano Safety features AI Core TPU / NPU / APU High-level inference architecture •AICore ח System Imageী ನೣ •Google Playܳ ా೧ ૑ࣘ੸ਵ۽ সؘ੉౟ •জীࢲח AI Edge SDK۽ ੽Ӕ •Low-Rank Adaptation(LoRA) ۨ੉য۽ জ ѐߊ੗о ݽ؛ ݏ୺ ࢸ੿ оמ https://developer.android.com/ai/aicore
9. AI Core Android on-device AI under the hood, I/O 24 Why is Fine-tuning important? •೐܁೐౟ ূ૑פয݂݅ਵ۽ח on-device ݽ؛ীࢲ ୽࠙ೞ૑ ঋ਺ •ઁড ઑѤ : model size, context window •LoRA Fine-tuningਸ ా೧ ੘਷ ݽ؛੉ ౠ੿ ࢎਊ ࢎ۹ীࢲ ڪযդ ࢿמਸ ߊൃೡ ࣻ ੓ب۾ ೣ Client app AI Edge SDK Gemini Nano Safety features AI Core TPU / NPU / APU LoRA https://developer.android.com/ai/aicore
10. Gemini API Integrate Gemini API Google AI Client SDK Vertex AI SDK for Firebase (Preview) (Preview) Add Generative AI to your Android app with the Gemini API, I/O 24
11. Gemini API h tt ps://developer.android.com/ai/generativeai#reference-apis-sdks Google APIs and SDKs for Gen AI on Android
12. Gemini API Integration 4ױ҅ Add Generative AI to your Android app with the Gemini API, I/O 24 Google AI Studioীࢲ ೐܁೐౟ܳ ೐۽షఋ੉ೝ Google AI Studioীࢲ API ః ߊә 01 02 ೐۽ં౟ী Google AI Client SDKী ؀ೠ ઙࣘࢿ ୶о Gemini Proܳ Kotlin ௏٘ী ా೤ 03 04
13. Google AI Studio Gemini API
14. Kotlin Code Gemini API < > Get Code
15. val model = GenerativeModel( modelName = "gemini-1.5-pro", apiKey = BuildConfig.apiKey, generationConfig = generationConfig { temperature = 1f topK = 64 topP = 0.95f maxOutputTokens = 8192 }, safetySettings = listOf( SafetySetting(HarmCategory.DANGEROUS_CONTENT, BlockThreshold.MEDIUM_AND_ABOVE), ) ) https://ai.google.dev/gemini-api/docs/safety-settings https://github.com/google/secrets-gradle-plugin
16. val model = GenerativeModel( modelName = "gemini-1.5-pro", apiKey = BuildConfig.apiKey, generationConfig = generationConfig { temperature = 1f topK = 64 topP = 0.95f maxOutputTokens = 8192 }, safetySettings = listOf( SafetySetting(HarmCategory.DANGEROUS_CONTENT, BlockThreshold.MEDIUM_AND_ABOVE), ) ) https://ai.google.dev/gemini-api/docs/safety-settings https://github.com/google/secrets-gradle-plugin
17. val model = GenerativeModel( modelName = "gemini-1.5-pro", apiKey = BuildConfig.apiKey, generationConfig = generationConfig { temperature = 1f topK = 64 topP = 0.95f maxOutputTokens = 8192 }, safetySettings = listOf( SafetySetting(HarmCategory.DANGEROUS_CONTENT, BlockThreshold.MEDIUM_AND_ABOVE), ) ) https://ai.google.dev/gemini-api/docs/safety-settings https://github.com/google/secrets-gradle-plugin
18. val model = GenerativeModel( modelName = "gemini-1.5-pro", apiKey = BuildConfig.apiKey, generationConfig = generationConfig { temperature = 1f topK = 64 topP = 0.95f maxOutputTokens = 8192 }, safetySettings = listOf( SafetySetting(HarmCategory.DANGEROUS_CONTENT, BlockThreshold.MEDIUM_AND_ABOVE), ) ) https://ai.google.dev/gemini-api/docs/safety-settings https://github.com/google/secrets-gradle-plugin
19. val model = GenerativeModel( modelName = "gemini-1.5-pro", apiKey = BuildConfig.apiKey, generationConfig = generationConfig { temperature = 1f topK = 64 topP = 0.95f maxOutputTokens = 8192 }, safetySettings = listOf( SafetySetting( HarmCategory.DANGEROUS_CONTENT, BlockThreshold.MEDIUM_AND_ABOVE ) ) ) https://ai.google.dev/gemini-api/docs/safety-settings https://github.com/google/secrets-gradle-plugin
20. scope.launch { val response = model.generateContent( content { image(bitmap) text("What is the object in this picture?") } ) } Prompting https://developer.android.com/ai/google-ai-client-sdk
21. scope.launch { val response = model.generateContentStream( content { image(bitmap) text("What is the object in this picture?") } ).collect { response -> outputContent += response.text } } Prompting https://developer.android.com/ai/google-ai-client-sdk
22. Gemini API New Project Template File > New Project > Gemini API Starter
23. Gemini API Onboarding tool Prototyping SDKs Google AI Studio ীࢲ ೐܁೐౟ܳ ࢸ҅೤פ׮. Gemini API ܳ জী ୶о೤פ׮. Google AI Client SDK ܳ ా೤೤פ׮. Vertex AI SDK for Firebase ܳ दب೧ ࠇפ׮.
24. Gemini in Android Studio Android Studio •Code Suggestions •App Quality Insights •Multimodal inputs (Preview)
25. Android Studio 2024.1.3 Canary 1 Ladybug ੉ߣ ࣁ࣌ীࢲ ࢎਊػ Android Studio ߡ੹ What’s new in preview(link)
26. Gemini ৬ ҕਬೡ ೐۽ં౟ ࢶఖೞӝ Context Awareness Control what you share
27. Gemini৬ ੹୓ ҕਬ Context Awareness
28. അ੤ ೐۽ં౟ ҕਬೡ Ѥ૑ ೐۽ં౟݃׮ ޛযࠁӝ Context Awareness
29. ೐۽ં౟ ௏٘ ҕਬೞ૑ ঋ਺ Context Awareness
30. Context Awareness .aiexclude ౵ੌ۽ ஶఫझ౟ ҕਬ ҳࢿ https://developer.android.com/studio/preview/gemini/aiexclude .gitignore ৬ زੌೠ ҳޙਵ۽ ੘ࢿ Intention actions ৬ code completion ನೣ
31. Context Awareness അ੤ ࠁҊ ੓ח ౵ੌ੉ AI ࢎਊ оמೠ ૑ ഛੋೞӝ
32. Context Awareness അ੤ ࠁҊ ੓ח ౵ੌ੉ AI ࢎਊ оמೠ ૑ ഛੋೞӝ
33. Context Awareness Gemini ࢎਊ Gemini ࢎਊ X
34. Gemini ହী ੑ۱ೠ ղਊ਷ ࢲߡ۽ ੹࣠ Context Awareness
35. Code Suggestion ௏٘ ੗ز ৮ࢿ https://developer.android.com/studio/preview/gemini/ai-code-completion Tab ఃܳ ׂ۞ ୶ୌਸ ࣻۅೞҊ Escܳ ׂ۞ ୶ୌਸ ૑਒פ׮. ౵ۄ޷ఠ۽ ߉਷ obj ੄ ࣘࢿ ч੉ String ژח Int ੌ ҃਋ ୹۱೧઱Ҋ, Ӓ ৻ীח “Unknown” ਵ۽ ୹۱
36. Code Suggestion Transform select code ࢶఖೠ ௏٘ ߸ജ Improve readability
37. Code Suggestion Suggest Improvements ѐࢶ ઁউ Make suggestions to improve this code: Code Block What would you like to do?
38. Code Suggestion • Copy • Insert at Cursor • Explore in Playground
39. Code Suggestion Explore in Playground
40. Code Suggestion Document Function
41. Code Suggestion ୶о۽ ѐࢶೞҊ र׮ݶ?
42. Code Suggestion Refine prompt “Describes receiving job types”
43. Code Suggestion Transformation Prompt History
44. Test Scenarios Test Scenarios ਬ׫ పझ౟ ੘ࢿ What's new in Android development tools, I/O 24
45. Test Scenarios Test Scenarios ਬ׫ పझ౟ ࢤࢿ Ѿҗ What's new in Android development tools, I/O 24
46. Test Scenarios // HomeViewModel.kt /** * Notify that the user updated the search query */ fun onSearchInputChanged(searchInput: String) { viewModelState.update { it.copy(searchInput = searchInput) } } ੘ࢿػ పझ౟ दաܻয় ࢎ۹ - ਗࠄ ೣࣻ What's new in Android development tools, I/O 24
47. Test Scenarios ੘ࢿػ పझ౟ दաܻয় ࢎ۹ - ࢤࢿػ ਬ׫ పझ౟ // Test that searchInput in viewModelState is updated with the provided searchInput @Test fun onSearchInputChanged_StateUpdate_EmptyString() { } @Test fun onSearchInputChanged_StateUpdate_SpecialCharacters() { } @Test fun onSearchInputChanged_StateUpdate_LongString() { } What's new in Android development tools, I/O 24
48. Test Scenarios ੘ࢿػ పझ౟ दաܻয় ࢎ۹ - ࢤࢿػ ਬ׫ పझ౟ // Test that searchInput in viewModelState is updated with the provided searchInput @Test fun onSearchInputChanged_StateUpdate_EmptyString() { } @Test fun onSearchInputChanged_StateUpdate_SpecialCharacters() { } @Test fun onSearchInputChanged_StateUpdate_LongString() { } What's new in Android development tools, I/O 24
49. Test Scenarios ੘ࢿػ పझ౟ दաܻয় ࢎ۹ - ࢤࢿػ ਬ׫ పझ౟ // Test that searchInput in viewModelState is updated with the provided searchInput @Test fun onSearchInputChanged_StateUpdate_EmptyString() { } @Test fun onSearchInputChanged_StateUpdate_SpecialCharacters() { } @Test fun onSearchInputChanged_StateUpdate_LongString() { } What's new in Android development tools, I/O 24
50. Commit Message Suggest Commit Message ழ޿ ݫद૑ ઁউ
51. Commit Message
52. Resource Translation Resource ߣ৉ೞӝ
53. Resource Translation Resource ߣ৉ೞӝ strings.xml > Gemini > Transform select code > “Translate to Korean” Gemini ؀ച ହী ֍যب Resource݅ ߣ৉ػ Ѿҗޛਸ ঳ਸ ࣻ ੓׮.
54. Resource Translation Resource ߣ৉ೞӝ
55. App Quality Insights Explain this exception from my app running on Google sdk_gphone64_arm64 with Android version 15: Exception: <Crash Stack> “Prompt” “Show insights”
56. Show insights Crash Analysis Report • The Exception • Location of the Exception • The Call Stack • How to Fix
57. @Composable fun Greeting(name: String, modifier: Modifier = Modifier) { Column { Text( modifier = modifier .clickable { onTextClicked() }, text = "Click", ) } } fun onTextClicked() { throw RuntimeException("Test exception") } Crash ਬߊ ௏٘
58. ೧Ѿ ߑߨ • onTextClicked Ѩష: MainActivity.kt ౵ੌ੄ 55ߣ૩ ઴۽ ੉زೞৈ onTextClicked ೣࣻ ղ੄ ௏٘ܳ ઱੄ Өѱ Ѩష೤פ׮. RuntimeExceptionਸ ੌਵఆ ࣻ ੓ח ֤ܻо ੓ח૑ ഛੋ ೤פ׮. • ٣ߡӒ: Android Studio੄ ٣ߡѢܳ ࢎਊೞৈ ௏٘ प೯ਸ ױ҅߹۽ ૓೯ೞҊ ৘৻о ߊࢤೞ ח ੿ഛೠ ૑੼ਸ ध߹೤פ׮. ߸ࣻ чҗ प೯ ൒ܴਸ Ѩࢎೞৈ ࢚ടਸ ੉೧೤פ׮. • Ҋ੄੸ੋ ৘৻ ഛੋ: పझ౟ ݾ੸ਵ۽ ৘৻ܳ Ҋ੄۽ ߊࢤदఅ ҃਋, ೧׼ ௏٘ܳ ઁѢೞѢա ઑ ੿೤פ׮. • ৘৻ ୊ܻ: ৘৻о Ҋ੄੸ੋ Ѫ੉ ইצ ҃਋, try-catch ࠶۾җ э਷ ৘৻ ୊ܻܳ ୶оೞৈ ࢚ട ਸ ਋ইೞѱ ୊ܻೞҊ জ੉ ୽جೞ૑ ঋب۾ ೤פ׮. 😎
59. Coming soon Function Name ୶ୌ Shift + F6 Rename What's new in Android development tools, I/O 24 Gemini > Rethink variable names
60. Coming soon + = Design.png Compose code Gemini LongContext Multimodal Input preview
61. Geminiীѱ Compose code ࢤࢿ दఃӝ Demo Demo
62. Jetnews Sample app https://github.com/android/compose-samples [рױೠ ಕ੉૑] • Ѩ࢝যܳ ੑ۱ೞח Textfield • Ӓ ইې ୶ୌ ఃਕ٘ ࢜۽਍ Ѩ࢝ ಕ੉૑ܳ ٜ݅যࠁ੗ Demo
63. Demo ਃ୒ࢎ೦җ ಁః૑ ૑੿ Generate a Jetpack Compose `SearchScreen` composable in package `com.example.jetnews.ui.search`.
64. Demo Layout ࢸݺ [Layout] - A Material 3 `TopAppBar` (all padding 10.dp) containing an `OutlinedTextField` with a search icon. - Below, a `LazyColumn` (padding 10.dp) displaying clickable hardcoded search recommendations from 1 to 10.
65. Demo ӝמ ࢸݺ [Functionality] - The search icon should trigger `submitSearch(query: String)` in `SearchScreen` using the text from the `OutlinedTextField`. - Clicking a recommendation should also trigger `submitSearch` in `SearchScreen`.
66. Demo ઁೠ ࢎ೦ ࢸݺ Include all necessary impo rt s with preview composable. Don’t omit code, provide the full solution. Do not include any previous response as context.
67. Demo Prompt Generate a Jetpack Compose `SearchScreen` composable in package `com.example.jetnews.ui.search`. [Layout] - A Material 3 `TopAppBar` (all padding 10.dp) containing an `OutlinedTextField` with a search icon. - Below, a `LazyColumn` (padding 10.dp) displaying clickable hardcoded search recommendations from 1 to 10. [Functionality] - The search icon should trigger `submitSearch(query: String)` in `SearchScreen` using the text from the `OutlinedTextField`. - Clicking a recommendation should also trigger `submitSearch` in `SearchScreen`. Include all necessary impo rt s with preview composable. Don’t omit code, provide the full solution. Do not include any previous response as context.
68. Demo ࢤࢿ Ѿҗ
69. хࢎ೤פ׮. ߓ೙઱ veronikapj GDG Korea Android