15分でわかる、Flutter ML 世界 @FlutterGakkai

第3回 FlutterGakkai
15分でわかる、Flutter ML 世界
Jai-Chang Park (＠Dreamwalker)

Transcript

1. Jai-Chang Park (ħDreamwalker) 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ Κͥͯ͘fFlutterͺML΀ᕴ຅Ίg ᩲ3ỹ FlutterGakkai 1
2. Park JaiChang (@Dreamwalker) • ᖃଐ: Dreamus Company • Flutter Seoul: Organizer • GDG Golang Korea: Organizer • @jaichangpark • Github: JAICHANGPARK • Flutter / Android / Go / Rust ᥦ࿭ᖖ෨ ᩲ3ỹ FlutterGakkai 2
3. ର৅ • ML(Machine Learning)ϧόϰΤϧϕΩϰΧϜϯ͹ᙓͩͱ͖ፖ • ෺ൃͩͱFlutterΧϜϯͽMLΤ᧱ᠭͩͱ͖ፖ • Flutterͽὢዳ͔͞Νፖ ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3
4. ኷ᶝ ! • Κ͔͡Ν᭳ዧ • Flutter͹ML (Machine Learning)᧱ᠭ͞൱ၙͼ΀Ħ • Flutter͹MLϧόϰΤͻ΀Κ͘ͽᑿᠭͩΐͫ͝Ħ • ͻͥ͝ΛᘿΓΝ΀͝Ρ͝ΜΐͭΥ🥹 • ⬇ • Flutter x ML(Machine Learning)αΩώ (ͽͼΞ΂͖͖͹ͫͿ..) ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 4
5. Agenda 1. Flutter 2. ML 3. Flutter x ML ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 5
6. 1. Flutter 1.1. FlutterͽͶ͖͸ 1.2. Platform Channel 6
7. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 1.1. FlutterͽͶ͖͸ Flutter is an open source framework by Google for building beautiful, natively compiled, multi-platform applications from a single codebase. https://flutter.dev/ 7
8. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 1.1. FlutterͽͶ͖͸ https://dart.dev/overview#platform 8
9. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 1.1. FlutterͽͶ͖͸ ͼͮDart᜔Ͳ͵ͱĦ🎯 • Flutter TeamΔ෺ൃᰮ࿴ͽ΁਺ၞ͡΀ϜϲεϮϤϸε᜔ͺϮϸτΩϥΤວᅝͩ͸͟ͱg • ͯ΀͘ͳDart᜔͞Flutter φ⑲ϥ΀᠓ཏͺ࿥᫿ͽରͫΝξθΧ͞ີ͝͵ͱg • React Native΁JS(JaveScript)Τᑿ͘gJaveScriptͯ΀ᥦମΤม๞ͩͽ͖͡g • (Dart == Google && Flutter == Google)͞ᬤ͵͸͖Νᅪg • Dart᜔΁ᡯʑAoT(Ahead-of-Time) Compiler΁ͼ͝͵ͱg • Dartφ⑲ϥ͞Flutterφ⑲ϥ΀ͱΓͽṐᅪͩ͸AoTω⑲ϰφά⑲ϸ(Toolchain)Τ෺ൃͫΝg • Dart VM(Virtual Machine)΀ᱠ᧱ặ 9
10. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 1.1. FlutterͽͶ͖͸ https://docs. fl utter.dev/resources/architectural-overview#architectural-layers Architectural layers 1. Framework - Dart Lang 2. Engine - C/C++ - Rendering : Skia, Impeller 3. Embedder - Platform-speci fi c 10
11. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 1.1. FlutterͽͶ͖͸ Architectural layers 1. Framework - Dart Lang 2. Engine - C/C++ - Rendering : Skia, Impeller 3. Embedder - Platform-speci fi c 11
12. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 1.1. FlutterͽͶ͖͸ • ϜϮψύϚή⑲ϥ΀࿸ၙΤFlutter Framework͹ᑿᠭͫΝͽ΁ͻͫ͘Ξ΂Κ ͖͹ͫ͝Ħ • ᆚ͚΂fAndroid, iOS΀Platform΀࿸ၙΤᑿ͖ͱ͖Ĉ • Platform Channel • FFI (Foreign Function Interface) 👇 12
13. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 1.2. Platform Channel 1. Hardware΀࿸ၙ • Bluetooth, BLE • Camera 2. ݖݶ • Permission https:// fl utter.dev/multi-platform/mobile 3. πϸκ⑲ᨼᏄďᲩĐ • Acc, Gyro, Mag 4. TCP/IP Socket 5. ͼͻg 13
14. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 1.2. Platform Channel • Kotlin or Java on Android • Swift or Objective-C on iOS • C++ on Windows • Objective-C on macOS • C on Linux https://docs. fl utter.dev/development/platform-integration/platform- channels?tab=type-mappings-c-plus-plus-tab 14
15. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 1.2. Platform Channel Method Channel • Asynchronous Method ! • ϜϮψύϚή⑲ϥͺᴢᙩͫΝͱΓͽ ᑿᠭͧΞΝφϨϒϰ Method Channel = Future Event Channel • Asynchronous! • ϜϮψύϚή⑲ϥ͹Event Streamďξύϯ⑲ϥĐΤᗀᙩͫ ΝͱΓͽᑿᠭͧΞΝφϨϒϰ Event Channel = Stream 15
16. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 1.2. Platform Channel Κ͔͡Ν᭳ዧ ͖Ζf ီ͖ͩͩfᏝࡶ͹ͫͿgᶶֶͬΈᶶ᠓͔ΝĦ ᬰ͵͸͜͡΋͟͹ͫ͝Ħ 😡🤯 16
17. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 1.2. Platform Channel ML ϧόϰ΁࿥Ꮳ᧣ͽ Native Platform ͹࣮ḝͧΞΐͫg ͶΐΜfᬰ͵͸͜͡ͺ᜽ͽርͳΐͫg 17
18. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 1. Flutter Recap (᠓ᛱ) 1. Flutter΁Google΀Flutter Team͝Λ෺ൃͩͱOpen Source UI Toolkit͹ ͔Νg • Dart, Multi-Platform, Declarative UI 2. Architecture • Framework, Engine, Embedder 3. Platform Channel • Method Channel, Event Channel 18
19. 2. ML 2.1. Machine LearningͽͶ͖͸ 2.2. ML΀ϮΩϛϮϯ 19
20. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 2.1. Machine LearningͽͶ͖͸ AI Arti fi cial Intelligence Machine Learning Deep Learning Χϰιϯοϥͺ ᱠ᧱ặ΀ઓ͖ 20
21. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 2.1. Machine LearningͽͶ͖͸ Data Input Black Box Model ෺ൃ Model ֶश Model Test Retraining Model Deploy User Feedback ෰ᔭ 21
22. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 2.2. ML΀ϮΩϛϮϯ • TensorFlow • TensorFlow.js • TensorFlow Lite (TinyML) • Keras • Pytorch • Other ML API Service • Ex) Open AI 2.0 22
23. 3. Flutter x ML 3.1. ᱠ᙮΀Flutter MLύϱϸώ 3.2. MLϧόϰΤ᧱ᠭͫΝ 3.3. Holobooth-Flutter Forward Edition 23
24. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ Machine Learning Model 1. ͫ͹ͽΚ͡ᥭΛΞͱϧόϰΤᑿ͘ 2. ᥦᐠ͹ᥭΝ 3. APIΤớᠭͫΝ 24
25. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ ͥͥ͹Ęᎈᬍ᠓ͼϢΩϸύ΁ Κ͡ᥭΛΞͱMLϧόϰΤϫ⑲λ⑲࣮᩺͞ͽ ᑿᠭ͹͟ΝΚ͘ͽͫΝͥͺ͹͔Νg 25 Console͹΀Α฼ͱใໝ ࣮ࡍͷΞϓϦԽ
26. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ 26 https://www.tensor fl ow.org/lite/android MobileΖEmbeddedỖ๕ͽᑿ͚Ν TensorFlow Lite Microcontrollers͹Δᑿ͚ t fl ite model
27. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ 27
28. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ 28 🤔ͥΞΛͫ΋͸ΤᬰΝᶶ᠓͔͞Μΐͫ͝Ħ ͫ͹ͽ͘ΐ͡ᥭΛΞͱϖψζ⑲νΤᑿᠭͫΝ΀΁Ħ https://pub.dev/
29. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ 29 Mar 14, 2019 JUN 3, 2020
30. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ 30 fi rebase_ml_vision 0.12.0+3 https://pub.dev/packages/ fi rebase_ml_vision OCT 21, 2021
31. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ 31 This package is now discontinued since these APIs are no longer available in the latest Firebase SDKs. As an alternative you can switch to Google's standalone ML Kit library via google_ml_kit for on-device vision APIs. For calling the Cloud Vision API from your app the recommended approach is using Firebase Authentication and Functions, which gives you a managed, serverless gateway to Google Cloud Vision APIs. For an example Functions project see the vision-annotate-images sample project.
32. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ 32 google_ml_kit 0.13.0 Access᥀: 2023-01-27 https://pub.dev/packages/google_ml_kit
33. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ 33 Google ML Kit - google_ml_kit ࿥᫿ 1. Vision APIs • ը૾ʹϥϕϧΛ෇͚ͯը૾ʹϥϕϧΛ෇͚ɺόʔ ίʔυɺςΩετɺإɺΦϒδΣΫτΛݕग़͢Δ APIg • Face mesh Τ᩹͡ ͫ΋͸ᑿᠭ൱ၙ 2. Natural Language APIs • 58 ᜡ᜔ණ΀຋༁ΤᙣᎭͩ͸຋༁ͩfጿᙩΤᩮᚳͫΝᥦ᝔ ᜡ᜔ॲሑ APIg • ͫ΋͸ᑿᠭ൱ၙ
34. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ 34 🤔ΰξτϥϧόϰ΁ ͻͫ͘Ξ΂͖͖͹ͫ͝Ħ ࿆΁ML Kit΀ϘϰώͧΞͱ ϧόϰΤᑿ͵͸͖ΝΚ͘͹ͫ͞…
35. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ 35 https://pub.dev/packages/ fi rebase_ml_model_downloader fi rebase_ml_model_downloader FirebaseͽΧψϜϲ⑲ώͧΞͱΰξτϥϧόϰΤυΫϸϲ⑲ώ͹͟ΐͫg Access᥀ 2023-01-27
36. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ 36 https://pub.dev/packages/t fl ite_ fl utter https://pub.dev/packages/t fl ite
37. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ 37 🤔 ͔΀t fl te͵͸ͻͥ͝υΫϸϲ⑲ώ͹͟ΐͫĦ 1. ᥭᕧͩͱֶᘯϧόϰΤt fl iteϚΦΩϰͽมỐ • https://www.tensor fl ow.org/lite/guide/model_maker?hl=ja • Tensor fl ow Lite Model Maker 2. Tensor fl ow Hub • https://www.tensor fl ow.org/hub?hl=ko • TensorFlow Hub ΁fύϱ⑲ϐϸεࡁΑ΀࿸ຂֶᘯϧόϰ΀ϯϢνύϯ͹ͫg
38. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ 38 🤔 PyTorch΁ᑿ͚ΐͭΥ͝Ħ
39. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ 39 🤔 Chat Bot(φϨψύϡψύ)΀Κ͘ͼμξϋϥ΁ĦĦ 1. Dialog fl ow • dialog fl ow_ fl utter 1.0.0 2. GPT3 from OpenAI • openai_gpt3_api • gpt_3_dart • fl utter_chatgpt_api • chat_gpt_sdk • chat_gpt_api • chat_gpt_ fl utter
40. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.2. MLϧόϰΤ᧱ᠭͫΝ 40 t fl iteΤυΫϸϲ⑲ώͩ͸ᑿ͘ PackageΤᑿ͘ Assets Folder Firebase ML Custom Tensor fl ow Hub ᥦᐠ͹ᥭΝ Firebase_ml_mod el_downloader t fl ite & t fl ite_ fl utter t fl ite model google_ml_kit End End Start Web͹ᑿ͘ TensorFlow.js ᥦᐠ͹PackageᥭΝ End
41. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.2. MLϧόϰΤ᧱ᠭͫΝ 41 dependencies - pubspec.yaml - PackageΤඦ୯ͽᱮ൰͹ ͟Νg ᆚ) PackageΤᑿ͘
42. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.3. Holobooth-Flutter Forward Edition 42 https://photobooth. fl utter.dev/ Δͩf ͥΞ͚֮͸͖ΐͫ͝Ħ Google I/O 22
43. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.3. Holobooth-Flutter Forward Edition 43 https://holobooth. fl utter.dev/#/
44. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.3. Holobooth-Flutter Forward Edition 44 1. Flutter 2. Firebase 3. TensorFlow.js detect the user’s face within the frame of the camera 4. MediaPipe estimates 468 3D face landmarks in real-time https://medium.com/ fl utter/how-its- made-holobooth-6473f3d018dd https://github.com/ fl utter/holobooth
45. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.3. Holobooth-Flutter Forward Edition 45 https://github.com/ fl utter/holobooth Access᥀: 2023-01-27
46. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.3. Holobooth-Flutter Forward Edition 46
47. ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ ΐͺΓ 47 1. Flutter • Google΀Flutter Team͝Λ෺ൃͩͱOpen Source UI Toolkit͹͔Νg • Dart, Multi-Platform, Declarative UI 2. Flutter x ML • TensorFlow, PyTorch, ML API • KeyPoint 👉 Κ͡ᥭΛΞͱMLϧόϰΤϫ⑲λ⑲࣮᩺͞ͽᑿᠭ͹͟ΝΚ͘ͽͫΝͥͺ͹͔Νg • 1) ͫ͹ͽΚ͡ᥭΛΞͱϧόϰΤᑿ͘ 2) ᥦᐠ͹ᥭΝ 3) APIΤớᠭͫΝ • Flutter - MLઐ༻ύοέʔδ͕ଘࡏ, google_ml_kit (MLKit) 3. Flutter Forward Holobooth • Tensor fl ow.js, MediaPipe
48. ͦਗ਼ௌ͔Μ͞ͺ͖ͦͨ͘ΐͩͱ Jai-Chang Park (ħDreamwalker)