15分でわかる、Flutter ML 世界 @FlutterGakkai

Slide 1

Slide 1 text

Jai-Chang Park (ħDreamwalker) 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ Κͥͯ͘fFlutterͺML΀ᕴ຅Ίg ᩲ3ỹ FlutterGakkai 1

Slide 2

Slide 2 text

Park JaiChang (@Dreamwalker) • ᖃଐ: Dreamus Company • Flutter Seoul: Organizer • GDG Golang Korea: Organizer • @jaichangpark • Github: JAICHANGPARK • Flutter / Android / Go / Rust ᥦ࿭ᖖ෨ ᩲ3ỹ FlutterGakkai 2

Slide 3

Slide 3 text

ର৅ • ML(Machine Learning)ϧόϰΤϧϕΩϰΧϜϯ͹ᙓͩͱ͖ፖ • ෺ൃͩͱFlutterΧϜϯͽMLΤ᧱ᠭͩͱ͖ፖ • Flutterͽὢዳ͔͞Νፖ ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3

Slide 4

Slide 4 text

኷ᶝ ! • Κ͔͡Ν᭳ዧ • Flutter͹ML (Machine Learning)᧱ᠭ͞൱ၙͼ΀Ħ • Flutter͹MLϧόϰΤͻ΀Κ͘ͽᑿᠭͩΐͫ͝Ħ • ͻͥ͝ΛᘿΓΝ΀͝Ρ͝ΜΐͭΥ🥹 • ⬇ • Flutter x ML(Machine Learning)αΩώ (ͽͼΞ΂͖͖͹ͫͿ..) ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 4

Slide 5

Slide 5 text

Agenda 1. Flutter 2. ML 3. Flutter x ML ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 5

Slide 6

Slide 6 text

1. Flutter 1.1. FlutterͽͶ͖͸ 1.2. Platform Channel 6

Slide 7

Slide 7 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 1.1. FlutterͽͶ͖͸ Flutter is an open source framework by Google for building beautiful, natively compiled, multi-platform applications from a single codebase. https://flutter.dev/ 7

Slide 8

Slide 8 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 1.1. FlutterͽͶ͖͸ https://dart.dev/overview#platform 8

Slide 9

Slide 9 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 1.1. FlutterͽͶ͖͸ ͼͮDart᜔Ͳ͵ͱĦ🎯 • Flutter TeamΔ෺ൃᰮ࿴ͽ΁਺ၞ͡΀ϜϲεϮϤϸε᜔ͺϮϸτΩϥΤວᅝͩ͸͟ͱg • ͯ΀͘ͳDart᜔͞Flutter φ⑲ϥ΀᠓ཏͺ࿥᫿ͽରͫΝξθΧ͞ີ͝͵ͱg • React Native΁JS(JaveScript)Τᑿ͘gJaveScriptͯ΀ᥦମΤม๞ͩͽ͖͡g • (Dart == Google && Flutter == Google)͞ᬤ͵͸͖Νᅪg • Dart᜔΁ᡯʑAoT(Ahead-of-Time) Compiler΁ͼ͝͵ͱg • Dartφ⑲ϥ͞Flutterφ⑲ϥ΀ͱΓͽṐᅪͩ͸AoTω⑲ϰφά⑲ϸ(Toolchain)Τ෺ൃͫΝg • Dart VM(Virtual Machine)΀ᱠ᧱ặ 9

Slide 10

Slide 10 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 1.1. FlutterͽͶ͖͸ https://docs. fl utter.dev/resources/architectural-overview#architectural-layers Architectural layers 1. Framework - Dart Lang 2. Engine - C/C++ - Rendering : Skia, Impeller 3. Embedder - Platform-speci fi c 10

Slide 11

Slide 11 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 1.1. FlutterͽͶ͖͸ Architectural layers 1. Framework - Dart Lang 2. Engine - C/C++ - Rendering : Skia, Impeller 3. Embedder - Platform-speci fi c 11

Slide 12

Slide 12 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 1.1. FlutterͽͶ͖͸ • ϜϮψύϚή⑲ϥ΀࿸ၙΤFlutter Framework͹ᑿᠭͫΝͽ΁ͻͫ͘Ξ΂Κ ͖͹ͫ͝Ħ • ᆚ͚΂fAndroid, iOS΀Platform΀࿸ၙΤᑿ͖ͱ͖Ĉ • Platform Channel • FFI (Foreign Function Interface) 👇 12

Slide 13

Slide 13 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 1.2. Platform Channel 1. Hardware΀࿸ၙ • Bluetooth, BLE • Camera 2. ݖݶ • Permission https:// fl utter.dev/multi-platform/mobile 3. πϸκ⑲ᨼᏄďᲩĐ • Acc, Gyro, Mag 4. TCP/IP Socket 5. ͼͻg 13

Slide 14

Slide 14 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 1.2. Platform Channel • Kotlin or Java on Android • Swift or Objective-C on iOS • C++ on Windows • Objective-C on macOS • C on Linux https://docs. fl utter.dev/development/platform-integration/platform- channels?tab=type-mappings-c-plus-plus-tab 14

Slide 15

Slide 15 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 1.2. Platform Channel Method Channel • Asynchronous Method ! • ϜϮψύϚή⑲ϥͺᴢᙩͫΝͱΓͽ ᑿᠭͧΞΝφϨϒϰ Method Channel = Future Event Channel • Asynchronous! • ϜϮψύϚή⑲ϥ͹Event Streamďξύϯ⑲ϥĐΤᗀᙩͫ ΝͱΓͽᑿᠭͧΞΝφϨϒϰ Event Channel = Stream 15

Slide 16

Slide 16 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 1.2. Platform Channel Κ͔͡Ν᭳ዧ ͖Ζf ီ͖ͩͩfᏝࡶ͹ͫͿgᶶֶͬΈᶶ᠓͔ΝĦ ᬰ͵͸͜͡΋͟͹ͫ͝Ħ 😡🤯 16

Slide 17

Slide 17 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 1.2. Platform Channel ML ϧόϰ΁࿥Ꮳ᧣ͽ Native Platform ͹࣮ḝͧΞΐͫg ͶΐΜfᬰ͵͸͜͡ͺ᜽ͽርͳΐͫg 17

Slide 18

Slide 18 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 1. Flutter Recap (᠓ᛱ) 1. Flutter΁Google΀Flutter Team͝Λ෺ൃͩͱOpen Source UI Toolkit͹ ͔Νg • Dart, Multi-Platform, Declarative UI 2. Architecture • Framework, Engine, Embedder 3. Platform Channel • Method Channel, Event Channel 18

Slide 19

Slide 19 text

2. ML 2.1. Machine LearningͽͶ͖͸ 2.2. ML΀ϮΩϛϮϯ 19

Slide 20

Slide 20 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 2.1. Machine LearningͽͶ͖͸ AI Arti fi cial Intelligence Machine Learning Deep Learning Χϰιϯοϥͺ ᱠ᧱ặ΀ઓ͖ 20

Slide 21

Slide 21 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 2.1. Machine LearningͽͶ͖͸ Data Input Black Box Model ෺ൃ Model ֶश Model Test Retraining Model Deploy User Feedback ෰ᔭ 21

Slide 22

Slide 22 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 2.2. ML΀ϮΩϛϮϯ • TensorFlow • TensorFlow.js • TensorFlow Lite (TinyML) • Keras • Pytorch • Other ML API Service • Ex) Open AI 2.0 22

Slide 23

Slide 23 text

3. Flutter x ML 3.1. ᱠ᙮΀Flutter MLύϱϸώ 3.2. MLϧόϰΤ᧱ᠭͫΝ 3.3. Holobooth-Flutter Forward Edition 23

Slide 24

Slide 24 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ Machine Learning Model 1. ͫ͹ͽΚ͡ᥭΛΞͱϧόϰΤᑿ͘ 2. ᥦᐠ͹ᥭΝ 3. APIΤớᠭͫΝ 24

Slide 25

Slide 25 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ ͥͥ͹Ęᎈᬍ᠓ͼϢΩϸύ΁ Κ͡ᥭΛΞͱMLϧόϰΤϫ⑲λ⑲࣮᩺͞ͽ ᑿᠭ͹͟ΝΚ͘ͽͫΝͥͺ͹͔Νg 25 Console͹΀Α฼ͱใໝ ࣮ࡍͷΞϓϦԽ

Slide 26

Slide 26 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ 26 https://www.tensor fl ow.org/lite/android MobileΖEmbeddedỖ๕ͽᑿ͚Ν TensorFlow Lite Microcontrollers͹Δᑿ͚ t fl ite model

Slide 27

Slide 27 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ 27

Slide 28

Slide 28 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ 28 🤔ͥΞΛͫ΋͸ΤᬰΝᶶ᠓͔͞Μΐͫ͝Ħ ͫ͹ͽ͘ΐ͡ᥭΛΞͱϖψζ⑲νΤᑿᠭͫΝ΀΁Ħ https://pub.dev/

Slide 29

Slide 29 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ 29 Mar 14, 2019 JUN 3, 2020

Slide 30

Slide 30 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ 30 fi rebase_ml_vision 0.12.0+3 https://pub.dev/packages/ fi rebase_ml_vision OCT 21, 2021

Slide 31

Slide 31 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ 31 This package is now discontinued since these APIs are no longer available in the latest Firebase SDKs. As an alternative you can switch to Google's standalone ML Kit library via google_ml_kit for on-device vision APIs. For calling the Cloud Vision API from your app the recommended approach is using Firebase Authentication and Functions, which gives you a managed, serverless gateway to Google Cloud Vision APIs. For an example Functions project see the vision-annotate-images sample project.

Slide 32

Slide 32 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ 32 google_ml_kit 0.13.0 Access᥀: 2023-01-27 https://pub.dev/packages/google_ml_kit

Slide 33

Slide 33 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ 33 Google ML Kit - google_ml_kit ࿥᫿ 1. Vision APIs • ը૾ʹϥϕϧΛ෇͚ͯը૾ʹϥϕϧΛ෇͚ɺόʔ ίʔυɺςΩετɺإɺΦϒδΣΫτΛݕग़͢Δ APIg • Face mesh Τ᩹͡ ͫ΋͸ᑿᠭ൱ၙ 2. Natural Language APIs • 58 ᜡ᜔ණ΀຋༁ΤᙣᎭͩ͸຋༁ͩfጿᙩΤᩮᚳͫΝᥦ᝔ ᜡ᜔ॲሑ APIg • ͫ΋͸ᑿᠭ൱ၙ

Slide 34

Slide 34 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ 34 🤔ΰξτϥϧόϰ΁ ͻͫ͘Ξ΂͖͖͹ͫ͝Ħ ࿆΁ML Kit΀ϘϰώͧΞͱ ϧόϰΤᑿ͵͸͖ΝΚ͘͹ͫ͞…

Slide 35

Slide 35 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ 35 https://pub.dev/packages/ fi rebase_ml_model_downloader fi rebase_ml_model_downloader FirebaseͽΧψϜϲ⑲ώͧΞͱΰξτϥϧόϰΤυΫϸϲ⑲ώ͹͟ΐͫg Access᥀ 2023-01-27

Slide 36

Slide 36 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ 36 https://pub.dev/packages/t fl ite_ fl utter https://pub.dev/packages/t fl ite

Slide 37

Slide 37 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ 37 🤔 ͔΀t fl te͵͸ͻͥ͝υΫϸϲ⑲ώ͹͟ΐͫĦ 1. ᥭᕧͩͱֶᘯϧόϰΤt fl iteϚΦΩϰͽมỐ • https://www.tensor fl ow.org/lite/guide/model_maker?hl=ja • Tensor fl ow Lite Model Maker 2. Tensor fl ow Hub • https://www.tensor fl ow.org/hub?hl=ko • TensorFlow Hub ΁fύϱ⑲ϐϸεࡁΑ΀࿸ຂֶᘯϧόϰ΀ϯϢνύϯ͹ͫg

Slide 38

Slide 38 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ 38 🤔 PyTorch΁ᑿ͚ΐͭΥ͝Ħ

Slide 39

Slide 39 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.1. ᱠ᙮΀Flutter MLύϱϸώ 39 🤔 Chat Bot(φϨψύϡψύ)΀Κ͘ͼμξϋϥ΁ĦĦ 1. Dialog fl ow • dialog fl ow_ fl utter 1.0.0 2. GPT3 from OpenAI • openai_gpt3_api • gpt_3_dart • fl utter_chatgpt_api • chat_gpt_sdk • chat_gpt_api • chat_gpt_ fl utter

Slide 40

Slide 40 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.2. MLϧόϰΤ᧱ᠭͫΝ 40 t fl iteΤυΫϸϲ⑲ώͩ͸ᑿ͘ PackageΤᑿ͘ Assets Folder Firebase ML Custom Tensor fl ow Hub ᥦᐠ͹ᥭΝ Firebase_ml_mod el_downloader t fl ite & t fl ite_ fl utter t fl ite model google_ml_kit End End Start Web͹ᑿ͘ TensorFlow.js ᥦᐠ͹PackageᥭΝ End

Slide 41

Slide 41 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.2. MLϧόϰΤ᧱ᠭͫΝ 41 dependencies - pubspec.yaml - PackageΤඦ୯ͽᱮ൰͹ ͟Νg ᆚ) PackageΤᑿ͘

Slide 42

Slide 42 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.3. Holobooth-Flutter Forward Edition 42 https://photobooth. fl utter.dev/ Δͩf ͥΞ͚֮͸͖ΐͫ͝Ħ Google I/O 22

Slide 43

Slide 43 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.3. Holobooth-Flutter Forward Edition 43 https://holobooth. fl utter.dev/#/

Slide 44

Slide 44 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.3. Holobooth-Flutter Forward Edition 44 1. Flutter 2. Firebase 3. TensorFlow.js detect the user’s face within the frame of the camera 4. MediaPipe estimates 468 3D face landmarks in real-time https://medium.com/ fl utter/how-its- made-holobooth-6473f3d018dd https://github.com/ fl utter/holobooth

Slide 45

Slide 45 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.3. Holobooth-Flutter Forward Edition 45 https://github.com/ fl utter/holobooth Access᥀: 2023-01-27

Slide 46

Slide 46 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ 3.3. Holobooth-Flutter Forward Edition 46

Slide 47

Slide 47 text

ᩲ3ỹ FlutterGakkai 15ᐠ͹Ρ͝ΝfFlutter ML ᕴ຅ ΐͺΓ 47 1. Flutter • Google΀Flutter Team͝Λ෺ൃͩͱOpen Source UI Toolkit͹͔Νg • Dart, Multi-Platform, Declarative UI 2. Flutter x ML • TensorFlow, PyTorch, ML API • KeyPoint 👉 Κ͡ᥭΛΞͱMLϧόϰΤϫ⑲λ⑲࣮᩺͞ͽᑿᠭ͹͟ΝΚ͘ͽͫΝͥͺ͹͔Νg • 1) ͫ͹ͽΚ͡ᥭΛΞͱϧόϰΤᑿ͘ 2) ᥦᐠ͹ᥭΝ 3) APIΤớᠭͫΝ • Flutter - MLઐ༻ύοέʔδ͕ଘࡏ, google_ml_kit (MLKit) 3. Flutter Forward Holobooth • Tensor fl ow.js, MediaPipe

Slide 48

Slide 48 text

ͦਗ਼ௌ͔Μ͞ͺ͖ͦͨ͘ΐͩͱ Jai-Chang Park (ħDreamwalker)