[DevTalksRomania]Building Augmented Reality Experiences with Flutter and AI-Powered Development

Slide 1

Slide 1 text

Kenichi Kambara (@korodroid) June 5, 2025 Building Augmented Reality Experiences with Flutter and AI-Powered Development Romania 2025

Slide 2

Slide 2 text

•Mobile App Development •Talks (21 Int’l/100+ Domestic) •Writings (10 Dev/Edu Books) •[Of fi cial] Evangelist at NTT TechnoCross e.g. AI-powered call analysis, livestock monitoring, and privileged access management •[Private] iplatform.org About me Kenichi Kambara (X:@korodroid)

Slide 3

Slide 3 text

Sekaiphone Pro(Flutter(+Kotlin/Swift)) Behind the Scenes of My Personal Project

Slide 4

Slide 4 text

My Product and What I’ve Learned Visited 50+ Countries

Slide 5

Slide 5 text

Agenda •My Motivation and Vision •Enhancing your Flutter App with AR •Common Design Pitfalls in AR UI/UX to Avoid •AI-Driven AR Development •Best Practices Based on Our Experience

Slide 6

Slide 6 text

My Motivation and Vision From Smartphones to the Seamless Integration of the Virtual ˍ Physical 🚀

Slide 7

Slide 7 text

Use Case 1: Same Place, No Face-to-Face

Slide 8

Slide 8 text

Use Case 1: My Solution with AR

Slide 9

Slide 9 text

Use Case 2: Understanding Unfamiliar Text

Slide 10

Slide 10 text

Use Case 2: My Solution with AR

Slide 11

Slide 11 text

Challenges to Overcome •Steps to Enhance a Flutter App with AR •Common Development Challenges and Solutions •Principles for Developing AR Features

Slide 12

Slide 12 text

About Flutter A framework that allows simultaneous development of apps for multiple platforms from a common source (Cross-platform) https://docs. fl utter.dev/

Slide 13

Slide 13 text

3 Unique features of Flutter 1. Beautiful UI & High Performance 2. Fast and ef fi cient development process 3. Uses Dart as the programming language

Slide 14

Slide 14 text

1. Beautiful UI & High Performance Flutter showcase https:// fl utter.dev/showcase

Slide 15

Slide 15 text

2. Fast and ef fi cient development process Hot reload https://docs. fl utter.dev/tools/hot-reload

Slide 16

Slide 16 text

3. Uses Dart as the programming language Very similar to Java/JavaScript import 'package:flutter/material.dart'; void main() { runApp( Center( child: Text( ‘Hello, Flutter’, textDirection: TextDirection.ltr, ), ), ); }

Slide 17

Slide 17 text

Enhancing your Flutter App with AR

Slide 18

Slide 18 text

Use Case 1: Camera Preview & Overlay (a) Create a Camera Preview Screen (b) Stack multiple Layers

Slide 19

Slide 19 text

(a) Create a Camera Preview Screen 1. Install the package 2. Set up required permissions 3. Initialize the camera & show the preview 4. Test the app

Slide 20

Slide 20 text

1. Install the package https://pub.dev/packages/camera

Slide 21

Slide 21 text

1. Install the package pubspec.yaml dependencies: flutter: sdk: flutter camera: ^0.10.0+2 permission_handler: ^10.2.0

Slide 22

Slide 22 text

1. Install the package pubspec.yaml dependencies: flutter: sdk: flutter camera: ^0.10.0+2 permission_handler: ^10.2.0

Slide 23

Slide 23 text

2. Set up required permissions [Android]AndroidManifest.xml [iOS]ios/Runner/Info.plist

Slide 24

Slide 24 text

3. Initialize the camera & show the preview main.dart 1/3

Slide 25

Slide 25 text

3. Initialize the camera & show the preview main.dart 2/3

Slide 26

Slide 26 text

3. Initialize the camera & show the preview main.dart 3/3

Slide 27

Slide 27 text

4. Test the app Device Emulator

Slide 28

Slide 28 text

(b) Stack multiple Layers A B A B

Slide 29

Slide 29 text

How to Stack multiple Layers ref. Stack: https://api. fl utter.dev/ fl utter/widgets/Stack-class.html

Slide 30

Slide 30 text

Code Example (Multiple Layers) A B Widget _stackTwoScreen() { return Stack( children: [ ScreenA, ScreenB, ], ); }

Slide 31

Slide 31 text

Use Case 2: Text Recognition and Overlay (c) Recognize Text [Translate recognized Text] (d) Overlay Text on Camera Preview (e) Adjust Text Position Lapte

Slide 32

Slide 32 text

Slide 33

Slide 33 text

1. Install the package pubspec.yaml dependencies: flutter: sdk: flutter camera: ^0.10.0+2 permission_handler: ^10.2.0 google_mlkit_text_recognition: ^0.10.0

Slide 34

Slide 34 text

2. Recognize Text from Camera Preview CameraImage RecognizedText InputImage Convert Recognize Text MLKit - TextRecognizer#processImage

Slide 35

Slide 35 text

2. Recognize Text from Camera Preview main.dart final TextRecognizer _textRecognizer = TextRecognizer(); void _processCameraImage(CameraImage image) async { if (_isDetecting) return; _isDetecting = true; try { final WriteBuffer allBytes = WriteBuffer(); for (var plane in image.planes) { allBytes.putUint8List(plane.bytes); } final bytes = allBytes.done().buffer.asUint8List(); // Create Metadata final InputImageMetadata metadata = InputImageMetadata( size: Size(image.width.toDouble(), image.height.toDouble()), rotation: InputImageRotation.rotation0deg, format: InputImageFormat.nv21, // Image format bytesPerRow: image.planes[0].bytesPerRow, );

Slide 36

Slide 36 text

2. Recognize Text from Camera Preview final InputImage inputImage = InputImage.fromBytes( bytes: bytes, metadata: metadata, ); final RecognizedText recognizedText = await _textRecognizer.processImage(inputImage); // Output recognized Text & Coordination for (TextBlock block in recognizedText.blocks) { final Rect boundingBox = block.boundingBox; final String text = block.text; // Output to console print('Text: $text'); print('BoundingBox: ${boundingBox.left}, ${boundingBox.top}, ${boundingBox.right}, $ {boundingBox.bottom}'); } } catch (e) { print('Error in recognizing text: $e'); } finally { _isDetecting = false; } }

Slide 37

Slide 37 text

Device Console Text: Milk BoundingBox: 123.0, 214.0, 552.0, 269.0 3. Test the app (on Device)

Slide 38

Slide 38 text

Device Emulator ? Challenge: Testing AR Features on an Emulator

Slide 39

Slide 39 text

1. Use the advanced Emulator Setting 2. Use the static Image instead of Camera Solution: How to test using Emulator

Slide 40

Slide 40 text

Using the advanced Emulator Setting •None •VirtualScene •Emulated •Webcam0

Slide 41

Slide 41 text

Camera Settings on Emulator Emulated VirtualScene Webcam

Slide 42

Slide 42 text

Using the static Image instead of Camera Camera Image (Camera Preview) Static Image (Assets)

Slide 43

Slide 43 text

Tips. How to test using Asset/Images final inputImage = InputImage.fromFilePath('assets/test_image.png'); final RecognizedText recognizedTextResult = await _textRecognizer.processImage(inputImage); Future processImage(InputImage inputImage) async { final result = await _channel.invokeMethod( 'vision#startTextRecognizer', { 'id': id, 'imageData': inputImage.toJson(), 'script': script.index }); return RecognizedText.fromJson(result); }

Slide 44

Slide 44 text

(d) Overlay Text on Camera Preview Lapte Lapte

Slide 45

Slide 45 text

Code Example (CameraPreview&Text) Widget _stackTwoScreens() { return Stack( children: [ PreviewScreen, OverlayTextScreen, ], ); } Lapte

Slide 46

Slide 46 text

Common Design Pitfalls in AR UI/UX to Avoid

Slide 47

Slide 47 text

Common Design Pitfalls in AR UI/UX to Avoid •Common user frustrations →e.g., unreadable text, visual overload, slow response •Accessibility issues in AR →e.g., low contrast, small font sizes ,and so on •What I learned from user feedback in my app →e.g., Couldn’t read the overlay in sunlight

Slide 48

Slide 48 text

e.g. Text Visibility on Camera Preview Milk Milk Milk

Slide 49

Slide 49 text

• Add a Semi-Transparent Background to the Text • Add a Shadow to the Text • Use Outlined Text • Dynamically Change the Text Color • Blur Part of the Background Solutions to Improve Text Visibility on Camera Preview

Slide 50

Slide 50 text

Solution examples Semi-Transparent Background Shadow to the Text Outlined Text

Slide 51

Slide 51 text

• Add a Semi-Transparent Background to the Text • Add a Shadow to the Text • Use Outlined Text • Dynamically Change the Text Color • Blur Part of the Background Solutions to Avoid Display Issues

Slide 52

Slide 52 text

Add a Semi-Transparent Background to the Text Widget _showOverlayTextScreen() { return Container( color: Colors.black.withOpacity(0.5), child: Text( text, style: textStyle, ), ); }

Slide 53

Slide 53 text

TextElement elementNew = TextElement( text: resultText, // e.g. “Milk”, boundingBox: element.boundingBox, cornerPoints: element.cornerPoints, symbols: element.symbols, recognizedLanguages: element.recognizedLanguages, confidence: element.confidence, angle: element.angle); Text: Milk BoundingBox: 123.0, 214.0, 552.0, 269.0 (e) Adjust Text Position

Slide 54

Slide 54 text

(e) Adjust Text Position Bounding box coordinates are based on the original image dimensions Essential to Scale

Slide 55

Slide 55 text

scaleX = constraints.maxWidth / imageSize.width; scaleY = constraints.maxHeight / imageSize.height; Positioned( left: element.boundingBox.left * scaleX, top: element.boundingBox.top * scaleY, width: element.boundingBox.width * scaleX, height: element.boundingBox.height * scaleY, child: Text( element.text, …) ); (e) Adjust Text Position

Slide 56

Slide 56 text

AI-Driven AR Development

Slide 57

Slide 57 text

How AI Helped Us Accelerate AR Development •Real Prompts from Our Work fl ow •AI for bug fi xing and optimization •Development speed boost

Slide 58

Slide 58 text

Mobile Apps Development with Generative AI Natural language prompts for: •Cultural UX Guidance •AR Story Generation •Natural Language Tech Support $POWFSTBUJPOBM"* 🧠Gemini 🧠ChatGPT 🧠Claude *%&*OUFHSBUFE"* 🛠Cursor 🛠Cline 🛠GitHub Copilot Embedded in your IDE for: •AR Code Optimization •Localization Bug Detection •AR Asset Integration

Slide 59

Slide 59 text

Use Case 1: Using Conversational AI English Arabic Japanese

Slide 60

Slide 60 text

Examples of Language Resources // English Resource "en": { // Top Screen "appName": "SekaiPhone Pro", "modeTalkSpeech": "Talk Mode\n(Speech)", "modeTalkText": "Talk Mode\n(Text)", "modePhone": "Phone Mode", "modeCamera": "Camera Mode", // Hoge Screen // … },

Slide 61

Slide 61 text

Preparations with Gemini

Slide 62

Slide 62 text

Use Case 2: Using IDE-Integrated AI Camera Preview "EEJOHUIFDBNFSBQBDLBHF 8SJUJOHUIFDBNFSBMPHJD #VJMEJOHUIF6*GPSQSFWJFXEJTQMBZ BOENPSF

Slide 63

Slide 63 text

Accelerating with Cursor

Slide 64

Slide 64 text

Important Issues to Consider User Permission 6TJOH4QFDJBM'FBUVSFTMJLF$BNFSB 4IPXJOHB1FSNJTTJPO%JBMPH )BOEMJOHUIF3FTVMU 0OFPGUIFLFZEJ ff FSFODFT

Slide 65

Slide 65 text

Accelerating with Cursor

Slide 66

Slide 66 text

Best Practices Based on Our Experience

Slide 67

Slide 67 text

Best Practices based on my experience •Behaviours depend on each device →Testing on many devices is essential •Many operations related on AR are heavier →Performance Tuning is also crucial •On-Device vs Cloud: Considerations →Depending on Use Case

Slide 68

Slide 68 text

Reference X: @korodroid

Slide 69

Slide 69 text

AR Features Can Make Your App More Engaging 🚀 Conclusion

Slide 70

Slide 70 text

Facebook:http://fb.com/kanbara.kenichi X:@korodroid LinkedIn:http://www.linkedin.com/in/korodroid Thank you so much Feel free to reach out for technical speeches, writing, or other inquiries