Compose Multiplatform × AI で作る、次世代アプリ開発支援ツールの設計と実装

近年、コード補完にとどまらず、LLM を活用してアプリケーション全体を支援・構築する取り組みが注目されています。本セッションでは、私が開発中の Compose Multiplatform ベースのデスクトップアプリ「ComposeFlow」の開発を通じて得られた、以下のような知見を共有します。

ComposeFlow は、モバイルアプリ開発の敷居を下げることを目指した AI ファーストなビジュアル開発環境です。ユーザーが自然言語で指示を出すと、LLM がそれを読み取り、アプリ構成の変更操作（例：UIコンポーネントの追加や削除、プロジェクト設定の変更など）を行います。このために、エディタ内のあらゆる操作を「tool」として定義し、LLM が直接呼び出せる仕組みを備えています。

本セッションでは、こうした AI に操作されることを前提とした開発環境を構築するうえでの技術的チャレンジやアーキテクチャ設計について掘り下げます。

具体的には：

- Android/iOS/Web アプリを対象としたビジュアルエディタを Compose Multiplatform で構築する方法

- アプリの構成情報を中間表現（YAML）として扱い、AI による生成や編集を可能にする仕組み

- GUI の各操作を API として定義し、LLM がそれらを tool として呼び出せる構造

- Cline や Cursor のようなコード補完ではなく、GUI を直接 AI に操作させるアプローチ

- Agent にどのようにツール群を公開するか、信頼性の担保や失敗時の再試行戦略などの運用知見

ComposeFlow 自体は現在クローズドソースですが、将来的にはオープンソースとして公開を予定しています。本セッションはプロダクトの紹介ではなく、「AI × GUI × Compose Multiplatform × Agent API 設計」という文脈での技術知見を共有することを目的としています。

Transcript

1. Compose Multiplatform × AI で作る、次世代アプリ 開発 支 援ツールの設計と実装 Design and Implementation of a Next-Generation App-Development Assistant Built with Compose Multiplatform and AI @thagikura Takeshi Hagikura
2. @thagikura Takeshi Hagikura Who am I? - 2023 Google Android Developer Relations 2025 - Founded ComposeFlow
3. Today’s goals 1. How visual editor works with Compose Multiplatform 2. How an AI agent works with the visual editor 3. Tips for making agentic workflows robust
4. Bit of intro of ComposeFlow Built-in AI agent Able to ask AI agent to any creation/modifications of the app Visual Editor Able to tweak any part of the app using the visual editor with real time preview Powered by Compose Multiplatform Generated apps and the ComposeFlow itself are powered by Compose Multiplatform Backend integration Firebase (Auth, Firestore and more features to come) External API integration
5. Before we begin…
6. Source code available at https://github.com/ComposeFlow/ComposeFlow
7. How visual editor works with Compose Multiplatform
10. Column Row Text Text Text Row Column Column Text Text Text Text Row Button Button Represents the node as tree structure
11. data class ComposeNode( val children: MutableList<ComposeNode> ) Each node is represented as Kotlin’s data class
12. data class ComposeNode( val children: MutableList<ComposeNode> ) { @Composable fun RenderedNodeInCanvas } It knows how to render it as Composable
13. data class ComposeNode( val children: MutableList<ComposeNode> = mutableStateListOf() ) { @Composable fun RenderedNodeInCanvas } Tree structure is represented using MutableStateList
14. When the tree structure is changed with a drag-drop operation Column Row Text Text Button
15. Recomposition happens real time. Because ComposeFlow is a Compose Multiplatform app Column Row Text Text Button
16. data class ComposeNode( val children: MutableList<ComposeNode> = mutableStateListOf() val modifier: MutableList<ModifierWrapper> = mutableStateListOf() ) { … } Modifier is expressed as MutableStateList as well
17. Our modifier wrapper knows how to express it as (Compose) Modifier sealed class ModifierWrapper( … ) { data class Alpha( val alpha: Float = 1f, ) : ModifierWrapper() { override fun toModifier(): Modifier = Modifier.alpha(alpha) data class Height( val height: Dp, ) : ModifierWrapper() { override fun toModifier(): Modifier = Modifier.height(height = height) }
18. Editor operations incur Recomposition in the modifier wrappers as well
19. By the way, as you probably know modifier order matters
21. From list of modifier wrappers, generating a combined Modifier using fold operator fun List<ModifierWrapper>.toModifierChain(): Modifier { val initial: Modifier = Modifier return fold(initial = initial) { acc, element -> acc.then(element.toModifier()) } } Different order changes the order of each modifier is applied
22. Actual Compose Modifier Modifier.width(100.dp) .height(120.dp) .padding(8.dp) .background(Color.Gray)
23. Actual Compose Modifier Modifier.width(100.dp) .height(120.dp) .padding(8.dp) .background(Color.Gray) class CombinedModifier( internal val outer: Modifier, internal val inner: Modifier) : Modifier { override fun <R> foldIn( initial: R, operation: (R, Modifier.Element) -> R): R = inner.foldIn(outer.foldIn(initial, operation), operation) } Translated as a combined Modifier Internally
24. How code generation works
25. It also supports see the code side by side
27. Or preview it in emulator or browser
28. data class ComposeNode( val children: MutableList<ComposeNode> ) { fun generateCode( … ): CodeBlock } Each node knows how to express itself as Kotlin code
29. data class ComposeNode( val children: MutableList<ComposeNode> ) { fun generateCode( … ): CodeBlock } Using KotlinPoet API
30. Column Row Text Text Button Column( modifier = Modifier.fillMaxWidth(), ) { // Code from children }
31. Column Row Text Text Button Button( onClick = { }, enabled = true, modifier = Modifier .padding(all = 8.dp), ) { Text( text = "Example button", ) }
32. data class ComposeNode( val children: MutableList<ComposeNode> ) { fun generateCode( … ): CodeBlock { val codeBlockBuilder = CodeBlock.builder() children.forEach { codeBlockBuilder.add( it.generateCode(…), ) } } Code generation happens recursively
33. Built-in template for Compose Multiplatform
34. Built-in template for Compose Multiplatform Generate code (mostly) in commonMain from ComposeFlow’s app structure
35. @Serializable data class ComposeNode( val children: MutableList<ComposeNode> ) Marked as @Serializable using Kotlin Serialization
36. @Serializable data class ComposeNode( val children: MutableList<ComposeNode> ) - id: "column_1" trait: !<ColumnTrait> { } label: "Column" level: 1 modifierList: - !<FillMaxWidth> { } - !<Weight> { } children: - id: "searchTextField" trait: !<TextFieldTrait> value: !<ValueFromCompanionState> { } placeholder: !<StringIntrinsicValue> value: "Search replies" leadingIcon: !<Outlined> "Search" trailingIcon: !<Outlined> "Close" singleLine: null shapeWrapper: !<Circle> { } transparentIndicator: true And serialized into Yaml
37. @Serializable data class Project( val screenHolder:
 … ScreenHolder
 … ) @Serializable data class ScreenHolder( val screens: MutableList<Screen> ) @Serializable data class Screen( val rootNode: MutableState<ComposeNode> )
38. @Serializable data class Project( … ScreenHolder
 … ) Entire project is also represented as a Yaml file id: “project-1" name: "blankproject" screenHolder: screens: - id: “screen-1" name: "Home" rootNode: id: “rootNode-1" trait: !<ScreenTrait> { } label: "Home" children: - id: "rootColumn" trait: !<ColumnTrait> { } label: "Root" level: 1 modifierList: - !<FillMaxWidth> { } - !<Weight> { } children: - id: "button1" trait: !<ButtonTrait> textProperty: ! value: "Button" enabled: value: true label: "Button"
39. - To enable visual editing (add metadata on top of the UI tree structure) - Due to the declarative nature of Compose App structure could be modeled declaratively Why Yaml?
40. Think of a simplest case - TextField - Text is changed with the same value as TextField
41. Can be abstracted as - String State (from TextField) - Text’s value is assigned from the state
42. val textField by blankViewModel .textFieldFlow.collectAsState() Text( text = textField, ) TextField( value = textField, onValueChange = { blankViewModel .onTextFieldUpdated(it) }, ) private val _textFieldFlow: MutableStateFlow<String> = MutableStateFlow("") public val textFieldFlow: StateFlow<String> = _textFieldFlow public fun onTextFieldUpdated( newValue: String ) { _textFieldFlow.value = newValue } In @Composable In ViewModel
43. Everything may not be abstracted that simply In that case, embedding custom code will be possible (Now we have official Kotlin LSP)
44. Summary 1. Each node is represented as Kotlin’s data class with tree structure
45. Summary 1. Each node is represented as Kotlin’s data class with tree structure 2. Drag-drop operations modifies the tree data structure
46. Summary 1. Each node is represented as Kotlin’s data class with tree structure 3. Each node knows how to render itself as @Composable and how to generate Kotlin code 2. Drag-drop operations modifies the tree data structure
47. Summary 1. Each node is represented as Kotlin’s data class with tree structure 3. Each node knows how to render itself as @Composable and how to generate Kotlin code 4. Generate code in commonMain from the project 2. Drag-drop operations modifies the tree data structure
48. Summary 1. Each node is represented as Kotlin’s data class with tree structure 3. Each node knows how to render itself as @Composable and how to generate Kotlin code 4. Those classes are marked as @Serializable and deserialized to Yaml 2. Drag-drop operations modifies the tree data structure 5. Those classes are marked as @Serializable and deserialized to Yaml
49. How an AI agent works with the visual editor
50. How an AI agent works with the visual editor Mostly focuses on client efforts in this session
51. What are AI agents …According to Anthropic Programs where freeform LLMs outputs control the workflow
54. fun onAddComposeNodeToContainerNode( project: Project, containerNodeId: String, composeNode: ComposeNode, indexToDrop: Int, ) Method used in visual editor for dropping a new Composable
55. fun onAddComposeNodeToContainerNode( project: Project, containerNodeId: String, composeNodeYaml: String, indexToDrop: Int, ) Define an equivalent method for LLM
56. fun onAddComposeNodeToContainerNode( project: Project, containerNodeId: String, composeNodeYaml: String, indexToDrop: Int, ) { … val composeNode: ComposeNode = decodeFromString(composeNodeYaml) … } Instead of ComposeNode, receive YAML as String
57. @LlmTool( name = "add_compose_node_to_container", description = "Adds a Compose UI component to a container node in the UI builder…”, ) fun onAddComposeNodeToContainerNode( project: Project, @LlmParam( description = "The ID of the container node where the component…”, ) containerNodeId: String, @LlmParam(
 description = "The YAML representation of the ComposeNode node…” ) composeNodeYaml: String, @LlmParam( description = "The position index where the component should be inserted…”, ) indexToDrop: Int, Add annotation for method definition and parameters
58. ./gradlew runKsp <===========--> 86% EXECUTING [10s] > IDLE > IDLE > IDLE Then define KSP for processing annotations
59. { "name": "add_compose_node_to_container", "description": "Adds a Compose UI component to a container node in the UI builder…”, "input_schema": { "type": "object", "properties": { "containerNodeId": { "type": "string", "description": "The ID of the container node where the component will be…” }, "composeNodeYaml": { "type": "string", "description": "The YAML representation of the ComposeNode node to be added…” }, "indexToDrop": { "type": "number", "description": "The position index where the component should be inserted…” } }, "required": [ "containerNodeId", "composeNodeYaml", "indexToDrop" ] } }
60. import { Anthropic } from '@anthropic-ai/sdk'; const anthropic = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY }); async function main() { const response = await anthropic.messages.create({ model: <model-name>, max_tokens: 1024, tools: [{ … }] } In server for calling LLM Pass the generated json (in Claude’s example in this case)
61. fun onAddComposeNodeToContainerNode( project: Project, containerNodeId: String, composeNodeYaml: String, indexToDrop: Int, ) { … val composeNode: ComposeNode = decodeFromString(composeNodeYaml) … } But how can we let LLMs generate the parsable YAML?
62. @Serializable data class ComposeNode( val children: MutableList<ComposeNode> ) For every Serializable data class
63. @Serializable data class ComposeNode( val children: MutableList<ComposeNode> ) export interface ComposeNode { children?: ComposeNode[]; } Convert Kotlin to TypeScript
64. export interface ComposeNode { children?: ComposeNode[]; } Then convert TypeScript to JSON schema “ComposeNode": { "properties": { "children": { "items": { "$ref": "#/definitions/ComposeNode" }, "type": "array" }, }, "type": "object" }
65. Then include JSON schema as part of prompt to LLM You are a helpful assistant for ComposeFlow, a visual development platform for Compose Multiplatform. **Your Task:** … By following the JSON schema “ComposeNode": { "properties": { "children": { "items": { "$ref": "#/definitions/ComposeNode" }, "type": "array" }, }, "type": "object" } …

66. Then include JSON schema as part of prompt to LLM You are a helpful assistant for ComposeFlow, a visual development platform for Compose Multiplatform. **Your Task:** … By following the JSON schema “ComposeNode": { "properties": { "children": { "items": { "$ref": "#/definitions/ComposeNode" }, "type": "array" }, }, "type": "object" } …
 If you do so, make sure to cache the prompt!
67. Side note A: Koog also supports creating JSON schema from data classes @Serializable @SerialName("SimpleWeatherForecast") @LLMDescription("Simple weather forecast for a location") data class SimpleWeatherForecast( @property:LLMDescription("Location name") val location: String, @property:LLMDescription("Temperature in Celsius") val temperature: Int, @property:LLMDescription("Weather conditions (e.g., sunny, cloudy, rainy)") val conditions: String ) JsonStructuredData.createJsonStructure<SimpleWeatherForecast>( schemaGenerator = BasicJsonSchemaGenerator.Default, examples = exampleForecasts )
68. Side note B: 1/2 Tried also VectorDB as embedding, but… “ComposeNode": { "properties": { "children": { "items": { "$ref": "#/definitions/ComposeNode" }, "type": "array" }, }, "type": "object" } query_embd = vo.embed( [query], model=“<model>", input_type="query" ).embeddings[0]
69. Side note B: 2/2 Including JSON schema in prompt was better In my case “ComposeNode": { "properties": { "children": { "items": { "$ref": "#/definitions/ComposeNode" }, "type": "array" }, }, "type": "object" } query_embd = vo.embed( [query], model=“<model>", input_type="query" ).embeddings[0]
70. Example Agentic workflow
71. Add navigation to the screen LLMs ComposeFlow
72. Add navigation to the screen LLMs ComposeFlow Append summarized app structure
73. LLMs { … "tool_calls": [ { "id": "toolu_01Bkbq4YKf4JafspQHvCTtT9", "index": 0, "type": "function", "function": { "name": "add_compose_node_to_container", "arguments": "{\"containerNodeId\": ... \”indexToDrop\": 0}" } } ], … } Generate a tool call to add NavigationDrawer
74. LLMs ComposeFlow Tool calls
75. LLMs ComposeFlow Tool calls Parse and dispatch the tool call
76. LLMs ComposeFlow Give LLMs feedback and keep iterating
77. LLMs ComposeFlow No more tool calls
78. LLMs ComposeFlow No more tool calls ✅ We are done!
79. Thoughts after building it - Other coding agents (Claude Code, Codex, etc) have similar mechanisms. They have tools like bash, gh, replace_file, etc. - Building an AI agent is not something to fear. Think of it as a simple while loop
80. Summary 1. Expose the same method as visual editor with expected parameters from LLMs
81. Summary 1. Expose the same method as visual editor with expected parameters from LLMs 2. Each exposed parameter should be parsable String instead of Kotlin classes
82. Summary 1. Expose the same method as visual editor with expected parameters from LLMs 2. Each exposed parameter should be parsable String instead of Kotlin classes 3. Annotate the method and parameters for KSP to produce JSON for tool cals
83. Summary 1. Expose the same method as visual editor with expected parameters from LLMs 2. Each exposed parameter should be parsable String instead of Kotlin classes 3. Annotate the method and parameters for KSP to produce JSON for tool cals 4. Pass the exposed methods as tool calls to LLMs
84. Summary 1. Expose the same method as visual editor with expected parameters from LLMs 2. Each exposed parameter should be parsable String instead of Kotlin classes 3. Annotate the method and parameters for KSP to produce JSON for tool cals 4. Pass the exposed methods as tool calls to LLMs 5. Define expected schemas for LLM (We follow @Serializable > TypeScript > JSON schema)
85. Summary 1. Expose the same method as visual editor with expected parameters from LLMs 2. Each exposed parameter should be parsable String instead of Kotlin classes 3. Annotate the method and parameters for KSP to produce JSON for tool cals 4. Pass the exposed methods as tool calls to LLMs 5. Define expected schemas for LLM (We follow @Serializable > TypeScript > JSON schema) 6. Let LLMs invoke tools given user’s request + current app structure
86. Summary 1. Expose the same method as visual editor with expected parameters from LLMs 2. Each exposed parameter should be parsable String instead of Kotlin classes 3. Annotate the method and parameters for KSP to produce JSON for tool cals 4. Pass the exposed methods as tool calls to LLMs 5. Define expected schemas for LLM (We follow @Serializable > TypeScript > JSON schema) 6. Let LLMs invoke tools given user’s request + current app structure 7. Parse and dispatch the tool calls. And keep iterating until user’s request is completed!
87. Knowledge for making agentic workflows reliable and robust
88. Give LLMs as much detailed feedback as possible
89. LLMs often make mistakes
90. Even with Claude Code
91. But in many cases they fix the issue on its own
92. And the same thing goes for ComposeFlow
93. ❌ Instead of sending an error feedback like: Error adding compose node to container
94. ✅ Send detailed feedback like: Value for 'textStyleWrapper' is invalid: Value for 'value' is invalid: Unknown type 'EnumWrapper'. Known types are: ContentScaleWrapper, FontStyleWrapper, NodeVisibility, TextAlignWrapper, TextDecorationWrapper, TextFieldColorsWrapper, TextOverflowWrapper, TextStyleWrapper at line 29, column 12 (path: children[0].children[0].trait) Context: 28: value: Caused by: io.composeflow.serializer.YamlLocationAwareException: Value for 'textStyleWrapper' is invalid: Valu for 'value' is invalid: Unknown type 'EnumWrapper'. Known types are: ContentScaleWrapper, FontStyleWrapper, NodeVisibility, TextAlignWrapper, TextDecorationWrapper, TextFieldColorsWrapper, TextOverflowWrapper, TextStyleWrapper at line 29, column 12 (path: children[0].children[0].trait) Ask Gemini 29: scalar @ YamlPath(segments=[com.charleskorn.kaml.YamlPathSegment$Root@6bfe9b57, MapElementKey(key=children, location=Location(line=7, column=1)), MapElementValue(location=Location(line=8 column=1)), ListEntry(index=0, location=Location(line=8, column=3)), MapElementKey(key=children, location=Location(line=27, column=3)), MapElementValue(location=Location(line=28, column=3)), ListEntry(index=0, location=Location(line=28, column=5)), MapElementKey(key=trait, location=Location(line=29 column=5)), MapElementValue(location=Location(line=29, column=12)), MapElementKey(key=colorWrapper,
95. In ComposeFlow, default Yaml serializer didn’t have the location where deserialization error happened
96. ✅ Implemented extended serializer to spot the location where deserialization error happens fun <T> KSerializer<T>.withLocationAwareExceptions(): LocationAwareSerializerWrapper<T> = LocationAwareSerializerWrapper(this) class LocationAwareSerializerWrapper<T>( private val delegate: KSerializer<T>, ) : KSerializer<T> { … }
97. Falls back to default value may be better user experience
98. In case of an deserialization error, it may be better to have a fallback value
99. For example we have Enum like @Serializable enum class TextAlignWrapper( val textAlign: TextAlign, ) : EnumWrapper { Left(TextAlign.Left), Right(TextAlign.Right), Center(TextAlign.Center), Justify(TextAlign.Justify), Start(TextAlign.Start), End(TextAlign.End), ; }
100. Even though we pass the JSON schema for enums as well, sometimes invalid values are sent Value for ‘textAlignWrapper' is invalid: Unknown type 'EnumWrapper'. Known types are Left, Right, Center, Justify, Start, End
101. In some cases, having a default value is better than letting LLMs retry @Serializable(TextAlignWrapper.TextAlignWrapperSerializer::class) enum class TextAlignWrapper( val textAlign: TextAlign, ) : EnumWrapper { Left(TextAlign.Left), Right(TextAlign.Right), Center(TextAlign.Center), Justify(TextAlign.Justify), Start(TextAlign.Start), End(TextAlign.End), ;
 object TextAlignWrapperSerializer : FallbackEnumSerializer<TextAlignWrapper>
 (TextAlignWrapper::class) }
102. Resources https://github.com/ComposeFlow/ComposeFlow/ 📖 https://docs.composeflow.io/
103. Thank you for your time!