Can LLMs Replicate 4 Years of Compose Migration? Exploring the boundaries of automation with 279 XML files from a real product

Information : https://2026.droidkaigi.jp/en/timetable/1231173/

日本語版 : https://speakerdeck.com/makun/makun-20260902-droidkaigi-2026-1

Transcript

1. Can LLMs Replicate 4 Years of Compose Migration? Exploring the boundaries of automation with 279 XML files from a real product Masahiro Saito
2. Masahiro Saito pixiv Inc. Mobile App Engineer and Engineering Recruitment Lead Responsible for planning and executing the Compose migration for the app covered in this session 2
3. Last year, I talked about leaving testing to AI DroidKaigi 2025 "Stop Writing Test Code: Automated Test Design and Generation for Asynchronous Processing with JetBrains AI Assistant" 3
4. I Tried a Multi-Year Compose Migration Again, but with an LLM Starting point With today's LLMs, couldn't much of this be automated? ▼ As I investigated There were several common patterns behind why it did not work ▼ The question I arrived at So how should we delegate the migration to AI efficiently? 4
5. Today's Agenda 1 What We Tested 2 Experiment Results 3 Discussion of the Results 4 Can We Delegate It If We Fill in the Missing Pieces? 5 Is It Worth Building a System for It? 6 Summary 5
6. The Target Is a Production App Palcy is a manga app co-developed by pixiv inc. and Kodansha Ltd. It has been in operation since 2018 and reached its 8th anniversary this August. The Compose migration had already been completed manually. In other words, we tested how far AI could handle XML migration in a real product, not sample code. © Kodansha Ltd. 6
7. Scope Current state Code already migrated to Compose Test target Roll back ▶ All 279 pre-migration XML layouts 7
8. Today's Agenda 1 What We Tested 2 Experiment Results 3 Discussion of the Results 4 Can We Delegate It If We Fill in the Missing Pieces? 5 Is It Worth Building a System for It? 6 Summary 8
9. Today's Questions 1 Can an LLM write Compose code? 2 Can it fix build errors on its own? 3 Can it make the visuals closer to the original? 4 Compared with the Compose implementation migrated by humans, what is missing? 9
10. The migration to Compose has already been completed, but... We do not provide information that is too close to the answer, such as the actual Compose implementation or Design System. What we want to observe is how far the LLM can get using only the information provided. We evaluate it using only a predefined scope of information. 10
11. That’s the Whole Validation Flow Input XML + limited context ▼ Generation The LLM generates Compose code ▼ Fix 1 Feed back compile errors and have the LLM fix them ▼ Fix 2 Show screenshots and have the LLM fix the result ▼ Answer check Compare against the actual migration result 11
12. Start with 30 Cases, Then Expand to All 279 Pilot validation Rollout Final scope 30 cases 249 cases 279 cases Compare approaches and decide on the method → Apply the same method to the remaining XML layouts → The entire set, not just a demo 12
13. Today's Agenda 1 What We Tested 2 Experiment Results 3 Discussion of the Results 結果Results 4 Can We Delegate It If We Fill in the Missing Pieces? 5 Is It Worth Building a System for It? 6 Summary 13
14. 279/279 Including the step of feeding errors back for fixes, 100% achieved a build PASS 14
15. Some Problems Can Be Fixed by Feeding Back Results and Iterating Highly effective Partial improvement build error size and layout When given the error log, the LLM could fix them itself Showing a screenshot sometimes made the result closer to the original 15
16. Today's Agenda 1 What We Tested 2 Experiment Results 3 Discussion of the Results 4 Can We Delegate It If We Fill in the Missing Pieces? 5 Is It Worth Building a System for It? 6 Summary 16
17. We Cannot Call This a "Successful Migration" Build PASS 100% ≠ Production-ready 100% 17
18. We Do Not Judge It Only by Whether It "Looks Similar" Compare in the same environment as the human migration result Review not only the image, but also structure, state, and implementation intent Also check whether the human chose a one-to-one replacement 18
19. If the Required Information Is Contained in the XML, the Overall Structure Is Reproduced Well Human migration LLM-genera ted * The original XML could not be captured due to Paparazzi limitations, so the manually migrated implementation was used as the comparison baseline 19
20. Case Where the Included Layout Was Not Provided Original XML <include> × 8 LLM-generated repeat(8) + // TODO <LinearLayout ...> <include ... /> <include ... /> <include ... /> ... (8 total) </LinearLayout> Column(...) { repeat(8) { index -> // TODO: generate rank{index+1} from // the included @layout/... later Box(...) { Text("rank${index + 1}") } } } → 20
21. The Visuals Were Close, but the Structure Was Missing Human migration LLM-generated 21
22. A Screen Containing a WebView In this experiment, AndroidView was prohibited by our rules. In practice, using the existing WebView from Compose is also an option. ＝ An effect of the rules we chose 22
23. A Screen Using MotionLayout In the actual human migration, the entire screen was redesigned rather than simply reproducing the XML. ＝ A product/specification decision about what should be built 23
24. Why Human Help Was Needed Missing input information resource / Design System Screen state determined at runtime Constraints imposed by the experiment rules Insufficient automated evaluation methods Scope decisions / final acceptance or rejection 24
25. Today's Agenda 1 What We Tested 2 Experiment Results 3 Discussion of the Results 4 Can We Delegate It If We Fill in the Missing Pieces? 5 Is It Worth Building a System for It? 6 Summary 25
26. Is the LLM's Capability Really the Cause? include case The included layout was not provided WebView case AndroidView was prohibited by the rules Could the cause be the input and rule settings rather than the LLM itself? 26
27. Providing the Included Layout Increased What the LLM Could Generate before repeat(8) + // TODO after repeat(8) + Row Column(...) { repeat(8) { index -> // TODO: generate rank{index+1} from // the included @layout/... later Box(...) { Text("rank${index + 1}") } } } repeat(8) { Row(...) { Box(...) Column(...) { ... // only tools: // text is available // the actual text is not // present in the input Text(text = "", ...) } } } → 27
28. raw pixel diff (percentage of pixels that differ in the image) Original XML LLM-generated differing pixels before 70.75% ↓ after providing the included layout 32.35% 28
29. By Adding the tools:text, the Content Correct Original XML before (generated Compose) after (after adding the text) → 29
30. The Content Improved, but the Metrics Got Worse → before → after raw pixel diff：4.34% → 9.20% before after edge SSIM：0.871 → 0.719 30
31. A Single Image Diff Metric Can’t Distinguish Between Diff Types Is the content correct? Does the visual theme match? Does the layout match? Is the difference caused by the comparison method itself? 31
32. The Automated Evaluation Itself Also Requires Validation 32
33. Full Automation Requires More Than Just "Generation" Input collection ▶ Generation ▶ Build ▶ Evaluation ▶ Accept / Reject 33
34. Today's Agenda 1 What We Tested 2 Experiment Results 3 Discussion of the Results 4 Can We Delegate It If We Fill in the Missing Pieces? 5 Is It Worth Building a System for It? 6 Summary 34
35. Building Mechanisms One by One Could Further Reduce Human Work Input Completion Evaluation Automatically collect related files Automatically add missing values to the input Automatically evaluate generated results 35
36. Compose Migration Is Basically a One-Way Process XML → Compose ↩ Once a screen has been migrated, it is not converted from XML again 36
37. Is Building an Additional Dedicated System Worth the Development Time? 37
38. Compare Doing It Manually Every Time with Building a System If humans do it every time Number of cases × human work time required per case If we build a dedicated system Time to build the system + human work that remains even after automation 38
39. The Number of Cases Likely to Need Additional Handling Varies Greatly by Factor tools_only_content 151 / 279 style_refs 77 / 279 runtime_list_content 74 / 279 custom_view 38 / 279 include_dependency 10 / 279 39
40. Even Without a Dedicated System, Work with AI Instead of Writing Everything by Hand Ask AI to migrate in small units → Review the result → Give additional instructions / make manual fixes if needed 40
41. Most of the Time Was Spent on Review 27 min 21 min Check and collect required information 10 min 20 min 10 min 3 min Case 1 Case 2 Case 3 Instructions to the LLM Review generated results Final check Direct code edits 0 min for all three cases 41
42. Where Could a System Reduce the Work? Check and collect required information Instructions to the LLM Across the three cases 5-10 minutes per case 42
43. First, Calculate Assuming It Can Be Built in One Day Build time (1 day = 8 hours) 480 min Time saved per case ÷ 5-10 min ＝ Break-even number of cases 48-96 The more time it takes to build, the more cases are required to justify it 43
44. Even for the Same Kind of "Fixable by Adding Missing Information" Problem, the Answer Can Differ include Candidate cases: 10 Human input: 1.5 min (one case) Simple extrapolation: roughly 15 min total The development cost we can justify is quite small 44
45. Even for the Same Type of "Problem That Improves If We Add Missing Information," the Answer Changes tools Candidate cases: 151 Human input: 6 min (one case) Simple extrapolation: about 15 hours total Further validation may be highly worthwhile 45
46. It Is Not Enough to Ask Only, "Can We Improve This Technically?" We also need to consider how many cases it affects and how many minutes of human work it can save 46
47. Today's Agenda 1 What We Tested 2 Experiment Results 3 Discussion of the Results 4 Can We Delegate It If We Fill in the Missing Pieces? 5 Is It Worth Building a System for It? 6 Summary 47
48. This Experiment Suggested a Better Way to Delegate Work to LLMs 1 Classify the characteristics of the XML 2 Gather the information required for each characteristic 3 Ask AI to perform the migration using a method suited to those characteristics 4 Have a human review the generated result 48
49. 1. First, Classify the XML Characteristics All required information is contained in the XML Information from included layouts is required Values exist only in tools attributes Content is determined at runtime Uses custom Views WebView / MotionLayout …… 49
50. 2. Change What We Give AI Based on the Characteristics Has include → Also provide the included XML Has values in tools → Also provide the required sample values Content is determined at runtime → Add state and data information WebView, etc. → Revisit the allowed implementation methods and rules Requires redesign → A human decides the direction 50
51. 3. Consider Further Automation Only for Repeated Work Identifying a pattern ≠ Building a system to automate everything Decide based on: number of cases × minutes saved per case 51
52. The Process We Arrived at for This Compose Migration XML ↓ Classify characteristics ↓ Gather the required information ↓ Ask AI to migrate based on the classification ↓ Human review ↘ For highly repetitive cases → Consider additional automation 52
53. Change How You Delegate to AI for Each Pattern. Decide Whether to Build a System Based on Scale. 53
54. Three Things to Consider When Delegating Work to AI 1 2 3 Separate the reasons it does not work Change how you delegate based on the reason Consider automation only for the repetitive parts What information, rules, or decisions are required? How many cases will it affect, and how much human work can it reduce? What is preventing AI from completing the task on its own? 54
55. Masahiro Saito 55