Share
Embed
Slide 1
Slide 1 text
DROIDKAIGI 2026 · TOKYO
How we moved from
Android Studio to Slack
Four months of production data from an
AI agent that fixes bugs where the bug
is reported.
Ahmed Ali
neverstop
Slide 2
Slide 2 text
HELLO
Ahmed Ali
12 years writing code
Mobile engineer
in/thisaay
Developer tools
At Infinum
DroidKaigi 2024
Cairo, Egypt
My passion is building developer
tools, and that is why I am
here today.
neverstop
02
Slide 3
Slide 3 text
I know you are tired of hearing AI, AI,
AI everywhere.
I am tired too.
So this talk has no concepts and no
predictions in it. Only what we ran in
production, and what it actually cost.
neverstop
03
Slide 4
Slide 4 text
How long does it take to fix a
one-line bug?
04
Slide 5
Slide 5 text
01
The classic cycle
Slide 6
Slide 6 text
HOW A BUG GOT FIXED
The classic cycle
01
neverstop
QA finds a bug
06
Slide 7
Slide 7 text
HOW A BUG GOT FIXED
The classic cycle
01
02
neverstop
QA finds a bug
Writes a ticket
06
Slide 8
Slide 8 text
HOW A BUG GOT FIXED
The classic cycle
01
02
03
neverstop
QA finds a bug
Writes a ticket
Waits to be assigned
06
Slide 9
Slide 9 text
HOW A BUG GOT FIXED
The classic cycle
01
02
03
04
neverstop
QA finds a bug
Writes a ticket
Waits to be assigned
A dev opens the project
06
Slide 10
Slide 10 text
HOW A BUG GOT FIXED
The classic cycle
01
02
03
04
05
neverstop
QA finds a bug
Writes a ticket
Waits to be assigned
A dev opens the project
Fix and open a PR
06
Slide 11
Slide 11 text
HOW A BUG GOT FIXED
The classic cycle
01
02
03
04
05
06
neverstop
QA finds a bug
Writes a ticket
Waits to be assigned
A dev opens the project
Fix and open a PR
Waits for review
06
Slide 12
Slide 12 text
Nothing here is wrong.
For a real feature, this is exactly
right.
neverstop
06
Slide 13
Slide 13 text
THE SAME CYCLE
Where the time actually goes
01
02
QA finds a bug
HANDOFF
Writes a ticket
HANDOFF
03
04
Waits to be assigned
HANDOFF
A dev opens the project
HANDOFF
05
06
neverstop
Fix and open a PR
HANDOFF
Waits for review
07
Slide 14
Slide 14 text
For a small bug, the handoffs
cost more than the fix.
07
Slide 15
Slide 15 text
SCOPE · 1 OF 3
The bugs were small
Some logic not working
A total that does not add up. A state that does
not reset.
neverstop
08
Slide 16
Slide 16 text
SCOPE · 2 OF 3
The bugs were small
UI that does not match the design
Wrong colour token, wrong icon, wrong
spacing.
neverstop
08
Slide 17
Slide 17 text
SCOPE · 3 OF 3
The bugs were small
Navigation issues
Back goes to the wrong place. A screen you
cannot leave.
neverstop
08
Slide 18
Slide 18 text
None of this is architecture.
Most of it is one line.
neverstop
08
Slide 19
Slide 19 text
02
The script
Slide 20
Slide 20 text
We were short on release. Still had
features to develop, and bugs to fix,
and sometimes really small bugs.
neverstop
10
Slide 21
Slide 21 text
MY WORKAROUND
What I did instead
01
neverstop
Copy the QA description
11
Slide 22
Slide 22 text
MY WORKAROUND
What I did instead
01
Copy the QA description
02
Leave my own worktree
neverstop
11
Slide 23
Slide 23 text
MY WORKAROUND
What I did instead
01
Copy the QA description
02
Leave my own worktree
Create a new one
03
neverstop
11
Slide 24
Slide 24 text
MY WORKAROUND
What I did instead
01
Copy the QA description
02
03
Leave my own worktree
Create a new one
04
Give Claude the same text
neverstop
11
Slide 25
Slide 25 text
MY WORKAROUND
What I did instead
01
Copy the QA description
02
Leave my own worktree
Create a new one
03
04
05
neverstop
Give Claude the same text
Go back to my worktree
11
Slide 26
Slide 26 text
It was faster than fixing it myself.
It was still a context switch.
neverstop
11
Slide 27
Slide 27 text
Claude fixed the bug.
I still switched context.
neverstop
12
Slide 28
Slide 28 text
THE FIRST VERSION
So I cut out the middle man
01
neverstop
Mention it in Slack
13
Slide 29
Slide 29 text
THE FIRST VERSION
So I cut out the middle man
01
Mention it in Slack
02
It starts a session in a fresh worktree
neverstop
13
Slide 30
Slide 30 text
THE FIRST VERSION
So I cut out the middle man
01
Mention it in Slack
02
It starts a session in a fresh worktree
It opens a PR, I review later
03
neverstop
13
Slide 31
Slide 31 text
∞
neverstop
AhmedBot
Named after me, because all it did was what I had been
doing by hand
13
Slide 32
Slide 32 text
THE FIRST THREAD
Message 1 of 3
QA WRITES IT THE WAY SHE ALWAYS DID
M
Mia 5:11 PM
Only one gift card showing in till summary
Pay one basket with two gift cards, 15 and 30. The till
summary shows 30 instead of 45.
neverstop
14
Slide 33
Slide 33 text
THE FIRST THREAD
Message 2 of 3
BUT A DEVELOPER STILL HAD TO POINT AT THE CODE
A
Ahmed 5:14 PM
I think we need to look into
FiscalDayTrackingServiceImpl and check why there is
an issue calculating the gift card
@AhmedBot fix
neverstop
14
Slide 34
Slide 34 text
THE FIRST THREAD
Message 3 of 3
IT ANSWERS WITH THE CAUSE, NOT JUST A PATCH
∞
AhmedBot APP 5:14 PM
Implementing the fix
Running Detekt (lint check)
Creating pull request
Root cause: the gift-card branch accumulated onto
creditCardAmount , so every new gift card restarted the sum.
Fix: one line — accumulate onto giftCardAmount instead.
PR opened
neverstop
14
Slide 35
Slide 35 text
WHAT I WROTE AT THE TIME
The bot participated in around 7 bugs. It fixed most of them, and
when it failed, it still provided a strong starting point. In some
cases, it quickly proved that a specific solution was not correct,
saving time.
neverstop
15
Slide 36
Slide 36 text
7
FIRST WEEK
bugs handled by a script one person
wrote in the middle of a release.
neverstop
16
Slide 37
Slide 37 text
03
When a script becomes
infrastructure
Slide 38
Slide 38 text
“Can you start AhmedBot? I
know you are on vacation.”
18
Slide 39
Slide 39 text
WHAT THAT MESSAGE MEANT
Five constraints
01
neverstop
Only ran while my laptop was awake
19
Slide 40
Slide 40 text
WHAT THAT MESSAGE MEANT
Five constraints
01
02
neverstop
Only ran while my laptop was awake
One Mac, too many Gradle builds
19
Slide 41
Slide 41 text
WHAT THAT MESSAGE MEANT
Five constraints
01
02
03
neverstop
Only ran while my laptop was awake
One Mac, too many Gradle builds
Every project validates differently
19
Slide 42
Slide 42 text
WHAT THAT MESSAGE MEANT
Five constraints
01
02
03
04
neverstop
Only ran while my laptop was awake
One Mac, too many Gradle builds
Every project validates differently
No machine holds every toolchain
19
Slide 43
Slide 43 text
WHAT THAT MESSAGE MEANT
Five constraints
01
02
03
04
05
neverstop
Only ran while my laptop was awake
One Mac, too many Gradle builds
Every project validates differently
No machine holds every toolchain
An agent with shell access
19
Slide 44
Slide 44 text
CONSTRAINT 01
The team asked me to start it while I was on vacation, and
while I was sick.
It had to become a service, not a person.
neverstop
20
Slide 45
Slide 45 text
CONSTRAINT 02
Several people mentioned it at once and my machine
fell over.
A queue, with limits per platform and
per project.
neverstop
21
Slide 46
Slide 46 text
CONSTRAINT 03
Android needs Gradle. Flutter needs FVM. Web needs
npm. A fix nobody verified is not a fix.
Every project declares how to prove its
own build.
neverstop
22
Slide 47
Slide 47 text
CONSTRAINT 04
Installing every one of those toolchains on the same
machine was never going to hold.
One image per platform.
neverstop
23
Slide 48
Slide 48 text
CONSTRAINT 05
An agent with shell access was running directly on a
machine with everything else on it.
One container per job. Deleted when
it finishes.
neverstop
24
Slide 49
Slide 49 text
ONE MORE THING HAD TO CHANGE
A bot named after one person cannot belong
to everyone
AhmedBot
Phantom
It works in a room it is not in. You only
ever see the pull request.
neverstop
25
Slide 50
Slide 50 text
So how does a new project get
in?
26
Slide 51
Slide 51 text
REGISTERING A PROJECT
Message 1 of 3
A REPOSITORY URL IS THE WHOLE REQUEST
A
Ahmed 12:51 PM
let’s add this one
github.com/company/kmp-toolkit
@Phantom register
neverstop
27
Slide 52
Slide 52 text
REGISTERING A PROJECT
Message 2 of 3
THEN IT INTERVIEWS YOU
Phantom APP 12:51 PM
What platform does this project use?
flutter
neverstop
android
jvm
ruby
dotnet
maui
ios
web
27
Slide 53
Slide 53 text
REGISTERING A PROJECT
Message 3 of 3
FOUR QUESTIONS LATER
Phantom APP 12:59 PM
All done. kmp-toolkit is live.
Lint ./gradlew detekt
Tests ./gradlew allTests
Anyone can trigger it with @Phantom toolkit fix
neverstop
27
Slide 54
Slide 54 text
Nineteen projects are registered.
I set up almost none of them.
neverstop
27
Slide 55
Slide 55 text
04
The new cycle
Slide 56
Slide 56 text
FOUR MONTHS LATER
Message 1 of 2
NO TICKET. NO DEVELOPER.
J
Jana 12:15 PM
[Bug] Inbox: “Return” option incorrect colour — should be
the colour as designed:
Policy documentation is ready
Return
Delete
Inbox — swipe actions, as designed
@Phantom fix
neverstop
29
Slide 57
Slide 57 text
FOUR MONTHS LATER
Message 2 of 2
IT NAMED THE EXACT WRONG TOKEN
Phantom APP 12:18 PM
Fixed! The swipe action used colorScheme.tertiary instead
of colorScheme.tertiaryContainer .
PR opened — one line changed.
neverstop
29
Slide 58
Slide 58 text
WHAT WAS DIFFERENT THIS TIME
No developer in the thread
01
neverstop
The bug title is the bug report
29
Slide 59
Slide 59 text
WHAT WAS DIFFERENT THIS TIME
No developer in the thread
01
The bug title is the bug report
02
Nobody added a technical pointer
neverstop
29
Slide 60
Slide 60 text
WHAT WAS DIFFERENT THIS TIME
No developer in the thread
01
The bug title is the bug report
02
Nobody added a technical pointer
03
The attached design is what it checked against
neverstop
29
Slide 61
Slide 61 text
HOW A BUG GETS FIXED NOW
The new cycle
01
neverstop
QA posts in a thread
30
Slide 62
Slide 62 text
HOW A BUG GETS FIXED NOW
The new cycle
01
QA posts in a thread
02
Mentions the bot
neverstop
30
Slide 63
Slide 63 text
HOW A BUG GETS FIXED NOW
The new cycle
01
QA posts in a thread
02
Mentions the bot
03
A verified PR appears
neverstop
30
Slide 64
Slide 64 text
HOW A BUG GETS FIXED NOW
The new cycle
01
QA posts in a thread
02
Mentions the bot
03
A verified PR appears
04
A developer reviews it
neverstop
30
Slide 65
Slide 65 text
The developer is still here.
Just at the end, where the
judgement is.
neverstop
30
Slide 66
Slide 66 text
05
What it actually did
Slide 67
Slide 67 text
378
FOUR MONTHS OF LOGS
agent runs across 19 projects: Android, Flutter,
iOS, JVM backends, .NET, Ruby and web. Most
of that volume is not Android.
neverstop
32
Slide 68
Slide 68 text
FOUR MONTHS OF LOGS
Who is using it?
yes, that one was the CEO
neverstop
QA
42.9%
152 runs
Software Engineering
39.3%
139 runs
Project Management
16.7%
59 runs
DevOps & Infrastructure
0.8%
3 runs
Leadership
0.3%
1 run
33
Slide 69
Slide 69 text
THE SAME DATA, TWO BARS
Six out of ten runs came from someone who is not a
software engineer.
60.7%
215 runs
39.3%
139 runs
QA, project management, infrastructure
Software engineers
QA 152 · PM 59 · DevOps 3 · Leadership 1
Mobile 103 · Java 22 · .NET 11
· JS 3
The people closest to the bug are the ones sending it.
neverstop
34
Slide 70
Slide 70 text
3.4
MEDIAN, MENTION TO PULL REQUEST
min
Nine out of ten finish inside ten minutes. Short
enough that QA stays in the thread and waits.
neverstop
35
Slide 71
Slide 71 text
$0.68
COST PER RUN
$255.85 of API spend, four months, the whole
production bot.
neverstop
36
Slide 72
Slide 72 text
THE ONE WE DID NOT PLAN FOR
108 of 378 runs were not fixes. They were
questions.
223
108
47
fix, change the code
explain, understand it
everything
else
We built something to change code. Almost a third of the value turned out to
be letting people read it: a PM checking how a rule works, QA confirming
expected behaviour before filing anything at all.
neverstop
37
Slide 73
Slide 73 text
199
FOUR MONTHS OF PULL REQUESTS
pull requests opened against production
codebases. Not experiments. Real branches,
real review, real merges.
neverstop
38
Slide 74
Slide 74 text
WHAT HAPPENED TO THEM · 1 OF 3
Every one of the 199, decided
Merged
79%
157 PRs
Closed unmerged
19%
38 PRs
Still open
2%
4 PRs
neverstop
39
Slide 75
Slide 75 text
WHAT HAPPENED TO THEM · 2 OF 3
Every one of the 199, decided
Merged
79%
157 PRs
Closed unmerged
19%
38 PRs
Still open
2%
4 PRs
neverstop
39
Slide 76
Slide 76 text
WHAT HAPPENED TO THEM · 3 OF 3
Every one of the 199, decided
Merged
79%
157 PRs
Closed unmerged
19%
38 PRs
Still open
2%
4 PRs
neverstop
39
Slide 77
Slide 77 text
87
OF THE 157 MERGED
went in exactly as the bot wrote them. No
human commit on the branch at all.
neverstop
40
Slide 78
Slide 78 text
28
PLUS 28 MORE
had a human commit, but it was branch
maintenance, not correction. Merging main in,
running the formatter, regenerating generated
files.
neverstop
40
Slide 79
Slide 79 text
73%
THE NUMBER I WOULD HAVE WANTED FOUR MONTHS AGO
of merged pull requests needed no substantive
human code edit. 115 of 157.
neverstop
40
Slide 80
Slide 80 text
AND THE OTHER SIDE OF IT
42 of them needed a real human
code change first.
One in four. That is the number I would want to hear if I
were sitting where you are.
neverstop
41
Slide 81
Slide 81 text
WHY THEY WERE MERGEABLE
The median merged pull request
+10 −3 2
lines added
lines removed
files touched
These are the one-line bugs from the start of the talk,
arriving finished.
neverstop
42
Slide 82
Slide 82 text
READ IT WITH THIS NEXT TO IT
Three out of four of those pull requests came from
one project.
76%
One codebase, tuned for months
24%
Eleven other
projects
One project with rules the bot had been learning against since March.
Expect your first month to look worse than my fourth.
neverstop
43
Slide 83
Slide 83 text
THE ONE I KEEP TURNING OVER
3 of the 157 merged pull requests drew an
inline review comment.
Read it as trust earned by small correct diffs. Or read it as
review getting thinner because the diffs look easy. I think
both are true.
neverstop
44
Slide 84
Slide 84 text
06
When it does not fix it
Slide 85
Slide 85 text
38
IT DOES NOT ALWAYS WORK
pull requests closed without merging. Nearly
one in five. Nobody merged them, and I am not
going to dress that up.
neverstop
46
Slide 86
Slide 86 text
BACK TO THE FIRST WEEK
A run that does not produce a fix still tells you
something
01
02
03
Add the missing context
Suggest another
approach
Take it seriously
Someone drops in the detail
the thread never had, and
runs it again.
A developer says where to
look, in the same thread,
without opening anything.
This one is not small. It
earned a developer, and now
everyone knows that.
Knowing an approach will not work is worth something. That is triage,
not failure.
neverstop
47
Slide 87
Slide 87 text
07
Then the industry shipped it
Slide 88
Slide 88 text
WHAT HAPPENED NEXT
Then the industry shipped it
MAR
Phantom’s first run
A container per job on our own machine, fixing bugs from a
Slack thread.
neverstop
49
Slide 89
Slide 89 text
WHAT HAPPENED NEXT
Then the industry shipped it
MAR
Phantom’s first run
APR
CodeRabbit Agent for Slack
An agent that lives in the thread, investigates and opens PRs.
Very close to Phantom.
neverstop
49
Slide 90
Slide 90 text
WHAT HAPPENED NEXT
Then the industry shipped it
MAR
Phantom’s first run
APR
CodeRabbit Agent for Slack
JUN
Anthropic ships Claude Tag
Tag @Claude in a channel and delegate the task.
neverstop
49
Slide 91
Slide 91 text
WHAT HAPPENED NEXT
Then the industry shipped it
MAR
Phantom’s first run
APR
CodeRabbit Agent for Slack
JUN
Anthropic ships Claude Tag
AUG
Slack ships Slack Code
A channel per task, opened by the agent and archived when the
work is done.
neverstop
49
Slide 92
Slide 92 text
We were about three months early.
That mostly proves the problem
was real.
neverstop
49
Slide 93
Slide 93 text
SO WHY KEEP OURS
Three reasons that held up
01
It lives where the bug is reported
QA and PMs do not open pull requests. They open Slack
threads.
neverstop
50
Slide 94
Slide 94 text
SO WHY KEEP OURS
Three reasons that held up
01
It lives where the bug is reported
02
It runs on our own machine, on our own repos
Including client code that cannot leave our infrastructure, and
Azure DevOps repos most tools do not support.
neverstop
50
Slide 95
Slide 95 text
SO WHY KEEP OURS
Three reasons that held up
01
It lives where the bug is reported
02
It runs on our own machine, on our own repos
03
It reads the same rules our developers do
The same project rule files a developer’s own agent reads, so
the code it writes looks like ours.
neverstop
50
Slide 96
Slide 96 text
Buy theirs, or build one that fits your
team. Both are reasonable answers
now.
50
Slide 97
Slide 97 text
08
If you want to build one
Slide 98
Slide 98 text
MENTION TO PULL REQUEST
What happens after you press enter
Container
the toolchain for that project
Slack thread
Orchestrator
the bug, the images
channel → project
Back in the thread
Orchestrator
Tool server
progress, cause, the PR
the only Slack credentials
the container has no Slack access
neverstop
Clone
Agent
works
Validate
52
Slide 99
Slide 99 text
INSIDE THE BASE IMAGE
The agent is baked in, and it cannot reach Slack
ONE CONTAINER, ONE
JOB
THE TOOLS IT MAY
CALL
Claude Code SDK
git, platform CLIs
No Slack token
Set the status
Post a message
Open a pull request
ORCHESTRATOR
Holds the only Slack
credentials in the system
✕ no path from the container to your workspace
Deleted when the job finishes.
neverstop
53
Slide 100
Slide 100 text
THE IMAGES
One base, one layer per platform
base git, the CLIs, the agent
android JDK 17, SDK 34
flutter needs the Android SDK
maui
The base image carries git, the
platform CLIs and the agent itself.
Everything above it is just a toolchain.
Adding a platform means adding one
layer, not redesigning anything.
jvm · dotnet · ruby · web
neverstop
54
Slide 101
Slide 101 text
THE ANDROID IMAGE
Running Gradle when nobody is watching
BASE
git and the platform CLIs
BASE
The Claude Code SDK
BASE
No Slack token, no workspace access
ANDROID
JDK 17
AGP 8 will not run on anything older. Gradle toolchains can still fetch another JDK to
compile with.
ANDROID
SDK 34, build tools 34
The platform the projects target, and nothing spare. Every extra package is minutes on a
cold image.
neverstop
55
Slide 102
Slide 102 text
WHY THE PULL REQUEST IS WORTH OPENING
The agent has to prove its own fix
Make the change
Run the project's linter
Three attempts to get clean. Each project names its
own linter and its own compile step, so the agent
runs what your CI would run.
neverstop
Read the errors, fix,
retry
Compile, then open the
PR
This is how it catches its
own mistakes.
56
Slide 103
Slide 103 text
WHY IT IS MINUTES, NOT HALF AN HOUR
Nothing clones from the network twice
01
A mirror per repository, kept warm
02
Jobs clone from the mirror, not the internet
03
One lock per repository
First run on a large Android repository: twenty minutes. Every run after
it: seconds.
neverstop
57
Slide 104
Slide 104 text
THE HONEST GAP
Where the architecture runs out
iOS is a passenger
Nothing here runs the app
There is no Swift toolchain image yet, so those
projects run on the plain base image. The agent
reads and changes Swift, but it cannot compile it,
so it cannot prove anything, and those pull requests
are drafts.
No emulator, no device, no screenshot of the fixed
state. Compile and lint tell you the code is sane. They
do not tell you the bug is gone. Someone still opens
the app.
neverstop
58
Slide 105
Slide 105 text
WHAT WE ACTUALLY BUILT
Not one bot. A place to plug agents in.
LIVE
LIVE
Fix a bug
Ask the
codebase
LIVE
TESTED
NEXT
Build an APK
Review a pull
request
Whatever we
need next
One orchestrator · one set of images · one
validation loop
neverstop
T H E I N F R A ST R U C T U R E
59
Slide 106
Slide 106 text
09
What we learned
Slide 107
Slide 107 text
IF YOU ARE THINKING ABOUT THIS
Three things we learned
01
Match the tool to the size of the problem
Small, well described bugs. Not architecture, not features.
neverstop
61
Slide 108
Slide 108 text
IF YOU ARE THINKING ABOUT THIS
Three things we learned
01
Match the tool to the size of the problem
02
Start with the small version
A script that works is a better argument than a proposal.
neverstop
61
Slide 109
Slide 109 text
IF YOU ARE THINKING ABOUT THIS
Three things we learned
01
Match the tool to the size of the problem
02
Start with the small version
03
The API bill is not the cost
Someone maintains this. Count that time before you call
it cheap.
neverstop
61
Slide 110
Slide 110 text
WHAT NONE OF US HAS SOLVED YET
The thread and the ticket still do not
know about each other.
61
Slide 111
Slide 111 text
TAKE IT WITH YOU
These slides are online
speakerdeck.com/thisaay
how-we-moved-from-android-studioto-slack
Every number in this talk, and the
parts I went past quickly.
neverstop
62
Slide 112
Slide 112 text
OVER TO YOU
Questions
neverstop
62
Slide 113
Slide 113 text
Every step we automated
made the fix faster.
None of them made the
handoff cheaper.
So we deleted the handoff.
Thank you. Ahmed Ali
THESE SLIDES
neverstop