Share
Embed
Transcript
75 slides
-
Unraveling the Internals of AppFunctions AppFunctions の内部実装を紐解く 2026/11/14 Kotlin Fest
2026 Slides (JA) Nosho Hiroaki 1 -
About me Nosho Hiroaki (hiroaki404) Android engineer / 6–8 years
of Kotlin Kotlin Fest 2025: "Getting Started with AI Agent Development using Koog" gentrans: a CLI translation tool built with Koog X: @hiroaki08181359 Plugin for IntelliJ-based IDEs Markdown translation / Local LLM support github.com/hiroaki404/gentrans 2 -
What are AppFunctions? A mechanism on Android that lets AI
agents discover and execute functions in various apps AIagent Similar to MCP or function calling ƒcallfunctions Gemini and others can use your app as a tool Officially announced as Android MCP at Android Dev Summit, October 2026 Notesapp developer.android.com/ai/appfunctions / The Android Show: I/O Edition 2026 / Android Dev Summit 2026 Mapapp Calendarapp 3 -
Today's session Follow the internals of AppFunctions through the source
code androidx and AOSP Build our own agent app that uses AppFunctions with Koog JetBrains' AI agent framework Motivation To be able to design good tools Technical curiosity To learn how AI agents on mobile devices are built with Kotlin For how to add AppFunctions to your own app and best practices, see the official guide 5 -
Agent and Tool Agent Agent The agent app that calls
functions AIagent e.g. Gemini Tool ƒcallfunctions The app that exposes functions e.g. your app Notesapp Mapapp Tool Calendarapp Usually we mostly care about the Tool side. Today we also look inside the Agent side 6 -
Code developers write on the Tool side /** * Books
a flight ... * @param fromCity The departure city. ... * @param toCity The arrival city, ... * @param date The departure date. * @return The confirmed booking, ... */ @AppFunctionDeclaration(isDescribedByKDoc = true) suspend fun bookFlight( fromCity: String, toCity: String, date: LocalDate, ): FlightBooking = ... Function description passed to the agent Function declaration (Manifest and XML configuration are also required) With the official android skills, setup and writing functions are easy Demo: tool/tripapp/src/main/java/.../BaseTripAppFunctionService.kt (excerpt) 7 -
Current limitations Agent: requires EXECUTE_APP_FUNCTIONS Regular apps cannot hold it
Only allowed apps such as Gemini can use it * This does not apply to an in-app agent calling its own app's functions Tool: anyone can write @AppFunctionDeclaration Until now: only EAP (Early Access Program) partners were called from Gemini Since late October: Connected Apps Platform (preview) started. Published to Gemini after Play review (Currently only US developers who applied and were approved) developer.android.com/ai/appfunctions / issuetracker b/490650368 #15 / Create and manage connected apps / Android Dev Summit 2026 8 -
How to test Try a Tool Check registration and execute
with cmd app_function Call it from the official Testing Agent Build an Agent (2 ways to get the permission) ① Rooted emulator/device ← demo ② am instrument (Testing Agent) # List registered functions adb shell cmd app_function list-app-functions \ --package <package name> # Execute a function adb shell cmd app_function execute-app-function \ --package <package name> --function '<function ID>' \ --parameters '<JSON>' # priv-app approach (today's demo) emulator -avd Pixel_10_Root -writable-system adb root && adb remount adb push privapp-permissions-....xml \ /system/etc/permissions/ adb push agent.apk /system/priv-app/ adb reboot Details (Japanese): https://qiita.com/hiro404/items/338300798ca31e7bef87 github.com/android/appfunctions (Testing Agent) 9 -
Versions and caveats Version androidx.appfunctions 1.0.0-beta01 Device Android 17 (API
37) emulator AOSP (system-side source) android17-release (Android 17) Caveats Today's content reflects the current state. It may change This session is based on my own investigation of the code and my own views. It is not an official position. Some parts are simplified. 10 -
Three questions 1. Developers only write an annotation and KDoc.
How can the agent find and choose functions? 2. How can it find apps installed after the agent app started? 3. For a function in another process, who authorizes it, and how does it run? 12 -
The big picture Worker Cataloger Toolapp ƒ Catalog Registerfunctions Oninstall
Functionlist Register dothework Checkcatalog Request Frontdesk Relayexecution Result Client Agent 13 -
Architecture Buildtime Toolapp @AppFunctionDeclaration(KDoc)/EntryPoint KSP(appfunctions-compiler) GeneratedKotlin GeneratedXML Service/Inventory Goesintoassets Oninstall/unlock
)llatsnita(KPAnideldnuB Writtenbydeveloper system_server Runtime PackageManager Listintent-filters/getProperty AppFunctionManagerService MetadataSyncAdapter enabledstate Runtime AppFunctionService onExecuteFunction(main) Binder Runtime @AppFunctionDeclarationfunctionbody ParseXML androidx(Kotlin) Binder(observe) Binder(search) PlatformAppFunctionReader Search/observeviaOSManager Staticmetadata apps-db AppFunctionManagerServiceImpl UIDmatch/permission/enabled Binder frameworkAppFunctionManager AttachcallingPackage ServiceHelper/RemoteServiceCaller Dispatchwithwhen Developer AppFunctionManager observe/executeAppFunction AppsIndexer AppSearchManagerService JoinSpec appfunctions-db Agentapp resolve→bindeachtime framework AppSearch(APEX) Binder(4calls) In-process Viafiles By the author. Versions used: androidx appfunctions 1.0.0-beta01 / AOSP android17-release / Demo: github.com/hiroaki404/appfunctions-koog-agent-demo 15 -
①–③ Registration flow Buildtime Tool Toolapp Atinstall Runtime ①KSP ⑧Functionbody
GenerateXML system_server OS Agent Agentapp Toolprocess ②AppsIndexer ③AppSearch ReadXML ⑥Checks Functionlist Permission/enabled ④search Getlist/observe ⑦bind CallTool ⑤execute Requestexecution KSP generates XML AppsIndexer reads the XML and registers the functions in the AppSearch database 16 -
④–⑧ Discovery and execution flow Buildtime Tool Toolapp system_server OS
Atinstall Runtime ①KSP ⑧Functionbody GenerateXML Toolprocess ②AppsIndexer ③AppSearch ReadXML ⑥Checks Functionlist Agent Agentapp Permission/enabled ④search Getlist/observe ⑦bind CallTool ⑤execute Requestexecution The agent gets the list from AppSearch and observes changes with a Flow Agent and Tool never connect directly 17 -
Tool The side that exposes functions Buildtime Tool Toolapp system_server
OS Agent Agentapp Atinstall Runtime ①KSP ⑧Functionbody GenerateXML Toolprocess ②AppsIndexer ③AppSearch ReadXML ⑥Checks Functionlist Permission/enabled ④search Getlist/observe ⑦bind CallTool ⑤execute Requestexecution 18 -
Functions of the demo tool apps The travel app and
messaging app from the opening demo They expose 5 functions App Function tripapp getTravelInfo Returns recommended spots tripapp saveItinerary Saves an itinerary tripapp searchItineraries Searches itineraries incl. past trips tripapp bookFlight Books a flight → Return String, Int → List<Spot> Itinerary → String String? → List<Itinerary> String, String, LocalDate → message sendMessage Sends a message to a contact String, String What it does github.com/hiroaki404/appfunctions-koog-agent-demo (tool/) Args FlightBooking → String 19 -
What developers do 1. build.gradle.kts : add the appfunctions dependency
and KSP 2. Write @AppFunctionDeclaration in an abstract Service annotated with @AppFunctionServiceEntryPoint (KDoc supported) 3. res/xml/app_metadata.xml : describe the whole app 4. AndroidManifest.xml : declare the generated Service and XML 20 -
① Gradle: add dependencies plugins { id("com.google.devtools.ksp") version "2.3.10" }
dependencies { Add the appfunctions and KSP dependencies implementation("androidx.appfunctions:appfunctions:1.0.0-beta01") ksp("androidx.appfunctions:appfunctions-compiler:1.0.0-beta01") } Demo: tool/tripapp/build.gradle.kts / tool/gradle/libs.versions.toml 21 -
② Write functions: EntryPoint and @AppFunctionDeclaration @AppFunctionServiceEntryPoint( serviceName = "TripAppFunctionService",
appFunctionXmlFileName = "trip_functions", ) abstract class BaseTripAppFunctionService : AppFunctionService() { /** * Books a flight ... * @param fromCity The departure city. ... * @param toCity The arrival city, ... * @param date The departure date. * @return The confirmed booking, ... */ @AppFunctionDeclaration(isDescribedByKDoc = true) suspend fun bookFlight( fromCity: String, toCity: String, date: LocalDate, ): FlightBooking = ... Name of the generated concrete class Name of the generated XML file Function description for the LLM Use KDoc as the description You can declare multiple functions @AppFunctionDeclaration(isDescribedByKDoc = true) suspend fun saveItinerary(itinerary: Itinerary): String = ... } Demo: tool/tripapp/src/main/java/.../BaseTripAppFunctionService.kt 22 -
③ app_metadata.xml: describing the whole app Describes the whole app
(not each function) The first entry point in the official guide Provided by the developer in res/xml/ <!-- res/xml/app_metadata.xml --> <AppFunctionAppMetadata xmlns:appfn= "http://schemas.android.com/apk/androidx.appfunctions" appfn:instructions="Travel app ... Whole-app description for the user's trips: searching saved itineraries, ... and booking flights. Operational Patterns: - To plan a new trip, call getTravelInfo ..., then pass the chosen spots ... to saveItinerary. - Flight numbers ... come from the bookFlight result. Constraints: - Dates written as text ... use yyyy-MM-dd ..." appfn:displayDescription= "@string/app_function_app_display_description" /> Demo: tool/tripapp/src/main/res/xml/app_metadata.xml (excerpt, reformatted) 23 -
④ Manifest: name the generated files <property> points to app_metadata.xml
Match serviceName / appFunctionXmlFileName with the <!-- AndroidManifest.xml (directly under <application>) --> <property android:name="android.app.appfunctions.app_metadata" android:resource= app_metadata.xml "@xml/app_metadata" /> <service Concrete Service name android:name=".TripAppFunctionService" Only the system can bind values above android:permission= With BIND_APP_FUNCTION_SERVICE , only the system can bind (required; it is a condition the system uses when searching) android:exported="true"> "android.permission.BIND_APP_FUNCTION_SERVICE" <property android:name="android.app.appfunctions.v2" android:value="trip_functions.xml" /> <property android:name="android.app.appfunctions.schema" android:value="app_functions_schema.xsd" /> <intent-filter> <action android:name= "android.app.appfunctions.AppFunctionService" /> </intent-filter> </service> Demo: tool/tripapp/src/main/AndroidManifest.xml 24 -
File generation with KSP KSP (Kotlin Symbol Processing): interprets code
and generates files Nextroundalsoreadsoutputs kspDebugKotlin(KSP) Source .kt Symbols KSPparsessource KSPprocessor Resolveneededtypes ReadKDoc,writefiles Generatedfiles .kt / .xml etc. Maincompileafterlastround compileDebugKotlin (main compile) Source +generatedfiles FIR Parse IR ACompilerPlugin Canrewritethis Bytecode KSP parses the source into symbols, so types, fields, methods, KDoc, etc. can be read. It never rewrites existing code; it only generates files 25 -
KSP entry point: AppFunctionCompiler The entry is AppFunctionCompiler, created by
a Provider registered in META-INF/services In the round where the required types are resolved, it runs the child processors (9 of them) in order ServiceEntryPoint Inventory // AppFunctionCompiler.kt class AppFunctionCompiler( private val processors: List<SymbolProcessor>, private val logger: KSPLogger, ) : SymbolProcessor { override fun process(resolver: Resolver) : List<KSAnnotated> { val deferred = shouldDeferAllProcessing(resolver) if (deferred.isNotEmpty()) { deferred } else { buildList { for (processor in processors) { Call child processors in order addAll(processor.process(resolver)) Serializable } } } ... androidx: appfunctions-compiler/.../AppFunctionCompiler.kt L44-64 (try/catch omitted) 26 -
AppFunctionServiceEntryPointProcessor Collects classes annotated with @AppFunctionServiceEntryPoint Also gets each function's
KDoc via docString Generates the concrete Service and the XML The XML is written with createNewFile using the xml extension // AppFunctionServiceEntryPointProcessor.kt override fun process(resolver: Resolver) : List<KSAnnotated> { ... val serviceEntryPoints = appFunctionSymbolResolver .resolveAnnotatedAppFunctionServiceEntryPoints() for (serviceEntryPoint in serviceEntryPoints) { generateAppFunctionService(serviceEntryPoint) generateXml(serviceEntryPoint, ...) } return emptyList() } Generate files // AppFunctionSymbolResolver.kt (collecting side) resolver.getSymbolsWithAnnotation(...) ... appFunctionDeclarations.map { AnnotatedAppFunction(it, it.docString) // KDoc } androidx: processors/AppFunctionServiceEntryPointProcessor.kt L101-122 / core/AppFunctionSymbolResolver.kt L37-63 (excerpt; try/catch and args omitted) 27 -
Generated files Concrete Service build/generated/ksp/debug/ Inventory ├── kotlin/dev/hiroaki404/.../tripapp/ │ ├──
TripAppFunctionService.kt Factory │ ├── $BaseTripAppFunctionService_ │ │ XML │ └── $SpotFactory.kt ... ×4 AppFunctionInventory.kt ├── kotlin/appfunctions_aggregated_deps/ ← Concrete Service ← Inventory ← Factory (index for aggregation ×5) └── resources/assets/ └── trip_functions.xml ← XML Function descriptions 28 -
Generated file ① Concrete Service Purpose Receives system calls and
routes them to the developer's function Contents onExecuteFunction gets the Request and metadata Branches on the ID with when and calls bookFlight IDs are const s in the class Args are cast from a Map Demo: generated code (excerpt) public class TripAppFunctionService : BaseTripAppFunctionService() { override fun onExecuteFunction( request, metadata, ...) = AppFunctionExecutionDispatcher .dispatchExecuteAppFunction( scope, request, metadata, ... ) { parameters -> when (request.functionIdentifier) { FUNCTION_ID_BOOK_FLIGHT -> this.bookFlight( parameters["fromCity"] as String, parameters["toCity"] as String, parameters["date"] as LocalDate) ... } } companion object { const val FUNCTION_ID_BOOK_FLIGHT = "...BaseTripAppFunctionService#bookFlight" } } Receives the Request By ID Type cast Function ID 29 -
Generated file ② Inventory: function metadata Purpose Holds the information
to convert the Request received by the Service into the defined argument types, and to convert the return value into the form to send back // $BaseTripAppFunctionService // _AppFunctionInventory.kt (generated) override val functionIdToMetadataMap = mapOf( "...BaseTripAppFunctionService#bookFlight" to ...BookFlightMetadataObject .APP_FUNCTION_METADATA, ... ) private object ...BookFlightMetadataObject { private val FROMCITY_PARAMETER_METADATA = Contents AppFunctionParameterMetadata( name = "fromCity", isRequired = true, Function metadata and type information dataType = ...FROMCITY_PRIMITIVE_DATA_TYPE, description = "The departure ...", A Map from function ID to metadata Built from the defined function's types, KDoc, etc. ID → metadata From KDoc ) ... val APP_FUNCTION_METADATA = ... } Demo: build/generated/ksp/debug/kotlin/.../$BaseTripAppFunctionService_AppFunctionInventory.kt (excerpt, 535 lines) 30 -
Generated file ③ XML: registered in the function list Purpose
Read by the system to register functions in AppSearch Contents Nearly the same metadata as the Inventory Descriptions come from KDoc (only the XML has them) Built by the ServiceEntryPoint processor with Java's DOM XML API Goes into the APK's assets/ <appfunctions> <appfunction> <id> ...BaseTripAppFunctionService#bookFlight </id> <enabledByDefault>true</enabledByDefault> <description> Books a flight ... </description> <parameters> <dataTypeMetadata> <isNullable>false</isNullable> <type>8</type> </dataTypeMetadata> <isRequired>true</isRequired> <name>fromCity</name> <description> The departure city. ... </description> </parameters> ... Demo: build/generated/ksp/debug/resources/assets/trip_functions.xml (excerpt, about 18KB) From KDoc 31 -
From KDoc to LLM-facing descriptions Inventory KDoc /** ... */
KSP docString Justextractsastring androidx DocstringUtils Splitstagsitself Readbyappatruntime Docsgoto2places XML ReadbyOSatinstall KSP passes just a string. @param is parsed by androidx with regular expressions androidx: appfunctions-compiler/.../core/DocstringUtils.kt L22-27 32 -
Code generation: building code with KotlinPoet KotlinPoet: a library (by
Square) that builds Kotlin source with Builders instead of string concatenation beginControlFlow( "when (%L.%L)", "request", "functionIdentifier") The concrete Service's when for (appFunction in The concrete Service and Inventory are generated with KotlinPoet serviceEntryPoint.appFunctions) { beginControlFlow("%L ->", getFunctionIdConstantPropertyName( appFunction)) add("this.%N(\n", function.simpleName.asString()) for (param in function.parameters) addStatement("%L[%S] as %T,", "parameters", paramName, param.type.toTypeName()) ... endControlFlow() Division of roles: KSP reads symbols and KDoc and provides the output files. KotlinPoet and Java's XML library build the contents Type cast } androidx: processors/AppFunctionServiceEntryPointProcessor.kt L284-355 (excerpt) 33 -
Summary: who uses the generated files, when, and how Toolapp
Buildtime Atinstall ConcreteService Tofunctionviawhen Inventory Restoreargtypes Factory Repackdataclasses Functionentrypoint KSP Runtime DispatchbyfunctionID Functionspec(Kotlin) Convertperspectypes Fordataclasses AppFunctionData⇄dataclass XML bind Functionspec(XML) system_server AppsIndexer RegisterXMLinAppSearch Whereagentsfindfunctions AppFunction ManagerService 34 -
Registering with the system Until the XML lands in AppSearch
(registered in the function list) Buildtime Tool Toolapp system_server OS Agent Agentapp Atinstall Runtime ①KSP ⑧Functionbody GenerateXML Toolprocess ②AppsIndexer ③AppSearch ReadXML ⑥Checks Functionlist Permission/enabled ④search Getlist/observe ⑦bind CallTool ⑤execute Requestexecution 35 -
From app install to registration system_server ToolAPK ①Install PackageManager ②ACTION_PACKAGE_ADDED
OnlynotestheManifest ③QueryXMLpath Manifest assets/*.xml AppsIndexer Registerreceiveratboot ⑤put ④ReadXMLviaAssetManager AppSearch appfunctions-db apps-db Enabled/disabled ⑥Sync Functiondefinitions AppFunctionManagerService Notifychanges Watchapps-dbchanges AppsIndexer in system_server registers functions in AppSearch 36 -
What is AppSearch? AppSearch is a high-performance on-device search solution
for managing locally stored, structured data. An on-device full-text search engine + document store // Sample from the official guide (shortened) @Document data class Note( @Document.Namespace val namespace: String, @Document.Id val id: String, @Document.StringProperty( indexingType = INDEXING_TYPE_PREFIXES ) val text: String, ) session.setSchema( SetSchemaRequest.Builder() .addDocumentClasses(Note::class.java) .build()) session.put( PutDocumentsRequest.Builder() .addDocuments(note).build()) session.search("fruit", searchSpec) developer.android.com/develop/ui/views/search/appsearch (2026-10-11. Quote from the opening) 37 -
AppSearch keywords Keyword Database Description in the official guide A
collection of documents conforming to a schema. Only the creating app (system) can open it Session The entry point for operations, returned when opening a DB Document A unit of data. Uniquely identified by Namespace and ID Schema The structure of the data Namespace Used to separate data from different sources * In AppFunctions, one function corresponds to one Document Create Put Search OpenDB,setschema PutDocuments Searchbyquery createSearchSession put search developer.android.com/develop/ui/views/search/appsearch (Database and session / Schema and schema types / Documents) 38 -
Storage and Session types Storage Where the index lives Shared
with other apps LocalStorage The app's data directory No PlatformStorage system_server (Android 12+) Yes PlayServicesStorage Google Play services Yes Session Connects to AppSearchSession One DB opened by name GlobalSearchSession Searches across all DBs it has access to The AppFunctions DB is in PlatformStorage developer.android.com/develop/ui/views/search/appsearch (Platform Storage, Local Storage or Play Services Storage, 2026-10-07) / androidx appsearch: Javadoc of AppSearchSession.java, GlobalSearchSession.java, PlatformStorage.java 39 -
AppsIndexer writes to AppSearch The AppSearch module has separate indexers
per data category AppsIndexer operates the AppSearch DB (apps-db) for AppFunctions AppsIndexer defines the Schema and puts Documents packages/modules/AppSearch/service/.../appsearch/ ├── contactsindexer/ Contacts ├── appsindexer/ App list + AppFunctions └── indexer/ Common periodic maintenance The package contains the implementation of the AppsIndexer used to index metadata about apps and app functions exposed by apps into AppSearch. AOSP AppSearch: appsindexer/package-info.java 40 -
AppsIndexer watches app installs At boot, system_server starts the AppSearch
module, and AppsIndexer starts watching Three triggers // AppsIndexerManagerService.java private void registerReceivers() { IntentFilter appChangedFilter = new IntentFilter(); appChangedFilter.addAction( Intent.ACTION_PACKAGE_CHANGED); Package changes (add, update, remove) appChangedFilter.addAction( First unlock / every 30 days appChangedFilter.addAction( On package-change notifications from PackageManager, it re-checks all packages Intent.ACTION_PACKAGE_ADDED); Install Intent.ACTION_PACKAGE_REPLACED); appChangedFilter.addAction( Intent.ACTION_PACKAGE_FULLY_REMOVED); appChangedFilter.addDataScheme("package"); mContext.registerReceiverForAllUsers( new AppsProviderChangedReceiver(), appChangedFilter, ...); Register receiver AOSP AppSearch: AppSearchModule.java L146-150 / appsindexer/AppsIndexerManagerService.java L132-149, L227-236 / AppsIndexerConfig.java L33 41 -
AppsIndexer gets the XML file from the APK 1. Enumerate
apps with an AppFunctionService via intent-filter (Services without BIND_APP_FUNCTION_SERVICE are excluded) 2. Get the XML path from the Service's <property> Open the XML at that path and read it with parseIntoMapForGivenSchemas // AppsUtil.java Intent appFunctionServiceIntent = new Intent(APP_FUNCTION_SERVICE_INTERFACE); List<ResolveInfo> services = packageManager .queryIntentServices(appFunctionServiceIntent, 0); ... // AppFunctionResolveInfo.java v2XmlProperty = packageManager.getProperty( APP_FUNCTION_V2_XML_PROPERTY_NAME, componentName); ... if (v2XmlProperty != null) { return v2XmlProperty; } ... return packageManager.getProperty( APP_FUNCTION_V1_XML_PROPERTY_NAME, componentName); ... // AppFunctionsIndexerUtil.java if (appFunctionXmlInfo.useSchemaForParsing()) { appFunctionXmlInfo.runWithXmlParser(packageManager, xmlParser -> packageAppFunctions.putAll( parser.parseIntoMapForGivenSchemas(...))); } AOSP AppSearch: AppsUtil.java L161-191 / AppFunctionResolveInfo.java L137-155 / AppFunctionsIndexerUtil.java L246-266 (excerpt) 42 -
Opening the APK and reading the XML Creates an AssetManager
from the Tool app's APK Uses AssetManager to open the APK's assets/ by path // AppFunctionResolveInfo.java (runWithXmlParser) Resources resources = packageManager.getResourcesForApplication( mPackageName); ... XmlPullParser parser = The KSP-generated trip_functions.xml is read here Each AppFunction (each <appfunction> tag in the XML) becomes one AppSearch document AOSP AppSearch: appsindexer/AppFunctionResolveInfo.java L456-477 (excerpt) XmlPullParserFactory.newInstance() .newPullParser(); try (InputStreamReader isr = new InputStreamReader( resources.getAssets() AssetManager .open(mXmlFile.getXmlFilePath()))) { parser.setInput(isr); action.accept(parser); } 43 -
Two AppSearch DBs DB apps-db Contents Function definitions (static) Written
by AppsIndexer Visibility Anyone who can see the Tool app Only the Tool app itself appfunctions-db Enabled/disabled (runtime) AppFunctionManagerService and apps with permissions such as EXECUTE_APP_FUNCTIONS Both are owned by android (system_server). Apps never write to them directly * Whether the Tool app is visible depends on the agent Manifest's <queries> / QUERY_ALL_PACKAGES (the demo uses the latter) AOSP AppSearch: AppSearchHelper.java L96, L284 / AOSP frameworks/base: AppFunctionRuntimeMetadata.java L49, MetadataSyncAdapter.java L305-320 44 -
Agent search How the function list is fetched Buildtime Tool
Toolapp system_server OS Agent Agentapp Atinstall Runtime ①KSP ⑧Functionbody GenerateXML Toolprocess ②AppsIndexer ③AppSearch ReadXML ⑥Checks Functionlist Permission/enabled ④search Getlist/observe ⑦bind CallTool ⑤execute Requestexecution 45 -
Observing and searching with AppFunctionManager For the Agent, androidx's AppFunctionManager
is the entry point Observe: observeAppFunctions() Notifies only changes via a Flow Search: searchAppFunctions() Gets the list of functions The agent app combines the two // AppFunctionDiscovery.kt (demo app) fun AppFunctionManager.discoverAppFunctions() : Flow<List<AppFunctionMetadata>> = observeAppFunctions() Notifies only changes .filterIsInstance< AppFunctionsChangeEvent.MetadataChanged>() .map { } .onStart { emit(Unit) } // initial .map { Re-fetch the list searchAppFunctions(AppFunctionSearchSpec()) } Demo: AppFunctionDiscovery.kt (reformatted) / androidx: AppFunctionManager.kt L205-207, L237-241 46 -
Down to AppSearch Both observing and searching go through an
intermediary class and call the OS AppFunctionManager Agentprocess androidx AppFunctionManager Delegate androidx The OS one in PlatformAppFunctionReader android.app.appfunctions , separate from androidx From there, two destinations Observe → AppFunctionManagerService Search → AppSearch framework android.app.appfunctions.AppFunctionManager Observe Notify Search watch AppSearch system_server AppFunctionManagerService Notifychanges androidx: AppFunctionManager.kt L635-643 / AOSP frameworks/base: AppFunctionManager.java L621-657, L702-750 apps-db(definitions) 47 -
Implementation of observeAppFunctions Inside a callbackFlow, registers an AppFunctionObserver with
the OS AppFunctionManager Turns callbacks into events and calls trySend Metadata changes (per package) MetadataChanged State changes (per function) → → StatesChanged androidx: PlatformAppFunctionReader.kt L166-191 (excerpt, reformatted) // PlatformAppFunctionReader.kt callbackFlow { val appFunctionObserver = object : AppFunctionObserver { override fun onAppFunctionMetadataChanged( changedPackageNames: Set<String> ) { trySend(AppFunctionsChangeEvent .MetadataChanged(changedPackageNames)) } override fun onAppFunctionStatesChanged(...) { trySend(AppFunctionsChangeEvent .StatesChanged(...)) } } val observation: AppFunctionObservation = appFunctionManager.observeAppFunctions( Runnable::run, appFunctionObserver) awaitClose { observation.cancel() } }.buffer(Channel.UNLIMITED) callbackFlow Send event 48 -
Implementation of searchAppFunctions Repacks the conditions into OS types package
/ function name / schemaName / schemaCategory / schemaVersion / scope The demo sets none functions → gets all visible No keyword filtering of descriptions Picking functions by meaning is the LLM's job suspendCancellableCoroutine: callback suspend → The OS searches AppSearch via GlobalSearchSession // PlatformAppFunctionReader.kt val platformSpec = searchFunctionSpec.toPlatformSearchSpec() return suspendCancellableCoroutine { cont -> appFunctionManager.searchAppFunctions( platformSpec, Runnable::run, ...) } // AppFunctionManagerHelper.java (OS) appSearchManager.createGlobalSearchSession( executor, sessionResult -> { GlobalSearchSession session = sessionResult.getResultValue(); ... SearchResults results = session.search( searchSpec .getStaticMetadataAppSearchQuery(), staticMetadataSearchSpec); }); androidx: PlatformAppFunctionReader.kt L62-103 / AOSP frameworks/base: AppFunctionManagerHelper.java L172-216 (both excerpts) 49 -
Summary: registration to observing and searching system_server Install AppsIndexer put
Agentprocess AppSearch Notifychanges AppFunctionManagerService NotifyAgentviaBinder Binder MetadataChanged observeAppFunctions Re-fetchlist searchAppFunctions The agent keeps observing. That's why apps installed later are found too 50 -
Agent function execution How a function in another process runs
Buildtime Tool Toolapp system_server OS Agent Agentapp Atinstall Runtime ①KSP ⑧Functionbody GenerateXML Toolprocess ②AppsIndexer ③AppSearch ReadXML ⑥Checks Functionlist Permission/enabled ④search Getlist/observe ⑦bind CallTool ⑤execute Requestexecution 51 -
The entry is executeAppFunction androidx's AppFunctionManager is the entry; it
runs via executeAppFunction() Goes through the OS AppFunctionManager Before calling, androidx searches AppSearch once more To get the metadata used to convert the return type Then system_server binds and calls the Tool's Service Agentprocess ①Lookupmetadata androidx AppFunctionManager androidx ②Execute PlatformAppFunctionReader androidx PlatformAppFunctionManagerApi framework android.app.appfunctions.AppFunctionManager system_server AppSearch apps-db(definitions) ①Search ②Execute AppFunctionManagerServiceImpl Toolprocess androidx: AppFunctionManager.kt L127-153 / PlatformAppFunctionManagerApi.kt L199-251 / PlatformAppFunctionReader.kt L105-117 ③Bindandcall AppFunctionService Callfunctionbody 52 -
system_server's pre-execution checks 1. The UID of the Agent's declared
package name matches the actual UID 2. User checks (multi-user, work profile) 3. Execute permission 4. The function is enabled ( apps-db + appfunctions-db joined via JoinSpec) 5. Find the Tool's Service (from apps-db) Only Services with BIND_APP_FUNCTION_SERVICE (declared in the Manifest) AOSP frameworks/base: AppFunctionManagerServiceImpl.java L386-560, L652-760 53 -
Execution permission checks Agent Has EXECUTE_APP_FUNCTIONS_SYSTEM Has EXECUTE_APP_FUNCTIONS No permission
Result Always OK OK for its own functions. Others OK only if the Tool is on the allowlist OK for its own functions only (intended for ML Kit in-app agents) allowlist: per Agent (signature), the list of Tools it may call The list is queried from a provider outside the system The Tool side can also restrict callers with @AppFunctionAccessLevel (SELF / SYSTEM / ANDROID_TRUSTED; default ANDROID_TRUSTED) Not just EXECUTE_APP_FUNCTIONS ; being on the allowlist is also required AOSP frameworks/base: CallerValidatorImpl.java L98-170 / allowlist/SystemAppFunctionAllowlistReader.java L126-186 54 -
Binding the service bindService per call BIND_FOREGROUND_SERVICE gives priority on
par with a Foreground Service * Only when the Agent has the permission // RemoteServiceCallerImpl.java public boolean bindAndRun() { boolean bindServiceResult = mContext.bindServiceAsUser( mIntent, this, mFlags, mUserHandle); Bind per call if (bindServiceResult) { ... mDirectServiceVulture = () -> { ... mCancellationSignal.cancel(); }; ... mCallerBinder.linkToDeath( mDirectServiceVulture, /* flags= */ 0); ... } ... } ... public void onCompleted() { AOSP frameworks/base: RemoteServiceCallerImpl.java L117-185 (excerpt) / AppFunctionManagerServiceImpl.java safeUnbind();L618-624 } Unbind when done 55 -
From the Tool app's process to function execution 1. The
framework's AppFunctionService checks the caller (is it system_server?) 2. Switches to the main thread 3. androidx's AppFunctionService looks up metadata off the main thread, then switches back 4. The KSP-generated concrete Service dispatches with when and calls the function body on the main thread Toolprocess:Binderthread Framework AppFunctionService(IAppFunctionService.Stub) Mainthread getMainExecutor androidx AppFunctionService Metadata GeneratedService onExecuteFunction when(functionIdentifier) Developer'sfunction AOSP frameworks/base: AppFunctionService.java L118-170 / androidx: AppFunctionService.kt L84-148 / Demo: generated Service 56 -
The whole flow (again) Buildtime Tool Toolapp system_server OS Agent
Agentapp Atinstall Runtime ①KSP ⑧Functionbody GenerateXML Toolprocess ②AppsIndexer ③AppSearch ReadXML ⑥Checks Functionlist Permission/enabled ④search Getlist/observe ⑦bind CallTool ⑤execute Requestexecution Agent and Tool never connect directly. The system is always in between 57 -
What I built tool: two tool apps tripapp: getTravelInfo /
saveItinerary / searchItineraries / bookFlight message: sendMessage appfunctions-koog-agent-demo/ ├── tool/ │ ├── tripapp/ Travel app │ └── message/ Messaging app └── agent/ Agent app agent: an agent app built with Koog Input field, responses, and tool-call display Shows the number and list of available tools LLM: gemini-3.5-flash-lite The agent is placed in priv-app on a rooted emulator to get the execution permission github.com/hiroaki404/appfunctions-koog-agent-demo (Koog 1.2.0) hiroaki404/appfunctions-koog-agent-demo 59 -
Normally, Koog tools are fixed at build time Tools are
written as classes Argument types are also fixed at build time But AppFunctions are only known at runtime Some apps are installed later // The usual way: a fixed tool that can only call bookFlight object BookFlightTool : SimpleTool<BookFlightTool.Args>() { @Serializable data class Args(val from: String, val to: String, val date: LocalDate) ... } val toolRegistry = ToolRegistry { tool(BookFlightTool) } Illustrative example (same idea as the demo's early fixed tool AddTool.kt, commit 6fe2519) 60 -
Building tools from metadata From the AppFunctionMetadata obtained by search
One function = one tool Build Koog's tool description, ToolDescriptor, at runtime The Tool's execute calls executeAppFunction Arguments are received as a JSONObject Same shape as Koog's McpTool Demo: agent/.../AppFunctionTool.kt / AgentRunner.kt (excerpt) Create one Tool per function // AgentRunner.kt val tools = appFunctions.mapNotNull { metadata -> val descriptor = metadata.toToolDescriptor() AppFunctionTool(manager, metadata, descriptor) } Build ToolDescriptor class AppFunctionTool( private val manager: AppFunctionManager, private val metadata: AppFunctionMetadata, descriptor: ToolDescriptor, ) : Tool<JSONObject, String>(..., descriptor) { override suspend fun execute( args: JSONObject, ): String { val parameters = ... // repack args val response = manager.executeAppFunction( ExecuteAppFunctionRequest( metadata.packageName, metadata.id, parameters)) return ... // response to JSON } } Args stay as JSON Run executeAppFunction 61 -
Converting between Koog types and AppFunctions types // Inside toToolDescriptor()
on p61: convert each arg's type fun AppFunctionMetadata.toToolDescriptor() = ToolDescriptor(name = ..., description = ..., requiredParameters = parameters.filter { ... }.map { ToolParameterDescriptor(it.name, ..., it.dataType.toToolParameterType(...)) }, ...) fun AppFunctionDataTypeMetadata.toToolParameterType( components: AppFunctionComponentsMetadata, ... ): ToolParameterType = when (this) { is AppFunctionStringTypeMetadata -> enumValues?.let { ToolParameterType.Enum(...) } ?: ToolParameterType.String is AppFunctionLongTypeMetadata -> ToolParameterType.Integer is AppFunctionReferenceTypeMetadata -> components.dataTypes[referenceDataType]!! .toToolParameterType(components, ...) ... } To Koog types ①Whenbuildingtools AppFunctionMetadata AppFunctionstypes ToolDescriptor Koogtypes ②Whenrunningtools Args(JSON) ReturnedbyLLM AppFunctionData ToexecuteAppFunction Follow references Demo: agent/.../AppFunctionSchemaConverter.kt (excerpt) 62 -
What reading the design revealed What I learned The agent
is an app holding a permission; the system lists and runs functions Tool apps and agent apps interact indirectly through the system It is built on Android's existing assets and Kotlin's, such as KSP, Flow, and KotlinPoet How what developers write is used Developers write When used How it gets there app_metadata.xml Runtime (if the agent reads it) Agent reads it directly via PackageManager KDoc / function definitions Registered at install; returned in search results KSP XML AppsIndexer AppSearch search results Service (implementation) When the function runs System binds and calls it → androidx: AppFunctionPackageMetadata.kt L115-130, AppFunctionSearchSpec.kt L95-125 → → What it affects Which app to use, and how → Function the LLM picks, and its args Processing, threads, errors, types 65 -
How to design a good Tool Write with when app_metadata
and KDoc are used in mind Put cross-function call order in app_metadata, and function descriptions in KDoc No need to stuff keywords for search, it seems (Depends on the agent implementation, though) Watch out for type complexity LLMs struggle with structured data. Test complex types especially well androidx: AppFunctionSearchSpec.kt L95-125 66 -
How to design a good Tool (cont.) Concurrency and waiting
Functions run on the main thread. Move heavy work off it with withContext ( Dispatchers.IO for I/O, Dispatchers.Default for computation) Timeouts are up to the agent. Hand long work to WorkManager and return "started" (e.g. downloads) Assume functions may run while the app is in use Some functions run only while the app shows a specific screen (@AppFunctionSignature) androidx: AppFunctionDeclaration.kt L26-30 / AOSP frameworks/base: AppFunctionService.java L128-135, ServiceConfig.java L47-50 67 -
How to design a good Tool (cont.) Safety (confirming function
execution) → The library has no confirmation mechanism generally don't expose destructive operations; if you do, ensure safety on the Tool side I prototyped a confirmation dialog and A2UI The Tool returns UI information, and the agent renders it with A2UI Approval parameters are defined by a custom agreement, so the agent can show a confirmation dialog AppFunctions probably has no such mechanism yet, so I hope for one in the future Demo: hiroaki404/appfunctions-koog-agent-demo (a2ui branch) 68 -
Good Tool design: other points Don't expose domain-layer or data-layer
methods as-is From the agent's view, it is the API layer Design inputs and outputs for agents, and add logic that assumes calls from agents Descriptions are read by the LLM. Write what the function does, not commands English is recommended (Google's answer) Arguments with default values become optional. But the values aren't used; null/0/false arrive Return errors as the AppFunctionException subclass matching the cause For details, see the official AppFunctions quality guidelines or the skill developer.android.com/ai/appfunctions/quality-guidelines github.com/android/skills/tree/main/device-ai/appfunctions Issue Tracker: b/551999767 (KDoc language) / AppFunctions Quality guidelines (2026-10-28) 69 -
The future of AppFunctions Restrictions and permissions Until now, both
Agent and Tool were limited to select apps Tool: likely to become widely available Agent: calling other apps remains privileged Own functions via ML Kit in-app agents When tools multiply, which gets chosen? androidx's AppFunctionManager cannot take keywords as a query It assumes fetching the whole list at once without filtering by content (Filtering by package name or function name is possible) The system does no ranking. Selection is up to the agent implementation and the LLM Like the situation with countless MCP servers and skills AOSP frameworks/base: AndroidManifest.xml L9492-9515 / SystemAppFunctionAllowlistReader.java L126-187 / ADS 2026 / d.android.com/distribute/aep 70 -
The future of AppFunctions (cont.) Engagement Tasks get done without
opening the app → what about ad and billing models? On the other hand, execution from agents creates a new inflow channel Functions can be enabled/disabled at runtime (e.g. only for paying users) Results can also lead users to open the app (PendingIntent) Security Prompt injection is often mitigated with guardrails on the agent side Still, Tools should be built without relying solely on the Agent or the system Don't expose whole DBs or files 71 -
Summary Question Answer Q1. With just an annotation and KDoc,
KSP generates XML, and AppsIndexer puts it into AppSearch. The how can it find and choose? LLM chooses after reading app_metadata and KDoc Q2. How can it find apps installed after the agent app started? Q3. For a function in another process, who authorizes it and how does it run? AppsIndexer registers them on PackageManager's install notifications. The change arrives via a Flow, and the agent searches again system_server checks it and mediates via Binder, running it in the Tool's process. Tools can also declare who may call them 72 -
References (1/2) Official documentation AppFunctions overview (including FAQ) https://developer.android.com/ai/appfunctions Add
AppFunctions to your app https://developer.android.com/ai/appfunctions/add-appfunctions AppFunctions quality guidelines https://developer.android.com/ai/appfunctions/quality-guidelines Create and manage connected apps https://developer.android.com/ai/appfunctions/create-manage-connected-apps AppSearch https://developer.android.com/develop/ui/views/search/appsearch androidx.appfunctions release notes https://developer.android.com/jetpack/androidx/releases/appfunctions Videos Android Dev Summit 2026 keynote https://www.youtube.com/watch?v=8Lmv7LGgL4E Integrate your app with the Android intelligence system https://www.youtube.com/watch?v=UsqDEfyOH2E Extend your app to Gemini and Search with the Connected Apps Platform https://www.youtube.com/watch?v=-4G-4w9FKDw The Android Show: I/O Edition 2026 https://www.youtube.com/watch?v=dXCCleAddEA 73 -
References (2/2) Source code and samples androidx appfunctions (1.0.0-beta01) https://github.com/androidx/androidx/tree/androidx-main/appfunctions
AOSP frameworks/base (Android 17) https://android.googlesource.com/platform/frameworks/base/+/refs/heads/android17-release/core/java/android/app/appfunctions/ https://android.googlesource.com/platform/frameworks/base/+/refs/heads/android17-release/services/appfunctions/ AOSP AppSearch (appsindexer) https://android.googlesource.com/platform/packages/modules/AppSearch/+/refs/heads/android17release/service/java/com/android/server/appsearch/appsindexer/ android/appfunctions (Testing Agent) https://github.com/android/appfunctions Koog https://github.com/JetBrains/koog Issue Tracker https://issuetracker.google.com/issues/490650368 (not called from Gemini) https://issuetracker.google.com/issues/539569496 (multi-module) https://issuetracker.google.com/issues/551999767 (KDoc language) Demo repository 74