Who Babysits The Agent at 3 a.m.? Automation Using Google AX

Handing a Jira ticket to an AI agent takes a minute. Keeping it alive for three hours is the hard part: it waits for a Slack reply, crashes halfway, or loops until the bill gets interesting. We look at three jobs teams already want agents for: turning tickets into pull requests, reviewing PRs, and fixing the alert that pages SRE at 3 a.m. Then we show how Google AX, Google's new open-source runtime for long-running agents, handles them on GKE with Workload Identity, LiteLLM, ArgoCD and HolmesGPT.

Transcript

1. Google Cloud Meetup Vienna 21.10.2026 Who Babysits The Agent at 3 a.m.? Automation Using Google AX Sebastian Waldbauer, Senior Software Engineer
2. THE RED LINE · Do it RIGHT. 7 CH AP TE RS Agenda One real job, start to finish: Maya's 3 a.m. alert. Then Q&A. 01 02 03 04 05 06 07 Setup Problem Meet AX The run Under the hood On GCP Decide Our stack today What hurts today What you write Alert to fix PR Pause and limits Deploy, security Costs, next step 3
3. SE TUP · Do it RIGHT. TOD AY, W ITHOUT AX Today, the alert wakes Maya Invoice service metrics invoice-worker on GKE fires Mimir Stores metrics Alertmanager sends Routes the alert Pager Calls Maya at 03:12 alert: InvoiceWorkerOOM expr: oom_events > 0 deploys wakes The rule we deployed T OD AY : AL L B Y H A N D ArgoCD syncs Deploys the merged fix PR Git repo Fix PR, hours later Setup Problem Manual digging opens Logs and guesswork Meet AX The run Under the hood Maya, on call Half asleep at 3 a.m. On GCP Decide 4
4. PR OBLEM · Do it RIGHT. T HE BI GGE R PI C TUR E Agents now run for hours 2022 2024 2025 2026 Chatbots Assistants Agents Long-running runs for: Seconds runs for: Seconds runs for: Minutes runs for: Hours to days Our tools were built for the first three. The fourth is a new kind of workload. Setup Problem Meet AX The run Under the hood On GCP Decide 5
5. PR OBLEM · Do it RIGHT. OUR STOR Y Maya's 3 a.m. page M Maya SRE, billing platform on-call TUE 03:1 2 0 3 : 20 03: 45 05: 00 Paged Uses an agent Agent asks Pod restarts “invoice-worker OOMKilled, again” › › A script on a pod, calling Gemini The pod waits, and still costs money. › An hour of digging gone. Start again. This talk: from alert to fix PR, with no babysitting. Setup Problem Meet AX The run Under the hood On GCP Decide 6
6. MEET AX · Do it RIGHT. WITH AX Same stack, AX takes the page Invoice service metrics invoice-worker on GKE fires Mimir Stores metrics Alertmanager sends Routes the alert HolmesGPT Turns it into a Task alert: InvoiceWorkerOOM expr: oom_events > 0 deploys creates Task The rule we deployed N E W: A X R EP L AC ES TH E 3 A. M . P A GE ArgoCD syncs Deploys the merged fix PR Git repo Hotfix now, fix PR later Agent sandbox starts Reads Mimir, finds the fix AX on GKE Stabilises, then fixes Service up in minutes, real fix by breakfast. Setup Problem Meet AX The run Under the hood On GCP Decide 7
7. MEET AX · Do it RIGHT. THE LAYER S Two layers, two jobs Agent Substrate AX Think of it as The runtime The dispatcher What it does Isolates, sleeps, wakes agents Turns alerts into Tasks Status Open source Preview Why AX? It runs hundreds of agents without a babysitter. Few agents? GKE Agent Sandbox (GA) gives one isolated pod per agent. Source: Google Cloud docs, 2026 Setup Problem Meet AX The run Under the hood On GCP Decide 8
8. MEET AX · Do it RIGHT. ONE M AN IFE ST You describe the job. AX does it. Maya's alert Alert › Task › Agent › Run › PR invoice-worker-oom.yaml A Task, a Workspace, a Model. One file. Apply it like any Kubernetes resource. $ ax apply -f invoice-worker-oom.yaml HolmesGPT writes this from the alert. Nothing else to deploy. Setup Problem apiVersion: ax.io/v1alpha1 kind: Task metadata: name: invoice-worker-oom spec: workspace: billing-platform # warm start model: gemini-flash serviceAccount: sa-oncall-agent # Workload Identity resources: cpu: "2" memory: 4Gi prompt: > Find why invoice-worker is OOMKilled, then open a fix PR. Meet AX The run Under the hood On GCP Decide 9
9. MEET AX · Do it RIGHT. FROM ALE RT TO TASK The alert picks the playbook 1 · A LER T C O MES I N 2 · L O O K U P TH E T Y P E 3 · TA S K I S C R E A TE D OOMKilled → Memory playbook alert: InvoiceWorkerOOM playbook: memory HighLatency → Latency playbook type: OOMKilled input: service: invoice-worker namespace: billing CrashLoop → Crash playbook service: invoice-worker mitigate: memory up to 2× Anything else → Page Maya playbooks.yaml · reviewed in Git Facts from the alert, not free text Fixed playbooks, changing facts. New alert type? Add one row. Setup Problem Meet AX The run Under the hood On GCP Decide 10
10. MEET AX · Do it RIGHT. THE AGEN T C ODE The agent: 20 lines of Python Maya's alert Alert › oncall_agent.py Task · › Agent › Run › PR LangGraph + LangChain 1 Model through the gateway 2 Four tools, nothing else import os from langchain_openai import ChatOpenAI from langgraph.prebuilt import create_react_agent from tools import read_file, edit_file, run_tests, open_pr # The LiteLLM gateway speaks the OpenAI API llm = ChatOpenAI( base_url="http://litellm:4000", model="gemini-flash") 1 agent = create_react_agent( llm, tools=[read_file, edit_file, run_tests, open_pr], prompt="Hotfix only if the playbook allows. Else PR + ask.") 2 3 # The task itself comes from AX, per alert task = os.environ["AX_TASK_PROMPT"] agent.invoke({"messages": [("user", task)]}, {"recursion_limit": 40}) Setup Problem 3 System message: the rules Hotfix only where the playbook allows 4 4 Meet AX The run Under the hood Task: new for each alert User message, from the Task On GCP Decide 11
11. THE RUN · Do it RIGHT. AL E RT → FI X P R Stable at 03:20, fix PR by 08:12 Maya's alert Alert › Task › Agent › › Run PR P AUSE D 4 H 27 MIN · N O C PU , N O TO KE N S 03:12 Alert → Task 03:13 Sandbox is up 03:20 03:35 Hotfix: 2Gi, stable Asks a question 27 min working Setup 4 h 27 min paused Problem Meet AX The run 08:02 Resumes 08:12 Opens the fix PR 31 steps logged Under the hood On GCP Decide 12
12. THE RUN · Do it RIGHT. SPED U P 5 hours in 30 seconds Maya's alert Alert › › Task Agent › Run › PR sebastian@posedio: ~/billing — ax watch $ ax get tasks -l source=holmesgpt task.ax.io/invoice-worker-oom Queued $ ax watch invoice-worker-oom 03:12:04 ax QUEUED task created from alert InvoiceWorkerOOMKilled 03:12:51 sandbox STARTED warm pool · billing-platform mounted · 4 tools 03:19:40 agent step 9 hotfix PR #516: memory 1Gi → 2Gi · auto-merged 03:21:09 tool pytest 51 passed, 1 failed (leak test) 03:27:33 agent step 14 fix: cap export cache at 256 MB 03:33:50 tool pytest 52 passed 03:35:12 slack ASK #billing-oncall: "Stable on 2Gi. Cap the cache, back to 1Gi?" 03:35:13 ax SUSPENDED checkpoint 38 MB · cpu 0 · tokens 0 08:02:41 slack REPLY @maya: "Yes, cap it, back to 1Gi" 08:02:42 ax RESUMED from checkpoint in 398 ms 08:09:55 tool pytest 52 passed 08:12:30 tool open_pr PR #517 opened · 3 commits 08:12:31 ax COMPLETED 31 steps · 27m active · 4h27m suspended $ ▌ [ax] invoice-worker-oom state: completed queued SUSPENDED running ✓ Setup Problem tokens: tokens: 0.9M 01.0M 280k 520k Meet AX steps: steps: 29 031 10 18 19 The run clock: clock: 08:12 03:12 03:13 03:21 03:35 06:00 08:02 08:12 Under the hood On GCP Decide 13
13. THE RUN · Do it RIGHT. TH E FIX P R What Maya finds at 08:12 Maya's alert Alert › Task › Agent › Run › PR Fil es changed PR #517 Cap the export cache opened by oncall-agent · 08:12 · 2 files src/export/cache.py - _cache = {} + _cache = LRUCache(maxsize=256) WHAT HAPPENED invoice-worker OOMKilled at 03:12. AL READY DEPL OYED Hotfix PR #516, auto-merged 03:19: memory 1Gi → 2Gi. Stable since 03:20. deploy/invoice-worker.yaml limits: memory: 2Gi # hotfix #516 + memory: 1Gi THIS PR ✓ pytest 52 passed · leak test added Caps the cache, sets memory back to 1Gi, as @maya approved at 08:02. Service already up. Maya reviews one small diff over coffee. Setup Problem Meet AX The run Under the hood On GCP Decide 14
14. UN DE R T HE H O OD · Do it RIGHT. SUSP END & RESUME Waiting costs nothing P L AI N K U BE R N E T E S P O D Idle for 4 h, still billed for CPU + memory Work A X TA S K 03:35 Agent asks Maya Working Work 08:02 Maya replies Working again Paused: no pod, no CPU, no tokens Save Restore Saved copy Memory, files, last step Like a saved game: AX stores the agent, hands the machine back, then loads it again. Setup Problem Meet AX The run Under the hood On GCP Decide 15
15. UN DE R T HE H O OD · Do it RIGHT. SPE ND L IM ITS Three limits stop a stuck agent S TU CK LO OP Same call, again Every call costs tokens GATE 1 · IN THE AGENT › Looping? Agent stops itself at step 40 GATE 2 · AX › GATE 3 · LI TELLM Too big? AX ends it at its CPU / memory cap › Too pricey? LiteLLM blocks calls over budget S TOP PED › Before the bill does Three gates in a row: if one misses the loop, the next one catches it. Setup Problem Meet AX The run Under the hood On GCP Decide 16
16. O N G OO G L E CLO U D · Do it RIGHT. R EFERENCE How AX runs on Google Cloud G O O GL E S E R V I C E S A X U S E S YO UR G KE C LUS T ER ADDED BY AX Vertex AI · Gemini models AX controller Agent Substrate Queues, pauses, resumes Memorystore for Redis · Event log Isolates and snapshots agents Cloud Storage · Sandbox snapshots ALR EADY TH ER E Invoice service Mimir + Alertmanager invoice-worker Workload Identity · Keyless access Metrics and alerts Artifact Registry · Agent images HolmesGPT LiteLLM + ArgoCD Creates the Task Gateway, deploys Cloud Logging · Audit trail AX adds two pieces. Everything else is a Google service you may already run. Setup Problem Meet AX The run Under the hood On GCP Decide 17
17. O N G OO G L E CLO U D · Do it RIGHT. P IP EL IN E One new step in your pipeline 1 AL ERT Prometheus invoice-worker OOMKilled 2 › TA SK 3 HolmesGPT Turns the alert into a Task › AG ENT 4 AX on GKE › Investigates in a sandbox, sleeps while it waits PR 5 Your Git repo Fix arrives as a pull request › DEPLO Y CI + ArgoCD Review, merge, ship as usual The o nly new s tep Maya keeps her tools. Your Git, CI and deploys stay the same. Setup Problem Meet AX The run Under the hood On GCP Decide 18
18. O N G OO G L E CLO U D · Do it RIGHT. SE CUR ITY The agent never holds a key H O W M A Y A ’ S A G E NT G E T S A C C E S S 1 Runs as its own identity 2 One Kubernetes account per agent › Gets a short-lived token Issued by Workload Identity, no secret stored 3 › Opens only two doors Vertex AI and Maya’s repo only I F T H E A G E N T I S T R I C K E D , I T C A N’ T … × Steal a key × Push to main × Call unknown sites × Hide what it did No key files in the sandbox It can only open a PR Only allow-listed sites Every call is logged Worst case: one repo, one PR, and a full log. Setup Problem Meet AX The run Under the hood On GCP Decide 19
19. DEC IDE · Do it RIGHT. W HE RE TH E MO NE Y G OE S Tokens are most of the bill P I LO T · ON E WOR KL OAD PRODU CTI O N F LEET ~20 tasks a day ~500 tasks a day One workload, one team Many teams, many workloads ~90% LLM tokens ~97% LLM tokens Sandbox compute ~1% Sandbox compute ~1% Platform ~9% Platform ~2% Setup Problem Meet AX The run Under the hood On GCP Decide 20
20. DEC IDE · Do it RIGHT. ADOPT OR W AIT Three yeses? Try AX. QU ES TI O N 1 yes Do agents run for hours? QU ES TI O N 2 yes Do you run Kubernetes? no QU ES TI O N 3 yes Many agents, or risky code? no OU TCO M E Pilot AX on GKE Start with one workload no Not needed yet Wait, or go managed Plain Jobs will do Use Cloud Run jobs Use a hosted runtime No new layer needed Three yes answers: take it seriously. One no: the grey box is the better next step. Setup Problem Meet AX The run Under the hood On GCP Decide 21
21. DEC IDE · Do it RIGHT. OUR RE COM ME ND AT I ON Start with one agent 1 Spike on kind 2 One workload 3 Pilot on GKE 4 Decide YO U GET YO U GET YO U GET YO U GET Team knows the Baseline: time, cost, Measured resume, A decision backed by primitives failures cost, isolation numbers Start small. Don't bet the platform. Setup Problem Meet AX The run Under the hood On GCP Decide 22
22. DEC IDE · Do it RIGHT. TAKE THIS WITH YOU AX on one page FIT Hours-long agents · Already on Kubernetes · Many or untrusted agents ON GCP GKE + Agent Substrate · Vertex AI · Memorystore · Cloud Storage B U D GE T Tokens are the bill · Budget per team in LiteLLM · Measure 10 tasks GUARD RAILS Hotfix only per playbook · Everything else a PR · Every call logged F I R ST S TE P Run the kind quickstart from github.com/google/ax, then point it at your noisiest alert. Setup Problem Meet AX The run Under the hood On GCP Decide 23
23. Do it RIGHT. CONTACT US: Weyringergasse 1-3/DG 1040 Wien www.posedio.com office@posedio.com THANK YOU!