Building Developer Products in the Agentic Age

This is a talk given by Ching-Wei Chen at the LeadDev New York Meetup on Mar 25, 2026.
The presentation uses a case study of a major Spotify outage to emphasize the need for proactive reliability and performance issue detection. It introduces Datadog APM Recommendations as a developer product that detects common anti-patterns like aggressive retries and N+1 queries. The rise of capable coding agents is "shrinking" the Software Delivery Loop, requiring a shift to building specialized tools for AI Agents.
This shift necessitates building specialized tools that are designed to be consumed directly by AI Agents, rather than just humans. Effective agent tools must be optimized to "save cognition" by compressing deep insights from vast data (like trace spans) and minimizing friction through discoverability and sensible defaults. Specialized tools for tasks like identifying latency bottlenecks have shown significant performance improvements in internal agent investigations. The talk concludes that agent tools should be efficient, broadly useful, and discoverable to achieve the simplicity of tools like grep

Transcript

1. Building Developer Products in the Agentic Age TALK Ching-Wei Chen Director of Engineering, Datadog LeadDev New York Meetup, Mar 25, 2026
2. Ching-Wei Chen (@cweichen) Director of Engineering, Datadog Previously: • VP of Engineering, Crossing Minds • Director of Engineering, Spotify 2
3. Building Developer Products in the Agentic Age
4. Story Time
5. 5 #incident-10631
6. Incident 10631 6 The Problem: Aggressive retries
7. Incident 10631 7 The Solution: Exponential backoff
8. Datadog APM Recommendations 8 Detect Resolve Proactively detects potential reliability or performance issues, including: • Aggressive Retries, N+1 queries, missing DB indexes, memory leaks, and more.
9. Meeting Developers Where They Are
10. 10 The Software Delivery Loop Plan Build Deploy Respond Observe
11. 11 The Software Delivery Loop Plan Build Deploy Respond Observe
12. 12 Recommendations Across the Loop Plan Observe
13. 13 Recommendations Across the Loop Observe Build
14. 14 Recommendations Across the Loop Observe Deploy
15. 15 Recommendations Across the Loop Observe Respond
16. 16 The Software Delivery Loop Plan Build Deploy Respond Observe
17. 17 The Agentic Software Delivery Loop
18. Building Tools for Agents
19. Who are you building for? 19 Humans: • Highly evolved visual processing system • Can find patterns and anomalies in vast data, but only with visual tools AI Agents: • Can interpret raw data directly • Don’t need visual representation to find patterns • Can consume data through MCP Tools, APIs, or CLIs
20. Tools: Survival of the fittest 20 “Nobody is coming for grep” Steve Yegge proposes an evolutionary view of Tools (including CLIs), where good Agents will favor Tools that “save cognition”, i.e. Tools that • Compress deep insights • Minimize friction • Are efficient • Have broad utility • Are discoverable • Inspired by Software Survival 3.0 by Steve Yegge
21. Minimize Friction • An Agent won’t use what it doesn’t know how to use.
22. Minimize Friction • This is where Tool Descriptions and sensible defaults can play a big part. * From MCP Tool Descriptions are Smelly! by Hasan, et al
23. Minimize Friction +31% lift on internal Agent evaluations
24. Compress Insights 24 • Trace data can be huge, 100’s of spans, up to 10’s of MBs in size (millions of tokens)
25. Compress Insights 25 • Processing these costs an Agent a lot of time and tokens • We built a specialized Tool to locate the latency bottleneck in a given Trace Real Bottleneck Misidentified Bottleneck
26. Compress Insights 26 • Processing these costs an Agent a lot of time and tokens • We built a specialized Tool to locate the latency bottleneck in a given Trace Real Bottleneck Misidentified Bottleneck +15% lift on Bits SRE Agent Investigations
27. Takeaways
28. 28 Apps: Meet the Agentic developer where they are
29. Tools: Be like grep 29 “Nobody is coming for grep” As you are designing your Tools for Agentic use, make sure to optimize for Tools that: • Compress deep insights • Minimize friction • Are efficient • Have broad utility • Are discoverable Inspired by Software Survival 3.0 by Steve Yegge
30. Thank you! We’re hiring! https://careers.datadoghq.com Ching-Wei Chen @cweichen