Everyone’s shipping AI agents that promise to triage incidents and fix dashboards — but how do you know if yours is actually good, or just good in the demo? This talk walks through building a real observability agent using grafana/ai-sdk (Grafana’s open-source Go SDK for tool-calling, streaming agents), then puts it under fire with grafana/o11y-bench, an open benchmark that grades agents on real Prometheus/Loki/Tempo tasks against a live Grafana stack — not just ‘did it produce an answer’ but ‘was the answer actually verifiable against ground truth.’ We’ll cover wiring an agent to query metrics, logs, and traces and make dashboard edits; where agents silently get subtle things wrong (a syntactically valid query that selects the wrong series); and what o11y-bench’s Pass@3 vs. Pass^3 metrics reveal about consistency vs. one-off luck. Live benchmark run included — open source.
Wei-Chin Call is a Staff Emerging Products Manager at Grafana Labs, where she helps shape the company’s next generation of agentic full-stack observability products. She joined Grafana Labs as a Senior Solutions Engineer, partnering with customers on technical proof-of-concepts and driving adoption of the Grafana Cloud stack. Prior to Grafana, Wei-Chin managed a Solutions Engineering team at Cribl and held technical evangelism and sales engineering leadership roles at AppDynamics, where she built sales enablement curricula and led cloud-native training initiatives. She is certified in OpenTelemetry (OTCA), Kubernetes (CKAD, CKA), and Prometheus (PCA).
Jun Ding is a Senior Solutions Engineer at Grafana Labs, where he partners with enterprise customers to design and adopt observability strategies using the Grafana Cloud stack. He brings a software engineering background with years of development experience, giving him a practical perspective on how teams build, operate, and troubleshoot modern applications. Prior to Grafana, Jun was a Senior Sales Engineer at AppDynamics, leading technical engagements and proofs of value for strategic enterprise accounts. Across Grafana and AppDynamics, he has helped customers modernize monitoring, visualization, alerting, and performance optimization programs while translating complex technical requirements into business outcomes. He was a two-time Grafana President’s Club honoree and recognized as the top West Region Solutions Engineer in FY2026.