Everyone’s shipping AI agents that promise to triage incidents and fix dashboards — but how do you know if yours is actually good, or just good in the demo? This talk walks through building a real observability agent using grafana/ai-sdk (Grafana’s open-source Go SDK for tool-calling, streaming agents), then puts it under fire with grafana/o11y-bench, an open benchmark that grades agents on real Prometheus/Loki/Tempo tasks against a live Grafana stack — not just 'did it produce an answer' but 'was the answer actually verifiable against ground truth.' We’ll cover wiring an agent to query metrics, logs, and traces and make dashboard edits; where agents silently get subtle things wrong (a syntactically valid query that selects the wrong series); and what o11y-bench’s Pass@3 vs. Pass^3 metrics reveal about consistency vs. one-off luck. Live benchmark run included — open source.