Getting Your Agents to Write (Good) Pipelines on Apache Spark

November 2, 10:35 AM–11:00 AM (PST) • Room 1

Agents are writing a ton of Spark code these days. Most of it is error ridden, and the errors are the boring kind: a column that doesn’t exist, a table nothing upstream declares, a session config invented from thin air. This talk covers what actually helps, drawn from my own work with OSS Spark. We’ll get into writing pipelines declaratively so structural mistakes get caught before any data moves and agent iterations are cheap, using Spark Connect to keep session context and credentials out of the agent’s hands, and how to structure your agent skills and prompts so your one-shots are much more successful. We’ll also discuss the tradeoffs between choosing the most performant model and managing tokens compared to changing the core infrastructure of your deployment.

OSA CON logo

Join OSA Con 2026

The anti-hype conference for open-source analytics and AI. Connect with engineers, maintainers, and technology leaders building the future of data.

November 2, 2026
San Francisco + Online

Register