Getting Your Agents to Write (Good) Pipelines on Apache Spark
Agents are writing a ton of Spark code these days. Most of it is error ridden, and the errors are the boring kind: a column that doesn’t exist, a table nothing upstream declares, a session config invented from thin air. This talk covers what actually helps, drawn from my own work with OSS Spark. We’ll get into writing pipelines declaratively so structural mistakes get caught before any data moves and agent iterations are cheap, using Spark Connect to keep session context and credentials out of the agent’s hands, and how to structure your agent skills and prompts so your one-shots are much more successful. We’ll also discuss the tradeoffs between choosing the most performant model and managing tokens compared to changing the core infrastructure of your deployment.
