General models can write the pipeline. They can't read the bill.

The fifth interview in MotherDuck’s series is with Josh Wills, who ran data engineering at Slack and now works on text and multimodal data at very large scale at DatologyAI.¹ The headline comes from his line that data engineering is the cache invalidation problem in a batch-processing context. The headline is fine, but two other things in the interview interested me more: how he works with agents today, and why he thinks they still can’t do data engineering well.

The agent writes the code; he still watches the bill

Agents now write nearly all of his code. He reviews, reads design documents, and hands off chores like fixing flaky tests and running database migrations.¹ The models are the general ones everyone uses: he dropped Opus 5 for OpenAI’s Codex and later went back to Opus 5.5.¹

His real complaint is cost. Agents write pipelines that produce the right answer and waste money doing it. You can vibe code a Databricks job that costs about $200,000 a day, he says, and no competent data engineer watching would have written it.¹ He blames training. Math, front-end work and SQL with an explain plan give fast feedback, and models are good at them. A pipeline that runs for hours across hundreds of machines is something models have barely trained on, and it gives no quick signal.¹ If a full run takes four hours, an agent gets six tries a day. A CSS change shows its result in seconds.

So he calls himself an emergency exception handler: the agent flies on autopilot, and he takes over when a pipeline breaks.¹

Slow feedback is where general models struggle

Josh draws the comparison himself. Biology and physics labs have to wait for the culture to grow, and data engineering has a lightweight version of that wait.¹ Follow that logic and bioengineering, chemical engineering or aerospace, where one try can take days and a lot of money, will be even harder for a general model.

Other evaluations point the same way. Epoch AI gave six frontier models 11 real tasks from its own work and concluded they can’t replace its staff yet. The models were reliable on well-defined coding and analysis, but missed the organisation’s unwritten standards and designed vague experiments.⁴ Nathan Lambert expects the most valuable superhuman abilities to land in math and coding.⁵

What Josh wants, then, is an agent that knows the trade. General models won’t give him that soon.

Three ways a specialist agent could arrive

Train for the domain. The missing pieces are training data and RL environments with long, large pipelines in them.¹ Whoever builds those environments can build a data engineering model.

Build a system around the general model. A large model writes the pipeline, a small decision model takes over part of its computation, and monitoring catches the rest. The MotherDuck piece itself mentions TypeSafe AI’s Jev as basically what Josh wanted.¹ Jev returns typed decisions with probabilities instead of text. The vendor says input costs $0.042 per million tokens with output free, and that in its own workflow evals Jev was 193.6 times faster and 444.6 times cheaper than frontier models, which it expects is the high end of real-world gains.²

What a model like this can replace is the part of a pipeline that is really a judgement. An upstream table changed: which downstream partitions need rebuilding? Does this batch need a full rerun? Which records get which label, and which are junk? Today those calls are made with full scans, stacks of rules or LLM calls, and none of them are cheap. Make each one a sub-second decision: when the model is confident, reuse what is already computed or compute only what changed; when it isn’t, fall back to the full job. That lands right on Josh’s core point, deciding what to precompute and what to compute on demand.¹ Hand part of that trade-off to a small model and you save on compute that runs every day.

It can’t replace the parts that have to be exact. Aggregates, reconciliations and finance figures have to be computed in full. Swap them for a probabilistic guess and you save money by getting the answer wrong. It is the same confidence routing I argued for in my last post: the small model takes what it is sure of, and everything else goes the reliable way.

Monitoring still matters, but only as a backstop. Databricks says plainly that its budgets are monitoring, not a hard cap, and suggests running a pacing alert every four hours.³ At Josh’s $200,000 a day, about $33,000 could be gone by the time it fires.

Lose the job altogether. Compute gets cheap enough, and agents smart enough and with long enough memory, that they simply process the data. Josh disagrees. Data engineering won’t go away, he says, because the precompute-or-compute-later trade-off keeps moving; he calls it a problem for the superintelligence.¹ Lambert, though, expects the cost of intelligence to fall almost exponentially.⁵ The cheaper it gets, the closer that third path comes.

I want that third path and I’m afraid of it. I want it because nobody would have to babysit a bill again, and data work would get far cheaper. I’m afraid of it because once that trade-off can be handed off too, there isn’t much left of a data engineer’s job.