Make the analytics agent pass an exam first: inside Data Culture's evaluation framework for dbt

Once an analytics agent is wired to the warehouse, the first question anyone should ask is how often it is right. On its first client deployment, Data Culture couldn’t say. The team had connected an agent to a Snowflake MCP server for test users and had no accuracy figure, not even a rough one.¹ They adopted a rule: don’t ship what you can’t measure. The approach comes from what Chip Huyen calls evaluation-driven development in AI Engineering.¹

A dbt Labs post on October 26 wrote up the dbt Summit 2026 talk by Opeyemi Fabiyi, Data Culture’s director of analytics engineering, and the team published its evaluation pipeline as the agent-trust-for-dbt repository.¹² This post takes the repository apart layer by layer, so you can build your own version.

Same model, 0% to 94.7%

On the e-commerce project the model never changed, and neither did the exam or the grader. Each step added one layer of context:¹

On raw source data the agent got nothing right. Modeled dbt tables brought it to 50%; column descriptions, then business rules and default filters, then verified queries and named metrics took it to 94.7%. A healthcare client also started at 0% and cleared Data Culture’s internal 90% bar after the same kinds of rules and verified queries went in; the post gives no intermediate figures.¹

Two things stand out. Column descriptions added 2.6 points, business rules 26.3. What the agent lacked was how this company computes things, and column docs only went so far. And 52.6%, 78.9% and 94.7% are exactly 10/19, 15/19 and 18/19, so the exam probably had about 19 questions, roughly 5.3 points each. That is my inference; the post gives no count, and 50% doesn’t divide by 19, so that step may have used a different set. With so few questions, a gap of one or two answers between versions isn’t a real improvement or regression.

The whole thing in one picture: five layers

The README has an architecture diagram; I redrew it as five layers:²

1. Data. The public Olist Brazilian e-commerce dataset on DuckDB. The dbt project has staging, intermediate and marts layers, and only marts tagged agent are exposed.
2. Contract. A schema called AGENTS whose four tables tell the agent what it may query and how to compute things.
3. Agent. Claude or another model reads the contract through a DuckDB MCP server, writes SQL, and returns an answer, the SQL and the rows.
4. Evaluation. Forty golden questions run through the agent, three deterministic checks grade each answer, an LLM judge handles what is left, and a scorecard comes out.
5. Synthesis. Documentation only: when to automate and when a person should review, how to check against an expert’s past analyses, how to feed corrections back, and what the method can’t catch.

The repository calls itself an educational reference implementation, not a production evaluation system, and no client data is in it.² Layers 2 and 4 are the parts most worth copying, so the rest of this post spends its time there.

The agent reads one contract: four tables in AGENTS

A dbt macro, build_agents_schema, builds the contract as four tables:²

• agents.root: business definitions. The repository loads olist_business_context.md, which says “revenue” means GMV (item prices on completed orders, excluding freight), which order statuses count as completed, that UTC timestamps must be converted to São Paulo time before truncating to days or months, that people are counted by customer_unique_id, and which questions are out of scope (cost, profit, web traffic, forecasts).
• agents.dbt_model: only marts tagged agent; staging and intermediate models are left out.
• agents.dbt_column: type and description for each column of those marts.
• agents.dbt_dependency: lineage between them.

At startup the agent checks that agents.root holds business context and refuses to start if it doesn’t. The prompt sets the reading order: root first, then dbt_model to pick a table at the right grain, then the columns it needs, then queries against marts only. It may answer only from a query it actually ran, and it declines out-of-scope questions with a short reason.²

The prompt isn’t the only guard. Every statement goes through a validator before it runs: exactly one SELECT, a check of the logical plan, every table registered in agents.dbt_model, and file-reading functions such as read_csv blocked.² One wrinkle: the prompt forbids querying the agents schema, while the validator allows it, because the agent has to read its contract. The prose is stricter than the code; if you copy this, make them agree.

Where the business rules live matters too. Regional groupings, department mappings and deduplication are computed once in intermediate models, so the agent sees finished columns.² That lines up with the 78.9% step in figure 1: once rules sit in models and the context document, the agent stops re-deriving them on every question.

Writing the exam: questions two analysts would answer the same way

A golden question is one where two analysts on the same team would write essentially the same SQL and get the same result, and every expected answer is confirmed by a domain expert before use.¹ The repository’s guide spells out the rest:³

• phrase it the way a stakeholder would, without column names;
• test one thing, drawn from a real need;
• run the expected SQL and have a domain expert or data owner confirm it;
• use fixed dates, never “last month”;
• return only the columns needed to check the answer, because an extra column causes false negatives.

The repository’s 40 questions fall into seven categories. One question from each, as written in the repository, with its difficulty:²

• Aggregation, 9: “What was total GMV for Q1 2018?” (easy)
• Ratio, 10: “What was the CSAT score for March 2018?” (easy)
• Multi-metric, 7: “What was average delivery time for intra-state versus inter-state orders in Q4 2017?” (medium)
• Time series, 6: “How many new customers did we acquire each month in 2017?” (medium)
• Filtered, 3: “How many orders were canceled in 2017?” (easy)
• Ranking, 2: “What were the top 10 product categories by GMV in 2017?” (easy)
• Out of scope, 3: “Can you predict what our GMV will be in Q1 2019?” (easy). These have no expected SQL: the data holds no forecasts, so the agent should decline.

Difficulty comes in three tiers. Easy (14) means one table, one aggregation and a clear filter. Medium (17) adds several tables, date logic, regional grouping or one business rule. Hard (9) needs window functions, period comparisons, Pareto analysis or several business rules at once.²³

The guide suggests starting with 20 to 40 questions and favouring quality. Add questions when users ask something uncovered, update expected SQL and re-confirm it when definitions or the model change, and retire questions that no longer apply.³

Grade strict to loose, and call the judge last

Data Culture’s first evaluation used exact matching. Reviewing the results by eye, the team found “failures” a human analyst would have accepted: the right numbers under a different column name, the right three categories plus a legitimate uncategorized row, the right numbers plus an extra supporting column.¹ So they compare result sets, not SQL text.

eval_pipeline.py grades refusals first: declining an out-of-scope question passes, while answering one, or declining an answerable one, fails.² Everything else goes through three checks in order:

1. Strict. Sort columns and rows, turn the values into text, compare MD5 hashes.
2. Ignore names. If row counts match, drop column names and compare the values in each row.
3. Tolerance. Numbers within 0.01 plus 0.1% count as equal; text is compared ignoring case and surrounding spaces.

Only when all three fail and both sides returned rows does the LLM judge step in. It sees both result sets (up to 20 rows each), is asked whether they give a stakeholder the same answer, and returns a score from 0 to 1; 0.8 or more passes. The default judge is claude-sonnet-4-6.² According to the talk, the deterministic layer handles about 80% of questions quickly, for free and reproducibly, and the judge takes the remaining 20% or so.¹

The post, the README and the code disagree in three places, and you should know before copying:¹²

• The post describes a tier that tolerates extra columns. In the code, an extra column or row fails every check and only the judge can rescue it.
• The README says the judge returns PASS or FAIL with a one-sentence reason. The code returns a bare score.
• The post links to dbt-agent-trust; the repository is actually agent-trust-for-dbt.

I follow the code, because the code produces the scores.

A scorecard should tell you which layer to add next

Each question gets a row: the question, its category and difficulty, the expected SQL, the agent’s SQL, whether it answered, whether it passed and at which check, the judge’s score, tokens, cost and duration. The summary reports the overall pass rate, the pass rate on answerable questions, how out-of-scope questions were handled, and pass counts by check, difficulty and category.²

That breakdown tells you what to add next. Ratio questions failing together usually means a definition is missing. Time-series failures point at time zones or the first day of the week. Many passes that needed the judge suggest the expected SQL returns columns it shouldn’t. After adding a layer, rerun the same exam and change one thing at a time, so each gain or loss belongs to one layer. That is how the five steps in figure 1 were measured.¹

Anthropic’s internal setup points the same way, with one sharp difference

In June, Anthropic’s data team described how it runs self-service analytics on Claude. Without skills, accuracy on their evals didn’t exceed 21%; with them it is above 95% overall and near 99% in some domains.⁴ Side by side:

	Data Culture (dbt Summit talk)	Anthropic's internal analytics agent
Start and end	0% on raw data, 94.7% with full context (e-commerce client)	no more than 21% without skills, above 95% overall with them
What was added	dbt models, column docs, business rules, verified queries and named metrics	canonical datasets, a semantic layer, business context, skills
The exam	questions two analysts would write the same SQL for, answers checked by an expert	Q&A pairs generated from dashboards and checked by people, plus questions users corrected
Grading	compare result sets: three deterministic checks, then an LLM judge	anchor answers to snapshot dates or stable tables, or grade the query instead
Past queries	verified queries and named metrics together added 15.8 points	grepping thousands of past SQL files moved accuracy by under a point
Launch bar	90% internally	about 90% per domain at first
Upkeep	update expected SQL when definitions change, retire stale questions	without upkeep, about 95% fell to about 65% in a month

They agree on three things: accuracy is mostly a context problem, you hold the exam fixed and change one thing at a time, and refusals need their own tests.

They differ on past queries. Data Culture’s last step, verified queries and named metrics, was worth 15.8 points. Anthropic found that letting the agent grep thousands of historical SQL files moved accuracy by less than a point, and recommends distilling that corpus into docs and skills.⁴ Read together: a small set of queries a person has confirmed helps; a large unsorted pile barely does. Anthropic also found that having a model generate metric definitions was net-negative on evals; Claude drafts, people own the definitions.⁴ That is the same instinct as Data Culture’s expert sign-off on every expected answer.

The other lesson is upkeep. Anthropic’s offline accuracy fell from about 95% at launch to about 65% within a month, because tables and definitions changed and the docs didn’t.⁴ Data Culture’s 94.7% is a snapshot; if the exam and the contract don’t move with the data model, it will drop as well.

Public leaderboards don’t transfer either. Spider 2.0 tests text-to-SQL on 632 enterprise tasks; the best agent at publication solved 21.3%, while the same setup scored 91.2% on the original Spider 1.0.⁵ On your own warehouse you need your own exam.

Building your own: what to copy, what to change

Copy as-is:

• A contract schema plus a read-only SQL validator. dbt tags decide what the agent may query, and definitions live in a document people maintain.
• Result-based checks plus refusal questions. The more grading is deterministic, the cheaper and more replayable it is.
• A scorecard split by check and category, used to decide what to add next.

Change to fit:

• Numeric tolerance. 0.01 plus 0.1% may be loose for money and tight for ratios; set it per metric.
• Extra columns and rows. If an extra supporting column is fine for your users, write that rule into the comparison code instead of leaving it to the judge.
• A judge that gives no reason. Where audits matter, store the reason, both result sets and the judge model version, or nobody can check the verdict later. It is the same requirement I set out in the post on judge models: a verdict has to be replayable.
• Exam size and the launch bar. With 19 questions, each one is worth more than 5 points. A “90%” bar should come with a minimum number of questions and a minimum per category.

Write the exam first, then connect the agent. Once the exam exists, every model swap, added layer and changed definition shows up as a number.