Three questions before a judge model replaces your filter script

Since October 9, a custom LLM-as-a-judge service policy in Unity Gateway can use OpenJev, built on Qwen3.5 4B, as its evaluator. It’s in Beta.³ A service policy is a checkpoint that every request to and response from a model service or MCP tool passes through.² You used to write one of two kinds: a SQL function that makes an exact call on keywords, tool names or arguments, or plain-language criteria for a chat model to judge.¹ OpenJev classifies without generating tokens, so it’s usually faster.³

Put simply, a decision model takes over part of what filter scripts do. I like that. But most Databricks teams work under heavy audit, and a fixed script guarantees that the same input always gets the same result and every step can be traced. So I read the docs next to what others have built and asked three questions. Can you see the confidence? Can you replay a verdict? What does a wrong one cost to undo?

In the gateway, OpenJev gives a verdict but no confidence

Inside a policy, OpenJev answers a two-option question: flag or allow. Its pick is the verdict, with no threshold to set. The flag option is worded to win when content may break your rules, so borderline content tends to get blocked. It gives no reason either.¹

The model does produce probabilities. Call it through the System One API, or run the ai_decide SQL function, and you get a probability for each option plus a confidence value.⁴⁵ A gateway policy collapses all of that into flag or allow.

The upside: Databricks has chosen the direction of error for you, which is to block more, and policies fail closed, so a missing field or any evaluation error counts as a denial.¹ The downside: the confidence-based routing I argued for in the d1 post isn’t possible here. Without a score, you can’t send the unsure cases to a larger model or a person.

You get a record, not a repeatable verdict

Auditing a script means replay: run the same input, get the same result, and you’ve shown the rule was applied. The Databricks docs say plainly that judge verdicts are non-deterministic, so the same input can get different verdicts, and that repeatable checks belong in SQL policies.²

So audit evidence has to move from replay to records. The unified trace table logs each policy’s verdict, and in Log mode (record, don’t block) what it would have done.² But delivery is best-effort, and Databricks calls it a complement to audit logs.⁷ The judge’s reasoning only shows up if you enable an inference table on the evaluator.² A judge model therefore needs extra record-keeping: the verdict, the prompt version and the evaluator version. Miss one and you can’t explain later why a call was blocked.

Rollback gets expensive once the verdict lands in a table

In the gateway, a policy only decides and never rewrites content. A wrong verdict blocks one call, and a prompt fix usually takes effect in a minute or two. There’s no history to repair.²

Pipelines are different. ai_classify and ai_decide label each row in SQL and write the labels to a table that downstream jobs read.⁵⁶ Both docs say Databricks may swap the underlying model when a better one appears; ai_decide warns that answers can vary between calls; ai_classify falls back to the legacy 1.0 unless you pin a version.⁵⁶ A hypothetical: in a 10-million-row history table, if just 1% of labels flip after a model change, that’s 100,000 rows to rewrite and explain to an auditor.

The rollback cost I worry about sits mostly in pipelines.

What others do: rules first, models for meaning, then distil

Databricks advises layering. Precise, repeatable rules go in SQL policies; only semantic checks that keywords can’t express go to a judge. Start in Log mode, review false positives and negatives on real traffic, then enforce. OpenJev works best on English.¹ Policies run in rank order and a denial stops the later ranks,² so the cheap, deterministic rules go first.

In May 2025, Josh Melton described a pipeline route on the Databricks Community technical blog. A large model labels the data, and those labels train classic models such as XGBoost and logistic regression. Semantic similarity and Levenshtein distance add a confidence signal to the labels, and cases below a threshold go to human review.⁸ My inference: a small model with fixed weights that your team holds gives the same output for the same input, and the platform can’t swap it out. That suits audits, though Melton’s post doesn’t discuss them.

How I’d use it: let the model judge, keep stored verdicts traceable

• Blocking decisions go in the gateway. SQL rules at rank 1, OpenJev only for semantic checks rules can’t express. Run Log mode first, with the evaluator’s inference table already on.
• Decisions that need confidence-based routing go in the pipeline with ai_decide, probabilities stored, and anything below the threshold sent to a person.
• Store each verdict with its model, function and prompt versions, append-only, with the function version pinned.
• Where a verdict must replay exactly, keep the script, or distil the labels into a small model you hold.

I still want decision models to replace filter scripts. First decide which way you want the model to be wrong, and make sure every verdict that lands in a table can be traced back.