d1 decides in 8 ms. Who catches the times it's wrong?

A decision in 8 ms

On October 7, Liquid AI put two open-weight decision models on Hugging Face: d1-3B and an experimental d1-omni-600M.¹ A decision model doesn’t write anything. You give it a question and a fixed set of answers, and it picks one in a single forward pass. Liquid says d1-3B answers one question in about 8 ms on an RTX 4090 and about 50 ms on a Jetson Orin Nano, a small edge board.¹ For a yes/no question it returns the probability of “yes”. For a multiple-choice question it returns the pick, a confidence value and a probability for each option.²

On quality, the vendor says d1-3B scores 48.57 on the public split of Decision Index v0.2.1, the best result under 10B parameters and level with Decider 35B-A3B, a model about 12 times its size. Its average across seven public benchmarks is 82.9.¹

TypeSafe AI’s Jev takes the same approach. It returns structured decisions with probabilities, and the company says responses take 70 to 500 ms end to end. It has not published public benchmark results.⁴

Multiply the error rate by the call volume

My first worry about d1 is the one I had about Jev: how accurate can a model this small be?

Start with the arithmetic. Teams put models like this into production because they handle huge volumes of classification, moderation and control requests at very low cost. That volume is exactly why the error rate has to be read together with it: the number of wrong decisions per day is the error rate times the number of calls. Picture a moderation pipeline making a million decisions a day. At 90% accuracy that is 100,000 posts wrongly blocked or let through every day. At 99% it is still 10,000. Every point of accuracy is worth a lot, but the absolute number of mistakes rarely becomes small enough to ignore. So the better questions are how many mistakes a day, what each one costs, and how many the business can absorb.

It also needs the right comparison. Today’s process makes mistakes too: human reviewers get tired, large models make things up, and both cost more. A small model earns its place if it makes fewer mistakes than the current setup on the same data, and if the mistakes it does make can be caught.

Part of my worry is backed by Liquid’s own numbers. Fewer parameters means less room for knowledge: the model card gives d1-3B a 23.8 on the knowledge part of Decision Index.² Context is less clear-cut. The model accepts 32K tokens,² but long inputs are expensive: a 3.4K-token description of the current state pushes a single call from about 8 ms to 102 ms on an RTX 4090 and to 1.64 seconds on the Orin Nano.¹ Whether accuracy also drops with long inputs, Liquid doesn’t say, so that part stays an open question.

All of these figures come from the vendor, and nobody has yet shown they hold up in production.⁵ The model card has no limitations section and no calibration error metric. “Calibrated” appears only in the description.² I want models like this to exist, but before deploying one I would treat it as a component that still has to prove itself.

Four things I’d do before going live

If I were adopting d1, I would finish these four first.

Measure the error rate on your own data, turn it into “wrong decisions per day”, and ask the business whether it can live with that number. A benchmark average says little about your use case.

Use the probabilities. Liquid’s docs list routing and cascades as use cases, but give no guidance on thresholds, abstaining, or handing a case to a larger model or a person.³ You have to design that part. Act on confident answers; send the rest to a bigger model or a reviewer.

Start with actions you can undo. A wrong tag or priority can be fixed later. A deleted post, a blocked payment or a control command sent to a machine has a cost per mistake, and you should price it before you ship.

Give it a way to get better with use. A general decision model knows nothing about your business, so in a real deployment you want to fine-tune it on your own data. d1’s weights are open,² so this is technically possible, but Liquid’s docs don’t describe how to do it yet.³ The practical route runs through the review step. Send low-confidence requests to people; their verdicts are labelled examples from your own business. Once enough have built up, fine-tune on them and measure the error rate again on the same data. The model fits your use case a little better each round, and each improvement comes with a number.

Small, fast decision models will bring AI to many places that could never afford a large model. The cheaper they get, the more they will be called, and the total number of mistakes grows with them. Before going live, multiply the error rate by the call volume.