Writing

Writing

New models and tools ship every day. I ask one question: will it hold up in production? Notes on data, AI and traceability, for the people who have to run them.

  1. Front view of a white phoropter, two mirrored banks of lens dials, each opening ringed by scales and metal knobs, against a light grey background

    AI × DataDeep dive10 min

    Make the analytics agent pass an exam first: inside Data Culture's evaluation framework for dbt

    Data Culture took an e-commerce analytics agent from 0% to 94.7% accuracy without changing the model, by adding context one layer at a time and grading it on a fixed exam. Using their open-source agent-trust-for-dbt repository, I draw the architecture as five layers, from the dbt foundation and the contract the agent reads to a grading cascade that ends in an LLM judge, and sort what you can copy as-is from what you should change.

  2. Over-the-shoulder view of a small aircraft cockpit, two pilots with four-stripe and three-stripe gold epaulettes, one hand on the throttle lever between them, a row of round gauges and two screens ahead

    AI ToolsShort take4 min

    One change through an Entire Trail: where agents step in, where people decide

    Entire, the company former GitHub CEO Thomas Dohmke started, launched a Git forge for people and agents on October 26. It swaps the pull request for a Trail, where review agents rescore every push and gates, CI and people decide whether a change merges. I walked one change through it using the official docs and mapped who does what at each step.

  3. A pegboard wall seen head-on, filled with hundreds of sewing thread cones, roughly grouped by colour into blocks of orange and red, blue and white, beige and brown

    Data EngineeringShort take3 min

    Liquid Clustering "wins three of four workloads", but only three were measured

    An InfoQ benchmark compares partitioning, ZOrder and Liquid Clustering on 50 million Delta Lake rows. I chart latency and data read for the three measured query types, point out that the fourth, a full-scan aggregation, repeats the second one's numbers, and lay out each layout's strengths and weak spots in one table.

  4. Top-down view of a desk where a hand holds a loupe over colour film negatives on a lit light table, with rows of negatives in clear sleeves hanging above and an orange desk lamp on the left

    AI × DataShort take3 min

    Before you give an agent memory, price the raw-history baseline

    Two October papers pit elaborate agent-memory systems against a plain baseline that keeps raw conversations and retrieves with reranking. The baseline wins six of seven settings and ingests in a fraction of the time. Build it first, measure write cost, accuracy and lost evidence, then choose a memory architecture.

  5. A hand holds a 100-euro banknote; on its right edge are the watermark area and a shiny security hologram, over a blurred grey-brown pavement

    GovernanceShort take3 min

    Claude's text now carries a watermark: what the EU requires, and what a watermark can prove

    To meet the EU AI Act, Anthropic has watermarked the text output of new Claude models since August. I explain what happened and how the watermark works, unpack what Article 50 asks of providers and publishers, and lay out what a watermark can and cannot answer.

  6. Black-and-white night shot of a pier, where a hooded street artist, back to the camera, draws a caricature on an easel with a lit Ferris wheel behind

    AI × DataShort take4 min

    Will an agent that writes summaries replace the dashboard? Brilliant Earth is halfway there

    Jeweler Brilliant Earth moved its analytics into Snowflake, took 100+ store users off a legacy BI tool and has an LLM write weekly reports from numbers its code computes. I use the case to answer one question. If visualization is the last step of the data pipeline, will an agent that summarizes take its place?

  7. A hermit crab carrying a white shell walks across a sandy beach, orange antennae reaching forward, the beige sand behind it blurred

    Data EngineeringShort take5 min

    500 GB in one weekend: Shipsidekick's move from Neon to ClickHouse Postgres, before and after

    Logistics SaaS Shipsidekick moved its Postgres from Neon to ClickHouse Managed Postgres and cut over in a weekend. I chart the core numbers before and after and the migration steps, and separate what came from architecture, what came from a bigger machine, and the bill the case study leaves out.

  8. Two old books stacked on each other, a pale purple tassel bookmark tucked between the pages of the top one and hanging over the spine

    Data EngineeringShort take4 min

    dbt State is free forever: 1,000 DATT a month is worth $94, and the rest is on your warehouse bill

    dbt Labs made dbt State free forever for self-hosted users, at 1,000 DATT a month. I explain how it decides to skip, clone or rebuild, how to use it in prod, staging and dev, and what the free allowance is actually worth.

  9. A trail signpost on a mountain top at dusk, its white boards pointing left and right with distances and walking times to each destination, ridges fading into the distance

    AI × DataShort take4 min

    Decision models have reached the warehouse. Copy the way they ask

    On October 19 Snowflake announced Decision Fast, a low-latency decision model in private preview that returns classifications, scores and routes with probabilities from a single SQL call. I draw its design in three steps and look at where a model like this saves the most money and time inside a warehouse.

  10. A red race car stopped in its pit box while five or six crew members in black suits and helmets change its tyres, with removed wheels on the ground

    AI × DataDeep dive8 min

    Letting an agent fix failed pipelines: a step-by-step build of Snowflake's 16-minute repair loop

    Snowflake's data team uses agents to repair failed data pipelines, averaging under 16 minutes from detection to a pull request waiting for review. I turn the setup into a tutorial you can follow, covering what you need first, how to build it, what it buys you, how to roll back and at what cost, and what the original post leaves out.

  11. Three rowers in one boat lean forward together, their four differently coloured oar blades resting on the water at the same moment

    Data EngineeringShort take5 min

    Databricks Multi-Statement Transactions Are GA: What Two Customers Actually Saved

    Databricks made multi-statement, multi-table transactions generally available on October 16, but gives each customer story a single quote. Here are the Fivetran and Accolade cases spelled out (the old two-step writes, what a failure halfway left behind, what one transaction changes) plus the path from BEGIN to END.

  12. Inside a logistics sorting center, several silver roller conveyor lines curve and merge in the foreground, with a wall of racking and stacked cardboard boxes behind

    Data EngineeringDeep dive8 min

    How Atlassian Cut Security Log Ingestion Costs by 80% at 44 Billion Events a Day

    Atlassian normalized two decades of messy security logs into OCSF on a Databricks lakehouse and reports an 80% cut in ingestion cost; here are the seven moves behind it, in the order the data travels, and why the 80% is attributed three different ways.

  13. A puddle on wet asphalt reflects an orange tower crane and a few buildings upside down, next to yellow road markings

    Data EngineeringShort take4 min

    Snowflake's Postgres Mirroring Is GA: How It Stacks Up Against Databricks Lakebase

    Snowflake Postgres data mirroring went GA on October 15; next to Databricks Lakebase it has the stronger consistency story but makes you choose between fresh and transactionally whole, while Databricks already syncs both ways with the Postgres-to-Delta leg still in preview.

  14. A railway construction site with new tracks laid on gravel ballast, a half-built concrete overpass and a crane in the distance, and no train on the line yet

    IndustryShort take5 min

    Lay the track before you buy the bullet train

    Snowflake's manufacturing survey says 41% of manufacturers are stuck in pilots and half still run on spreadsheets. EU, US and Chinese figures measure different things but point the same way. AI is the train, the data foundation is the track, and most companies haven't laid it yet.

  15. A volleyball line judge holding an orange-red flag at the edge of the court, players blurred behind

    AI × DataShort take5 min

    Three questions before a judge model replaces your filter script

    Databricks now lets a small classifier, OpenJev, act as the judge in Unity Gateway service policies. I like decision models taking over filter scripts, but in audited work you need answers on confidence, replay and rollback first.

  16. A wall of wooden card-catalogue drawers, one pulled open to show a stack of index cards

    GovernanceShort take3 min

    The GDPR is strict, but individuals still can't see whether their data is protected

    An IAPP column asks whether predictability and coherence can make up for the EU's missing regulatory convergence. My view is that the rules can be strict, but only regulators and companies can check them. Individuals get a statement.

  17. A turnstile in a dim station, its side lit by a red sign with a green arrow

    GovernanceDeep dive7 min

    How agents get fenced in when the answer must be 100% right

    Financial regulatory reporting demands zero errors, and an AICon talk lays out a full set of controls for agents. The logic holds, but it buys accuracy with constraints, and non-standard data created upstream is still there.

  18. A refinery at blue hour: dense pipes, towers and tanks lit up against the night sky

    AI × DataShort take5 min

    General models can write the pipeline. They can't read the bill.

    Josh Wills says agents write data pipelines that are correct and wildly expensive. I read that as a missing feedback problem, and it points to what data engineering is waiting for, a specialist agent.

  19. A large sum, 1% times 1,000,000, with the result 10,000 below it in blue

    AI ToolsShort take4 min

    d1 decides in 8 ms. Who catches the times it's wrong?

    Liquid AI open-sourced d1, a small model that answers without writing text. I want models like this, but before production I'd multiply its error rate by its call volume.