Databricks Multi-Statement Transactions Are GA: What Two Customers Actually Saved

On October 16 Databricks made multi-statement transactions generally available.¹ The feature had been in public preview since March.⁷

The docs explain the problem with a transfer: moving 100 from account 1 to account 2 takes three statements, subtract 100 from account 1, add 100 to account 2, write an audit log row.² If each statement commits on its own and the job dies after the first, account 1 has paid, account 2 never received, and the 100 has vanished with no log entry. A multi-statement transaction wraps the three into one unit across Unity Catalog managed Delta or Iceberg tables: all three take effect together, or a failure in any of them undoes all three.¹²

Data pipelines are full of jobs where several steps must succeed together, which is why I rate this feature highly. The announcement, though, gives Fivetran and Accolade one quote each and never says how they worked before or what they saved. Here are both stories filled in: the scenario, before, and after.

Fivetran writes two rows per change, and the risk lived between them

Fivetran’s History Mode keeps every version of a record as a Type 2 slowly changing dimension.¹ _fivetran_start and _fivetran_end bound each version, and _fivetran_active marks the one current version.⁵ When customer 42 changes address, two writes follow: close v1 (set it FALSE, end it a millisecond before the new version starts), then insert v2 as active.⁵

Before. Egidio Terra of Fivetran says guarantees used to cover a single statement, so the writes ran separately, guarded by reconciliation logic and checks on later syncs.¹ The top row of Figure 1 shows what a failure halfway leaves. Close first and fail the insert, and customer 42 has no current row. Insert first and fail the close, and they have two, so per-customer totals double. Reports read the wrong state until the next sync repairs it.

After (bottom row of Figure 1). The UPDATE and the INSERT sit inside one BEGIN ATOMIC … END, so a change lands whole or not at all, and Fivetran deleted a whole class of consistency-repair code.¹ Until commit only the current session sees the changes; after it, the table log holds one entry.²³ The state the reconciliation guarded against can’t happen.

Accolade runs about 1,700 claims transactions a day without a second warehouse

Accolade is a healthcare technology company.⁶ Engineering manager Akshay Ajgaonkar says it runs roughly 1,700 production transactions a day for incoming claims and liabilities, each of which must complete reliably, and that these now run on the governed lakehouse with no separate warehouse.¹ The quote names no tables or earlier system; the customer story says Accolade’s data used to be spread across platforms including Redshift and Snowflake, at real cost.⁶

The scenario (my illustration). One claim touches at least three tables: a claims detail row, the member’s outstanding liability, and a processing log. At month end the claims total must match the liability change.

Before (top row of Figure 2). With each statement committing alone, a claim can land while the liability update fails. Teams either keep the job in a transactional warehouse, paying for another copy of the data and a second set of permissions, or stay on the lakehouse and write compensation jobs. Databricks’ list of warehouse problems (extra copies, fragmented governance, consistency workarounds) prices exactly those options.¹

After (bottom row of Figure 2). All three statements go in one BEGIN ATOMIC. A business check fits too: if a balance goes negative, SIGNAL an error and the claim row rolls back with it.²³ Concurrent batches only conflict when they touch the same member’s row, and Unity Catalog row filters and masks apply to writes.¹

Run the numbers: 1,700 a day is about 620,000 a year. If one split commit in a thousand broke halfway, that is 1.7 half-written claims a day, over 600 a year to find and explain. In a transaction a failed claim rolls back whole and reruns.

The path from BEGIN to END

The first read of a table takes a snapshot that the rest of the transaction reuses,² and changes stay staged until commit.³ Spanning tables works because of catalog commits: Unity Catalog, rather than each table’s own log, coordinates the commit.⁴ Nothing is locked; conflicts on the same rows or schema are checked at commit, and an error, SIGNAL, conflict, dropped session or the 48-hour limit rolls everything back. Interactive BEGIN TRANSACTION … COMMIT detects conflicts per table, so the docs recommend BEGIN ATOMIC for scheduled jobs.²

Author’s note: the savings are repair code, and three things to check first