How Atlassian Cut Security Log Ingestion Costs by 80% at 44 Billion Events a Day

Atlassian’s security team handles 44 billion security events a day. They come from products the company acquired over twenty years, and each product brought its own log format, timestamp style and proprietary schema.¹ On October 15, Databricks and Atlassian engineers published a joint blog post on how they normalized all of it into one open schema on a Databricks lakehouse.¹

The reported results: ingestion cost down 80%, more than 500 production detection signals on a single schema, and analysts 60% more productive.¹ The customer story adds that instantly searchable logs went from 30 days to 12 months.²

It is a textbook case of cutting cost on very large data volumes. The savings come from a chain of decisions that depend on each other: decide what to collect, normalize before the data lands, let AI do the mapping inside a set of gates, and only then tune the queries. Below are those moves in the order the data travels.

Twenty years of acquisitions left logs that would not line up

The pain showed up with the analysts first. They spent much of their time cleaning data after it landed, data sat in different places in different shapes so cross-platform correlation barely worked, and the SIEM (security information and event management) pipelines could not keep up with the volume.¹

The old SIEM had three structural problems. It kept data in proprietary storage for 30 days, so older logs had to be rehydrated from cold storage at a cost in time and money. It was licensed per gigabyte ingested, so the bill grew with the data. And it used a vendor-specific query language, so skills did not carry over.²

A major security incident forced the issue. The investigation needed close to a petabyte of rehydrated data, the old system could not cope, and the team moved to Databricks in the middle of the incident.²

The common schema is OCSF, the Open Cybersecurity Schema Framework. AWS, Cisco, IBM and Splunk were among its backers when it launched in 2022, and it joined the Linux Foundation in November 2024 with more than 900 contributors and 200 participating organizations.⁴ Here is the whole data path.

Decide what to collect by scoring each log on value and cost

At Data + AI Summit, the team described a value-versus-cost rubric for deciding which logs are worth ingesting.³ In the customer story’s words, the rubric keeps the lake full of signal and the noise out.²

That is hard to do well under a per-gigabyte SIEM license, where every new log type is another line on the invoice and the question becomes how much you can afford. Once the data sits in your own storage, the question becomes what is worth keeping.

Some rough arithmetic, with an assumed average of 1 KB per event: 44 billion events a day is about 44 TB. Every 10% you drop at intake is 4.4 TB a day you never store, transform or query, and no later step has to pay for it. The first cut belongs at the front door; query tuning comes last.

Transform before landing: Atlassian chose ETL on purpose

Modern data platforms lean toward ELT: land the raw data, transform it when you need it. Atlassian deliberately went the other way and transforms in flight. The blog gives three reasons: real-time detection, independence from any one vendor’s format, and compression while the data moves.¹ The customer story puts a number on the last one: Parquet files 20% smaller than the JSON strings they replaced.²

ELT buys flexibility, but its downside is exactly where Atlassian started: every analyst cleaning the same data after it lands. ETL moves that work to the entrance and does it once. The price is that mappings must exist up front, and unmapped fields are easy to lose. The Go library’s automatic handling of unmapped fields¹ is there to close that gap.

My reading is that ETL pays off here because two conditions hold: the target schema is stable and shared, and the volume is large enough that per-analyst cleaning gets absurdly expensive. Atlassian meets both.

Forty thousand fields in Go code, so errors fail at compile time

OCSF has roughly 40,000 fields.¹ The team picked it over ECS and UDM because it was the only fully open option, with strong typing, validation, compression efficiency, extensibility and Linux Foundation backing.¹

The decisive step was embedding the whole schema in Go structs, which the team calls TypeGo. Invalid records fail at compile time, nested objects are supported natively, and the code is unit tested.¹ Alongside it sits an open-source Go OCSF library with faster validation.¹ ACID transactions in the storage layer then let schema versions migrate without breaking downstream consumers.¹

The schema’s size looked daunting at first and turned into an asset, because the meaning travels with the data: a field means the same thing in every source.¹ For data engineers, this is a data contract at its strictest. The contract lives in code, and data that breaks it does not compile.

Where the 80% came from depends on which source you read

The 80% is the number everyone repeats, yet the three sources attribute it differently:

• The blog ties it to standardizing security telemetry on OCSF.¹
• The Summit abstract ties it to moving to a file notification loader.³
• The customer story calls it an 80% cut in operational ingest costs and quotes Niels Heijmans, Atlassian’s chief security architect, on “ingestion overhead”.²

These do not contradict each other, but they are not the same claim. Putting the earlier sections together, there are at least four possible sources of savings: no more per-gigabyte SIEM licensing, logs dropped by the intake rubric, compression in flight, and a different loading method. As I understand file notification loading, storage events tell the loader that new files have arrived, so it never has to keep listing directories, and the savings grow with volume.

None of the three sources gives a breakdown. Before you use the 80% to convince your own leadership, find the matching line in your own bill: licensing, storage or compute.

AI writes the mappings, and a person checks every stage

Mapping started as a six-step manual procedure: collect the logs, transform them to TypeGo, route, test, configure and ship through a pull request. The blog describes the move from “mapping as a service” to a self-service platform.¹ Figure 2 shows the current flow.¹

The strict schema helps again here. The agent cannot produce invalid output, so it invents fewer mappings and burns fewer tokens on failed attempts.¹

The pace changed dramatically. Zachary Schmerber, the engineer behind the work, says it took him a year to map about a third of today’s mapped data and two months to map the other two-thirds.¹ The blog places that quote in its section on automation but does not credit the speed-up to automation alone. Per month, the difference is roughly twelvefold.

In my post on judge models I argued that before a model makes judgments in a heavily audited setting, you should ask three questions: can uncertain cases be routed elsewhere, can a verdict be replayed, and what does a wrong verdict cost to roll back. Atlassian’s flow answers all three. Human confirmation sits at the front, regression tests replay every change, and the canary and soak period keep rollback cheap by catching problems before production.

One shared field turns rule-by-rule alerts into one alert per user

Detection gets the most direct benefit. One rule used to raise one alert. Now several signals cluster into a single alert on standardized OCSF values, and the team’s Alert Explorer tool groups findings that way: the user field is the same field whether it appears in an authentication event or a network event.¹

Enrichment, which adds user and asset context to an alert, gets simpler too. Instead of each detection doing its own, one enrichment API is called on one standardized username.¹ Cross-domain correlation, user behavior analytics and spotting anomalies across many routers or vulnerability scanners at once all became practical.¹ Detection code moved from proprietary SIEM queries to PySpark, with a shared Python library of reusable functions.² The blog credits the 60% productivity gain to one repeatable process across every tool.¹

Query tuning comes last, and each speed-up comes from a different query

There are plenty of performance numbers, all from different queries:

• Partitioning and Z-ORDER took a “representative” query from about 45 seconds to 2–3 seconds, and Graviton clusters with Photon cut query cost and runtime by about two-thirds.¹
• The customer story reports Photon cutting a query from 17 seconds to 5, and more than 21 billion events queried in under a minute.²
• The Summit abstract says scheduled jobs and day-long searches went from minutes to seconds.³

These numbers cannot be added up or used to confirm each other; they only show the direction. Liquid Clustering is next on the list to evaluate.¹ Access runs through Unity Catalog, because the lake holds sensitive material such as insider threat investigations.²

Tuning comes last for a simple reason. The intake rubric and the shared schema decide what gets stored and in what shape; query tuning can only speed up what is left. Do it in the other order and you are making the noise faster.

Four things to copy, and three to discount

Worth copying:

• An intake rubric: score every log type on value and cost, and drop what is not worth collecting.
• A typed schema at the entrance: it does not have to be Go; JSON Schema or Protobuf can also stop bad data at the door.
• A gate chain for AI mappings: human confirmation, local tests, canary, soak and regression tests, with none skipped.
• Enrichment and grouping on shared fields: write the enrichment once and let every detection use it.

Worth discounting:

• The material is co-written by Databricks and Atlassian, and none of the numbers are independently verified.
• The performance figures come from different queries, and the three sources attribute the 80% differently.
• Not every company can match Atlassian’s engineering investment: its own Go library and mapping IDE, and an OCSF schema maintainer on the team.¹

If you build audit or compliance log platforms, this case deserves a close read. Audit logs look a lot like security logs: many sources, high volume, and every record has to be traceable. A shared schema, validation at the entrance, and a test and review record for every mapping change carry over directly.