I spent the early years of my career in the automotive industry, where safety is the most critical design parameter and you are always thinking: what if it fails at highway speed? Just one system working well is not enough; a failure in one can cascade into others and immediately become fatal. It also means that when I claim the most instructive thing is happening in self-driving cars, you should account for my bias. For years, we called self-driving the hardest problem in AI, until the robotics community gently informed us that the actual final boss is folding laundry. But until someone ships a fleet of a million towel-folding robots reporting telemetry, the driving people still have the best data story in the business, and that story is worth stealing.
When the industry decided that self-driving was the biggest problem to solve, it faced the same question we at DeepTempo did: where is the data? When Tesla wants Autopilot to get better at a scenario, they do not send an engineer to go find examples of it. The fleet does that. Every car runs the model in shadow mode, and every time the shadow model's prediction diverges from what the human driver actually did, that moment gets flagged, shipped back, matched against similar cases, annotated, and folded into the next training run. Karpathy called it the data engine, and it is widely considered the hardest part of Tesla's autonomy stack to replicate.
Waymo built the complementary version: they leaned toward simulation. Real autonomous miles are scarce relative to everything that can go wrong on a road, so a critic model flags suboptimal driving from the fleet, and those edge cases get replayed and varied inside a closed-loop simulator, generating orders of magnitude more training scenarios than the physical world could ever supply. For the situations too dangerous to wait for, they stage them deliberately at closed test facilities.
Two very different companies arrived at the same conclusion, and by now it is how the most serious teams in the industry are approaching the data problem for domain-specific models: the scenario space is effectively unbounded, so you cannot get there by having humans manually investigate every failure. Manual triage does not just cost money; it caps how fast the model can improve, because every retraining cycle waits on a queue of unexamined cases. The only way out is to automate the relationship between outcome and data. When the model gets something wrong, the system itself should be able to answer: what did the model see, what was missing or distorted in the data, and what would it have needed to get it right. If an attack slipped past a detection model, "the model missed it" is not the answer. the pipeline must be able to answer why it happened.
At DeepTempo, we built that flywheel for cybersecurity. LogLM, our foundation model for threat detection, always snatches the attention for its purpose-built encoder-only design. But a model is a function of its architecture and its data, and the data is the half nobody hands you. We built both from scratch. The model architecture is inspired by the transformer but modified to handle the nuances of cyber logs, so we ended up creating our own model library rather than just pulling one from HuggingFace. That customization was necessary because of our data modality; in most cases, if you share the same modality, you will be fine using the tokenizers and dataloaders designed for LLMs. The challenge arises when you shift left. You have to start thinking from first principles: what does a sequence look like, what does a token look like, and there isn't a package to inspire you. So you go back to the whiteboard. I can talk for hours about the model, but this blog is about the second half: the data flywheel that keeps the model honest and fed with new information.
The uncomfortable truth about security telemetry
Security telemetry in the current enterprise is deeply fragmented. What arrives at a detection layer is not one signal but dozens: flow records, DNS queries, TLS handshakes, HTTP transactions, authentication events, and the list goes on. Each is a different lens on the same network, each owned by a different tool with a different format and a different idea of time, and attacks do not respect the boundaries between any of them. A serious detection layer has to be designed with the full surface in mind. We designed for all of it from the beginning, and we started with NetFlow.
We started there deliberately. You cannot install an agent on everything, but everything touches the network, and flow records are the cheapest complete picture of that fact: they cover devices no endpoint tool will ever see, keep working when the payload is encrypted, and cannot be wiped by an attacker who owns the endpoint, because they live on infrastructure the attacker never touched. Above all, flow captures the one behavior no attacker can skip. Tooling changes and malware changes, but reconnaissance, lateral movement, and exfiltration are communication by definition. A zero-day still has to phone home. And flow is, by far, the harshest proving ground for the data problem, so a pipeline that survives NetFlow has earned the right to expand.
None of these logs is a neutral observation. Each is the output of some system that made choices, and the choices leave marks. NetFlow in the wild is not a dataset; it is a negotiation between a dozen vendors with their own opinions about sampling, timeouts, and much more. Some exporters split a single long-lived connection at arbitrary timeout boundaries, so one SSH session can look like fifteen unrelated flows. And the distortions only compound as you move up the stack: a recursive resolver collapses thousands of clients into one DNS query, TLS 1.3 is deleting the handshake fields defenders spent a decade learning to fingerprint, and whether a Kerberos event exists at all depends on an audit policy someone set years ago, so identical attacker behavior is visible in one enterprise and absent from another.
Attackers understand this better than most defenders. They tune scans to stay under sampling floors, pace beacons to blend into keep alive traffic, and tunnel command and control through DNS because flow records make it look like name resolution. These distortions are not just noise to be cleaned; they are terrain the adversary navigates on purpose.
Train on this raw material without correcting for it, and the model learns the quirks of the collectors instead of the behavior of the network. It will be very confident and very wrong. In all the our experimentation, this has one of the most important factor for models stability in production.
The flywheel
So we made a deliberate decision: stop treating data preparation as a preprocessing step and start treating it as the product. It is the single best engineering decision we have made; I only wish we had made it sooner. The result is a loop with five stages, and the whole point is that it is a loop: every stage feeds the next, and the last stage feeds the first.
One principle runs through every stage, and it is among the most important aspects of building a flywheel like this: understand how the data is generated at the system level, and replicate that generation process inside the flywheel. We established above that no log is a neutral observation; a flow record is the output of a specific exporter with its own sampling policy, timeout behavior, and position in the topology. The consequence is strict: if the data we manufacture inside it does not carry the same fingerprints as the data arriving from production, the model will learn to tell the two apart, and everything downstream of that is quietly corrupted. The flywheel is not just a loop for collecting data. It is a loop for faithfully reproducing the system that generates it.
1. Discover. New data enters from three directions. Open-source datasets, which are useful but overfished; everyone trains on the same handful of academic captures, which is one reason so many detection models generalize poorly. Honeypots and network sensors we operate ourselves, which intercept real traffic in the wild, everything from routine background noise to opportunistic scanning to genuine intrusion attempts. This is pretraining fuel: an encoder learning the structure of behavior needs volume and realism far more than it needs labels, and there is no substitute for traffic that nobody curated. And production deployments, where the traffic of real enterprise networks (financial institutions, telecom operators, universities) shapes what we go looking for next. This is opt-in, never a requirement of deploying with us.
2. Process. Every log, regardless of source, goes through our proprietary normalization pipeline, our EDA pipeline, and more than 50 custom-designed quality checks that go beyond typical ML data validation, before the model ever sees it. This is where two hard problems meet: the fragmentation of the systems producing the data, and what a model does with that fragmentation if it is allowed through. A pretrained encoder has no labels downstream to catch bad data; whatever gets in, artifact or behavior, gets absorbed into the representation. So we are deliberately selective. The goal was never to ingest everything; it is to ingest quality and diversity, because a million near-identical flows teach the model less than a thousand genuinely distinct behaviors. The checks enforce quality: fidelity to production traffic, no artifact left to become a shortcut, no duplication turning training into memorization. The continuously running EDA layer enforces diversity and catches drift before the model discovers it the hard way.
3. Simulate. Real attack traffic is rare by design; attackers do not publish datasets. So we manufacture it, in two complementary ways: one for breadth, one for depth.
Breadth comes from our attack traffic synthesis pipeline: attack behaviors defined as DSLs, run through Monte Carlo generation and other proven techniques so no two instances look alike. These are derived from the live threat landscape, pulled continuously from open sources like MITRE Caldera and other intel feeds. One canonical exfiltration teaches the model that specific transfer; ten thousand statistically varied ones teach it what data leaving a network looks like. Crucially, synthetic traffic passes through the same generation conditions as real traffic, sampling, timeouts, exporter quirks and all.
Depth comes from the cyber range, where we execute complete attack scenarios and capture the traffic end to end. Deterministic attacks, scripted playbooks of known tradecraft, give us reproducibility. Agentic attacks, where AI agents like Stanford's ARTEMIS are handed an objective and left to plan and adapt, give us what no script can: the improvisation of a live adversary. Playbooks encode what we already know; agents surprise us on purpose. Between them, the range produces what synthesis alone cannot: full kill chains, reconnaissance through lateral movement to impact, wrapped in the benign noise of a living network.
4. Evaluate. Evaluation is the most important part of the flywheel, because it decides what all the other parts were worth. Nothing gets promoted on intuition, and nothing gets evaluated only once. Every candidate model runs against internal benchmarks that combine held-out open-source datasets, synthesized attacks, and captures from the range, none of which the model has seen in training. The aggregate score is the least interesting number we produce. Aggregate metrics hide failures: a model can post a better overall detection rate while quietly regressing on one attack family or one exporter type. So evaluation is sliced, per behavior, per technique, per data source, and a challenger replaces the champion only when its improvement survives statistical guards built to separate real gains from benchmark noise: confidence intervals, not point estimates.
The slices that fail are the flywheel's steering wheel. Every weakness becomes a targeting order for stage three, synthesize more of this, run range scenarios against that, and the hardest cases get promoted into both the training set and the benchmark, so the eval suite grows harder as the model grows stronger. Since releasing Vigil, our open-source AI SOC, at RSA this year, evaluation also extends past offline metrics: candidate findings run through Vigil's triage and investigation workflows, scoring LogLM the way a SOC actually consumes it. Evaluation is the brake on the flywheel, and the brake is what lets you go fast: trusted evals let us retrain aggressively, because regressions get caught before they reach a customer.
5. Feed back. Deployments close the loop. What the model flags, what gets confirmed or rejected, and where the traffic distribution has drifted all flow back into stage one. Vigil sharpened this stage considerably. Where feedback used to arrive as unstructured analyst judgment, a finding that passes through Vigil comes back as a structured verdict: severity from triage, disposition from investigation, a MITRE technique mapping, a record of what the responder did about it. That is the difference between "an analyst dismissed this" and a labeled training example that explains itself. Because Vigil is open source, the adjudication layer is not a black box; every team that runs Vigil runs the same machinery we run internally. As feedback flows into training, we ensure the model does not trade what it knew for what it just learned: updates are designed so there is no catastrophic forgetting, and the model accumulates rather than swaps.
Then the wheel turns again. This flywheel is what enables our model training cadence, and the cadence is the point. A flywheel that turns once is just a wheel.
Our fleet, our simulator, our test track
Back to the autonomous driving parallel, because the mapping is nearly one to one: production deployments and honeypots are our fleet, the synthesis pipeline is our simulator, and the cyber range is our closed test track, staging what we cannot wait for the real world to hand us on a schedule.
There is one place the analogy breaks, and it is what makes our version harder. Tesla's data source is cooperative. Drivers want to drive well; every mile generates honest signal. Our subject is an adversary who actively works to not generate data, who blends into benign traffic on purpose, and who changes behavior when detected. Autonomous driving mines edge cases from a willing world. We have to manufacture ours from an unwilling one. That is why simulation and range exercises are not optional accelerants the way they are in theirs; they are load-bearing.
What building the flywheel taught us
Data quality failures are silent. They do not crash and they do not spike your loss, which is exactly what makes them dangerous. The only defense we have found is instrumenting the pipeline as seriously as the model, and I now consider a data pipeline without instrumentation to be broken by definition.
Synthetic data is a scalpel, not a firehose. Early on we were tempted to drown the model in generated attacks. Narrow, deliberately varied synthesis is what actually helped. Generate against your weaknesses, not your convenience.
Evals are the highest-leverage artifact you will build. Our benchmarks outlived three major model versions. Architectures came and went; the evals told us whether each change was real. If I could keep only one artifact from this entire effort, it would be the evaluation suite, not the model weights. we released a part of it in open-source benchmark - SOCBench (socbench.org)
The loop beats the snapshot. Any individual stage of our pipeline could be copied. The compounding is what cannot: every deployment sharpens where we place our sensors, which improves the training data, which improves the model, which earns more deployment. Accumulated turns of the flywheel are not something a competitor can shortcut with a bigger training run, and I sleep better because of it.
The real moat
The security industry has a long habit of shipping detection built on stale academic datasets, or on sprawling ensembles of narrow supervised models each trained to spot one labeled attack, and then being surprised when they fall over on the traffic that never appeared in the labels. That is not what LogLM is. It is a single foundation model that learns the behavior of a network the way a language model learns language, from the structure of the data itself rather than from a catalog of known-bad signatures. But a foundation model only earns that name if what it learns from is right, and the lesson we keep relearning is that a model can only be as good as its data. Architecture is a design decision you make once and refine. The data engine is an operating discipline you run continuously, forever.
We chose to make the unglamorous part a core competency alongside the model itself. This is why one purpose-built foundation model, not a fleet of brittle classifiers, holds its own in production networks: it generalizes to behavior it was never explicitly shown, because it learned what normal and abnormal look like rather than memorizing a list. And the flywheel is the part of our stack that scales past NetFlow: every new telemetry type we take on rides the same loop, because the loop was never about flows. It was about knowing, for every outcome the model produces, exactly what data led there.
In automotive, we were never allowed to ship a system whose failures we could not trace back to a cause. I could not unlearn that habit, so we built our detection layer the same way. And there is one more reason the driving story was the right one to steal. As Vigil earns more autonomy, turn by turn, triaging, investigating, and adjudicating with less human steering, the flywheel approaches the thing we borrowed the metaphor from: a pipeline that drives itself, with humans setting the destination and holding the brake. I cannot wait to get there.
