We gave three frontier reasoning models, GPT, Claude, and Gemini, four expert personas, a large context budget, investigative tools, and a generous per-investigation budget. We gave our purpose-built encoder-only model, LogLM, nothing but the raw flows. Then we asked all of them the same question over the same network traffic: what here is malicious?
Before I dive into the what, why, and how of the setup, one thing to flag up front: this blog is part of our open-source initiative, SOCBench. The goal of SOCBench is to bring the community together to create datasets and methodologies that evaluate the work AI is being asked to do in cybersecurity operations. Check out the repo. PRs and stars are welcome.
At DeepTempo, we build a vertical foundation model (LogLM) for cybersecurity, a purpose-built encoder-only model that understands logs related to cybersecurity operations. While designing it, we cared about cost, efficacy, latency, and reliability, the factors that decide whether anyone can actually run detection in production. With Mythos and GPT-cyber, frontier models are showing off their offensive cyber capabilities and there are benchmarks like CyBench that evaluates them of CTF type of tasks, but there is no open benchmark that exhaustively evaluates on them all things research and production viability. Hence, SOCBench: an open benchmark for AI in cybersecurity operations.
Detections are core of how a Security Operation Center (SOC) operates. So, we took that as first task in the Benchmark. We took these frontier models and LogLM out for a spin to check their capability of doing detection on the raw NetFlow data. This post covers findings from our evaluation on diverse dataset from Stratosphere Labs and some other sources. This evaluation, and the ones to follow, focus on four axes that matter in production: cost, efficacy, latency, and reliability. Key findings from this round:
- LLMs can tell when an evaluation unit has something suspicious. Verdict F1 on the 1,205 shared eval units lands between 0.86 and 0.93 (best persona per provider). LogLM hits 0.954 F1 and 0.888 MCC on the same evaluation set.
- On clean traffic, whether it sits inside a malware capture or in a fully benign one, the LLMs flag 36 to 86percent as malicious. LogLM stays under 2 percent in both settings. For real SOC operations, lower false positives means less analyst fatigue, and this gap is what decides whether a system can go inline at all.
- LLMs struggle to explain why a unit was bad. Per-flow F1 is 27 to 42 percentage points lower than verdict F1 in every provider's best persona. They detect, but they cannot localize.
- Per-alert cost averages $0.057 to $0.150 across the three LLM providers (single persona). LogLM is under $0.0001. At telco scale (about 70M flow events per minute, batched into roughly 2 billion alerts per day at 50 flows per alert), the LLM bill runs to hundreds of millions of dollars per day.
- All evaluations are zero-shot.
While you are excited to read more of the how this evaluation was conducted, I will request you to consider checking out the code and star our GitHub Repo SOCBench - I promise I will not ask again. So lets get into it.
Why SOCBench?
Every vendor pitch I read in security right now leads with “AI in the SOC” or “AI-powered cyber.” Almost none of them publish how they evaluated the system, or how their AI does on the telemetry a defender actually sees. AI models and systems are black boxes, and the only way of finding their true capabilities is to run domain-specific evaluations on them. SOCBench is a call to action for the community to build evaluation methodologies and datasets so that we know whether the systems we are deploying can actually be useful for cyber operations. RSAC this year was instructive: I talked with multiple SOC vendors and asked one question, how is your product different from others? No one had a satisfactory answer. During the same time, we also launched our open-source project to provide answer to some of those question and more questions from the community who is forced to deal with black box approach to AI in cyber, do check out the repo: VIGIL-SOC. Vigil has already hit 200 stars, and lead us to building SOCBench in open - one is system powered by agents, another is keeping it honest.
Detection on NetFlow is Hard and Noisy
NetFlow is often considered hard as it is very noisy and correlation becomes challenging because of the huge volume of logs generated. As vast majority of traffic is now encrypted, NetFlow bypasses this limitation because it doesn't care about the content as it only looks at the traffic patterns, allowing defenders to still flag malicious behavior even if the payload is completely hidden. NetFlow can be enabled across all modern routers, switches, and firewalls providing visibility into scenarios where perimeter defenses are completely blind.
Detection on NetFlow is challenging as signal of compromise is rarely in single flow. It lives in structure of the traffic, in distributions and rhythms across thousands of connections, against a background of benign activity that looks superficially identical. Reasoning through the individual flows or reasoning through some of them manually is challenging and first task becomes to recognize what exactly to investigate. This falls under direct strength of modern days LLMs powered by Agents. They are extraordinary at tasks that decompose into narratable steps. You can debug a program you have never seen because each broken assumption produces a trace you can read. You can chain a multi-step exploit because each stage telegraphs the next. There is a thread. NetFlow detection has no thread. You can look at one flow and learn almost nothing. You can look at a hundred and start to see a shape. You can look at ten thousand, and the distribution itself becomes the signal. That is what a purpose-built model learns.
As we build a foundation model to learn these behaviors, I also wanted to keep us honest. SOCBench started as keeping us honest exercise and eventually we decided that community can benefit from it. Hence, we made is open-source and expect the community to come together to keep everyone talking about AI in Cyber - Honest.
The Face-off between LLMs vs LogLM
Agents Powered by LLMs. As a starting point, we started with three pinned models (GPT-5.4, claude-opus-4–7, gemini-2.5-pro) running as multi-turn agents inside a bounded loop. They got read-only investigative tools (list_pairs, get_pair_timelines, get_flows, host_rollups, port/protocol matrix, rarity_stats). Each LLM is provided with Four Expert personas with their own loop budgets and tool allowlists, large context budgets, structured-output enforcement, a cost cap per investigation, and a ground-truth firewall that strips label fields from every tool response. Each persona takes in playbooks and characteristics of how they will actually behave in real-world scenario e.g., SOC analyst gets less time than Detection Engineer to work through a task.
LogLM. Single-shot. No tools. No personas. No multi-turn loop. We hand it the raw flows; it returns predictions. The model is an encoder. It looks at the whole picture at once and decides whether the interaction looks like a compromise.
The asymmetry is on purpose. I wanted the LLMs to fail, if they were going to fail, in the most flattering possible setup. Allow them every engineered advantage - tools, personas and keep them honest on time and cost. If the result still came out the way it did, then the result is not about scaffolding. It is about capturing the nuances of the domain, in this case, cybersecurity, and specifically NetFlow and flow-like logs. In all cases, models are considered to be running inline with the logs.
Operational Metrics are as important as Efficacy
As LLMs and AI based systems move from research phase to production deployment, they must be measured on four axes: cost, efficacy, latency and reliability. In this benchmark, and all that will be released in future, will cover all these aspects. While measuring cost, latency and reliability is obvious - well we can debate and work on better approach together if you don't agree - but I needed to put efficacy measurement grounded in real-world needs of cyber. Efficacy measurement is broken down into three aspects:
Verdict. Can the model tell that a given evaluation unit (a conversation between two IPs, or a host fanning out to many destinations) is malicious? This is the bare minimum. If you fail this, nothing else matters.
False Positives. Cyber attacks are rare and most of traffic in day to day operation is clean. When you point the model at this clean traffic, does it stay quiet, or does it creates serious noise in regular work? A model that emits false positives 50 percent of the time on clean traffic is not deployable, no matter how well it does on actual malware.
Explainability. Once the model raises an alert, can it tell the analyst which flows drove the call? In a benchmark where most malicious units are nearly pure malware, this looks like a localization test. In production, where units are mixed (a host running normal traffic and command-and-control on the same connection at the same time), it is the difference between an alert that narrows the analyst’s scope and one that hands back the entire haystack. Per-flow efficacy is the closest single number for this.
How an evaluation unit actually looks like?
Before we get into the numbers, I want to show you a real evaluation unit - I was calling it events - but a thoughtful commenter on my LinkedIn post highlighted that "event" is interpreted differently by different people in cyber operations. So, I will avoid the word entirely. In SOCBench, the evaluation unit is something different, it is scoring boundary. One unit produces one verdict. One verdict, in production is one alert.
This is a unit he-9fe954c104be63de from benchmark-v0, a host_egress unit. Source IP 192.168.1.195 fanning out over a five-minute window, ten flows total. Gold label: mixed (Some flows are malicious, some are not). The model never sees that label. It sees these ten rows:
flow dst_ip t_rel proto sport dport b_in b_out
1 185.244.25.235 0.0 TCP 48986 6667 158 91
2 147.231.100.5 39.2 UDP 123 123 90 90
3 185.244.25.235 94.7 TCP 48986 6667 158 91
4 147.231.100.5 103.2 UDP 123 123 90 90
5 147.231.100.5 169.2 UDP 123 123 90 90
6 185.244.25.235 192.9 TCP 48986 6667 158 91
7 147.231.100.5 233.2 UDP 123 123 90 90
8 77.78.107.252 282.2 UDP 123 123 0 90
9 185.244.25.235 284.5 TCP 48986 6667 158 91
10 147.231.100.5 297.2 UDP 123 123 90 90In this case, gold labels says, with labels hidden from LLMs to see, flows 1,3,6 and 9 are labelled C&C. They are the same source talking to the same destination on TCP port 6667 (IRC, the protocol used by a particular Stratosphere malware family’s beacon channel) at regular intervals. The other six flows are NTP traffic on UDP port 123, which is just the host doing what hosts do. A correct verdict is “malicious.” A correct localization is the four-flow short list {1, 3, 6, 9}.
If a model is raising an alert, it shall produce a list of candidate flow leading to that alert. If not done right, the analyst has to triage all ten flows to figure out which of these are bad, on real network where the unit might be a thousand flows instead of ten, this is the entire workload.
The Methodology
The methodology is in the harness; the load bearing pieces are summarized in the diagram.
Overall harness considers seven steps. where possible deterministic, content-addressed and reproducible.
1. Ingest the raw NetFlow capture.
2. Build index: normalize to the canonical schema, assign stable flow_id, derive pair and host rollups, and emit a dataset_hash so anyone re-running the pipeline lands on the same bytes.
3. Construct eval units: pair_timeline and host_egress, both produced by label-agnostic rules, so the unit-typing itself does not leak ground truth.
4. Stratified sample across the six strata (unit type x gold label), seeded.
5. ReAct Agent Loop: each unit goes through four personas, each with persona-scoped read-only tools, in a bounded ReAct loop of REASON, OBSERVE, ACT. Three providers run in parallel: Anthropic, OpenAI, and Gemini. LogLM is single-shot in its own track.
6. Score against hidden ground truth on three F1 lenses (per-flow, per-pair, per-host) plus verdict accuracy, cost, latency, and reliability.
7. Aggregate and compare: per-model and per-persona leaderboard, ablation deltas, cost-vs-quality plot.
The dataset. benchmark-v0, about 757,641 flows, 37.3 percent malicious. Four Stratosphere malware captures and four benign captures, normalized to a canonical NetFlow schema. Per-flow labels. We stratified the sample across six strata (pair_timeline and host_egress crossed with benign, malicious, and mixed), 500 units per stratum, capped at 1,500 total, seeded. The Anthropic run completed 1,205 of those under its $0.50 per-rendering budget; the results below use that 1,205-unit shared slice so all three providers are scored on the exact same units.
The eval units. Two kinds. A pair_timeline is one source/destination conversation, time-ordered. A host_egress is one source IP fanning out to many destinations in a window (assigned when a source reaches ten distinct destinations in any five-minute bucket; the threshold is label-agnostic, so the typing rule does not leak ground truth). Units larger than 1,000 flows are split into contiguous sub-windows.
The ReAct agent loop. For each unit times persona times provider, the harness runs a bounded multi-turn agent loop. Tool calls are read-only. The final answer is a strict JSON schema (verdict, confidence, predicted malicious flow indices, predicted malicious destinations, rationale) declared natively to each provider. If the model produces an invalid structured response, the harness records a failure. We do not silently repair anything.
The four personas differ in the loop budget and the tool allowlist. Goal was to closely simulate the real-world behavior of these personas e.g., how much time they have to respond to an alert. The Persona X tool matrix is fixed across providers, so the same agent runs in the same shape against every model:
Persona turns calls wall Tools beyond base
soc_analyst 4 6 60s base only
threat_analyst 8 12 120s + top_destinations, pair_stats
adversary_hunter 10 16 150s + port_proto_matrix, rarity_stats
detection_engineer 12 20 180s same as adversary_hunter
Base tools (every persona):
list_pairs, get_pair_timeline, get_flows, host_rollup, submit_assessmentThe submit_assessment call is what closes the loop and emits the structured verdict.
The clamp. A model only gets credit for `flow_ids` it actually saw in a tool response during the investigation. No credit for guessed IDs. This is the one asymmetry: the clamp applies to the LLM agents but not to single-shot LogLM, which sees the whole unit. I disclose it instead of hiding it.
The metrics. Two layers of measurement.
At the unit level, verdict: binary, malicious or benign. We report verdict accuracy and verdict F1 against gold. This is the unit-level number a CISO reads, and the one the results section leads with.
Inside the unit, three F1 lenses for explainability, each computed per unit and then macro-averaged across units:
- per-flow: predicted-malicious set vs gold, flow by flow.
- per-pair: a
(src_ip, dst_ip)is positive if any of its flows is. - per-host: same, keyed by source IP. The meaningful lens for
host_egressunits.
We report per_flow_f1_macro: macro per-flow F1 across all 1,205 units. We also compute per_flow_f1_macro_malicious which restricts the macro to malicious and mixed units; that variant strips out benign units (where empty-prediction-on-empty-gold scores a free 1.0) and is published in the harness for reference. We also report a composite, effective_per_flow_f1, which is per-flow F1 times the first-pass-valid rate. Accuracy times reliability in one number.
Cost. Per call: prompt, output, reasoning, and cached tokens, priced against a pinned pricing.yaml snapshot. We cap each rendering at $0.50 to keep the experiment bounded.
Results
LogLM ran against the exact same flow set the LLMs were scored on: 354,923 flows across the 1,205 SOCBench eval units. LogLM's verdict and per-flow predictions are at sequence grain; the LLM agents emit one verdict per unit and one set of predicted-malicious flow IDs per unit. To compare cleanly, LogLM's measured Recall and TNR are projected onto the SOCBench eval-unit class prior (62 percent positive, 38 percent benign).
Best Persona Per Provider
Configuration Recall Precision F1 MCC
----------------------------- ------ --------- ----- -----
LogLM 92.2% 98.7% 95.4% 88.8%
Claude Opus 4.7 / soc_analyst 99.2% 86.8% 92.6% 84.0%
Gemini 2.5 Pro / soc_analyst 96.6% 80.5% 87.8% 67.9%
GPT-5.4 / threat_analyst 80.0% 93.6% 86.3% 69.3%LLMs pooled across all 4 personas (no persona tuning):
Configuration Recall Precision F1 MCC
-------------------------------- ------ --------- ----- -----
LogLM 92.2% 98.7% 95.4% 88.8%
Claude Opus 4.7 (4-persona pool) 99.2% 78.7% 87.8% 69.3%
Gemini 2.5 Pro (4-persona pool) 96.4% 74.8% 84.2% 55.4%
GPT-5.4 (4-persona pool) 87.7% 73.4% 79.9% 39.9%Per-flow F1 (malicious-only, macro across captures):
Configuration Per-flow F1
----------------------------- -----------
LogLM 98.9%
Claude Opus 4.7 / soc_analyst 65.3%
Gemini 2.5 Pro / soc_analyst 52.2%
GPT-5.4 / threat_analyst 44.4%Four things to notice.
1. The LLMs win Recall because they alarm on everything. The cost of that recall shows up in Precision. LogLM is the only model above 95 percent precision; the best LLM persona (GPT-5.4 / threat_analyst) reaches 93.6 percent, but its Recall drops to 80 percent.
2. On F1 and MCC, the prior-aware summaries, LogLM wins by 3 to 21 points against the best LLM persona, and by 8 to 49 points against the pooled LLM (the realistic "deployed without persona tuning" cell).
3. The per-flow F1 gap is the explainability story. LogLM at 98.9 percent means when it flags a unit, it points at the exact malicious flows. The best LLM persona at 65.3 percent means even the strongest agent gets only two-thirds of the localization right; the next best is at 52.2; the worst at 44.4.
4. Without persona tuning the LLMs lose another 5 to 15 points on every metric. LogLM has no persona to tune; 92.2 / 98.7 / 95.4 / 88.8 holds as-is.
Verdict
Best persona per provider on the 1,205 shared eval units lands between 0.86 and 0.93 verdict F1. Anthropic's soc_analyst hits 0.926, Gemini's soc_analyst 0.878, OpenAI's threat_analyst 0.863. LogLM verdict F1 at sequence grain is 0.959 (4 malware captures pooled), or 0.954 when projected onto the SOCBench eval-unit class prior.
Verdict F1 alone hides a lot. Below is the full per-provider per-persona table on the combined split (all 1,205 units, positive class = {malicious, mixed}). Acc, Prec, Rec, and F1 are the standard binary metrics. TNR (true negative rate) is how often the agent stays quiet on benign units. MCC (Matthews correlation coefficient) is the imbalance-robust quality score, bounded in [-1, +1], where 0 is random and 1 is perfect.
Provider Persona TP FP TN FN Acc Prec Rec F1 TNR MCC
--------------- ------------------ --- --- --- --- ----- ----- ----- ----- ----- -----
Claude Opus 4.7 soc_analyst 501 76 375 4 0.916 0.868 0.992 0.926 0.831 0.840
Claude Opus 4.7 threat_analyst 648 207 248 11 0.804 0.758 0.983 0.856 0.545 0.615
Claude Opus 4.7 adversary_hunter 698 176 280 3 0.845 0.799 0.996 0.886 0.614 0.693
Claude Opus 4.7 detection_engineer 635 212 241 1 0.804 0.750 0.998 0.856 0.532 0.629
Gemini 2.5 Pro soc_analyst 626 152 296 22 0.841 0.805 0.966 0.878 0.661 0.679
Gemini 2.5 Pro threat_analyst 604 321 127 31 0.675 0.653 0.951 0.774 0.283 0.327
Gemini 2.5 Pro adversary_hunter 710 213 236 16 0.805 0.769 0.978 0.861 0.526 0.596
Gemini 2.5 Pro detection_engineer 689 200 253 28 0.805 0.775 0.961 0.858 0.558 0.592
GPT-5.4 soc_analyst 677 266 182 54 0.729 0.718 0.926 0.809 0.406 0.403
GPT-5.4 threat_analyst 589 40 415 147 0.843 0.936 0.800 0.863 0.912 0.693
GPT-5.4 adversary_hunter 632 289 161 113 0.664 0.686 0.848 0.759 0.358 0.238
GPT-5.4 detection_engineer 689 341 117 49 0.674 0.669 0.934 0.779 0.255 0.266Two things to notice. Recall is uniformly high (0.80 to 1.00); the LLMs rarely miss a malicious unit. The precision/TNR axis is where the configurations sort, and that is what drops MCC. Claude soc_analyst is the only persona above 0.80 MCC. GPT threat_analyst is the precision outlier: 94 percent precision, 91 percent TNR, but the lowest recall in the table, the only configuration that meaningfully says no to malicious. Gemini threat_analyst is the precision floor (65 percent) and sits near the bottom on every other axis.
Caveats are on the headline number itself. Verdict is one bit per unit: bad or not. That is enough to raise an alert. It is not enough to act on one. And on a benchmark this skewed toward malicious traffic, F1 of the positive class flatters the model in a way MCC and TNR do not.
Are they quiet on benign traffic? - False Positives
We ran each system against two kinds of clean traffic and measured how often it raised a verdict of malicious. The first column is the benign sequences that sit inside the four malware captures (a network that does have malware on it, but the specific flow being scored is not part of the attack). The second column is fully clean captures with no malicious traffic at all: a university Linux box, a home Linux box, an HTTPS website, an xDSL Linux client, plus the Stratosphere normal-* series
Provider FP on benign mixed with malware Verdict FP on benign-only traffic
Anthropic 36% 39%
OpenAI 53% 43%
Gemini 41% 86%
LogLM <1% <2%In production, almost everything is benign. At 36 percent FP, the model is alerting on more than a third of clean traffic. At 86 percent, on nearly every conversation a normal machine has with the internet. None of these are deployable, and the gap to LogLM here is the one that decides whether the system can ride a live network.
A partial mechanism: an autoregressive reasoner trained on a large corpus about security threats has read about all the ways a flow might be malicious. The prior that any given flow is suspicious, in the model's head, is higher than the prior in the actual traffic distribution. The bias does not surface in a benchmark designed around accuracy on malicious units. It surfaces the moment the model is asked to stay quiet.
Explainability
Best persona per provider, intra-model gap between verdict F1 and per-flow F1 (overall macro across all 1,205 units):
Provider / persona Verdict F1 Per-flow F1 Drop
Anthropic / soc_analyst 0.926 0.653 27 pts
Gemini / soc_analyst 0.878 0.522 36 pts
OpenAI / threat_analyst 0.863 0.444 42 pts
LogLM 0.954 0.989 noneThe same LLM that gets the verdict right cannot, with the same confidence, point at the flows that drove the call. The minimum intra-model gap is 27 points; the largest is 42.
Against LogLM, the gap is wider. On the four malware captures pooled, LogLM hits per-flow F1 of 98.9 percent; the best LLM persona (Claude / soc_analyst) reaches 65.3 percent on the same task. The next best LLM is at 52.2; the worst at 44.4.
Operationally, on the 10-flow sample from earlier, the gap is the difference between predicting {1, 3, 6, 9} and predicting "look at all 10." On a 1,000-flow unit, it is the difference between a focused investigation and a full re-triage. An alert that cannot localize does not save the analyst any time.
Where the LLMs catch up
Units where almost every flow is malicious. There, the optimal strategy is "predict everything malicious," which is degenerate but scores well on per-flow F1. Any verdict-correct system does well in that regime. The harder regime is mixed units, where a host runs C&C and NTP at the same time. That is where the per-flow gap reopens, and that is the regime that dominates real network traffic.
Cost, Latency, Reliability
Per-alert cost. In production you run one persona, not four. Numbers below are the average cost of a single persona on one eval unit (total provider spend divided by 1,205 units and again by 4 personas):
Model $/alert
----------------------------- -------
Anthropic / Claude Opus 4.7 $0.150
Gemini 2.5 Pro $0.062
OpenAI / GPT-5.4 $0.057
LogLM <$0.0001Projected to volume (assuming 50 flows per alert on average):
Volume Anthropic Gemini OpenAI
1M alerts/day $150k/day $62k/day $57k/day
~70M flow events/min (~2B alerts/day at telco scale) ~$302M/day ~$125M/day ~$115M/dayAt one million alerts a day, the cheapest LLM stack is roughly $60,000 of inference, every day, before a single analyst looks at the output. At telco scale (about 70M flow events per minute, batched into roughly 2 billion alerts per day at 50 flows per alert), the LLM bill runs to hundreds of millions of dollars per day.
Latency. Mean wall-clock per unit: Anthropic 37.5s, Gemini 30.2s, OpenAI 18.7s. LogLM is single-shot encoder inference, dominated by I/O. Inline detection with an LLM agent does not keep up. Out-of-band, you accept tens of seconds of detection lag per unit.
Reliability. First-pass-valid rate: OpenAI 98.8 percent, Gemini 93.9 percent, Anthropic 89.7 percent. Roughly 1 in 10 Anthropic investigations had to retry. The composite effective_per_flow_f1 discounts F1 by this rate, and the gap to LogLM widens further.
Does the scaffolding help?
The harness supports two ablations: tools_off (only submit_assessment available) and playbooks_off (persona playbook removed). The full sweep against the 1,205 shared units is next on the list. The prediction this post commits to is that the deltas will be small. If they are not, the prediction is wrong, and the post will be updated. The commands are socbench run --ablation tools_off and socbench aggregate.
Why Purpose-built wins the job?
LogLM is an encoder. It learns the distribution of benign network traffic and recognizes when traffic deviates from it. The strong recall and the near-zero benign FP come from the same capability: knowing what normal looks like, well enough to see when it is broken.
An autoregressive agent reasons from local cues, one tool response at a time. Each flow in isolation looks superficially suspicious. A connection to an unfamiliar IP on a non-standard port could be malicious, and the model has read about plenty of cases where it was. So the model flags it. It flags the actually-malicious flow and most flows that look like the actually-malicious flow. The base architecture is not wrong to find these things suspicious. It is wrong to make them load-bearing. The two failure modes compound. High benign FP is the prior pulling the model toward "suspicious" on neutral evidence. Low per-flow F1 is the inability to discriminate between flows once the unit has been flagged. A perfect-benign-quiet LLM would still leave the explainability gap. A perfect-localization LLM would still drown the analyst in false alarms. Deployable detection needs both, and they come from the same kind of model.
This is not a claim that frontier reasoning models are bad at security work. Enumeration, exploit chaining, detection-rule synthesis from prose, triage explanation, narrative generation: those decompose into narratable steps and they are autoregressive-friendly. Flow-level detection is not.
Limitations
One dataset, four malware families, and four different benign environments. Benchmark needs to be extended to much diverse environments and more complex attacks.
What to do with this?
Detection engineer or threat researcher: Clone the harness, plug in your API keys, swap in your dataset. Repo: SOCBENCH [https://github.com/DeepTempo/socbench]
CISO. Three questions for any vendor selling you AI in the SOC. (1) What is the per-alert inference cost at your traffic volume? (2) What is the false-positive rate on your benign traffic, not theirs? (3) When the model flags a unit malicious, can it tell your analyst which specific flows drove the call? If any answer is unclear, the system is not ready.
Model builder. The interesting frontier is not "make the LLM agent better at flow-level detection." It is "let the LLM do what it is good at, on top of a recognition layer that has already done the discrimination." Detection should be encoder work. Investigation, triage, escalation, rule synthesis, narrative: those are the seats where an LLM agent earns its keep.
SOCBench is detection-only for now. Triage, hunting, IR, detection engineering, threat intel are on the way. If you want to shape what gets measured, the repo is open. We are building our open-source AI SOC based on findings of the benchmark and more benchmarks that we run internally - do checkout the repo: VIGIL-SOC
