Can open-source LLMs actually outperform frontier models in analyzing cybersecurity network telemetry? We extended the SOCBench framework to see if we could balance detection accuracy while maintaining data sovereignty and cut costs through the implementation and application of open-source LLMs (Sec-8, Seneca, GLM-5.2). We found that open-source models rival frontier-level classification for specific tasks while being 12 times cheaper. Despite this, LogLM’s value and accuracy outperform both open-source and frontier models.
What first interested me about SOCBench was the application of stress-testing existing general and fine-tuned LLMs on raw telemetry generated in cybersecurity operations. Generally, first intuition is to handover the telemetry directly to the LLMs and let them do the classification directly. What differs with SOCBench is the addition of an agentic harness in the benchmark which drastically improves results that raw models alone can derive. code is available here: https://github.com/DeepTempo/socbench/tree/opensource
So, I set out to extend the SOCBench framework to the open-source world. I wanted to find the sweet spot, where we could maintain high detection accuracy while slashing costs and keeping data strictly in house. That's one of the core design principles of our open-source AI SOC - VIGIL.
Going into this project, I knew that frontier models, while potentially demonstrating high accuracy, would struggle to scale to the level that NetFlow would require. As such, I decided to look at this project with the following concepts in mind:
The Cost/Accuracy Differential. Frontier models, while accurate, are extremely expensive. Local models, while regarded as less accurate out of the box, possess significant advantages of being cheaper over massive volumes due to being run on local hardware while remaining relatively lightweight in nature. I wondered if the value proposition could increase- such as the F1 score remaining roughly the same as compared with frontier models, while reducing cost dramatically. The first benchmark runs covering frontier models demonstrate this, which you can find more details about here.
Domain specialization vs. Raw scale. Does knowledge dramatically increase the effectiveness of models? We employ agents with subtasks to specifically address this issue, but does finetuned domain knowledge make a difference, allowing the local LLMs to punch above their weight class in comparison to sheer parameter count?
Data Sovereignty. In cybersecurity, shipping NetFlow transcripts and internal enterprise telemetry to third-party hosted APIs introduces massive compliance, privacy and sovereignty liabilities, not to mention export lag.
As such, I began looking into a variety of open-source LLMs, and settled on Foundation-Sec-8b (Abbreviated as Sec-8) and Seneca-Cybersecurity-LLM-x-QwQ-32B-Q4_Medium-Version (Abbreviated as Seneca), both quantized and domain-specific fine-tuned versions of existing local models: Llama3.1-8b and Qwen2.5-32B, respectively, while also focusing on a recently-released frontier open-source model, GLM-5.2.
While I expected this to be a simple project, adapting the open source harness proved difficult. In the case of Claude, Gemini and OpenAI, the adapter built to run for these benchmarks was extremely simple, where the vendor handles tool-call structure, error handling, and reasoning versus output separation on the server side. For open-source models, I had to build all of this myself.
Process
My first few runs of sec8 were complete messes, yielding zeros across the board. The model wasn’t failing to find threats, but instead, it refused to wrap its final answers in the required submit_assessment tool call. Instead, it would narrate bare JSON into the transcript:
{
"verdict": "malicious",
"rationale": "Suspicious traffic detected on port 443...",
"malicious_destinations": ["10.0.0.5"]
}Because the harness expected a function call, these turns were treated as plain text prose until the turn budget was exhausted, invalidating each testing unit invalid.
As a result, I realized that I needed to develop my own parser, which extracted the distinct JSON objects by their internal signature and extracted keys such as verdict and rationale, and map them to the benchmark’s schema.
This resulted in the following results (after a 40-unit initial test):
Metric Before Parser Fix After Parser Fix
-------------------------------------------------------------------
Text-only turns 326 137
Schema-valid submissions 2/40 36/40
Forced final turns 38/40 0/40
-------------------------------------------------------------------Note: Turns refers to the number of agentic reasoning steps. This was entirely a parser fix, not a prompt fix. The model’s raw behavior remained unaltered, instead the harness stopped throwing away scorable answers that already existed inside the LLM transcript. Forced final turns are turns forced by the harness if the model was not submitting out of its own volition.
With the reasoning parser active, I compared the open-source models through the full benchmark- around 1500 units across 4 personas (6000 total renderings). This was relatively trivial for the extrapolation across the other open source models. The results were the following:
Metric Foundation-Sec-8B Seneca-QwQ-32B GLM 5.2
---------------------------------------------------------------------------------------------------
Coverage
units scored 4,349 4,064 474 (22% partial)
first-pass valid 0.725 0.677 0.366
Verdict (malicious vs benign)
precision 0.656 0.671 0.326
recall 0.882 0.798 0.936
F1 0.752 0.729 0.484
accuracy 0.641 0.712 0.603
cov-adj verdict F1 0.650 0.507 0.190
cov-adj recall 0.648 0.430 0.134
TP / FP / FN / TN 2370/1244/316/419 1573/773/398/1320 88/182/6/198
Flow-level
per-flow F1 0.373 0.459 0.551
native-lens F1 0.376 0.538 0.578
effective per-flow f1 0.270 0.308 0.203
Cost
effective cost for 1,500 units $76.85 $137.48 ~$1,326
normalized cost for 1 row $0.05 $0.09 $0.88
---------------------------------------------------------------------------------------------------Note: We evaluated normalized f1 values, excluding adapter failures as valid results in comparison to the frontier models. Approximate dollar value for benchmarks that were not run to 6000 units.
Persona Sec8 (F1) Seneca (F1) GLM-5.2
--------------------------------------------------------------------------------
Adversary Hunter 0.357 0.417 0.609
Detection Engineer 0.366 0.416 0.544
SOC Analyst 0.377 0.511 0.711
Threat Analyst 0.391 0.495 0.340
--------------------------------------------------------------------------------Key Takeaways:
For each of the local models, I analyzed the transcripts to explore what happened and how one might improve them. It is important to note that the same harness was applied to each one, and a work in progress, and thus the models may not have performed as well as they could have, given a more well developed harness.
Analysis
Sec8: Llama-3.1-8B fine-tuned by Cisco
At an 8B parameter footprint, it achieves a Verdict F1 of 0.752, landing close to frontier heavyweights like GPT-5.4 (0.797). It possesses an impressive security “instinct,” boasting a high verdict recall of 0.882, meaning it rarely misses a compromised host.
Despite this, the moment that the task demands localization, Sec8’s performance degrades, with its per-flow F1 dropping to 0.373. In exploring the benchmark transcript, I found that Sec8 commonly hallucinates threats.
When its tool calls were scope-clamped against actual network traffic, roughly 99% of its flagged malicious destinations were entirely fabricated. It confidently cross-referenced ports, IPs, and beaconing behaviors that never existed in the input transcripts.
Seneca: Qwen-2.5-32B finetuned on cyber data
In contrast, Seneca achieves a per-flow F1 of 0.459, easily out-localizing closed frontier systems like Gemini 2.5 Pro (0.399) and GPT-5.4 (0.308). The 32B model leverages its parameter capacity to systematically drill into network pairs and accurately separate malicious traffic from background noise.
Seneca fails through its first-pass validity rate sitting extremely low at 0.677, with a 30.4% failure rate. This run was run on a 4xL4 configuration, which led to considerable instability. This could likely run better if run on a better GPU configuration.
GLM-5.2: 753B parameter open-source model by Z.ai
In my opinion, GLM-5.2 is the most interesting. GLM-5.2 has the most parameters, yet least amount of domain knowledge. Even in comparison to frontier models, it was able to achieve a per-flow F1 score of 0.551, exceeding all frontier and local models. Despite this, its overall F1 score was 0.484 and first-pass validity rate was 0.366. Its recall also rivals frontier models at 0.936.
In evaluating the results, I found that it simply called too many tools. By being a “chatty” reasoning model, similar to OpenAI’s GPT-5.4 (see blog), I gave it a 1.5x tool call multiplier, which it excessively used.
On large host_egress units (which make up 73% of the benchmark), it over-suspected every single destination, and was force-terminated by the harness on 84% of large units, completely erasing its highly accurate analysis.
Persona Opus budget GLM budget Opus turns GLM turns Opus forced GLM forced
(1.0x) (1.5x) used used
---------------------------------------------------------------------------------------------------
SOC Analyst 4 6 4.1 6.6 29% 85%
Threat Analyst 8 12 6.2 12.4 3% 88%
Adversary Hunter 10 15 6.8 15.3 2% 88%
Detection Engineer 12 18 7.8 17.6 3% 74%
---------------------------------------------------------------------------------------------------What's Next?
With the popularity of AI-harnesses, it is extremely important to integrate and refine the evaluation and agent harness in SOCBench. I am confident that not only can we improve the output, but also reduce token cost in the process. It would also be interesting to test the local LLMs in perfectly identical computing environments, isolating this testing feature which evidently resulted in a lot of costly errors from this run.
I think it’s interesting to note that the open-source LLMs did much better than I expected, and in some cases rivaled the frontier models themselves. Ultimately, while open-source models proved to save nearly 92% as compared to the frontier models, when compared directly to a purpose-built network detection model such as LogLM, the cost and accuracy difference simply does not compare. Considering LogLM’s impressive 0.954 F1 score, and costs less than $0.0001 per row, LogLM proves to be dramatically more cost effective and accurate relative to every single model we have benchmarked.
Some disclaimers: while open-source LLMs do make a lot of sense in specific scenarios, running them on cloud infrastructure proved expensive, yet not quite as expensive as the frontier LLMs. It is to be noted that these models are niche and not hosted on any specific OpenRouter supported infrastructure, and that the per token cost for their base models is extremely cheap relative to the costs in this benchmark, due to api endpoints being highly optimized for throughput as opposed to the GPU allocation method we utilized.
