Picture for me, if you will, a world where security threats are proactively detected, where cybersecurity teams no longer play an endless game of catchup with attackers. This is in stark contrast to today’s reality, where organizations are perpetually racing to patch vulnerabilities and respond to breaches after they occur. For many, this proactive future may seem out of reach.
It turns out the key to the problem may come from an unlikely place. Logs are something that every enterprise stores. Many logs simply expire and disappear with time, but many are required to be stored either due to practical use or regulations. One such type is the lowly flow log. In 1996, CISCO released a new kind of log called a netflow log. This log would track the data entering and exiting network interfaces. It contained useful information such as source and destination IP and the total size of the data sent. Due to the nature of the data, these logs can get very long and be filled with traffic that is hard to find any real value from. For this reason, many cybersecurity tools ignore or only minimally interact with netflow. There are not nearly as many rule sets out there for netflow as there are for some other log types. Netflow is the perfect example of something pretty much everyone has but almost no one uses.
The thing is, netflow can reveal some very interesting things about a network. With a long enough context window and adequate knowledge of the network, you can start to understand different types of events. You can even start to see what normal behavior looks like for not just a specific network but most networks. So why is that? Well, the answer is actually pretty simple. Netflow is the perfect replay of the behavior of the network, not the contents of the network. You can see things from a different perspective. Instead of getting caught up in packet headers and other details, you simply see the flow of network traffic as it is. To a human, this is hard to absorb. To an AI with a large context window, this is actually a goldmine of information.
So how would we start to build this out? First, we need to build a new kind of AI. We can’t simply fine-tune GPT2.5 to read these logs. We need something more specialized if it is going to be truly effective. We need to use a LogLM. A LogLM is a model that is trained on specific logs to understand them and recognize normal patterns. It will identify abnormal patterns by simply trying to predict what it should expect to happen next, and if the actual outcome is different enough, it will mark the event or actually the sequence of events as anomalous.
Think about it in the context of the old haystack analogy. If you have a haystack and you need to find a needle, you would have to sort through each and every piece of the haystack to find what you are looking for. If, however, you could find a way to just ignore every piece of the haystack that was hay, all you would see would be the needles. In this way, we can metaphorically burn the haystack down and see what is left. This is exactly what a LogLM does. It operates as if nothing is happening out of the ordinary until something out of the ordinary occurs. All it has to do is keep track of event sequences, and when they complete, if nothing was wrong, it can just let them pass and forget about them. But when it sees a new event that it doesn’t recognize, well, that is a different story. To be clear, it doesn’t know if the event is an attack. It isn’t a rule set that checks every event. It just knows that something isn’t right. This allows it to label the sequence of events as an anomaly and then feed it to the next step for identification.
Think for a moment about what I just said. The model didn’t need to understand what the existing attack was. This could have been an old attack that everyone in the world had rules written for, or it could have been something totally new. There is no difference to the model. All it needed to know was that the sequence of events was out of place on the network it was monitoring. Suddenly, we are talking about proactive security. This is something that isn’t playing catchup to the attackers. This is something the attacker has to work to defeat. Is it foolproof? No, nothing is, but does it make it exponentially harder for an attacker to succeed in their endeavors? Absolutely. This is the opposite of security theatre. This is proactive detection, and the possibilities are endless.
Up to this point in my writing, I have focused on netflow, but let’s take a look for a second at other log types. What if we apply these same concepts to VPC or defender logs? Well, quite simply, we’ve basically created a system that can detect anomalies across all aspects of an organization’s infrastructure — from network traffic to cloud resources to endpoint behavior — without needing to be explicitly programmed for each new type of threat. LogLMs are in their early days, but the initial promise they show is exceptional. F1 scores that have been out of reach traditionally are well within the capabilities of foundation LogLMs like the one we have been working on at DeepTempo. LogLMs are the future of cybersecurity, and while there will most likely never be a single solution, they will be a critical component of cybersecurity systems moving forward.
If you want to play with our LogLM give it a try Here.
