You probably think you understand AI security. Most people do — until they realize we’re not just fighting hackers anymore. We’re entering an era where machines attack machines, where algorithms hunt algorithms, and where the very AI systems we built to protect us are becoming weapons against each other.
I used to think cybersecurity was about stopping human attackers. Then I discovered what’s actually happening in the shadows of our AI infrastructure, and it changed everything I thought I knew about digital warfare.
The Machine vs. Machine Reality
While security teams obsess over traditional threats, a parallel war is already underway. Weaponizing AI is proving to be a potent catalyst driving new, more complex cybersecurity threats, reshaping the cybersecurity landscape for years to come. From rogue attackers to sophisticated advanced persistent threat (APT) and nation-state attack teams, weaponizing large language models (LLMs) is the new tradecraft of choice.
This isn’t science fiction. MIT Principal Research Scientist Una-May O’Reilly develops artificial agents that reveal AI models’ security weaknesses by mimicking threat actors. They can process cyber knowledge, plan attack steps, and come to informed decisions within a campaign. What MIT is doing defensively, attackers are already doing maliciously.
The evidence is mounting from authoritative sources. Cybercriminals are inevitably adopting Artificial Intelligence (AI) techniques to evade the cyberspace and cause greater damage without being noticed. Researchers in cybersecurity domain have not researched the concept behind AI-powered cyberattacks enough to understand the level of sophistication this type of attack possesses.
Why Traditional Defenses Are Failing
Here’s the uncomfortable truth: our current security paradigm was built for human attackers. Current defenses are largely reactive — each new attack typically requires identification, human response, and design intervention to prevent it. They are inadequate to address the ever increasing scale, severity and adaptive strategies of malicious parties.
The problem runs deeper than most security leaders realize. The core capabilities of human beings are AI’s blind spots; “humanness” is simply not yet (or possibly ever) replicable by artificial intelligence. We have yet to build an effective security tool that can operate without human intervention but we are close. On the other hand attackers don’t need human-like AI — they need AI that can outmaneuver our defenses faster than we can respond.
The Attack Surface Explosion
AI-to-AI attacks aren’t just theoretical. AI-powered cyberattacks leverage AI or machine learning (ML) algorithms and techniques to automate, accelerate, or enhance various phases of a cyberattack. This includes identifying vulnerabilities, deploying campaigns along identified attack vectors, advancing attack paths, establishing backdoors within systems, exfiltrating or tampering with data, and interfering with system operations.
The scope is staggering. The attacks use many tactics, such as evasion, poisoning, model replication, and exploiting conventional software vulnerabilities. They include various malicious actors, from regular users to skilled red teams, who focus on attacking machine learning models in environments such as cloud-hosted, on-premises, and edge installations.
What makes this particularly dangerous is the identity crisis in our systems. Up to 85% of identity-related breaches are caused by hacking of machine identities, according to ReliaQuest, and Gartner’s 2024 IAM Leadership Survey found that 54% of organizations saw an increase in overall identity breaches. Machine identities — service accounts, API keys, AI agents — are becoming the primary attack vectors, and most organizations can’t even see them, let alone secure them.
The Research Arms Race
The academic community is racing to understand this threat. Adversarial machine learning is an active research area. A quick Google Scholar search reveals nearly 10,000 papers published on this topic in 2024 alone (as of the end of May). The arms race continues as new attacks and defense methods are proposed.
Government agencies are taking notice. The Artificial Intelligence Security Incident Response Team (AISIRT) will analyze and respond to threats and security incidents emerging from advances in AI and machine learning (ML). The team will also lead research efforts in incident analysis and response and vulnerability mitigation involving AI and ML systems. When Carnegie Mellon creates a dedicated AI security incident response team, you know the threat is real.
The Department of Defense is particularly concerned. Our Secure AI Lab is working to make machine learning as secure as possible for the DoD and Intelligence Community. We organize our work into a find-fix-verify paradigm, where we find machine learning vulnerabilities by developing new adversarial attacks, fix vulnerabilities by developing defenses and mitigations to known attacks, and verify, within a given system, that vulnerabilities have been properly mitigated via adversarially focused test and evaluation.
The Evolution of Attack Sophistication
What’s happening goes far beyond traditional hacking. This research project will develop a novel machine learning (ML) approach using a coevolutionary algorithm that is integrated with an Artificial Intelligence planner. The resulting system will be applied to a use case in the form of a cyber game wherein it will assume the roles of two automated game players that compete against each other. Researchers are literally building AI systems that learn to attack other AI systems through evolutionary competition.
The sophistication is accelerating. AI’s ability to learn from data and continuously evolve makes it an invaluable tool in developing more resilient and scalable cybersecurity solutions by addressing challenges such as insider threats that pose significant risks from within the organization, making them difficult to detect using conventional methods. But this same capability makes AI attacks adaptive and persistent in ways human attackers never could be.
The Policy and Industry Response
The security industry is struggling to keep pace. Mixed results with AI implementations are pushing security leaders to focus on narrower use cases with more measurable impacts. Organizations are realizing that broad AI security approaches aren’t working — they need targeted defenses for specific attack vectors.
Georgetown University’s Center for Security and Emerging Technology is studying the policy implications. In July 2022, the Center for Security and Emerging Technology (CSET) at Georgetown University and the Program on Geopolitics, Technology, and Governance at the Stanford Cyber Policy Center convened a workshop of experts to examine the relationship between vulnerabilities in artificial intelligence systems and more traditional types of software vulnerabilities. The consensus: AI vulnerabilities require fundamentally different approaches than traditional cybersecurity.
What This Means for Organizations
The implications are sobering. Other security risks are tied to vulnerabilities within models themselves, rather than social engineering. Adversarial machine learning and data poisoning, where inputs and training data are intentionally designed to mislead or corrupt models, can damage AI systems themselves. Your AI systems aren’t just tools — they’re targets.
The research trajectory is clear. The concept of adversarial machine learning has been around for a long time, but the term has only recently come into use. With the explosive growth of ML and artificial intelligence (AI), adversarial tactics, techniques, and procedures have generated a lot of interest and have grown significantly. What was once academic research is becoming operational reality.
The Path Forward
We need a new security paradigm. Our vision is autonomous cyber defenses that anticipate and take measures against counter attacks. Traditional reactive security isn’t enough when attacks happen at machine speed.
The future of cybersecurity isn’t about better firewalls or smarter analysts — it’s about building AI systems that can defend against other AI systems. The question isn’t whether this machine-vs-machine warfare will happen. It’s already happening. The question is whether we’ll be ready for it.
Organizations that don’t adapt their security strategies for AI-to-AI attack vectors aren’t just falling behind — they’re leaving themselves defenseless in a war they don’t even know they’re fighting.
The hidden war between machines has begun. The only question left is which side will win.
