Last week I wrote about the move from a SNAFU loop to an OODA loop. Some of the reaction to that blog was that perhaps it could be useful to expand upon the DevOps and SRE side of this necessary transition. The following blog required a little more work finding patterns and articles in the attic of my mind than anticipated. Please note that this blog is not intended as an indictment of current or past cyber, SRE, or DevOps approaches, rather, it simply shares some lessons learned from a similar transition. Now that the attackers have transitioned to a flavor of machine-speed intelligence - we must as well. I’m confident that lessons learned from the transition to SREs and DevOps can be helpful. I hope you agree.
To set the stage, roughly fifteen years ago, operations went through a phase transition. Cloud was here, mind- boggling in its scale and capabilities, and mobile too was a force for change. The old ways of carefully running systems, with their locked-down change windows and more waterfall-like artifacts, just didn’t work. The result was a culture and technical transition that changed nearly everything. Read the DevOps Handbook, the Phoenix Project, the Google SRE handbook, and Accelerate as a starting point if you’d like to learn more. Or just ask your favorite AI about all of the above and/or look for the writings of John Willis who is an amazing guide to thinking through these approaches, and a scholar on the still under-appreciated work of Edward Deming.
Here is the crux of the situation. As cloud and mobile hit, we saw that the systems being operated had outgrown the humans operating them: too many services, too many machines, too many alerts for any team of people to manage by hand, however skilled and caffeinated. Google's answer was Site Reliability Engineering, sometimes described as what happens when you ask software engineers to design and own an operations function. We also called this transition DevOps. Companies such as Netflix demonstrated what the transition looked like in practice, building event-driven automation platforms (in their case Winston, built atop StackStorm, which I cofounded) that let engineers hand their repeatable diagnostic and remediation work to software.
Cybersecurity now faces the same transition, with a few important differences. Most notably today, our pressure comes from adversaries. In the DevOps world, our problems came from accidental failures across utterly enormous scale, resulting in states that are impossible to forecast. We were not facing an adversary that probed for weaknesses, adapted to countermeasures, or accelerated when defenders slowed down. As discussed in my last blog on the SNAFU to OODA loop transition, AI-enabled attackers do all of the above.
The Zero Day Clock, to pick one source, shows the mean interval between disclosure and exploitation falling from roughly a year in 2021 to negative two hours as of this blog. Mandiant's M-Trends 2026 puts mean time to exploit at negative seven days; in any case, exploitation now routinely precedes the patch. Defense that depends on humans reading alert queues, or time-travel, to see tomorrow’s attacks today so they can write or adapt their detections, are impractical because they cannot operate within today’s intervals.
Here are a handful of lessons from the transition to SRE and high degrees of automation:
1. The pager is a design defect
Google's SRE organization enforced a rule that still sounds radical: engineers may spend no more than half their time on operational work. Anything beyond that threshold is called toil, and toil is treated as a defect to be engineered away rather than a cost to be staffed. Netflix made much the same point. In our SRE meetups when talking about Winston / StackStorm, their engineers showed a photograph of a physical runbook: a binder seven to ten inches thick. Imagine searching it at 2 a.m.
As Netflix and others pointed out, waking a human to execute a documented procedure is a failure of system design. Winston focused StackStorm explicitly to serve as Tier-1 support, so that engineers could hand repeatable diagnostic and remediation tasks to software that runs in response to events.
Incidentally, last week I spoke to the head of SRE at a top 5 security vendor whose cloud service spans approximately 120 data centers plus many on-prem and secure environments; he reports they use StackStorm to automate approximately 83% of all operational incidents. Side note - yes, that made my day.
By comparison, generally speaking, a SOC today is that ten-pound run-book binder, rendered as processes, hard-to-use dashboards, and limited automation. Tier-1 triage, the copying of indicators between consoles, the many-tab investigation of an alert that turns out to be a misconfigured scanner: all of it is toil in the SRE sense, repetitive, automatable, error-prone, and increasing at least linearly with alert volume – and more often geometrically with new attack surfaces of exposed interfaces and automated threat and vulnerability exploits. The lesson is that human judgment is the scarcest resource in defense, and every hour spent on procedures that software could execute is an hour subtracted from the work only humans can do.
2. The operator was redefined, and the operators became more important
Perhaps the most consequential element of the SRE transition was how people scaled themselves to address the challenge and opportunity. Google staffed SRE with software engineers and bound them to the 50% rule; this way, operating a system and improving it became the same job. The sysadmin who executed procedures becomes an engineer who builds the systems that execute procedures, still carries the pager, works the queue, and converts incidents into design input to boost resiliency. The title change to SRE reflected changes in capabilities: fluency in a console mattered less; fluency in software, statistics, and systems thinking mattered more. Those who made the transition to become SREs found their work elevated and their compensation as well.
Security operations teams are starting to undergo the same redefinition.
As software and AI agents absorb triage and investigation, the analyst's value should migrate up the stack: from executing playbooks to authoring the skills that unlock the agent to perform them; from disposing of alerts to evaluating the machine judgment that disposes of them; from console fluency to a working command of detection engineering, adversary behavior, and the statistical literacy needed to read an SLO or a LogLM’s precision curve. Machine speed operators become supervisors of the system: they learn when to trust the system, when to intervene, and how to turn each intervention into an approach to improve the system.
I personally like the metaphor of a Montessori teacher; they mostly design environments where learning happens rather than teaching in the traditional sense. Organizations that allow for this transition will retain their best people through it. Those that treat the SOC as a cost center to be automated away will lose the judgment their machine-speed intelligence needs to grow and learn.
3. Quantified risk replaced the pretense of perfection
As briefly mentioned above, one new aspect of SRE thinking was the error budget. As the SRE handbook from Google puts it, and as everyone sort of realized around the same time, 100% reliability is typically the wrong target: users cannot distinguish 99.999% from 100%, and the marginal cost of all those nines is massive. So we shifted to defining explicit service-level objectives (SLOs), and started defining the gap between the objective and perfection as a budget, and teams then started to spend that budget deliberately on velocity and change. The error budget avoided the structural conflict between development and operations by giving both sides a shared, quantitative mechanism for deciding how much risk to take.
Security still mostly operates on assumptions and mindsets the SRE movement avoids. In security, we speak of preventing all breaches while measuring almost nothing about detection itself. The transition to machine speed defense requires the security equivalent of the SLO: explicit objectives for detection coverage, false positive rates, and above all dwell time, the interval between compromise and confident detection. To be clear, many security practitioners already abide by the mantra that the goals of incident detection are that they become recoverable, shallow and loud or easy to see. Once those goals exist, we can craft an operational budget against them, an SLO for detection. Call it a Detection Level Objective (DLO). With a DLO in hand, machine speed intelligence platforms like DeepTempo’s Intelligent Defense Platform have something concrete to be measured against. A defense that cannot state its DLO has no principled way to decide what its machines are permitted to do and how they are doing.
4. Autonomy, like trust, is earned slowly and lost quickly
Neither Google nor Netflix leaped from manual operations to closed-loop automation. Google’s SRE handbook describes a typical evolution: from no automation, to externally maintained scripts, to automation owned by the service team, to systems that require no intervention at all.
Netflix's production usage of our Winston/StackStorm platform followed the same trend. Their runbooks are clustered into four patterns of increasing consequence: squelching false positives, gathering diagnostics for the on-call engineer, applying remediation only under conditions verified to be safe, and brokering events to existing tools.
Speaking of which - canarying is a key concept taken for granted by SREs that cyber perhaps ought to adopt, and perhaps inevitably will: every consequential change proves itself against a small blast radius before addressing a large one. This is basically like sandboxing to explode payloads, but for automated ops.
AI systems in the SOC can begin where errors are cheap (enrichment, deduplication, diagnostic assembly), demonstrate their judgment in shadow against human dispositions, and be promoted to consequential action only as its measured performance, measured in part by the Detection Level Objective (DLO), earns the promotion. Staging and canarying are crucial steps that allow us to move faster as trust is earned.
5. Platforms beat scripts and avoid handoffs that destroy context
DevOps folks talk about flow, i.e., optimizing the whole system rather than any silo, reducing batch sizes, and reducing handoffs, because handoffs between teams lose context and often result in rework. Netflix internalized this architecturally. StackStorm turns our scripts into reliable workflows. It was and is a scaffolding in today’s parlance and includes a number of primitives we are now building into the Vigil open source AI SOC: runbooks written as code then and markdown today, stored in version control, isolated per team, promoted through dev, test, and prod branches, and deployed to a highly available, region-isolated fleet. Via StackStorm and the amazing teams at Netflix and elsewhere, operational knowledge became a software artifact with a lifecycle, subject to review, testing, versioning, and rollback.
Compare this to the SOC; setting aside all the reasons we got here, the SOC is organized almost entirely around handoffs. Tier 1 escalates to Tier 2, Tier 2 to incident response, and so on. As we’ve learned the hard way, at each boundary context evaporates a little bit, and the investigation is basically restarted.
Detection logic, meanwhile, all too often lives as vendor-managed signatures and tribal knowledge rather than as versioned, testable artifacts, whether rules or models, that the defending organization owns. Surely detection and response deserve the full software lifecycle, and the investigative flow should be continuous from signal to disposition, carried by a platform that preserves context rather than a tiered structure that sheds it.
6. Incidents are curriculum
A lot of DevOps is about feedback and learning: shorten and amplify the loops that carry information from production back to the people who can act on it, and build a culture in which failure produces knowledge rather than blame. The SRE practice of the blameless postmortem is crucial here. Outages become inputs to improvements, and the organization compounds and learns.
Security generates this feedback in volume and then, generally speaking, discards it. Analyst dispositions can be valuable labeled examples. False positives can often be used to teach the system. At human speed, this learning loop closes through wiki pages and tuning tickets, assuming that homework is completed. At machine speed, the loop must terminate in the detection layer itself: dispositions and outcomes flowing back as training and tuning signals. Ideally, a foundation model like a LogLM acts as a substrate, encoding a beyond-human understanding of how the system actually performs, allowing for a huge improvement in DLO that further improves instead of degrades over time. Foundation models allow for safe learning across environments.
The interval, again
Let’s review this journey into DevOps lore and lessons learned.
First, in DevOps land, we came to treat human execution of documented procedures as a defect. We eventually allowed the operator to become the engineer and supervisor of the system that replaces the paperweight procedures. Measure what matters and set objectives like the DLO against it: This builds in a reality-based understanding of systems failure rates within environments compounding complexity. Grant autonomy in stages using those measurements. Build the loop as a platform that preserves context. And close the loop by feeding every outcome back into the system's judgment; note that this does imply that each enterprise must own their learning loop, a trend much broader than cyber security. This is the path operations walked from the pager to a largely self-healing service with the help of platforms like dear old StackStorm. This seems very much like the path security should walk from the alert queue to machine-speed defense.
AI is here now and can help us further accelerate.
And I’ll take that up in a blog to come. Your feedback would be amazing. Many thanks to reviewers and contributors, including Colin McNamara and Julie Tsai. They made this draft much better. Any remaining confusing sections are all my fault.
