Why This Study Matters
If you’ve been following the conversation around AI in security operations, you understand that there’s lots of hype, but not a lot of data. For SOC leaders, that’s a problem because you don’t want to waste time on experimentation. It’s one thing to hear vendors talk about efficiency gains, and another to see hard evidence that AI actually improves SOC effectiveness.
That’s why the new report from the Cloud Security Alliance (CSA) is so important. It’s the first independent, data-driven study to benchmark how AI SOC agents affect analyst performance in alert investigations. With 148 participants across a range of experience levels, this was a robust study that everyone involved in security operations needs to see.
What the CSA Study Looked At
The study had two groups of analysts complete two investigation scenarios:
- One group of analysts with AI assistance (through the Dropzone AI platform)
- One group working manually with AWS GuardDuty and Microsoft Sentinel
Both groups were given the same two escalated alert investigations—an AWS S3 bucket alert and a Microsoft Entra failed logins alert. These are Tier 2 investigations, the kinds of alerts that already passed initial triage and need careful analysis.
The goal was to see whether AI could improve analyst speed, accuracy, and consistency.
The Results: Faster, More Accurate, More Consistent
Across every metric, the AI-assisted group outperformed the manual group:
- Accuracy: Analysts with AI support were 22–29% more likely to reach the correct conclusion across the two scenarios.
- Speed: AI-augmented analysts completed the investigations 45–61% faster across the two scenarios.
- Completeness: AI-assisted analysts kept their work thorough, while the manual group showed a drop-off in detail over time.
- Fatigue resistance: With AI, analysts sustained their quality from one investigation to the next while the manual group showed signs of fatigue.
These are important results for SOC leaders to consider, especially given that alert volumes continue to increase and most organizations do not have funds to hire enough new analysts. The CSA benchmark study shows that AI SOC analysts like Dropzone AI will make a material impact on SOC effectiveness.
See the full results! You can read the CSA report here: Beyond the Hype: A Benchmark Study of AI Agents in the SOC.
How Analysts Felt About Using AI
How does it feel to be “AI augmented?” This is an important question when you’re considering whether or not to bring in AI SOC agent technology. The CSA study included questions to measure participants’ sentiment toward AI, both before and after completing the investigation scenarios.
- Even though baseline attitudes toward AI were already positive (8.6/10), 94% of participants responded, “Yes – I view AI more positively” after using it.
- When asked to describe the Dropzone AI platform, the most common words were “Efficient,” “Helpful,” “Time-saving,” and “Intuitive.”
- While measured results proved this out, it’s also important that 94% of participants in the AI-Assisted group felt that AI sped up their investigation. Perceptions matter!
In other words, analysts not only performed better with AI, they also liked using it. For SOC managers worried about adoption hurdles, that’s an encouraging sign.
What This Means for SOCs
The CSA study focused on Tier 2 investigations, but it’s worth remembering that AI SOC agents also completely automate Tier 1 triage: the work of weeding out false positives. That’s what Dropzone AI already does today.
But the CSA study shows that AI also helps analysts work faster and more accurately on escalated alerts. For overworked SOC teams, this dual benefit—automation of Tier 1 and augmentation of Tier 2—is exactly what’s needed to keep up with today’s alert volume.
