Executive Summary

Over several months, Okta has used frontier AI models to proactively hunt for vulnerabilities and secure Okta’s products as part of OpenAI’s Daybreak Cyber Partner Program and Anthropic's Project Glasswing. 

These partnerships allowed us to access the frontier models GPT 5.5 Cyber Preview (TAC) as well as Mythos Preview to scan millions of lines of code in multiple languages and across very different tech stacks. 

We set out to achieve two goals: the first was to use these frontier models to find vulnerabilities so we can better protect our customers, and the second was to learn from the experience to inform our future AI-driven security work.

This blog post aims to add new insights to an emerging body of work from those organizations who have accessed these frontier capabilities ahead of any public release. We expect many Okta customers, especially those in technology and financial services, are now in the process of planning for their own adoption of these capabilities. Throughout this post, we share how we built our pipeline, how we conducted the scanning, and what we learned, in the hopes that others can find value from our experience. 

Our key finding was that we achieved the best results when human experts and AI agents worked together and taught each other.

What We Built

We started by looking at the excellent work already published in the community around using AI for vulnerability scanning, bringing together the best ideas that would work for our goals. (See the appendix for key references we used as inspiration.)

This research inspired the design and implementation of a new system from scratch to be able to conduct this type of scanning. We decided to create an architecture that is flexible enough to evolve and isn’t locked to a single vendor. We also wanted a system that could be repurposed for tasks beyond vulnerability discovery. 

Here’s what we built based on our core architectural goals:

• Secure: Using inspiration from many orchestration projects (listed in the appendix), we chose to build a custom lightweight orchestrator in Python. We designed it to have strong isolation to reduce the attack surface.

• Flexible and extensible: We designed our system so it could use any model, with any coding harness or any agentic SDK, with minimal development effort. We also built it to allow us to quickly deploy new workflows using configuration files, similar to a Directed Acyclic Graph (DAG) system.

• Inspectable: We created real-time tracking, allowing human reviewers to follow what happened from a dashboard. We also captured the full sessions of each pipeline step and agent.

Scanning Pipelines

A key design decision is how to build our scanning pipelines - that is, the sequence of steps and loops we used to find vulnerabilities. 

Instead of telling our agent to “go find vulnerabilities” and allow it to wander off, we took a graph-engineering approach. We defined complex steps and sequences with iterations, loops, and branching, without losing control of the agent. 

Each step is executed as a full isolated Codex or Claude Code session using progressive context loading - this means that we carefully controlled what files each step gave to the next one (though the agent could intentionally decide it needed to read additional findings from a previous step). We did this to reduce so-called “context bloat,” where an agent starts to make poorer decisions because it has too much information in its context window. 

We also chose to develop four different pipelines to scan our code. While running different pipelines resulted in more work, we wanted to use the opportunity to improve our approach to using AI models as part of vulnerability discovery.

Figure 1: Madara-Scorecard pipeline is a linear 8-step flow with a feedback loop that re-hunts coverage gaps up to three times before consolidating findings. Each step is executed as a full isolated Codex or Claude Code session using progressive context loading.

Using four distinct pipelines allowed us to analyze which steps worked better, providing feedback on how we could create new, specialized pipelines based on product type, tech stack, or a specific type of vulnerability we needed to find. Our research demonstrated that a different sequence of steps is more likely to find a different type of vulnerability.

Figure 2: Using the Sink-Thorough pipeline, dual parallel scanners feed into a linear analysis chain, then fan out to four parallel revalidation chunk loops before merging and consolidating results.

Of course, having four different pipelines scan hundreds of repositories with both models (GPT Cyber and Mythos) came with a tradeoff - we accepted that this would result in duplicate findings, a potentially higher false positive rate, and a more burdensome triage process. 

How We Scanned

Before we could unleash the agents, we had to decide exactly what we wanted to scan. To ensure we delivered the greatest impact in the shortest amount of time, we started by scanning the code that matters most to protect: the software our customers install themselves. We deployed our agents across the entire code base, letting them go “wide” first. Even though we knew this approach would generate many results (including false positives), we accepted this trade-off because we felt that even false positives could be valuable to provide ideas for future research.

We had both humans and AI agents perform scanning in various ways. We took two main approaches, which happened in parallel: allowing AI agents to autonomously scan across our code repositories, and pairing human experts with AI agents to direct them to investigate an area or idea they had. The goal was to see if the expert could help the agent get to better or more novel results.

A Novel Approach to Capturing Sessions

We chose to capture all of the executed step sessions. This allowed human experts to go back at any point in time to review what an agent did. Even though the pipeline was complex, if the experts observed that the model did something interesting at a certain point in the pipeline, they could go back and explore that further. This also allowed us to fully validate what happened in the pipeline. This approach challenges the commonly-held view that autonomous AI is an opaque system - we wanted to have a way to inspect it, even interactively.

This concept might sound familiar if you’ve done exploit development, forensics, or malware analysis. Going back to a specific moment of analysis is helpful, and that’s the concept we applied. One of our engineers was inspired by the idea of “time travel debugging” in exploit development - meaning a developer can record a program's execution and move both forward and backward through its operational history. If you’ve ever taken a snapshot of a virtual machine to look at a system before and after malware executed, it’s a similar concept. Though the concept isn’t novel, our application of it was.

To do this, we identified the minimum information we needed to capture to reconstruct a session, since Codex and Claude Code collected a lot of extra information we didn’t need. We implemented logic in our orchestration system to keep only what was essential. Once a step in a pipeline was complete and the session was captured, the expert could launch Codex or Claude Code and load it as an interactive session for that specific step and then “guide” the model on what to do next. We used session viewer tools to help with this task, allowing experts to look at the traces in an HTML viewer to inspect what the model did. Since the pipeline has independent steps, this approach allowed us to change different steps in the process to explore how that changed our result.

This approach unlocked a lot of value for us. If human experts observed something interesting in the middle of the model conducting the scanning, they could load it. 

We learned that when you pair the model with someone who can steer it, you get a great result.

How We Triaged

After we scanned using the four pipelines and our collaborative human-agent approach, we were left with many findings from the models, including duplicates. We tackled the task of triaging those investigative leads and validating if they were true exploitable vulnerabilities, and we needed to do it quickly. As we’ve done throughout the project, we paired our human experts with our AI agents to complete this as effectively as possible.

We prioritized triaging the highest severity findings from our scans as well as any findings that could potentially be assigned a CVE, since any true positives would have the greatest impact. Any potential CVE findings were only triaged manually by humans, since it was crucial to validate this as quickly and accurately as possible. As we triaged, we observed that the models regularly over-estimated the importance of findings, and we downgraded the criticality due to the real-world conditions being different than the model assumed. For example, a model might have assumed the code would be directly exposed to the Internet, but in reality it was not configured that way. 

After our agents did their triaging work, humans verified whether the results and verdicts were accurate. When findings were false positives, those human experts investigated to figure out why, so they could make improvements to our pipelines to reduce similar false positives in the future.

What We Learned

Partner humans and agents for the best results

Throughout the project, our best results paired human experts with AI agents. The scanning process was an exercise in reciprocal mentoring between us and our agents. We first allowed agents to scan the code without additional context so they could teach us new approaches and techniques, and we then taught the agents how to focus using our institutional knowledge around the codebase. For example, we tried giving the models “hints” about previous vulnerabilities as additional context.

We had experts in that codebase work together with the agent, pushing it more into exploring a specific attack surface, product area, or complexity. We found that our results were more impactful when an expert guided the agents, and we were able to uncover findings that the autonomous agent may have missed. For example, a human expert might know certain areas of code containing high risk functionality such as complex authentication flows or cryptographic primitives. Instead of the agent wandering the code base trying to find these on its own, the agents performed better when guided as close to the potential vulnerability as possible.

Agents were sometimes like hunting dogs getting us close to a vulnerability. The agents often reported findings that weren’t true vulnerabilities as-is, but the human expert took that as an investigative lead, pushed the agent further, connected additional context, and ultimately found additional vulnerabilities.

This project would not have been possible without the hard work of our Product Security engineers. Before reporting any vulnerabilities found by these models, it required a major effort to triage, reproduce, and report these vulnerabilities over the course of many weeks.

Experiment and iterate to continuously improve

We intentionally set out with the goal to use this project as a learning experience. It would have been simpler to use one pipeline, but instead we used four. It would have been simpler and cheaper if we hadn’t captured sessions, or insisted on collaboration between agents with human expertise. If an experiment didn’t work the first time, we iterated and tried it again. This iterative approach didn’t just help us learn, it also led to better results because we didn’t just scan once and declare victory.

Our work doesn’t stop here. We’ve captured a rich data set and we have a plethora of ideas for what to do next and how we can improve. We will keep iterating and learning as we continue this critical work to find gaps and close them before threats can exploit them.