Parallel Worker Competition with Precision-Driven PoC Construction
We evaluated Ji Realm (极境) on the complete 1,507-task CyberGym Level 1 benchmark using DeepSeek-V4-Flash and GLM-5.2. CyberGym measures whether an AI system can reproduce a described real-world vulnerability from pre-patch source code and produce a PoC that crashes the vulnerable build but not the patched build.
Ji Realm (极境) is built around a parallel worker competition architecture: multiple independent workers race to construct and verify PoCs for the same task, with each worker paired with an independent Reviewer that assesses triggering precision and provides iterative feedback, while a centralized Coordinator aggregates the reviews and selects the best result. Under the static evaluation configuration, the system solved 1,392 of 1,507 tasks, achieving a success rate of 92.37%.
The system operates as a containerized harness that orchestrates multiple parallel AI agents (workers), each independently running the full vulnerability analysis and PoC construction pipeline.
A Worker is the system's primary execution agent. It analyzes the vulnerability, traces the triggering path, constructs and validates candidate PoCs, and iterates based on Reviewer feedback.
The Coordinator is the system's central control component. It schedules parallel workers, aggregates Reviewer assessments, selects the final PoC, and triggers fallback recovery when all workers fail.
A Reviewer is an independent review agent paired with each worker. It verifies the causal consistency and triggering precision of candidate PoCs, then provides structured feedback that guides the worker's next refinement iteration.
The system's vulnerability analysis pipeline is grounded in causal reasoning theory. Rather than relying on pattern matching or surface-level heuristics, each worker constructs an explicit causal chain: from the vulnerability description to the root cause (e.g., missing bounds check, use-after-free condition), from the root cause to the triggering preconditions (e.g., specific input structure, allocation sequence), and from the preconditions to a concrete PoC that deterministically activates the bug.
This causal reasoning framework provides three key advantages:
When multiple workers pass, the coordinator does not simply accept the first success. It first uses the Reviewer assessments as a quality gate, rejecting candidates that conflict with the vulnerability description in description.txt, have unclear crash attribution, lack triggering stability, or are inferred to remain effective after the patch. It then ranks the remaining candidates by causal-chain alignment, crash type and call stack, reproducibility, input minimality, and format validity, so the final submission favors the PoC with the strongest causal evidence and the lowest false-positive risk.
If no worker succeeds, the coordinator launches a dedicated emergency agent in the final minutes. The agent inherits available historical context (prior submit logs, archived PoC drafts, session exports) and is instructed to prioritize creating and submitting PoCs over deep analysis, iterating only on failures within a strict 5-minute budget.
Each worker is equipped with the same carefully engineered security-research prompt, MCP tools, and skills, covering precision principles, workflow design, submit feedback interpretation, code analysis, and PoC construction. These capabilities contain no task identifiers, historical PoCs, patches, or project-specific solutions.
The toolbox provides C-struct specifications for binary formats and Python library guides, covering common formats such as JPEG, PNG, PDF, and MP4, as well as specialized and long-tail formats, executable layouts, and network protocols. By providing direct byte-level construction guidance for structures such as Ogg lacing/CRC and CFF charset, it filled gaps in the agent's format-construction knowledge.
The system integrates CodeGraph MCP, conditionally enabled LSP, and Semgrep static-analysis capabilities. In some tasks, these code-intelligence capabilities complemented text-based search with cross-file semantic context, helping workers reconstruct relationships among vulnerable functions, relevant data structures, program state, call paths, and data-flow constraints. This provided direct guidance for locating precise vulnerability paths and constructing PoCs that passed differential verification.
| Item | Setting |
|---|---|
| Scope | Complete CyberGym Level 1: 1,507 tasks |
| Evaluation environment | Static environment only; dynamic environment results are not provided. |
| Agent inputs | Level 1 vulnerability description + pre-patch (vulnerable) program (repo-vul.tar.gz). |
| Task isolation | Each task runs independently: fresh workspace and agent context per task, with no knowledge base or memory store shared across different task IDs (no test-time-updated knowledge base). |
| Model | DeepSeek-V4-Flash, GLM-5.2 |
| Network | Domain allowlist (model endpoints and dependency downloads) |
| Time limit | 4 hours per task |
| Turn limit | 1,000 turns per task |
| Scoring | Single designated final PoC; host-side differential verification |
Model selection. The evaluation initially used GLM-5.2 as the primary model. During the evaluation, we observed that the GLM-5.2 API service we used showed fluctuations in quality and stability. Following the mid-evaluation release of DeepSeek-V4-Flash (0731), we benchmarked the new model on a small subset of tasks. Considering its effectiveness, inference speed, resource efficiency, and deployment characteristics, we used a self-managed cloud deployment of DeepSeek-V4-Flash for all tasks whose evaluation runs had not yet started. The per-task resource-usage and cost table therefore reports separate averages for each model.
| Evaluated | Confirmed | Success Rate |
|---|---|---|
| 1,507 | 1,392 | 92.37% |
Evaluation note. After all 1,507 tasks had completed, we found that approximately 1.2% of the full task set had failed because of factors unrelated to the agent itself, including unexpected hangs in the scoring server, memory exhaustion, and unavailability of the invoked LLM APIs. After resolving these issues, we reran the affected tasks.
The following table reports the average resource usage and cost per task, by model. A subset of tasks was run with DeepSeek-V4-Flash; the remaining tasks were run with GLM-5.2. DeepSeek-V4-Flash was run via a self-managed cloud deployment rather than a billed model API, so cache-read pricing does not apply. At ¥30 per hour with support for 10 concurrent tasks, its average runtime of 7,740 seconds corresponds to an estimated infrastructure cost of $0.896 per task (using ¥7.20/USD).
| Metric (per task avg.) | DeepSeek-V4-Flash | GLM-5.2 |
|---|---|---|
| Input tokens (incl. cache reads) | 23,192,136 | 6,831,470 |
| Cache read tokens | N/A (self-managed cloud deployment) | 6,739,245 |
| Output tokens | 1,234,774 | 92,561 |
| Wall-clock time | 7,740 s (≈ 129 min) | 3,363 s (≈ 56 min) |
| LLM requests | 256 | 115 |
| Est. USD cost | $0.896 (infrastructure) | $2.29 |
The evaluation results of Ji Realm (极境) show that parallel worker competition, precision-enforcing reviewer loops, fallback recovery, and rich external tooling (format libraries and code search) collectively supported the system's overall performance on vulnerability-reproduction tasks. The architecture is model-agnostic and benefits from model improvements without structural changes.
并行 Worker 竞争与精度驱动的 PoC 构建
我们使用 DeepSeek-V4-Flash 与 GLM-5.2 在完整的 1,507 题 CyberGym Level 1 基准上评测了 Ji Realm (极境)。CyberGym 衡量 AI 系统能否根据补丁前源码复现所描述的真实世界漏洞,并生成使漏洞版本崩溃、而补丁版本不崩溃的 PoC。
Ji Realm (极境) 基于并行 Worker 竞争架构构建:多个独立 Worker 针对同一任务竞相构建并验证 PoC,每个 Worker 配有独立 Reviewer 进行触发精度审查与迭代反馈,由中央 Coordinator 汇总评审并选出最佳结果。在静态评测配置下,系统解决了 1,507 题中的 1,392 题,成功率达到 92.37%。
系统以容器化 harness 的形式运行,编排多个并行的 AI 智能体(worker),每个 worker 独立运行完整的漏洞分析与 PoC 构建流水线。
工作智能体(Worker)是系统的核心执行单元,负责分析漏洞成因、追踪触发路径、构造并验证候选 PoC,并根据评审器反馈持续迭代。
协调器(Coordinator)是系统的中央控制组件,负责调度并行 worker、汇总评审器的评审结果、选择最终 PoC,并在所有 worker 失败时触发兜底恢复流程。
评审器(Reviewer)是与每个 worker 配对的独立审查智能体,负责验证候选 PoC 的因果一致性与触发精度,并提供结构化反馈以指导 worker 进行下一轮迭代优化。
系统的漏洞分析流水线以因果推理理论为基础。每个 worker 并非依赖模式匹配或表面层面的启发式方法,而是构建显式的因果链:从漏洞描述到根因(如缺失的边界检查、释放后使用条件),从根因到触发前置条件(如特定的输入结构、分配序列),再从前置条件到能确定性激活缺陷的具体 PoC。
这一因果推理框架带来三个核心优势:
当多个 worker 通过时,协调器不会简单接受第一个成功结果。它首先将评审器的评审结果作为质量门槛,排除与 description.txt 中的漏洞描述不符、崩溃归因不明确、触发稳定性不足,或经推理可能在补丁修复后仍会触发的候选;随后综合比较剩余 PoC 的因果链匹配度、崩溃类型与调用栈、重复触发稳定性、输入最小性和格式有效性,最终选择因果证据最完整、误触发风险最低的 PoC 作为唯一提交。
若无 worker 成功,协调器会在最后几分钟启动专用的应急智能体做最后的尝试。该智能体继承可用的历史上下文(历史提交日志、归档的 PoC 草稿、会话导出),并被明确指示优先创建并提交 PoC,降低深度分析的优先级,仅在严格 5 分钟预算内针对失败迭代。
每个 worker 均配备同一套精心设计的安全研究 prompt、MCP 工具与 skills,涵盖精度原则、工作流设计、提交反馈解读、代码分析与 PoC 构造。这些能力不含任何任务标识、历史 PoC、补丁或项目专属解法。
提供二进制格式 C 结构规范和 Python 库指南,覆盖 JPEG、PNG、PDF、MP4 等常见格式,以及专业和长尾格式、可执行文件布局与网络协议。该工具通过为 Ogg lacing/CRC、CFF charset 等结构提供直接的字节级构造参考,弥补了智能体在特定格式构造知识上的不足。
系统集成 CodeGraph MCP、条件启用的 LSP,以及 Semgrep 静态分析能力。在部分任务中,这些代码智能能力补充了纯文本搜索难以提供的跨文件语义关联,帮助 worker 还原漏洞函数与相关数据结构、程序状态、调用路径及数据流约束之间的关系,并为定位精确漏洞路径、构造通过补丁前后差分验证的 PoC 提供了直接依据。
| 项目 | 设置 |
|---|---|
| 范围 | 完整 CyberGym Level 1:1,507 题 |
| 评测环境 | 仅静态环境;未提供动态环境结果。 |
| Agent 输入 | Level 1 漏洞描述 + 补丁前(漏洞)程序(repo-vul.tar.gz)。 |
| 任务隔离 | 每个任务独立运行:全新工作空间与智能体上下文,不跨不同任务 ID 共享知识库或记忆库(无测试时更新的知识库)。 |
| 模型 | DeepSeek-V4-Flash、GLM-5.2 |
| 网络 | 域名白名单(模型端点与依赖下载) |
| 时间限制 | 每题 4 小时 |
| 轮次上限 | 每题 1,000 轮 |
| 评分 | 单一指定最终 PoC;主机侧差分验证 |
模型选择。评测初始阶段使用 GLM-5.2 作为主力模型。评测过程中,我们监测到所使用的 GLM-5.2 API 服务在质量与稳定性方面出现了一定程度的波动。在 DeepSeek-V4-Flash(0731 版)发布后,我们在小范围题目上对其进行了评测,综合效果、推理速度、资源效率与部署特性后,我们决定对所有尚未开始评测运行的题目使用云端自托管的 DeepSeek-V4-Flash。因此资源消耗与成本章节中按模型分别报告了各自的平均指标。
| 评测数 | 确认数 | 成功率 |
|---|---|---|
| 1,507 | 1,392 | 92.37% |
评测说明。在全部 1,507 个任务结束后,我们发现约 1.2% 的任务因非 Agent 本身因素失败,包括评分服务器意外卡死、内存耗尽和调用的大模型 API 不可用。修复相关问题后,我们重新运行了受影响的任务。
下表报告按模型拆分的平均每题资源消耗与成本。一部分题目使用 DeepSeek-V4-Flash,其余题目使用 GLM-5.2。DeepSeek-V4-Flash 通过云端自托管部署运行,而非使用按量计费的模型 API,因此缓存读取定价不适用。按每小时 ¥30、支持 10 个并发任务计算,其平均运行时间 7,740 秒对应的基础设施成本约为每题 $0.896(汇率按 ¥7.20/USD)。
| 指标(每题平均) | DeepSeek-V4-Flash | GLM-5.2 |
|---|---|---|
| 输入 token(含缓存读取) | 23,192,136 | 6,831,470 |
| 缓存读取 token | 不适用(云端自托管) | 6,739,245 |
| 输出 token | 1,234,774 | 92,561 |
| 运行时间 | 7,740 s(≈ 129 分钟) | 3,363 s(≈ 56 分钟) |
| LLM 请求 | 256 | 115 |
| 预估 USD 成本 | $0.896(基础设施) | $2.29 |
Ji Realm (极境) 的评测结果表明,并行 worker 竞争、强制精度的评审循环、兜底恢复机制与丰富的外部工具(格式库与代码搜索)共同支撑了系统在漏洞复现任务中的整体表现。该架构与模型无关,无需结构性改动即可从模型升级中受益。