Minerva: Reinforcement Learning with Verifiable Rewards for Cyber Threat Intelligence LLMs

Md Tanvirul Alam; Aritran Piplai; Ionut Cardei; Nidhi Rastogi; Peter J Worth Jr

Accepted at TMLR · 2026

Minerva: Reinforcement Learning with Verifiable Rewards
for Cyber Threat Intelligence LLMs

Md Tanvirul Alam¹Aritran Piplai²Ionut Cardei³Nidhi Rastogi¹Peter J Worth Jr³

¹Rochester Institute of Technology²University of Texas at El Paso³Florida Atlantic University

Read the paper Code OpenReview

Public security knowledge becomes 16 verifier-checkable CTI tasks. MinervaRL combines GRPO with verified, filtered teacher traces for hard prompts, then learns from those traces on the original prompts. — Public security knowledge. Checkable answers. A stronger learning signal.Open overview

The idea

Better answers start
with better feedback.

CTI analysts turn messy security information into structured outputs: weakness identifiers, attack techniques, severity vectors, and more. These outputs can often be checked against public standards, making CTI a natural setting for reinforcement learning with verifiable rewards.

Minerva pairs a curated training suite with MinervaRL, a method that supplements GRPO with verified teacher traces when a model struggles to produce a fully correct answer.

Read the full abstract

Cyber threat intelligence (CTI) analysts routinely convert noisy, unstructured security artifacts into standardized, automation-ready representations. Although large language models (LLMs) show promise for this task, existing approaches remain brittle when producing structured CTI outputs and have largely relied on supervised fine-tuning (SFT). In contrast, CTI standards and community-maintained resources define canonical identifiers and schemas that enable deterministic verification of model outputs. We leverage this structure to study reinforcement learning with verifiable rewards (RLVR) for CTI tasks. We introduce Minerva, a unified dataset and training pipeline spanning multiple CTI subtasks, each paired with task-specific verifiers that score structured outputs and identifier predictions. To address reward sparsity during rollout, we propose MinervaRL, an auxiliary self-training mechanism that generates additional verified trajectories and distills them back into the model. Averaged across four backbones and 12 CTI benchmarks, MinervaRL improves the mean score by 15.8 percentage points over the corresponding base models and by 4.3 points over GRPO.

verifier-checkable training tasks: 16
curated training examples: 32,000
points over base models: +15.8pp
points over standard GRPO: +4.3pp

Score improvements are averaged across four backbones and 12 evaluation benchmarks. pp = percentage points.

01 / Minerva-CTI

Security knowledge with checkable answers.

Minerva-CTI turns public standards, knowledge bases, and detection resources into 16 training tasks. Each example pairs an analyst-style prompt with a canonical target and a task-specific verifier. The split contains 32,000 training and 1,200 validation instances.

Vulnerability mapping

Map a CVE description to a canonical weakness or a CVSS severity vector.

CVE descriptionCWE / CVSS

Detection engineering

Connect detection rules and emulation tests to ATT&CK techniques.

Rule or testATT&CK technique

Adversary behavior

Identify techniques, tactics, and mitigations from attack scenarios.

Attack scenarioStandardized identifiers

Threat context

Link actor behavior and CAPEC examples to structured security knowledge.

Behavior or exampleActor / pattern / weakness

Representative task groups. Sources include MITRE ATT&CK/CWE/CAPEC, CVE/NVD, Mappings Explorer, Sigma, Atomic Red Team, Microsoft Sentinel, and Splunk Security Content.

02 / MinervaRL

When rewards are sparse, learn from verified traces.

MinervaRL keeps the standard GRPO loop and adds an auxiliary self-training path. Prompts without a fully verified rollout receive extra help from a teacher. Accepted traces are distilled back onto the original, answer-free prompts.

01
Sample and verify
Generate eight on-policy rollouts per prompt and score them with the task verifier. Continue the standard GRPO update.
02
Help the hard prompts
If no rollout is fully verified, an EMA teacher generates traces conditioned on the target labels. Verify them and filter for quality.
03
Learn on the original prompt
Periodically distill accepted traces onto the original prompt, without the target labels. This adds a supervised learning signal.

The distinction matters. Labels help generate the teacher trace; the student learns from the original, answer-free prompt. The auxiliary update supplements the GRPO objective.

Explore the full method diagram from the paper

The standard GRPO update and the auxiliary ACR generation, filtering, buffer, and periodic distillation paths. — Solid arrows show synchronous training; dashed arrows show periodic distillation. Click the figure to enlarge.

03 / Main results

Higher average scores across all four backbones.

Across 12 CTI benchmarks and four open-weight backbones, MinervaRL improves the mean score by 15.8 percentage points over the corresponding base models and by 4.3 points over GRPO. It has the highest average within each backbone group in the reported comparisons.

Llama-3.1-8B: Base 48.6, GRPO 56.0, MinervaRL 60.2. Llama-3.2-3B: 33.1, 42.1, 48.3. Qwen3-8B: 37.6, 47.0, 53.4. Qwen3-4B: 27.6, 47.6, 48.0.

All controlled baselines · 12-task macro average
Backbone	Base	STaR-CTI	DART-CTI	GRPO	LUFFY-CTI	MinervaRL
Llama-3.1-8B	48.6	49.7	53.7	56.0	57.0	60.2
Llama-3.2-3B	33.1	37.3	44.0	42.1	45.4	48.3
Qwen3-8B	37.6	36.8	42.9	47.0	53.2	53.4
Qwen3-4B	27.6	45.4	39.4	47.6	44.0	48.0

Paper Table 1. Scores are percentages; the macro average combines the 12 task metrics. Llama backbones start from Instruct checkpoints; Qwen3 backbones start from Base checkpoints. STaR, DART, and LUFFY are the CTI adaptations used in the paper.

View scores for all 12 benchmarks

Full scores for the six controlled variants of each backbone
Backbone / method	CKT	CyberMetric	SOCEval	RCM	VSP	ATE	RMS	ElasticRule	APTNER	LANCE	AnnoCTR	AZERG	Avg.
Llama-3.1-8B / Base	67.6	83.2	64.8	48.4	76.0	17.4	6.7	14.4	33.5	78.2	50.7	42.8	48.6
Llama-3.1-8B / STaR-CTI	70.0	81.8	61.7	57.4	73.8	29.2	9.6	21.3	33.1	80.0	46.5	32.1	49.7
Llama-3.1-8B / DART-CTI	71.3	82.4	61.5	64.0	73.0	37.8	27.7	31.0	34.0	74.8	47.1	39.3	53.7
Llama-3.1-8B / GRPO	71.9	85.4	63.0	66.3	82.6	32.0	30.9	32.2	32.7	86.2	49.5	39.5	56.0
Llama-3.1-8B / LUFFY-CTI	73.9	83.7	63.3	63.8	79.6	40.6	34.3	33.8	35.4	82.4	49.7	43.1	57.0
Llama-3.1-8B / MinervaRL	73.9	84.2	64.7	68.8	87.6	48.4	42.1	40.5	34.1	84.6	50.3	43.7	60.2
Llama-3.2-3B / Base	71.6	77.0	55.7	15.2	2.8	1.0	0.4	1.2	25.4	77.8	35.9	32.8	33.1
Llama-3.2-3B / STaR-CTI	64.3	70.5	50.3	36.4	53.4	4.2	7.0	1.9	21.4	76.3	33.8	28.0	37.3
Llama-3.2-3B / DART-CTI	73.0	78.2	54.5	54.9	61.2	16.8	17.8	11.6	22.5	73.8	38.7	25.7	44.0
Llama-3.2-3B / GRPO	71.2	77.8	58.3	48.9	56.5	5.2	15.4	5.3	27.5	70.6	38.5	29.7	42.1
Llama-3.2-3B / LUFFY-CTI	70.6	78.5	55.4	43.8	71.6	19.4	24.2	20.4	15.0	68.0	46.0	32.1	45.4
Llama-3.2-3B / MinervaRL	71.0	78.1	54.6	57.2	77.1	21.8	29.3	20.1	16.7	82.2	37.9	33.7	48.3
Qwen3-8B / Base	71.7	69.3	63.8	52.6	69.0	13.4	6.5	9.0	31.3	45.1	9.8	9.8	37.6
Qwen3-8B / STaR-CTI	37.9	49.3	65.1	36.8	68.7	15.6	3.5	4.9	37.9	55.7	40.0	26.4	36.8
Qwen3-8B / DART-CTI	39.2	55.7	65.2	47.1	72.7	14.0	6.5	16.4	37.9	76.4	46.5	36.6	42.9
Qwen3-8B / GRPO	69.8	72.7	69.0	60.5	68.8	23.6	8.2	18.1	37.7	66.1	37.8	31.1	47.0
Qwen3-8B / LUFFY-CTI	78.6	88.3	64.8	65.3	73.2	29.4	25.8	32.4	34.3	74.5	45.1	26.8	53.2
Qwen3-8B / MinervaRL	77.8	88.2	67.5	64.8	79.4	32.0	20.1	22.0	37.4	74.9	43.0	33.1	53.4
Qwen3-4B / Base	45.6	50.1	56.1	45.6	76.5	4.2	6.5	4.6	1.2	18.0	16.6	6.5	27.6
Qwen3-4B / STaR-CTI	76.1	88.2	66.1	21.1	80.2	5.8	6.6	12.3	31.8	62.0	46.9	47.7	45.4
Qwen3-4B / DART-CTI	45.4	42.1	60.0	54.6	73.8	20.6	5.8	17.1	27.6	47.5	43.2	35.1	39.4
Qwen3-4B / GRPO	74.2	85.8	63.9	60.8	88.1	20.2	3.8	20.4	32.4	58.2	39.6	24.0	47.6
Qwen3-4B / LUFFY-CTI	54.1	56.5	60.9	59.9	72.6	17.6	10.3	23.4	29.2	60.4	46.7	36.2	44.0
Qwen3-4B / MinervaRL	70.0	80.0	64.0	59.9	80.0	25.4	5.1	27.5	28.0	56.7	50.4	29.1	48.0

The paper additionally reports Llama-Primus-Merged and Foundation-Sec-8B-Instruct as external security-SFT references. Consult Table 1 for those two rows.

Download the reported data (JSON) ↓

Beyond the average

34 of 48 point comparisons
favor MinervaRL over GRPO.

Task-level gains vary. Qwen3-4B has the smallest mean gain, and some tasks regress. Paired 95% bootstrap intervals give a more careful view:

28137

Above zeroOverlaps zeroBelow zero

Intervals reflect variation over evaluation examples for fixed checkpoints, rather than variation across training seeds.

04 / Training dynamics

Fewer stalled prompts. Earlier validation progress.

The training curves connect the method to its motivation. MinervaRL reduces the fraction of prompts whose sampled rollout group earns no verifier reward and reaches the best GRPO validation accuracy earlier in training.

Paper training curves comparing GRPO and MinervaRL across the four backbones. Llama-3.1-8B also includes the GRPO-12 rollout control. — Fraction of prompts whose rollout group has maximum reward zero. This diagnostic differs from the method’s no-fully-verified-rollout gate.

Paper validation curves for four backbones. Dashed lines mark best GRPO accuracy and crosses mark the first MinervaRL step to reach it. — Curves use five-step rolling means. Dashed lines mark best GRPO accuracy; crosses mark when MinervaRL first reaches it.

Progress has a compute cost. In the paper’s 10-step Llama-3.1-8B timing study, MinervaRL took 654.5 seconds versus 469.8 for GRPO, a 39.3% increase. Earlier progress in training steps is not a claim of lower cost per step.

05 / Scope & limitations

A useful step, with clear limits.

The strongest gains concentrate on structured, verifier-aligned tasks. Results do not imply uniform improvement on every task or establish reliability for autonomous security decisions.

What the study establishes

Verifiable CTI tasks provide useful reward signals, and auxiliary self-training improves aggregate performance across the evaluated backbone groups.

What remains open

Independent training-seed variation, broader gate and hyperparameter ablations, multilingual CTI, and reducing the cost of trace generation and filtering.

Minerva is intended for defensive CTI analysis. Its outputs support analyst judgment and require validation before operational use.

Build on this work

Citation

Accepted in Transactions on Machine Learning Research, 2026.

@article{alam2026minerva,
  title = {Minerva: Reinforcement Learning with Verifiable Rewards for Cyber Threat Intelligence LLMs},
  author = {Alam, Md Tanvirul and Piplai, Aritran and Cardei, Ionut and Rastogi, Nidhi and Worth, Jr., Peter J},
  journal = {Transactions on Machine Learning Research},
  year = {2026},
  url = {https://openreview.net/forum?id=w1IMu9exKC}
}

Read the full paper ↗Explore the code ↗OpenReview discussion ↗

Minerva: Reinforcement Learning with Verifiable Rewards
for Cyber Threat Intelligence LLMs

Better answers start
with better feedback.