Reinforcement Learning with Evolving Rubrics as Rewards for Audio Reasoning

¹University of Maryland, College Park, ²University of Illinois Urbana-Champaign,
³University of Illinois Chicago, ⁴Microsoft Research, ⁵MBZUAI

Overview

Outcome-based rewards supervise only the final answer, letting the model answer without attending to the audio. Process-based rewards score the reasoning, but with coarse, hand-crafted, fixed criteria. AudioRubrics instead rewards the reasoning process with self-evolving, audio-grounded rubrics: per-sample rubrics are synthesized from the raw waveform, then regenerated and reweighted from the model's own rollouts, so the reward keeps targeting the current policy's weaknesses.

Motivation of AudioRubrics. — **Figure 1.** Evolving rubrics push the reasoning from superficial guesses toward deep, faithful, and evidence-grounded explanations.

Method	Process Supervision	Fine-grained Criteria	Audio- Grounded	Automatic Generation	Evolve with Policy
R1-AQA	✗	✗	✗	✗	✗
Omni-R1	✗	✗	✗	✗	✗
Audio-Reasoner	✓	✗	✓	✗	✗
Audio-Thinker	✓	✗	✗	✓	✗
CESAR	✓	✓	✗	✗	✗
AudioRubrics	✓	✓	✓	✓	✓

Table 1. Comparison with existing post-training designs for audio reasoning.

Method

(i) Audio-grounded initialization. Rubrics are generated from the raw waveform, so every criterion is anchored in the acoustic evidence actually present in the clip.

(ii) Evolving rubrics. At each step, saturated criteria are pruned and harder ones are elicited from the rollouts, so the evaluation standard co-evolves with the policy.

(iii) Overthinking penalty. A linear length penalty counterbalances the rubric reward, keeping the reasoning informative but concise.

$R_i = R^{\text{out}}_i + \gamma\, R^{\text{rub}}_i + \delta\, R^{\text{over}}_i$

Main Results

AudioRubrics achieves the best overall accuracy on all three benchmarks among models of similar size, and its audio perception improves tangibly: on the MMSU perception split it surpasses the best comparable baseline by 4.3 points, an 8.9% relative improvement.

Models	MMAU Test-mini	MMAR	MMSU
Proprietary Models
GPT-4o-Audio	62.50	63.50	56.38
GPT-audio-1.5	74.90	72.00	63.40
Gemini-3-Flash	77.50	76.10	78.28
Gemini-3.1-Pro	77.60	77.50	82.60
Open-source Models
Qwen2.5-Omni-7B	65.20	56.70	60.57
MiMo-Audio (7B)	74.90	56.70	55.18
Audio-Flamingo 3 (7B)	74.26	60.10	62.30
Training-based Models
Omni-R1 (7B)	74.60	58.20	62.20
Ke-Omni-R (7B)	74.60	60.90	62.08
Audio-Thinker (7B)	73.70	65.30	--
CESAR (7B)	77.10	62.70	64.24
AudioRubrics (7B)	78.00	65.80	65.86

Table 2. Overall accuracy (%) on MMAU Test-mini, MMAR, and MMSU. See the paper for full per-dimension results and additional baselines.

Analysis

1. A moderate rubric weight works best. The rubric reward consistently improves over GRPO across the entire range, peaking at $\gamma = 0.5$.

2. The gains scale with the capability of the rubric generator and judge. Replacing Gemini-3.1-Pro with the weaker GPT-audio-1.5 drops performance below the GRPO baseline.

Rubric weight and judge analysis. — **Figure 3.** (a) Impact of the rubric weight $\gamma$. (b, c) Accuracy and training dynamics with different rubric generators and judges.

3. The rubric set keeps upgrading. The share of newly evolved rubrics adopted for the reward grows from roughly 35% early in training to about 60% by the end.

4. Stable reasoning length. Vanilla GRPO collapses toward near-zero length, removing the length penalty leads to unbounded growth, and the full method stays stable throughout.

Ratio of newly-evolved rubrics. — **Figure 4.** Ratio of newly-evolved rubrics adopted during training.

Response length dynamics. — **Figure 5.** Response length during training under three reward configurations.

BibTeX

@article{yu2026audiorubrics, title={Reinforcement Learning with Evolving Rubrics as Rewards for Audio Reasoning}, author={Yu, Fangxu and Feng, Tao and Min, Dehai and Lin, Zinan and Xu, Weijia and Xu, Michael and Yu, Philip S. and Liu, Ge and Zhou, Tianyi}, journal={arXiv preprint arXiv:2608.02831}, year={2026} }