Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling

Multimodal large language models exhibit a critical flaw when evaluating evidence: they often prioritize plausible narratives over perceptually accurate answers, a phenomenon known as Perceptual Judgment Bias. This occurs when visual and textual cues conflict, leading to unreliable judgments. Researchers have identified and analyzed this bias, which undermines the reliability of automated evaluators¹. To mitigate this issue, techniques such as perceptual perturbation and reward modeling can be employed to improve the models' perceptual judgment. By addressing this weakness, multimodal large language models can become more trustworthy and effective evaluators. The implications of this research extend beyond technology, influencing policy, security, and workforce dynamics. So what matters to practitioners is that mitigating Perceptual Judgment Bias is crucial for developing reliable and unbiased AI evaluators.