Win by Silence: Deletion Non-Monotonicity, Autonomous Exploitation, and Typed-State Gating in LLM Plan Evaluation

Researchers have identified a critical flaw in plan evaluators used for large language models (LLMs), where deleting certain transitions in a plan can actually increase its score. This phenomenon, known as deletion non-monotonicity, occurs when an interior transition is removed and its predecessor is retargeted, while retaining downstream value. The score change can be calculated using the formula Delta_k = (prod_{i1. This discovery has significant implications for the development of LLMs, as it suggests that current evaluation methods may be flawed. The findings are based on a study of a 26-route cohort, where all 57 instances exhibited this behavior. This matters to practitioners because it highlights the need to re-examine evaluation metrics for LLM-generated plans to ensure they are accurately assessing plan quality.