When are likely answers right? On Sequence Probability and Correctness in LLMs

Large language models' decoding methods often rely on shifting probability mass towards more likely outputs, either locally or globally. The effectiveness of these methods hinges on the alignment between sequence probability and correctness. Research investigates when the conditional probability of a continuation given a prompt actually corresponds to the correct output. This alignment is crucial, as it determines the success of decoding methods in generating accurate responses. The study explores the relationship between sequence probability and correctness, shedding light on the underlying mechanisms of large language models¹. By understanding when sequence probability aligns with correctness, developers can refine their models to produce more accurate outputs. This, in turn, has significant implications for the development and deployment of AI systems, affecting not only technology but also policy, security, and workforce dynamics. So what matters to practitioners is that a deeper understanding of sequence probability and correctness can inform the design of more reliable and trustworthy large language models.