Large Language Models are being utilized in increasingly complex tasks, which often involve subjective decision-making and require trustworthy outputs. To address this challenge, researchers are focusing on developing uncertainty-aware generation and decision-making capabilities for these models. This involves designing algorithms that can quantify and manage ambiguity in their outputs, enabling more reliable and informed decision-making. The importance of this development lies in the fact that many real-world applications of LLMs, such as those in policy and security, require a high degree of trustworthiness and transparency1. As AI models become more pervasive, their ability to handle ambiguity and uncertainty will be crucial in determining their effectiveness and reliability. The development of uncertainty-aware LLMs has significant implications for practitioners, as it will enable them to make more informed decisions and mitigate potential risks associated with AI-driven systems.