Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment

Large Language Models (LLMs) are being utilized in dialogue assistants to aid software developers, but current evaluation benchmarks primarily focus on functional correctness, neglecting the assessment of Non-Functional Requirements (NFRs). NFRs are inherently vague and context-dependent, making their evaluation challenging. Researchers have identified a critical gap in evaluating the quality and accuracy of LLM-based conversations when handling NFRs, which involve multiple aspects of a program¹. This gap has significant implications for the development of reliable and efficient software systems. The evaluation of LLM-based dialogue assistants' ability to handle NFRs is crucial, as it directly affects the overall quality and performance of the software. As AI technology continues to advance, the ability to accurately assess and satisfy NFRs in multi-turn dialogues becomes increasingly important. This matters to practitioners because inaccurate or unsatisfactory handling of NFRs can lead to significant security and functionality issues in software systems.