Formalize, Don't Optimize: The Heuristic Trap in LLM-Generated Combinatorial Solvers

Large Language Models (LLMs) often struggle to directly solve complex combinatorial problems, prompting the development of neuro-symbolic systems that leverage LLMs to generate executable solvers. A key consideration in designing these systems is how the LLM represents the solver and whether it should also optimize search. Researchers have introduced CP-SynC-XL, a comprehensive benchmark comprising 100 combinatorial problems with 4,577 instances, to investigate this issue¹. This benchmark allows for the evaluation of LLM-generated solvers and their ability to formalize problems effectively. The study reveals that LLMs tend to fall into a heuristic trap, prioritizing optimization over formalization, which can lead to suboptimal solutions. By recognizing this trap, developers can design more effective neuro-symbolic systems that focus on formalizing problems rather than optimizing search. This matters to practitioners because it highlights the importance of careful LLM design to avoid common pitfalls and ensure the generation of accurate and reliable solvers.