Open, Closed and Broken: Prompt Fuzzing Finds LLMs Still Fragile Across Open and Closed Models

Researchers at Palo Alto Unit42 have exposed the vulnerability of large language models (LLMs) to genetic algorithm-inspired prompt fuzzing, a technique that exploits the fragility of LLM guardrails. This method, which involves generating and testing numerous prompts to evade security controls, has been shown to be effective across both open and closed models. The study reveals that LLMs can be manipulated using scalable evasion methods, highlighting critical security implications for GenAI systems. Specifically, the research demonstrates that prompt fuzzing can be used to bypass safety measures and elicit undesirable responses from LLMs¹. The findings suggest that LLMs are still fragile and require more robust security controls to prevent exploitation. This matters to practitioners because it underscores the need for more rigorous testing and validation of LLMs to ensure their security and reliability in real-world applications.