The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems

The prevailing discourse on AI safety overlooks a crucial aspect: subtle, insidious failures that can have far-reaching consequences. Unlike overt failures, these hidden safety-critical challenges are often embedded in complex systems, making them difficult to detect. They can manifest as plausible, distributed errors that become normalized within workflows, rather than spectacular, localized incidents. The absence of instrumentation for these failures means they can persist undetected, potentially leading to catastrophic outcomes. A case in point is the lack of visibility into errors that can arise from interactions between components, which can have a profound impact on system reliability. The failure to address these hidden challenges can have significant implications for security, policy, and workforce dynamics, as evidenced by developments in AI from major vendors like ARM¹. This oversight matters to practitioners because it underscores the need for a more nuanced approach to AI safety, one that prioritizes the detection and mitigation of subtle, yet potentially catastrophic, failures.