Learning, Potential, and Retention: An Approach for Evaluating Adaptive AI-Enabled Medical Devices

Evaluating adaptive AI models in medical devices poses significant challenges due to continuous updates of both models and evaluation datasets. A new approach has been proposed to address this issue, focusing on three key measurements: learning, potential, and retention. Learning assesses model improvement on current data, while potential evaluates dataset-driven performance shifts, and retention examines knowledge preservation. This framework enables a more comprehensive understanding of adaptive AI models in medical devices, allowing for more accurate performance assessments¹. The introduction of this approach is crucial in the medical field, where AI-enabled devices are becoming increasingly prevalent. By providing a more nuanced evaluation framework, manufacturers and regulators can better understand the capabilities and limitations of these devices. This, in turn, can lead to improved patient outcomes and more effective device development, so the ability to accurately evaluate adaptive AI models is essential for ensuring the safety and efficacy of AI-enabled medical devices.