Large language model non-compliance with FDA guidance for clinical decision support devices

Abstract

Large language models (LLMs) show considerable promise for clinical decision support (CDS) but none is currently authorized by the Food and Drug Administration (FDA) as a CDS device. We evaluated whether two popular LLMs could be induced to provide unauthorized, device­like CDS, in violation of FDA’s requirements. We found that LLM output readily produced device­like decision support across a range of scenarios despite instructions to remain compliant with FDA guidelines.