Accuracy Is Not Risk: Why Medical AI Should Not Report Only Its Most Likely Label

arXiv:2611.01312v1 Announce Type: cross Abstract: Background. Many medical AI models are trained to reproduce a single reference label and hand the most likely label to the next step in a clinical workflow. A classifier can select that label correctly while providing an inadequate summary for estimating downstream risk. Methods. We use Jeffrey's rule of conditioning to specify when probabilities over intermediate states, such as diagnoses or imaging categories, can be carried forward to an outcome risk. We separate two errors of reporting only the most likely label, discarding the probability of the other states and applying the risk of patients who have the leading diagnosis to a patient who only probably has it. We also give conditions under which the label and the full distribution lead to the same decision, and we propose guardrails for device developers, regulatory reviewers, and health insurers. Results. In a hypothetical chest-pain differential, reporting only…