Audit Exposes Label and Split Flaws in Public Thyroid AI Dataset — Research Library

Key Highlights

Nine ultrasound images carry both benign and papillary carcinoma labels across 19 files, creating internally inconsistent ground truth.
Case-level cross-validation yields cytology ensemble AUROC of 0.958 and ultrasound ensemble AUROC of 0.768.
Image-level folds contaminated 99.8 percent of cases yet changed AUROC by only −0.002 and +0.019, prompting scrutiny of shortcut learning.

Context and Core Dynamics

A peer-reviewed audit published in BMC Medical Imaging interrogates a public thyroid imaging resource released by Pang and colleagues, which contains 387 thyroid ultrasound images and 1,332 fine-needle aspiration cytology blocks and has supported studies distinguishing benign thyroid nodules from papillary carcinoma. Behnam Kiani Kalejahi of Central Asian University in Tashkent and Mohammad Javad Rajabi of INTI International University in Malaysia asked what reported performance on this archive demonstrates and what it does not.

The auditors first identified label-level contradictions: nine ultrasound images appeared under both benign and papillary carcinoma labels, accounting for 19 files. The same image was therefore presented to algorithms as cancer and as absence of cancer, injecting noise into training and making accuracy difficult to interpret because the ground truth is internally inconsistent. These conflicting cases were excluded from the audited analysis.

Evaluation design was the second concern. The researchers built a frozen case-level partition, keeping all images from one case on one side of the split, and evaluated five ImageNet-initialised convolutional network backbones plus an ensemble under group-stratified five-fold cross-validation, with thresholds selected on validation data and case-clustered 95 percent confidence intervals. Cytology ensemble AUROC was 0.958, 95 percent CI 0.938–0.975; ultrasound ensemble AUROC was 0.768, 95 percent CI 0.716–0.819. Retraining two backbones with image-level folds, where 99.8 percent of cases were contaminated across folds, changed AUROC by only −0.002 and +0.019, suggesting the models were not primarily memorising individual cases and raising the question of which signal they used. The audit then probed shortcut learning by detecting the sonographic sector and retraining all five backbones and the ensemble with that region removed.

Caregiving Application and Social Impact

For the International AI Caregiving Association (IAICA), the practical lesson is that thyroid nodule decision support should not be adopted on benchmark AUROC alone. Older adults frequently undergo ultrasound and cytology in specialised care, and inconsistent labels or leaky splits can inflate confidence in triage, affecting referral decisions, biopsy avoidance and patient anxiety. Case-level validation, transparent reporting and external checks are needed before such tools enter eldercare or rehabilitation pathways.

Ethics and Compliance Note

The audit also raises governance questions for data provenance, consent and de-identification in public medical image archives. Regulators, clinicians and patients should treat benchmark performance as provisional until models are validated at case level and tested for shortcut reliance. Clear reporting of label conflicts, split design, confidence intervals and external validation is an ethical and compliance expectation, not an optional refinement.