Evaluation of large language models for VI-RADS reports: a comparative analysis of zero-shot and few-shot prompting

Abstract

Introduction Accurate preoperative staging is vital in bladder cancer management, particularly for assessing muscle invasion. Multiparametric MRI (mpMRI) combined with the Vesical Imaging-Reporting and Data System (VI-RADS) offers a non-invasive and standardized approach for tumor stratification. However, inter-observer variability among radiologists remains a significant limitation. This study investigates the performance of three large language models (LLMs)-ChatGPT (OpenAI; GPT-5.2), Gemini (Google; Gemini 2.0), and Copilot (Microsoft; Copilot based on GPT-5 architecture)-in classifying bladder lesions according to VI-RADS using zero-shot and few-shot prompting strategies. Materials and Methods A synthetic dataset of 100 simulated bladder cancer cases was developed from expert-crafted synthetic radiology reports based on structured text descriptors and expert consensus, comprising 20 cases for each VI-RADS category (1-5). Each case included structured imaging descriptors aligned with VI-RADS scoring criteria. The LLMs were evaluated under zero-shot (no examples provided) and few-shot (with illustrative examples) prompting conditions. Performance metrics included accuracy, macro F1 scores, and Cohen's kappa, with statistical significance assessed using McNemar's test. Results Few-shot prompting significantly improved the classification performance of ChatGPT (OpenAI; GPT-5.2) (accuracy: 94%, F1: 0.939) and Gemini (Google; Gemini 2.0) (accuracy: 83%, F1: 0.823) compared to zero-shot. In contrast, Copilot (Microsoft; Copilot based on GPT-5 architecture)'s accuracy (53%) and F1 score (0.503) declined under few-shot conditions. ChatGPT (OpenAI; GPT-5.2) demonstrated the highest consistency in identifying high-risk lesions (VI-RADS 4-5), followed by Gemini (Google; Gemini 2.0). Conclusion Few-shot prompting enhances LLM performance in VI-RADS classification, particularly for ChatGPT (OpenAI; GPT-5.2) and Gemini (Google; Gemini 2.0). These findings highlight the potential of AI tools to support radiological decision-making in bladder cancer staging. Further studies using real imaging data and explainable AI are warranted.