This study introduces a GPT-4–powered multimodal AI agent specifically engineered to support clinical decision-making in precision oncology by integrating vision transformers for detecting microsatellite instability and KRAS and BRAF mutations, MedSAM for radiology segmentation, genomics classifiers, retrieval-augmented generation (RAG), and structured medical databases (OncoKB, PubMed, clinical guidelines).
Evaluated on 20 realistic multimodal patient cases, the AI agent autonomously used appropriate tools with 87.5% accuracy, reached correct clinical conclusions in 91.0% of cases and accurately cited relevant oncology guidelines 75.5% of the time.
Compared to GPT-4 alone, the integrated AI agent drastically improved decision-making accuracy from 30.3% to 87.2%.
The AI agent also showed proficiency in multistep reasoning - chaining outputs from one tool as inputs for others.
These findings demonstrate that integrating language models with precision oncology and search tools substantially enhances clinical accuracy, establishing a robust foundation for deploying AI-driven personalized oncology support systems.
【MORE】