Leveraging artificial intelligence and augmented reality for enhanced interaction and learning in phytomedicine

Abstract

Medicinal plants are essential to both traditional and modern healthcare systems, providing therapeutic compounds widely used in herbal remedies and pharmaceuticals. Their continued relevance spans centuries of indigenous knowledge and plays a significant role in global health, especially in regions where access to conventional medical care is limited. Despite their benefits, accurate identification and access to structured, trustworthy medicinal information remain major challenges. Many medicinal plants closely resemble toxic or ineffective species, raising risks of misidentification. Additionally, critical knowledge about their preparation, uses, and side effects is often undocumented, fragmented, or orally transmitted, limiting accessibility for the general public. Existing identification tools typically rely on expert input or static databases, offering limited interactivity, poor recognition accuracy, and little real-time feedback. To address these limitations, we present an interactive platform that combines Computer Vision, Augmented Reality (AR), and Retrieval Augmented Generation (RAG) to support education and plant based diagnostics. Users can capture plant images or input symptoms via a web interface for real-time analysis. A Vision Transformer model trained on over 10,000 annotated images achieved 98.65% identification accuracy, outperforming ResNet50, EfficientNet, and VGG16. After identification, the platform simultaneously renders a 3D AR model using Unity and ARCore and retrieves relevant medicinal knowledge from a structured Pinecone database. The content is contextualized using a GPT based language model. Users can interact with the AR model in real time rotating, scaling, and exploring while receiving tailored explanations. Evaluation results show a 30% improvement in knowledge retention compared to traditional learning methods, offering a promising step in AI-driven healthcare, education, and biodiversity conservation.