Part of the TPC Seminar Series

---

Abstract:

Large language models (LLMs) have demonstrated promising capabilities in understanding textual data and are increasingly being adopted to help researchers accelerate scientific discovery through knowledge extraction (information retrieval), knowledge distillation (summarizing key findings and methodologies into concise forms), and knowledge synthesis (aggregating information from multiple scientific sources to address complex queries, generate hypothesis and formulate experimental plans). However, scientific data often exists in both visual and textual modalities. Vision language models (VLMs) address this by incorporating a pretrained vision backbone for processing images and a cross-modal projector that adapts image tokens into the LLM dimensional space, thereby providing richer multimodal comprehension. Nevertheless, off-the-shelf VLMs show limited capabilities in handling domain-specific data. We propose two intelligent assistants finetuned from LLaVA, to help scale scientific discovery in low-dose radiation therapy (LDRT), a benign approach used in the treatment of cancer-related illnesses. Using multilingual data from 42673 scientific documents, we devise complex reasoning and detailed description tasks for visual question answering (VQA) benchmarks. Our preliminary experiments show that the scientific assistants, trained on 50,882 image-text data entries, outperform base LLaVA models by +1.8 points (52.1%) for v1.5-13b and +0.97 points (29.1%) for v1.6-vicuna-13b in overall performance, using Qwen2-72B-Instruct as the judge.

Biography:

Chibuike Robinson Umeike is a fourth-year Ph.D. student in Civil, Construction, and Environmental Engineering at the University of Alabama, where he is also pursuing a Master’s degree in Computer Science. He is deeply committed to leveraging the concerted advantage of vision and language modalities for science and engineering applications. Robinson’s research has been featured in peer-reviewed IEEE and ASCE publications and presented at international conferences. Recently, he completed a research internship at Argonne National Laboratory’s Data Science and Learning (DSL) division under the supervision of Neil Getty and Fangfang Xia, where he worked on developing scalable solutions for large vision-language models to accelerate scientific discovery and improve multimodal data comprehension.

Additional Zoom Information:

Meeting ID: 161 029 4961
Passcode: 418036

+16692545252,,1610294961#<tel:+16692545252,1610294961>,,,,*418036# US (San Jose)
+16469641167,,1610294961#<tel:+16469641167,1610294961>,,,,*418036# US (US Spanish Line)

Dial by your location

• +1 669 254 5252<tel:+16692545252> US (San Jose)
• +1 646 964 1167<tel:+16469641167> US (US Spanish Line)
• +1 646 828 7666<tel:+16468287666> US (New York)
• +1 415 449 4000<tel:+14154494000> US (US Spanish Line)
• +1 551 285 1373<tel:+15512851373> US (New Jersey)
• +1 669 216 1590<tel:+16692161590> US (San Jose)