Credit: Argonne National Laboratory
Scientists have an image problem. Each year, over a million scientific articles are published, most of which contain complex figures with multiple embedded images, graphs, and illustrations woven throughout the text. Sadly, effectively searching and extracting these images for use by deep learning models has proven challenging.
To solve this issue, scientists at the U.S. Department of Energy’s Argonne National Laboratory and Northwestern University have developed a groundbreaking new software tool called EXSCLAIM!. This tool – which stands for extraction, separation, and caption-based natural language annotation for images – promises to revolutionize how researchers access and utilize the vast troves of visual data buried within the scientific literature.
EXSCLAIM!’s success has to do with its innovative “query-to-dataset” approach inspired by generative AI tools like ChatGPT and DALL-E. By processing both the images and surrounding text from figure captions, the software pulls images containing specific visual content and creates descriptive labels using the natural language from the caption.
“While existing methods often struggle with the compound layout problem, EXSCLAIM! employs a new approach to overcome this,” lead author Eric Schwenker, a former Argonne graduate student, was quoted in an Argonne article about the project. “Our software is effective at identifying sharp image boundaries, and it excels in capturing irregular image arrangements.”
EXSCLAIM! has already proven its worth by constructing a self-labeled dataset of over 280,000 nanostructure images from electron microscopy literature. While initially focused on materials science, this amazing tool is designed to be adaptable across any scientific field dealing with large amounts of published image data.
Discussed in length in the official research paperEXSCLAIM! establishes a scalable pipeline for curating meaningful image and language information from scientific publications. It combines rule-based natural language processing techniques with image recognition to automatically extract images from figures, separate them into individual images, and annotate them with relevant keywords from the caption text.
The researchers highlight that while rule-based caption processing allows for modeling specific syntax patterns, it struggles to generalize across the wide variety of caption styles found in the literature. To address this, they plan to incorporate transformer-based natural language processing (NLP) models, which have proven effective at generalizing across diverse contexts.
As the explosion of scientific imaging data continues, tools like EXSCLAIM! will play a vital role in enabling researchers to effectively navigate, search, and analyze the vast amounts of visual information locked within the various literature. By bridging the gap between images and language, this incredible software opens new frontiers for accelerating scientific discovery through advanced computer vision and multi-modal learning techniques.
Related
