Building KnowledgeSeeker - A RAG application platform for societal good

One is to ask a research question, and get the generated answer along with the references. 
The user can select which sources to use in answering their question. Sources can either be their own uploaded research or utilising research archives such as PubMed, ClinicalTrials.gov or Arxiv. 

The other major feature is the Lookouts. Here the user can specify their research area of interest as well as specifying how long time range the lookout should take into account in searching and ranking relevant research. Apart from that, the user can also specify locations on where the research was conducted. When a lookout is created, relevant sources from ClinicalTrials.gov are fetched and ranked, where the top number of results specified by the user are shown in the interface. The user can read these results and their AI generated summary, as well as explore the original research source at hand.

How did we create it? Our project development

After two weeks of coding, we developed our first prototype. 

This version allowed users to upload multiple PDFs and query them via a search bar. The system generates responses using Retrieval-Augmented Generation (RAG), leveraging various open-source large language models and embedding models. To find relevant articles, we utilized similarity search within a Postgres vector database that stores all the embeddings. We built the application using the Streamlit framework.

Our prototype successfully answered questions on topics explicitly mentioned in the documents. Additionally, we experimented with different embedding models to compare their performance. Since our queries were in Swedish while the papers were in English, we noticed that language often influenced the results more than context. For instance, text chunks containing Swedish words were automatically prioritized. Moreover, when generating responses, the answers were frequently returned in English.

To simplify integration with various LLMs, we decided to use the command R large language model and its corresponding embedding model from Cohere. For the new functionality, we revamped our framework using React Next.js for the frontend, FastAPI for the backend, and maintained the Postgres database. We introduced the ability to automatically retrieve new articles from research archives like arXiv, PubMed, and ClinicalTrials, eliminating the need for users to manually find and upload research articles beforehand. Through APIs, we extract abstracts and summaries, which are then used as input for the LLM instead of processing entire papers.

To prevent irrelevant text snippets—such as references or side notes—from being selected, we developed a custom algorithm using Unstructured and Spacy to filter out unnecessary information. This classifier achieved over 99% accuracy on test data. To further enhance the quality of the selected text chunks, we used LLMs to extract meaningful content and re-rank the chunks based on their relevance to the query. To address language-related issues, we decided to work exclusively with English texts for this functionality.

Researchers provided highly positive feedback on our application, noting its accuracy, relevance, and potential utility.

• Accurate and Insightful Responses: Lisa Wendby from Mind appreciated the tool’s ability to identify research gaps, calling it "helpful for understanding where research is lacking."
• Valuable for Researchers: Researchers found the application "very valuable," especially for its transparency in showing which articles responses are based on. The researchers noted it could be beneficial for both researchers and staff at Mind.
• Handling Complex Queries: The researchers were impressed with how the application addressed more abstract questions, offering insightful responses on complex topics like familial transmission of suicide risk.
• Performance and Potential: Researchers observed that while the application’s performance varied with uploaded articles, it showed significant promise, particularly when integrated with PubMed. One researcher praised the application for its "very promising" potential and appreciated that responses are based on identifiable references.

In the final version of our application, we introduced a feature called "Lookout." 

This functionality allows users to create and maintain lists of relevant research articles on specific topics, which are automatically updated whenever new studies are published. Integrated with Zapier and Google Cloud Scheduler, "Lookout" can send email updates to subscribers, including summaries of the latest research. Users can also filter these updates by specific time windows, such as the last month or the last three months, and by regions like Sweden or Europe.

Lookouts:

This enhancement with lookout further strengthens the application’s utility, providing researchers and professionals with timely, targeted updates on their areas of interest.

Further improvements could include the design being enhanced, additional archives like Google Scholar being integrated, and sources such as Wikipedia being incorporated. Multilingual support, including Nordic languages, could also be added, and the services might be extended to support more organizations.

As we’re ending our project within the talent program AI for Impact - we’re really proud of the application we’ve built. We’re happy that what started as a brief for an application aimed for Mind and other non-profits to use to find the latest research, also resulted in an additional use case for the broader research community to find research and identify research gaps.

See more from our project description on MyAI 📄: https://lnkd.in/dNCYRzRr