Mendel’s New Neuro-Symbolic AI Outperforms GPT-4 in Cohort Retrieval

Mendel, a leader in Clinical AI, today announced the results of its latest research on Neuro-Symbolic AI where Mendel’s Clinical AI system can automate the identification of patient cohorts from unstructured and structured EMRs, outperforming GPT-4 in several benchmarks. Mendel’s unique clinical AI approach couples large language models (LLMs) with its proprietary hypergraph reasoning engine. The research unveiled by Mendel showed how it is able to power significant advancements in Automatic Cohort Retrieval (ACR), a fundamental task for clinical research and patient care. This research findings can be read in full here.

Transforming cohort retrieval

Identifying patient cohorts is essential for clinical trials, retrospective studies, and other healthcare applications. Traditional methods relying on automated queries of structured data combined with manual curation are time-consuming and often yield low-quality results. Mendel’s AI offerings utilize a unique approach that couples a world-class clinical LLM trained to understand structured and unstructured text with a proprietary reasoning engine infused with medical knowledge reviewed by medical professionals to apply a clinician’s mind to complex and varied medical situations. This ability to apply clinical reasoning to ACR has been demonstrated to offer significant improvements over existing Retrieval-Augmented Generation (RAG) and LLM techniques.

“Our latest research at Mendel marks a significant milestone in the field of AI in general, and healthcare in particular,” said Wael Salloum, Cofounder and Chief Science Officer at Mendel. “We are the leader in clinical reasoning by coupling LLMs with our hypergraph reasoning, enhancing both the effectiveness and efficiency of patient cohort retrieval. This work is critical in paving the way for more robust and scalable clinical reasoning. This breakthrough underscores our commitment to advance the AI field to transform clinical research and improve patient outcomes.”

Key findings of the study include:

This research introduces two types of reasoning to the AI field:

1. Longitudinal Reasoning: Mendel’s neuro-symbolic architecture outperformed pure LLM approaches by efficiently handling the longitudinal nature of unstructured Electronic Medical Records (EMRs). As a patient’s record unfolds over time, the system reasons over the emerging facts, contrasting, rejecting, and consolidating them into a symbolic patient journey. Unlike LLM-only approaches, this approach processes a patient’s EMR just once, offline, to construct a journey that can be queried repeatedly at minimal cost.
2. Large-Scale Reasoning: Mendel’s integration of real-time hypergraph reasoning and a clinical LLM achieved higher Precision and Recall in cohort retrieval tasks. Unlike LLM-only solutions, which process the entire patient database for each query—making them infeasible for healthcare applications—Mendel’s approach maintains a fixed cost per query, regardless of the database size.

Benchmark and evaluation

Mendel’s research introduces a new benchmark task for ACR, featuring a comprehensive query dataset and an evaluation framework. The study compared the performance of Retrieval Augmented Generation (RAG) and LLM-based solutions and Mendel’s neuro-symbolic systems, providing a detailed analysis of their effectiveness and efficiency.

In the evaluation, Mendel had a 1.4K patient data set, and Mendel evaluated several embeddings and found Ada outperformed others. The evaluation report compares Ada with GPT-4 (RAG) to Mendel’s Neuro-symbolic System, Hypercube. F1 score is the key metric used to evaluate the accuracy of models, balancing both precision (how many of the results are relevant) and recall (how many relevant results were identified). This score provides a comprehensive measure of the model’s performance.

Below are the sample results of F1 scores:

1. Medium complexity with most queries (52)
2. Broad cohort size
3. Document count is greater than 74

Future implications

The findings underscore the transformative potential of Mendel’s Neuro-Symbolic AI system by combining LLMs with domain-specific knowledge embedded in hypergraphs. This approach enhances the accuracy and efficiency of cohort retrieval, facilitating more precise patient stratification and targeted interventions for new therapies. It also paves the way for broader applications in clinical research and patient care.

Mendel is offering a free “TryMe” demo to showcase the company’s Neuro-Symbolic AI system.

Explore AITechPark for the latest advancements in AI, IOT, Cybersecurity, AITech News, and insightful updates from industry experts!