What MM-KG is actually doing

MM-KG (Multimodal Knowledge Graph) is a representation and retrieval layer that sits between raw clinical data and downstream models. It is designed for clinical questions where you need to connect a specific patient’s multimodal evidence with external biomedical knowledge 1.

Instead of shoving everything into an LLM context window as text, MM-KG:

  • Represents patient observations (from EHR text, imaging, genomic, and biospecimen data) as one layer in a typed graph.
  • Represents biomedical concepts as a separate layer in the same graph.
  • Connects these layers via explicit alignment edges.
  • Performs query-conditioned retrieval to extract a compact subgraph per question for use by a large language model (LLM) or a graph neural network (GNN) 1.

The core idea is that the link between “this particular patient’s evidence” and “this general biomedical relation” is a first-class, inspectable, deletable object in your system, not just something implicit in embeddings or text concatenation.

MM-KG is instantiated over two real clinical datasets, MIMIC-IV and ADNI, and used to answer clinical questions that require both patient data and biomedical knowledge 1.

Step-by-step: from multimodal evidence to answer-bearing subgraph

The MM-KG pipeline has three main stages 1:

  1. Harmonize multimodal patient evidence.
  2. Align the harmonized observations to a biomedical knowledge graph.
  3. Retrieve a query-conditioned subgraph for downstream models.

Each stage is where you’d wire in orchestration and control flow in a deployed agentic system.

1. Modality-specific harmonizers

First, MM-KG uses modality-specific harmonizers to convert heterogeneous clinical data into a unified set of typed observations 1. The modalities explicitly handled are:

  • EHR text
  • Imaging
  • Genomic data
  • Biospecimen data 1

Each harmonizer:

  • Takes raw data in its modality.
  • Produces typed observations.
  • Maps these observations to UMLS concepts 1.

So at this point you have a graph layer whose nodes and/or edges are standardized observations, each associated with UMLS concepts. The type system and mapping to UMLS are what let you connect this patient-specific graph to a general biomedical knowledge layer.

For an agent stack, this stage is effectively the adapter set: four different modality-specific tools all normalizing into one schema keyed by UMLS concept identifiers.

2. Route-prioritized alignment to a biomedical KG

Next, MM-KG uses a route-prioritized aligner to link the harmonized observations to a biomedical knowledge graph 1.

Mechanically, this means:

  • The patient observation layer and the biomedical knowledge layer remain separate.
  • Alignment happens through explicit edges that say “this observation is aligned with that concept/relation in the biomedical KG” 1.

“Route-prioritized” implies that the aligner is not just greedily linking everything possible; it is prioritizing alignment routes that are useful for downstream reasoning. The specifics of how routes are prioritized are not detailed in the source, but the result is a unified, typed, two-layer graph with:

  • Patient observation nodes.
  • Biomedical concept nodes.
  • Alignment edges that explicitly encode which concepts are grounded in which patient evidence 1.

For builders, this is key: alignment is not the same thing as retrieval. MM-KG precomputes or maintains a graph where evidence-to-knowledge links are already made and can be traced or ablated later.

3. Query-conditioned retrieval

Once the MM-KG is built for a patient, it still might be large. MM-KG then uses query-conditioned retrieval to select a compact subgraph relevant to a specific clinical question 1.

This retrieval stage takes as input:

  • A question/query.
  • The full MM-KG for a patient.

And returns:

  • A small subgraph (“packet”) tailored to that question, which is then passed to either:
  • A large language model, or
  • A graph neural network 1.

This is a different control pattern than vanilla retrieval-augmented generation (RAG). Instead of retrieving independent documents or chunks, you retrieve connected structures that already encode how patient evidence and biomedical knowledge relate to one another.

The paper also compares different retrieval strategies, showing that query-conditioned retrieval can reach 0.731 AUROC with 6.8× less context than the strongest generic policy 1. In contrast, static knowledge graph context—i.e., knowledge dropped in without tailoring to the query—“gives no consistent gain on ordinary outcome prediction” 1.

Operationally, this means that blindly including a big slice of a biomedical KG in your model’s prompt is mostly wasted tokens for outcome prediction tasks, whereas targeted subgraph retrieval actually changes outcomes.

How this fits in a clinical LLM stack

Putting the pieces together, a MM-KG-style stack for a clinical assistant grounded in multimodal data looks like:

  1. Data ingestion & harmonization
  • EHR text, imaging, genomic, biospecimen → modality-specific harmonizers.
  • Harmonizers output typed observations mapped to UMLS concepts 1.
  1. Graph construction
  • Build a typed graph layer of patient observations.
  • Attach a biomedical knowledge graph as a second layer.
  • Use a route-prioritized aligner to add explicit alignment edges between layers 1.
  1. Question-time orchestration
  • Receive a clinical query.
  • Run query-conditioned retrieval over MM-KG to select a compact subgraph 1.
  1. Downstream reasoning
  • Feed the subgraph to:
  • A large language model (e.g., via text serialization of nodes/edges), or
  • A graph neural network 1.
  1. Attribution & intervention
  • Because patient-evidence nodes, biomedical-concept nodes, and alignment edges are explicit and separated, you can:
  • Trace which edges and nodes support a particular answer.
  • Delete specific relations and measure their effect 1.

This contrasts with traditional predictive systems that “rarely represent such links explicitly, so they can neither be traced to their evidence sources nor removed to measure their contributions” 1.

Why the interaction matters (and how they isolate it)

MM-KG’s evaluation design is explicitly set up to separate:

  • The effect of patient evidence alone.
  • The effect of biomedical knowledge alone.
  • The interaction between the two 1.

They do this with a 2×2 design that controls access to patient evidence and biomedical knowledge 1. For questions that require both sources, the results are:

  • Neither source alone performs far above chance 1.
  • Their combination yields a drug-controlled AUROC interaction of:
  • +0.194 on MIMIC, and
  • +0.299 on ADNI 1.

So the improvement is not simply “more data is better.” It is specifically about correctly wiring patient evidence into biomedical relations.

For production design, the implication from the paper’s results is:

  • Adding knowledge without aligning it to patient evidence (e.g., dumping static KG context) does not reliably help.
  • Correctly modeling the interaction—via explicit alignment edges and query-conditioned retrieval—is what moves the needle.

Localizing answers to individual relations

One strong property of MM-KG is that the system can be ablated at the level of “the single relation that actually carries the answer”:

  • On a held-out five-candidate ranking task, MM-KG:
  • Outperforms MindMap by +0.131 Hits@1 1.
  • Leads an adapted GraphCare on items that require consulting the patient 1.

Then, critically:

  • Deleting the single answer-bearing relation from the retrieved packet “returns Hits@1 to the no-knowledge baseline” 1.

That means:

  • The answer is tightly localized in the graph.
  • The system is sensitive to that single edge.
  • You can turn off the contribution of a particular biomedical relation and watch performance degrade back to where it would have been without knowledge.

For agentic systems, this gives you a concrete unit of attribution and control:

  • If you need to justify an answer, you can point to the specific relation and its aligned patient evidence.
  • If you need to test robustness or remove suspect knowledge, you can delete or modify that relation and re-run.

Context budget and routing policies

On the context side, MM-KG’s query-conditioned retrieval is explicitly compared to a “strongest generic policy” 1:

  • Query-conditioned retrieval achieves 0.731 AUROC.
  • It uses 6.8× less context than that strongest generic policy 1.

At the same time:

  • Static knowledge graph context—i.e., context that is not query-conditioned—“gives no consistent gain on ordinary outcome prediction” 1.

For builders, this is direct evidence against “just stuff more KG into the prompt”:

  • The routing policy for which parts of the KG you expose per query is crucial.
  • A naive top-k or static stub of the KG is not sufficient to harvest the value of the knowledge graph.

Instead, you want something shaped like MM-KG’s query-conditioned routing over a structured graph.

Why this matters for clinical agents

The paper’s overall conclusion is that:

“Knowledge graphs thus benefit clinical LLMs not as background context but as explicit links between multimodal patient evidence and the relation a question requires, and MM-KG makes these links retrievable, traceable, and testable.” 1

The key operational points for agentic systems are:

  • Representation: Separate patient observations and biomedical concepts into layered, typed graph structures 1.
  • Alignment: Use explicit alignment edges, produced by modality-specific harmonizers and a route-prioritized aligner, to connect multimodal evidence to knowledge 1.
  • Routing: Use query-conditioned retrieval to build compact, connected subgraphs per question, rather than static or generic KG context 1.
  • Attribution and testing: Make answer-bearing relations a first-class object you can identify and delete, revealing their causal role in model performance 1.

This shifts where you put engineering effort: less in scaling static context, more in designing alignment and routing over explicit graph structures that encode the interaction between a specific patient and general biomedical knowledge.

What is not documented

The sources do not document:

  • The internal schema of the typed graph, beyond the existence of separate patient and biomedical layers and explicit alignment edges.
  • The specific algorithms or architectures used for modality-specific harmonizers, the route-prioritized aligner, or query-conditioned retrieval.
  • The exact form of the biomedical knowledge graph, beyond that it is aligned via UMLS concepts.
  • The details of how the compact subgraphs are serialized or fed into large language models or graph neural networks.
  • Any system-level metrics such as latency, memory footprint, or scalability characteristics of MM-KG in deployment.