What is a context graph?

A context graph is a way of structuring discrete information in the context window of a large language model (LLM), where “nodes” represent pieces of information and “edges” represent the relationships between them. These graphs represent the knowledge that an LLM will have access to when making an inference and they are a form of knowledge graph.

The context window is what an LLM uses to generate text in response to a user query or tool call. Commonly, this context is unstructured (plain text, files or images) and the relationships between the different pieces of information are not explicitly represented.

A context graph explicitly models those relationships, turning the context window from a simple a prompt string into a first-class system of record where entities, documents, facts, user intents and their relationships are structured and connected.

Context graphs can be traversed, filtered and composed of multiple sub-graphs. They also encode information about relationships (in information science these are called ontologies). For instance, an organization has a name and contains employees who have names and job titles. Explicitly modeling these relationships in a structured way helps rigorously configure the logic that an LLM uses to generate responses.

A context graph gives generative artificial intelligence (GenAI) systems—and the AI models and AI agents within them—well-structured data that reduces hallucinations and preserves its full context.

Without structure, determining what should go into the LLM context is shallow, in that it simply relies on top-k retrieval, and brittle, in that it misses indirect relationships. With a context graph, an LLM can be supported in reasoning across sources with representations of the reliability and relevance of those sources to maintain an evolving knowledgebase.
In practice, this means that a context graph not only represents information retrieved from a vector search or CRM, it can also provide context about that information. 

Using a newspaper article for example, the context graph can provide the LLM with information on how old an article is, who uploaded it to a system, or how individuals mentioned in a newspaper story are related to one another. This structure gives LLMs a wealth of information to use when generating responses alongside better observability and explainability to the end user of that generated response.

This capability is especially important in real-world enterprise AI systems, in determining high-stakes pricing, or in healthcare where model outputs are used for medical decision-making. 

A context graph is one element of broader context engineering practice. In context engineering, the goal is to control what information an LLM receives—and how it is structured—through techniques such as reordering, summarization or compression. A context graph is a powerful tool because it can both dramatically reduce the amount of information that the LLM intakes at inference and give that information additional structure and meaning.

The context graph itself is not typically what the model directly consumes. Instead, it is a pre-processing and orchestration layer that determines what the model should see at inference time.

At a practical level, a context graph consists of these components:

·      Nodes: discrete unites of information such as documents, chunks of text information, entities extracted from prompts or the results of tool calls, or past interactions

·      Edges: relationships between nodes, such as“references” (a document references an entity), or “is similar to” (one entity is similar to another).

·      Attributes: metadata about nodes or edges such as timestamps, embeddings, confidence scores, provenance or relevance. These help a system sort or prioritize one node or edge over another.

These components are closely related to ideas from graph theory and knowledge graphs but applied here to dynamic context assembly for AI systems.

In a standard LLM workflow, context is a linear sequence of tokens placed into a prompt. With a context graph, context is a subgraph which is selected and assembled into a prompt.

Rather than simply retrieving top-k similar chunks, a context graph enables a data scientist to retrieve nodes, expand additional information via edges and even rank subgraphs contained within a larger graph. The selected subgraph can then be converted into a linear prompt, which will represent that structure.

For example, relationships like “Document A supports B” or “Person C is part of Organization D” can be explicitly represented. The order of information can be highly meaningful, and a graph will represent that.

Large language models like GPT don’t inherently understand relationships beyond what fits into their context window. A context graph can improve LLM results in several key respects:

-              Retrieval quality

-              Multi-step reasoning

-              Coherence

-              Noise reduction

-              Control and transparency

Retrieval quality

Traditional retrieval approaches often depend on embedding similarity alone, which can surface text that is lexically or semantically similar but not actually useful for the task. A context graph adds structure by incorporating relationships, so the system can retrieve not only what is similar to a query but also what is logically or contextually connected. This leads to more relevant and complete inputs.

Multi-step reasoning

Many questions require connecting information that is not directly similar to the query but is linked through intermediate concepts. A context graph enables the system to follow these connections, effectively performing multi-hop retrieval. This allows the model to produce answers that reflect deeper reasoning rather than shallow pattern matching.

Coherence

When context is assembled from a graph, the selected pieces of information are more likely to be related to each other via the graph rather than solely related to the query. As a result, the model’s output can be more consistent and logically structured.

By representing prior interactions, user preferences and evolving topics as part of the graph, the system can build context that persists across sessions. This capability allows for more personalized and context-aware responses without overwhelming the model with raw conversation history.

When relationships such as causality, dependency or contradiction are encoded and then reflected in the prompt, the model is better guided in how to interpret the information. This reduces ambiguity and helps the model align its reasoning with the intended structure.

Noise reduction

A context graph also helps reduce noise. In purely similarity-based systems, irrelevant but superficially similar content can crowd out important information. Graph-based filtering can prioritize nodes that are central, well-connected or supported by multiple sources. This leads to cleaner prompts and more reliable outputs.

Control and transparency

Context is assembled from a structured system, it becomes easier to trace why certain information was included and how different pieces are connected. This improves transparency and makes it easier to debug, audit and refine the system. Over time, these feedback loops  can indirectly improves the quality of model outputs.

Building context graphs into an AI system requires careful design to create a responsive and informative architecture. The goal is to turn raw data and interactions into structured, queryable context that can be assembled into high-quality prompts. Typically, this looks like a pipeline with the following distinct, but highly integrated, stages:

-              Data ingestion and preprocessing

-              Dual storage (vector index and graph store)

-              Retrieval

-              Graph traversal

-              Ranking

-              Prompt construction

-              Prompt execution

Data ingestion and preprocessing

In this first step, sources such as documents, application programming interfaces (APIs) and user inputs are collected and broken into manageable units such as passages or semantic chunks. Each chunk is then enriched.

Enrichment includes generating embeddings, extracting entities and optionally producing summaries. Embeddings provide a fast, flexible way to locate relevant information in a high-dimensional space. The embeddings are produced using a model that maps text into high-dimensional vectors so that semantic similarity can be computed efficiently.

Dual storage (vector index and graph store)

Next comes dual storage: a vector index and a graph store. This approach is called graph retrieval augmented generation, or GraphRAG.

A vector index store has optimizations specifically to enable similarity search. Each chunk’s embedding is stored so that, given a query embedding, you can quickly retrieve semantically similar chunks.

In parallel, a graph structure is built where each chunk, entity or concept becomes a node. Edges are created based on relationships such as co-occurrence, citation, shared entities, temporal sequence or explicit links. This graph is conceptually similar to systems used in knowledge graphs, but it is often more dynamic and application-specific.

The key design decision is that the graph is not just a static representation of facts. Rather, it is a representation of current state, continuously updated as new data arrives and as users interact with the system.

For example, if two pieces of information are frequently retrieved together, an edge can be strengthened or created between them. If a user session introduces a new concept, it can be added as a node and linked to existing nodes. This turns the graph into a living representation of context.

Retrieval

When a user submits a query, the retrieval process begins. One strategy to do this turns queries into embeddings which are then used to perform a similarity search in the vector index. This produces an initial set of candidate nodes. At this stage, a traditional retrieval system might stop and pass these directly to the model. With a context graph however, the graph is used to expand and refine the result set.

Graph traversal

Graph traversal is then applied to the initial candidates. For each retrieved node, the system explores neighboring nodes based on edge types and weights.

For example, it might include nodes that are strongly connected through shared entities or that represent prerequisite concepts. This allows the system to perform multi-hop retrieval, where relevant context is not just directly similar to the query but also indirectly related through meaningful connections.

A graph database like Neo4j using a query language like the Neo4j Cypher query language can make this step better conform to graph concepts.

Ranking

After traversal, a ranking step occurs. The system scores nodes based on a combination of metrics such as embedding similarity, graph centrality, edge weights, recency and source reliability. The result is a selected subgraph that represents the most relevant and coherent context for the query.

Prompt construction

Prompt construction bridges the gap between structured data and the linear input format required by the model. Because a model like GPT expects a linear sequence of tokens, the selected subgraph must be transformed into a structured prompt.

This involves ordering the nodes, summarizing or compressing content if necessary, and explicitly encoding relationships when they matter. For instance, the prompt might include statements that clarify that one document supports or contradicts another, or that one concept depends on another. This step is crucial because it translates graph structure into a form the model can interpret.

Prompt execution

Finally, the constructed prompt is sent to the language model. The model generates a response based on the curated context. Optionally, the system can feed the response back into the graph.

For example, it can extract new entities, create new nodes or update relationships based on what was generated or how the user reacts. This closes the loop and enables continuous improvement.

This architecture overcomes the limitations of similarity-based retrieval (such as cosine-similarity) by incorporating structure, history and reasoning paths into the context that is ultimately presented to the model.

To help illustrate how a context graph can work in a real-world setting, imagine a physician evaluating a patient admitted with chest pain. The physician asks an agentic system: “What are the most likely causes of this patient’s chest pain given their history and current lab results?”

In a basic retrieval setup, the system embeds the query and retrieves relevant documents about chest pain, heart attack and gastrointestinal causes. These documents are concatenated and sent to a model such like OpenAI’s GPT or IBM Granite. The model produces an answer but may miss important connections or include irrelevant conditions.

With a context graph, the system behaves differently because it has already structured medical knowledge and patient data into interconnected nodes. The graph could contain nodes that represent symptoms (“chest pain” and “shortness of breath”), conditions (“myocardial infarction” and “reflux disease”) and patient specific information from charts and tests (age, blood pressure and clinical findings). These nodes are linked through relationships such as “symptom of X,” “risk factor for Y” or “commonly co-occurs with Z.”

When the physician enters their query, the agent would identify key nodes from patient data, such as chest pain and elevated troponin levels. These nodes become entry points into the graph. Instead of simply retrieving similar text as in a RAG system, the agent would traverse the graph.

Graph traversal should indicate that chest pain and elevated troponin strongly links to myocardial infarction. It could also see edges connecting chest pain to pulmonary embolism or acid reflux, but those paths would be weaker. This kind of specific and rigorous weighting means that the agent is less likely to be influenced by irrelevant diagnoses.

The resulting subgraph includes likely diagnoses, supporting evidence and relevant guidelines, which is then transformed into a structured prompt.

Context graphs help the agent reflect clinical reasoning rather than a generic summary of information. By using relationships between symptoms, lab results, and conditions, the system prioritizes diagnoses that are supported by the patient’s data rather than those that are merely textually similar to the query. The output should prioritize myocardial infarction, explain the significance of troponin, and suggest immediate actions such as further cardiac evaluation.

The graph also provides traceability because the context is assembled symptoms and findings represented in a graph. That graph can be audited later if the clinician wants to understand model output more deeply or to help improve the underlying data. As more patient cases and outcomes are added, the graph can strengthen or adjust relationships and thus improve future recommendations.