Graph Engine internals

#Module boundaries

Graph Engine is the representation and computation layer of graph-driven analysis. The agent requests specific evidence; Graph Engine accesses the active graph or specified dataset according to tool arguments, executes queries, algorithms, or learning tasks, and returns identifiable objects, values, structures, and state. Its construction subagent prepares or optimizes graph schemas when needed; reading, statistics, and learning on existing graphs do not require reconstruction for each analysis.

Graph Engine execution pathOpen diagram ↗Open diagram ↗

Graph Engine exposes HTTP and MCP adapters. Backend uses HTTP for data and jobs; the agent invokes graph tools through MCP. graph_engine is the top-level entry point, graph_data_service implements HTTP, graph state, and computation, and mcp_service adapts the tool protocol. They share tool specifications and resource read/write policies.

HTTP tool forwarding uses POST /sessions/{session_id}/tools/{tool_name} with {"arguments": {...}}. GraphDataClient injects user, session, run, generation, and operation identity. This is an internal service entrypoint; browsers use the Backend public API. Construction tools have additional dataset-scoped adaptation.

#How graph results become analysis evidence

Graph results need both their values or objects and the context required to interpret them. Node names, relationship directions, weight semantics, graph version, and returned scope jointly determine what a result supports. HTTP or MCP transport success is not business success; success flags, errors, truncation, and terminal job states all inform subsequent analysis.

Versioning and resource scheduling keep related operations aligned with the correct input, but cannot remove limitations caused by missing data or an unsuitable method. A failed bounded path search cannot prove global unreachability, communities do not establish business classes, and link-prediction scores do not establish causation. The following sections detail these data and algorithm contracts.

create_analysis_report returns an analysis_report.v2 structure for the interface; it does not automatically create a Markdown or PDF file. For a downloadable file, Agent Runtime workspace tools generate content under Ephemeral and publish it to Derived with workspace_publish(path=..., name=...), explicitly including the extension in name. Workspace tools are separate from the 72 default Graph Engine tools.

#Graph representation and relationship identity

GraphData is a read-only graph container. Computation uses contiguous integer node indices, with _id2node / _node2id mapping business names to indices and back. The two rows of edge_index store source and destination indices. Node/edge features, types, and labels use separate arrays. Mutations publish a new graph object or revision instead of modifying a graph under analysis.

Property graphs may contain multiple relationships between the same endpoints. An endpoint pair identifies candidate relationships, not a unique edge_id. Use returned IDs when selecting or editing a relationship. Generic append returns UNSUPPORTED_PROPERTY_GRAPH_APPEND for constructed property graphs to avoid collapsing relationship identities and properties into a plain edge list.

GraphData converts edge_index to int64 and requires two rows. Name/index mappings must be provided together and agree. Features use float32 and training/validation/test masks use boolean arrays. Row i of node_feat, item i of node_label, and id2node[i] refer to the same node and must not be reordered independently. Edge attributes must also align with edge_index columns.

import numpy as np
from graph_data_service.graph_data import GraphData

# Three named nodes and three directed edges.
graph = GraphData(
    num_nodes=3,
    edge_index=np.array([[0, 0, 1], [1, 2, 2]], dtype=np.int64),
    id2node={0: "customer:10", 1: "order:100", 2: "region:1"},
    node2id={"customer:10": 0, "order:100": 1, "region:1": 2},
    edge_weight=np.array([2.0, 5.0, 1.0]),
    semantic_type="directed",
)
# Columns represent customer:10 -> order:100,
# customer:10 -> region:1, and order:100 -> region:1.

This example illustrates array/name alignment, not the default direction of commerce foreign keys. Source-table construction determines row nodes and FK direction in the construction module; manually imported graphs require the caller to define edge semantics.

Business properties and model features are stored separately. Order amounts and status strings in properties.data do not automatically become node_feat. Single-graph learning without node_feat uses normalized [in-degree, out-degree, total-degree, 1] features; this does not mean the model reads business fields. Graph classification instead uses node-category IDs in its dataset format. graph_metadata.construction_graph preserves the original property graph, including properties and relationship identities, through conversions, version switches and queries.

#Graph direction and edge storage

Direction is a semantic property, not just adjacency symmetry. Two accounts following each other form two directed relationships, while one undirected relationship may also use two stored records. Graph Engine separates semantic_type from the physical edge organization described by storage_type.

Direction resolution considers explicit semantic_type, metadata, predefined dataset rules, and structural evidence in that order. A symmetric structure can remain unknown without other information. Cora, CiteSeer, and PubMed have predefined undirected semantics. to_undirected creates a new graph and may add reverse array records; logical edge counts and node degrees account for these storage copies.

edge_index = [[0, 1], [1, 0]]

semantic_type = "directed"   -> two directed edges
semantic_type = "undirected" -> one logical undirected edge
no semantic evidence        -> symmetry alone is ambiguous

Degree and neighbor count also differ. In an undirected graph, a self-loop contributes two to degree but identifies only one distinct neighbor. Directed graphs report incoming, outgoing, and total degree. Several relationships may connect the same neighbor, so unique-neighbor counts cannot replace relationship counts.

#Graph versions and caches

Loading, undirected conversion, subgraph extraction, and undo change the active graph version. get_graph_info reports graph_version, list_graph_versions lists versions, and load_graph_version selects a saved version. Constructed datasets have a separate revision; load_constructed_graph explicitly makes a constructed result the session’s active graph.

NetworkX representations and entity-search indexes are derived data. Version-keyed reuse avoids repeated conversion by tools; a changed version must use its corresponding representation. Rebuilding a cache is a graph read and must not publish a new revision. The first conversion still incurs a cost proportional to graph size.

Versions are isolated per session, with graph data in NPZ files and a graph_versions.json manifest. Ordinary save_snapshot calls prune history using GRAPH_VERSION_HISTORY_LIMIT, which defaults to 50; controlled publish_snapshot calls currently do not apply that pruning. New version numbers remain monotonic after switching to an older revision. Isolated workers write staging data; the Graph Engine parent process checks generation and graph_version before atomically publishing the manifest and recording a run/job/generation receipt, preventing stale executions from overwriting newer state.

Graph objects, algorithm conversions, and search indexes are version-bound. GraphCompute caches the NetworkX representation and EntitySearchIndex retains its creation graph_version, rebuilding when versions differ. Creating a read cache does not increment graph_version. Read the actual version before recording an experiment or comparing results across runs.

Construction revision, session graph_version, and execution_generation serve different purposes. revision protects dataset edits from stale overwrites; graph_version selects algorithm input; execution_generation prevents a stopped, older run from writing results. One identifier cannot substitute for the other checks.

#Parallel-edge semantics

Weights must be finite numeric values; booleans and nonnumeric values are invalid. Their meaning depends on the algorithm: path cost, flow/cut capacity, or strength for communities and some centralities. Establish the business interpretation before calling a tool.

NetworkX conversion retains simple graph representations for ordinary graphs. Constructed property graphs use MultiGraph or MultiDiGraph when endpoints have distinct relationship identifiers. Reverse storage copies of the same undirected relationship are deduplicated. Business edge_id remains in attributes while NetworkX multiedge keys use edge indices; the two identifiers are not interchangeable.

Suppose A→B has parallel edges weighted 2 and 5. Shortest paths interpret them as alternative costs and use 2 after projection. Maximum flow interprets them as parallel capacity and uses 7. Both follow their algorithm definitions. Establish what the weight means before choosing a tool; a transaction amount is not automatically a distance.

Methods without a declared multigraph policy return UNSUPPORTED_MULTIGRAPH_OPERATION. Parameter choices also matter: only some community methods are approved, while ego_graph and node_subset preserve relationships and other subgraph methods have separate restrictions. Validation runs before computation to prevent silent relationship merging.

#Tool dependencies and concurrency

Each tool declares reads, writes, barrier, and background. Resources are qualified by identity: graph, models, and documents are generally session-scoped, while datasets and construction:{dataset_id} are user/dataset-scoped. RAW, WAR, and WAW conflicts create dependency edges. Independent reads can overlap; unknown tools are conservative barriers.

load_constructed_graph (writes graph)
  ├─ node_centrality       (reads graph)
  ├─ community_detection  (reads graph)
  └─ shortest_path        (reads graph)
       ↓ all finished
load_graph_version        (writes graph)

subgraph_extraction writes a new version by default and is read-only only with create_version=false. Failed or approval-blocked predecessors cause TOOL_DEPENDENCY_BLOCKED. The model must obtain real results before submitting dependent IDs or revisions; scheduling does not fabricate arguments.

Agent-side tool_dag schedules calls within a model turn; Graph Engine’s tool_scheduler controls admission across requests. Background jobs retain resource leases until execution actually finishes, not merely until job_id is returned. Status checks and cancellation avoid waiting on graph write leases so tasks remain controllable.

Cross-request scheduling preserves arrival order among conflicting requests. If read R1 is running and write W2 is queued, a new read R3 cannot bypass W2; otherwise continuous readers could starve the writer. Different users do not conflict, but sessions for the same user can share datasets or construction:{dataset_id}.

Resource dependencies and compute quotas are separate controls. Dependencies prevent reading changing data; semaphores limit simultaneous ordinary or heavy computation. Two training jobs that only read the same graph may still serialize because of GPU use, random-number state, or per-user quotas.

#Workers and background jobs

The compatibility path for NetworkX algorithms uses a bounded ProcessPool, while other synchronous handlers may use threads. With an execution host and run identity, controlled execution uses terminable isolated WorkerProcess instances with stop confirmation and persisted terminal state. Cancelling observation does not stop computation; capacity and leases are reclaimed when execution actually exits.

get_job_status returns status and progress, with ordinary completed output under job.result; get_job_result also retrieves results. GraphMCPTools waits for final output from graph-tool job handles, supports nested jobs, and rejects cycles or missing output. Supervised infer loads a trained model and runs a forward pass; pretraining infer queries saved node vectors. Graph-classification dataset binding is separate from active single-graph binding.

{
  "success": true,
  "job": {
    "job_id": "JOB_ID",
    "status": "done",
    "progress": 1,
    "result": {
      "success": true
    }
  },
  "progress_updates": []
}

This shape illustrates get_job_status field placement: final business output lives in job.result. Actual status values and result fields depend on the job; after_seq supports incremental progress polling.

#Entity search and indexing

search_node and search_edge resolve descriptions into identifiers for graph tools. Node indexes contain names, types, labels, degrees, and neighbor previews. Semantic text prefers node_text, otherwise names, types, and labels. The index does not automatically turn every business field in properties.data into searchable text.

Entity semantic search depends on Graph Engine’s own embedding configuration. EMBEDDING_PROVIDER explicitly selects sentence-transformers, openai or lexical. Automatic selection first tries local Sentence Transformers, requiring its dependency and model weights, then OpenAI when OPENAI_API_KEY is set, and otherwise lexical hashing. The current OpenAI branch calls the fixed api.openai.com/v1/embeddings endpoint; Agent Runtime chat-model or EMBEDDING_BASE_URL settings do not override it. Lexical mode does not provide semantic ranking for entities. Index-encoding failures also disable semantic search; inspect semantic_enabled and retrieval errors. Explicit openai selection requires OPENAI_API_KEY; failure to initialize a forced provider does not run the automatic selection sequence. Initial entity indexing encodes node and edge documents for the entire graph, so remote reachability, latency and quota also affect availability.

Confidence is rule-based rather than probability-calibrated. Exact matches receive high confidence. Non-exact queries shorter than three characters are ambiguous, and score gaps below 0.05 can also trigger ambiguity. With semantic_enabled=false, entity search falls back to exact resolution and structural filters. Non-exact semantic scores are zero; this is not continued semantic ranking.

{
  "query": "",
  "filters": {
    "node_type": "customer",
    "min_degree": 2
  },
  "sort": {
    "field": "degree",
    "order": "desc"
  },
  "top_k": 10
}

This example filters candidate customers by type and degree. Confirm actual node_type values from graph or node information for your dataset. rerank currently reports configuration state without running a separate cross-encoder. If ambiguous=true, refine the request with type, endpoint, or node ID.

Parallel-edge retrieval has an additional limitation: search_edge resolves exact endpoints to only the last relationship, and indexed properties are also looked up by endpoints. It does not exhaustively enumerate relationships sharing endpoints. Inspect and edit individual relationships using the actual edge_id from get_constructed_graph; best or ambiguous=false does not establish uniqueness.

#Document chunking and context retrieval

LightweightRAG maintains a document index separate from node search. upload_document stores session documents plus doc_index.json and doc_embeddings.npy; retrieve_context returns text chunks based on query vectors. Relevant text does not establish a graph relationship; graph tools still provide the structural evidence.

The default splitter uses 512 whitespace-separated items with 128 items of overlap and a stride of 384. These are not model-tokenizer tokens; a long Chinese paragraph without spaces is not split every 512 characters. Prepare appropriately segmented Chinese text and inspect actual chunk counts and lengths.

Providers include local Sentence Transformers, a remote embedding API, or lexical hashing. Document retrieval uses 384-dimensional hash vectors in lexical mode. Because Python hashing depends on the process, old chunks are re-encoded in the current process rather than mixing hash spaces. Dimension changes require rebuilding the session document index.

Controlled document writes stage a version under .rag_versions, then rag_manifest.json selects the active directory. Readers therefore resolve one complete index/vector set. Returned doc_id, chunk_id, offset, and score identify cited text. offset is the chunk ordinal, not a character offset in the source document.

#Storage and scaling boundaries

The array representation needs roughly 16E bytes for the [2,E] int64 edge index, another 4E for ordinary float32 edge weights, and 4NF for N nodes with F float32 features. These figures exclude strings, Python objects, properties, NetworkX representations, retrieval indexes, and worker copies. Source-file size alone is not a memory estimate.

Structured large graphs also have a separate partitioned-snapshot path. prepare_large_graph streams an uploaded edge-list or triples file, and run_large_graph_algorithm submits a persistent job against a published version without first creating a full GraphData or NetworkX graph. Inspect get_large_graph_capabilities for exact, approximate or bounded semantics and resource limits, then read summaries, previews and artifact references. Partitioned versions and ordinary active-graph representations have separate access paths; a large-graph preview is not a fully loaded session graph.

GRAPH_STORAGE_BACKEND currently selects local or nebula. NebulaGraph mode requires an enabled connection. Some queries run in the database; NetworkX algorithms materialize data within capability and size limits. Defaults are 50,000 nodes and 200,000 edges, controlled by NEBULA_MAX_MATERIALIZE_NODES and NEBULA_MAX_MATERIALIZE_EDGES. materialized_scope, target_in_scope, and complete_for_query describe coverage; absence within a bounded hop neighborhood does not establish global unreachability.

Graph Engine owns source files, graph revisions, construction reports, domain experience, and model artifacts. Agent Runtime owns messages, runs, memory, and traces. NebulaGraph storage is configurable, but algorithms do not all run inside the database: remote queries and bounded materialization are separate paths.