# Agent runtime mechanics

How a graph data agent selects evidence, organizes tool calls, and interprets results, supported by sessions, skills, concurrency, approval continuation, and background jobs.

## Creating an agent for a request

Agent Runtime hosts iGraph’s graph data agent on Agno AgentOS. The agent interprets questions, identifies evidence needs, selects skills and tools, and uses actual results to guide further analysis and interpretation. Graph Engine handles graph schemas, queries, algorithms, and training; Backend handles users, projects, and browser-facing APIs. For each request, a factory assembles the model, database, graph tools, skills, learning components, and domain instructions. Model configuration and user skills therefore belong to the request environment rather than a dedicated persistent agent process per user.

The factory resolves model, reasoning, planning, and output options, then loads model configuration using the default workspace context injected by Backend. The model must be available in the catalog with a matching provider, protocol, endpoint, and credentials. These are model-service settings; browser users do not need roles or login permissions. Missing configuration prevents model invocation, and naming another model in a prompt does not change its connection settings.

| Option | Behavior |
| --- | --- |
| model | Selects a standard model or the current user’s custom model; an omitted value uses the service default. |
| reasoning_mode | direct uses direct execution; staged enables reasoning and instructions to check evidence and tool prerequisites. |
| planning_mode | never disables automatic session planning; auto keeps planning available; always maintains goals, steps, and progress and enables reasoning. |
| output_format | chat, report, and json are output options. json disables Markdown formatting; it does not itself supply business JSON Schema validation. |

The default workspace owner and session context propagate through service calls, and tool calls include the current run identifier. Execution control adds generation and operation IDs; internal interfaces continue to use service credentials. Backend, Agent Runtime, Graph Engine API, and MCP must share the same internal service key. Removing user sign-in does not remove this requirement.

In addition to MCP graph tools, the runtime installs session workspace tools to list visible files, execute restricted commands, manage the session environment, and publish intermediate files as Derived outputs. Workspace commands keep Uploaded, Derived, and Environment read-only and write intermediate results to Ephemeral; publishing creates a deliverable version. These tools are separate from Graph Engine computation tools.

## Sessions, runs, and graph state

A session contains an ongoing conversation. A run executes one request and may include multiple model turns, tool calls, and an approval pause. Sessions retain runs, messages, tool results, and session state. The agent includes history and session state in context, with a normal history window of the most recent 20 runs. This is a model-context setting, not deletion of older persisted history or a promise that every historical item appears in every model request.

Graph workflows distinguish a saved source, a constructed dataset, and a graph loaded into the current session. Uploading SQL, CSV, or TXT supplies source data. active_dataset_id identifies a dataset binding but does not by itself prove that an analyzable graph is loaded. The agent reads construction status and revision from data_state and verifies the current graph with get_graph_info. A ready constructed dataset can be reused through load_constructed_graph without rebuilding it for every analysis.

Backend refreshes the current user and project data scope for each run. construction_sources identifies saved datasets attached to the current message, while available_constructed_datasets lists constructed datasets available for loading or optimization. Before scoped tools execute, the runtime checks dataset_id and returns DATASET_SCOPE_REQUIRED for an out-of-scope value. A successful load updates the active dataset, ready status, and revision in session state. The model must not infer IDs from filenames or cross project boundaries merely because datasets share an owner.

For example, after an order-table upload, a user asks which customers connect multiple communities. The agent first identifies the attached dataset and its construction state, loads a reusable graph, then obtains centrality and community results. Even if no active dataset is bound, it must check the session graph before concluding that no graph exists. Customer findings, visualization, and reports should use the verified graph and tool evidence.

## From the analysis question to the next tool call

The question, session state, and tool results jointly determine analysis steps. Current domain instructions require checking uncertain graph state, favor traditional graph algorithms suited to the question, and use graph learning when task definition, features, labels, and training conditions are sufficient. These rules guide evidence gathering without forcing every request through upload, construction, training, and reporting.

| Current situation | Analysis action | Condition for proceeding |
| --- | --- | --- |
| Ambiguous object names | Locate candidates with search_node / search_edge and refine types, endpoints, or the user description. | Pass objects to dependent graph operations only after confirming their actual identifiers. |
| The question needs structural explanation | Select paths, connectivity, centrality, communities, or local analysis using relationship semantics. | Check result scope and metric meaning, then investigate any remaining evidence gaps. |
| The question needs prediction | Verify the task, features, supervision, and data split before calling the appropriate train / infer. | Use an actual model_id and completed job results, and explain the evaluation protocol. |
| Facts are available and need explanation | Organize a textual answer, using visualize_subgraph or create_analysis_report for structural views or reports when needed. | Ground metrics and observations in obtained results and distinguish inference from unverified hypotheses. |

The report tool receives prior raw responses in source_results under actual tool names. The model can explain connections, compare metrics, or propose next steps from these results, but report generation is not an independent proof of business correctness. The construction SQL reference executor validates fixed queries for relational graph schema changes; causal claims, business risk, and predictive validity each require appropriate evidence and evaluation.

## Skill discovery, loading, and review

Skills organize analysis procedures as reusable instructions describing when a problem-solving approach applies, its steps, and supporting references. Each skill starts at SKILL.md and may include other resources that Agno discovers and reads on demand. State checks, paths and connectivity, visualization, and statistical-reporting skills guide different stages of evidence gathering; construction skills cover graph schema preparation and optimization. Skills guide tool use without pruning MCP definitions to a fixed subset whenever a skill matches.

The loader combines built-in skills with active skills for the current user. Bundled resources are materialized in the runtime skill directory with names normalized to lowercase letters, digits, and hyphens. User skills have a separate user directory, while candidates live in a review directory. Candidate files are excluded from LocalSkills, so generating a skill draft and making it available to subsequent tasks are distinct stages.

The management API supports listing, creation, editing, deletion, approval, and rejection. Creation can target the active user directory or review_queue. Proposed reusable procedures can remain candidates while the user checks applicability, steps, and risk. Approval moves the candidate into the active directory, optionally renames it, and records an approval timestamp and playbook_id. An existing target returns a conflict instead of being overwritten. Rejection can delete files or leave a rejection record; retained candidates still do not become active skills.

> **Review enables instructions, not tool authorization**
> Approving a skill makes it available as an instruction source for later tasks. Clearing a graph and persisting graph optimizations still follow their tool-specific confirmation rules. Parameters suggested by a skill must also be checked against real graph, dataset, and model identifiers.

The current workbench provides a skill list, full-content inspection, .md / .zip upload, and approval or rejection of candidates. User and candidate skills can be deleted; built-in skills have no delete action. The API also supports creating and editing skills; an API capability does not imply that every operation has a matching UI button.

## What memory and learning retain

LearningMachine organizes user profiles, user memory, session context, reusable knowledge, and decision logs. These serve different timescales: stable preferences, durable user facts, task progress and plans, generalized experience, and significant decisions. Conversation history remains in session storage. Persisting a tool result in a run does not automatically promote it into long-term memory.

Reusable knowledge uses PostgreSQL pgvector and hybrid retrieval and requires the resolved embedding model and credentials. When EMBEDDING_API_KEY is unset, configuration falls back through OPENAI_API_KEY, LLM_API_KEY, and QWEN_API_KEY, with corresponding base-URL fallbacks. Check whether the resolved service supports the embedding model instead of inferring capability from a single variable. Without the required configuration, reusable knowledge is disabled while user profiles, user memory, and session context follow their own settings. PROPOSE does not mean a conversation necessarily generates, approves, and installs a SKILL.md; knowledge proposals and skill review are separate mechanisms.

Reusable knowledge requires an embedding model and credential. It uses PostgreSQL with pgvector and hybrid search. Without those settings, learned knowledge stays disabled while profile, memory, and session context retain their own configurations. PROPOSE means experience is handled as proposed knowledge; it does not mean every conversation automatically creates, approves, and installs a SKILL.md. Knowledge proposals and skill-directory review are separate mechanisms with separate state.

Retention is bounded by stability and verification. A durable preference such as “lead reports with conclusions and metrics” is appropriate memory; a full node list from one query, an unverified guess, or a transient error is not. With execution controls enabled, stopped runs do not start new learning extraction; embedding and vector writes participate in cancellation, and late memory writes check the stop state. Later runs may receive limited previews of confirmed tool results from interrupted work, but unfinished answers do not become conclusions, long-term memory, or new authorization.

The frontend management panel lists, searches, and deletes saved memory records. Managing those records is distinct from LearnedKnowledge vector retrieval. Diagnose memory issues by checking persistence, visibility to the current user, and embedding retrieval separately.

## MCP discovery and invocation identity

Graph tools are discovered from Graph Engine over MCP and exposed to the model as native tool calls. Business tools cover loading, construction, analysis, machine learning, and reports; register_session, delete_graph_session, and set_session_config are session-management entry points. The available set depends on discovery and feature configuration; for example, MCP_ENABLE_REPORTING_TOOLS can disable reporting tools. The model uses tool names, descriptions, and schemas without a separate text protocol replacing native calls.

Discovery and each invocation own their connection lifecycle. The toolkit is marked initialized only after successful discovery, so failed discovery cannot leave an initialized but empty toolkit. Concurrent calls use separate temporary MCP sessions, preventing one branch from closing another branch’s connection. Background-job polling also uses controlled temporary sessions, with connection and cancellation ownership tied to the corresponding run and operation.

The runtime injects internal call context: X-User-ID, X-Session-ID, X-Run-ID, and X-Agent-ID. X-User-ID identifies the fixed workspace owner rather than a browser login credential. Execution control adds X-Execution-Generation and X-Operation-ID. Tool results remain scoped to the session, and datasets pass project-scope checks; model-authored business parameters cannot switch ownership.

Successful transport does not imply successful business execution. The runtime interprets success and status fields, restores structured failures from MCP error text, and passes them to dependency scheduling and persisted tool history. If a graph load returns success=false, subsequent centrality analysis cannot treat it as a successfully loaded graph. The answer should explain the failure and identify the scope, revision, or parameter correction needed.

## Resource-aware scheduling within a model turn

A model may emit several native tool calls in one turn. The runtime preserves their relative ordering and builds a directed acyclic graph from resource contracts. A later call waits if it reads or writes a resource written by an earlier call, or writes a resource the earlier call reads. Only independent nodes run concurrently. The default concurrency limit is 4; AGENT_TOOL_MAX_CONCURRENCY is clamped to 1–32.

Resources include the active graph, datasets, construction results, model files, documents, and session configuration. Graph information, centrality, community, and path analysis mainly read the graph and can run together. Loading, version switching, and clearing write the graph and must be ordered against those reads. train writes model resources, so dependent infer calls wait for training. Construction resources can be narrowed by dataset_id. subgraph_extraction is treated as read-only only for explicit create_version=false. Unknown tools are conservative barriers.

A typical batch starts with load_constructed_graph followed by degree_analysis, node_centrality, and community_detection. Once loading completes, the three read branches can start together; a later version switch waits for them. The DAG handles resource conflicts, not missing parameters. If shortest_path needs node IDs returned by search_node, search must finish first and the next model turn must use those actual IDs. Likewise, a new model_id from training must be read before submitting inference that requires it.

When a predecessor fails, pauses for confirmation, or remains an unfinished job, dependent nodes are not executed and receive TOOL_DEPENDENCY_BLOCKED. Completed independent branches still provide valid evidence. After resolving the predecessor, blocked steps must be resubmitted; they are not an implicit queue waiting to resume. Events stream as work completes, while tool results are assembled in original batch order for deterministic model context.

## Approval pauses and continuation

Graph operations requiring confirmation pause through Agno’s native approval mechanism. clear_graph and commit_graph_optimization require confirmation: one clears the active graph, and the other persists a dataset optimization after preview. Read-only previews and loading saved datasets do not require that confirmation. An explicit request to train authorizes that training run without repeated approvals for epochs, polling, or immediately following read-only inference. Skill review and tool approval do not replace one another.

The paused record contains the tool call ID, tool name, and proposed arguments. The frontend displays them and offers approve or reject; the feedback API additionally supports a modify decision with patched_args. Backend rereads the authoritative paused record and checks run_id, tool_call_id, and tool name. Stale cards, resumed runs, or mismatched tools cause a conflict. Approval uses the persisted arguments; only modify uses replacement arguments, preventing a stale browser copy from changing the operation.

Continuation keeps the original run_id and preserves completed tools and outputs, updating only the decision for the pending call. With execution controls enabled, only a persisted paused phase whose executor has stopped may continue. Its generation increases so late acknowledgements from the earlier phase cannot update the new phase. Duplicate feedback returns accepted or in-progress conflicts. If a response is lost, recover session state before resubmitting a write. The current feedback API handles one pending operation at a time and rejects multiple simultaneous requirements.

For optimization, read real node identifiers and revision, preview the operations, explain the changes, and commit after confirmation. Preview does not persist changes. A revision conflict requires rereading, previewing, and confirming against the new base. After commit, load the newest saved version before verifying graph information or visualizing it. Resubmit any downstream analysis blocked by the DAG so approval, persistence, and presentation refer to the same verifiable result.

## When a background job is complete

Training, inference, and other heavy operations may return async=true and job_id. This means the job was accepted, not that its business result is ready. The graph-tool wrapper calls get_job_status until completion, failure, or cancellation before completing the tool call seen by the model. Progress events expose stages, progress, and messages while waiting, without repeatedly submitting training or writes.

Progress has its own sequence cursor: after_seq marks what has been consumed and progress_cursor marks the position reached by the response. The runtime validates job identity, state, strictly increasing sequences, and remaining progress pages. Even a done job is drained of remaining progress when progress_has_more is true before its final result is delivered. Transient transport failures retry only the same read-only status query with the same job and cursor, for up to three attempts; business work is not resubmitted.

A job may hand its result to a child job. The runtime follows the new job_id, resets the progress cursor, and rejects missing or cyclic child identities. Only the last job’s actual result completes the tool call with async=false. Failure or cancellation produces success=false and blocks dependent nodes. If a job claims completion without a usable final result, the runtime reports the result as unavailable rather than treating queue acceptance or a progress percentage as proof of success.

For users, completed training should map to an actual model ID, final metrics, and model files returned by the tool; completed analysis should map to graph results or report outputs. Cross-session experiment comparisons within a project should use Backend-verified project_training_summary to match configuration, metrics, and model IDs. Queue acceptance or partial training logs support a progress update, not invented accuracy, rankings, or performance comparisons.

The machine-learning view displays available training and inference records, training curves, and final metrics. The project training endpoint aggregates experiments for comparison. Reports can be inspected in the results view, and published files downloaded from the workspace. Metrics from different datasets, tasks, or evaluation splits need separate interpretation even when they appear in one list.

## Reconnecting event streams

SSE sends text deltas, tool lifecycle events, approval requests, and graph-job progress to the browser. It observes a run: disconnecting, refreshing, or changing the viewed session does not itself cancel the task. AgentOS owns execution. Reconnection should return to the original run_id and read existing events. Resending the question can start new business work and is not a substitute for event recovery.

Delta recovery uses original event indices. The recovery endpoint verifies user, session, agent, and run ownership, then checks whether the cursor is inside the in-memory event window. The initial cursor may be -1, and subsequent indices must advance continuously. Missing buffers, trimmed windows, and invalid cursors produce REPLAY_UNAVAILABLE or an in-stream recovery notice. The client should then synchronize saved run state and rebuild messages, tool cards, and pending feedback, rather than append renumbered database events to the old sequence.

Event reconnection reads existing events without recreating the business agent, rediscovering tools, or invoking the model again. Approval continuation is different: it supplies a decision for a specific pending tool and advances the original run through continue. Persistence recovery is another operation that retries saving existing results. Keeping these paths distinct prevents a refresh from retraining, a confirmation from discarding prior outputs, or a database retry from repeating graph mutations.

> **Keep the two cursor types separate**
> The SSE event index locates events within a run. A graph job’s progress_seq locates progress records for one job_id. A child job starts its own progress sequence; that does not reset the run’s SSE cursor.

## Cancellation, idempotency, and generations

> **Check deployment capabilities first**
> The current .env.example sets IGRAPH_RUN_CANCEL_ENABLED=true; the code fallback remains false when the variable is absent. The frontend enables stopping only when GET /api/capabilities returns run_cancel_v1 / job_cancel_v1=true, with Backend checking runtime and graph-service capabilities. edit_latest_turn_v1 separately controls latest-turn editing. The following mechanisms apply when these capabilities are enabled.

Durable execution controls are enabled by IGRAPH_RUN_CANCEL_ENABLED with a valid internal service key. A new request registers start_request_id and a content fingerprint before preparation and agent launch. Retrying the same ID and content reuses the run_id without resubmitting the question; different content under the same ID returns START_IDEMPOTENCY_CONFLICT. An active run, pending approval, or unconfirmed final save blocks new admission in the session. Clients should use can_send and active_run to manage input availability.

Cancellation has its own cancel_request_id and targets exactly one of run_id or start_request_id. The latter allows stopping a just-submitted request before its run_id is known: the service can record cancellation intent first and reject a late start. A cancellation ID is bound to its target and cannot be reused for another task. Each run has a generation, starting at 1 and increasing on approval continuation. Old-generation acknowledgements cannot overwrite current state or admit new jobs for the current phase.

Once cancellation is accepted, the runtime signals preparation, agent execution, and graph jobs without waiting for one branch to finish before notifying another. RunScope tracks model requests, tools, embedding calls, and other asynchronous work. Each branch closes its own connections and iterators. Stop checks run before operations and after waits so network failures during cancellation do not trigger new work. Jobs retain parent-child relationships, and cancelling a job also covers its descendants.

Receiving a stop request and fully stopping execution are different states. cancel_status advances from requested or stopping to confirmed only after preparation, the agent, and relevant jobs have exited and their final states are persisted. The API may return 202 before confirmation. Silence in the text stream is not evidence that computation stopped. During a database outage, the service attempts to stop known local work but returns CANCEL_NOT_CONFIRMED rather than claiming a durable acknowledgement.

## Persisting final state reliably

Session messages and run results are stored in native AgentOS session records in PostgreSQL. With durable execution control enabled, separate control tables hold runs, jobs, generations, cancellation intent, and exit evidence without duplicating chat history. Short transactions and PostgreSQL transaction-level advisory locks coordinate registration, admission checks, and cancellation intent within user and session boundaries. Transactions do not wait for remote computation or worker exit.

Persisting results and proving executor exit require separate evidence. Before native writes, the runtime captures the scrubbed run payload, including completed tools, partial text, approval requirements, and cancellation metadata, and retains a retryable local spool. Spool files use temporary-file atomic replacement; the database remains the authoritative session store. On failure, the controller merges only that run into the latest session and reads it back to verify the full result. It does not rerun the model, training, or business tools to repair a save.

While persistence is unconfirmed, terminal_persisted remains false and new requests cannot pass the unfinished session boundary. Retry data remains until database read-back and ledger acknowledgement, preventing executor exit with missing results from being mistaken for normal completion. After restart, the controller scans unfinished records and reconciles process-liveness evidence with saved checkpoints. Merely failing to find a task object in memory is not proof that execution stopped.

A stable paused run can be persisted as cancelled without creating a business agent to generate a cancellation answer. If the executor is proven to have exited before completion, recovery retains existing input, tool facts, and partial text and marks the run interrupted or erroneous. Later discussion can distinguish completed and unexecuted work and reread formal outputs when needed. Cancellation and recovery preserve verifiable results without presenting unfinished text as a completed report or repeating graph and model mutations during save retries.

| Client-visible field | Meaning |
| --- | --- |
| run_status | The run phase, such as preparing, running, paused, completed, cancelled, or error. |
| execution_stopped | Whether the required stop evidence is complete; it does not alone confirm result persistence. |
| terminal_persisted | Whether the native terminal or stable paused phase has been verified in storage. |
| cancel_status | Cancellation intent and confirmation progress; confirmed means stop and save requirements are satisfied. |
| execution_generation, revision | Identify execution phases and control-state changes; they are not the graph’s construction_revision. |
| can_send | Whether the session can admit the next request; use this for UI admission state. |

Reconciliation after a service restart restores saved display state, persists already available results, or marks interrupted execution. It does not promise to resume model inference or graph computation from the interruption point. Original SSE replay buffers belong to the old process; use snapshots and persisted records when those buffers are lost.