#Paper and question source: GABench
The four graph-analysis experiments use questions from our work, GABench. Its 10,400 tasks assess agents’ ability to plan, use graph tools, and answer from their results.
#Overview of test questions
| Task family | What the questions ask |
|---|---|
| Graph retrieval | Retrieve node attributes, edges, and graph statistics. |
| Graph theory | Compute centrality, shortest paths, and global structural properties. |
| Graph machine learning | Perform node classification, link prediction, and graph classification. |
| Open-ended graph question answering | Use graph evidence to develop insights, predictions, and explanations. |
- NAG: Numerical-Attribute Graph, with numerical node attributes.
- TAG: Text-Attributed Graph, with textual node attributes.
- TPG: Text-Paired Graph, paired with a graph-level text description.
#Experimental setup and reading the results
The four graph-analysis experiments use GLM-5-turbo as the base model; rows identify the agent harness. A model-only row in the source results denotes OpenClaw. These four tables report only SR (task success rate), as a percentage with two decimal places; higher is better.
Each SR column is ranked independently: the highest score is bold, the second-highest score is underlined, and ties share the same marking. Retrieval and theory compare OpenClaw and iGraph; machine learning and open-ended question answering compare Hermes, Claude Code, OpenClaw, and iGraph.
#Graph retrieval
Retrieval SR is reported at the node, edge, and graph levels.
| Agent harness | NAGNumerical-Attribute Graph | TAGText-Attributed Graph | TPGText-Paired Graph | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Node | Edge | Graph | Node | Edge | Graph | Node | Edge | Graph | |
| OpenClaw | 8.33 | 6.25 | 6.67 | 72.91 | 18.18 | 27.27 | 60.00 | 11.11 | 42.86 |
| iGraph | 31.00 | 94.74 | 18.67 | 100.00 | 100.00 | 82.50 | 100.00 | 100.00 | 100.00 |
#Graph theory
Graph-theory SR is reported at the node, edge, and graph levels.
| Agent harness | NAGNumerical-Attribute Graph | TAGText-Attributed Graph | TPGText-Paired Graph | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Node | Edge | Graph | Node | Edge | Graph | Node | Edge | Graph | |
| OpenClaw | 24.00 | 26.81 | 7.40 | 61.02 | 9.52 | 21.66 | 28.00 | 13.33 | 10.49 |
| iGraph | 37.53 | 38.14 | 45.00 | 94.44 | 88.24 | 88.89 | 100.00 | 100.00 | 100.00 |
#Graph machine learning
NC denotes node classification, LP link prediction, and GC graph classification. Scores measure the agent’s success rate (SR) on the corresponding tasks.
| Agent harness | NAGNumerical-Attribute Graph | TAGText-Attributed Graph | TPGText-Paired Graph | ||
|---|---|---|---|---|---|
| NC | LP | NC | LP | GC | |
| Hermes | 61.54 | 90.90 | 54.35 | 87.18 | 7.14 |
| Claude Code | 95.23 | 78.57 | 86.79 | 89.53 | 88.89 |
| OpenClaw | 26.66 | 65.18 | 19.36 | 36.77 | 26.74 |
| iGraph | 90.00 | 100.00 | 95.00 | 100.00 | 85.00 |
#Open-ended graph question answering
Insight and Prediction identify the two open-ended task categories. Both are compared using SR.
| Agent harness | NAGNumerical-Attribute Graph | TAGText-Attributed Graph | TPGText-Paired Graph | |||
|---|---|---|---|---|---|---|
| Insight | Prediction | Insight | Prediction | Insight | Prediction | |
| Hermes | 0.00 | 8.22 | 1.28 | 15.12 | 16.28 | 8.11 |
| Claude Code | 7.63 | 3.70 | 18.59 | 20.00 | 29.63 | 20.69 |
| OpenClaw | 7.65 | 3.22 | 13.56 | 10.47 | 14.29 | 12.50 |
| iGraph | 25.00 | 8.60 | 55.00 | 42.50 | 35.00 | 21.00 |
#Graph construction agent
The baseline graph is built using primary-key and foreign-key (PK/FK) rules. Results compare query cost after the construction agent optimizes the graph schema. Optimization queries guide the search and evaluation of candidate graph schemas; held-out queries assess improvements on a separate query set.
| Dataset | Optimization queries | Held-out queries |
|---|---|---|
| Amazon | 27.33% | 53.20% |
| AVS | 14.80% | 20.71% |
| MAG | 11.79% | 19.91% |
| StackExchange | 8.40% | 6.74% |
For each dataset, select the supplied result with lower costs on both optimization and held-out queries: the first result set for Amazon and AVS, and the second for MAG and StackExchange. Both reductions in a row come from the same result set and retain the reported values. This is a per-dataset best-result summary.
Percentages indicate cost reduction relative to PK/FK rule-based construction; larger reductions are better. Rows represent different datasets and are not ranked against one another.