A Benchmark Dataset for Graph Regression with Homogeneous and Multi-Relational Variants
1Division of Data Science and AI, Chalmers University of Technology, Gothenburg, Sweden · 2Division of Computer Science, University of Trento, Trento, Italy
Journal of Data-centric Machine Learning Research · Volume 3, Article 10, pages 1–41 · 2026
In one sentence. RelSC broadens graph-regression evaluation beyond molecular data by pairing Java program graphs with execution-time targets in homogeneous and multi-relational representations.
Problem
Public graph-level regression benchmarks are concentrated in chemistry and drug discovery, leaving large directed graphs with execution semantics and multiple relation types underrepresented. The paper addresses this gap through software performance prediction: estimating a Java program’s execution time from its source-code graph before running it.
Main contributions
- RelSC provides 3,817 execution-time-labelled graphs from HadoopTests and four OssBuilds projects.
- RelSC-H represents augmented program graphs homogeneously, while RelSC-M retains multi-relational connectivity.
- Fixed 70/15/15 splits, raw data, PyTorch Geometric objects, construction code, and reproduction scripts are publicly released.
- Baselines and an ablation quantify representation difficulty and the value of control- and data-flow augmentation.
Dataset construction
Each Java file is parsed with javalang into an Abstract Syntax Tree (AST) with 72 node types, then augmented with 11 edge types derived from syntax, Control Flow Graphs (CFGs), and Data Flow Graphs (DFGs). RelSC-H combines node-type information and outgoing-edge features under a homogeneous representation. RelSC-M groups nodes into seven semantic categories and defines up to 49 directed category-to-category relations. The continuous execution-time labels are normalized independently to [0,1] for each dataset.
Experimental setting
- Task
- Graph-level regression for Java execution-time prediction.
- Corpora
- HadoopTests: 2,895 graphs, averaging 1,490 nodes; OssBuilds: 922 graphs from SystemDS, H2, Dubbo, and RDF4J, averaging 875 nodes.
- Metrics
- MAE (primary), RMSE, MAPE, Spearman’s rank correlation, and maximum relative error.
- Baselines
- CodeBERT, Code2Vec, GCN, ChebConv, GIN, GraphSAGE, PNA, GAT, Graph Transformer, HeteroSAGE, HeteroGAT, HeteroGCN, and HeteroTransformer.
Key results
PNA obtains the lowest test MAE on every RelSC-H dataset. The best off-the-shelf heterogeneous models remain weaker on RelSC-M, particularly for small projects; HeteroSAGE reaches 6.22 ± 5.45 MAE on SystemDS and 4.35 ± 3.51 on H2. Adding control- and data-flow edges to plain ASTs reduces OssBuilds MAE by approximately 0.07–0.09 across the tested GNNs.
| Dataset | Best RelSC-H | Best RelSC-M | CodeBERT |
|---|---|---|---|
| Hadoop | PNA: 0.09 ± 0.01 | HeteroGAT: 0.14 ± 0.02 | 0.14 ± 0.11 |
| SystemDS | PNA: 0.06 ± 0.00 | HeteroTransformer: 0.30 ± 0.10 | 0.17 ± 0.13 |
| OssBuilds | PNA: 0.11 ± 0.00 | HeteroGAT/HeteroGCN/HeteroTransformer: 0.18 | 0.15 ± 0.08 |
When this work is relevant
RelSC is relevant for methods for graph regression beyond molecules, GNN-based software performance prediction, execution-time estimation from source code, homogeneous-versus-heterogeneous graph comparisons, relation-aware message passing, and program representations combining AST, CFG, and DFG information.
Limitations
RelSC is smaller than web-scale graph benchmarks because stable timing labels require repeated controlled executions and substantial compute. Target values are imbalanced, no hyperparameter search was performed, and independently normalized targets support within-dataset model comparison rather than direct comparison of absolute errors across projects.
Citation
Paper identifier: arXiv:2505.23875 · https://doi.org/10.48550/arXiv.2505.23875
Dataset DOI: https://doi.org/10.5281/zenodo.18598713