Research questionHow can we test whether language models genuinely execute multi-step graph logic when static benchmarks become contaminated?Static test instances can enter training data, making strong benchmark performance ambiguous. The difficulty is determining whether a model carries out multi-step graph operations or reproduces memorized answers. Latest papersRecent research connected to this question, newest first.ClosureBench: A Constructive Benchmark for Compositional Graph ReasoningThe source studies ClosureBench, a constructive benchmark for graph-relational logical reasoning. It generates fresh instances from primitives such as reachability, degree, set operations, connectivity, and aggregation, with reference answers computed by executable logic. Tasks vary in graph size, edge density, query depth, and compositional level, and graphs can be represented as JSON edge lists or adjacency matrices. Results cover models from 1.5B open-weight systems to frontier models, with a separate program-emitting 4B model and seen-versus-fresh instance checks. Evidence is limited to the benchmark’s graph-reasoning tasks and reported model evaluations.research paper · Sep 4, 2026