cid 90af4588e27418cb3b5606a5f42d527949cbe4982e30ca73e7e3b3990e372be5 · original · open access · via openalex
Title: SemVerBench: Benchmarking LLM Comprehension of Version-Constraint Resolution Semantics Authors: Qibai Chen; Zeming Liu Venue: arXiv (Cornell University) Year: 2026 Canonical: doi:10.48550/arxiv.2609.11180 URL: https://doi.org/10.48550/arxiv.2609.11180 Collector: openalex OA: true ABSTRACT Large language model (LLM) coding agents constantly decide whether a version satisfies a constraint such as ^1.2.3 or =2.0,1.2 means =1.3.0) traps every model on Cargo (near 60%), and although standard PEP 440 prefix matching is universal, on zero-pad/post-release corner cases GPT-5.1 collapses (0/26) while Claude stays at 97-100% (verified on a 67-item oracle-validated set). Opus significantly outperforms all other models, and Sonnet outperforms the OpenAI models (McNemar). The failures look more like an activation/application gap than a knowledge gap: injecting the rule or a light correct hint recovers most errors, whereas interval decomposition does not, and models are at ceiling on the basic forms of the same rules. An author-stratified analysis finds no statistically significant self-favoritism. Because the task is verifiable and a free, 100%-correct resolver exists, tool delegation reaches ~100%: coding agents should delegate version resolution to a resolver rather than reason about versions in-head.