StateBench: Testing how language models track state changes over long documents
huggingface ·
AllenAI has introduced StateBench, a new benchmark designed to evaluate how well language models track state changes across long documents. The benchmark categorizes tasks into four complexity levels: r-trivial, aperiodic, soluble, and general finite, which correspond to increasing degrees of variable assignment and dependency. Researchers tested various architectures, including standard transformers, Gated DeltaNet, and hybrid models, as well as frontier commercial models like Gemini 3.8 Flash and GPT-6. While pretrained models generally struggled with harder state-tracking problems as input length increased, models trained directly on the task showed significant architectural differences. Specifically, Gated DeltaNet and hybrid models demonstrated superior generalization capabilities compared to transformers, which failed even when explicitly trained on the hardest tasks. Frontier models also exhibited sharp declines in accuracy on complex sequences, highlighting sensitivity issues where missing a single update leads to incorrect results. The study suggests that recurrent and hybrid architectures may offer more robust solutions for long-context state tracking than traditional transformer designs. StateBench aims to provide researchers with a finer-grained tool to identify model limitations and guide the development of future architectures.
huggingface