RePo: コンテキスト再配置を備えた言語モデル

We’re excited to introduce RePo: Language Models with Context Re-Positioning,

Interactive Website: https://pub.sakana.ai/repo/

Paper (arxiv): https://arxiv.org/abs/2512.14391

Code: https://github.com/SakanaAI/repo

Summary

Standard language models process information as a rigid linear sequence where the only signal for structure is a fixed token index, forcing them to treat physical proximity as semantic relevance. Cognitive Load Theory suggests this is inefficient. Just as humans struggle when key facts are buried in noise, models waste finite capacity managing disorganized inputs instead of focusing on deep reasoning.

RePo breaks this bottleneck by allowing models to actively reorganize their context. Instead of using a fixed index, our module learns to assign positions based on content relevance. This lets the model dynamically pull relevant distant information closer and push noise away, effectively reshaping the attention geometry to match the problem structure.

This flexibility yields significant gains in robustness. RePo outperforms standard encodings on noisy contexts, structured data, and long-range dependencies while maintaining competitive general performance. It represents a step toward models that intelligently curate their own working memory rather than passively accepting input order.