再帰型トランスフォーマー：より深い実効深度と効率的なデコーディング

The Recurrent Transformer:Greater Effective Depth and Efficient DecodingCostin-Andrei Oncescu∗ Depen Morwani Samy JelassiAlexandru Meterez Mujin Kwun Sham KakadeHarvard UniversityAbstractTransformers process tokens in parallel but are temporally shallow: at position t, each layer attends tokey–value pairs computed based on the previous layer, yielding a depth capped by the number of layers. Recurrentmodels offer unbounded temporal depth but suffer from optimization instability and historically underutilizemodern accelerators. We introduce the Recurrent Transformer, a simple architectural change where each layerattends to key–value pairs computed off its own activations, yielding layerwise recurrent memory while preservingstandard autoregressive decoding cost. We show that the architecture can emulate both (i) a conventionalTransformer and (ii) token-to-token recurrent updates under mild assumptions, while avoiding optimizationinstability. Naively, prefill/training appears bandwidth-bound with effective arithmetic intensity near 1 becausekeys and values are revealed sequentially; we give an exact tiling-based algorithm that preserves the mathematicalcomputation while reducing HBM traffic from Θ(N 2) to Θ(N log N ), increasing effective arithmetic intensityto Θ(N/ log N ) for sequence length N . On 150M and 300M parameter C4 pretraining, Recurrent Transformersimprove cross-entropy over a parameter-matched Transformer baseline and achieve the improvement with fewerlayers (fixed parameters), suggesting that recurrence can trade depth for width, thus reducing KV cache memoryfootprint and inference latency. Code is available at https://github.com/geniucos/recurrent-transformer1 IntroductionTransformers [Vaswani et al., 2017] are highly effective sequence models, but their computation across positionsis structurally shallow: within each layer, position t attends to key–value pairs computed from the previous layerembeddings, allowing essentially at most one interaction per layer between any pair of positions. A growing bodyof theory studies the fundamental limitations implied by bounded depth in attention models, including circuit-complexity characterizations of what low-depth Transformers can and cannot represent [Merrill et al., 2022, Liuet al., 2023]. These perspectives motivate architectures that achieve greater effective depth.We introduce the Recurrent Transformer (RT), a simple modification of how key–value pairs are computedthat makes each layer temporally recurrent. In a standard Transformer, at layer ℓ, the key–value pair at position tis computed from the layer-(ℓ − 1) representation at that position and can then be attended to by later positionst′ > t. In the Recurrent Transformer, by contrast, the key–value pair at position t in layer ℓ is computed fromthat position’s output at layer ℓ, rather than from its layer-(ℓ − 1) representation. Consequently, a later positiont < t′ at layer ℓ attends to a representation at t that already reflects layer ℓ attention and MLP computation.Importantly, Recurrent Transformer performs this recurrence separately within each layer, so each layer maintains itsown key–value memory. This differs from the Feedback Transformer [Fan et al., 2020], which uses a shared memoryacross layers, and this layerwise separation is a key reason why our architecture can be implemented efficiently.We motivate Recurrent Transformer’s design through lenses of representation, optimization and computationalefficiency:(i) Representational perspective. Recurrent Transformers retains per-token key–value memory just like aTransformer, but increase the space of computations that can be expressed within a single layer by allowinglater positions to attend to representations that have already undergone attention and MLP processing. Undermild assumptions, Recurrent Transformers can emulate standard Transformer behavior; conversely, by restricting∗Correspondence to: concescu@g.harvard.edu1arXiv:2604.21215v1 [cs.LG] 23 Apr 2026