The residual stream is the backbone of modern Transformers, yet it is usually treated as a single additive channel. We propose a two-axis framework in which sequence position and layer depth provide distinct pathways for information to flow and be mixed.
Within this view, causal depth-wise residual attention — attention that mixes representations across layers — turns out to be equivalent to sequence-axis short sliding-window attention. This duality connects depth-wise and sequence-wise designs, clarifying how locality along one axis corresponds to structure along the other and suggesting a unified design space for residual architectures.
