Recurrent fast-weight memories and selective state-space models compress a growing context into a bounded state. Because every new token updates that state, their writes can be viewed as online continual-learning rules — at inference time the model is, in effect, continually learning from its own context.
Falcon develops this fast-weight attention perspective, connecting attention, recurrent memory, and state-space updates through the lens of continual learning over a bounded memory.
