Out-of-equilibrium disordered systems may preserve a memory of external driving that can be read out at a later time. I will present one form of memory that does this in a remarkable fashion. The system remembers multiple values from a series of training inputs yet forgets nearly all of them at long times despite a continual repetition of the inputs. However, if noise is added to the system, all the memories can be preserved indefinitely. This provides a concrete example of 'plasticity' in memory formation. Despite these surprising features, the cause of this memory formation is easily understood and appears to be generic.