One Noise to Rule Them All