First, we challenge the standard practice of uniform parameter allocation in fine-tuning. We introduce MiMi (Mini-Adapters), which demonstrates that adaptation requirements are heterogeneous across network depth. By dynamically allocating capacity through iterative neuron selection, MiMi achieves superior accuracy–efficiency trade-offs.

Transitioning from static to sequential efficiency, we address the stability–plasticity dilemma: the tension between retaining prior knowledge and adapting to new tasks. We identify that existing « first-session adaptation » methods prioritise stability at the cost of rigidity. To resolve this, we propose PLASTIC (Plasticity-Enhanced Test-Time Adaptation in Class-Incremental Learning), a framework that integrates test-time adaptation to allow plasticity throughout the model’s lifespan. Complementing this, we introduce CoFiMA (Continual Fisher-Weighted Model Averaging) to ensure stability. By reformulating continual learning as a model merging problem weighted by Fisher information, CoFiMA consolidates knowledge from sequential tasks without the computational overhead of complex regularisation or large replay buffers.

Finally, we extend these principles to vision–language models, where storing raw images for rehearsal introduces storage and data-sharing constraints. We propose QUAD (QUestion-only replay with Attention Distillation) for continual visual question answering. QUAD decouples multimodal continuity from visual storage, showing that attention-consistency distillation allows models to retain visual grounding using text-only queries.

Taken together, these contributions show that efficient and continual adaptation need not depend on storing past data or retraining the full model. A common principle underlies them: adaptation is driven by information the model already encodes rather than by replaying stored examples.