There are these things called Loras (Low Ranked adaptions) which do exactly this.

"A language model with billions of parameters may be LoRA fine-tuned with only several millions of parameters." -- Wikipedia