⚡️ PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models

Significantly improved finetuned perf by simply changing the initialization of LoRA's AB matrix from Gaussian/zero to principal components.

On GSM8K, Mistral-7B fine-tuned with PiSSA achieves an accuracy of 72.86%, outperforming LoRA’s 67.7% by 5.16%.

▪Github: https://github.com/GraphPKU/PiSSA
▪Paper: https://arxiv.org/abs/2404.02948

@opendatascience

Data Science by ODS.ai 🦜

⚡️ PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models

Significantly improved finetuned perf by simply changing the initialization of LoRA's AB matrix from Gaussian/zero to principal components.

On GSM8K, Mistral-7B fine-tuned with PiSSA achieves an accuracy of 72.86%, outperforming LoRA’s 67.7% by 5.16%.

▪Github: https://github.com/GraphPKU/PiSSA
▪Paper: https://arxiv.org/abs/2404.02948

@opendatascience

hottg.com/opendatascience/1994

13.7K viewsedited  Apr 9 at 08:06