LoRA (Low-Rank Adaptation) & rsLoRA (Rank-Stabilized LoRA)

Freezes pretrained weight matrices $W_0 \in \mathbb{R}^{d_{\text{out}}\times d_{\text{in}}}$ and injects trainable low-rank matrices $B \in \mathbb{R}^{d_{\text{out}}\times r}, A \in \mathbb{R}^{r \times d_{\text{in}}}$ ($r \ll \min(d_{\text{in}}, d_{\text{out}})$): $W = W_0 + \gamma_r B A$ where standard LoRA sets $\gamma_r = \frac{\alpha}{r}$ (which causes gradient norm collapse as $r$ increases), while rsLoRA sets $\gamma_r = \frac{\alpha}{\sqrt{r}}$ so activation and gradient variances remain invariant across high ranks $r \in \{64, 128, 256\}$.