QLoRA (4-Bit NormalFloat Quantized LoRA) & LoftQ

Quantizes the frozen base LLM to 4-bit NormalFloat (NF4) (information-theoretically optimal quantile quantization for normal weight distributions) with Double Quantization of quantization scales and Paged Optimizers, backpropagating BF16 gradients exclusively into FP16/BF16 LoRA adapters. LoftQ initializes $(Q, A, B)$ jointly via alternating SVD so $Q + BA \approx W_0$ at step 0, eliminating quantization discrepancy before post-training begins.