REU Project

Large language models, like Llama-4 and GPT-5, provide the foundation upon which they can be adapted to a specialized domain through fine-tuning methods. Existing methods in post-training, like full-parameter fine tuning, can be computationally expensive and memory exhaustive, even with some parameter reduction found in commonly used low-rank adaptation. We first examine existing tensor-based adaptation methods, and their reported impact on parameter count and accuracy based on varying compression techniques. Furthermore, we attempt to develop a unified framework for low tensor rank adaptations that remains agnostic to non-uniform model weights, reflecting developments in recent architecture. This framework will also generalize different tensor compression methods found, highlighting the tradeoffs and benefits to accuracy and parameter count associated with each factorization.