Cornell Relax ML Lab

A novel framework for calculating GraB updates in a layer-wise fashion, integrating seamlessly into the PyTorch Automatic Differentiation (AD) engine that makes it possible to employ optimal convergence rates in the pre-training of Large Language Models (LLMs).

GraB-sampler is the optimal permutation-based SGD data sampler for PyTorch