A 124M-parameter GPT-2 model was trained on the 10B fineweb-edu dataset (https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) using Karpathy's llm library (https://github.com/karpathy/build-nanogpt). Training took 3 hours on 4 H100(80G) GPU, yielding the following graphs:
Settings:
Model Parameters: 124M
Tokens: 10B
Batch Size: 48
Max Sequence Length: 1024
Total Batches: 196608
Warmup Steps: 1906
Max Steps: 50862
GPUs: 4
GPU Memory Usage: 62567 MB
Training Time: 3 hours (12 GPU hours)
Inference Providers
NEW
This model isn't deployed by any Inference Provider.
🙋
Ask for provider support