IFM/latent-feedback-checkpoints

Latent Feedback Is a Compute and Data Amplifier: checkpoints

These are the trained models of Latent Feedback Is a Compute and Data Amplifier (paper: alphaXiv, code: ifm-ai/LoRM, folder works/latent_feedback).

There are 174 runs, one folder each, named <arm>-<size>-u<U/N>-f<F>:

• Arms: transformer, and feedback, the latent feedback Transformer: each position reads the previous position's top-layer state, trained with two passes on 85% of the updates.
• Sizes: 60M, 134M, 271M and 778M parameters. The paper fits its law on 60M to 271M and holds 778M out.
• Grid: U/N is unique training tokens per parameter, and F is FLOPs per unique token in Transformer epochs.
• Choice of run: each folder is the best learning rate and weight decay of its cell. The paper's cell value is the minimum of a quadratic fitted to all of the cell's runs (the best run when that minimum lies outside the grid), so it can differ from the checkpoint's loss by up to 0.002.
• Seeds: -s1 and -s2 are the extra seeds at 134M, U/N 12.5 and 100.

Each folder holds:

• checkpoint.pt: the fp32 model weights, under model.
• config.yaml: the run's config, which make_config.py writes.
• metrics.jsonl: the training and validation log. For latent feedback, the validation loss the paper reports is eval/nll@P4.

Size	U/N	F
60M	12.5, 50, 100, 400	1 to 4
134M	12.5, 25, 50, 100, 200, 400	1 to 4
271M	12.5, 25, 50, 100, 200, 350	1 to 4
778M	12.5	1 to 4
778M	50	1, 2
778M	100	1

To download and score one run:

hf download IFM/latent-feedback-checkpoints --include 'feedback-134m-u100-f4/*' --local-dir runs
python -m lorm.eval.shards runs/feedback-134m-u100-f4 --data data/d3/dolma3_val_000000.bin --out val.json --passes 1,4