cagliostro-v3

A 146M parameter decoder-only language model pretrained from scratch on 75B tokens of open web, synthetic textbook and mathematics data. It is the third model in the cagliostro line and the first to clear an Index of 26 on the Open SLM Leaderboard metric.

The training run is complete. 75.00B tokens, 762,939 steps, learning rate decayed to zero.

Results

Zero-shot, measured with lm-evaluation-harness and the leaderboard's own ArithMark-3 script, on the exported float32 weights in this repository.

Benchmark	Metric	Score
HellaSwag	acc_norm	42.51
ARC-Easy	acc_norm	54.88
ARC-Challenge	acc_norm	28.75
PIQA	acc_norm	67.46
ArithMark-3	acc_norm	43.70
Open SLM Index		26.55

The Index is the leaderboard's own formula, (N(HellaSwag,25) + N(CombinedARC,25) + N(PIQA,50) + 0.65*N(ArithMark,25)) / 3.65 where N(v,c) = 100(v-c)/(100-c) and CombinedARC is the mean of ARC-Easy and ARC-Challenge.

For context against other models at this scale, using the leaderboard's published figures:

Model	Params	Tokens	Index
SmolLM2-135M	135M	2T	27.13
cagliostro-v3	146M	75B	26.55
SmolLM-135M	135M	600B	25.74
GPT-X2.5-135M	135M	75B	25.17
Haidass1.5-143M	143M	400B	25.07
BananaMind-2-Pro	139M	100B	24.96

SmolLM2-135M is 0.58 Index ahead on roughly 27 times the training tokens. Token counts for the other models are as published by their authors.

No single benchmark carries the cooldown gain. All five move together, which is the signature of the learning rate depression lifting rather than the model acquiring something new in the last 11B tokens.

The shape of the result is lopsided. cagliostro-v3 beats both models shown on ArithMark-3 by more than 4 points, which is what tripling the mathematics share during the cooldown bought. Across the whole board it places third on that benchmark, behind MobileLLM-R1-140M-base at 65.70 and palmer-006 at 52.70. It trails on PIQA, where GPT-X2.5-135M sits at 69.42 against our 67.46, and PIQA carries the heaviest weight in the Index at 0.548 per point. That single task is most of the remaining gap to SmolLM2-135M.

The flat stretch from 30B to 64B is not a stall. A warmup-stable-decay schedule holds the learning rate at its peak for the first 85% of the run, and constant peak learning rate depresses zero-shot multiple-choice accuracy even while validation loss keeps improving. The recovery from 22.0 to 26.6 is what the cooldown unlocks, not new knowledge appearing in the last 9B tokens.

The sharp drop in training loss at 63.75B is the data mixture changing, not the model improving. Validation loss on a fixed held-out set is the honest line, and it moves smoothly.

Model details

Field	Value
Parameters	146,352,000
Non-embedding parameters	85.7%
Layers	30
Hidden size	640
Intermediate size	1,536
Attention heads	10
Key/value heads	5
Attention	Grouped query attention with cross-head subspace attenuation
Activation	SwiGLU
Normalization	RMSNorm, eps 1e-6
Positional encoding	RoPE, theta 100,000
Context length	2,048
Vocabulary	32,768 BPE
Embeddings	Tied input and output
Logit cap	15.0
Weights	float32 safetensors

The architecture is defined in this repository. trust_remote_code=True is required because CagliostroForCausalLM is not part of transformers.

Training data

Two mixtures. The first covers the stable phase, the second takes over when the cooldown begins at 85% of the run.

Source	Stable phase	Cooldown
FineWeb-Edu (deduplicated)	43.7%	37.0%
DCLM-Baseline	28.3%	5.0%
Cosmopedia v2	16.0%	25.0%
FineMath 3+	5.0%	15.0%
OpenMathInstruct-2	3.0%	13.0%
InfiWebMath 3+	2.0%	0.0%
SmolTalk	2.0%	5.0%

Mathematics rises from 10% to 28% during the cooldown. No source exceeds 0.4 epochs across the full run, so nothing is repeated enough to memorize.

Training setup

Field	Value
Optimizer	AdamW, weight decay 0.01
Schedule	Warmup-stable-decay
Warmup	2,000 steps
Stable phase	steps 2,000 to 648,498 at peak learning rate
Cooldown	114,441 steps, cosine to zero
Tokens per step	98,304
Total steps	762,939
Precision	bfloat16 with float32 master weights
Hardware	one RTX 5090
Throughput	90,000 to 103,000 tokens per second
Wall clock	about 9 days

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "bench-labs/cagliostro-v3"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, dtype=torch.float32)

ids = tok("The capital of France is", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=32, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

This is a base model with no instruction tuning and no chat template. It completes text.

Reproducing the evaluation

pip install lm-eval
python -m lm_eval --model hf \
  --model_args pretrained=bench-labs/cagliostro-v3,dtype=float32,trust_remote_code=True \
  --tasks hellaswag,arc_easy,arc_challenge,piqa \
  --num_fewshot 0 --batch_size 8 --device cuda:0

ArithMark-3 uses the script linked from the leaderboard, pointed at the same model id. Evaluate in float32. A bfloat16 round trip moves logits by about 1.3e-1 at this logit cap, which is enough to change borderline multiple-choice answers, while float32 agrees with the training weights to 3.2e-05.

Provenance

This repository holds the full training history. A checkpoint was pushed every 30 minutes from the first step, giving 363 commits between 11 and 20 September 2026. Any intermediate checkpoint can be retrieved by revision, and the benchmark trajectory in the chart above can be reproduced from them.

Limitations

English only. 2,048 token context. No instruction tuning, no safety tuning, no RLHF. At 146M parameters it confabulates freely and should not be relied on for factual questions. The mathematics ability measured by ArithMark is arithmetic and short symbolic work, not general mathematical reasoning.

License

Apache-2.0. The training data is drawn from FineWeb-Edu and FineMath (ODC-By), DCLM-Baseline and OpenMathInstruct-2 (CC-BY-4.0), Cosmopedia v2 and SmolTalk (Apache-2.0).