TECHNICAL.md

CaptionBERT-8192 v2 / v2-B — Technical Companion

Models: AbstractPhil/captionbert-8192-v2 · AbstractPhil/captionbert-8192-v2-B Data: AbstractPhil/captionbert-8192-v2-consensus · AbstractPhil/conceptual-captions-12m-webdataset-berts

---

1. Overview

captionbert-8192-v2 and captionbert-8192-v2-B are sibling 58.3M-parameter text encoders trained from scratch by consensus distillation: the training target is the agreement of five BERT-family teachers (bert-base, distilbert, roberta-base, albert-base-v2, ModernBERT-base), computed over 27–32 million image captions, and neither model ever saw a similarity label. Each ships with an AMOE adapter library — three task-anchored adapters dispatched by a graded router over the frozen backbone — that lifts the eight-task benchmark mean from .6077/.6031 to .7287/.7294, past every teacher. This document records the full recipe, the ten-model benchmark run under one shared harness, the adapter-library findings including a transplant/portability study, and the measured laws and limits the family established.

2. Training recipe

Architecture (identical for both backbones)

Parameters	58,308,864 (0.53× bert-base, so the compression claim survives)
Layers / hidden / heads	12 / 512 / 8
FFN	2048
Positions	8,192
Output	768-d, mean-pooled, L2-normalized

The consensus target

Each of the five teachers embeds the CC12M LLaVA-NeXT captions. Each teacher then receives one global whitened-Procrustes map into bert-base's coordinate frame — bert-base serves as a fixed reference member of the ensemble. The consensus vector for a caption is the normalized centroid of the five aligned embeddings. The student regresses onto these precomputed vectors and contrasts against them; the similarity structure it learns is entirely implicit in teacher agreement.

Objective and optimization

• Loss: InfoNCE (temperature 0.07) + per-sample MSE against the precomputed consensus vector.
• Optimizer: Adam, weight decay 0.
• Schedule: lr 6e-4, cosine decay, batch 2048.

The two runs

	v2	v2-B
Corpus	26.9M rows (54 of 66 chunks; 56 usable before the ModernBERT repair)	31,905,616 rows (66/66 chunks, repaired corpus)
Steps	52,548	62,312
Wall time	5h24m	~6.4h
Hardware	single RTX 6000 Pro	single RTX 6000 Pro
Data filtering	none	none

v2-B reuses the same alignment maps as v2. This is load-bearing: refitting the maps on the repaired corpus would place the new consensus targets in a different coordinate frame, and the loss carries no signal about frame identity.

Alignment-fit discipline (the positive control)

The five alignment maps were fit on a stratified random sample of 24k rows (N/d = 31.3) and reported out of sample. The worst teacher map scores 1,833× chance out-of-sample and retains 95% of its in-sample R@1. A retired predecessor audit in this program, fit at N/d = 4.9, retained 3.8% — that single ratio separates a map at the null from a map carrying real structure.

The maps also recovered a lineage nobody supplied: out-of-sample alignment cosine orders distilbert .625 ≫ roberta .372 > albert .331 ≈ ModernBERT .327 — exactly the architectural distance of each teacher from bert-base (DistilBERT is literally distilled from it).

Reference-member alignment pins the frame

Because the consensus is expressed in a fixed member's coordinates, the gap between the student's frame and the target frame collapsed monotonically toward zero during training (+.0235 → +.0070 → +.0015 → +.0005 → +.0010 final). This line therefore ships with no rotation matrix. The vision-side consensus line of the same research program aligns to a running mean — a frame-free target — and must ship a 0.5MB rotation. The same construction, measured on both sides, yields opposite deployment requirements.

3. Results

Training-side gauges

gauge	v2	v2-B
Target cosine	.8410	.8421
Embedding effective rank	102.9	103.1
Self-cosine (anisotropy)	+.0031	+.0030

The two backbones are indistinguishable on every training-side gauge.

The eight-task benchmark: 10 models, one harness

All models are mean-pooled and L2-normalized, with no task tuning, run through one shared harness under identical conditions. This table superseded every earlier comparison in the campaign (earlier tables had mixed a 2-task teacher run with an 8-task run — and mis-identified the best teacher as a result). Scores are Spearman correlations.

model	params	STS-B	SICK-R	STS12	STS13	STS14	STS15	STS16	BIOSSES	mean
bert-base	109.5M	.4729	.5865	.3087	.5988	.4773	.6029	.6373	.5469	.5289
ModernBERT-base	149.0M	.4215	.5479	.3527	.4247	.3795	.5349	.4174	.5630	.4552
roberta-base	124.6M	.5436	.6296	.3211	.5631	.4522	.6134	.6198	.5777	.5401
albert-base-v2	11.7M	.4784	.5364	.3101	.4831	.3809	.5542	.5491	.4863	.4723
distilbert	66.4M	.5717	.6424	.4344	.6490	.5410	.6663	.6854	.5162	.5883
captionbert-8192-v2	58.3M	.5747	.6526	.5051	.5995	.5452	.7136	.6776	.5933	.6077
captionbert-8192-v2-B	58.3M	.5752	.6548	.5012	.6037	.5470	.7146	.6782	.5500	.6031
v2 + arms	63.2M	.7684	.7391	.6682	.7557	.6921	.8055	.7626	.6382	.7287
v2-B + arms (native)	63.2M	.7675	.7374	.6706	.7381	.6945	.8109	.7695	.6472	.7294
all-MiniLM-L6-v2	22.7M	.8203	.7758	.7237	.8058	.7559	.8539	.7899	.8144	.7925

Reading the table row by row:

• bert-base (.5289) — the reference member whose frame every teacher was aligned into; the consensus construction is literally "agreement expressed in bert-base's coordinates," and the student beats it by +.0788 at half the size.
• ModernBERT-base (.4552) — the largest model in the table and the lowest mean: parameter count buys nothing on this benchmark. It was also the teacher whose embeddings went missing from 10 corpus chunks, triggering the repair that produced v2-B.
• roberta-base (.5401) — the previously presumed best teacher, demoted by the 8-task view; its row documents that the earlier 2-task comparison had picked the wrong bar.
• albert-base-v2 (.4723) — the 11.7M-parameter floor of the teacher family, showing the consensus was carried by teacher agreement rather than by uniformly strong parents.
• distilbert (.5883) — the actual best teacher, and therefore the bar the whole "student exceeds its teachers" claim is measured against.
• captionbert-8192-v2 (.6077) — the headline row: exceeds all five teachers at 13% of their combined 461M parameters, with the margin concentrated in STS12, where every teacher collapses to .31–.43 and both students hold ~.50.
• captionbert-8192-v2-B (.6031) — the data-volume null made visible: 19% more data sits in the same row of the table (see §6).
• v2 + arms (.7287) — what ~5M adapter parameters on a frozen backbone buy: +.1210, alongside an effective-rank jump from 36.6 to 57.6.
• v2-B + arms (.7294) — the best artifact in the family, and the closing point of the portability arc: native adapters recover 100% of what transplanted adapters lost (§5).
• all-MiniLM-L6-v2 (.7925) — the supervised reference, trained on 1B+ curated pairs; the adapted pair closes to within .063 of it, converting the recipe's ceiling into a concrete distance.

Replication at scale

"Student exceeds its teachers" had failed to replicate at v1 scale (26M parameters / 500k rows), where the student sat at the teacher median. At v2 scale it replicates: .6077 against the best teacher's .5883, at 13% of the teachers' combined 461M parameters, with no similarity label ever seen. The earlier conclusion failed at that scale; the modality itself supports the result.

Geometry tracks capability

Across all ten models on STS-B sentences, self-cosine (anisotropy) and effective rank track capability: roberta-base sits at the bottom at +.9594 self-cosine / effective rank 19.8, all-MiniLM-L6-v2 at the top at +.0251 / 86.7, and the adapted pair between them at +.0964 / 57.6 and +.0984 / 55.5. No training stack anywhere in the table contains an isotropy objective. Isotropy is the property that moved, unasked.

4. The AMOE adapter library

Each backbone carries a library of three anchors — adapters trained for equivalence, simplification, and paraphrase — dispatched by a graded router (sinh/cosh gating) over the frozen backbone. Anchors toggle bit-exactly (asserted at load, never assumed), so a single artifact serves as both the bare baseline and the adapted model.

Collective and per-arm numbers

• v2 + arms: .7287 eight-task mean. v2-B + its own native arms: .7294. The two are statistically tied (collective-level seed spread .003–.005); the tiebreak is provenance, and v2-B is the complete-corpus backbone with no asterisk — the best artifact in the family.
• Eight for eight: the 3-arm collective beats every one of its own members on every task. The sharpest column is BIOSSES: the equivalence anchor alone reads .5845, below the bare backbone's .5933, while the collective reads .6382 — the mixture rescues a task where its strongest member is a liability.
• Geometry moves with capability: the arms lift effective rank 36.6 → 57.6 (v2) and 36.1 → 55.5 (v2-B), and cut self-cosine +.1396 → +.0964 — the first evidence in this line that adaptation adds usable directions to the representation rather than only rotating existing ones.

The capacity control

In greedy roster selection, a completely random (untrained) arm gains +.0045 by itself — so every candidate is judged against random-at-same-size rather than against the bare backbone. Under that bar:

• Paraphrase cleared random by +.0146 and joined the roster.
• Lexical (WordNet) and topical (Specter) candidates scored below random and were rejected.
• Every 4th-arm candidate — including a second random arm — scored below the 3-arm collective, so selection halted at exactly the arm count the conserved-amplitude arithmetic predicted (measured amplitude shares .4987 / .1944 / .1810, sum .874).

A masked member is a damped member

Masking an anchor leaves the dispatch budget unrenormalized. The simplification anchor reads .7400 trained-and-run alone, and .6104 when read masked inside the collective. Mixture margins must therefore be quoted against solo runs, never against damped masked rows — an earlier +.0213 margin claim was corrected to +.0124 on exactly this.

Sharper routing is worse

Routing temperature .10 / .05 / .02 gives seed-mean STS-B .7506 / .7492 / .7441 (2 seeds, monotone) while routing separation rises — the router committed harder and lost. Graded blending carries the value of the dispatch; hard selection destroys it.

5. Anchors are backbone-bound (the transplant study)

The two backbones are indistinguishable on every gauge available: eight STS tasks within ±.0042 (mean delta +.0009 excluding the 100-row BIOSSES), effective rank 36.6 vs 36.1, self-cosine +.1396 vs +.1411. Yet:

• v2's arm collective attached to v2-B unchanged falls .7287 → .6863 — 68.8% of the arm gain retained, a 31% loss with no error and no warning. The collective drops more than its worst member (−.0425 vs −.0330): the dispatch compounds the mismatch.
• The recovery arc, with preregistered pass bars: re-aligning only the 1,536 routing keys (800 steps, anchors frozen) recovers 29% (.6987); retraining the three anchors on v2-B (1,500 steps each) recovers 100% (.7294, above v2's own .7287). Decomposition: 71% of the loss lives in the anchors (.0307), 29% in the routing keys (.0124).
• The mechanism: adapters read the residual stream; task gauges read the pooled output. The stream carries backbone identity that the pooled output does not — which makes 1.6M adapter parameters a more sensitive instrument for backbone identity than the entire 8-task suite.
• Three hypotheses died in the retrain, all refuted by the same result: "v2-B is less adaptable" (it reaches .7294), "v2-B is over-specialized to captions" (the self-cosine signal was noise), and "v2-B is worse out of domain" (BIOSSES inverts to .6472 on v2-B with native anchors vs .6382 on v2 — the deficit was an artifact of foreign anchors).
• Operational cost of doing it right: one native anchor set per backbone, ~18 minutes of training.

6. Laws established by this family

Each law below carries the measurement that earned it.

Measurement discipline

Established on a retired predecessor audit in this program; enforced throughout this family.

1. A contrastive objective whose two views share one forward pass is underdetermined. The cheapest tell: retrieval scores invariant in gallery size.
2. No retrieval number is quoted until the query set is shown to vary — length histogram, pooled-query effective rank, pairwise cosine — checked before the first arm was trained.
3. An alignment map is never scored on the rows it was fit to. At N/d = 4.9 the predecessor retained 3.8% of in-sample R@1 out of sample; at N/d = 31.3 this family retained 95%.
4. Effective rank is always reported beside CV. CV is dimension-dependent; a CV target is implicitly a claim about effective dimension.
5. A control must be able to fail before its null is citable.
6. The round-trip law: a diagnostic of the form f(x, g⁻¹(g(x))) measures the operator, never the input; every diagnostic feature needs data on both sides.
7. Mimicry and capability are separate gauges: student→target retrieval and task scores live on separate lines with separate names.
8. Inspect the input distribution before designing the experiment. Validity picks the split; convenience gets no vote.

Dispatch laws

Measured on the adapter campaign.

9. The amplitude budget is conserved. Arms split a fixed sum — measured shares .4987 / .1944 / .1810, sum .874, as predicted — so arm count is an empirical outcome, never a design choice.
10. Sharper routing is worse (2 seeds, monotone): temperature .10/.05/.02 → STS-B .7506/.7492/.7441. Graded blending carries the value.
11. The amplitude decomposition runs before roster selection — it is a pre-flight instrument.
12. The one-task exemption: cross-task destruction by always-on expert stacks is absent where every arm serves one task; the exemption is scoped strictly to that condition.
13. The capacity control is the baseline: a random arm at the same size (+.0045 by itself) is the correct comparator for any roster claim.
14. A masked member is a damped member: masking leaves the dispatch budget unrenormalized (simplification: .7400 solo vs .6104 masked; a +.0213 margin corrected to +.0124).

Transfer and reliability laws

Established on the v2-B leg.

15. Anchors are backbone-bound: 71% of transplant loss lives in the anchors, 29% in the routing keys; one native anchor set per backbone costs ~18 minutes, and a hardcoded adapter repo in a loader is a silent 31% loss with no error.
16. A completed checkpoint is unresumable into a new corpus: epoch == epochs makes the training loop an empty range that reprints the previous run's metrics as its own.
17. Skipping a pipeline stage means fetching its outputs: a refit alignment map lands in a different frame with no signal in the loss.
18. A benchmark must be independent of the artifact it measures: the harness had briefly resolved the adapters through the model's own remote loader before this was caught.
19. A distribution shift is judged on p99 and max, never the mean: a .906 mean-length ratio fired a false alarm while p99 matched to 1% and the repaired maxima were shorter.

7. Known limits and honest caveats

The consensus-rank ceiling

The consensus target itself has effective rank 28.7 of 768 (26.9–29.6 across 56 chunks): five BERT-family teachers agree on only ~29 directions. The student builds 102.9 effective directions in-domain (rank grew monotonically through training and never turned down), yet only 33.4–36.6 of them survive on STS-B sentences — the ~74 caption-built directions stay in-domain. For scale: all-MiniLM-L6-v2 uses 86.7–94.3 effective directions on the same sentences at 2.6× fewer parameters. The limit is set by teacher agreement, and the student's in-domain rank shows capacity to spare; the named next lever is heterogeneous teachers, and it is measurable at consensus-build time, before any training is spent.

One caveat inside this story: the out-of-domain effective rank of 36.6 sits within ~4 points of bert-base's 32.8, so part of that column may be a property of the evaluation data itself. The clean control — bert-base embedded on the CC12M captions — was never run and is the top open item.

The data-volume null

19% more rows for 19% more steps — the complete repaired corpus versus the 54-chunk subset — moved nothing: 8-task mean .6031 vs .6077, where the entire −.0046 is BIOSSES; excluding it, +.0009. Training metrics are equally flat (target cosine .8421 vs .8410, effective rank 103.1 vs 102.9). Under this recipe, more of the same distribution leaves a ~29-direction target where it is. (Corpus provenance, for completeness: ModernBERT embeddings had been missing from 10 of 66 chunks; the repair was gate-verified at cosine 1.00000, minimum 1.00000, against a re-embed, and the "long captions broke it" hypothesis was refuted — p99 caption lengths were identical across repaired and never-missing chunks, and the repaired maxima were shorter.)

Statistical and provenance caveats

• The v2-B backbone is single-seed. The arm and temperature sweeps carry 2 seeds; collective-level seed spread is .003–.005.
• The .7287 / .7294 comparison is between each line's own shipped checkpoint selection: v2 ships its best checkpoint, v2-B its final. The two v2-B checkpoints differ materially (max weight delta 1.319e-01). It is the right comparison — each number belongs to its line's shipped artifact — and it is stated here so the reader can weigh it.
• BIOSSES is 100 rows with a Spearman standard error of ~0.10; every BIOSSES delta in this document is sub-sigma except where stated.
• The supervised reference all-MiniLM-L6-v2 (.7925, trained on 1B+ curated pairs) sits .063 above the adapted pair; that gap is the concrete distance this recipe has left to close.

References and attribution

Models

• BERT (Devlin et al.) — teacher and fixed reference member (bert-base); the coordinate frame every alignment map targets.
• DistilBERT (Sanh et al.) — teacher; the best single teacher on the eight-task suite.
• RoBERTa (Liu et al.) — teacher.
• ALBERT (Lan et al.) — teacher.
• ModernBERT (Warner et al.) — teacher.
• MiniLM (Wang et al.) via Sentence-BERT (Reimers & Gurevych) — the supervised reference all-MiniLM-L6-v2.

Data

• CC12M (Changpinyo et al.), obtained via pixparse/cc12m-wds — source of the caption corpus.
• MTEB (Muennighoff et al.) — the STS-B / SICK-R / STS12–16 / BIOSSES evaluation suites.

Methods

• InfoNCE (van den Oord et al.) — the contrastive term of the training loss.
• Orthogonal Procrustes (Schönemann, 1966) — the whitened alignment maps into the reference frame.
• Knowledge distillation (Hinton et al.) — the lineage of the consensus-distillation objective.
• LoRA (Hu et al.) — the lineage of the zero-initialized adapter arms.
• Mixture-of-Experts (Shazeer et al.) — the dispatch lineage behind the AMOE adapter library.
• Transformer (Vaswani et al.) — the backbone architecture.

Infrastructure

• PyTorch — training and evaluation stack.
• RunPod and Google Colab — compute environments.

Related artifacts: captionbert-8192-v2 · captionbert-8192-v2-B · captionbert-8192-v2-consensus · conceptual-captions-12m-webdataset-berts · prior installment: geometric-memory-ft4

---

Trained and evaluated in 2026 by AbstractPhil. All artifacts — both backbones, their adapter libraries, the consensus dataset, and the caption corpus — are at huggingface.co/AbstractPhil.