---
license: apache-2.0
language: [en]
pipeline_tag: text-generation
base_model: DedeProGames/Kiyo-135M
tags: [diffusion-language-model, masked-diffusion, autoregressive-conversion, llama, kiyo]
---

# Kiyo-Diffusion-135M

Masked diffusion language model adapted from [DedeProGames/Kiyo-135M](https://huggingface.co/DedeProGames/Kiyo-135M): same Llama
weights, bidirectional attention, a new `<|mask|>` token, the masked-diffusion (MDLM/LLaDA) objective
and the AR shift operation (DiffuLLaMA/Dream recipe). No timestep embedding. Research use only.

```bash
python kiyo_diffusion.py generate --model DedeProGames/Kiyo-Diffusion-135M --prompt "The capital of France is"
```

## Diffusion adaptation

The autoregressive Kiyo-135M was converted in place — causal attention replaced by bidirectional
attention, a `<|mask|>` token added, and the objective switched to masked denoising — then trained
for **250M tokens** on the same data mix the base model was pretrained on
(fineweb-edu 45% / dclm-baseline 35% / finemath 10% / stack-v3 10%). Keeping the data distribution
identical is what protects the base knowledge: this is continued pretraining under a new objective,
not a domain shift.

| Setting | Value |
|---|---|
| Tokens | 250M (1908 steps × 131,072 tokens) |
| Learning rate | 1e-4 → 1e-5, cosine, 200 warmup |
| Attention annealing | causal → bidirectional over the first 381 steps (20%) |
| Batch | 131,072 tokens global, sequence length 1024 |
| Precision | bf16 |
| Hardware | 2× RTX 3060 12GB (DDP), ~15k tokens/s |

### Benchmark: BananaMind Base Bench 1.1

All 31 training checkpoints were evaluated on
[BananaMind/BananaMind-Base-Bench-1.1](https://huggingface.co/datasets/BananaMind/BananaMind-Base-Bench-1.1)
— 350 four-way text-completion items across 7 categories, reporting a fixed-scale Overall Elo.

![Overall Elo per checkpoint](diffusion_adaptation_elo.png)

| Checkpoint | Overall Elo | Accuracy |
|---|---:|---:|
| Kiyo-135M, before conversion (official causal metric) | 1125.9 | 67.7% |
| After conversion, 0 training steps | 968.7 | 50.0% |
| Step 500 (65M tokens) | 1054.1 | 60.3% |
| **Final, step 1908 (250M tokens)** | **1055.4** | **59.1%** |

**What the curve shows.** Opening the attention costs 157 Elo immediately. The first 500 steps
(65M tokens) recover 85 of those points — 54% of the gap — and essentially all of the recovery
happens there. Across the remaining 1408 steps the Overall Elo stays flat at 1045 ± 10 while the
training NELBO keeps falling (4.47 → 4.06): the model keeps getting better at denoising without
regaining measurable benchmark capability.

For a 135M-parameter model, diffusion adaptation appears to saturate near 65M tokens, and the
residual ~70 Elo gap looks structural to bidirectional attention at this scale rather than a debt
that more tokens would repay. Per category, `world_knowledge` and `context_tracking` recover closest
to the original — bidirectional context helps them — while `code_completion` suffers most
(0.86 → 0.50), which is unsurprising for a strongly sequential task.

### Scoring methodology

A masked diffusion LM has no causal next-token likelihood, so the official metric of this benchmark
(mean conditional token log-probability) cannot be reproduced. Checkpoints are scored instead with a
Monte-Carlo ELBO (`n_mc=64`), the diffusion analogue used by LLaDA.

To make checkpoints comparable to one another, mask samples are drawn from a seed derived from the
item index, so every checkpoint is evaluated against identical masks. Without this, replicate runs
of the same checkpoint differed by as much as 16 Elo; with it, replicates are identical.

**The dashed baseline in the chart is therefore not an ELBO-equivalent score.** It marks where the
model started, measured the only way an autoregressive model can be measured. Elo values here are
comparable within this chart only, and not with the official leaderboard.