HmarBERT-mini

HmarBERT-mini is a small, 4-layer BERT model trained from scratch for the Hmar language (hmr, ISO 639-3), part of the Zo languages family.

At roughly 16.9M parameters (~32 MB in FP16), it runs quickly on laptops, phones, or basic CPUs.

---

Model Details

• Architecture: 4-layer BERT encoder (384 hidden size, 6 attention heads)
• Parameters: 16.91M (~32.2 MB FP16 / ~67.6 MB FP32)
• Vocabulary: 24,576 WordPiece tokens (uncased)
• Max Sequence Length: 512 tokens
• Masking: Whole-word masking (6-epoch cycles, 100% token coverage)
• Training length: 48 epochs (8 cycles)

---

Training

Training ran in two phases:

1. Epochs 1–24: ~265k short sentences (hmar-heritage-org/sentences) to learn basic grammar, affixes, and sentence structure.
2. Epochs 25–48: ~33k paragraphs (hmar-heritage-org/paragraphs, ~3.86M words) from scripture, prose, essays, and online comments to handle full context up to 512 tokens.

---

Tokenizer

• Accents: Typing without accents (tirin, van, Pathien) maps to the same tokens as formal text with accents (tîrin, vân, Pathien).
• Emojis and symbols: Cleaned during preprocessing so they don't produce accidental [UNK] tokens.
• Hmar characters: Preserves all standard letters and digraphs (ṭ, kh, th, ng, ch).

---

Quickstart

from transformers import pipeline

unmasker = pipeline("fill-mask", model="azinamotoe/HmarBERT-mini")

# Example 1: Prose
text1 = "Chu zan chun lung an lêng deu rieu leiin a ruolnu chu hla rikawt insaktir dingin kan [MASK] a."
print(unmasker(text1)[0]['sequence'])
# -> "...hla rikawt insaktir dingin kan ngên a."

# Example 2: Scripture
text2 = "A tîrin Pathienin hnuoi le [MASK] a siem a."
print(unmasker(text2)[0]['sequence'])
# -> "A tîrin Pathienin hnuoi le vân a siem a."

---

Evaluation

For test results, accuracy curves, and qualitative examples across different registers, see:

• BENCHMARK.md — Evaluation report and case studies
• benchmark_metrics.json — Training loss and learning rate history
• benchmark_predictions.jsonl — Model predictions on the test set

---

Acknowledgements

Trained on open datasets curated by the Hmar Heritage Foundation (hmar-heritage-org).