Sunbird/sunflower_language_classification_v2

Sunflower Language ID v2

Language identification across 64 African languages, fine-tuned from google/t5-efficient-tiny with a replacement SentencePiece vocabulary trained on the target corpus.

Why v2

v1 used the stock T5 English-C4 vocabulary, which has no byte fallback and could not represent large parts of this language set — Amharic was 47% <unk>, Yoruba 20%. That damage falls hardest on short input, where there is no redundancy to absorb a lost token. v2 ships a 32k byte-fallback vocabulary trained on balanced text from all 64 languages, measured at 0.00% <unk>.

Accuracy by input length

Held-out split, 64 languages, ~16,000 sentences per bucket.

Input length	v1	v2
1 word	0.201	0.405
2 words	0.477	0.597
3 words	0.614	0.701
5 words	0.749	0.805
8 words	0.832	0.864
full sentence	0.907	0.928

Usage — two things that must match training

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

REPO = "yigagilbert/sunflower_language_classification_v2"

# 1. use_fast=False. The fast conversion drops SentencePiece byte_fallback and
#    reintroduces <unk> (0.47% on Amharic), which is the bug this model fixes.
tok = AutoTokenizer.from_pretrained(REPO, use_fast=False)
mdl = AutoModelForSequenceClassification.from_pretrained(REPO).eval()

# 2. Normalise exactly as training did: lowercase + collapse whitespace.
def normalize_text(s):
    return " ".join(s.lower().split())

def predict(text):
    enc = tok(normalize_text(text), return_tensors="pt", truncation=True, max_length=64)
    with torch.no_grad():
        probs = mdl(**enc).logits[0].softmax(-1)
    top = probs.argmax().item()
    return mdl.config.id2label[top], probs[top].item()

print(predict("Webale nyo okutuyamba"))   # ('lug', 0.62)

Skipping either step silently degrades short-text accuracy — passing raw-case text to a model trained on lowercase is what caused the original regression.

Supported languages

ISO 639-3 codes. "Short" is accuracy on 1-3 word input; chance is ~0.016.

Code	Language	Short		Code	Language	Short
ach	Acholi	0.605		luc	Aringa	0.479
adh	Adhola (Jopadhola)	0.292		lug	Luganda	0.692
afr	Afrikaans	0.866		luo	Dholuo	0.705
aka	Akan	0.505		luy	Luyia	0.228
alz	Alur	0.453		mhi	Ma'di	0.268
amh	Amharic	0.940		mlg	Malagasy	0.906
bam	Bambara	0.476		myx	Masaaba	0.727
bem	Bemba	0.746		nbl	Southern Ndebele	0.579
bfa	Bari	0.468		nuj	Nyole	0.338
cgg	Chiga (Rukiga)	0.193		nya	Nyanja (Chichewa)	0.810
dag	Dagbani	0.485		nyn	Nyankole	0.582
din	Dinka	0.696		nyo	Nyoro (Runyoro)	0.203
eng	English	0.698		orm	Oromo	0.838
ewe	Ewe	0.809		pcm	Nigerian Pidgin	0.564
fra	French	0.655		pok	Pokoot	0.047
ful	Fulah	0.511		rub	Gungu	0.439
gwr	Gwere	0.256		ruc	Ruuli	0.114
hau	Hausa	0.709		run	Rundi (Kirundi)	0.627
ibo	Igbo	0.819		rwm	Amba	0.031
kab	Kabyle	0.674		sna	Shona	0.813
kau	Kanuri	0.321		som	Somali	0.864
kdi	Kumam	0.376		sot	Southern Sotho	0.556
kdj	Karamojong	0.432		swa	Swahili	0.814
keo	Kakwa	0.421		teo	Teso (Ateso)	0.793
kik	Kikuyu	0.796		tlj	Talinga-Bwisi	0.417
kin	Kinyarwanda	0.615		tsn	Tswana	0.747
koo	Konzo	0.702		ttj	Tooro (Rutooro)	0.523
kpz	Kupsabiny	0.667		wol	Wolof	0.669
laj	Lango	0.252		xho	Xhosa	0.635
lgg	Lugbara	0.593		xog	Soga (Lusoga)	0.497
lin	Lingala	0.845		yor	Yoruba	0.885
lsm	Saamia	0.401		zul	Zulu	0.640

Known limitations

Several pairs are genuinely close to inseparable on short input, and the confusions are symmetric because the languages overlap in reality:

Pair	Rate	Note
cgg to nyn	34.2%	Chiga/Nyankole, often treated as one language
nyo to ttj	30.4%	Nyoro/Tooro, likewise
run to kin	19.7%	Kirundi/Kinyarwanda, mutually intelligible
nbl to zul	18.6%	Nguni cluster
pcm to eng	14.0%	Nigerian Pidgin vs English

For a single word, expect ~0.41 accuracy across 64 classes (chance is ~0.016). If your deployment only serves a known subset of languages, mask the logits to that subset — it recovers a large amount of accuracy for free.

Training

• Base: google/t5-efficient-tiny, embeddings re-initialised for the new vocab
• 60,000 steps, batch 64, lr 1e-3 cosine, bf16, best checkpoint by short-text (1-3 word) accuracy
• Augmentation: log-uniform random crops biased toward short spans, light character noise
• Logit adjustment (tau=1.0) with the class prior clipped to 50:1
• Languages with under 1,400 unique examples excluded as unlearnable