twinkle-ai/Formosa-Vision
Viewer โข Updated โข 2.79k โข 233 โข 13
AI & ML interests
Traditional Chinese language models
Recent Activity
View all activity
lianghsunย
updated 2
datasets 3 days ago
Auto-update dataset - 2026-10-19
#65 opened 6 days ago
by 
whats2000
Auto-update dataset - 2026-10-12
#64 opened 12 days ago
by whats2000
Auto-update dataset - 2026-10-12
#64 opened 12 days ago
by whats2000
cointeleportingย
updated a
dataset 14 days ago
Auto-update dataset - 2026-09-28
#63 opened 27 days ago
by whats2000
Auto-update dataset - 2026-09-28
#63 opened 27 days ago
by whats2000
Auto-update dataset - 2026-09-21
#62 opened about 1 month ago
by whats2000
Auto-update dataset - 2026-09-21
#62 opened about 1 month ago
by whats2000
Auto-update dataset - 2026-09-14
#61 opened about 1 month ago
by whats2000
win10ย
authored a
paper about 2 months ago
Auto-update dataset - 2026-09-07
#60 opened about 2 months ago
by whats2000
Auto-update dataset - 2026-08-31
#59 opened about 2 months ago
by whats2000
lianghsunย
updated a
model about 2 months ago
lianghsunย
published a
model about 2 months ago
lianghsunย
updated a
dataset about 2 months ago
lianghsunย
published a
dataset about 2 months ago
Post
395
๐น๐ผ Releasing https://huggingface.co/lianghsun/tw-tokenizer-v1 โ a tokenizer trained from scratch for Traditional Chinese (Taiwan).
**46% better Chinese compression than Qwen3.8-27B with 81% of its vocab (201K vs 248K), and English essentially untouched (4.657 vs 4.674 chars/token).**
The gain isn't from the regex โ it's the corpus. Qwen carries **27,364 Simplified-only multi-char tokens**, 11% of its vocab, dead weight for Traditional Chinese. Train on pure Traditional and that waste never appears.
Recent work is skeptical that compression predicts quality (Lotz et al. 2025 measured ฯ = โ0.59), so we validated two levels deeper:
**Segmentation** โ boundary hit rate against jieba: **85.6%** vs Qwen's 77.8%. Single-character tokens: **17.6%** vs 41.7%.
**Downstream** โ trained a 270M model from scratch with each tokenizer, compared bits-per-character (the only metric fair across tokenizers). At equal compute: **4.434 vs 4.591**, a 3.4% win โ with 13% fewer parameters. Same token budget means our model saw 440M characters vs 308M: **43% more data for the same compute**.
Also: 6-char cap on pure-CJK tokens (long tokens obscure orthographic info โ Haslett, CL 2025), NFC not NFKC, 1,024 reserved tokens.
Known limits (weak Tรขi-lรด support, small-scale downstream validation, vocab sweep hadn't flattened) are in the card.
๐ https://huggingface.co/lianghsun/tw-tokenizer-v1
**46% better Chinese compression than Qwen3.8-27B with 81% of its vocab (201K vs 248K), and English essentially untouched (4.657 vs 4.674 chars/token).**
The gain isn't from the regex โ it's the corpus. Qwen carries **27,364 Simplified-only multi-char tokens**, 11% of its vocab, dead weight for Traditional Chinese. Train on pure Traditional and that waste never appears.
Recent work is skeptical that compression predicts quality (Lotz et al. 2025 measured ฯ = โ0.59), so we validated two levels deeper:
**Segmentation** โ boundary hit rate against jieba: **85.6%** vs Qwen's 77.8%. Single-character tokens: **17.6%** vs 41.7%.
ๅฐๆฅญ็ด ้คใ็น่ณชๆ็ถๅ
ฌๅๅฏฉๆฅๅชๅ
ours: ['ๅฐๆฅญ็ด ้ค', 'ใ', '็น่ณช', 'ๆ็ถ', 'ๅ
ฌๅ', 'ๅฏฉๆฅ', 'ๅชๅ']
Qwen: ['ๅฐๆฅญ', '็ด ', '้ค', ...] โ ใ็ด ้คใsplit mid-word**Downstream** โ trained a 270M model from scratch with each tokenizer, compared bits-per-character (the only metric fair across tokenizers). At equal compute: **4.434 vs 4.591**, a 3.4% win โ with 13% fewer parameters. Same token budget means our model saw 440M characters vs 308M: **43% more data for the same compute**.
Also: 6-char cap on pure-CJK tokens (long tokens obscure orthographic info โ Haslett, CL 2025), NFC not NFKC, 1,024 reserved tokens.
Known limits (weak Tรขi-lรด support, small-scale downstream validation, vocab sweep hadn't flattened) are in the card.
๐ https://huggingface.co/lianghsun/tw-tokenizer-v1
Auto-update dataset - 2026-08-24
#58 opened 2 months ago
by whats2000