Post
131
Zero-v1.0-144M is currently training. So far, it has completed 26% of its training run, with about 98 billion tokens to go.
Current Val PPL: 13.705
Zero-v1.0 uses a custom architecture consisting of RMSNorm, RoPe, SwiGLU, Engram Conditional Memory, mHC, and XSA GQA.
Current Val PPL: 13.705
Zero-v1.0 uses a custom architecture consisting of RMSNorm, RoPe, SwiGLU, Engram Conditional Memory, mHC, and XSA GQA.