ModernBERT

Encoder-only baseline, 149M params, 8192 context

4 episodes mention this Visit official site →

Episodes

#6014: Why Every AI Model Is Secretly a Decoder

Encoder, decoder, or both? The three transformer shapes, why one won, and why BERT still refuses to die.

Oct 15
#5624: One Forward Pass: Building a Bounded Classifier

No generation, no parsing — just label scores in a single pass. How to build a small encoder classifier that actually works.

Sep 28
#5431: The Other Half of Hugging Face: Why BERT Still Out-Downloads Llama

Encoder models pull over a billion downloads a month. Decoder models pull 397 million. The AI conversation and the download counter are describing ...

Sep 21
#4458: How Headroom Compresses Agent Context Without Breaking Outputs

A deep dive into Headroom's content-aware compression pipeline that promises 20-95% token savings without changing agent answers.

Jul 24