Transformer Architecture
215 episodes · Page 3 of 9
#5431: The Other Half of Hugging Face: Why BERT Still Out-Downloads Llama
Encoder models pull over a billion downloads a month. Decoder models pull 397 million. The AI conversation and the download counter are describing ...
#5430: Two Boxes: ASR and the Text Fixer Behind It
Punctuation, casing, ITN, disfluency — the four-job layer between raw ASR output and text you can actually read.
#5419: The Attention Budget in Your Pocket
Why phone dictation runs out of room, and how bounded attention windows buy you punctuation without blowing your memory budget.
#5410: Adapters: 102KB That Reshapes a 403GB Model
A 102KB adapter file changes how a 403GB base model behaves — without ever merging into it. Here's how model adapters actually work.
#5408: When Small NLP Models Beat the LLM
Feature extraction, fill-mask, token classification — the classic NLP tasks still have a job. Here's when a small model beats a frontier API.
#5396: Teaching a Small Model to Stop Spelling Out Numbers
Your ASR pipeline is fine until someone dictates "three point two" and gets "three point two" spelled out. Here's how inverse text normalization ac...
#5289: Why World's Fairs Built Absurd Transit
Moving sidewalks, monorails, and robot boxers: what we lost when technology's grand public spectacles became trade shows.
#5188: DeepSeek's Point Release That Isn't
DeepSeek shipped a whole new architecture and called it a point release. Here's what actually changed inside the model.
#5184: AI Is a Number Factory, Not a Wordsmith
Under the prose, every AI model is just matrices of floating-point math. So where does the randomness actually come from?
#5182: DeepSeek V4.1 Flash: 1M Context, 437x Smaller KV Cache
DeepSeek V4.1 Flash landed with a 1M-token window and a KV cache 437x smaller than V1. Here's what actually changed — and why the middle of your co...
#5089: How Runway Overruns Happen and the EMAS Safety Net
A Prime Air 767 overran at Miami. We break down the physics, the safety engineering, and why EMAS beds matter.
#5041: GPT-6 Astra, 100K GPUs, and the AGI Question
OpenAI's GPT-6 Astra arrives with an AGI claim and a 100K-GPU training run. We break down what Stargate LLC actually is and whether the scale justi...
#4999: Why AI Can't Just Put a Circle on Your Photo
Why multimodal AI models can't simply circle a Dremel bit without rebuilding your entire image from scratch.
#4911: How Airline SOPs Vary Between Operators
Why the same plane type can fly differently depending on which airline's procedures the crew follows.
#4789: How LLMs Actually Know When to Stop
Three layers of stop signals, from EOS tokens to context limits — and why none of them really "know" when to stop.
#4774: When a Token Isn't a Word Anymore
Images, audio, and code all use "tokens" — but they're completely different things. Here's how tokenization actually works across modalities.
#4723: Are Model Families Actually Different Models?
Claude Opus, Sonnet, and Haiku aren't trim levels — they're different models sharing a brand name.
#4695: The Unfenced Runway: When Airfields Blend Into Town Life
What happens when a working airfield has no fence, no guards, and a grandmother using the runway as a shortcut?
#4670: Why AI Weights Are Indecipherable
Can you actually read an AI model's weights like a book? We explore why the answer is mostly no — and what researchers are doing about it.
#4669: What "Distribution" Really Means in AI Models
Unpacking what "distribution" actually means under the hood — and why training data variety shapes model behavior.
#4667: How Transformers Killed the Robot Voice
From espeak's robotic squawk to neural voices with added "ums" — how transformers made speech synthesis human.
#4628: Living Airside: Can You Make an Airport Your Home?
We explore whether you can actually live in airports, hopping between airside hotels without ever entering a country.
#4619: The Token Trap: Why AI Context Windows Confuse Everyone
Token counts are everywhere, but they don't tell you what AI can actually do. Here's what should replace them.
#4605: Why Chatterbox Still Leads Open-Source TTS
We break down the architecture of Chatterbox, the open-source TTS model powering this podcast, and why it remains state of the art.