Quantization
28 episodes
#6134: Cheap Tokens, Expensive Scripts
A model with worse list prices can cost less per script — if the cache hits. The bill hides in tokens, not rates.
#6052: What's Actually in the Hugging Face ASR Catalog?
Four real architectural bets sit underneath thousands of Hugging Face ASR repos. Here's how to tell the bets from the repackagings.
#5900: Serving DeepSeek-V4.1-Flash on Your Own GPU
Five inference engines, one frontier MoE, and a shortlist that collapses depending on which GPU you already bought.
#5873: Abliteration: How $5 Undoes a Model's Guardrails
A rank-one weight edit costs under $5 and strips refusal behavior from open models. Then an automated tool beat human experts at it.
#5868: Fine-Tuning Parakeet for Um-Free, Hebrish Speech
Parakeet runs great on a phone — but fine-tuning it happens on a GPU. Here's the actual recipe for dropping ums and catching Hebrew.
#5460: Four Small Models, One Android Phone: Does It Actually Work?
A chained on-device dictation pipeline — VAD, ASR, cleanup — and why "it feels smooth" isn't the same as knowing it works.
#5456: Parakeet vs Whisper: Picking a Phone ASR Model
Why Whisper loses on Android, why Parakeet v2 beat v3, and how to benchmark speech-to-text without any tooling.
#5416: Xet, Buckets, and Auto-Pulling Model Weights
Hugging Face swapped its storage backend to Xet with barely a ripple. So why can't a bucket auto-pull new upstream weights?
#5411: Fine-Tuning at 4-Bit vs 16-Bit: What It Really Costs
QLoRA cuts fine-tuning VRAM 15x and cost up to 85% — but you pay in training time, quality, and safety alignment.
#5410: Adapters: 102KB That Reshapes a 403GB Model
A 102KB adapter file changes how a 403GB base model behaves — without ever merging into it. Here's how model adapters actually work.
#5384: Android ASR Runtimes: LiteRT, ExecuTorch, and Why Your Phone Has No VRAM
Why does your phone have no VRAM number? A tour of Android's runtime layer and what it takes to run ASR locally.
#5004: What Precision Is Your Model Actually Running At?
When you call an API, are you getting the model you think you are? Inference providers quantize more than they admit.
#4822: Cloud-Local AI Hybrid: Does It Actually Work?
Can you pair Claude's reasoning with a local Qwen 7B for savings? The gap is wider than you think.
#3322: Desk Pets: Local AI on Your Desktop
Are desk pets useful tools or expensive Tamagotchis? We break down the local AI, privacy tradeoffs, and psychology behind these devices.
#2840: How Long Must a Password Actually Be?
The surprising math behind how long your password needs to be to survive a brute-force attack.
#2041: The "MPEG Moment" for AI: Llamafile & Native Models
Why are we squeezing massive cloud models onto desktops? Meet the "native" AI revolution.
#2017: The Art of Squeezing AI Models onto Your GPU
Those cryptic letters on Hugging Face actually map how much brain power you trade for speed.
#1559: Dark Knowledge: The Art of AI Model Distillation
Discover how model distillation transfers "dark knowledge" from massive AI giants into tiny, efficient models that live in your pocket.
#1479: The Speed of Thought: Inside the New Era of Inference
The war for model size is over. Explore the engineering breakthroughs making massive AI models faster than human thought.
#1099: Digital Recalls: Why Your AI Is Losing Its Edge
Is your AI getting lazier? Explore the "digital recall" and why the world’s most advanced models are secretly taking steps backward.
#1094: The CPU-First Era: Why AI is Moving Back to the Processor
Is the GPU's reign over? Discover how modern CPUs and clever optimization are bringing powerful AI models to the hardware you already own.
#1085: The Tokenization Lie: How AI Actually Processes Media
Think 1,000 tokens equals 750 words? For audio and video, that rule is a lie. Discover the hidden math behind multimodal AI.
#992: Beyond the Digital Sandwich: The Future of Voice AI
Is speech recognition dead? Explore how multimodal models are replacing the "digital sandwich" with true intent-based reasoning.
#477: The Hard Trade-Offs of On-Device AI Agents
Can your phone finally think for itself? Explore the hardware and software breakthroughs bringing agentic AI to the palm of your hand.