AI Inference
549 episodes · Page 2 of 23
#6382: Choosing AI Models on Cost and Performance
Model prices vary thirty-eight-fold per index point, meaning ratio formulas love cheap options regardless of actual quality.
#6380: Choosing the Right Voice Activity Detector
A compact model file can hide a massive runtime, while Cobra bundles its proprietary detector and engine into just 450 kilobytes.
#6379: Porting Voice Models to Core ML and MLX
Core ML packages separate computation specs from raw weights, while MLX runs on CPU and GPU without touching the Neural Engine.
#6378: Running Speaker Identification Locally on Phones
Exported voice embedding models like CAM++ run offline on mobile devices in packages as small as twenty-eight megabytes.
#6375: Why Modern AI Rests on Statistics
When polygenic risk scores derived from European data are applied globally, prediction accuracy drops to 25 percent in African populations.
#6369: The Most Popular GPUs - Today's Cloud Demand
RunPod charges about $1.34 an hour for serverless RTX 4090 access, while a community pod runs for roughly 34 cents.
#6368: What Does It Take To Train An AI Model From Scratch?
nanochat trains a BPE tokenizer in Rust in roughly 34 seconds, yet it can quietly disadvantage languages like Korean by nearly half.
#6366: Why Training Recipes Matter More Than Model Size
Altering learning-rate timing alone improved global word error rates by twelve absolute percentage points in speech recognition experiments.
#6355: Why Model Downgrades Are Rarely About Quantization
A caching bug and a shift to medium reasoning effort once reduced Anthropic internal benchmark scores by about three percent.
#6349: Splitting Model Layers Across the Internet
Splitting model layers over a wide-area network yields under two tokens per second until scheduling, speculation, and pipelining step in.
#6347: Splitting LLM Layers Across AMD and Nvidia GPUs
A two-node llama.cpp setup running a 70B model jumps from three tokens per second on gigabit Ethernet to six on 2.5-gigabit.
#6345: How vLLM Squeezes More Requests into GPU Memory
Early serving engines wasted up to 80% of KV cache memory on unspent reservations; vLLM cut that slack to under 4% using virtual memory paging.
#6343: Why Live Video Generation Still Struggles With Timing
A streaming video model can render fifty-eight frames a second while still delivering pictures that are noticeably late.
#6341: Why Fast Speech Detectors Still Struggle with Onsets
Aalto and Genelec experiments reveal human ears can detect sound timing shifts down to half a millisecond at certain frequencies.
#6317: Choosing Model Precision, Quantization, and Format
Quantizing Qwen 2.5 9B to Q4_K_M boosted generation speed 2.7 times over BF16, while prompt processing rates stayed roughly unchanged.
#6316: Why Protein Design Ignores Astronomical Counts
A hundred amino acids can form roughly 10 to the 130th possible sequences, far exceeding the number of atoms in the observable universe.
#6315: Inside China's Mixed-Language AI Models
Domestic Chinese models typically draw sixty to eighty percent of their training data from Chinese, leaving massive room for English and code.
#6314: Why Forgotten Runpod Workloads Keep Billing
Runpod stops pods at a zero balance, but attached volume disks double in cost when stopped and unattached pods vanish entirely.
#6312: Choosing Between Modal, RunPod, and Vast.ai
RunPod lists an H100 SXM at $3.99 an hour on Secure Cloud, but its Community marketplace drops it to $2.69.
#6310: Why Model APIs Lack Dedicated Few-Shot Fields
Zou and colleagues found models suffered steep drops at 16,000 tokens on tasks requiring learning across an entire demonstration set.
#6309: Why DeepMind Systems Live Outside the Gemini API
AlphaEvolve saved 0.7% of Google's compute, but running it requires an enterprise cloud environment to evaluate candidate code.
#6302: Why Fine-Tuning Whisper Won't Stop Duplicated Text
Whisper evaluates transcripts against a gzip compression threshold of 2.4, rejecting repetitive text before retrying at higher temperatures.
#6295: When Splitting Serverless Apps Earns Its Keep
Modal apps enforce atomic deployment, meaning all grouped functions redeploy in place together while scaling independently.
#6289: Why Hermes and Pi Top the OpenRouter Coding Table
Hermes Agent and pi logged 1.66 trillion and 1.41 trillion tokens on OpenRouter, topping Claude Code despite fundamentally opposite designs.