SentencePiece
Google subword tokenization library
Episodes
-
#6105: Не только Китай: ИИ в России, Индии и ЯпонииКитай занимает почти всё внимание в разговорах об ИИ, но Россия, Индия и Япония тоже создают мощные региональные модели с необычной архитектурой. -
#2982: Why Your TTS Model Nails "Shabbat" but Not "Keren HishtalmutWhy multilingual TTS models handle loanwords but fail at niche vocabulary — and what you can do about it. -
#2060: The Tokenizer's Hidden Tax on Non-English TextWhy does a simple greeting in Mandarin cost more to process than in English? It's the tokenizer's hidden inefficiency. -
#1680: Beyond China: AI in Russia, India, JapanChina dominates the AI conversation, but Russia, India, and Japan are building powerful regional models with unique architectures. -
#1084: Why AI Models Can’t Read and Your Bill Is RisingWhy does the same prompt cost more on different models? Discover the "invisible wall" of tokenization and how it shapes AI perception.