词元(Token)是大语言模型处理文本的最小单位,由分词器把文字切分为子词、符号或单字,核心是模型的「计量与计费单位」,能力包括上下文按词元计长、推理按词元产出与 API 按词元计价。它解决的是计算机无法直接理解连续文字的表示问题;对使用者而言,词元直接影响成本、速度与可用上下文长度,是必须理解的基础概念。
理解词元的核心是掌握计量规则,它可以帮助企业实现以下目标:
让成本可预算。中英文的词元换算比例不同,掌握换算关系后账单估算与配额管理不再靠猜测,财务口径也能提前锁定。分词器对中文通常按字或词切分,对英文按子词切分,同一段内容的双语词元数可相差数倍,跨语言成本测算必须分别进行。主流平台均提供词元计数接口,应用侧在发送前即可预估消耗并做拦截或压缩,避免超限报错。
让上下文不溢出。按词元长度规划提示与文档组装,避免触顶后被静默截断,关键信息丢失的风险被消除。上下文窗口、输入与输出的计费单价均按词元口径标注,采购与选型时按统一口径折算才能真实比价。
让速度可优化。输出词元数是延迟的主要来源,控制回答长度与结构即可显著提速,交互体验更流畅。
让选型更精准。不同模型的分词器不同,同一段文本的词元数存在差异,实际成本对比必须按词元口径计算才有结论。
总结来看,词元是大模型世界的「度量衡」,读懂它才能把 AI 用得又好又省。
翻译
A token is the smallest unit of text processed by large language models, produced by tokenizers that split text into subwords, symbols or characters — the metering and billing unit of model computing.
The core of understanding tokens is mastering the metering rules, and it helps enterprises achieve the following goals:
Budgetable costs: Chinese-English token ratios differ, so mastering conversion makes billing estimates and quota management predictable.
No context overflow: assembling prompts by token length avoids silent truncation and the loss of critical information.
Optimizable speed: output token count drives latency, so controlling answer length visibly improves responsiveness.
Sharper model selection: tokenizers differ across models, so real cost comparisons must be computed on token counts.
In summary, tokens are the measurement unit of the large-model world; reading them well means using AI both better and cheaper.