模型路由是根据请求特征动态选择最合适模型处理任务的机制,核心能力是在质量、成本与延迟之间做逐请求的权衡,把简单请求交给小模型、把困难请求交给大模型。它解决的是所有请求都用最强模型带来的成本浪费,也解决了静态架构无法随模型能力演进调整的问题。
并非每个问题都需要最贵的模型,但事前判断难度又不总是可靠。
难度评估:用轻量分类器或启发式规则判断请求的复杂度。
级联策略:先用小模型处理,对置信度不足的请求再升级到大模型。
能力匹配:按任务类型(代码、长文理解、多模态等)路由到专精模型。
降级兜底:在高负载或预算约束下自动切换到更经济的配置。路由策略本身也需要评估:若难度判断存在系统性偏差,大请求被误判为小请求就会同时损失质量与成本收益,因此策略需要与业务目标一同校准。缓存与前缀复用是配套手段:把可复用的提示前缀预计算并缓存,能让同一模型在多次调用间的实际成本大幅低于名义单价。路由还可与预算策略绑定,为不同业务线设定不同的用量上限。
模型路由把模型选择从一次性架构决策变成了可按请求优化的运行时策略,是推理成本控制的关键手段,也让同一套应用能同时服务成本敏感与质量敏感的请求。
翻译
Model routing selects the most suitable model per request based on the characteristics of that request, balancing quality against cost and latency by sending easy requests to small models and hard ones to capable ones. It addresses the waste that comes from running every request through the most expensive model available.
Not every question needs the priciest model, yet predicting difficulty reliably in advance turns out to be hard.
Difficulty assessment: a lightweight classifier or a set of heuristics judges how complex a request is.
Cascading: a small model handles the request first, and only those with insufficient confidence are escalated to a larger model.
Capability matching: requests are routed to models specialised for the task type, such as code, long-document understanding or multimodal input.
Fallback and degradation: under heavy load or a budget constraint the system automatically switches to a more economical configuration.The routing policy itself has to be evaluated, because a systematic bias in difficulty estimation that sends hard requests to small models loses quality and the cost saving at once; the policy must therefore be calibrated against business objectives.Caching and prefix reuse are complementary techniques: precomputing and caching reusable prompt prefixes makes the effective cost of repeated calls to the same model considerably lower than its list price.Routing can also be bound to budget policy, setting different consumption ceilings for different business lines.
Routing turns model selection from a one-off architectural decision into a runtime strategy that can be optimised per request, and it is central to controlling inference cost.