知识蒸馏是把大模型或集成模型的输出行为迁移到更小模型上的训练方法，核心是用软标签传递教师模型的暗知识，能力包括 logits 蒸馏、中间层特征对齐与蒸馏损失设计。它解决的是大模型推理成本高、难以在边缘或高并发场景部署的问题，并不解决小模型自身容量不足带来的性能上限问题。
知识蒸馏的核心是行为迁移，它可以帮助企业实现以下目标：
让部署成本显著下降。参数规模缩小后，推理延迟与单次成本同步降低，可在更高并发下运行。教师模型的输出分布以温度参数软化，类别之间的细微差异被保留，传递的信息比硬标签更稠密。
让暗知识被传递。教师的输出分布携带类别之间的相对关系，这些信息比硬标签更稠密。蒸馏损失与任务损失联合优化，学生在拟合教师行为的同时仍以真实标签为准，二者不会相互替代。
让小模型超越同规模独立训练。学生在蒸馏目标下常能达到接近教师、在特定任务上甚至更好的表现。
让部署位置更灵活。蒸馏后模型可在边缘设备、端侧或受限算力环境中独立运行，不再依赖云端调用。
总之，知识蒸馏可以帮助企业实现模型部署从依赖昂贵大模型到使用轻量模型的转变，显著降低推理成本与延迟，让模型能力覆盖到边缘与高并发场景，并通过暗知识传递提升小模型的独立表现，是一种以少量精度换取大幅成本收益的工程路径。
翻译
Knowledge distillation is a training method that transfers the output behavior of a large model or ensemble into a smaller model. Its core is conveying the teacher’s dark knowledge through soft labels, with capabilities covering logit distillation, intermediate layer feature alignment and distillation loss design. It solves the problem of high inference cost that prevents deploying large models at the edge or under high concurrency, but it does not solve the performance ceiling imposed by the small model’s own capacity.
At the heart of knowledge distillation is behavior transfer, which can help companies achieve the following:
Cut deployment cost substantially. With fewer parameters, inference latency and unit cost fall together, allowing operation at higher concurrency.
Convey dark knowledge. The teacher’s output distribution carries relative relations between classes, which is denser information than hard labels.
Let small models beat independent training at the same size. Under the distillation objective, students often approach the teacher and sometimes surpass it on specific tasks.
Make deployment location flexible. Distilled models run independently on edge devices, on-device or under constrained compute, no longer depending on cloud calls.
In short, knowledge distillation can help enterprises shift model deployment from depending on expensive large models to using lightweight ones, significantly reduce inference cost and latency, extend model capability to edge and high-concurrency scenarios, and improve small-model standalone performance through dark knowledge transfer — an engineering path trading a little accuracy for large cost savings.
