- 2024-06-23MODEL COMPRESSION VIA DISTILLATION AND QUANTIZATION翻译
摘要：深度神经网络（DNNs）继续取得重大进展，解决了从图像分类到翻译或增强学习的任务。这个领域的一个受到广泛关注的方面是在资源受限环境中（如移动或嵌入式设备）高效执行深度模型。本文聚焦于这一问题，并提出了两种新的压缩方法，这两种方法共同利用了权重量化和大型网络（称为“教师”网络）
- 2024-04-07DISTILLM: Towards Streamlined Distillation for Large Language Models
本文是LLM系列文章，针对《DISTILLM:TowardsStreamlinedDistillationforLargeLanguageModels》的翻译。DISTILLM：面向大型语言模型的流线蒸馏摘要1引言2背景3DISTILLM4实验5分析与讨论6相关工作7结论摘要知识蒸馏（KD）被广泛用于将教师模型压缩为
- 2023-12-042023ICCV_FSI Frequency and Spatial Interactive Learning for Image Restoration in Under-Display Camer
三.Network 1. 2.FLB：没看懂是怎么分离的水平和竖直方向 3.SLB：每一层保留一半的通道特征用于细化，其余的在特征重构后输出（没看懂）。Multi-distillationNetwork 超分辨网络的Multi-distillationNetwork(2019ACMMM_LightweightImageSuper-ResolutionwithIn
- 2023-11-06Linkless Link Prediction via Relational Distillation
目录概符号说明LLP代码GuoZ.,ShiaoW.,ZhangS.,LiuY.,ChawlaN.V.,ShahN.andZhaoT.Linklesslinkpredictionviarelationaldistillation.ICML,2023.概从GNN教师模型蒸馏到MLP学生模型.符号说明\(G=(\mathcal{V,E})\),无向图;\(\mathbf{A}\in
- 2023-10-318 Innovative BERT Knowledge Distillation Papers That Have Changed The Landscape of NLP
8InnovativeBERTKnowledgeDistillationPapersThatHaveChangedTheLandscapeofNLPContemporarystate-of-the-artNLPmodelsaredifficulttobeutilizedinproduction.Knowledgedistillationofferstoolsfortacklingsuchissuesalongwithseveralothe
- 2023-10-18论文阅读：Knowledge Distillation via the Target-aware Transformer
摘要Knowledgedistillationbecomesadefactostandardtoimprovetheperformanceofsmallneuralnetworks.知识蒸馏成为提高小型神经网络性能的事实上的标准。Mostofthepreviousworksproposetoregresstherepresentationalfeaturesfromtheteachertothes
- 2023-09-26Unbiased Knowledge Distillation for Recommendation
目录概UnKD代码ChenG.,ChenJ.,FengF.,ZhouS.andHeX.Unbiasedknowledgedistillationforrecommendation.WSDM,2023.概考虑流行度偏差的知识蒸馏,应用于推荐系统.UnKDMotivation就不讲了,感觉不是很强烈.方法很简单,就是将按照流行度给items进行
- 2023-09-22DE-RRD: A Knowledge Distillation Framework for Recommender System
目录概DE-RRDDistillationExperts(DE)RelaxedRankingDistillation(RRD)代码KangS.,HwangJ.,KweonW.andYuH.DE-RRD:Aknowledgedistillationframeworkforrecommendersystem.CIKM,2020.概知识蒸馏应用于推荐系统(同时迁移隐层+输出层特征).DE-RRD
- 2023-09-22Topology Distillation for Recommender System
目录概TopologyDistillationFullTopologyDistillation(FTD)HierarchicalTopologyDistillation(HTD)代码KangS.,HwangJ.,KweonW.andYuH.Topologydistillationforrecommendersystem.KDD,2021.概一种基于关系的知识蒸馏,这种关系的处理比较特殊.Topolog
- 2023-07-25模型蒸馏
https://www.microsoft.com/en-us/research/blog/three-mysteries-in-deep-learning-ensemble-knowledge-distillation-and-self-distillation/