Awesome Knowledge-Distillation

Different forms of knowledge

Self-Knowledge Distillation

[ICCV 2019] Be Your Own Teacher: Improve the Performance of CNN via Self Distillation

• Zhang, Linfeng, et al. “Be your own teacher: Improve the performance of convolutional neural networks via self distillation.” Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.
• code: https://github.com/ArchipLab-LinfengZhang/pytorch-self-distillation-final
• blog: [ICCV 2019] Be Your Own Teacher: Improve the Performance of CNN via Self Distillation

[CVPR 2020] Regularizing Class-wise Predictions via Self-knowledge Distillation

• Yun, Sukmin, et al. “Regularizing class-wise predictions via self-knowledge distillation.” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.
• code: https://github.com/alinlab/cs-kd
• blog: [CVPR 2020] Regularizing Class-Wise Predictions via Self-Knowledge Distillation

[CVPR 2021] Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillation

• Ji, Mingi, et al. “Refine myself by teaching myself: Feature refinement via self-knowledge distillation.” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.
• code: https://github.com/MingiJi/FRSKD
• blog: [CVPR 2021] Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillation

KD for ViTs

[ICML 2021] Training data-efficient image transformers & distillation through attention

• Touvron, Hugo, et al. “Training data-efficient image transformers & distillation through attention.” International conference on machine learning. PMLR, 2021.
• code: https://github.com/facebookresearch/deit
• blog: [ICML 2021] Training data-efficient image transformers & distillation through attention

References

• survey: https://github.com/FLHonker/Awesome-Knowledge-Distillation