image captioning经典论文分类整理+部分有源码
巫马越彬
Attention-Based Methods
O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. CVPR 2015.
K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. ICML 2015.
https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Image-Captioning、https://github.com/yunjey/show-attend-and-tell
P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang. Bottom-up and top-down attention for image captioning and visual question answering. CVPR 2018. https://github.com/peteanderson80/bottom-up-attention
J. Gu, J. Cai, G. Wang, and T. Chen. Stack-captioning: Coarse-to-fine learning for image captioning. AAAI 2018.
L. Huang, W. Wang, J. Chen, and X.-Y. Wei. Attention on attention for image captioning. ICCV, 2019.
W. Jiang, L. Ma, Y.-G. Jiang, W. Liu, and T. Zhang. Recurrent fusion network for image captioning. ECCV 2018.
Attention-Based Methods that Consider Spatial and Semantic Relations between Image Elements
Image captioning: Transforming objects into words. NIPS 2019. S. Herdade, A. Kappeler, K. Boakye, and J. Soares.
X-linear attention networks for image captioning. CVPR, 2020. Y. Pan, T. Yao, Y. Li, and T. Mei. https://github.com/Panda-Peter/image-captioning
F. Liu, X. Ren, Y. Liu, K. Lei, and X. Sun. Exploring and distilling cross-modal information for image captioning. IJCAI, 2020.
Meshed-memory transformer for image captioning. CVPR 2020. M. Cornia, M. Stefanini, L. Baraldi, and R. Cucchiara.
Oscar: Object semantics aligned pre-training for vision-language tasks. ECCV 2020. X. Li, X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong, F. Wei, et al.https://github.com/microsoft/Oscar
Unified vision-language pre-training for image captioning and vqa. AAAI 2020. L. Zhou, H. Palangi, L. Zhang, H. Hu, J. Corso, and J. Gao.
Show, control and tell: A framework for generating controllable and grounded captions. CVPR 2019. M. Cornia, L. Baraldi, and R. Cucchiara.https://github.com/aimagelab/show-control-and-tell
Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning. 2017-CVPR. Jiasen Lu2, Caiming Xiong, Devi Parikh.
Graph-Based Methods for Spatial and Semantic Relations between Image Elements
Auto-encoding scene graphs for image captioning. CVPR, 2019. X. Yang, K. Tang, H. Zhang, and J. Cai.
J. Gu, S. Joty, J. Cai, H. Zhao, X. Yang, and G. Wang. Unpaired image captioning via scene graph alignments. ICCV 2019.
Yiwu Zhong, Liwei Wang, et al. Comprehensive Image Captioning via Scene Graph Decomposition. ECCV 2020.
Combining Attention-Based Methods and Graph-Based Methods
T. Yao, Y. Pan, Y. Li, and T. Mei. Exploring visual relationship for image captioning. ECCV 2018. https://github.com/airsplay/VisualRelationships
S. Chen, Q. Jin, P. Wang, and Q. Wu. Say as you wish: Fine-grained control of image caption generation with abstract scene graphs. CVPR 2020.
Convolutional-Based Methods
J. Aneja, A. Deshpande, and A. G. Schwing. Convolutional image captioning. CVPR 2018.
Q. Wang and A. B. Chan. Cnn+ cnn: Convolutional decoders for image captioning. CoRR, 2018. https://github.com/qingzwang/GHA-ImageCaptioning
Unsupervised Methods and Reinforcement Learning
C. Chen, S. Mu, W. Xiao, Z. Ye, L. Wu, and Q. Ju. Improving image captioning with conditional generative adversarial nets. AAAI 2019. https://github.com/Anjaney1999/image-captioning-seqgan
X. Liu, H. Li, J. Shao, D. Chen, and X. Wang. Show, tell and discriminate: Image captioning by self-retrieval with partially labeled data. ECCV 2018.
Towards Unsupervised Image Captioning with Shared Multimodal Embeddings. ICCV2019.
Generating Multi-Style Captions
SentiCap: Generating Image Descriptions with Sentiments. Alexander Mathews 2016. 数据集
StyleNet: Generating Attractive Visual Captions with Styles. Chuang Gan et al. CVPR 2017.
“Factual” or “Emotional”: Stylized Image Captioning with Adaptive Learning and Attention. Tianlang Chen et al. CVPR 2018.
SemStyle: Learning to Generate Stylised Image Captions using Unaligned Text. Mathews A et al. CVPR 2018.
Beyond Narrative Description: Generating Poetry from Images by Multi-Adversarial Training. ACM MM 2018.
Engaging image captioning via personality. K. Shuster, S. Humeau, H. Hu, A. Bordes, and J. Weston. CVPR 2019.
Mscap: Multi-style image captioning with unpaired stylized text. L. Guo, J. Liu, P. Yao, J. Li, and H. Lu. CVPR 2019.
Unsupervised Stylish Image Description Generation via Domain Layer Norm. Cheng-Kuan Chen et al. AAAI 2019.
MemCap: Memorizing Style Knowledge for Image Captioning. Wentian Zhao, et al. AAAI 2020.
Human-like Controllable Image Captioning with Verb-specific Semantic Roles. Long Chen, Zhihong Jiang, Jun Xiao, Wei Liu. CVPR 2021.
3M: Multi-style image caption generation using Multi-modality features under Multi-UPDOWN model. Chengxi Li and Brent Harrison. arkiv2021.
StyleM: Stylized Metrics for Image Captioning Built with Contrastive N-grams. Chengxi Li and Brent Harrison. arkiv2022. 风格化描述度量指标
Papers about image caption按年份和会议补充
(2015-2020):
CVPR 2019:
Unsupervised Image Captioning - Yang F et al, CVPR 2019. https://github.com/fengyang0317/unsupervised_captioning
Pointing Novel Objects in Image Captioning - Li Y et al, CVPR 2019.
Context and Attribute Grounded Dense Captioning - Yin G et al, CVPR 2019.
Look Back and Predict Forward in Image Captioning - Qin Y et al, CVPR 2019.
Self-critical n-step Training for Image Captioning - Gao J et al, CVPR 2019.
Intention Oriented Image Captions with Guiding Objects - Zheng Y et al, CVPR 2019.
Describing like humans: on diversity in image captioning - Wang Q et al, CVPR 2019. https://github.com/qingzwang/DiversityMetrics
Adversarial Semantic Alignment for Improved Image Captions - Dognin P et al, CVPR 2019.
Fast, Diverse and Accurate Image Captioning Guided By Part-of-Speech - Aditya D et al, CVPR 2019.
Good News, Everyone! Context driven entity-aware captioning for news images - Biten A F et al, CVPR 2019.
CapSal: Leveraging Captioning to Boost Semantics for Salient Object Detection - Zhang L et al, CVPR 2019.
Dense Relational Captioning: Triple-Stream Networks for Relationship-Based Captioning - Kim D et al, CVPR 2019.
Exact Adversarial Attack to Image Captioning via Structured Output Learning With Latent Variables - Xu Y et al, CVPR 2019.
AAAI 2019
Meta Learning for Image Captioning - Li N et al, AAAI 2019. https://github.com/facebookresearch/LaMCTS、https://github.com/linnanwang/AlphaX-NASBench101
Learning Object Context for Dense Captioning - Li X et al, AAAI 2019. https://github.com/ttengwang/ESGN
Hierarchical Attention Network for Image Captioning - Wang W et al, AAAI 2019. https://github.com/ltguo19/VSUA-Captioning
Improving Image Captioning with Conditional Generative Adversarial Nets - Chen C et al, AAAI 2019.
ICCV 2019
Hierarchy Parsing for Image Captioning - Yao T et al, ICCV 2019.
Entangled Transformer for Image Captioning - Li G et al, ICCV 2019.
Reflective Decoding Network for Image Captioning - Ke L at al, ICCV 2019.
Learning to Collocate Neural Modules for Image Captioning - Yang X et al, ICCV 2019.
NeurIPS 2019
Adaptively Aligned Image Captioning via Adaptive Attention Time - Huang L et al, NeurIPS 2019.
Variational Structured Semantic Inference for Diverse Image Captioning - Chen F et al, NeurIPS 2019.
Aligning Visual Regions and Textual Concepts for Semantic-Grounded Image Representations - Liu F et al, NeurIPS 2019.
IJCAI 2019
Image Captioning with Compositional Neural Module Networks - Tian J et al, IJCAI 2019.
Exploring and Distilling Cross-Modal Information for Image Captioning - Liu F et al, IJCAI 2019.
Swell-and-Shrink: Decomposing Image Captioning by Transformation and Summarization - Wang H et al, IJCAI 2019.
Hornet: a hierarchical offshoot recurrent network for improving person re-ID via image captioning - Yan S et al, IJCAI 2019.
AAAI 2020
MemCap: Memorizing Style Knowledge for Image Captioning - Zhao et al, AAAI 2020.
Unified Vision-Language Pre-Training for Image Captioning and VQA - Zhou L et al, AAAI 2020.
Show, Recall, and Tell: Image Captioning with Recall Mechanism - Wang L et al, AAAI 2020.
Reinforcing an Image Caption Generator using Off-line Human Feedback - Hongsuck Seo P et al, AAAI, 2020.
Interactive Dual Generative Adversarial Networks for Image Captioning - Liu et al, AAAI 2020.
Feature Deformation Meta-Networks in Image Captioning of Novel Objects - Cao et al, AAAI 2020.
Joint Commonsense and Relation Reasoning for Image and Video Captioning - Hou et al, AAAI 2020.
Learning Long- and Short-Term User Literal-Preference with Multimodal Hierarchical Transformer Network for Personalized Image Caption - Zhang et al, AAAI 2020.
CVPR 2020
Normalized and Geometry-Aware Self-Attention Network for Image Captioning - Guo L et al, CVPR 2020.
Object Relational Graph with Teacher-Recommended Learning for Video Captioning - Zhang Z et al, CVPR 2020.
More Grounded Image Captioning by Distilling Image-Text Matching Model.
Better Captioning with Sequence-Level Exploration.
ECCV 2020
Length-Controllable Image Captioning - Deng C et al, ECCV 2020.
Captioning Images Taken by People Who Are Blind - Gurari D et al, ECCV 2020.
Towards Unique and Informative Captioning of Images - Wang Z et al, ECCV 2020.
Learning Visual Representations with Caption Annotations - Sariyildiz M et al, ECCV 2020. https://github.com/MicPie/clasp
SODA: Story Oriented Dense Video Captioning Evaluation Framework - Fujita S et al, ECCV 2020.
TextCaps: a Dataset for Image Captioning with Reading Comprehension - Sidorov O et al, ECCV 2020.
Compare and Reweight: Distinctive Image Captioning Using Similar Images Sets - Wang J et al, ECCV 2020.
Learning to Generate Grounded Visual Captions without Localization Supervision - Ma C et al, ECCV 2020.
Fashion Captioning: Towards Generating Accurate Descriptions with Semantic Rewards - Yang X et al, ECCV 2020.
NeurIPS 2020
Diverse Image Captioning with Context-Object Split Latent Spaces - Mahajan S et al, NeurIPS 2020.
RATT: Recurrent Attention to Transient Tasks for Continual Image Captioning - Chiaro R et al, NeurIPS 2020.
CVPR 2021
Towards Accurate Text-based Image Captioning with Content Diversity Exploration. Guanghui Xu et al. CVPR2021.
Image Change Captioning by Learning from an Auxiliary Task. Mehrdad Hosseinzadeh and Yang Wang.
FAIEr: Fidelity and Adequacy Ensured Image Caption Evaluation. Sijin Wang et al.
Improving OCR-based Image Captioning by Incorporating Geometrical Relationship. Jing Wang et al.
Scan2Cap: Context-aware Dense Captioning in RGB-D Scans. Dave Zhenyu Chen et al.
CVPR2022
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation.Junnan Li Dongxu Li Caiming Xiong Steven Hoi.2022-CVPR. https://github.com/salesforce/BLIP
未完
2022-02-13
by littleoo
类似资料：