# 📄 Multimodal Prompt

## Paper List

<div style="line-height:0.2em;">


[**InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output**](https://arxiv.org/abs/2407.03320) （**2024.07.03**）

<font color="gray">Pan Zhang, Xiao-wen Dong, Yuhang Zang, Yuhang Cao, Rui Qian, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-4-red)  [![](https://img.shields.io/badge/Github%20Stars-2.0k-blue)](https://github.com/internlm/internlm-xcomposer)

---

[**LLaRA: Supercharging Robot Learning Data for Vision-Language Policy**](https://arxiv.org/abs/2406.20095) （**2024.06.28**）

<font color="gray">Xiang Li, Cristina Mata, Jong Sung Park, Kumara Kahatapitiya, Yoo Sung Jang, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-3-red)  [![](https://img.shields.io/badge/Github%20Stars-77-blue)](https://github.com/lostxine/llara)

---

[**Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs**](https://arxiv.org/abs/2406.20098) （**2024.06.28**）

<font color="gray">Sukmin Yun, Haokun Lin, Rusiru Thushara, Mohammad Qazim Bhat, Yongxin Wang, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-33-blue)](https://github.com/mbzuai-llm/web2code)

---

[**LLaVolta: Efficient Multi-modal Models via Stage-wise Visual Context Compression**](https://arxiv.org/abs/2406.20092) （**2024.06.28**）

<font color="gray">Jieneng Chen, Luoxin Ye, Ju He, Zhao-Yang Wang, Daniel Khashabi, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-20-blue)](https://github.com/beckschen/llavolta)

---

[**Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs**](https://arxiv.org/abs/2406.16860) （**2024.06.24**）

<font color="gray">Shengbang Tong, Ellis L Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-67-red)  [![](https://img.shields.io/badge/Github%20Stars-1.4k-blue)](https://github.com/cambrian-mllm/cambrian)

---

[**VoCo-LLaMA: Towards Vision Compression with Large Language Models**](https://arxiv.org/abs/2406.12275) （**2024.06.18**）

<font color="gray">Xubing Ye, Yukang Gan, Xiaoke Huang, Yixiao Ge, Ying Shan, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-10-red)  [![](https://img.shields.io/badge/Github%20Stars-58-blue)](https://github.com/Yxxxb/VoCo-LLaMA)

---

[**Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Models**](https://arxiv.org/abs/2406.08487) （**2024.06.12**）

<font color="gray">Yi-Fan Zhang, Qingsong Wen, Chaoyou Fu, Xue Wang, Zhang Zhang, etc </font>

![](https://img.shields.io/badge/Citations-1-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-11-red)  [![](https://img.shields.io/badge/Github%20Stars-90-blue)](https://github.com/yfzhang114/slime)

---

[**An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models**](https://arxiv.org/abs/2406.05130) （**2024.06.07**）

<font color="gray">Xiongtao Zhou, Jie He, Yuhua Ke, Guangyao Zhu, V'ictor Guti'errez-Basulto, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-1-red)

---

[**Leveraging Visual Tokens for Extended Text Contexts in Multi-Modal Learning**](https://arxiv.org/abs/2406.02547) （**2024.06.04**）

<font color="gray">Alex Jinpeng Wang, Linjie Li, Yiqi Lin, Min Li, Lijuan Wang, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-6-blue)](https://github.com/showlab/VisInContext)

---

[**DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models**](https://doi.org/10.48550/arXiv.2405.20985) （**2024.05.31**）

<font color="gray">Linli Yao, Lei Li, Shuhuai Ren, Lean Wang, Yuanxin Liu, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis**](https://doi.org/10.48550/arXiv.2405.21075) （**2024.05.31**）

<font color="gray">Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-3-green)

---

[**ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Models**](https://doi.org/10.48550/arXiv.2405.15738) （**2024.05.24**）

<font color="gray">Chunjiang Ge, Sijie Cheng, Ziming Wang, Jiale Yuan, Yuan Gao, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-1-green)  [![](https://img.shields.io/badge/Github%20Stars-78-blue)](https://github.com/alibaba/conv-llava)

---

[**Prompt-Aware Adapter: Towards Learning Adaptive Visual Tokens for Multimodal Large Language Models**](https://doi.org/10.48550/arXiv.2405.15684) （**2024.05.24**）

<font color="gray">Yue Zhang, Hehe Fan, Yi Yang .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**Probing Multimodal LLMs as World Models for Driving**](https://doi.org/10.48550/arXiv.2405.05956) （**2024.05.09**）

<font color="gray">Shiva Sreeram, T. Wang, Alaa Maalouf, G. Rosman, S. Karaman, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-31-blue)](https://github.com/sreeramsa/drivesim)

---

[**Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning**](https://doi.org/10.48550/arXiv.2405.05615) （**2024.05.09**）

<font color="gray">Shibo Jie, Yehui Tang, Ning Ding, Zhi-Hong Deng, Kai Han, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-1-green)  [![](https://img.shields.io/badge/Github%20Stars-38-blue)](https://github.com/jieshibo/memvp)

---

[**Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers**](https://doi.org/10.48550/arXiv.2405.05945) （**2024.05.09**）

<font color="gray">Peng Gao, Le Zhuo, Ziyi Lin, Chris Liu, Junsong Chen, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-8-green)

---

[**Auto-Encoding Morph-Tokens for Multimodal LLM**](https://doi.org/10.48550/arXiv.2405.01926) （**2024.05.03**）

<font color="gray">Kaihang Pan, Siliang Tang, Juncheng Li, Zhaoyu Fan, Wei Chow, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-2-green)  [![](https://img.shields.io/badge/Github%20Stars-37-blue)](https://github.com/dcdmllm/morphtokens)

---

[**EALD-MLLM: Emotion Analysis in Long-sequential and De-identity videos with Multi-modal Large Language Model**](https://doi.org/10.48550/arXiv.2405.00574) （**2024.05.01**）

<font color="gray">Deng Li, Xin Liu, Bohao Xing, Baiqiang Xia, Yuan Zong, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**CofiPara: A Coarse-to-fine Paradigm for Multimodal Sarcasm Target Identification with Large Multimodal Models**](https://doi.org/10.48550/arXiv.2405.00390) （**2024.05.01**）

<font color="gray">Hongzhan Lin, Zixin Chen, Ziyang Luo, Mingfei Cheng, Jing Ma, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-1-green)  [![](https://img.shields.io/badge/Github%20Stars-4-blue)](https://github.com/lbotirx/cofipara)

---

[**Training-Free Unsupervised Prompt for Vision-Language Models**](https://doi.org/10.48550/arXiv.2404.16339) （**2024.04.25**）

<font color="gray">Sifan Long, Linbin Wang, Zhen Zhao, Zichang Tan, Yiming Wu, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-5-blue)](https://github.com/wlb12345/tfup)

---

[**AAPL: Adding Attributes to Prompt Learning for Vision-Language Models**](https://doi.org/10.48550/arXiv.2404.16804) （**2024.04.25**）

<font color="gray">Gahyeon Kim, Sohee Kim, Seokju Lee .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-26-blue)](https://github.com/Gahyeonkim09/AAPL)

---

[**Cantor: Inspiring Multimodal Chain-of-Thought of MLLM**](https://doi.org/10.48550/arXiv.2404.16033) （**2024.04.24**）

<font color="gray">Timin Gao, Peixian Chen, Mengdan Zhang, Chaoyou Fu, Yunhang Shen, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-1-green)

---

[**MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI**](https://doi.org/10.48550/arXiv.2404.16006) （**2024.04.24**）

<font color="gray">Kaining Ying, Fanqing Meng, Jin Wang, Zhiqiang Li, Han Lin, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-18-green)

---

[**Unified Scene Representation and Reconstruction for 3D Large Language Models**](https://doi.org/10.48550/arXiv.2404.13044) （**2024.04.19**）

<font color="gray">Tao Chu, Pan Zhang, Xiao-wen Dong, Yuhang Zang, Qiong Liu, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**BRAVE: Broadening the visual encoding of vision-language models**](https://arxiv.org/abs/2404.07204) （**2024.04.10**）

<font color="gray">Ouguzhan Fatih Kar, A. Tonioni, Petra Poklukar, Achin Kulshrestha, Amir Zamir, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-22-red)

---

[**ORacle: Large Vision-Language Models for Knowledge-Guided Holistic OR Domain Modeling**](https://arxiv.org/abs/2404.07031) （**2024.04.10**）

<font color="gray">Ege Ozsoy, Chantal Pellegrini, Matthias Keicher, Nassir Navab </font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**MoMA: Multimodal LLM Adapter for Fast Personalized Image Generation**](https://arxiv.org/abs/2404.05674) （**2024.04.08**）

<font color="gray">Kunpeng Song, Yizhe Zhu, Bingchen Liu, Qing Yan, A. Elgammal, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-11-red)  [![](https://img.shields.io/badge/Github%20Stars-152-blue)](https://github.com/bytedance/MoMA)

---

[**Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs**](https://arxiv.org/abs/2404.05719) （**2024.04.08**）

<font color="gray">Keen You, Haotian Zhang, E. Schoop, Floris Weers, Amanda Swearngin, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-57-red)

---

[**MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens**](https://arxiv.org/abs/2404.03413) （**2024.04.04**）

<font color="gray">Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Deyao Zhu, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-23-red)  [![](https://img.shields.io/badge/Github%20Stars-442-blue)](https://github.com/Vision-CAIR/MiniGPT4-video)

---

[**ViTamin: Designing Scalable Vision Models in the Vision-Language Era**](https://arxiv.org/abs/2404.02132) （**2024.04.02**）

<font color="gray">Jienneg Chen, Qihang Yu, Xiaohui Shen, Alan Yuille, Liang-Chieh Chen </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-16-red)  [![](https://img.shields.io/badge/Github%20Stars-147-blue)](https://github.com/beckschen/vitamin)

---

[**Segment Any 3D Object with Language**](https://arxiv.org/abs/2404.02157) （**2024.04.02**）

<font color="gray">Seungjun Lee, Yuyang Zhao, Gim Hee Lee </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-7-red)

---

[**Iterated Learning Improves Compositionality in Large Vision-Language Models**](https://arxiv.org/abs/2404.02145) （**2024.04.02**）

<font color="gray">Chenhao Zheng, Jieyu Zhang, Aniruddha Kembhavi, Ranjay Krishna </font>

![](https://img.shields.io/badge/Citations-1-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-9-red)

---

[**Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models**](https://doi.org/10.48550/arXiv.2403.18814) （**2024.03.27**）

<font color="gray">Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-3-green)  [![](https://img.shields.io/badge/Github%20Stars-3.1k-blue)](https://github.com/dvlab-research/minigemini)

---

[**Visual CoT: Unleashing Chain-of-Thought Reasoning in Multi-Modal Language Models**](https://arxiv.org/abs/2403.16999) （**2024.03.25**）

<font color="gray">Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-16-red)  [![](https://img.shields.io/badge/Github%20Stars-63-blue)](https://github.com/deepcs233/visual-cot)

---

[**Hierarchical Text-to-Vision Self Supervised Alignment for Improved Histopathology Representation Learning**](https://arxiv.org/abs/2403.14616) （**2024.03.21**）

<font color="gray">Hasindri Watawana, Kanchana Ranasinghe, Tariq Mahmood, Muzammal Naseer, Salman Khan, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-19-blue)](https://github.com/hasindri/hlss)

---

[**MyVLM: Personalizing VLMs for User-Specific Queries**](https://arxiv.org/abs/2403.14599) （**2024.03.21**）

<font color="gray">Yuval Alaluf, Elad Richardson, Sergey Tulyakov, Kfir Aberman, Daniel Cohen-Or </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-13-red)

---

[**MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?**](https://arxiv.org/abs/2403.14624) （**2024.03.21**）

<font color="gray">Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-17-red)

---

[**PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model**](https://arxiv.org/abs/2403.14598) （**2024.03.21**）

<font color="gray">Zheng Zhang, Yeyao Ma, Enming Zhang, Xiang Bai </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-8-red)  [![](https://img.shields.io/badge/Github%20Stars-141-blue)](https://github.com/zamling/psalm)

---

[**SC-Tune: Unleashing Self-Consistent Referential Comprehension in Large Vision Language Models**](https://arxiv.org/abs/2403.13263) （**2024.03.20**）

<font color="gray">Tongtian Yue, Jie Cheng, Longteng Guo, Xingyuan Dai, Zijia Zhao, etc </font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**The First to Know: How Token Distributions Reveal Hidden Knowledge in Large Vision-Language Models?**](https://arxiv.org/abs/2403.09037) （**2024.03.14**）

<font color="gray">Qinyu Zhao, Ming Xu, Kartik Gupta, Akshay Asthana, Liang Zheng, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-3-red)  [![](https://img.shields.io/badge/Github%20Stars-8-blue)](https://github.com/qinyu-allen-zhao/lvlm-lp)

---

[**3D-VLA: A 3D Vision-Language-Action Generative World Model**](https://arxiv.org/abs/2403.09631) （**2024.03.14**）

<font color="gray">Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, etc </font>

![](https://img.shields.io/badge/Citations-1-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-24-red)

---

[**UniCode: Learning a Unified Codebook for Multimodal Large Language Models**](https://arxiv.org/abs/2403.09072) （**2024.03.14**）

<font color="gray">Sipeng Zheng, Bohan Zhou, Yicheng Feng, Ye Wang, Zongqing Lu </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-5-red)

---

[**DeepSeek-VL: Towards Real-World Vision-Language Understanding**](https://arxiv.org/abs/2403.05525) （**2024.03.08**）

<font color="gray">Haoyu Lu, Wen Liu, Bo Zhang, Bing-Li Wang, Kai Dong, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-61-red)  [![](https://img.shields.io/badge/Github%20Stars-1.9k-blue)](https://github.com/deepseek-ai/deepseek-vl)

---

[**VLM-PL: Advanced Pseudo Labeling approach Class Incremental Object Detection with Vision-Language Model**](https://arxiv.org/abs/2403.05346) （**2024.03.08**）

<font color="gray">Junsu Kim, Yunhoe Ku, Jihyeon Kim, Junuk Cha, Seungryul Baek </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-5-red)

---

[**TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document**](https://arxiv.org/abs/2403.04473) （**2024.03.07**）

<font color="gray">Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-33-red)  [![](https://img.shields.io/badge/Github%20Stars-1.5k-blue)](https://github.com/yuliang-liu/monkey)

---

[**Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models**](https://arxiv.org/abs/2403.03003) （**2024.03.05**）

<font color="gray">Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, etc </font>

![](https://img.shields.io/badge/Citations-1-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-21-red)  [![](https://img.shields.io/badge/Github%20Stars-188-blue)](https://github.com/luogen1996/llava-hr)

---

[**RegionGPT: Towards Region Understanding Vision Language Model**](https://arxiv.org/abs/2403.02330) （**2024.03.04**）

<font color="gray">Qiushan Guo, Shalini De Mello, Hongxu Yin, Wonmin Byeon, Ka Chun Cheung, etc </font>

![](https://img.shields.io/badge/Citations-1-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-14-red)

---

[**Non-autoregressive Sequence-to-Sequence Vision-Language Models**](https://arxiv.org/abs/2403.02249) （**2024.03.04**）

<font color="gray">Kunyu Shi, Qi Dong, Luis Goncalves, Zhuowen Tu, S. Soatto </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-8-red)

---

[**Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers**](https://arxiv.org/abs/2402.19479) （**2024.02.29**）

<font color="gray">Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, etc </font>

![](https://img.shields.io/badge/Citations-3-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-32-red)  [![](https://img.shields.io/badge/Github%20Stars-438-blue)](https://github.com/snap-research/panda-70m)

---

[**Tower: An Open Multilingual Large Language Model for Translation-Related Tasks**](https://doi.org/10.48550/arXiv.2402.17733) （**2024.02.27**）

<font color="gray">Duarte M. Alves, José P. Pombal, Nuno M. Guerreiro, Pedro H. Martins, Joao Alves, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-1-green)  [![](https://img.shields.io/badge/Github%20Stars-498-blue)](https://github.com/epfllm/megatron-llm)

---

[**ShapeLLM: Universal 3D Object Understanding for Embodied Interaction**](https://doi.org/10.48550/arXiv.2402.17766) （**2024.02.27**）

<font color="gray">Zekun Qi, Runpei Dong, Shaochen Zhang, Haoran Geng, Chunrui Han, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-1-green)  [![](https://img.shields.io/badge/Github%20Stars-78-blue)](https://github.com/qizekun/ShapeLLM)

---

[**VRP-SAM: SAM with Visual Reference Prompt**](https://doi.org/10.48550/arXiv.2402.17726) （**2024.02.27**）

<font color="gray">Yanpeng Sun, Jiahui Chen, Shan Zhang, Xinyu Zhang, Qiang Chen, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**GROUNDHOG: Grounding Large Language Models to Holistic Segmentation**](https://arxiv.org/abs/2402.16846) （**2024.02.26**）

<font color="gray">Yichi Zhang, Ziqiao Ma, Xiaofeng Gao, Suhaila Shakiah, Qiaozi Gao, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-21-red)

---

[**Genie: Generative Interactive Environments**](https://arxiv.org/abs/2402.15391) （**2024.02.23**）

<font color="gray">Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-103-red)

---

[**LLMBind: A Unified Modality-Task Integration Framework**](https://doi.org/10.48550/arXiv.2402.14891) （**2024.02.22**）

<font color="gray">Bin Zhu, Peng Jin, Munan Ning, Bin Lin, Jinfa Huang, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-1-green)  [![](https://img.shields.io/badge/Github%20Stars-11-blue)](https://github.com/PKU-YuanGroup/LLMBind)

---

[**How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Prompts**](https://arxiv.org/abs/2402.13220) （**2024.02.20**）

<font color="gray">Yusu Qian, Haotian Zhang, Yinfei Yang, Zhe Gan </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-19-red)  [![](https://img.shields.io/badge/Github%20Stars-8-blue)](https://github.com/qinyu-allen-zhao/lvlm-lp)

---

[**Video ReCap: Recursive Captioning of Hour-Long Videos**](https://arxiv.org/abs/2402.13250) （**2024.02.20**）

<font color="gray">Md Mohaiminul Islam, Ngan Ho, Xitong Yang, Tushar Nagarajan, Lorenzo Torresani, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-20-red)  [![](https://img.shields.io/badge/Github%20Stars-144-blue)](https://github.com/md-mohaiminul/VideoRecap)

---

[**An Empirical Study Into What Matters for Calibrating Vision-Language Models**](https://doi.org/10.48550/arXiv.2402.07417) （**2024.02.12**）

<font color="gray">Weijie Tu, Weijian Deng, Dylan Campbell, Stephen Gould, Tom Gedeon .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models**](https://doi.org/10.48550/arXiv.2402.05935) （**2024.02.08**）

<font color="gray">Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-1-green)  [![](https://img.shields.io/badge/Github%20Stars-2.6k-blue)](https://github.com/alpha-vllm/llama2-accessory)

---

[**Binding Touch to Everything: Learning Unified Multimodal Tactile Representations**](https://doi.org/10.48550/arXiv.2401.18084) （**2024.01.31**）

<font color="gray">Fengyu Yang, Chao Feng, Ziyang Chen, Hyoungseob Park, Daniel Wang, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-1-green)

---

[**InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model**](https://doi.org/10.48550/arXiv.2401.16420) （**2024.01.29**）

<font color="gray">Xiao-wen Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-4-green)  [![](https://img.shields.io/badge/Github%20Stars-2.0k-blue)](https://github.com/internlm/internlm-xcomposer)

---

[**Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities**](https://doi.org/10.48550/arXiv.2401.14405) （**2024.01.25**）

<font color="gray">Yiyuan Zhang, Xiaohan Ding, Kaixiong Gong, Yixiao Ge, Ying Shan, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-83-blue)](https://github.com/ailab-cvc/m2pt)

---

[**MM-LLMs: Recent Advances in MultiModal Large Language Models**](https://doi.org/10.48550/arXiv.2401.13601) （**2024.01.24**）

<font color="gray">Duzhen Zhang, Yahan Yu, Chenxing Li, Jiahua Dong, Dan Su, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-3-green)

---

[**Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models**](https://doi.org/10.48550/arXiv.2401.03105) （**2024.01.06**）

<font color="gray">Xin He, Longhui Wei, Lingxi Xie, Qi Tian .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**Learning to Prompt with Text Only Supervision for Vision-Language Models**](https://doi.org/10.48550/arXiv.2401.02418) （**2024.01.04**）

<font color="gray">Muhammad Uzair Khattak, Muhammad Ferjad Naeem, Muzammal Naseer, L. V. Gool, F. Tombari .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-76-blue)](https://github.com/muzairkhattak/protext)

---

[**Instruct-Imagen: Image Generation with Multi-modal Instruction**](https://doi.org/10.48550/arXiv.2401.01952) （**2024.01.03**）

<font color="gray">Hexiang Hu, Kelvin C.K. Chan, Yu-Chuan Su, Wenhu Chen, Yandong Li, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-2-green)

---

[**G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Model**](https://arxiv.org/abs/2312.11370) （**2023.12.18**）

<font color="gray">Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-12-red)  [![](https://img.shields.io/badge/Github%20Stars-106-blue)](https://github.com/pipilurj/g-llava)

---

[**Lyrics: Boosting Fine-grained Language-Vision Alignment and Comprehension via Semantic-aware Visual Objects**](https://doi.org/10.48550/arXiv.2312.05278) （**2023.12.08**）

<font color="gray">Junyu Lu, Ruyi Gan, Di Zhang, Xiaojun Wu, Ziwei Wu, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-2-green)

---

[**LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models**](https://arxiv.org/abs/2312.02949) （**2023.12.05**）

<font color="gray">Hao Zhang, Hongyang Li, Feng Li, Tianhe Ren, Xueyan Zou, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-28-red)  [![](https://img.shields.io/badge/Github%20Stars-296-blue)](https://github.com/ux-decoder/llava-grounding)

---

[**Sequential Modeling Enables Scalable Learning for Large Vision Models**](https://arxiv.org/abs/2312.00785) （**2023.12.01**）

<font color="gray">Yutong Bai, Xinyang Geng, K. Mangalam, Amir Bar, Alan Yuille, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-219-red)  [![](https://img.shields.io/badge/Github%20Stars-1.7k-blue)](https://github.com/ytongbai/LVM)

---

[**LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models**](https://doi.org/10.48550/arXiv.2311.17043) （**2023.11.28**）

<font color="gray">Yanwei Li, Chengyao Wang, Jiaya Jia .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-622-blue)](https://github.com/dvlab-research/llama-vid)

---

[**MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers**](https://doi.org/10.48550/arXiv.2311.15475) （**2023.11.27**）

<font color="gray">Yawar Siddiqui, A. Alliegro, Alexey Artemov, Tatiana Tommasi, Daniele Sirigatti, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-646-blue)](https://github.com/lucidrains/meshgpt-pytorch)

---

[**An Embodied Generalist Agent in 3D World**](https://doi.org/10.48550/arXiv.2311.12871) （**2023.11.18**）

<font color="gray">Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-1-green)  [![](https://img.shields.io/badge/Github%20Stars-282-blue)](https://github.com/embodied-generalist/embodied-generalist)

---

[**Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning**](https://doi.org/10.48550/arXiv.2311.10709) （**2023.11.17**）

<font color="gray">Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, S. Azadi, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-2-green)

---

[**MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning**](https://doi.org/10.48550/arXiv.2311.10537) （**2023.11.16**）

<font color="gray">Xiangru Tang, Anni Zou, Zhuosheng Zhang, Yilun Zhao, Xingyao Zhang, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-188-blue)](https://github.com/gersteinlab/medagents)

---

[**Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding**](https://doi.org/10.48550/arXiv.2311.08046) （**2023.11.14**）

<font color="gray">Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, Li Yuan .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-3-green)  [![](https://img.shields.io/badge/Github%20Stars-711-blue)](https://github.com/pku-yuangroup/chat-univi)

---

[**EviPrompt: A Training-Free Evidential Prompt Generation Method for Segment Anything Model in Medical Images**](https://doi.org/10.48550/arXiv.2311.06400) （**2023.11.10**）

<font color="gray">Yinsong Xu, Jiaqi Tang, Aidong Men, Qingchao Chen .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model**](https://doi.org/10.48550/arXiv.2311.05348) （**2023.11.09**）

<font color="gray">Jinjin Xu, Liwu Xu, Yuzhe Yang, Xiang Li, Yanchun Xie, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-2-green)  [![](https://img.shields.io/badge/Github%20Stars-120-blue)](https://github.com/OPPOMKLab/u-LLaVA)

---

[**Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challenges**](https://doi.org/10.48550/arXiv.2311.03287) （**2023.11.06**）

<font color="gray">Chenhang Cui, Yiyang Zhou, Xinyu Yang, Shirley Wu, Linjun Zhang, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-5-green)  [![](https://img.shields.io/badge/Github%20Stars-51-blue)](https://github.com/gzcch/bingo)

---

[**Levels of AGI: Operationalizing Progress on the Path to AGI**](https://doi.org/10.48550/arXiv.2311.02462) （**2023.11.04**）

<font color="gray">Meredith Ringel Morris, Jascha Narain Sohl-Dickstein, Noah Fiedel, T. Warkentin, Allan Dafoe, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-1-green)

---

[**Woodpecker: Hallucination Correction for Multimodal Large Language Models**](https://arxiv.org/abs/2310.16045) （**2023.10.24**）

<font color="gray">Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-47-red)  [![](https://img.shields.io/badge/Github%20Stars-572-blue)](https://github.com/bradyfu/woodpecker)

---

[**3D-GPT: Procedural 3D Modeling with Large Language Models**](https://doi.org/10.48550/arXiv.2310.12945) （**2023.10.19**）

<font color="gray">Chunyi Sun, Junlin Han, Weijian Deng, Xinlong Wang, Zishan Qin, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-1-green)

---

[**BiLL-VTG: Bridging Large Language Models and Lightweight Visual Tools for Video-based Texts Generation**](https://doi.org/10.48550/arXiv.2310.10586) （**2023.10.16**）

<font color="gray">Ji Qi, Kaixuan Ji, Jifan Yu, Duokang Wang, Bin Xu, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**MiniGPT-5: Interleaved Vision-and-Language Generation via Generative Vokens**](https://doi.org/10.48550/arXiv.2310.02239) （**2023.10.03**）

<font color="gray">Kaizhi Zheng, Xuehai He, Xin Eric Wang .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-832-blue)](https://github.com/eric-ai-lab/minigpt-5)

---

[**Kosmos-2.5: A Multimodal Literate Model**](https://doi.org/10.48550/arXiv.2309.11419) （**2023.09.20**）

<font color="gray">Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**Investigating the Catastrophic Forgetting in Multimodal Large Language Models**](https://doi.org/10.48550/arXiv.2309.10313) （**2023.09.19**）

<font color="gray">Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**Physically Grounded Vision-Language Models for Robotic Manipulation**](https://arxiv.org/abs/2309.02561) （**2023.09.05**）

<font color="gray">Jensen Gao, Bidipta Sarkar, F. Xia, Ted Xiao, Jiajun Wu, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-39-red)

---

[**Physically Grounded Vision-Language Models for Robotic Manipulation**](https://doi.org/10.48550/arXiv.2309.02561) （**2023.09.05**）

<font color="gray">Jensen Gao, Bidipta Sarkar, F. Xia, Ted Xiao, Jiajun Wu, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-1-green)

---

[**Point-Bind&Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following**](https://arxiv.org/abs/2309.00615) （**2023.09.01**）

<font color="gray">Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-44-red)  [![](https://img.shields.io/badge/Github%20Stars-382-blue)](https://github.com/ziyuguo99/point-bind_point-llm)

---

[**PointLLM: Empowering Large Language Models to Understand Point Clouds**](https://doi.org/10.48550/arXiv.2308.16911) （**2023.08.31**）

<font color="gray">Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-14-green)  [![](https://img.shields.io/badge/Github%20Stars-427-blue)](https://github.com/openrobotlab/pointllm)

---

[**PE-MED: Prompt Enhancement for Interactive Medical Image Segmentation**](https://doi.org/10.48550/arXiv.2308.13746) （**2023.08.26**）

<font color="gray">Ao Chang, Xing Tao, Xin Yang, Yuhao Huang, Xinrui Zhou, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**SeamlessM4T-Massively Multilingual & Multimodal Machine Translation**](https://doi.org/10.48550/arXiv.2308.11596) （**2023.08.22**）

<font color="gray">Seamless Communication, Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-1-green)  [![](https://img.shields.io/badge/Github%20Stars-10.5k-blue)](https://github.com/facebookresearch/seamless_communication)

---

[**Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenes**](https://doi.org/10.48550/arXiv.2308.08769) （**2023.08.17**）

<font color="gray">Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, Zhou Zhao .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-5-green)  [![](https://img.shields.io/badge/Github%20Stars-50-blue)](https://github.com/Chat-3D/Chat-3D)

---

[**VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use**](https://doi.org/10.48550/arXiv.2308.06595) （**2023.08.12**）

<font color="gray">Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-1-green)  [![](https://img.shields.io/badge/Github%20Stars-45-blue)](https://github.com/mlfoundations/VisIT-Bench)

---

[**3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment**](https://doi.org/10.48550/arXiv.2308.04352) （**2023.08.08**）

<font color="gray">Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-7-green)  [![](https://img.shields.io/badge/Github%20Stars-173-blue)](https://github.com/3d-vista/3D-VisTA)

---

[**UniVTG: Towards Unified Video-Language Temporal Grounding**](https://doi.org/10.48550/arXiv.2307.16715) （**2023.07.31**）

<font color="gray">Kevin Lin, Pengchuan Zhang, Joya Chen, Shraman Pramanick, Difei Gao, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-1-green)  [![](https://img.shields.io/badge/Github%20Stars-299-blue)](https://github.com/showlab/univtg)

---

[**Med-Flamingo: a Multimodal Medical Few-shot Learner**](https://doi.org/10.48550/arXiv.2307.15189) （**2023.07.27**）

<font color="gray">Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, C. Zakka, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-3-green)  [![](https://img.shields.io/badge/Github%20Stars-367-blue)](https://github.com/snap-stanford/med-flamingo)

---

[**OBJECT 3DIT: Language-guided 3D-aware Image Editing**](https://doi.org/10.48550/arXiv.2307.11073) （**2023.07.20**）

<font color="gray">Oscar Michel, Anand Bhattad, Eli VanderBilt, Ranjay Krishna, Aniruddha Kembhavi, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-15-blue)](https://github.com/allenai/object-edit)

---

[**Brain2Music: Reconstructing Music from Human Brain Activity**](https://doi.org/10.48550/arXiv.2307.11078) （**2023.07.20**）

<font color="gray">Timo I. Denk, Yu Takagi, Takuya Matsuyama, A. Agostinelli, Tomoya Nakai, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**(Ab)using Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs**](https://doi.org/10.48550/arXiv.2307.10490) （**2023.07.19**）

<font color="gray">Eugene Bagdasaryan, Tsung-Yin Hsieh, Ben Nassi, Vitaly Shmatikov .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-1-green)  [![](https://img.shields.io/badge/Github%20Stars-56-blue)](https://github.com/ebagdasa/multimodal_injection)

---

[**MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots**](https://arxiv.org/abs/2307.08715) （**2023.07.16**）

<font color="gray">Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, etc </font>

![](https://img.shields.io/badge/Citations-13-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-21-red)

---

[**HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Models**](https://arxiv.org/abs/2307.06949) （**2023.07.13**）

<font color="gray">Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-70-red)  [![](https://img.shields.io/badge/Github%20Stars-158-blue)](https://github.com/JiauZhang/hyperdreambooth)

---

[**Multimodal Prompt Learning for Product Title Generation with Extremely Limited Labels**](https://doi.org/10.48550/arXiv.2307.01969) （**2023.07.05**）

<font color="gray">Bang Yang, Fenglin Liu, Zheng Li, Qingyu Yin, Chenyu You, etc .  - 【Annual Meeting of the Association for Computational Linguistics】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**Multimodal Prompt Retrieval for Generative Visual Question Answering**](https://doi.org/10.48550/arXiv.2306.17675) （**2023.06.30**）

<font color="gray">Timothy Ossowski, Junjie Hu .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMs**](https://doi.org/10.48550/arXiv.2306.17842) （**2023.06.30**）

<font color="gray">Lijun Yu, Yong Cheng, Zhiruo Wang, Vivek Kumar, Wolfgang Macherey, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-1-green)

---

[**Multimodal Prompt Learning in Emotion Recognition Using Context and Audio Information**](https://doi.org/10.3390/math11132908) （**2023.06.28**）

<font color="gray">Eunseo Jeong, Gyu-Min Kim, Sangwoo Kang .  - 【Mathematics】</font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-8-red)

---

[**Palm: Predicting Actions through Language Models @ Ego4D Long-Term Action Anticipation Challenge 2023**](https://doi.org/10.48550/arXiv.2306.16545) （**2023.06.28**）

<font color="gray">Daoji Huang, Otmar Hilliges, L. Gool, Xi Wang .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-9-blue)](https://github.com/dandoge/palm)

---

[**Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic**](https://doi.org/10.48550/arXiv.2306.15195) （**2023.06.27**）

<font color="gray">Ke Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-711-blue)](https://github.com/shikras/shikra)

---

[**DocEdit: Language-Guided Document Editing**](https://doi.org/10.1609/aaai.v37i2.25282) （**2023.06.26**）

<font color="gray">Puneet Mathur, R. Jain, Jiuxiang Gu, Franck Dernoncourt, Dinesh Manocha, etc .  - 【AAAI Conference on Artificial Intelligence】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**PromptIR: Prompting for All-in-One Blind Image Restoration**](https://arxiv.org/abs/2306.13090) （**2023.06.22**）

<font color="gray">Vaishnav Potlapalli, Syed Waqas Zamir, Salman Khan, F. Khan </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-40-red)  [![](https://img.shields.io/badge/Github%20Stars-297-blue)](https://github.com/va1shn9v/promptir)

---

[**Unleashing the AI revolution: exploring the capabilities and challenges of large language models and text‐to‐image AI programs**](https://doi.org/10.1002/uog.26297) （**2023.06.17**）

<font color="gray">A. Youssef .  - 【Ultrasound in Obstetrics and Gynecology】</font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-2-red)

---

[**Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration**](https://doi.org/10.48550/arXiv.2306.09093) （**2023.06.15**）

<font color="gray">Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-1.5k-blue)](https://github.com/lyuchenyang/macaw-llm)

---

[**Towards AGI in Computer Vision: Lessons Learned from GPT and Large Language Models**](https://doi.org/10.48550/arXiv.2306.08641) （**2023.06.14**）

<font color="gray">Lingxi Xie, Longhui Wei, Xiaopeng Zhang, Kaifeng Bi, Xiaotao Gu, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-2-green)

---

[**Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding**](https://doi.org/10.48550/arXiv.2306.02858) （**2023.06.05**）

<font color="gray">Han Zhang, Xin Li, Lidong Bing .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-4-green)  [![](https://img.shields.io/badge/Github%20Stars-2.6k-blue)](https://github.com/damo-nlp-sg/video-llama)

---

[**HeadSculpt: Crafting 3D Head Avatars with Text**](https://doi.org/10.48550/arXiv.2306.03038) （**2023.06.05**）

<font color="gray">Xiaoping Han, Yukang Cao, K. Han, Xiatian Zhu, Jiankang Deng, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-3-green)

---

[**Meta-Learning For Vision-and-Language Cross-lingual Transfer**](https://arxiv.org/abs/2305.14843) （**2023.05.24**）

<font color="gray">Hanxu Hu, Frank Keller </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-4-red)

---

[**LayoutGPT: Compositional Visual Planning and Generation with Large Language Models**](https://arxiv.org/abs/2305.15393) （**2023.05.24**）

<font color="gray">Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-41-red)  [![](https://img.shields.io/badge/Github%20Stars-265-blue)](https://github.com/weixi-feng/layoutgpt)

---

[**Cream: Visually-Situated Natural Language Understanding with Contrastive Reading Model and Frozen Large Language Models**](https://arxiv.org/abs/2305.15080) （**2023.05.24**）

<font color="gray">Geewook Kim, Hodong Lee, Daehee Kim, Haeji Jung, Sanghee Park, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-19-red)

---

[**EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought**](https://arxiv.org/abs/2305.15021) （**2023.05.24**）

<font color="gray">Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-80-red)

---

[**OverPrompt: Enhancing ChatGPT Capabilities through an Efficient In-Context Learning Approach**](https://arxiv.org/abs/2305.14973) （**2023.05.24**）

<font color="gray">Jiazheng Li, Runcong Zhao, Yulan He, Lin Gui </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-1-red)

---

[**In-Context Demonstration Selection with Cross Entropy Difference**](https://arxiv.org/abs/2305.14726) （**2023.05.24**）

<font color="gray">Dan Iter, Reid Pryzant, Ruochen Xu, Shuohang Wang, Yang Liu, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-10-red)  [![](https://img.shields.io/badge/Github%20Stars-3.4k-blue)](https://github.com/microsoft/lmops)

---

[**Abductive Commonsense Reasoning Exploiting Mutually Exclusive Explanations**](https://arxiv.org/abs/2305.14618) （**2023.05.24**）

<font color="gray">Wenting Zhao, Justin T. Chiu, Claire Cardie, Alexander M. Rush </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-16-red)

---

[**Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks**](https://arxiv.org/abs/2305.14201) （**2023.05.23**）

<font color="gray">Tiedong Liu, Bryan Kian Hsiang Low </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-66-red)  [![](https://img.shields.io/badge/Github%20Stars-173-blue)](https://github.com/liutiedong/goat)

---

[**Masked Path Modeling for Vision-and-Language Navigation**](https://doi.org/10.48550/arXiv.2305.14268) （**2023.05.23**）

<font color="gray">Zi-Yi Dou, Feng Gao, Nanyun Peng .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**ChatCoT: Tool-Augmented Chain-of-Thought Reasoning on Chat-based Large Language Models**](https://arxiv.org/abs/2305.14323) （**2023.05.23**）

<font color="gray">Z. Chen, Kun Zhou, Beichen Zhang, Zheng Gong, Wayne Xin Zhao, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-27-red)  [![](https://img.shields.io/badge/Github%20Stars-41-blue)](https://github.com/rucaibox/chatcot)

---

[**LM-Switch: Lightweight Language Model Conditioning in Word Embedding Space**](https://arxiv.org/abs/2305.12798) （**2023.05.22**）

<font color="gray">Chi Han, Jialiang Xu, Manling Li, Y. Fung, Chenkai Sun, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-11-red)

---

[**Enhancing Cross-lingual Natural Language Inference by Soft Prompting with Multilingual Verbalizer**](https://arxiv.org/abs/2305.12761) （**2023.05.22**）

<font color="gray">Shuang Li, Xuming Hu, Aiwei Liu, Yawen Yang, Fukun Ma, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-12-red)  [![](https://img.shields.io/badge/Github%20Stars-2-blue)](https://github.com/thu-bpm/softmv)

---

[**A Benchmark on Extremely Weakly Supervised Text Classification: Reconcile Seed Matching and Prompting Approaches**](https://arxiv.org/abs/2305.12749) （**2023.05.22**）

<font color="gray">Zihan Wang, Tianle Wang, Dheeraj Mekala, Jingbo Shang </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-9-red)  [![](https://img.shields.io/badge/Github%20Stars-5-blue)](https://github.com/zihanwangki/x-tc)

---

[**Enhance Reasoning Ability of Visual-Language Models via Large Language Models**](https://arxiv.org/abs/2305.13267) （**2023.05.22**）

<font color="gray">Yueting Yang, Xintong Zhang, Wenjuan Han </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-3-red)

---

[**VLAB: Enhancing Video Language Pre-training by Feature Adapting and Blending**](https://doi.org/10.48550/arXiv.2305.13167) （**2023.05.22**）

<font color="gray">Xingjian He, Sihan Chen, Fan Ma, Zhicheng Huang, Xiaojie Jin, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-1-green)

---

[**InstructVid2Vid: Controllable Video Editing with Natural Language Instructions**](https://arxiv.org/abs/2305.12328) （**2023.05.21**）

<font color="gray">Bosheng Qin, Juncheng Li, Siliang Tang, Tat-Seng Chua, Yueting Zhuang </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-12-red)

---

[**Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning**](https://arxiv.org/abs/2305.12295) （**2023.05.20**）

<font color="gray">Liangming Pan, Alon Albalak, Xinyi Wang, William Yang Wang </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-74-red)  [![](https://img.shields.io/badge/Github%20Stars-201-blue)](https://github.com/teacherpeterpan/logic-llm)

---

[**LogiCoT: Logical Chain-of-Thought Instruction-Tuning Data Collection with GPT-4**](https://arxiv.org/abs/2305.12147) （**2023.05.20**）

<font color="gray">Hanmeng Liu, Zhiyang Teng, Leyang Cui, Chaoli Zhang, Qiji Zhou, etc </font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**SelfzCoT: a Self-Prompt Zero-shot CoT from Semantic-level to Code-level for a Better Utilization of LLMs**](https://doi.org/10.48550/arXiv.2305.11461) （**2023.05.19**）

<font color="gray">IokTong Lei, ZhiDong Deng .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**Controlling the Extraction of Memorized Data from Large Language Models via Prompt-Tuning**](https://doi.org/10.48550/arXiv.2305.11759) （**2023.05.19**）

<font color="gray">Mustafa Safa Ozdayi, Charith S. Peris, Jack G. M. FitzGerald, Christophe Dupuy, Jimit Majmudar, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-24-blue)](https://github.com/amazon-science/controlling-llm-memorization)

---

[**RCOT: Detecting and Rectifying Factual Inconsistency in Reasoning by Reversing Chain-of-Thought**](https://doi.org/10.48550/arXiv.2305.11499) （**2023.05.19**）

<font color="gray">Tianci Xue, Ziqi Wang, Zhenhailong Wang, Chi Han, Pengfei Yu, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**TreePrompt: Learning to Compose Tree Prompts for Explainable Visual Grounding**](https://doi.org/10.48550/arXiv.2305.11497) （**2023.05.19**）

<font color="gray">Chenchi Zhang, Jun Xiao, Lei Chen, Jian Shao, Long Chen .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**Efficient Prompting via Dynamic In-Context Learning**](https://doi.org/10.48550/arXiv.2305.11170) （**2023.05.18**）

<font color="gray">Wangchunshu Zhou, Yuchen Jiang, Ryan Cotterell, Mrinmaya Sachan .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers**](https://arxiv.org/abs/2305.07185) （**2023.05.12**）

<font color="gray">L. Yu, Daniel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-148-red)

---

[**Prompt Tuning Inversion for Text-Driven Image Editing Using Diffusion Models**](https://doi.org/10.48550/arXiv.2305.04441) （**2023.05.08**）

<font color="gray">Wenkai Dong, Song Xue, Xiaoyue Duan, Shumin Han .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**Prompt What You Need: Enhancing Segmentation in Rainy Scenes with Anchor-based Prompting**](https://doi.org/10.48550/arXiv.2305.03902) （**2023.05.06**）

<font color="gray">Xiaoyuan Guo, Xiang Wei, Q. Su, Hui-Huang Zhao, Shunli Zhan .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**Edit Everything: A Text-Guided Generative System for Images Editing**](https://arxiv.org/abs/2304.14006) （**2023.04.27**）

<font color="gray">Defeng Xie, Ruichen Wang, Jian Ma, Chen Chen, Haonan Lu, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-13-red)  [![](https://img.shields.io/badge/Github%20Stars-17-blue)](https://github.com/defengxie/edit_everything)

---

[**ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding System**](https://arxiv.org/abs/2304.14407) （**2023.04.27**）

<font color="gray">Junke Wang, Dongdong Chen, Chong Luo, Xiyang Dai, Lu Yuan, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-16-red)

---

[**mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality**](https://arxiv.org/abs/2304.14178) （**2023.04.27**）

<font color="gray">Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-150-red)  [![](https://img.shields.io/badge/Github%20Stars-2.0k-blue)](https://github.com/x-plug/mplug-owl)

---

[**Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models**](https://doi.org/10.48550/arXiv.2304.09337) （**2023.04.18**）

<font color="gray">Stephen Brade, Bryan Wang, Maurício Sousa, Sageev Oore, Tovi Grossman .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**Towards Robust Prompts on Vision-Language Models**](https://arxiv.org/abs/2304.08479) （**2023.04.17**）

<font color="gray">Jindong Gu, A. Beirami, Xuezhi Wang, Alex Beutel, Philip H. S. Torr, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-19-red)

---

[**Visual Instruction Tuning**](https://arxiv.org/abs/2304.08485) （**2023.04.17**）

<font color="gray">Haotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae Lee </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-871-red)  [![](https://img.shields.io/badge/Github%20Stars-17.9k-blue)](https://github.com/haotian-liu/LLaVA)

---

[**Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved With Text**](https://arxiv.org/abs/2304.06939) （**2023.04.14**）

<font color="gray">Wanrong Zhu, Jack Hessel, Anas Awadalla, S. Gadre, Jesse Dodge, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-62-red)  [![](https://img.shields.io/badge/Github%20Stars-883-blue)](https://github.com/allenai/mmc4)

---

[**Segment Everything Everywhere All at Once**](https://doi.org/10.48550/arXiv.2304.06718) （**2023.04.13**）

<font color="gray">Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-3-green)  [![](https://img.shields.io/badge/Github%20Stars-14.1k-blue)](https://github.com/IDEA-Research/Grounded-Segment-Anything)

---

[**Efficient Multimodal Fusion via Interactive Prompting**](https://arxiv.org/abs/2304.06306) （**2023.04.13**）

<font color="gray">Yaowei Li, Ruijie Quan, Linchao Zhu, Yezhou Yang </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-57-red)

---

[**ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning**](https://arxiv.org/abs/2304.05613) （**2023.04.12**）

<font color="gray">Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-118-red)

---

[**Prompt Pre-Training with Twenty-Thousand Classes for Open-Vocabulary Visual Recognition**](https://doi.org/10.48550/arXiv.2304.04704) （**2023.04.10**）

<font color="gray">Shuhuai Ren, Aston Zhang, Yi Zhu, Shuai Zhang, Shuai Zheng, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-249-blue)](https://github.com/amazon-science/prompt-pretraining)

---

[**Video ChatCaptioner: Towards Enriched Spatiotemporal Descriptions**](https://doi.org/10.48550/arXiv.2304.04227) （**2023.04.09**）

<font color="gray">Jun Chen, Deyao Zhu, Kilichbek Haydarov, Xiang Li, Mohamed Elhoseiny .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-3-green)  [![](https://img.shields.io/badge/Github%20Stars-447-blue)](https://github.com/vision-cair/chatcaptioner)

---

[**Vita-CLIP: Video and text adaptive CLIP via Multimodal Prompting**](https://arxiv.org/abs/2304.03307) （**2023.04.06**）



![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-60-red)  [![](https://img.shields.io/badge/Github%20Stars-100-blue)](https://github.com/talalwasim/vita-clip)

---

[**TagGPT: Large Language Models are Zero-shot Multimodal Taggers**](https://arxiv.org/abs/2304.03022) （**2023.04.06**）



![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-15-red)  [![](https://img.shields.io/badge/Github%20Stars-53-blue)](https://github.com/tencentarc/taggpt)

---

[**Segment Anything**](https://doi.org/10.48550/arXiv.2304.02643) （**2023.04.05**）

<font color="gray">A. Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-9-green)  [![](https://img.shields.io/badge/Github%20Stars-45.4k-blue)](https://github.com/facebookresearch/segment-anything)

---

[**ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding with GPT and Prototype Guidance**](https://doi.org/10.48550/arXiv.2303.16894) （**2023.03.29**）

<font color="gray">Ziyu Guo, Yiwen Tang, Renrui Zhang, Dong Wang, Zhigang Wang, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-9-green)  [![](https://img.shields.io/badge/Github%20Stars-55-blue)](https://github.com/ivan-tang-3d/viewrefer3d)

---

[**TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs**](https://doi.org/10.48550/arXiv.2303.16434) （**2023.03.29**）

<font color="gray">Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-8-green)

---

[**MEDIMP: Medical Images and Prompts for renal transplant representation learning**](https://arxiv.org/abs/2303.12445) （**2023.03.22**）

<font color="gray">Leo Milecki, Vicky Kalogeiton, Sylvain Bodard, Dany Anglicheau, Jean-Michel Correas, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-6-red)

---

[**CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognition**](https://arxiv.org/abs/2303.11313) （**2023.03.20**）



![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-41-red)  [![](https://img.shields.io/badge/Github%20Stars-209-blue)](https://github.com/deeptibhegde/clip-goes-3d)

---

[**MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action**](https://doi.org/10.48550/arXiv.2303.11381) （**2023.03.20**）

<font color="gray">Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, etc .  - 【ArXiv】</font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-913-blue)](https://github.com/microsoft/MM-REACT)

---

[**Visual Prompt Multi-Modal Tracking**](https://arxiv.org/abs/2303.10826) （**2023.03.20**）



![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-81-red)  [![](https://img.shields.io/badge/Github%20Stars-234-blue)](https://github.com/jiawen-zhu/vipt)

---

[**Audio Visual Language Maps for Robot Navigation**](https://doi.org/10.48550/arXiv.2303.07522) （**2023.03.13**）

<font color="gray">Chen Huang, Oier Mees, Andy Zeng, W. Burgard .  - 【ArXiv】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptions**](https://doi.org/10.48550/arXiv.2303.06594) （**2023.03.12**）

<font color="gray">Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaoqian Shen, Wenxuan Zhang, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-4-green)  [![](https://img.shields.io/badge/Github%20Stars-447-blue)](https://github.com/vision-cair/chatcaptioner)

---

[**Text-Visual Prompting for Efficient 2D Temporal Video Grounding**](https://doi.org/10.1109/CVPR52729.2023.01421) （**2023.03.09**）

<font color="gray">Yimeng Zhang, Xin Chen, Jinghan Jia, Sijia Liu, Ke Ding .  - 【Computer Vision and Pattern Recognition】</font>

![](https://img.shields.io/badge/Citations-3-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-47-red)  [![](https://img.shields.io/badge/Github%20Stars-12-blue)](https://github.com/intel/TVP)

---

[**Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models**](https://arxiv.org/abs/2303.04671) （**2023.03.08**）

<font color="gray">Chenfei Wu, Sheng-Kai Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-523-red)  [![](https://img.shields.io/badge/Github%20Stars-34.5k-blue)](https://github.com/microsoft/visual-chatgpt)

---

[**Multimodal Parameter-Efficient Few-Shot Class Incremental Learning**](https://doi.org/10.48550/arXiv.2303.04751) （**2023.03.08**）

<font color="gray">Marco D’Alessandro, Alberto Alonso, Enrique Calabr'es, M. Galar .  - 【ArXiv】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning**](https://arxiv.org/abs/2303.02861) （**2023.03.06**）

<font color="gray">Zhen Wang, R. Panda, Leonid Karlinsky, R. Feris, Huan Sun, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-90-red)

---

[**Multimodal Prompting with Missing Modalities for Visual Recognition**](https://doi.org/10.48550/arXiv.2303.03369) （**2023.03.06**）

<font color="gray">Yi-Lun Lee, Yi-Hsuan Tsai, Wei-Chen Chiu, Chen-Yu Lee .  - 【ArXiv】</font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-145-blue)](https://github.com/yilunlee/missing_aware_prompts)

---

[**Multimodal Chain-of-Thought Reasoning in Language Models**](https://doi.org/10.48550/arXiv.2302.00923) （**2023.02.02**）

<font color="gray">Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, G. Karypis, etc .  - 【ArXiv】</font>

![](https://img.shields.io/badge/Citations-6-green)  [![](https://img.shields.io/badge/Github%20Stars-3.7k-blue)](https://github.com/amazon-science/mm-cot)

---

[**LVP-M3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translation**](https://doi.org/10.48550/arXiv.2210.15461) （**2022.10.19**）

<font color="gray">Hongcheng Guo, Jiaheng Liu, Haoyang Huang, Jian Yang, Zhoujun Li, etc .  - 【Conference on Empirical Methods in Natural Language Processing】</font>

![](https://img.shields.io/badge/Citations-2-green)

---

[**CoHOZ: Contrasive Multimodal prompt Tuning for Hierarchical Open-set Zero-shot Recognition**](https://doi.org/10.1145/3503161.3548021) （**2022.10.10**）

<font color="gray">Ning Liao, Yifeng Liu, Li Xiaobo, Chenyi Lei, Guoxin Wang, etc .  - 【Proceedings of the 30th ACM International Conference on Multimedia】</font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-5-red)

---

[**VIMA: General Robot Manipulation with Multimodal Prompts**](https://doi.org/10.48550/arXiv.2210.03094) （**2022.10.06**）

<font color="gray">Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, etc .  - 【ArXiv】</font>

![](https://img.shields.io/badge/Citations-15-green)  [![](https://img.shields.io/badge/Github%20Stars-243-blue)](https://github.com/vimalabs/VIMABench)

---

[**Learning to Prompt for Vision-Language Models**](https://doi.org/10.1007/s11263-022-01653-1) （**2022.09.01**）

<font color="gray">Kaiyang Zhou, Jingkang Yang, Chen Change Loy, Ziwei Liu </font>

![](https://img.shields.io/badge/Citations-1-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-700-red)  [![](https://img.shields.io/badge/Github%20Stars-1.6k-blue)](https://github.com/kaiyangzhou/coop)

---

[**Visual Prompt Tuning**](https://doi.org/10.48550/arXiv.2203.12119) （**2022.03.23**）

<font color="gray">Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, S. Belongie, etc .  - 【European Conference on Computer Vision】</font>

![](https://img.shields.io/badge/Citations-104-green)  [![](https://img.shields.io/badge/Github%20Stars-518-blue)](https://github.com/KMnP/vpt)

---

[**Multimodal Few-Shot Learning with Frozen Language Models**](https://arxiv.org/abs/2106.13884) （**2021.06.25**）

<font color="gray">Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. Eslami, Oriol Vinyals, etc .  - 【Neural Information Processing Systems】</font>

![](https://img.shields.io/badge/Citations-173-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-531-red)

---

[**MPT: Multimodal Prompt Tuning for Event Detection**](https://api.semanticscholar.org/04b49c9356aed1be7b2b4aa5dccd0cfce31c416d) 



![](https://img.shields.io/badge/Citations-0-green)

---

[**Similarity-Aware Multimodal Prompt Learning for Fake News Detection**](https://doi.org/10.2139/ssrn.4347542) 

<font color="gray">Ye Jiang, Xiaomin Yu, Yimin Wang, Xiaoman Xu, Xingyi Song, etc .  - 【SSRN Electronic Journal】</font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-2-red)


</div>

# CONTINUE...
