# 📄 Evaluation and Reliability

## Paper List

<div style="line-height:0.2em;">


[**CELLO: Causal Evaluation of Large Vision-Language Models**](https://arxiv.org/abs/2406.19131) （**2024.06.27**）

<font color="gray">Meiqi Chen, Bo Peng, Yan Zhang, Chaochao Lu </font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-4-blue)](https://github.com/opencausalab/cello)

---

[**PrExMe! Large Scale Prompt Exploration of Open Source LLMs for Machine Translation and Summarization Evaluation**](https://arxiv.org/abs/2406.18528) （**2024.06.26**）

<font color="gray">Christoph Leiter, Steffen Eger </font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**Revisiting Referring Expression Comprehension Evaluation in the Era of Large Multimodal Models**](https://arxiv.org/abs/2406.16866) （**2024.06.24**）

<font color="gray">Jierun Chen, Fangyun Wei, Jinjing Zhao, Sizhe Song, Bohuai Wu, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-5-blue)](https://github.com/jierunchen/ref-l4)

---

[**OR-Bench: An Over-Refusal Benchmark for Large Language Models**](https://doi.org/10.48550/arXiv.2405.20947) （**2024.05.31**）

<font color="gray">Justin Cui, Wei-Lin Chiang, I. Stoica, Cho-Jui Hsieh .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**TimeChara: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Models**](https://doi.org/10.48550/arXiv.2405.18027) （**2024.05.28**）

<font color="gray">Jaewoo Ahn, Taehyun Lee, Junyoung Lim, Jin-Hwa Kim, Sangdoo Yun, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**Subtle Biases Need Subtler Measures: Dual Metrics for Evaluating Representative and Affinity Bias in Large Language Models**](https://doi.org/10.48550/arXiv.2405.14555) （**2024.05.23**）

<font color="gray">Abhishek Kumar, Sarfaroz Yunusov, Ali Emami .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models**](https://doi.org/10.48550/arXiv.2405.10299) （**2024.05.16**）

<font color="gray">R. Sukthanker, Arber Zela, B. Staffler, Jorg K. H. Franke, Frank Hutter .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-1-green)  [![](https://img.shields.io/badge/Github%20Stars-9-blue)](https://github.com/automl/hw-gpt-bench)

---

[**Multimodal LLMs Struggle with Basic Visual Network Analysis: a VNA Benchmark**](https://doi.org/10.48550/arXiv.2405.06634) （**2024.05.10**）

<font color="gray">Evan M. Williams, K. Carley .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-1-blue)](https://github.com/evanup/vna_benchmark)

---

[**Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models**](https://doi.org/10.48550/arXiv.2405.02287) （**2024.05.03**）

<font color="gray">Piotr Padlewski, Max Bain, Matthew Henderson, Zhongkai Zhu, Nishant Relan, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-6-green)  [![](https://img.shields.io/badge/Github%20Stars-139-blue)](https://github.com/reka-ai/reka-vibe-eval)

---

[**Causal Evaluation of Language Models**](https://doi.org/10.48550/arXiv.2405.00622) （**2024.05.01**）

<font color="gray">Sirui Chen, Bo Peng, Meiqi Chen, Ruiqi Wang, Mengying Xu, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-1-green)  [![](https://img.shields.io/badge/Github%20Stars-42-blue)](https://github.com/OpenCausaLab/CaLM)

---

[**IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages**](https://doi.org/10.48550/arXiv.2404.16816) （**2024.04.25**）

<font color="gray">Harman Singh, Nitish Gupta, Shikhar Bharadwaj, Dinesh Tewari, Partha Talukdar .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-1-green)  [![](https://img.shields.io/badge/Github%20Stars-24-blue)](https://github.com/google-research-datasets/indic-gen-bench)

---

[**MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI**](https://doi.org/10.48550/arXiv.2404.16006) （**2024.04.24**）

<font color="gray">Kaining Ying, Fanqing Meng, Jin Wang, Zhiqiang Li, Han Lin, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-18-green)

---

[**Evaluating LLMs at Detecting Errors in LLM Responses**](https://arxiv.org/abs/2404.03602) （**2024.04.04**）

<font color="gray">Ryo Kamoi, Sarkar Snigdha Sarathi Das, Renze Lou, Jihyun Janice Ahn, Yilun Zhao, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-4-red)  [![](https://img.shields.io/badge/Github%20Stars-19-blue)](https://github.com/psunlpgroup/realmistake)

---

[**Do Large Language Models Rank Fairly? An Empirical Study on the Fairness of LLMs as Rankers**](https://arxiv.org/abs/2404.03192) （**2024.04.04**）

<font color="gray">Yuan Wang, Xuyang Wu, Hsin-Tai Wu, Zhiqiang Tao, Yi Fang </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-4-red)

---

[**Unsolvable Problem Detection: Evaluating Trustworthiness of Vision Language Models**](https://doi.org/10.48550/arXiv.2403.20331) （**2024.03.29**）

<font color="gray">Atsuyuki Miyai, Jingkang Yang, Jingyang Zhang, Yifei Ming, Qing Yu, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-51-blue)](https://github.com/atsumiyai/upd)

---

[**ERBench: An Entity-Relationship based Automatically Verifiable Hallucination Benchmark for Large Language Models**](https://arxiv.org/abs/2403.05266) （**2024.03.08**）

<font color="gray">Jio Oh, Soyeon Kim, Junseok Seo, Jindong Wang, Ruochen Xu, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-1-blue)](https://github.com/dilab-kaist/erbench)

---

[**Benchmarking the Text-to-SQL Capability of Large Language Models: A Comprehensive Evaluation**](https://arxiv.org/abs/2403.02951) （**2024.03.05**）

<font color="gray">Bin Zhang, Yuxiao Ye, Guoqing Du, Xiaoru Hu, Zhishuai Li, etc </font>

![](https://img.shields.io/badge/Citations-1-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-26-red)

---

[**Beyond Specialization: Assessing the Capabilities of MLLMs in Age and Gender Estimation**](https://arxiv.org/abs/2403.02302) （**2024.03.04**）

<font color="gray">Maksim Kuprashevich, Grigorii Alekseenko, Irina Tolstykh </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-3-red)  [![](https://img.shields.io/badge/Github%20Stars-268-blue)](https://github.com/wildchlamydia/mivolo)

---

[**A Cognitive Evaluation Benchmark of Image Reasoning and Description for Large Vision Language Models**](https://arxiv.org/abs/2402.18409) （**2024.02.28**）

<font color="gray">Xiujie Song, Mengyue Wu, Ke Zhu, Chunhao Zhang, Yanyi Chen </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-7-red)  [![](https://img.shields.io/badge/Github%20Stars-2-blue)](https://github.com/xiujiesong/cog-bench)

---

[**Evaluating Very Long-Term Conversational Memory of LLM Agents**](https://doi.org/10.48550/arXiv.2402.17753) （**2024.02.27**）

<font color="gray">A. Maharana, Dong-Ho Lee, S. Tulyakov, Mohit Bansal, Francesco Barbieri, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**Semantic Mirror Jailbreak: Genetic Algorithm Based Jailbreak Prompts Against Open-source LLMs**](https://doi.org/10.48550/arXiv.2402.14872) （**2024.02.21**）

<font color="gray">Xiaoxia Li, Siyuan Liang, Jiyi Zhang, Hansheng Fang, Aishan Liu, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarization**](https://arxiv.org/abs/2402.13249) （**2024.02.20**）

<font color="gray">Liyan Tang, Igor Shalyminov, Amy Wing-mei Wong, Jon Burnsky, Jake W. Vincent, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-11-red)  [![](https://img.shields.io/badge/Github%20Stars-22-blue)](https://github.com/amazon-science/tofueval)

---

[**How Well Can LLMs Negotiate? NegotiationArena Platform and Analysis**](https://doi.org/10.48550/arXiv.2402.05863) （**2024.02.08**）

<font color="gray">Federico Bianchi, P. Chia, Mert Yüksekgönül, Jacopo Tagliabue, Daniel Jurafsky, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-1-green)  [![](https://img.shields.io/badge/Github%20Stars-51-blue)](https://github.com/vinid/negotiationarena)

---

[**Can Large Language Models Understand Context?**](https://doi.org/10.48550/arXiv.2402.00858) （**2024.02.01**）

<font color="gray">Yilun Zhu, Joel Ruben Antony Moniz, Shruti Bhargava, Jiarui Lu, Dhivya Piraviperumal, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-1-green)

---

[**Evaluating Large Language Models for Generalization and Robustness via Data Compression**](https://doi.org/10.48550/arXiv.2402.00861) （**2024.02.01**）

<font color="gray">Yucheng Li, Yunhao Guo, Frank Guerin, Chenghua Lin .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-24-blue)](https://github.com/liyucheng09/llm-compressive)

---

[**PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models**](https://doi.org/10.48550/arXiv.2401.15042) （**2024.01.26**）

<font color="gray">Haochen Tan, Zhijiang Guo, Zhan Shi, Lu Xu, Zhili Liu, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-1-green)  [![](https://img.shields.io/badge/Github%20Stars-7-blue)](https://github.com/namco0816/proxyqa)

---

[**VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks**](https://doi.org/10.48550/arXiv.2401.13649) （**2024.01.24**）

<font color="gray">Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-1-green)

---

[**Principled Instructions Are All You Need for Questioning LLaMA-1/2, GPT-3.5/4**](https://doi.org/10.48550/arXiv.2312.16171) （**2023.12.26**）

<font color="gray">S. Bsharat, Aidar Myrzakhan, Zhiqiang Shen .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-3-green)  [![](https://img.shields.io/badge/Github%20Stars-861-blue)](https://github.com/vila-lab/atlas)

---

[**TouchStone: Evaluating Vision-Language Models by Language Models**](https://doi.org/10.48550/arXiv.2308.16890) （**2023.08.31**）

<font color="gray">Shuai Bai, Shusheng Yang, Jinze Bai, Peng Wang, Xing Zhang, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-74-blue)](https://github.com/ofa-sys/touchstone)

---

[**Shepherd: A Critic for Language Model Generation**](https://doi.org/10.48550/arXiv.2308.04592) （**2023.08.08**）

<font color="gray">Tianlu Wang, Ping Yu, Xiaoqing Tan, Sean O'Brien, Ramakanth Pasunuru, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-201-blue)](https://github.com/facebookresearch/shepherd)

---

[**Self-consistency for open-ended generations**](https://doi.org/10.48550/arXiv.2307.06857) （**2023.07.11**）

<font color="gray">Siddhartha Jain, Xiaofei Ma, Anoop Deoras, Bing Xiang .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-1-green)

---

[**Jailbroken: How Does LLM Safety Training Fail?**](https://doi.org/10.48550/arXiv.2307.02483) （**2023.07.05**）

<font color="gray">Alexander Wei, Nika Haghtalab, J. Steinhardt .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-19-blue)](https://github.com/cassidylaidlaw/hidden-context)

---

[**Towards Measuring the Representation of Subjective Global Opinions in Language Models**](https://doi.org/10.48550/arXiv.2306.16388) （**2023.06.28**）

<font color="gray">Esin Durmus, Karina Nyugen, Thomas Liao, Nicholas Schiefer, Amanda Askell, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-23-blue)](https://github.com/salt-nlp/culturebank)

---

[**On the Reliability of Watermarks for Large Language Models**](https://doi.org/10.48550/arXiv.2306.04634) （**2023.06.07**）

<font color="gray">John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-2-green)  [![](https://img.shields.io/badge/Github%20Stars-471-blue)](https://github.com/jwkirchenbauer/lm-watermarking)

---

[**SETI: Systematicity Evaluation of Textual Inference**](https://arxiv.org/abs/2305.15045) （**2023.05.24**）

<font color="gray">Xiyan Fu, Anette Frank </font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**From Words to Wires: Generating Functioning Electronic Devices from Natural Language Descriptions**](https://arxiv.org/abs/2305.14874) （**2023.05.24**）

<font color="gray">Peter Jansen </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-5-red)  [![](https://img.shields.io/badge/Github%20Stars-6-blue)](https://github.com/cognitiveailab/words2wires)

---

[**Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples**](https://arxiv.org/abs/2305.15269) （**2023.05.24**）

<font color="gray">Abulhair Saparov, Richard Yuanzhe Pang, Vishakh Padmakumar, Nitish Joshi, Seyed Mehran Kazemi, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-28-red)  [![](https://img.shields.io/badge/Github%20Stars-97-blue)](https://github.com/asaparov/prontoqa)

---

[**EvEval: A Comprehensive Evaluation of Event Semantics for Large Language Models**](https://arxiv.org/abs/2305.15268) （**2023.05.24**）

<font color="gray">Zhengwei Tao, Zhi Jin, Xiaoying Bai, Haiyan Zhao, Yanlin Feng, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-13-red)

---

[**Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructions**](https://arxiv.org/abs/2305.15083) （**2023.05.24**）

<font color="gray">Jiahuan Li, Hao Zhou, Shujian Huang, Shanbo Chen, Jiajun Chen </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-12-red)

---

[**HuatuoGPT, towards Taming Language Model to Be a Doctor**](https://arxiv.org/abs/2305.15075) （**2023.05.24**）

<font color="gray">Hongbo Zhang, Junying Chen, Feng Jiang, Fei Yu, Zhihong Chen, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-39-red)  [![](https://img.shields.io/badge/Github%20Stars-997-blue)](https://github.com/freedomintelligence/huatuogpt)

---

[**Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models**](https://arxiv.org/abs/2305.15074) （**2023.05.24**）

<font color="gray">Daman Arora, Himanshu Gaurav Singh, Mausam </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-23-red)  [![](https://img.shields.io/badge/Github%20Stars-38-blue)](https://github.com/hgaurav2k/jeebench)

---

[**Is GPT-4 a Good Data Analyst?**](https://arxiv.org/abs/2305.15038) （**2023.05.24**）

<font color="gray">Liying Cheng, Xingxuan Li, Lidong Bing </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-61-red)  [![](https://img.shields.io/badge/Github%20Stars-178-blue)](https://github.com/damo-nlp-sg/gpt4-as-dataanalyst)

---

[**ImageNetVC: Zero-Shot Visual Commonsense Evaluation on 1000 ImageNet Categories**](https://arxiv.org/abs/2305.15028) （**2023.05.24**）

<font color="gray">Heming Xia, Qingxiu Dong, Lei Li, Jingjing Xu, Ziwei Qin, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-8-red)

---

[**Sentiment Analysis in the Era of Large Language Models: A Reality Check**](https://arxiv.org/abs/2305.15005) （**2023.05.24**）

<font color="gray">Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, Lidong Bing </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-105-red)  [![](https://img.shields.io/badge/Github%20Stars-64-blue)](https://github.com/damo-nlp-sg/llm-sentiment)

---

[**A RelEntLess Benchmark for Modelling Graded Relations between Named Entities**](https://arxiv.org/abs/2305.15002) （**2023.05.24**）

<font color="gray">Asahi Ushio, Jose Camacho Collados, S. Schockaert </font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models**](https://arxiv.org/abs/2305.14985) （**2023.05.24**）

<font color="gray">Haoxuan You, Rui Sun, Zhecan Wang, Long Chen, Gengyu Wang, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-21-red)  [![](https://img.shields.io/badge/Github%20Stars-30-blue)](https://github.com/hxyou/idealgpt)

---

[**GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP**](https://arxiv.org/abs/2305.14976) （**2023.05.24**）

<font color="gray">Md Tawkat Islam Khondaker, Abdul Waheed, El Moatez Billah Nagoudi, Muhammad Abdul-Mageed </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-32-red)

---

[**Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback**](https://arxiv.org/abs/2305.14975) （**2023.05.24**）

<font color="gray">Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-34-red)

---

[**PESCO: Prompt-enhanced Self Contrastive Learning for Zero-shot Text Classification**](https://arxiv.org/abs/2305.14963) （**2023.05.24**）

<font color="gray">Yau-Shian Wang, Ta-Chung Chi, Ruohong Zhang, Yiming Yang </font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**Evaluating NLG Evaluation Metrics: A Measurement Theory Perspective**](https://arxiv.org/abs/2305.14889) （**2023.05.24**）

<font color="gray">Ziang Xiao, Susu Zhang, Vivian Lai, Q. Vera Liao </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-9-red)

---

[**ByteSized32: A Corpus and Challenge Task for Generating Task-Specific World Models Expressed as Text Games**](https://arxiv.org/abs/2305.14879) （**2023.05.24**）

<font color="gray">Ruoyao Wang, Graham Todd, Eric Yuan, Ziang Xiao, Marc-Alexandre Cot'e, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-7-red)  [![](https://img.shields.io/badge/Github%20Stars-16-blue)](https://github.com/cognitiveailab/bytesized32)

---

[**Estimating Large Language Model Capabilities without Labeled Test Data**](https://arxiv.org/abs/2305.14802) （**2023.05.24**）

<font color="gray">Harvey Yiyun Fu, Qinyuan Ye, Albert Xu, Xiang Ren, Robin Jia </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-12-red)  [![](https://img.shields.io/badge/Github%20Stars-1-blue)](https://github.com/harvey-fin/icl-estimate)

---

[**Faithful Low-Resource Data-to-Text Generation through Cycle Training**](https://arxiv.org/abs/2305.14793) （**2023.05.24**）

<font color="gray">Zhuoer Wang, Marcus Collins, Nikhita Vedula, Simone Filice, Shervin Malmasi, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-11-red)  [![](https://img.shields.io/badge/Github%20Stars-9-blue)](https://github.com/edillower/cyclenlg)

---

[**Large Language Models as Counterfactual Generator: Strengths and Weaknesses**](https://arxiv.org/abs/2305.14791) （**2023.05.24**）

<font color="gray">Yongqi Li, Mayi Xu, Xin Miao, Shen Zhou, T. Qian </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-4-red)

---

[**ChatGPT and Simple Linguistic Inferences: Blind Spots and Blinds**](https://arxiv.org/abs/2305.14785) （**2023.05.24**）

<font color="gray">Victoria Basmov, Yoav Goldberg, Reut Tsarfaty </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-9-red)

---

[**Measuring the Knowledge Acquisition-Utilization Gap in Pretrained Language Models**](https://arxiv.org/abs/2305.14775) （**2023.05.24**）

<font color="gray">Amirhossein Kazemnejad, Mehdi Rezagholizadeh, Prasanna Parthasarathi, Sarath Chandar </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-5-red)

---

[**Using Natural Language Explanations to Rescale Human Judgments**](https://arxiv.org/abs/2305.14770) （**2023.05.24**）

<font color="gray">Manya Wadhwa, Jifan Chen, Junyi Jessy Li, Greg Durrett </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-8-red)  [![](https://img.shields.io/badge/Github%20Stars-2-blue)](https://github.com/manyawadhwa/explanation_based_rescaling)

---

[**Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models**](https://arxiv.org/abs/2305.14763) （**2023.05.24**）

<font color="gray">Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-42-red)

---

[**ECHo: Event Causality Inference via Human-centric Reasoning**](https://arxiv.org/abs/2305.14740) （**2023.05.24**）

<font color="gray">Yuxi Xie, Guanzhen Li, MingSung Kan </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-3-red)

---

[**In-Context Demonstration Selection with Cross Entropy Difference**](https://arxiv.org/abs/2305.14726) （**2023.05.24**）

<font color="gray">Dan Iter, Reid Pryzant, Ruochen Xu, Shuohang Wang, Yang Liu, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-10-red)  [![](https://img.shields.io/badge/Github%20Stars-3.4k-blue)](https://github.com/microsoft/lmops)

---

[**I Spy a Metaphor: Large Language Models and Diffusion Models Co-Create Visual Metaphors**](https://arxiv.org/abs/2305.14724) （**2023.05.24**）

<font color="gray">Tuhin Chakrabarty, Arkadiy Saakyan, Olivia Winn, Artemis Panagopoulou, Yue Yang, etc </font>

![](https://img.shields.io/badge/Citations-2-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-24-red)  [![](https://img.shields.io/badge/Github%20Stars-8-blue)](https://github.com/tuhinjubcse/visualmetaphors)

---

[**Gender Biases in Automatic Evaluation Metrics: A Case Study on Image Captioning**](https://arxiv.org/abs/2305.14711) （**2023.05.24**）

<font color="gray">Haoyi Qiu, Zi-Yi Dou, Tianlu Wang, Asli Celikyilmaz, Nanyun Peng </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-9-red)

---

[**Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models**](https://arxiv.org/abs/2305.14710) （**2023.05.24**）

<font color="gray">Jiashu Xu, Mingyu Derek Ma, Fei Wang, Chaowei Xiao, Muhao Chen </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-18-red)

---

[**Can Transformers Learn to Solve Problems Recursively?**](https://arxiv.org/abs/2305.14699) （**2023.05.24**）

<font color="gray">Shizhuo Dylan Zhang, Curt Tigges, Stella Rose Biderman, M. Raginsky, T. Ringer </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-22-red)

---

[**Enabling Large Language Models to Generate Text with Citations**](https://arxiv.org/abs/2305.14627) （**2023.05.24**）

<font color="gray">Tianyu Gao, Howard Yen, Jiatong Yu, Danqi Chen </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-95-red)  [![](https://img.shields.io/badge/Github%20Stars-413-blue)](https://github.com/princeton-nlp/alce)

---

[**Attentiveness to Answer Choices Doesn't Always Entail High QA Accuracy**](https://arxiv.org/abs/2305.14596) （**2023.05.24**）

<font color="gray">Sarah Wiegreffe, Matthew Finlayson, Oyvind Tafjord, Peter Clark, Ashish Sabharwal </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-4-red)

---

[**WikiChat: A Few-Shot LLM-Based Chatbot Grounded with Wikipedia**](https://arxiv.org/abs/2305.14292) （**2023.05.23**）

<font color="gray">Sina J. Semnani, Violet Z. Yao, Heidi C. Zhang, M. Lam </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-33-red)

---

[**Efficient Open Domain Multi-Hop Question Answering with Few-Shot Data Synthesis**](https://arxiv.org/abs/2305.13691) （**2023.05.23**）

<font color="gray">Mingda Chen, Xilun Chen, Wen-tau Yih </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-8-red)

---

[**Learn from Mistakes through Cooperative Interaction with Study Assistant**](https://arxiv.org/abs/2305.13829) （**2023.05.23**）

<font color="gray">Danqing Wang, Lei Li </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-6-red)

---

[**RET-LLM: Towards a General Read-Write Memory for Large Language Models**](https://arxiv.org/abs/2305.14322) （**2023.05.23**）

<font color="gray">Ali Modarressi, Ayyoob Imani, Mohsen Fayyaz, Hinrich Schütze </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-25-red)  [![](https://img.shields.io/badge/Github%20Stars-18.4k-blue)](https://github.com/tloen/alpaca-lora)

---

[**ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding**](https://arxiv.org/abs/2305.14196) （**2023.05.23**）

<font color="gray">Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, Omer Levy </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-19-red)  [![](https://img.shields.io/badge/Github%20Stars-17-blue)](https://github.com/tau-nlp/zero_scrolls)

---

[**Evaluating Factual Consistency of Summaries with Large Language Models**](https://arxiv.org/abs/2305.14069) （**2023.05.23**）

<font color="gray">Shiqi Chen, Siyang Gao, Junxian He </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-16-red)  [![](https://img.shields.io/badge/Github%20Stars-7-blue)](https://github.com/hkust-nlp/llmeval_sum_factual)

---

[**Polyglot or Not? Measuring Multilingual Encyclopedic Knowledge Retrieval from Foundation Language Models**](https://arxiv.org/abs/2305.13675) （**2023.05.23**）

<font color="gray">Tim Schott, Daniel Furman, Shreshta Bhat </font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**Detecting and Mitigating Indirect Stereotypes in Word Embeddings**](https://arxiv.org/abs/2305.14574) （**2023.05.23**）

<font color="gray">Erin George, Joyce Chew, Deanna Needell </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-1-red)

---

[**PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documents**](https://arxiv.org/abs/2305.14564) （**2023.05.23**）

<font color="gray">Simeng Sun, Yang Liu, Shuohang Wang, Chenguang Zhu, Mohit Iyyer </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-37-red)  [![](https://img.shields.io/badge/Github%20Stars-46-blue)](https://github.com/simengsun/pearl)

---

[**Unraveling ChatGPT: A Critical Analysis of AI-Generated Goal-Oriented Dialogues and Annotations**](https://arxiv.org/abs/2305.14556) （**2023.05.23**）

<font color="gray">Tiziano Labruna, Sofia Brenna, Andrea Zaninello, Bernardo Magnini </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-10-red)

---

[**Sources of Hallucination by Large Language Models on Inference Tasks**](https://arxiv.org/abs/2305.14552) （**2023.05.23**）

<font color="gray">Nick McKenna, Tianyi Li, Liang Cheng, Mohammad Javad Hosseini, Mark Johnson, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-49-red)  [![](https://img.shields.io/badge/Github%20Stars-13-blue)](https://github.com/teddy-li/llm-nli-analysis)

---

[**LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond**](https://arxiv.org/abs/2305.14540) （**2023.05.23**）

<font color="gray">Philippe Laban, Wojciech Kry'sci'nski, Divyansh Agarwal, Alexander R. Fabbri, Caiming Xiong, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-22-red)  [![](https://img.shields.io/badge/Github%20Stars-57-blue)](https://github.com/salesforce/factualnlg)

---

[**Automatic Model Selection with Large Language Models for Reasoning**](https://arxiv.org/abs/2305.14333) （**2023.05.23**）

<font color="gray">Xu Zhao, Yuxi Xie, Kenji Kawaguchi, Junxian He, Qizhe Xie </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-18-red)  [![](https://img.shields.io/badge/Github%20Stars-28-blue)](https://github.com/xuzhao0/model-selection-reasoning)

---

[**Cascaded Beam Search: Plug-and-Play Terminology-Forcing For Neural Machine Translation**](https://arxiv.org/abs/2305.14538) （**2023.05.23**）

<font color="gray">Fr'ed'eric Odermatt, B'eni Egressy, Roger Wattenhofer </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-3-red)

---

[**CREATOR: Disentangling Abstract and Concrete Reasonings of Large Language Models through Tool Creation**](https://arxiv.org/abs/2305.14318) （**2023.05.23**）

<font color="gray">Cheng Qian, Chi Han, Yi R. Fung, Yujia Qin, Zhiyuan Liu, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-39-red)

---

[**Deduction under Perturbed Evidence: Probing Student Simulation Capabilities of Large Language Models**](https://arxiv.org/abs/2305.14507) （**2023.05.23**）

<font color="gray">Shashank Sonkar, Richard Baraniuk </font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**Prompt position really matters in few-shot and zero-shot NLU tasks**](https://arxiv.org/abs/2305.14493) （**2023.05.23**）

<font color="gray">Junyu Mao, Stuart E. Middleton, Mahesan Niranjan </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-3-red)

---

[**CGCE: A Chinese Generative Chat Evaluation Benchmark for General and Financial Domains**](https://arxiv.org/abs/2305.14471) （**2023.05.23**）

<font color="gray">Xuanyu Zhang, Bingbing Li, Qing Yang </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-3-red)  [![](https://img.shields.io/badge/Github%20Stars-930-blue)](https://github.com/duxiaoman-di/xuanyuan)

---

[**Dancing Between Success and Failure: Edit-level Simplification Evaluation using SALSA**](https://arxiv.org/abs/2305.14458) （**2023.05.23**）

<font color="gray">David Heineman, Yao Dou, Mounica Maddela, Wei Xu </font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**Pre-training Language Models for Comparative Reasoning**](https://arxiv.org/abs/2305.14457) （**2023.05.23**）

<font color="gray">Mengxia Yu, Zhihan Zhang, Wenhao Yu, Meng Jiang </font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**Having Beer after Prayer? Measuring Cultural Bias in Large Language Models**](https://arxiv.org/abs/2305.14456) （**2023.05.23**）

<font color="gray">Tarek Naous, Michael J. Ryan, Wei Xu </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-25-red)  [![](https://img.shields.io/badge/Github%20Stars-5-blue)](https://github.com/tareknaous/camel)

---

[**On Robustness of Finetuned Transformer-based NLP Models**](https://arxiv.org/abs/2305.14453) （**2023.05.23**）

<font color="gray">Pavan Kalyan Reddy Neerudu, Subba Reddy Oota, Mounika Marreddy, Venkateswara Rao Kagita, Manish Gupta </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-12-red)  [![](https://img.shields.io/badge/Github%20Stars-3-blue)](https://github.com/pavanneerudu/robustness-of-transformers-models)

---

[**Is Information Extraction Solved by ChatGPT? An Analysis of Performance, Evaluation Criteria, Robustness and Errors**](https://arxiv.org/abs/2305.14450) （**2023.05.23**）

<font color="gray">Ridong Han, T. Peng, Chaohao Yang, Benyou Wang, Lu Liu, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-35-red)  [![](https://img.shields.io/badge/Github%20Stars-118-blue)](https://github.com/ridonghan/evaluation-of-chatgpt-on-information-extraction)

---

[**Exploring Contrast Consistency of Open-Domain Question Answering Systems on Minimally Edited Questions**](https://arxiv.org/abs/2305.14441) （**2023.05.23**）

<font color="gray">Zhihan Zhang, Wenhao Yu, Zheng Ning, Mingxuan Ju, Meng Jiang </font>

![](https://img.shields.io/badge/Citations-1-green)  [![](https://img.shields.io/badge/Github%20Stars-3-blue)](https://github.com/ytyz1307zzh/minimally_edited_questions)

---

[**Domain-Expanded ASTE: Rethinking Generalization in Aspect Sentiment Triplet Extraction**](https://arxiv.org/abs/2305.14434) （**2023.05.23**）

<font color="gray">Yew Ken Chia, Hui Chen, Wei Han, Guizhen Chen, Sharifah Mahani Aljunied, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-1-red)

---

[**TaDSE: Template-aware Dialogue Sentence Embeddings**](https://arxiv.org/abs/2305.14299) （**2023.05.23**）

<font color="gray">Minsik Oh, Jiwei Li, Guoyin Wang </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-1-red)

---

[**WebIE: Faithful and Robust Information Extraction on the Web**](https://arxiv.org/abs/2305.14293) （**2023.05.23**）

<font color="gray">Chenxi Whitehouse, Clara Vania, Alham Fikri Aji, Christos Christodoulopoulos, Andrea Pierleoni </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-16-red)

---

[**Exploring Representational Disparities Between Multilingual and Bilingual Translation Models**](https://arxiv.org/abs/2305.14230) （**2023.05.23**）

<font color="gray">Neha Verma, Kenton Murray, Kevin Duh </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-7-red)

---

[**How Old is GPT?: The HumBEL Framework for Evaluating Language Models using Human Demographic Data**](https://arxiv.org/abs/2305.14195) （**2023.05.23**）

<font color="gray">Anthony Sicilia, Jennifer C. Gates, Malihe Alikhani </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-3-red)

---

[**Out-of-Distribution Generalization in Text Classification: Past, Present, and Future**](https://arxiv.org/abs/2305.14104) （**2023.05.23**）

<font color="gray">Linyi Yang, Y. Song, Xuan Ren, Chenyang Lyu, Yidong Wang, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-6-red)

---

[**Target-Agnostic Gender-Aware Contrastive Learning for Mitigating Bias in Multilingual Machine Translation**](https://arxiv.org/abs/2305.14016) （**2023.05.23**）

<font color="gray">Minwoo Lee, Hyukhun Koh, Kang-il Lee, Dongdong Zhang, Minsung Kim, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-6-red)

---

[**LLM-powered Data Augmentation for Enhanced Crosslingual Performance**](https://arxiv.org/abs/2305.14288) （**2023.05.23**）

<font color="gray">Chenxi Whitehouse, Monojit Choudhury, Alham Fikri Aji </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-36-red)

---

[**FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation**](https://arxiv.org/abs/2305.14251) （**2023.05.23**）

<font color="gray">Sewon Min, Kalpesh Krishna, Xinxi Lyu, M. Lewis, Wen-tau Yih, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-100-red)  [![](https://img.shields.io/badge/Github%20Stars-238-blue)](https://github.com/shmsw25/factscore)

---

[**On Learning to Summarize with Large Language Models as References**](https://arxiv.org/abs/2305.14239) （**2023.05.23**）

<font color="gray">Yixin Liu, Alexander R. Fabbri, Pengfei Liu, Dragomir Radev, Arman Cohan </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-38-red)  [![](https://img.shields.io/badge/Github%20Stars-40-blue)](https://github.com/yixinl7/sumllm)

---

[**Reducing Sensitivity on Speaker Names for Text Generation from Dialogues**](https://arxiv.org/abs/2305.13833) （**2023.05.23**）

<font color="gray">Qi Jia, Haifeng Tang, Kenny Q. Zhu </font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-2-blue)](https://github.com/jiaqisjtu/speakernamesensitivity)

---

[**Self-Critique Prompting with Large Language Models for Inductive Instructions**](https://arxiv.org/abs/2305.13733) （**2023.05.23**）

<font color="gray">Rui Wang, Hongru Wang, Fei Mi, Yi Chen, Ruifeng Xu, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-9-red)

---

[**ChipGPT: How far are we from natural language hardware design**](https://arxiv.org/abs/2305.14019) （**2023.05.23**）

<font color="gray">Kaiyan Chang, Ying Wang, Haimeng Ren, Mengdi Wang, Shengwen Liang, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-21-red)

---

[**Prompt-Based Monte-Carlo Tree Search for Goal-Oriented Dialogue Policy Planning**](https://arxiv.org/abs/2305.13660) （**2023.05.23**）

<font color="gray">Xiao Yu, Maximillian Chen, Zhou Yu </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-19-red)  [![](https://img.shields.io/badge/Github%20Stars-8-blue)](https://github.com/jasonyux/gdpzero)

---

[**Improving Self-training for Cross-lingual Named Entity Recognition with Contrastive and Prototype Learning**](https://arxiv.org/abs/2305.13628) （**2023.05.23**）

<font color="gray">Ran Zhou, Xin Li, Lidong Bing, E. Cambria, Chun Miao </font>

![](https://img.shields.io/badge/Citations-3-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-22-red)  [![](https://img.shields.io/badge/Github%20Stars-8-blue)](https://github.com/damo-nlp-sg/contproto)

---

[**Better Low-Resource Entity Recognition Through Translation and Annotation Fusion**](https://arxiv.org/abs/2305.13582) （**2023.05.23**）

<font color="gray">Yang Chen, Vedaant Shah, Alan Ritter </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-7-red)

---

[**Cross-functional Analysis of Generalisation in Behavioural Learning**](https://arxiv.org/abs/2305.12951) （**2023.05.22**）

<font color="gray">Pedro Henrique Luz de Araujo, Benjamin Roth </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-5-red)

---

[**Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrations**](https://arxiv.org/abs/2305.13299) （**2023.05.22**）

<font color="gray">Chenglei Si, Dan Friedman, Nitish Joshi, Shi Feng, Danqi Chen, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-34-red)  [![](https://img.shields.io/badge/Github%20Stars-7-blue)](https://github.com/noviscl/ambigprompt)

---

[**LM vs LM: Detecting Factual Errors via Cross Examination**](https://arxiv.org/abs/2305.13281) （**2023.05.22**）

<font color="gray">Roi Cohen, May Hamri, Mor Geva, A. Globerson </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-45-red)

---

[**SPARSEFIT: Few-shot Prompting with Sparse Fine-tuning for Jointly Generating Predictions and Natural Language Explanations**](https://arxiv.org/abs/2305.13235) （**2023.05.22**）

<font color="gray">Jesus Solano, Oana-Maria Camburu, Pasquale Minervini </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-4-red)

---

[**SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluation**](https://arxiv.org/abs/2305.13194) （**2023.05.22**）

<font color="gray">Elizabeth Clark, Shruti Rijhwani, Sebastian Gehrmann, Joshua Maynez, Roee Aharoni, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-14-red)

---

[**Kanbun-LM: Reading and Translating Classical Chinese in Japanese Methods by Language Models**](https://arxiv.org/abs/2305.12759) （**2023.05.22**）

<font color="gray">Hao Wang, Hirofumi Shimizu, Daisuke Kawahara </font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-11-blue)](https://github.com/nlp-waseda/kanbun-lm)

---

[**Beyond Labels: Empowering Human with Natural Language Explanations through a Novel Active-Learning Architecture**](https://arxiv.org/abs/2305.12710) （**2023.05.22**）

<font color="gray">Bingsheng Yao, Ishan Jindal, Lucian Popa, Yannis Katsis, Sayan Ghosh, etc </font>

![](https://img.shields.io/badge/Citations-1-green)

---

[**llm-japanese-dataset v0: Construction of Japanese Chat Dataset for Large Language Models and its Methodology**](https://arxiv.org/abs/2305.12720) （**2023.05.22**）

<font color="gray">Masanori Hirano, Masahiro Suzuki, Hiroki Sakaji </font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-75-blue)](https://github.com/masanorihirano/llm-japanese-dataset)

---

[**AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback**](https://arxiv.org/abs/2305.14387) （**2023.05.22**）

<font color="gray">Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-98-red)  [![](https://img.shields.io/badge/Github%20Stars-1.3k-blue)](https://github.com/tatsu-lab/alpaca_eval)

---

[**Multilingual Holistic Bias: Extending Descriptors and Patterns to Unveil Demographic Biases in Languages at Scale**](https://arxiv.org/abs/2305.13198) （**2023.05.22**）

<font color="gray">M. Costa-jussà, Pierre Yves Andrews, Eric J. M. Smith, Prangthip Hansanti, Christophe Ropers, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-6-red)

---

[**Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design**](https://arxiv.org/abs/2305.13035) （**2023.05.22**）

<font color="gray">Ibrahim M. Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, L. Beyer </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-12-red)  [![](https://img.shields.io/badge/Github%20Stars-2.0k-blue)](https://github.com/google-research/big_vision)

---

[**MultiTabQA: Generating Tabular Answers for Multi-Table Question Answering**](https://arxiv.org/abs/2305.12820) （**2023.05.22**）

<font color="gray">Vaishali Pal, Andrew Yates, E. Kanoulas, M. de Rijke </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-18-red)  [![](https://img.shields.io/badge/Github%20Stars-19-blue)](https://github.com/kolk/multitabqa)

---

[**Cross-lingual Transfer Can Worsen Bias in Sentiment Analysis**](https://arxiv.org/abs/2305.12709) （**2023.05.22**）

<font color="gray">Seraphina Goldfarb-Tarrant, Björn Ross, Adam Lopez </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-7-red)

---

[**Model Analysis&Evaluation for Ambiguous Question Answering**](https://arxiv.org/abs/2305.12483) （**2023.05.21**）

<font color="gray">Konstantinos Papakostas, Irene Papadopoulou </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-7-red)

---

[**TheoremQA: A Theorem-driven Question Answering dataset**](https://arxiv.org/abs/2305.12524) （**2023.05.21**）

<font color="gray">Wenhu Chen, Ming Yin, Max Ku, Yixin Wan, Xueguang Ma, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-24-red)  [![](https://img.shields.io/badge/Github%20Stars-152-blue)](https://github.com/wenhuchen/theoremqa)

---

[**Evaluating the Performance of Large Language Models on GAOKAO Benchmark**](https://arxiv.org/abs/2305.12474) （**2023.05.21**）

<font color="gray">Xiaotian Zhang, Chun-yan Li, Yi Zong, Zhengyu Ying, Liang He, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-9-red)  [![](https://img.shields.io/badge/Github%20Stars-489-blue)](https://github.com/openlmlab/gaokao-bench)

---

[**Can NLP Models Correctly Reason Over Contexts that Break the Common Assumptions?**](https://arxiv.org/abs/2305.12096) （**2023.05.20**）

<font color="gray">Neeraj Varshney, Mihir Parmar, Nisarg Patel, Divij Handa, Sayantan Sarkar, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-4-red)

---

[**Evaluation of medium-large Language Models at zero-shot closed book generative question answering**](https://arxiv.org/abs/2305.11991) （**2023.05.19**）

<font color="gray">Ren'e Peinl, Johannes Wirth </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-12-red)

---

[**Separating form and meaning: Using self-consistency to quantify task understanding across multiple senses**](https://doi.org/10.48550/arXiv.2305.11662) （**2023.05.19**）

<font color="gray">Xenia Ohmer, Elia Bruni, Dieuwke Hupkes .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**OPT-R: Exploring the Role of Explanations in Finetuning and Prompting for Reasoning Skills of Large Language Models**](https://arxiv.org/abs/2305.12001) （**2023.05.19**）

<font color="gray">Badr AlKhamissi, Siddharth Verma, Ping Yu, Zhijing Jin, Asli Celikyilmaz, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-15-red)

---

[**Examining the Inter-Consistency of Large Language Models: An In-depth Analysis via Debate**](https://doi.org/10.48550/arXiv.2305.11595) （**2023.05.19**）

<font color="gray">Kai Xiong, Xiao Ding, Yixin Cao, Ting Liu, Bing Qin .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**TELeR: A General Taxonomy of LLM Prompts for Benchmarking Complex Tasks**](https://doi.org/10.48550/arXiv.2305.11430) （**2023.05.19**）

<font color="gray">Shubhra (Santu) Karmaker, Dongji Feng .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-1-green)

---

[**Efficient Prompting via Dynamic In-Context Learning**](https://doi.org/10.48550/arXiv.2305.11170) （**2023.05.18**）

<font color="gray">Wangchunshu Zhou, Yuchen Jiang, Ryan Cotterell, Mrinmaya Sachan .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**Qualifying Chinese Medical Licensing Examination with Knowledge Enhanced Generative Pre-training Model**](https://doi.org/10.48550/arXiv.2305.10163) （**2023.05.17**）

<font color="gray">Jiageng Wu, X. Wu, Zhaopeng Qiu, Minghui Li, Yefeng Zheng, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-1-green)

---

[**Can Language Models Solve Graph Problems in Natural Language?**](https://doi.org/10.48550/arXiv.2305.10037) （**2023.05.17**）

<font color="gray">Heng Wang, Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, etc .  - 【arXiv.org】</font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-93-blue)](https://github.com/arthur-heng/nlgraph)

---

[**AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models**](https://arxiv.org/abs/2304.06364) （**2023.04.13**）

<font color="gray">Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-87-red)  [![](https://img.shields.io/badge/Github%20Stars-668-blue)](https://github.com/ruixiangcui/agieval)

---

[**GPTEval: NLG Evaluation using GPT-4 with Better Human Alignment**](https://arxiv.org/abs/2303.16634) （**2023.03.29**）

<font color="gray">Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, etc </font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-179-red)

---

[**How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Tasks**](https://doi.org/10.48550/arXiv.2303.00293) （**2023.03.01**）

<font color="gray">Xuanting Chen, Junjie Ye, Can Zu, Nuo Xu, Rui Zheng, etc .  - 【ArXiv】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**Bounding the Capabilities of Large Language Models in Open Text Generation with Prompt Constraints**](https://doi.org/10.48550/arXiv.2302.09185) （**2023.02.17**）

<font color="gray">Albert Lu, Hongxin Zhang, Yanzhe Zhang, Xuezhi Wang, Diyi Yang .  - 【ArXiv】</font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-27-blue)](https://github.com/salt-nlp/bound-cap-llm)

---

[**Evaluating the Robustness of Discrete Prompts**](https://doi.org/10.48550/arXiv.2302.05619) （**2023.02.11**）

<font color="gray">Yoichi Ishibashi, D. Bollegala, Katsuhito Sudoh, Satoshi Nakamura .  - 【ArXiv】</font>

![](https://img.shields.io/badge/Citations-0-green)  [![](https://img.shields.io/badge/Github%20Stars-3-blue)](https://github.com/livnlp/prompt-robustness)

---

[**Controlling for Stereotypes in Multimodal Language Model Evaluation**](https://doi.org/10.48550/arXiv.2302.01582) （**2023.02.03**）

<font color="gray">Manuj Malik, Richard Johansson .  - 【BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**Large Language Models Can Be Easily Distracted by Irrelevant Context**](https://doi.org/10.48550/arXiv.2302.00093) （**2023.01.31**）

<font color="gray">Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, etc .  - 【ArXiv】</font>

![](https://img.shields.io/badge/Citations-3-green)  [![](https://img.shields.io/badge/Github%20Stars-49-blue)](https://github.com/google-research-datasets/gsm-ic)

---

[**Emergent Analogical Reasoning in Large Language Models**](https://doi.org/10.48550/arXiv.2212.09196) （**2022.12.19**）

<font color="gray">Taylor W. Webb, K. Holyoak, Hongjing Lu .  - 【ArXiv】</font>

![](https://img.shields.io/badge/Citations-5-green)  [![](https://img.shields.io/badge/Github%20Stars-37-blue)](https://github.com/taylorwwebb/emergent_analogies_llm)

---

[**Discovering Language Model Behaviors with Model-Written Evaluations**](https://doi.org/10.48550/arXiv.2212.09251) （**2022.12.19**）

<font color="gray">Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, etc .  - 【ArXiv】</font>

![](https://img.shields.io/badge/Citations-8-green)  [![](https://img.shields.io/badge/Github%20Stars-217-blue)](https://github.com/anthropics/evals)

---

[**Constitutional AI: Harmlessness from AI Feedback**](https://doi.org/10.48550/arXiv.2212.08073) （**2022.12.15**）

<font color="gray">Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, John Kernion, etc .  - 【ArXiv】</font>

![](https://img.shields.io/badge/Citations-21-green)  [![](https://img.shields.io/badge/Github%20Stars-202-blue)](https://github.com/anthropics/constitutionalharmlessnesspaper)

---

[**On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning**](https://doi.org/10.48550/arXiv.2212.08061) （**2022.12.15**）

<font color="gray">Omar Shaikh, Hongxin Zhang, William B. Held, Michael Bernstein, Diyi Yang .  - 【ArXiv】</font>

![](https://img.shields.io/badge/Citations-2-green)  [![](https://img.shields.io/badge/Github%20Stars-23-blue)](https://github.com/salt-nlp/chain-of-thought-bias)

---

[**Demystifying Prompts in Language Models via Perplexity Estimation**](https://doi.org/10.48550/arXiv.2212.04037) （**2022.12.08**）

<font color="gray">Hila Gonen, Srini Iyer, Terra Blevins, Noah A. Smith, Luke Zettlemoyer .  - 【ArXiv】</font>

![](https://img.shields.io/badge/Citations-5-green)

---

[**Solving math word problems with process- and outcome-based feedback**](https://doi.org/10.48550/arXiv.2211.14275) （**2022.11.25**）

<font color="gray">J. Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, etc .  - 【ArXiv】</font>

![](https://img.shields.io/badge/Citations-3-green)

---

[**Holistic Evaluation of Language Models**](https://doi.org/10.48550/arXiv.2211.09110) （**2022.11.16**）

<font color="gray">Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, etc .  - 【ArXiv】</font>

![](https://img.shields.io/badge/Citations-40-green)  [![](https://img.shields.io/badge/Github%20Stars-1.8k-blue)](https://github.com/stanford-crfm/helm)

---

[**Can language models handle recursively nested grammatical structures? A case study on comparing models and humans**](https://doi.org/10.48550/arXiv.2210.15303) （**2022.10.27**）

<font color="gray">Andrew Kyle Lampinen .  - 【ArXiv】</font>

![](https://img.shields.io/badge/Citations-3-green)

---

[**Prompting GPT-3 To Be Reliable**](https://doi.org/10.48550/arXiv.2210.09150) （**2022.10.17**）

<font color="gray">Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, etc .  - 【ArXiv】</font>

![](https://img.shields.io/badge/Citations-9-green)  [![](https://img.shields.io/badge/Github%20Stars-75-blue)](https://github.com/noviscl/gpt3-reliability)

---

[**An Interpretability Evaluation Benchmark for Pre-trained Language Models**](https://doi.org/10.48550/arXiv.2207.13948) （**2022.07.28**）

<font color="gray">Ya-Ming Shen, Lijie Wang, Ying Chen, Xinyan Xiao, Jing Liu, etc .  - 【ArXiv】</font>

![](https://img.shields.io/badge/Citations-0-green)

---

[**Re-Examining Calibration: The Case of Question Answering**](https://arxiv.org/abs/2205.12507) （**2022.05.25**）

<font color="gray">Chenglei Si, Chen Zhao, Sewon Min, Jordan L. Boyd-Graber .  - 【Conference on Empirical Methods in Natural Language Processing】</font>

![](https://img.shields.io/badge/Citations-0-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-23-red)  [![](https://img.shields.io/badge/Github%20Stars-2-blue)](https://github.com/noviscl/calibrateqa)

---

[**Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing**](https://doi.org/10.48550/arXiv.2205.12253) （**2022.05.24**）

<font color="gray">Linlu Qiu, Peter Shaw, Panupong Pasupat, Tianze Shi, Jonathan Herzig, etc .  - 【Conference on Empirical Methods in Natural Language Processing】</font>

![](https://img.shields.io/badge/Citations-11-green)

---

[**The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning**](https://arxiv.org/abs/2205.03401) （**2022.05.06**）

<font color="gray">Xi Ye, Greg Durrett </font>

![](https://img.shields.io/badge/Citations-10-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-88-red)  [![](https://img.shields.io/badge/Github%20Stars-13-blue)](https://github.com/xiye17/textualexplincontext)

---

[**Training Verifiers to Solve Math Word Problems**](https://arxiv.org/abs/2110.14168) （**2021.10.27**）

<font color="gray">Karl Cobbe, V. Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, etc .  - 【ArXiv】</font>

![](https://img.shields.io/badge/Citations-181-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-272-red)  [![](https://img.shields.io/badge/Github%20Stars-934-blue)](https://github.com/openai/grade-school-math)

---

[**BBQ: A hand-built bias benchmark for question answering**](https://doi.org/10.18653/v1/2022.findings-acl.165) （**2021.10.15**）

<font color="gray">Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, etc .  - 【Findings】</font>

![](https://img.shields.io/badge/Citations-19-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-77-red)  [![](https://img.shields.io/badge/Github%20Stars-70-blue)](https://github.com/nyu-mll/bbq)

---

[**BARTScore: Evaluating Generated Text as Text Generation**](https://arxiv.org/abs/2106.11520) （**2021.06.22**）

<font color="gray">Weizhe Yuan, Graham Neubig, Pengfei Liu .  - 【Neural Information Processing Systems】</font>

![](https://img.shields.io/badge/Citations-172-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-231-red)  [![](https://img.shields.io/badge/Github%20Stars-307-blue)](https://github.com/neulab/BARTScore)

---

[**Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning**](https://doi.org/10.18653/v1/2021.acl-long.102) （**2021.06.13**）

<font color="gray">Bill Yuchen Lin, Seyeon Lee, Xiaoyang Qiao, Xiang Ren .  - 【Annual Meeting of the Association for Computational Linguistics】</font>

![](https://img.shields.io/badge/Citations-20-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-76-red)  [![](https://img.shields.io/badge/Github%20Stars-22-blue)](https://github.com/INK-USC/XCSR)

---

[**Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity**](https://doi.org/10.18653/v1/2022.acl-long.556) （**2021.04.18**）

<font color="gray">Yao Lu, Max Bartolo, Alastair Moore, S. Riedel, Pontus Stenetorp .  - 【Annual Meeting of the Association for Computational Linguistics】</font>

![](https://img.shields.io/badge/Citations-170-green)  ![](https://img.shields.io/badge/Mendeley%20Readers-310-red)  [![](https://img.shields.io/badge/Github%20Stars-451-blue)](https://github.com/RUCAIBox/LLMBox)


</div>

# CONTINUE...