开源AI工具：Benchmarking

Chain-of-Thought Hub: Measuring LLMs' Reasoning Performance

"A fantasy graph illustrating a chain of stars in a dark night with blue sky, digital art, super resolution". Midjourney V5

----

By Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng, Tushar Khot, Wenhu Chen

From University of Edinburgh, University of Washington, Allen Institute for AI, University of Waterloo

[paper] [blog] [twitter]

Recently, there are a lot of progress in LLMs. Many claim that a small model less than 10B can achieve comparable performance to GPT-3.5. Really?

In a casual conversation, the distinction between GPT-3.5 and GPT-4 can be subtle. The difference comes out when **\the complexity of the task reaches a sufficient threshold\** — GPT-4 is more reliable, creative, and able to handle much more nuanced instructions than GPT-3.5. -- GPT-4 release blog

The key differentiator is whether a model can do complex tasks, like the old saying: "chit-chat is cheap, show me the reasoning." This is why we compile a list of complex reasoning tasks including math (GSM8K), science (MATH, TheoremQA), symbolic (BBH), knowledge (MMLU, C-Eval), coding (HumanEval), factual (SummEdits), and long-context (RepoBench, Qspr, QALT, BkSS) to measure the models' performance on challenging tasks.

More importantly, we envisage large language models to become the next-generation computational platform and foster an ecosystem of LLM-based new applications. When this comes, chain-of-thought prompt engineering will be the next-generation system calls and shell scripts.

The credibility of chain-of-thought hub comes from the very carefully mediculously picked datasets and models that can clearly help the development of LLMs. The resutls and scripts from Chain-of-thought Hub is being used and referred by leading industrial and academic organizations in the space of large language models. We devide the tasks into three categories: main, experimental, and long-context. Main: datasets that are stable and consistently referred by places where LLMs are built. Experimental: datasets that has the potential to test future LLM capabilities. * Long-context: datasets that require reasoning over very long context, an important direction of future LLMs.

<details> <summary>[List of datasets we consider]</summary>

| Section | Dataset | Description | | ------- | ------- | ----------- | | Main | GSM8K | Grade-level math word problems | | Main | MATH | Competition-level math and science problems | | Main | MMLU | Multi-discipline knowledge | | Main | BBH | Challenging language and symbolic reasoning | | Main | HumanEval | Python coding | | Main | C-Eval | Chineses multi-discipline knowledge | | Experimental | TheoremQA | Theorem proving | | Experimental | SummEdits | Factual reasoning | | Long Ctx | Qspr | Question answering over research papers | | Long Ctx | QALT | Multiple-choice questions over long articles and stories | | Long Ctx | BkSS | Reordering of summaries of parts of novels | </details>

[Call for contribution]: would love to invite community members to: Send a PR to fill in a missing number in the table Raise an issue to suggest / brainstorm a new task / benchmark that measures reasoning over very long context Raise an issue to suggest / brainstorm a new task / benchmark that measures complex API calls and tool usage Raise an issue to suggest other good tasks / benchmarks that can clearly differentiate models' performance * Raise an issue to suggest a new model that can be added to the table

[UPDATE 20231210]: Add Gemini, Yi-34B, DeepSeek 67B Update long-context -- we will have more updates on this section * Preview of Mistral 7B8E MoE model results <details> <summary>Mistral 7B 8E looks approximately comparible with Yi34B / LLaMA2 70B / DeepSeek 67B</summary>

</details>

[UPDATE 20230620]: Seperate main (datasets that are stable and consistently referred by places where LLMs are built) and experimental (datasets that has the potential to test future LLM capabilities) leaderboards. Add long-context section (experimental)

<details> <summary>[Previous updates]</summary> [UPDATE 20230609]: Add evaluation scripts on MMLU for LLaMA and Falcon

[UPDATE 20230601]: Add SummEdits

[UPDATE 20230527]: Add TheoremQA, add Vicuna, Alpaca, InstructCodeT5. </details>

More about the tasks

• GSM8K: 8k elementary school math. -- Performance improvements on this dataset directly translate to daily math abilities when interacting with LLMs
• MMLU: 15k problems under 57 subjects, high school and college knowledge
• MATH (Hard!): 12k problems within 7 categories, very hard math and natural science. All current models struggle.
• BBH: 6.5k problems within 23 subsets, symbolic and text reasoning
• HumanEval: a classical handwritten dataset of 164 Python problems for evaluating coding capability.
• C-Eval: a collection of 13k multi-choice questions spanning 52 disciplines of knowledge test in Chinese.
• TheoremQA (Hard!): 800 QA pairs covering 350+ theorems spanning across Math, EE&CS, Physics and Finance.
• SummEdits: 6.3k factual consistency reasoning problems within 10 domains.

What's different than other important evaluation?

• HeLM uses answer-only prompting, we use chain-of-thought promoting
• HeLM evaluates everything. We only focus on complex reasoning, the key differentiator of LLMs' capability.
• Open LLM Leaderboard evaluates open-sourced language models. We consider most leading models.
• Currently, the performance of LLaMA 65B on Open LLM Leaderboard is just 48.8, which is significantly lower than the 63.4 reported in the paper. This casts doubts on the comparison between LLaMA and Falcon.
• In our reproduction, we got 61.4 using the MMLU official prompt + greedy decoding + fp16. Our results favors the original LLaMA number and cast doublts on the results of Open LLM Leaderboard.
• Our evaluation script is rather straightforward, most parameters are default, no fancy prompt engineering. We encourage the community to try out our scripts and reproduce our results.
• According to Nathan Lambert, HuggingFace is currently redoing the backend of Open LLM Leaderboard, and the results may change (Jun 10 2023).
• Chatbot Arena evaluates chatbot models, which is more user-oriented at deployment. Our evaluation is more developer-oriented, and we consider on not only chatbots but also base models.