LiveBench

Installation Quickstart

We recommend using a virtual environment to install LiveBench.

python -m venv .venv
source .venv/bin/activate

To generate answers with API models (i.e. with gen_api_answer.py), conduct judgments, and show results:

cd LiveBench
pip install -e .

To score results on the coding tasks (code_completion and code_generation), you will also need to install the required dependencies:

cd livebench/code_runner
pip install -r requirements_eval.txt

Note that, to evaluate the agentic coding questions, you will need docker installed and available (i.e. the command docker --version should work). This will be checked prior to such tasks being run.

Note about local models: Local model inference is unmaintained. We highly recommend serving your model on an OpenAI compatible API using vllm and performing inference using run_livebench.py.

Our repo contains code from LiveCodeBench and IFEval.

Usage

cd livebench

Evaluating New Models and Configuring API Parametersdee

Any API-based model for which there is an OpenAI compatible endpoint should work out of the box using the --api-base and --api-key (or --api-key-name) arguments. If you'd like to override the name of the model for local files (e.g. saving it as deepseek-v3 instead of deepseek-chat), use the --model-display-name argument. You can also override values for temperature and max tokens using the --force-temperature and --max-tokens arguments, respectively.

If you'd like to have persistent model configuration without needing to specify command-line arguments, you can create a model configuration document in a yaml file in livebench/model/model_configs. See the other files there for examples of the necessary format. Important values are model_display_name, which determines the answer .jsonl file name and model ID used for other scripts, and api_name, which provides a mapping between API providers and names for the model in that API. For instance, Deepseek R1 can be evaluated using the Deepseek API with a name of deepseek-reasoner and the Together API with a name of deepseek-ai/deepseek-r1. api_kwargs allows you to set overrides for parameters such as temperature, max tokens, and top p, for all providers or for specific ones. Once this is set, you can use --model <model_name> with the model_display_name value you put in the yaml document when running run_livebench.py.

When performing inference, use the --model-provider-override argument to override the provider you'd like to use for the model.

We have also implemented inference for Anthropic, Cohere, Mistral, Together, and Google models, so those should also all work immediately either by using --model-provider-override or adding a new entry to the appropriate configuration file.

If you'd like to use a model with a new provider that is not OpenAI-compatible, you will need to implement a new completions function in completions.py and add it to get_api_function in that file; then, you can use it in your model configuration.

Evaluating New Questions

If you want to create your own set of questions, or try out different prompts, etc, follow these steps:

• Create a question.jsonl file with the following path (or, run python download_questions.py and update the downloaded file): livebench/data/live_bench/<category>/<task>/question.jsonl. For example, livebench/data/reasoning/web_of_lies_new_prompt/question.jsonl. Here is an example of the format for question.jsonl (it's the first few questions from web_of_lies_v2):

{"question_id": "0daa7ca38beec4441b9d5c04d0b98912322926f0a3ac28a5097889d4ed83506f", "category": "reasoning", "ground_truth": "no, yes, yes", "turns": ["In this question, assume each person either always tells the truth or always lies. Tala is at the movie theater. The person at the restaurant says the person at the aquarium lies. Ayaan is at the aquarium. Ryan is at the botanical garden. The person at the park says the person at the art gallery lies. The person at the museum tells the truth. Zara is at the museum. Jake is at the art gallery. The person at the art gallery says the person at the theater lies. Beatriz is at the park. The person at the movie theater says the person at the train station lies. Nadia is at the campground. The person at the campground says the person at the art gallery tells the truth. The person at the theater lies. The person at the amusement park says the person at the aquarium tells the truth. Grace is at the restaurant. The person at the aquarium thinks their friend is lying. Nia is at the theater. Kehinde is at the train station. The person at the theater thinks their friend is lying. The person at the botanical garden says the person at the train station tells the truth. The person at the aquarium says the person at the campground tells the truth. The person at the aquarium saw a firetruck. The person at the train station says the person at the amusement park lies. Mateo is at the amusement park. Does the person at the train station tell the truth? Does the person at the amusement park tell the truth? Does the person at the aquarium tell the truth? Think step by step, and then put your answer in **bold** as a list of three words, yes or no (for example, **yes, no, yes**). If you don't know, guess."], "task": "web_of_lies_v2"}

• If adding a new task, create a new scoring method in the process_results folder. If it is similar to an existing task, you can copy that task's scoring function. For example, livebench/process_results/reasoning/web_of_lies_new_prompt/utils.py can be a copy of the web_of_lies_v2 scoring method.
• Add the scoring function to gen_ground_truth_judgment.py here.

- Run and score models using --question-source jsonl and specifying your task. For example:

 
python gen_api_answer.py --bench-name live_bench/reasoning/web_of_lies_new_prompt --model claude-3-5-sonnet --question-source jsonl
python gen_ground_truth_judgment.py --bench-name live_bench/reasoning/web_of_lies_new_prompt --question-source jsonl
python show_livebench_result.py --bench-name live_bench/reasoning/web_of_lies_new_prompt