Environment Setup

Follow these steps to set up the environment for <span style="font-family: Consolas;">FasterPy</span>:

conda create -n fsp python=3.11
conda activate fsp
pip install -r requirements.txt

Evaluation Setup and Execution

We modified the evaluation code from madaan/pie-perf: Training language models to make programs faster, and the usage is similar.

Step 1. Save the generations in a jsonl file with the following fields:

{
    "slow_code_col": "the column name for the input code",
    "model_generated_potentially_faster_code_col": "slow_code_col after applying awesome_optimization. This is the code that will be evaluated. You can also provide a list of different candidates here, and the evaluation will be done for each candidate"
}

Step 2. Next, we need to provide the path to the file with some metadata. We call it the reference_file but providing references are optional. The main purpose of this file is to provide information like the language of the code, the problem id, etc. The file should have slow_code_col (same as the generations file) and problem_id. We join the generations file and the references file on the slow_code_col to get the problem id.

Step 3. Finally, we need to provide the path to the file with the actual test cases. We call it the inputs_outputs_basepath. This is a directory with the following structure:

inputs_outputs_basepath/{problem_id}/{inputs, outputs}.txt

Step 4. In addition to these, we need to provide some information about the run. Specifically, the number of times each program should be run, the number of programs to evaluate, the timeout, and so on.

We wrap all of this information is provided in a yaml file sample_eval_config.yaml. Here is an example:

model_generated_outputs_path: "to_be_eval/test-qw2.5-7b-i.jsonl"
inputs_outputs_basepath: "codenet/public_test_cases"
reference_file_path: "py_reference.jsonl"
output_report_file_path: "eval-results/test-qw2.5-7b-i-report.jsonl"
language: "python"
num_problems_to_evaluate: -1
num_trials: 4
ignore_first_k: 1
max_time_per_run: 3
temp_dir: null
model_generated_potentially_faster_code_col: "model_generated_potentially_faster_code_col"
slow_code_col: "slow_code_col"
reference_code_col: "reference_code_col"
is_prompt_based: false
cpu_number: 1

python src/codenet_eval/run_eval.py --eval_config eval/sample_eval_config.yaml

🚀 Quick Start