---
title: "CLI Commands"
description: ""
position: 3
---
This page documents all available NeMo Gym CLI commands.

<Note>
Each command has both a short form (such as `ng_run`) and a full form (such as `nemo_gym_run`). They are functionally identical.

</Note>

## Quick Reference

```bash
# Display help
ng_help

# Get detailed help for any command
ng_run +help=true
ng_test +h=true
```

---

## Server Management

Commands for running, testing, and managing NeMo Gym servers.

### `ng_run` / `nemo_gym_run`

Start NeMo Gym servers for agents, models, and resources.

This command reads configuration from YAML files specified via `+config_paths` and starts all configured servers. The configuration files should define server instances with their entrypoints and settings.

**Configuration Parameter**

| Parameter | Type | Description |
| --- | --- | --- |
| `config_paths` | List[str] | Paths to YAML configuration files. Specify using Hydra: `+config_paths="[file1.yaml,file2.yaml]"` |

**Example**

```bash
# Start servers with specific configs
config_paths="resources_servers/example_single_tool_call/configs/example_single_tool_call.yaml,\
responses_api_models/openai_model/configs/openai_model.yaml"
ng_run "+config_paths=[${config_paths}]"
```

---

### `ng_test` / `nemo_gym_test`

Test a specific server module by running its pytest suite and optionally validating example data.

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `entrypoint` | str | Entrypoint for this command. Must be a relative path with two parts (such as `responses_api_agents/simple_agent`). |
| `should_validate_data` | bool | Whether to validate the example data (examples, metrics, rollouts, and so on) for this server. Default: `False`. |

**Example**

```bash
ng_test +entrypoint=resources_servers/example_single_tool_call
```

---

### `ng_test_all` / `nemo_gym_test_all`

Run tests for all server modules in the project.

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `fail_on_total_and_test_mismatch` | bool | Fail if the number of server modules does not match the number with tests. Default: `False`. |

**Example**

```bash
ng_test_all
```

---

### `ng_dev_test` / `nemo_gym_dev_test`

Run core NeMo Gym tests with coverage reporting. Runs pytest with the `--cov` flag.

**Example**

```bash
ng_dev_test
```

---

### `ng_init_resources_server` / `nemo_gym_init_resources_server`

Initialize a new resources server with template files and directory structure.

**Example**

```bash
ng_init_resources_server +entrypoint=resources_servers/my_server
```

---

## Data Collection

Commands for collecting verified rollouts for RL training.

### `ng_collect_rollouts` / `nemo_gym_collect_rollouts`

Perform a batch of rollout collection.

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `agent_name` | str | The agent to collect rollouts from. |
| `input_jsonl_fpath` | str | The input data source to use to collect rollouts, in the form of a file path to a JSONL file. |
| `output_jsonl_fpath` | str | The output data JSONL file path. |
| `limit` | Optional[int] | Maximum number of examples to load and take from the input dataset. |
| `num_repeats` | Optional[int] | The number of times to repeat each example to run. Useful if you want to calculate mean@k, such as mean@4 or mean@16. |
| `num_repeats_add_seed` | bool | When num_repeats &gt;1, add a "seed" parameter on the Responses create params. |
| `num_samples_in_parallel` | Optional[int] | Limit the number of concurrent samples running at once. |
| `responses_create_params` | Dict | Overrides for the `responses_create_params`, such as `temperature` and `max_output_tokens`. Refer to [Generation Parameters](#generation-parameters). |
| `resume_from_cache` | bool | Resume an interrupted run by skipping rows already completed. Default: `False`. Refer to [Resume Interrupted Runs](#resume-interrupted-runs). |

**Example**

```bash
ng_collect_rollouts \
    +agent_name=example_single_tool_call_simple_agent \
    +input_jsonl_fpath=weather_query.jsonl \
    +output_jsonl_fpath=weather_rollouts.jsonl \
    +limit=100 \
    +num_repeats=4 \
    +num_samples_in_parallel=10
```

#### Generation Parameters

Sampling parameters such as `temperature`, `max_output_tokens`, and `top_p` are not standalone CLI flags — they are passed as overrides inside `responses_create_params` using Hydra's nested dot syntax. Overrides are merged into each input row's existing `responses_create_params` with a **shallow** merge (top-level keys only):

```bash
ng_collect_rollouts \
    +agent_name=example_single_tool_call_simple_agent \
    +input_jsonl_fpath=weather_query.jsonl \
    +output_jsonl_fpath=weather_rollouts.jsonl \
    ++responses_create_params.temperature=1.0 \
    ++responses_create_params.top_p=1.0 \
    ++responses_create_params.max_output_tokens=4096
```

The same syntax works for `ng_e2e_collect_rollouts`. Top-level fields such as `temperature` and `max_output_tokens` are straightforward. For nested objects (for example, `++responses_create_params.reasoning.effort=low`), the entire nested dict replaces the row's existing value at that key — other fields under the same nested object are not preserved.

#### Resume Interrupted Runs

Setting `+resume_from_cache=true` lets you restart the **same command** after a crash or interruption and pick up only the rows that have not finished yet. It works for both `ng_collect_rollouts` and `ng_e2e_collect_rollouts`, across any environment.

How it works:

- **Materialized inputs.** On the first run, the fully expanded input rows (after `num_repeats`, `limit`, `prompt_config`, and any overrides) are written to a sidecar file next to your output. The path is derived from `output_jsonl_fpath` by appending `_materialized_inputs` to the stem — so `rollouts.jsonl` produces `rollouts_materialized_inputs.jsonl`.
- **Incremental output.** Results are flushed to `output_jsonl_fpath` after each completed rollout, so partial output survives a crash.
- **Matching.** On resume, completed work is matched by `(task_index, rollout_index)` against the materialized inputs, and already-completed rows are skipped. The run prints a summary such as the number of original input rows, rows already done, and rows that still need to be run.
- **Fallback.** If either the materialized inputs or the output file is missing, resume is skipped and the run starts fresh. With the default `resume_from_cache=False`, existing output is cleared before the run.

<Warning>
If you change the config, schema, or data between runs, the materialized inputs become stale and resume will diff against the old expansion. Delete the `*_materialized_inputs.jsonl` file (and the output file) to start fresh.
</Warning>

### `ng_e2e_collect_rollouts` / `nemo_gym_e2e_collect_rollouts`

Spin up all necessary servers and perform a batch of rollout collection using each dataset inside the provided configs.

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `output_jsonl_fpath` | str | The output data JSONL file path. |
| `num_samples_in_parallel` | Optional[int] | Limit the number of concurrent samples running at once. |
| `responses_create_params` | Dict | Overrides for the `responses_create_params`, such as `temperature` and `max_output_tokens`. Refer to [Generation Parameters](#generation-parameters). |
| `resume_from_cache` | bool | Resume an interrupted run by skipping rows already completed. Default: `False`. Refer to [Resume Interrupted Runs](#resume-interrupted-runs). |

**Examples**

```bash
ng_e2e_collect_rollouts \
    +output_jsonl_fpath=weather_rollouts.jsonl \
    +num_samples_in_parallel=10
```

```bash
config_paths="responses_api_models/openai_model/configs/openai_model.yaml,\
resources_servers/math_with_judge/configs/math_with_judge.yaml"
ng_e2e_collect_rollouts \
    "+config_paths=[${config_paths}]" \
    ++wandb_project= \
    ++wandb_name= \
    ++wandb_dir= \
    ++output_jsonl_fpath=results/test_e2e_rollout_collection/aime24.jsonl \
    ++split=validation
```

Example using GPT-OSS 120B remote vLLM endpoint
```bash
experiment_name=rollouts/test_001
config_paths="responses_api_models/openai_model/configs/openai_model.yaml,\
resources_servers/math_with_judge/configs/math_with_judge.yaml"
ng_e2e_collect_rollouts \
    "+config_paths=[${config_paths}]" \
    +skip_venv_if_present=true \
    +wandb_project=gym-dev \
    +wandb_name=$(date +%Y%m%d)/$experiment_name \
    ++output_jsonl_fpath=results/$experiment_name.jsonl \
    ++overwrite_metrics_conflicts=true \
    ++split=validation \
    ++policy_model_name=openai/gpt-oss-120b \
    ++policy_api_key=dummy_key \
    ++policy_base_url=http://0.0.0.0:10240/v1 \
    ++responses_create_params.reasoning.effort=low \
    ++responses_create_params.temperature=1.0 \
    ++responses_create_params.top_p=1.0 &> eval_gptoss120b.log &
```

---

### `ng_reward_profile` / `nemo_gym_reward_profile`

Computes statistics on rewards and task difficulty for rollouts collected with `ng_collect_rollouts` with `num_repeats` &gt;1. This outputs a new "reward profiled" dataset, where each task in the dataset has metrics like the average reward, standard deviation, min/max, and pass rate. This is useful in filtering tasks before training for difficulty, variance, or creating a curriculum. 

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `input_jsonl_fpath` | str | Path to the original task dataset JSONL file. |
| `rollouts_jsonl_fpath` | str | Path to the rollouts file from `ng_collect_rollouts` (must have been run with `num_repeats` &gt;1). |
| `output_jsonl_fpath` | str | Output file path for the reward profiled dataset. |
| `pass_threshold` | Optional[float] | Reward threshold for computing pass rate. If not specified, pass rate metrics are not included. |

**Output Fields**

Each output row contains all original task fields plus:
- `avg_reward`: Average reward across all rollouts
- `std_reward`: Standard deviation of rewards
- `min_reward`: Minimum reward observed
- `max_reward`: Maximum reward observed
- `total_samples`: Number of rollout samples
- `pass_rate`, `pass_rate_total`, `pass_rate_passed`, `pass_threshold`: (Only if `pass_threshold` is specified)

**Example**

```bash
ng_reward_profile \
    +input_jsonl_fpath=tasks.jsonl \
    +rollouts_jsonl_fpath=rollouts.jsonl \
    +output_jsonl_fpath=profiled_tasks.jsonl \
    +pass_threshold=1.0
```

---

## Data Management

Commands for preparing and viewing training data.

### `ng_prepare_data` / `nemo_gym_prepare_data`

Prepare and validate training data, generating metrics and statistics for datasets.

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `output_dirpath` | str | Directory path where processed datasets and metrics will be saved. |
| `mode` | Literal["train_preparation", "example_validation"] | Processing mode. Use `train_preparation` to prepare train and validation datasets for training, or `example_validation` to validate example data for PR submission. |
| `should_download` | bool | Whether to automatically download missing datasets from remote registries. Default: `False`. |
| `overwrite_metrics_conflicts` | bool | Whether or not to overwrite metrics conflicts. Default: `False`. |

**Example**

```bash
config_paths="resources_servers/example_multi_step/configs/example_multi_step.yaml,\
responses_api_models/openai_model/configs/openai_model.yaml"
ng_prepare_data "+config_paths=[${config_paths}]" \
    +output_dirpath=data/example_multi_step \
    +mode=example_validation
```

---

### `ng_materialize_prompts` / `nemo_gym_materialize_prompts`

Apply a prompt template to raw JSONL data, producing materialized JSONL with populated `responses_create_params.input` for RL training.

Each input row must **not** already have a populated `responses_create_params.input`; the command applies the prompt template from `prompt_config` to each row, fills in the input, and preserves the row's other fields.

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `input_jsonl_fpath` | str | Raw JSONL data (rows without `responses_create_params.input`). |
| `prompt_config` | str | Path to the prompt YAML file to apply. |
| `output_jsonl_fpath` | str | Output path for the materialized JSONL with populated prompts. |

**Example**

```bash
ng_materialize_prompts \
    +input_jsonl_fpath=data/my_dataset.jsonl \
    +prompt_config=/path/to/my_prompt.yaml \
    +output_jsonl_fpath=my_dataset_materialized.jsonl
```

<Note>
**Which data-preparation command should I use?**

- **`ng_materialize_prompts`** — a focused, standalone step that applies a prompt template to raw rows to populate `responses_create_params.input`. No servers are started. Use it when you have raw data and just need to turn it into prompt-ready rows.
- **`ng_prepare_data`** — the full preparation pipeline for training: it can download missing datasets, validate data, and compute dataset metrics, writing train/validation splits and metrics artifacts. Use it to prepare and validate datasets for training or PR submission.
</Note>

---

## Dataset Registry - GitLab

Commands for uploading, downloading, and managing datasets in GitLab Model Registry.

### `ng_upload_dataset_to_gitlab` / `nemo_gym_upload_dataset_to_gitlab`

Upload a local JSONL dataset artifact to GitLab.

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `dataset_name` | str | The dataset name. |
| `version` | str | The version of this dataset. Must be in the format `x.x.x`. |
| `input_jsonl_fpath` | str | Path to the JSONL file to upload. |

**Example**

```bash
ng_upload_dataset_to_gitlab \
    +dataset_name=example_multi_step \
    +version=0.0.1 \
    +input_jsonl_fpath=data/train.jsonl
```

---

### `ng_download_dataset_from_gitlab` / `nemo_gym_download_dataset_from_gitlab`

Download a JSONL dataset from GitLab Model Registry.

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `dataset_name` | str | The dataset name. |
| `version` | str | The version of this dataset. Must be in the format `x.x.x`. |
| `artifact_fpath` | str | The filepath to the artifact to download. |
| `output_fpath` | str | Path where the downloaded dataset will be saved. |

**Example**

```bash
ng_download_dataset_from_gitlab \
    +dataset_name=example_multi_step \
    +version=0.0.1 \
    +artifact_fpath=train.jsonl \
    +output_fpath=data/train.jsonl
```

---

### `ng_delete_dataset_from_gitlab` / `nemo_gym_delete_dataset_from_gitlab`

Delete a dataset from GitLab Model Registry. Prompts for confirmation.

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `dataset_name` | str | Name of the dataset to delete from GitLab. |

**Example**

```bash
ng_delete_dataset_from_gitlab +dataset_name=old_dataset
```

---

## Dataset Registry - HuggingFace

Commands for uploading and downloading datasets to/from HuggingFace Hub.

### `ng_upload_dataset_to_hf` / `nemo_gym_upload_dataset_to_hf`

Upload a JSONL dataset to HuggingFace Hub with optional GitLab deletion after successful upload.

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `hf_token` | str | HuggingFace API token for authentication. |
| `hf_organization` | str | HuggingFace organization name where the dataset will be uploaded. |
| `hf_collection_name` | str | HuggingFace collection name for organizing datasets. |
| `hf_collection_slug` | str | Alphanumeric collection slug found at the end of the collection URI. |
| `dataset_name` | str | Name of the dataset. Will be combined with domain and resources server name. |
| `input_jsonl_fpath` | str | Path to the local JSONL file to upload. |
| `resource_config_path` | str | Path to resources server config file. Used to extract domain for naming convention. |
| `hf_dataset_prefix` | str | Prefix prepended to dataset name. Default: `NeMo-Gym`. |
| `delete_from_gitlab` | Optional[bool] | Delete the dataset from GitLab after successful upload to HuggingFace. Default: `False`. |

**Example**

```bash
resource_config_path="resources_servers/example_multi_step/configs/example_multi_step.yaml"
ng_upload_dataset_to_hf \
    +dataset_name=my_dataset \
    +input_jsonl_fpath=data/train.jsonl \
    +resource_config_path=${resource_config_path} \
    +delete_from_gitlab=true
```

---

### `ng_download_dataset_from_hf` / `nemo_gym_download_dataset_from_hf`

Download a JSONL dataset from HuggingFace Hub to local filesystem.

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `output_fpath` | str | Local file path where the downloaded dataset will be saved. |
| `hf_token` | str | HuggingFace API token for authentication. |
| `artifact_fpath` | str | Name of the artifact file to download from the repository. |
| `repo_id` | str | HuggingFace repository ID in format `organization/dataset-name`. |

**Example**

```bash
ng_download_dataset_from_hf \
    +repo_id=NVIDIA/NeMo-Gym-Math-example_multi_step-v1 \
    +artifact_fpath=train.jsonl \
    +output_fpath=data/train.jsonl
```

---

### `ng_gitlab_to_hf_dataset` / `nemo_gym_gitlab_to_hf_dataset`

Upload a JSONL dataset to HuggingFace Hub and automatically delete from GitLab after successful upload.

This command always deletes the dataset from GitLab after uploading to HuggingFace. Use `ng_upload_dataset_to_hf` if you want optional deletion control.

**Parameters**

Same as `ng_upload_dataset_to_hf` but `delete_from_gitlab` is not available. This command always deletes.

**Example**

```bash
resource_config_path="resources_servers/example_multi_step/configs/example_multi_step.yaml"
ng_gitlab_to_hf_dataset \
    +dataset_name=my_dataset \
    +input_jsonl_fpath=data/train.jsonl \
    +resource_config_path=${resource_config_path}
```

---

## Configuration & Help

Commands for debugging configuration and getting help.

### `ng_dump_config` / `nemo_gym_dump_config`

Display the resolved Hydra configuration for debugging purposes.

**Example**

```bash
ng_dump_config "+config_paths=[<config1>,<config2>]"
```

---

### `ng_help` / `nemo_gym_help`

Display a list of available NeMo Gym CLI commands.

**Example**

```bash
ng_help
```

---

### `ng_version` / `nemo_gym_version`

Display NeMo Gym version and system information.

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `json_format` | bool | Output in JSON format for programmatic use. Default: `False`. Can be specified with `+json=true`. |

**Example**

```bash
# Display version information
ng_version

# Output as JSON
ng_version +json=true
```

---

### `ng_pip_list` / `nemo_gym_pip_list`

Each server has its own isolated virtual environment. To inspect the packages:

**Parameters**

| Parameter | Type | Description |
| --- | --- | --- |
| `entrypoint` | str | The relative entrypoint path to the server directory |
| `format` | Optional[str] | Output format for pip list. Options: 'columns' (default), 'freeze', 'json'. Default: `None`. |
| `outdated` | bool | List outdated packages. Default: `False`. |

**Examples**

```bash
# List all packages
ng_pip_list +entrypoint=resources_servers/example_single_tool_call

# Output as JSON
ng_pip_list +entrypoint=resources_servers/example_single_tool_call +format=json

# Check for outdated packages
ng_pip_list +entrypoint=resources_servers/example_single_tool_call +outdated=true
```

---

### `ng_status` / `nemo_gym_status`

View all currently running NeMo Gym servers and their health status.

**Example**

```bash
ng_status

NeMo Gym Server Status:

[1] ✓ example_single_tool_call (resources_servers/example_single_tool_call)
{
    'server_type': 'resources_servers',
    'name': 'example_single_tool_call',
    'port': 58117,
    'pid': 89904,
    'uptime_seconds': '0d 0h 0m 41.5s',
}
[2] ✓ example_single_tool_call_simple_agent (responses_api_agents/simple_agent)
{
    'server_type': 'responses_api_agents',
    'name': 'simple_agent',
    'port': 58118,
    'pid': 89905,
    'uptime_seconds': '0d 0h 0m 41.5s',
}
[3] ✓ policy_model (responses_api_models/openai_model)
{
    'server_type': 'responses_api_models',
    'name': 'openai_model',
    'port': 58119,
    'pid': 89907,
    'uptime_seconds': '0d 0h 0m 41.5s',
}

3 servers found (3 healthy, 0 unhealthy)

```

---

## Getting Help

For detailed help on any command, run it with `+help=true` or `+h=true`:

```bash
ng_run +help=true
ng_collect_rollouts +h=true
```

This will display all available configuration parameters and their descriptions.

---

## Re-install Gym and dependencies
```bash
ng_reinstall
```

This will re-install Gym and its dependencies into the currently activated Python virtual environment.
