# VerifIF Resource Server

A NeMo Gym resource server that integrates **VerifIF** (Verifiable Instruction Following) validators for comprehensive instruction-following evaluation in reinforcement learning training.

## Overview

This resource server provides two types of validators:

1. **Fast Rule-Based Validators (~50+)**: CPU-efficient validators for structural constraints like word count, keyword presence, formatting, punctuation, etc.

2. **LLM Judge Validators (~27)**: Async LLM-as-a-judge validators for semantic/stylistic constraints like tone, formality, politeness, and linguistic patterns.

3. **Custom LLM Judge Questions**: Free-form yes/no questions evaluated by an LLM judge.

The environment is intentionally generalistic and ships with datasets targeting **MultiChallenge**, **InverseIFEval**, **CFBench**, and **SysBench** — data will be released soon.

## Quick Start

### 1. Set up environment variables

Create `env.yaml` in your Gym root:

```yaml
policy_base_url: https://api.openai.com/v1
policy_api_key: your-openai-api-key
policy_model_name: gpt-5-2025-08-07  # or gpt-4.1-2025-04-14
```

### 2. Start the servers

```bash
cd /path/to/Gym
source .venv/bin/activate
gym env start \
    --resources-server verifif \
    --model-type openai_model
```

### 3. Run a test

```bash
gym eval run --no-serve \
    --agent verifif_simple_agent \
    --input resources_servers/verifif/data/example.jsonl \
    --output results.jsonl
```

## Architecture

```
verifif/
├── app.py                    # Main resource server (TuringVIFResourcesServer)
├── vif_validators/           # Validation logic
│   ├── __init__.py
│   ├── validator.py          # Fast rule-based validators
│   ├── data_loader.py        # Instruction definitions & prompts
│   ├── instruction_definition.csv
│   ├── subinstruction_definition.csv
│   └── evaluation_modes.csv
├── configs/
│   └── verifif.yaml       # Server configuration
├── data/
│   └── example.jsonl         # Example dataset
├── tests/
│   ├── __init__.py
│   └── test_app.py           # Unit tests (47 tests)
├── requirements.txt
└── README.md
```

## Supported Instructions

### Fast Validators (Rule-Based)

| Category | Instructions |
|----------|-------------|
| **Length Constraints** | `number_words`, `number_characters`, `sentence_length`, `word_repetition`, `unique_words`, `word_length`, `avg_word_length`, `paragraph_length` |
| **Keywords** | `existence`, `frequency`, `forbidden_words`, `letter_frequency`, `alliteration`, `palindrome_word`, `positioning`, `vowel_count`, `consonant_count` |
| **Format** | `json_format`, `numbered_list`, `bullet_lists`, `title`, `multiple_sections`, `number_paragraphs`, `sentences_per_paragraph`, `nested_list`, `table`, `heading_depth`, `sentence_count`, `sentence_endings` |
| **Case** | `all_caps`, `lowercase`, `alternating`, `first_letter_cap`, `capital_word_frequency`, `lowercase_word_frequency`, `*_target` variants, `case_ratio`, `vowel_consonant_balance` |
| **Punctuation** | `no_comma`, `no_period`, `question_exclaim`, `end_rule` |
| **Start/End** | `start_checker`, `end_checker`, `wrap_checker`, `quotation` |
| **Detectable Content** | `postscript`, `number_placeholders`, `numeric_inclusion` |

### LLM Judge Validators

| Category | Instructions |
|----------|-------------|
| **Stylistic** | `tone_formality`, `emotional_tone`, `politeness`, `descriptive_level`, `literary_style`, `sentence_tone_consistency`, `voice`, `figurative_language`, `tone_transition`, `emotive_adjectives`, `sensory_detail`, `rhythm_pattern` |
| **Linguistic** | `pragmatic_context`, `speech_act`, `syntactic_pattern`, `grammatical_mood`, `morphological_form`, `phonological_pattern`, `sound_symbolism` |
| **Situational** | `role_based`, `task_specific`, `audience_alignment`, `contextual_scenario`, `perspective`, `emotional_alignment`, `cultural_context`, `temporal_context`, `environment_setting` |

## Configuration

### Server Config (`configs/verifif.yaml`)

```yaml
verifif:
  resources_servers:
    verifif:
      entrypoint: app.py
      domain: instruction_following
      # Reward aggregation mode
      aggregation_mode: all  # all | any | mean | min | max
      # LLM Judge configuration - uses policy model by default
      judge_base_url: ${policy_base_url}
      judge_api_key: ${policy_api_key}
      judge_model: ${policy_model_name}  # Any OpenAI-compatible model (OpenAI, vLLM, etc.)
```

### Reward Aggregation Modes

The `aggregation_mode` setting controls how individual check verdicts are combined into the final reward:

| Mode | Behavior | Output Range |
|------|----------|-------------|
| `all` (default) | All checks must pass (logical AND) | 0.0 or 1.0 |
| `any` | At least one check must pass (logical OR) | 0.0 or 1.0 |
| `mean` | Average of binary per-check scores | [0.0, 1.0] continuous |
| `min` | Minimum score (strictest) | 0.0 or 1.0 |
| `max` | Maximum score (most lenient) | 0.0 or 1.0 |

Override in your experiment YAML:

```yaml
env:
  nemo_gym:
    verifif:
      resources_servers:
        verifif:
          aggregation_mode: mean
```

### Model Support

The judge can be **any OpenAI-compatible chat endpoint** — OpenAI, Azure, vLLM, SGLang, TGI, or an in-cluster Gym `vllm_model` / `local_vllm` server (set `judge_server_name` to auto-discover the URL). OpenAI reasoning models (`gpt-5*`, `o1*`, `o3*`, `o4-mini*`) are auto-routed to the Responses API; everything else uses Chat Completions.

## Dataset Format

Each entry in your JSONL dataset should have:

```json
{
  "id": 1,
  "instructions": [
    {"instruction_id": "length_constraints:number_words", "relation": "at least", "num_words": 50},
    {"instruction_id": "keywords:existence", "keywords": ["research", "methodology"]},
    {"instruction_id": "stylistic:tone_formality", "tone_level": "formal"}
  ],
  "llm_judge": [
    {"uid": 1, "content": "Does the response contain a clear call-to-action?"}
  ],
  "responses_create_params": {
    "input": [{"role": "user", "content": "Write about research methodology..."}]
  }
}
```

### Field Descriptions

| Field | Required | Description |
|-------|----------|-------------|
| `id` | Optional | Request identifier |
| `instructions` | Yes | List of instruction objects with `instruction_id` and parameters |
| `llm_judge` | Optional | List of custom yes/no questions for LLM judge evaluation |
| `responses_create_params` | Yes | Parameters for the LLM response generation |

## Running Tests

```bash
cd /path/to/Gym
source .venv/bin/activate
pytest resources_servers/verifif/tests/ -v
```

## API Endpoints

### POST /verify

Validates a response against instructions.

**Request Body:**
```json
{
  "id": 1,
  "instructions": [...],
  "llm_judge": [{"uid": 1, "content": "Is the response professional?"}],
  "response": {
    "output": [{"content": [{"text": "...response text..."}]}]
  }
}
```

**Response:**
```json
{
  "reward": 1.0,
  "follow_all_instructions": true,
  "follow_instruction_list": [true, true, true],
  "validation_results": [
    {"instruction": "length_constraints:number_words", "status": "Passed", "message": "Word count: 75 (at least 50)"},
    {"instruction": "stylistic:tone_formality", "status": "Passed", "message": "The response maintains formal tone..."},
    {"instruction": "llm_judge_1", "status": "Passed", "message": "Yes, the response includes..."}
  ]
}
```

## Integration Notes

### Async Design

- All LLM judge calls use `NeMoGymAsyncOpenAI` for non-blocking I/O
- LLM validators run in parallel via `asyncio.gather`
- Fast validators run synchronously (CPU-bound, ~1-5ms each)
- Follows NVIDIA NeMo Gym integration guidelines (no extra threads/processes)

### Performance

| Validator Type | Typical Latency |
|----------------|-----------------|
| Fast (rule-based) | 1-5ms |
| LLM Judge (chat models, e.g. GPT-4.1, Qwen3) | 500-2000ms |
| LLM Judge (reasoning models, e.g. GPT-5) | 1000-5000ms |

### Scaling

For high-throughput training:
1. Use rate limiting for LLM judge API calls
2. Consider micro-batching LLM evaluations (future enhancement)
3. Cache definition lookups (already implemented)
4. Use GPT-4.1 for faster judge evaluations when reasoning is not critical

## Troubleshooting

### Common Issues

| Issue | Solution |
|-------|----------|
| `401 Unauthorized` | Check `policy_api_key` in `env.yaml` |
| `400 Bad Request` with GPT-5 | Ensure you're using the latest `app.py` with Responses API support |
| `ModuleNotFoundError` | Run `ray stop --force` and restart servers |
| Server won't start | Delete `.venv` in `resources_servers/verifif/` and restart |

### Debugging

View server logs:
```bash
# Check terminal output from gym env start
# Or view Ray dashboard at http://127.0.0.1:8265
```

## Contributing

1. Add new fast validators to `vif_validators/validator.py`
2. Add instruction definitions to the CSV files
3. Write unit tests in `tests/test_app.py`
4. Update this README with new instructions

## License

Apache-2.0
