# Test Plan: list_vulnerability_types Tool

## Overview

This test plan provides comprehensive testing guidance for the `list_vulnerability_types` MCP tool. This tool returns the complete list of vulnerability types (rule names) available in a Contrast organization.

### Migration Notes

**This plan replaces:**
- `test-plan-list_vulnerability_types.md` (original at root level)

The tool signature and behavior remain unchanged; this is a format/location update.

### Tool Signature

**MCP Tool Name:** `list_vulnerability_types`

**Parameters:** None (no parameters required)

### Response Structure

**Returns:** `SingleToolResponse<List<String>>`

```java
SingleToolResponse {
    List<String> data,           // Alphabetically sorted list of rule names
    String message,              // Warnings or info messages
    boolean found                // Always true if request succeeds
}

// Example data:
[
  "cmd-injection",
  "crypto-bad-mac",
  "path-traversal",
  "sql-injection",
  "xss-reflected",
  "xxe"
]
```

### Critical Behavioral Notes

| Behavior | Description |
|----------|-------------|
| **No Parameters** | Tool takes no input parameters |
| **Organization-Specific** | Results depend on organization's configured rules |
| **Alphabetically Sorted** | Results are sorted alphabetically |
| **Trimmed/Cleaned** | Rule names are trimmed (no whitespace) |
| **No Duplicates** | Each rule name appears once |
| **Dynamic List** | Fetched dynamically, reflects current configuration |

---

## 1. Basic Functionality Tests

### Test Case 1.1: Retrieve All Vulnerability Types - Success

**Objective:** Verify tool returns vulnerability types successfully.

**Prerequisites:**
- Organization with rules configured

**Test Steps:**
1. Call `list_vulnerability_types` with no parameters

**Expected Results:**
- Response status: success
- `data` is array of strings
- `data` contains at least 1 rule name
- All items are non-empty strings
- `found: true`

---

### Test Case 1.2: Common OWASP Types Present

**Objective:** Verify common vulnerability types are present.

**Test Steps:**
1. Call `list_vulnerability_types`
2. Check for common types

**Expected Results:**
- Contains common types like:
  - `sql-injection` or SQL-related rules
  - `xss-reflected` or `xss-stored` (cross-site scripting)
  - `path-traversal` or directory traversal rules
  - `cmd-injection` or command injection rules
  - `crypto-*` rules (cryptography)
  - `xxe` (XML external entity)
- At least 5 common types present

---

### Test Case 1.3: Alphabetical Sorting

**Objective:** Verify results are sorted alphabetically.

**Test Steps:**
1. Call `list_vulnerability_types`
2. Verify sorting

**Expected Results:**
- Each item is lexicographically <= the next item
- First item starts with early letter (a-c typically)
- Last item starts with later letter (t-z typically)
- Example valid order: ["cmd-injection", "crypto-bad-mac", "sql-injection"]

**Verification Logic:**
```
For each pair (items[i], items[i+1]):
  assert items[i].compareTo(items[i+1]) <= 0
```

---

### Test Case 1.4: Data Format - Trimmed and Clean

**Objective:** Verify all rule names are properly formatted.

**Test Steps:**
1. Call `list_vulnerability_types`
2. Examine each item

**Expected Results:**
- No empty strings in list
- No null values
- No leading whitespace on any rule name
- No trailing whitespace on any rule name
- All rule names follow pattern: lowercase letters, numbers, hyphens
- Pattern: `/^[a-z0-9-]+$/`

---

### Test Case 1.5: No Duplicates

**Objective:** Verify no duplicate rule names.

**Test Steps:**
1. Call `list_vulnerability_types`
2. Check for duplicates

**Expected Results:**
- No duplicate entries
- `Set(results).size === results.length`

---

## 2. Empty/Edge Case Tests

### Test Case 2.1: Organization with No Rules

**Objective:** Handle organization with no rules configured.

**Prerequisites:**
- Empty organization or simulate no rules

**Test Steps:**
1. Call `list_vulnerability_types` in empty org

**Expected Results:**
- Response status: success (not error)
- `data` is empty array `[]`
- `data` is NOT null
- `found: true` (request succeeded, just no data)
- May have warning in `message`

---

### Test Case 2.2: Large Rule Set (50+ Rules)

**Objective:** Test with many rules configured.

**Prerequisites:**
- Organization with 50+ rules

**Test Steps:**
1. Call `list_vulnerability_types`
2. Measure response time
3. Verify all rules returned

**Expected Results:**
- Response completes within 5 seconds
- All rules returned
- Sorting still correct
- No duplicates

---

## 3. Integration Tests

### Test Case 3.1: Use Results with search_vulnerabilities

**Objective:** Verify rule names work as filter values.

**Test Steps:**
1. Call `list_vulnerability_types` and capture results
2. Select 1-2 rule names (e.g., "sql-injection")
3. Call `search_vulnerabilities(vulnTypes="sql-injection")`
4. Verify no filter errors

**Expected Results:**
- `search_vulnerabilities` accepts rule names without error
- Returned vulnerabilities match the specified types
- No "invalid rule type" errors

**Example Flow:**
```
1. types = list_vulnerability_types()
   → ["cmd-injection", "sql-injection", "xss-reflected"]

2. vulns = search_vulnerabilities(vulnTypes="sql-injection,xss-reflected")
   → All returned vulnerabilities have type in ["sql-injection", "xss-reflected"]
```

---

### Test Case 3.2: Use Results with search_app_vulnerabilities

**Objective:** Verify rule names work with app-scoped tool.

**Test Steps:**
1. Call `list_vulnerability_types` to get types
2. Call `search_app_vulnerabilities(appId="xyz", vulnTypes="sql-injection")`

**Expected Results:**
- Filter works correctly
- No errors
- Returns only matching types

---

### Test Case 3.3: Case Sensitivity Verification

**Objective:** Verify returned names are correct case for filtering.

**Test Steps:**
1. Call `list_vulnerability_types`
2. Use exact returned values with search tools
3. Try modified case (uppercase)

**Expected Results:**
- Exact case match works
- Search tools may handle case conversion
- No case-related errors

---

## 4. Response Structure Tests

### Test Case 4.1: Response Schema Validation

**Objective:** Verify response structure.

**Test Steps:**
1. Call `list_vulnerability_types`
2. Examine response

**Expected Results:**
- Response is `SingleToolResponse` structure
- `data` is array of strings
- `message` is string or null
- `found` is boolean

---

### Test Case 4.2: Message Field Content

**Objective:** Check message field for info/warnings.

**Test Steps:**
1. Call `list_vulnerability_types`
2. Check `message` field

**Expected Results:**
- If no issues: `message` may be null or empty
- If warnings: `message` contains helpful text
- No errors for normal operation

---

## 5. Error Handling Tests

### Test Case 5.1: API Connection Failure

**Objective:** Test handling when Contrast API unavailable.

**Prerequisites:**
- Simulate API failure (invalid credentials, network issue)

**Test Steps:**
1. Configure invalid credentials
2. Call `list_vulnerability_types`

**Expected Results:**
- Returns error response
- Error message: "Failed to retrieve vulnerability types" or similar
- No crash or hang
- Error is user-friendly (not raw stack trace)

---

### Test Case 5.2: Timeout Handling

**Objective:** Test handling of slow responses.

**Prerequisites:**
- Network with delays

**Test Steps:**
1. Introduce network delays
2. Call `list_vulnerability_types`

**Expected Results:**
- Returns within reasonable timeout
- Either success with data or error message
- No indefinite hang

---

### Test Case 5.3: Repeated Calls Consistency

**Objective:** Verify consistent results across calls.

**Test Steps:**
1. Call `list_vulnerability_types` 3 times
2. Compare results

**Expected Results:**
- All 3 calls return identical results
- No variation between calls
- Consistent performance

---

## 6. Performance Tests

### Test Case 6.1: Response Time Benchmarks

**Objective:** Measure and verify acceptable performance.

**Test Steps:**
1. Call tool and measure response time
2. Repeat 5 times
3. Calculate average

**Expected Results:**
- Small org (1-10 rules): < 1 second
- Medium org (11-50 rules): < 2 seconds
- Large org (50+ rules): < 5 seconds
- Consistent across calls

---

## Test Execution Guidelines

### Pre-Test Setup
1. Verify MCP server running with valid credentials
2. Know approximate number of rules in test organization
3. Have at least one application with vulnerabilities for integration testing

### Test Data Recommendations
- Organization with 10+ rules configured
- At least common OWASP types enabled
- Access to second organization if testing empty case

### Success Criteria
The `list_vulnerability_types` tool passes testing if:
- Basic functionality test passes (TC 1.1)
- Results are alphabetically sorted (TC 1.3)
- All rule names are properly formatted (TC 1.4)
- Integration with search tools works (TC 3.1, 3.2)
- Error handling is graceful (TC 5.1)
- No crashes or hangs
- Performance is acceptable (< 5 seconds)

---

## Test Coverage Summary

| Category | Test Cases | Coverage |
|----------|------------|----------|
| Basic Functionality | 5 | Core behavior verification |
| Empty/Edge Cases | 2 | No data, large data |
| Integration | 3 | Use with search tools |
| Response Structure | 2 | Schema validation |
| Error Handling | 3 | Failures, timeouts |
| Performance | 1 | Response time |

**Total: 16 test cases**

---

## Appendix: Expected Behavior Examples

### Successful Response Example
```json
{
  "data": [
    "cmd-injection",
    "crypto-bad-ciphers",
    "crypto-bad-mac",
    "path-traversal",
    "sql-injection",
    "xss-reflected",
    "xss-stored",
    "xxe"
  ],
  "message": null,
  "found": true
}
```

### Empty Organization Response
```json
{
  "data": [],
  "message": "No rules returned from Contrast API",
  "found": true
}
```

### Error Response Example
```json
{
  "data": null,
  "message": "Failed to retrieve vulnerability types: Connection refused",
  "found": false
}
```

---

## Logging Verification

Check MCP server logs (`/tmp/mcp-contrast.log`) for:
- Info log: "Retrieving all vulnerability types..."
- Info log: "Retrieved {count} vulnerability types"
- No error or warning logs in success case

---

## References

- **Tool Implementation**: `tool/vulnerability/ListVulnerabilityTypesTool.java`
- **Related Tools**: `search_vulnerabilities`, `search_app_vulnerabilities`
- **Old Test Plan**: `test-plan-list_vulnerability_types.md` (root level)
