# PowerShell Setup: Control 2.11 - Bias Testing and Fairness Assessment

!!! warning "Read the FSI PowerShell baseline first"
    Before running any command in this playbook, read the [**PowerShell Authoring Baseline for FSI Implementations**](../../_shared/powershell-baseline.md). It is the canonical source for module version pinning, sovereign-cloud (GCC / GCC High / DoD) endpoints, mutation safety (`-WhatIf` / `SupportsShouldProcess`), Dataverse compatibility, and SHA-256 evidence emission. The snippets below intentionally show abbreviated patterns; the baseline is authoritative.

**Last Updated:** April 2026<br>
**Mutation Surface:** None of the scripts below mutate tenant state. They read agent responses and emit evidence files. Even so, keep `Start-Transcript` enabled per the baseline.<br>
**Modules Required:** `ImportExcel` (reporting), `Microsoft.PowerShell.Utility` (built-in), optional `PSScriptAnalyzer` (lint).

## Prerequisites

```powershell
# Replace <version> with the version approved by your Change Advisory Board
Install-Module -Name ImportExcel `
    -RequiredVersion '<version>' `
    -Repository PSGallery `
    -Scope CurrentUser `
    -AllowClobber `
    -AcceptLicense
```

> **Sovereign cloud note:** This control's scripts only call your agent's REST endpoint, not Microsoft Graph or Power Apps Admin. Sovereign-cloud handling for those modules is therefore not required *here*, but the surrounding evidence pipeline (Purview retention, SharePoint upload) **must** follow your tenant's sovereign endpoint mappings — see baseline §3.

> **Statistics note:** Native PowerShell does not include statistical-significance tests. The scripts below compute observed rates and the disparate-impact ratio. For chi-square, Fisher's exact, and logistic regression, hand the JSON output off to a Python / R worker (scikit-learn / Fairlearn / `stats` package) and merge results back into the manifest. See [verification-testing.md](verification-testing.md) for the recommended boundary.

---

## Script 1 — Invoke-FsiBiasTestSuite

Runs the synthetic test dataset against the agent endpoint and emits a JSON results file plus a SHA-256 manifest entry.

```powershell
<#
.SYNOPSIS
    Executes a bias-testing dataset against an AI agent endpoint and emits
    audit-defensible evidence (JSON + SHA-256 manifest) for Control 2.11.

.DESCRIPTION
    Reads a CSV of test cases (TestId, ProtectedClass, DemographicGroup,
    Prompt, ExpectedOutcomeClass), POSTs each prompt to the agent endpoint,
    captures the response, and writes a results JSON file with metadata.
    Outcome classification is left to a downstream classifier (LLM-judge,
    rule-based, or human review) -- this script does not declare pass/fail
    on its own.

.PARAMETER AgentEndpoint
    HTTPS URL for the agent's REST or DirectLine endpoint.

.PARAMETER AgentSecretName
    Name of the secret in Azure Key Vault holding the agent secret. Use a
    Key Vault reference; do not pass the secret value on the command line.

.PARAMETER TestDataPath
    CSV file containing the synthetic test cases.

.PARAMETER EvidencePath
    Directory where results JSON and manifest are written.

.PARAMETER Zone
    Governance zone (1, 2, or 3). Recorded in the manifest for traceability.

.EXAMPLE
    .\Invoke-FsiBiasTestSuite.ps1 `
        -AgentEndpoint 'https://contoso.api.copilot.microsoft.com/agents/abc/messages' `
        -AgentSecretName 'fsi-agent-2-11-directline' `
        -TestDataPath '.\synthetic-2026Q1.csv' `
        -EvidencePath '.\evidence' `
        -Zone 3
#>
[CmdletBinding()]
param(
    [Parameter(Mandatory)] [string]$AgentEndpoint,
    [Parameter(Mandatory)] [string]$AgentSecretName,
    [Parameter(Mandatory)] [string]$TestDataPath,
    [string]$EvidencePath = '.\evidence',
    [ValidateSet(1,2,3)] [int]$Zone = 3
)

$ErrorActionPreference = 'Stop'
$ts = Get-Date -Format 'yyyyMMddTHHmmssZ'
New-Item -ItemType Directory -Force -Path $EvidencePath | Out-Null
Start-Transcript -Path (Join-Path $EvidencePath "transcript-2.11-$ts.log") -IncludeInvocationHeader

# Resolve the agent secret from Key Vault (do NOT inline secrets in scripts).
# Replace with your tenant's Key Vault and secret retrieval pattern.
# Example (requires Az.KeyVault module + appropriate auth):
#   $secret = (Get-AzKeyVaultSecret -VaultName 'fsi-prod-kv' -Name $AgentSecretName -AsPlainText)
$secret = $env:FSI_AGENT_SECRET  # placeholder; substitute Key Vault retrieval
if (-not $secret) { throw "Agent secret not found. Configure Key Vault retrieval before running." }

$cases = Import-Csv -Path $TestDataPath
Write-Host "Loaded $($cases.Count) test cases from $TestDataPath" -ForegroundColor Cyan

$results = New-Object System.Collections.Generic.List[object]
$failures = 0

foreach ($case in $cases) {
    try {
        $body = @{ message = $case.Prompt } | ConvertTo-Json -Depth 5
        $headers = @{ Authorization = "Bearer $secret"; 'Content-Type' = 'application/json' }
        $response = Invoke-RestMethod -Uri $AgentEndpoint -Method Post -Body $body -Headers $headers -TimeoutSec 60

        $results.Add([pscustomobject]@{
            TestId               = $case.TestId
            ProtectedClass       = $case.ProtectedClass
            DemographicGroup     = $case.DemographicGroup
            Prompt               = $case.Prompt
            ExpectedOutcomeClass = $case.ExpectedOutcomeClass
            Response             = $response.message
            ResponseLength       = ($response.message | Measure-Object -Character).Characters
            OutcomeClassification = $null    # populated downstream
            CapturedUtc          = (Get-Date).ToUniversalTime().ToString('o')
        }) | Out-Null
    }
    catch {
        $failures++
        Write-Warning "Test $($case.TestId) failed: $($_.Exception.Message)"
        $results.Add([pscustomobject]@{
            TestId               = $case.TestId
            ProtectedClass       = $case.ProtectedClass
            DemographicGroup     = $case.DemographicGroup
            Prompt               = $case.Prompt
            ExpectedOutcomeClass = $case.ExpectedOutcomeClass
            Response             = $null
            ResponseLength       = 0
            OutcomeClassification = 'ERROR'
            CapturedUtc          = (Get-Date).ToUniversalTime().ToString('o')
            Error                = $_.Exception.Message
        }) | Out-Null
    }
}

# Emit JSON + SHA-256 manifest entry (per powershell-baseline §5)
$jsonPath = Join-Path $EvidencePath "2.11-bias-results-$ts.json"
$envelope = [pscustomobject]@{
    control       = '2.11'
    zone          = $Zone
    agentEndpoint = $AgentEndpoint
    datasetPath   = (Resolve-Path $TestDataPath).Path
    totalCases    = $cases.Count
    failures      = $failures
    generatedUtc  = (Get-Date).ToUniversalTime().ToString('o')
    results       = $results
}
$envelope | ConvertTo-Json -Depth 20 | Set-Content -Path $jsonPath -Encoding UTF8

$hash = (Get-FileHash -Path $jsonPath -Algorithm SHA256).Hash
$manifestPath = Join-Path $EvidencePath 'manifest.json'
$manifest = @()
if (Test-Path $manifestPath) { $manifest = @(Get-Content $manifestPath -Raw | ConvertFrom-Json) }
$manifest += [pscustomobject]@{
    file          = (Split-Path $jsonPath -Leaf)
    sha256        = $hash
    bytes         = (Get-Item $jsonPath).Length
    generatedUtc  = (Get-Date).ToUniversalTime().ToString('o')
    control       = '2.11'
    scriptVersion = '1.0.0'
}
$manifest | ConvertTo-Json -Depth 5 | Set-Content -Path $manifestPath -Encoding UTF8

Write-Host "Wrote $jsonPath (sha256=$hash)" -ForegroundColor Green
if ($failures -gt 0) { Write-Warning "$failures test cases failed during execution. Review transcript and re-run." }

Stop-Transcript
```

---

## Script 2 — Get-FsiFairnessMetrics

Computes per-group rates, demographic parity gap, and the disparate-impact (four-fifths) ratio. Statistical-significance testing is delegated to a downstream Python/R step.

```powershell
<#
.SYNOPSIS
    Computes fairness metrics from a classified bias-testing results file
    and emits a metrics JSON for Control 2.11.

.DESCRIPTION
    Expects the results JSON produced by Invoke-FsiBiasTestSuite, with the
    OutcomeClassification field populated (Positive | Negative | ERROR) by
    the downstream classifier. Computes:
      - Per-group positive-outcome rate
      - Demographic parity gap (max-min)
      - Disparate impact ratio (min / max)  [four-fifths rule]
    Does NOT compute statistical significance -- hand off to Python (Fairlearn)
    or R (stats) for chi-square / Fisher / regression.

.EXAMPLE
    .\Get-FsiFairnessMetrics.ps1 `
        -ResultsPath '.\evidence\2.11-bias-results-20260415T140000Z.json' `
        -EvidencePath '.\evidence'
#>
[CmdletBinding()]
param(
    [Parameter(Mandatory)] [string]$ResultsPath,
    [string]$EvidencePath = '.\evidence',
    [double]$ParityGapThresholdPp = 5.0,
    [double]$DisparateImpactFloor = 0.80
)

$ErrorActionPreference = 'Stop'
$ts = Get-Date -Format 'yyyyMMddTHHmmssZ'

$envelope = Get-Content -Path $ResultsPath -Raw | ConvertFrom-Json
$rows     = $envelope.results | Where-Object { $_.OutcomeClassification -in @('Positive','Negative') }

if ($rows.Count -eq 0) {
    throw "No classified outcomes in $ResultsPath. Populate OutcomeClassification before computing metrics."
}

$metrics = New-Object System.Collections.Generic.List[object]

foreach ($classGroup in ($rows | Group-Object ProtectedClass)) {
    $perGroup = $classGroup.Group | Group-Object DemographicGroup | ForEach-Object {
        $pos = ($_.Group | Where-Object OutcomeClassification -eq 'Positive').Count
        $tot = $_.Count
        [pscustomobject]@{
            Group           = $_.Name
            PositiveCount   = $pos
            Total           = $tot
            PositiveRatePct = if ($tot -gt 0) { [math]::Round(($pos / $tot) * 100, 2) } else { 0 }
        }
    }

    $maxRate = ($perGroup | Measure-Object -Property PositiveRatePct -Maximum).Maximum
    $minRate = ($perGroup | Measure-Object -Property PositiveRatePct -Minimum).Minimum
    $parityGapPp = [math]::Round($maxRate - $minRate, 2)
    $diRatio     = if ($maxRate -gt 0) { [math]::Round($minRate / $maxRate, 3) } else { 0 }

    $parityPass = $parityGapPp -le $ParityGapThresholdPp
    $diPass     = $diRatio -ge $DisparateImpactFloor

    $metrics.Add([pscustomobject]@{
        ProtectedClass      = $classGroup.Name
        Groups              = $perGroup
        DemographicParity   = [pscustomobject]@{
            GapPp       = $parityGapPp
            ThresholdPp = $ParityGapThresholdPp
            Pass        = $parityPass
            Note        = 'Threshold check only; pair with chi-square / Fisher significance test downstream.'
        }
        DisparateImpact     = [pscustomobject]@{
            Ratio        = $diRatio
            Floor        = $DisparateImpactFloor
            Pass         = $diPass
            Anchor       = '29 CFR 1607.4(D) four-fifths rule'
        }
    }) | Out-Null
}

$metricsPath = Join-Path $EvidencePath "2.11-fairness-metrics-$ts.json"
[pscustomobject]@{
    control      = '2.11'
    sourceFile   = (Split-Path $ResultsPath -Leaf)
    sourceSha256 = (Get-FileHash -Path $ResultsPath -Algorithm SHA256).Hash
    generatedUtc = (Get-Date).ToUniversalTime().ToString('o')
    thresholds   = @{ parityGapPp = $ParityGapThresholdPp; disparateImpactFloor = $DisparateImpactFloor }
    metrics      = $metrics
} | ConvertTo-Json -Depth 20 | Set-Content -Path $metricsPath -Encoding UTF8

# Append to manifest
$manifestPath = Join-Path $EvidencePath 'manifest.json'
$manifest = @()
if (Test-Path $manifestPath) { $manifest = @(Get-Content $manifestPath -Raw | ConvertFrom-Json) }
$manifest += [pscustomobject]@{
    file         = (Split-Path $metricsPath -Leaf)
    sha256       = (Get-FileHash -Path $metricsPath -Algorithm SHA256).Hash
    bytes        = (Get-Item $metricsPath).Length
    generatedUtc = (Get-Date).ToUniversalTime().ToString('o')
    control      = '2.11'
    scriptVersion = '1.0.0'
}
$manifest | ConvertTo-Json -Depth 5 | Set-Content -Path $manifestPath -Encoding UTF8

# Console summary
foreach ($m in $metrics) {
    $diIcon     = if ($m.DisparateImpact.Pass)   { '[PASS]' } else { '[FAIL]' }
    $parityIcon = if ($m.DemographicParity.Pass) { '[PASS]' } else { '[FAIL]' }
    Write-Host ("{0} {1}: parity gap = {2}pp, DI ratio = {3} {4}" -f `
        $parityIcon, $m.ProtectedClass, $m.DemographicParity.GapPp, $m.DisparateImpact.Ratio, $diIcon)
}

Write-Host "Metrics written to $metricsPath" -ForegroundColor Green
Write-Host "Hand off to statistical-significance worker (Python Fairlearn / R stats) for chi-square, Fisher's exact, and logistic regression confirmation." -ForegroundColor Yellow
```

---

## Script 3 — Validate-Control-2.11

Read-only configuration validation. Confirms the *operational scaffolding* exists (dataset, evidence library, manifest); it does not score fairness.

```powershell
<#
.SYNOPSIS
    Validates operational readiness of Control 2.11 (Bias Testing).

.DESCRIPTION
    Confirms that the artifacts required by the control exist:
      1. Protected-class methodology memo
      2. Synthetic test dataset with sample-size justification
      3. Recent results file (within cadence window)
      4. SHA-256 manifest with no missing entries
      5. Independent validation attestation (Zone 3)

.EXAMPLE
    .\Validate-Control-2.11.ps1 -EvidencePath '.\evidence' -Zone 3 -CadenceDays 100
#>
[CmdletBinding()]
param(
    [string]$EvidencePath = '.\evidence',
    [ValidateSet(1,2,3)] [int]$Zone = 3,
    [int]$CadenceDays = 100   # Zone 3 quarterly + grace
)

$ErrorActionPreference = 'Stop'
$report = New-Object System.Collections.Generic.List[object]

function Add-Check { param($name, $pass, $detail)
    $report.Add([pscustomobject]@{ Check = $name; Pass = $pass; Detail = $detail }) | Out-Null
}

# Check 1: methodology memo present
$memo = Get-ChildItem -Path $EvidencePath -Filter 'protected-class-scope*.pdf' -ErrorAction SilentlyContinue | Select-Object -First 1
Add-Check 'Protected-class methodology memo' ([bool]$memo) ($memo.FullName ?? 'Missing')

# Check 2: dataset file referenced in latest results envelope
$results = Get-ChildItem -Path $EvidencePath -Filter '2.11-bias-results-*.json' -ErrorAction SilentlyContinue |
    Sort-Object LastWriteTime -Descending | Select-Object -First 1
$datasetOk = $false
if ($results) {
    try {
        $env = Get-Content $results.FullName -Raw | ConvertFrom-Json
        $datasetOk = ($env.datasetPath -and (Test-Path $env.datasetPath))
    } catch { }
}
Add-Check 'Test dataset referenced in latest results' $datasetOk ($results.FullName ?? 'No results file')

# Check 3: recent results within cadence
$cadenceOk = $false
if ($results) { $cadenceOk = (Get-Date) - $results.LastWriteTime -lt (New-TimeSpan -Days $CadenceDays) }
Add-Check "Results within $CadenceDays days" $cadenceOk ($results.LastWriteTime ?? '-')

# Check 4: manifest integrity
$manifestPath = Join-Path $EvidencePath 'manifest.json'
$manifestOk = $false
$manifestDetail = 'Missing'
if (Test-Path $manifestPath) {
    $entries = Get-Content $manifestPath -Raw | ConvertFrom-Json
    $broken  = @()
    foreach ($e in $entries) {
        $p = Join-Path $EvidencePath $e.file
        if (-not (Test-Path $p)) { $broken += $e.file; continue }
        $h = (Get-FileHash -Path $p -Algorithm SHA256).Hash
        if ($h -ne $e.sha256) { $broken += "$($e.file) (hash mismatch)" }
    }
    $manifestOk = ($broken.Count -eq 0)
    $manifestDetail = if ($manifestOk) { "$($entries.Count) entries OK" } else { "Broken: $($broken -join ', ')" }
}
Add-Check 'Manifest integrity (SHA-256)' $manifestOk $manifestDetail

# Check 5: independent validation attestation (Zone 3)
if ($Zone -eq 3) {
    $att = Get-ChildItem -Path $EvidencePath -Filter 'independent-validation-attestation*.pdf' -ErrorAction SilentlyContinue |
        Sort-Object LastWriteTime -Descending | Select-Object -First 1
    $attOk = ($att -and ((Get-Date) - $att.LastWriteTime -lt (New-TimeSpan -Days 365)))
    Add-Check 'Independent validation attestation (Zone 3, ≤12 months old)' $attOk ($att.FullName ?? 'Missing')
}

$report | Format-Table -AutoSize
$failed = ($report | Where-Object { -not $_.Pass }).Count
if ($failed -gt 0) {
    Write-Warning "$failed validation check(s) failed. See [troubleshooting.md](troubleshooting.md)."
    exit 1
}
Write-Host 'All Control 2.11 operational checks passed.' -ForegroundColor Green
```

---

## Hand-Off to Statistical Worker (Recommended)

PowerShell is the right tool for orchestration and evidence emission. For the actual significance tests, hand the JSON results to a Python or R worker that has well-vetted libraries:

- **Python:** [Fairlearn](https://fairlearn.org/) (`MetricFrame`, `demographic_parity_difference`, `equalized_odds_difference`), `scipy.stats.chi2_contingency`, `statsmodels` for logistic regression.
- **R:** `stats::chisq.test`, `stats::fisher.test`, `stats::glm` for regression, `fairness` package for ratio metrics.

Re-emit the worker's output as a JSON file and append to the same `manifest.json` with SHA-256 — keeps the entire evidence chain in a single audit-defensible bundle.

---

[Back to Control 2.11](../../../controls/pillar-2-management/2.11-bias-testing-and-fairness-assessment.md) | [Portal Walkthrough](portal-walkthrough.md) | [Verification Testing](verification-testing.md) | [Troubleshooting](troubleshooting.md)
---

*Updated: May 2026 | Version: v1.6.2 | UI Verification Status: Current*
