# Azure Support Agent - Bug Hunting Plan

> Date: 2026-06-10
> Scope: Azure Support Agent application-wide bug hunt

## 1. Goal

Find bugs that matter in real use: broken workflows, bad data persistence, auth/scope leaks, AI/tool execution failures, stale UI state, weak error handling, performance cliffs, and unsafe edge cases.

## 2. Bug Hunt Matrix

Use this matrix for planning and tracking.

| Area | Workflow | Risk | Test type | Expected result | Actual result | Severity | Owner |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  |  |  |  |  |  |

Severity scale:

- **P0:** data loss, cross-tenant leak, unsafe write, auth bypass, app unusable.
- **P1:** major workflow broken, AI/tool generation unusable, dashboard/build/run failures.
- **P2:** important UX/data bug with workaround.
- **P3:** polish, copy, layout, minor edge case.

## 3. Test Identities

Prepare these identities before the bug hunt:

- Admin user.
- Regular user.
- Auditor/read-only user.
- User with no Azure connection.
- User with broken Azure connection.
- User with read-only Azure connection.

## 4. Test Data

Prepare or seed the following:

- At least 2 workloads.
- At least 1 architecture linked to a workload.
- At least 1 architecture memory with meaningful content.
- At least 1 assessment run with findings.
- At least 1 workbook with a successful run.
- At least 1 workbook with a failed run.
- At least 1 Monitor dashboard generated by AI.
- At least 1 Azure connection with a Log Analytics workspace id.
- At least 1 Azure connection without a Log Analytics workspace id.

## 5. Highest-Risk Areas First

### 5.1 Auth, Tenant, Role, and Scope Isolation

Test cases:

1. Regular user cannot access admin endpoints/UI.
2. Auditor cannot mutate settings, dashboards, connections, users, or workbooks.
3. User cannot see another tenant's chats, workloads, dashboards, assessments, or memories.
4. Admin-only Monitor dashboard authoring is blocked for non-admin.
5. `DEV_AUTH=false` requires valid login.
6. Session expiration redirects cleanly.
7. Logout invalidates the session.
8. SSO/OIDC/SAML failure paths do not leave partial users or sessions.
9. Azure connection secrets are never returned to the frontend.
10. Prompt/model/API keys are masked and preserved correctly on update.

Likely bug classes:

- Missing `require_admin`.
- Missing tenant filter.
- Registry JSON objects with blank tenant id visible to everyone.
- Frontend hiding buttons but backend still allowing calls.

### 5.2 Chat and Agent Turn Reliability

Test cases:

1. New chat -> ask simple question -> response streams live.
2. Start long turn -> navigate away -> return -> live stream continues.
3. Start turn -> refresh browser -> partial assistant content is persisted.
4. Start multiple chats streaming simultaneously.
5. Delete chat while stream is running.
6. Delete all chats while stream is running.
7. Tool-call max iteration exhaustion still produces a final answer.
8. Tool error returns useful answer, not an empty bubble.
9. SSE parser handles `\r\n\r\n` and `\n\n`.
10. Image attachment does not break history or streaming.
11. Scope clarification works: ambiguous VM/resource question asks subscription first.
12. Selected workload/architecture memory is injected only when appropriate.

Likely bug classes:

- Race conditions in stream state.
- Duplicate assistant messages.
- Missing persisted activity.
- Stale active chat refs.
- Empty final answer after tool loop cap.
- Backend cancellation losing partial content.

### 5.3 Azure/Entra Tooling and Command Runner

Test cases:

1. MCP tool list loads and caches.
2. Azure MCP unavailable gives a useful error.
3. Entra MCP disabled hides tools and blocks execution.
4. Broken Azure connection surfaces clear status.
5. Read-only commands run.
6. Mutating commands require approval or are blocked.
7. Command runner rejects shell chaining, redirection, pipes outside quotes, and subshells.
8. Resource Graph KQL with pipes inside quotes works.
9. Log Analytics widget query requires workspace id and errors clearly if missing.
10. Built-in network tools block `127.0.0.1`, private IPs, link-local, `169.254.169.254`, and denied hosts.
11. Allowed public web/TCP ping works.
12. Timeouts do not hang the backend.

Likely bug classes:

- Command injection bypass.
- KQL rejected because of `|`.
- SSRF gap on redirects or DNS rebinding.
- Bad error propagation from subprocess.
- Tool catalog stale after settings change.

## 6. Feature Workflows

### 6.1 Workloads, Inventory, and Cost

Test cases:

1. Create workload from subscription, resource group, and resource.
2. Exclude child resources from selected parent scope.
3. Refresh workload from origin.
4. Inventory filters by subscription, region, resource group, resource type, and workload.
5. Map region multi-select filters both map and left menu.
6. Location map narrow-window layout.
7. Cost/inventory staleness indicator turns bold red after 6 hours.
8. Empty inventory state is clear.
9. Resource Graph query-too-long fallback/partitioning works.
10. Subscription names show instead of only IDs.

Likely bug classes:

- Filters not syncing across panes.
- Workload membership drift.
- Subscription name/id mismatch.
- Map label overlap.
- Resource Graph truncation silently hiding data.

### 6.2 Assessments

Test cases:

1. Run assessment for workload.
2. SSE progress streams correctly.
3. Findings table loads.
4. Sort every sortable column.
5. All default filters/checkboxes selected when entering screen.
6. Filter by pillar, severity, status, and resource.
7. Waive finding.
8. Expired waiver stops applying.
9. Create ticket from finding.
10. Create/edit/delete/run custom check.
11. Assessment history and baseline/diff work.
12. Soft-deleted assessment does not show in active views.

Likely bug classes:

- Default selection state wrong.
- Sorting mutates source incorrectly.
- Waiver scoring mismatch.
- Findings count differs from details.
- SSE completion not refreshing final data.

### 6.3 Architectures and Architecture Memory

Test cases:

1. Create architecture manually.
2. Link architecture to workload.
3. Save link and reload.
4. Rebuild architecture from workload.
5. Create memory for architecture.
6. Fill all memory sections and save.
7. Markdown preview matches content.
8. Version/revision restore works.
9. Multiple architectures linked to the same workload prompt user where needed.
10. Memory is used by Deep Investigation.
11. Memory is used by Monitor dashboard generation.
12. Deleting architecture handles memory safely.

Likely bug classes:

- Link save not persisted.
- Memory revisions losing sections.
- Stale selected architecture/workload.
- AI generation overwriting user-edited memory.
- Markdown preview layout overflow.

### 6.4 Workbooks, Playbooks, and Automations

Test cases:

1. Create workbook with `az`.
2. Create workbook with Resource Graph KQL.
3. Create workbook with Log Analytics KQL if workspace configured.
4. Test run unsaved workbook.
5. Save workbook and run.
6. Failed run stores error and status.
7. AI summary/severity/extract works.
8. Workbook tile appears in Monitor.
9. Import/export workbook.
10. Build playbook from multiple workbook steps.
11. Scheduled task fires once when due.
12. Scheduler does not double-run the same task.
13. Manual run records `TaskRun`.
14. Custom agent tool permissions are respected.

Likely bug classes:

- Param interpolation mistakes.
- Command read/write misclassification.
- Run history missing tenant/user.
- Scheduler duplicate dispatch.
- Background task exceptions swallowed without visible status.

### 6.5 Monitor 2.0

Test cases:

1. Built-in dashboard loads.
2. Create dashboard.
3. Add/remove/move/resize widget.
4. Save dashboard, reload, layout persists.
5. Restore previous dashboard version.
6. Add chart widget from static data, app telemetry, Resource Graph, Log Analytics, Azure metrics, workbook ref, web ping, and TCP ping.
7. Widget editor preview works.
8. Per-widget live refresh works.
9. Pause live stops refresh.
10. Web/TCP ping history accumulates.
11. AI builds one widget from prompt.
12. AI builds dashboard from workload for all archetypes:
    - full-stack
    - SRE live
    - incident
    - security
    - cost/capacity
    - executive
13. Generated dashboard contains `ai_design` metadata.
14. Design brief appears in modal.
15. Dry-run repair creates gap widgets instead of broken charts.
16. Critic notes appear when critic score is low.
17. Missing resource id shows useful state.
18. Missing Log Analytics workspace shows useful state.
19. Large dashboard scroll/resize performance is acceptable.

Likely bug classes:

- Widget renderer crashes on odd table shapes.
- Recharts blank canvas with empty/mixed data.
- Dashboard save drops widget fields.
- Legacy tiles migration creates duplicate IDs.
- AI-generated data source has unsupported fields.
- Dry-run uses wrong tenant.
- App telemetry cache stale too long.
- Ping sampler duplicates samples excessively.

### 6.6 Admin and Settings

Test cases:

1. Update LLM provider and model.
2. Blank API key preserves existing secret.
3. Refresh model list for each provider.
4. Test provider streams diagnostics.
5. Hide/unhide models.
6. Built-in tools admin toggle applies immediately.
7. Network egress allow/deny lists apply.
8. Azure connection create/edit/test/default/delete.
9. Log Analytics workspace id persists.
10. App settings save/reload.
11. AI prompts override/reset.
12. Audit log records sensitive actions without leaking secrets.

Likely bug classes:

- Masked secrets overwritten with mask.
- Provider selection stale until restart.
- Admin UI saving wrong provider's model/key.
- Settings apply on frontend only, not backend.
- Audit metadata leaking key snippets.

## 7. Frontend UX and Responsiveness

Test widths:

- 360px mobile.
- 768px tablet.
- 1366px desktop.
- Wide desktop.

Screens to inspect:

- Chat.
- Inventory map.
- Assessments detail.
- Architecture Memory.
- Monitor dashboard/editor.
- Admin provider settings.
- Workbooks editor.
- Automations.

Look for:

- Text overflow in buttons/cards.
- Sticky headers covering content.
- Nested scroll traps.
- Modals too tall to close.
- Tables without horizontal scroll.
- Dashboard widgets unreadable after resize.
- Empty states that look broken.
- Buttons with text/icons wrapping badly.

## 8. Performance Bug Hunt

Measure:

1. Initial app load.
2. `/admin/monitor` aggregation time.
3. Monitor widget run time.
4. App telemetry widget burst refresh.
5. Inventory load with many resources.
6. Assessment run duration by workload size.
7. Resource Graph query fan-out.
8. LLM streaming time-to-first-token.
9. Frontend FPS while dragging Monitor widgets.
10. Backend memory after long AI dashboard build.

Targets:

- Monitor datasource catalog < 50ms.
- Cached app telemetry widget < 100ms.
- Dashboard render has no obvious jank with 20 widgets.
- SSE first visible event within about 1 second after backend starts work.
- No unbounded growth in ping history or live stream maps.

## 9. Security Bug Hunt

Attack-style tests:

1. Try regular user calling admin endpoints directly.
2. Try tenant A ids in tenant B requests.
3. Try prompt injection asking model to approve write.
4. Try command injection in workbook body.
5. Try SSRF through web ping redirect.
6. Try private IP via DNS name.
7. Try huge tool output.
8. Try malformed JSON registry files.
9. Try stored XSS in chat messages, workbook output, markdown widgets, architecture memory, and notification text.
10. Try uploading large images.
11. Try path traversal in import/export endpoints.
12. Try model/provider config with hostile base URL.

## 10. Suggested Execution Order

Run the bug hunt in this order:

1. **Smoke pass**
   Confirm app boots, login, chat, monitor, admin, workloads, and inventory.

2. **P0/P1 safety pass**
   Auth, tenant scope, secrets, command runner, SSRF, and approvals.

3. **Core workflow pass**
   Chat, Azure tools, workloads, inventory, and assessments.

4. **Knowledge/AI pass**
   Architectures, memory, workbooks, and Monitor AI generation.

5. **Scheduler/background pass**
   Automations, workbook runs, ping sampler, and notifications.

6. **Frontend responsive pass**
   Key screens at multiple widths.

7. **Performance pass**
   Monitor, inventory, assessment, and streaming.

8. **Regression pass**
   Re-run smoke plus top critical workflows after fixes.

## 11. Automation to Add

Backend test scripts:

- `test_auth_scope.py`
- `test_command_runner_safety.py`
- `test_builtin_tools_ssrf.py`
- `test_monitor_datasources.py`
- `test_monitor_ai_author.py`
- `test_workbook_executor.py`
- `test_assessment_scoring.py`
- `test_architecture_memory_revisions.py`

Frontend/browser tests:

- Login and route smoke.
- Chat streaming smoke.
- Monitor add/save/reload widget.
- Build dashboard from workload.
- Assessment filter/sort/default selection.
- Inventory map filter behavior.
- Admin LLM provider save/test.

Performance probes:

- Monitor widget burst.
- Inventory large result render.
- Assessment run with synthetic workload.
- SSE time-to-first-event.

## 12. Bug Report Template

```md
### Title
Short, action-oriented.

### Severity
P0 / P1 / P2 / P3

### Area
Chat / Monitor / Assessments / Admin / Security / etc.

### Steps
1.
2.
3.

### Expected
What should happen.

### Actual
What happened.

### Evidence
Screenshot, console error, API response, log snippet.

### Suspected Cause
File/function if known.

### Fix Notes
Smallest safe fix.

### Regression Test
What test prevents it from coming back.
```

## 13. First Bug-Hunt Sprint

### Day 1

- Auth/admin endpoint access matrix.
- Command runner and SSRF tests.
- Chat stream persistence/navigation tests.
- Monitor widget run/save/reload tests.
- Assessment table filters/sort/default selection.

### Day 2

- Architecture Memory save/revision/AI use.
- Workload-to-AI Monitor dashboard generation for all archetypes.
- Inventory map filters.
- Workbooks run/preview/tile behavior.
- Performance smoke on Monitor and Inventory.
- Fix all P0/P1 immediately, queue P2/P3.
