# CodeFRAME Testing Guide

> Sprint 1 manual testing checklist; for current test organization and counts see
> [`README.md`](README.md) and [`SPRINTS.md`](SPRINTS.md). For the canonical E2E
> workflows that Sprint 10 tests should exercise, see the E2E section in
> [`PRD.md`](PRD.md).

## Sprint 1 Manual Test Checklist

### Setup Requirements

#### Environment Setup
- [ ] Clone repository: `git clone https://github.com/frankbria/codeframe.git`
- [ ] Navigate to project: `cd codeframe`
- [ ] Create virtual environment: `python -m venv venv`
- [ ] Activate virtual environment: `source venv/bin/activate`
- [ ] Install Python dependencies: `pip install -e .`
- [ ] Install Node dependencies: `cd web-ui && npm install && cd ..`

#### Configuration
- [ ] Create `.env` file in project root
- [ ] Add ANTHROPIC_API_KEY: `ANTHROPIC_API_KEY=your-key-here`
- [ ] Verify `.env` is in `.gitignore`

#### Service Startup
- [ ] Start Status Server: `python -m codeframe.ui.server`
  - Expected: Server starts on port 8000
  - Expected: "Status Server running on http://localhost:8000" message
- [ ] Start Web UI (new terminal): `cd web-ui && npm run dev`
  - Expected: UI starts on port 3000
  - Expected: "Local: http://localhost:3000" message

---

### Test 1: Project Creation (cf-8, cf-11)

#### 1.1 Basic Project Initialization
- [ ] Run: `codeframe init test-project`
- [ ] Verify: Project directory created at `./test-project`
- [ ] Verify: `.codeframe/` directory exists in `test-project/`
- [ ] Verify: Database file created: `.codeframe/state.db`
- [ ] Verify: Success message displayed

#### 1.2 Database Verification
- [ ] Open database: `sqlite3 test-project/.codeframe/state.db`
- [ ] Run: `.tables`
- [ ] Verify: All tables exist: projects, tasks, agents, blockers, memory, context_items, checkpoints, changelog
- [ ] Run: `SELECT * FROM projects;`
- [ ] Verify: Project entry exists with name='test-project', status='init'
- [ ] Exit sqlite: `.exit`

#### 1.3 Dashboard Integration
- [ ] Open browser: http://localhost:3000
- [ ] Verify: "test-project" appears in projects list
- [ ] Verify: Status shows "init"
- [ ] Verify: Created timestamp is recent

---

### Test 2: Database CRUD Operations (cf-8)

#### 2.1 Project Operations
- [ ] Create second project: `codeframe init project2`
- [ ] Verify: Both projects appear in dashboard
- [ ] Verify: Both have separate `.codeframe/state.db` files
- [ ] Update project status (via API or direct DB): `UPDATE projects SET status='planning' WHERE name='test-project'`
- [ ] Refresh dashboard
- [ ] Verify: Status updated to "planning"

#### 2.2 Agent Creation
- [ ] Use Python REPL or script:
  ```python
  from codeframe.platform_store.database import Database
  db = Database("test-project/.codeframe/state.db")
  db.initialize()
  agent_id = db.create_agent("lead-1", "lead", "claude", "directive")
  agent = db.get_agent("lead-1")
  print(agent)
  ```
- [ ] Verify: Agent created successfully
- [ ] Verify: Agent details correct (type=lead, provider=claude, maturity=directive)

#### 2.3 Memory Storage
- [ ] Continue in Python REPL:
  ```python
  project = db.get_project(1)
  memory_id = db.create_memory(
      project_id=project['id'],
      category='pattern',
      key='auth_pattern',
      value='JWT with refresh tokens'
  )
  memories = db.get_project_memories(project['id'])
  print(memories)
  ```
- [ ] Verify: Memory entry created
- [ ] Verify: Can retrieve memory by project_id

---

### Test 3: Anthropic Provider Integration (cf-9)

#### 3.1 Provider Initialization
- [ ] Test provider creation:
  ```python
  from codeframe.agents.providers.anthropic_provider import AnthropicProvider
  provider = AnthropicProvider(api_key="test-key")
  ```
- [ ] Verify: Provider initializes without error
- [ ] Verify: Model defaults to "claude-3-5-sonnet-20241022"

#### 3.2 Message Sending (Mock/Test)
- [ ] Run unit tests: `ANTHROPIC_API_KEY="test-key" pytest tests/test_anthropic_provider.py -v`
- [ ] Verify: All 17 tests pass
- [ ] Verify: Message formatting tests pass
- [ ] Verify: Error handling tests pass

---

### Test 4: Lead Agent Lifecycle (cf-9, cf-10)

#### 4.1 Lead Agent Creation
- [ ] Test Lead Agent initialization:
  ```python
  from codeframe.agents.lead_agent import LeadAgent
  from codeframe.platform_store.database import Database

  db = Database("test-project/.codeframe/state.db")
  db.initialize()

  lead = LeadAgent(
      agent_id="lead-test-1",
      provider_name="anthropic",
      api_key="test-key",
      database=db
  )
  ```
- [ ] Verify: Lead Agent creates successfully
- [ ] Verify: Agent entry created in database
- [ ] Verify: Default maturity level is "directive"

#### 4.2 Agent Status Management
- [ ] Update agent status:
  ```python
  lead.update_status("working")
  agent = db.get_agent("lead-test-1")
  print(agent['status'])
  ```
- [ ] Verify: Status updates to "working"
- [ ] Verify: Database reflects change

---

### Test 5: Project Creation API (cf-11)

#### 5.1 API Endpoint Testing
- [ ] Send POST request to create project:
  ```bash
  curl -X POST http://localhost:8000/api/projects \
    -H "Content-Type: application/json" \
    -d '{"name": "api-test-project", "description": "Test via API"}'
  ```
- [ ] Verify: 200 OK response
- [ ] Verify: Response includes project_id and status
- [ ] Verify: Project directory created
- [ ] Verify: Database initialized

#### 5.2 Project Listing API
- [ ] Send GET request:
  ```bash
  curl http://localhost:8000/api/projects
  ```
- [ ] Verify: Returns JSON array
- [ ] Verify: Contains all created projects
- [ ] Verify: Each project has id, name, status, created_at

#### 5.3 WebSocket Real-Time Updates
- [ ] Open browser developer console on dashboard
- [ ] Create new project via CLI: `codeframe init websocket-test`
- [ ] Verify: Dashboard updates automatically (no refresh needed)
- [ ] Verify: Console shows WebSocket message received
- [ ] Check WebSocket connection: Network tab → WS → Messages
- [ ] Verify: `project_created` event appears

---

### Test 6: Agent Lifecycle Management (cf-10)

#### 6.1 Agent State Transitions
- [ ] Test agent lifecycle:
  ```python
  from codeframe.agents.lead_agent import LeadAgent
  from codeframe.platform_store.database import Database
  from codeframe.core.models import AgentMaturity

  db = Database("test-project/.codeframe/state.db")
  db.initialize()

  lead = LeadAgent("lead-lifecycle", "anthropic", "test-key", db)

  # Test state transitions
  lead.update_status("idle")
  lead.update_status("working")
  lead.update_status("blocked")
  lead.update_status("completed")

  # Test maturity progression
  lead.update_maturity(AgentMaturity.D2)
  lead.update_maturity(AgentMaturity.D3)
  ```
- [ ] Verify: All status transitions succeed
- [ ] Verify: Maturity level updates correctly
- [ ] Verify: Database reflects all changes

#### 6.2 Agent Error Handling
- [ ] Test invalid transitions:
  ```python
  lead.update_status("invalid_status")  # Should handle gracefully
  ```
- [ ] Verify: Error handled without crash
- [ ] Verify: Agent remains in valid state

---

### Test 7: End-to-End Integration

#### 7.1 Complete Project Workflow
- [ ] Initialize new project: `codeframe init e2e-test`
- [ ] Verify: Project appears in dashboard immediately
- [ ] Verify: Database created with all tables
- [ ] Create Lead Agent programmatically
- [ ] Store memory entry
- [ ] Update project status to "planning"
- [ ] Verify: All changes visible in dashboard
- [ ] Verify: WebSocket events fire for all updates

#### 7.2 Multi-Project Handling
- [ ] Create 3 projects simultaneously:
  ```bash
  codeframe init project-a &
  codeframe init project-b &
  codeframe init project-c &
  wait
  ```
- [ ] Verify: All 3 projects created successfully
- [ ] Verify: Each has independent database
- [ ] Verify: Dashboard shows all 3 projects
- [ ] Verify: No database conflicts

---

### Test 8: Error Conditions & Edge Cases

#### 8.1 Database Errors
- [ ] Test duplicate project: `codeframe init test-project` (already exists)
- [ ] Verify: Error message displayed
- [ ] Verify: No corruption of existing project

#### 8.2 API Errors
- [ ] Send invalid JSON to API:
  ```bash
  curl -X POST http://localhost:8000/api/projects \
    -H "Content-Type: application/json" \
    -d 'invalid json'
  ```
- [ ] Verify: 400 Bad Request response
- [ ] Verify: Helpful error message

#### 8.3 Missing Configuration
- [ ] Remove ANTHROPIC_API_KEY from `.env`
- [ ] Try to create Lead Agent
- [ ] Verify: Clear error about missing API key
- [ ] Restore API key

---

### Test 9: Performance Verification

#### 9.1 Response Times
- [ ] Measure API response time:
  ```bash
  time curl http://localhost:8000/api/projects
  ```
- [ ] Verify: Response time < 500ms (p95 requirement)

#### 9.2 Database Performance
- [ ] Create 100 memory entries:
  ```python
  for i in range(100):
      db.create_memory(1, 'test', f'key_{i}', f'value_{i}')
  ```
- [ ] Query all memories: `db.get_project_memories(1)`
- [ ] Verify: Query completes in < 1 second

---

### Test 10: Automated Test Suite Verification

#### 10.1 Run All Tests
- [ ] Run complete test suite:
  ```bash
  ANTHROPIC_API_KEY="test-key" pytest -v
  ```
- [ ] Verify: 111 tests pass (100% pass rate)
- [ ] Verify: No warnings or errors

#### 10.2 Test Coverage
- [ ] Run with coverage:
  ```bash
  ANTHROPIC_API_KEY="test-key" pytest --cov=codeframe --cov-report=html
  ```
- [ ] Verify: Overall coverage > 90%
- [ ] Verify: Database module > 92%
- [ ] Open `htmlcov/index.html` to review

---

## Definition of Done Verification

### Sprint 1 Completion Criteria
- [ ] ✅ All 9 tasks complete (cf-8 through cf-13)
- [ ] ✅ 111 automated tests passing at 100%
- [ ] ✅ Zero mock data in production code
- [ ] ✅ Database operations tested at >80% coverage (actual: 92%)
- [ ] ✅ API response time <500ms (p95)
- [ ] ✅ WebSocket reconnect works automatically
- [ ] ✅ Can run `codeframe init` and see project in dashboard
- [ ] ✅ Lead Agent can be created with valid API key
- [ ] ✅ No critical bugs blocking sprint review

### Code Quality Checks
- [ ] ✅ All code follows Python PEP 8 style guide
- [ ] ✅ No hardcoded API keys or secrets in code
- [ ] ✅ All database operations use parameterized queries
- [ ] ✅ Error handling implemented for all external calls
- [ ] ✅ TDD followed for all features (RED-GREEN-REFACTOR)

### Documentation Complete
- [ ] ✅ TESTING.md created with manual test checklist
- [ ] ✅ README.md updated with setup instructions
- [ ] ✅ AGILE_SPRINTS.md reflects actual progress
- [ ] ✅ All API endpoints documented

---

## Test Results Template

```markdown
## Sprint 1 Manual Test Execution Results
**Date**: YYYY-MM-DD
**Tester**: [Name]
**Environment**: [OS, Python version, Node version]

### Setup
- [ ] All setup steps completed successfully
- Issues found: [None | List issues]

### Test 1: Project Creation
- [ ] Passed | [ ] Failed
- Issues found: [None | List issues]
- Notes:

### Test 2: Database CRUD
- [ ] Passed | [ ] Failed
- Issues found: [None | List issues]
- Notes:

[Continue for all tests...]

### Overall Assessment
- Total Tests Run: X
- Tests Passed: Y
- Tests Failed: Z
- Critical Issues: [List]
- Sprint 1 Ready for Demo: [ ] Yes | [ ] No
```

---

## Troubleshooting Guide

### Common Issues

**Issue**: Database file not found
- **Solution**: Ensure you're in the correct project directory, run `codeframe init` first

**Issue**: WebSocket connection fails
- **Solution**: Check Status Server is running on port 8000, check browser console for errors

**Issue**: API key error
- **Solution**: Verify `.env` file exists with valid ANTHROPIC_API_KEY

**Issue**: Import errors
- **Solution**: Activate virtual environment, reinstall with `pip install -e .`

**Issue**: Dashboard doesn't update
- **Solution**: Check WebSocket connection in browser DevTools, restart Status Server

---

## Next Steps (Sprint 2)

After completing Sprint 1 manual testing:
1. Document any critical bugs and fix before demo
2. Prepare sprint demo showing working features
3. Review Sprint 2 tasks: Socratic Discovery phase
4. Plan Sprint 2 implementation starting with CLI foundation

**Sprint 1 Status**: COMPLETE ✅
**Total Test Cases**: 111 automated + 10 manual test scenarios
**Pass Rate**: 100%
**Ready for Production**: Foundation components ready for Sprint 2 integration
