# worker_plan agent instructions

Scope: FastAPI worker service and shared `worker_plan_internal`/`worker_plan_api`
packages used by frontends and database workers. Keep interfaces stable across
consumers.

## Guidelines
- Preserve the public API contract in `worker_plan/app.py`:
  - Keep request/response shapes and endpoint paths backward compatible.
  - Avoid renaming response fields like `run_id`, `run_dir`, `display_run_dir`.
- Artifact contract: `/runs/{run_id}/zip` must not include
  `track_activity.jsonl` in downloadable zips.
- Maintain the run directory conventions (`PlanExe_...`); run outputs go under
  `{PLANEXE_CONFIG_PATH}/run/`.
- When changing pipeline behavior, keep the subprocess invocation in
  `start_pipeline_subprocess` consistent with `worker_plan_internal`.
- Keep `PlanExeDotEnv.load().update_os_environ()` early so `.env` overrides work.
- CRITICAL: `worker_plan_api` must stay lightweight. Allowed imports are
  stdlib, `typing`, `dataclasses`, `enum`, and `pydantic`. Do not import
  `llama_index`, `fastapi`, `httpx`, `numpy`, `pandas`, or `torch` there.
- Keep planning logic in `worker_plan_internal` (pipeline stages:
  prompt parsing -> LLM planning -> file/report output).
- Other services may import `worker_plan_api` for types/helpers; they must not
  import `worker_plan_internal`.
- If new environment variables or endpoints are added, update
  `worker_plan/README.md` and any Railway docs.

## LLM Error Handling
- When an LLM call fails inside a pipeline task, raise `LLMChatError` (from
  `worker_plan_internal.llm_util.llm_errors`), **not** a bare `ValueError`.
  `LLMChatError` preserves the root cause and generates an `error_id` UUID for
  cross-referencing logs with `usage_metrics.jsonl` rows.
- Standard pattern (30+ call sites):
  ```python
  from worker_plan_internal.llm_util.llm_errors import LLMChatError
  try:
      result = llm_executor.run(execute_function)
  except PipelineStopRequested:
      raise
  except Exception as e:
      llm_error = LLMChatError(cause=e)
      logger.debug(f"LLM chat interaction failed [{llm_error.error_id}]: {e}")
      logger.error(f"LLM chat interaction failed [{llm_error.error_id}]", exc_info=True)
      raise llm_error from e
  ```
- For tasks with multiple LLM calls, use the `message` parameter to
  distinguish them: `LLMChatError(cause=e, message="LLM chat interaction 2 failed")`.
- Error classification: `classify_error()` in `usage_metrics.py` maps raw
  exception strings to short categories (`invalid_json`, `timeout`,
  `rate_limit`, etc.) stored in `usage_metrics.jsonl`. Unknown errors preserve
  a truncated `error_detail` field.

## Pipeline Stages (`worker_plan_internal/plan/nodes/`)

Each Luigi pipeline task lives in its own file under `stages/`. This enables:
- Multiple agents working on different stages without merge conflicts
- `self_improve/` targeting individual step files
- Easy DAG insertion (create new file, update downstream `requires()`)

### Convention for new stages

1. Create `stages/<stage_name>.py` with one task class
2. Import `PlanTask` from `worker_plan_internal.plan.run_plan_pipeline`
3. Import upstream task dependencies from sibling stage files
4. Declare dependencies via `requires()` returning upstream task(s)
5. Add the new task to `stages/full_plan_pipeline.py`'s `requires()` dict

### Framework location

`run_plan_pipeline.py` contains only shared framework:
- `PlanTask` (base class for all stages)
- `ExecutePipeline`, `HandleTaskCompletionParameters`, `PipelineProgress`
- `_task_class_to_step_label`, `configure_logging`
- `__main__` entry point

## Testing
- Prefer unit tests over manual server checks. Run `python test.py` from repo
  root; worker tests live under `worker_plan/worker_plan_internal/**/tests` and
  `worker_plan/worker_plan_api/tests`.
- Error handling tests: `worker_plan/tests/test_llm_errors.py` and
  `worker_plan/tests/test_usage_metrics.py`.
