Litmux提示词测试框架

Quick Start

```bash pip install litmux

cp .env.example .env

Examples

See examples/ for three ready-to-run projects:

• 01-quickstart — minimal single-model test
• 02-multi-model — compare across providers
• 03-generate-and-eval — AI-generated dataset + LLM judge

---

Cloud (Optional, Free)

Sync results to a hosted dashboard for history, trends, and team visibility.

litmux login       # one-time browser auth
litmux run         # results auto-sync
litmux dashboard   # open app.litmux.dev

The CLI works fully offline. Cloud is opt-in.

---

Configuration

models:
  - provider: openai | anthropic | google | huggingface
    model: string
    temperature: 0.0
    max_tokens: 1024

defaultTest:
  assert:
    - type: cost-less-than
      value: 0.01

tests:
  - name: string
    prompt: path/to/prompt.txt
    inputs: { variable: "value" }
    assert:
      - type: contains
        value: "expected"

evals:
  - name: string
    prompt: path/to/prompt.txt
    dataset: path/to/data.csv
    input_mapping: { prompt_var: csv_column }
    expected: csv_column
    assert: [...]
    judge:
      criteria: "..."
      threshold: 7.0

Environment Variables

---

Add at least one: OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_API_KEY, HF_TOKEN

litmux init # scaffold a project litmux run # run tests against all configured models ```

No database, no cloud account, no Docker.

---

.github/workflows/litmux.yml

- run: litmux run --ci env: OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} ```

---

`litmux compare` — side-by-side model outputs

litmux compare

---