# Regenerate dataset answers with an EAGLE target model

EAGLE drafters learn best when the assistant turns in the training data are
produced by the same model that will serve as the inference target. Most
public chat datasets were generated by other models, so their assistant
tokens are off-distribution for the drafter. This guide walks through using
`nemo_automodel.components.speculative.regenerate` to replace those answers
with fresh ones from the target model.

## When to use this

- You want to train an EAGLE-3 drafter for a specific target model (say
  `meta-llama/Llama-3.1-8B-Instruct`), but the only conversational data you
  have was generated by a different model.
- You already have a curated prompts dataset (e.g. ShareGPT, UltraChat, an
  internal chat corpus) and want to align its answer distribution with the
  target.

If a regenerated dataset already exists for your target on the Hub (e.g.
`frankleeeee/PerfectBlend-Regenerated-Llama-3.1-8B-Instruct`), just point
the EAGLE-3 recipe's `train_data_path` at it — no need to regenerate.

## Two-step flow

The script talks to an OpenAI-compatible chat completion endpoint, so it
expects the target model to already be served. We use SGLang for this; vLLM
or any other OpenAI-compatible server works too.

### Step 1 — start the target server

In one shell:

```bash
python -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-8B-Instruct \
    --port 30000
```

Wait for the `Uvicorn running on http://0.0.0.0:30000` line before moving
on. For multi-GPU targets pass `--tp 2` (or higher) to shard the weights.

### Step 2 — regenerate

In another shell, point the regeneration script at the running server:

```bash
python -m nemo_automodel.components.speculative.regenerate \
    --input-data Aeala/ShareGPT_Vicuna_unfiltered \
    --output-dir ./regenerated/sharegpt_llama31_8b \
    --target-server http://localhost:30000/v1 \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --concurrency 64 \
    --shard-size 1000
```

For each sample the script:

1. Loads the `messages` column from the input dataset (HF Hub repo id,
   local parquet / parquet dir, or JSON / JSONL — same loader as
   `ChatDataset`).
2. Drops every trailing `assistant` turn, keeping the leading
   `system / user / ...` context (multi-turn conversations keep their
   intermediate assistant turns).
3. Calls `/v1/chat/completions` on the target server with that prompt
   and appends the response as the new assistant turn.
4. Writes the rebuilt conversations to `shard-NNNNNN.parquet` files of
   `--shard-size` rows each.

The script is resumable: rerunning with the same `--output-dir` and
`--resume` skips any shard indices already on disk. Resume is guarded by a
`manifest.json` in the output directory; if you change the input dataset,
split, target model, or shard sizing, the script will fail fast instead of
silently mixing shards from incompatible runs.

## Plugging it into EAGLE-3 training

The output directory is a parquet dataset with a `messages` column, which
is exactly what `ChatDataset` (used by `build_eagle3_dataloader`) consumes.
Point your EAGLE-3 recipe config at it:

```yaml
recipe_args:
  target_model_name_or_path: meta-llama/Llama-3.1-8B-Instruct
  train_data_path: ./regenerated/sharegpt_llama31_8b
  val_data_path: null
```

## Tuning knobs

| Flag | Default | Notes |
|---|---|---|
| `--concurrency` | 32 | In-flight requests; raise to saturate the target server. |
| `--shard-size` | 1000 | Smaller shards = more frequent checkpointing, more files. |
| `--max-new-tokens` | 1024 | Cap per-answer length. |
| `--temperature` | 0.0 | Greedy by default; EAGLE drafters are typically trained against argmax-style answers. |
| `--top-p` | 1.0 | Only relevant with `temperature > 0`. |
| `--timeout-s` | 600 | Per-request timeout. Bump for very long generations. |
| `--max-retries` | 3 | Retries on 5xx, 429, and transport errors with exponential backoff. |
| `--split` | `train` | Supports HF slice syntax, e.g. `train[:10000]`. |
| `--shuffle-seed` | unset | Optional shuffle before slicing. |

## Pitfalls

- **Wrong model name.** The `--model` flag is what gets sent in the
  OpenAI-compatible payload. It must match the name SGLang serves — by
  default SGLang uses `--model-path` as the served name, but if you pass
  `--served-model-name` to SGLang, mirror that here.
- **Server not warm.** Send one curl request to
  `http://localhost:30000/v1/chat/completions` first to confirm the server
  is up; otherwise the script will retry-then-fail on the first batch.
- **Tokenizer mismatch.** The regenerated dataset is consumed by
  `ChatDataset`, which applies the *target* model's chat template at
  training time. Make sure the EAGLE-3 recipe's tokenizer comes from the
  same model id you used for `--model` here, otherwise the loss mask
  alignment will silently drift.
