---
name: ember-train
description: Train AND serve a specialized model on a confidential dataset through Ember's private-training pipeline (the `pl` CLI) — dataset in, verified fine-tuned model out, optionally served as a metered OpenAI-compatible endpoint. Use when the user mentions Ember training, ZOA Training, private fine-tuning, training on confidential data (transcripts, emails, support threads), the pl pipeline, pl serve, or embersovereignty.com/train.
---

# Training a specialized model on confidential data with Ember

`pl pipeline` turns a raw confidential dataset into a fine-tuned open-weight
model with cryptographic proof of custody: the data is encrypted on the
owner's machine under a fresh data key, and that key is released **only** to
a trainer running in an attested confidential VM (Intel TDX) whose exact
workload — container digest, base model, hyperparameters — passed policy.
The run ends with a LoRA adapter plus a receipt signed inside the enclave:
base model, dataset digests, and a held-out eval score (baseline → final),
verifiable long after the VM is gone. Nobody outside the enclave — Ember
included — can read the data.

Human-readable companion: https://embersovereignty.com/train/

## Prerequisites

- Python ≥ 3.10, then: `pip install https://embersovereignty.com/train/private-learning.tar.gz`
- The `claude` CLI, signed in — dataset distillation and eval generation run
  through it locally (`claude -p`; a subscription, no API key). Hundreds of
  calls for a real corpus: budget hours, not minutes, for prep.
- One of the two custody setups under "Training backends" below.

## The custody rules (the pipeline enforces them — don't fight them)

1. **The workdir must be outside any git repository.** Everything derived
   from the source data (chunks, distilled examples, eval questions, the
   adapter) is as confidential as the data itself. The config loader hard-fails
   on a workdir inside a repo. Never commit anything from the workdir.
2. **Eval questions come only from held-out documents** (every Nth document;
   `holdout_every`). Never mix them into training.
3. **The adversarial eval filter is mandatory and will discard most
   questions.** Every generated question is posed to a no-context generalist
   model; anything it can guess is dropped. Keeping only ~30–40% is normal
   and correct — an eval that a stranger can pass measures nothing. If fewer
   than `min_eval_questions` survive, the run warns: the score will be noisy;
   prefer adding held-out data over lowering the bar.
4. **No personal names in distilled output** (built into the prompts): the
   tuned model may later be served to third parties per-token.

## Quickstart

```sh
pl pipeline init --config my-model.json
# edit the config (below), then:
pl pipeline run --config my-model.json --until assemble   # prep only — no GPU, no money
# inspect <workdir>/train.jsonl and <workdir>/eval.jsonl, then:
pl pipeline run --config my-model.json                     # seal → train → collect → verify
pl pipeline status --config my-model.json                  # per-stage progress any time
```

Stages: `ingest → distill → evalgen → filter → assemble → submit → collect
→ verify`. Completed stages are recorded in `<workdir>/pipeline-state.json`
and skipped on re-run; `distill`/`evalgen` resume mid-stage, so a killed run
loses nothing. `--redo STAGE` re-runs a stage and everything after it.
Always run `--until assemble` first and show the user dataset stats before
anything that costs GPU time.

## The config

```json
{
  "name": "acme-support-model",
  "workdir": "~/pl-workdirs/acme-support",
  "source": { "format": "docs-jsonl", "path": "~/exports/support-threads.jsonl" },
  "domain": "Acme's internal payment-infrastructure engineering — ledger design, reconciliation, incident history, and the reasoning behind those choices",
  "subject_hint": "Acme Corp's internal payment infrastructure",
  "holdout_every": 10,
  "prep": { "model": "sonnet", "parallel": 8 },
  "custody": { "backend": "hosted", "trainer_url": "http://TRAINER_IP:8000",
               "hosted": { "session_token_file": "~/.ember/session.json" } }
}
```

- `domain` is the highest-leverage field: one rich sentence describing what
  the model should be a deep expert on. It drives both distillation and eval
  generation. `subject_hint` is a short label shown to the eval-filter model.
- `eval_jsonl` (optional): bring your own MCQ eval and skip generation+filter.

### Source formats

| `source.format` | shape |
|---|---|
| `docs-jsonl` | one JSON object per line: `{"text": "...", "title"?, "date"? ("YYYY-MM-DD"), "id"?, "notes"?}` — the easiest target; convert emails/chat/tickets to this |
| `text` | a directory of `.md`/`.txt`, one document per file; a leading `YYYY-MM-DD` in the filename becomes the date |
| `transcripts` | `<path>/otter-transcripts/*.md` + `<path>/gemini-transcripts/*.docx` (meeting-transcript exports; duplicates deduped by date+title) |
| `sft-jsonl` | ready `{"messages": [{"role": "user"|"assistant", ...}]}` rows — skips distillation; held-out rows still drive eval generation |

Dates matter: the holdout split is every Nth document in date order.
Rough sizing from the reference run: ~750k words of transcripts distilled to
~1,300 training examples and supported a ~90-question filtered eval.

## Training backends

### `hosted` — Ember runs the infrastructure (default)

The data key is sealed to Ember's release policy engine (a threshold network
running the attestation gates; approval policy lives on-chain), so the run
needs no key ceremony from you. Requirements:

- An Ember account: sign in at https://embersovereignty.com/console (email
  code). Put the session token in `~/.ember/session.json` as
  `{"session_token": "..."}` or export `EMBER_SESSION_TOKEN`.
- A trainer for the job: hosted trainer VMs are provisioned per engagement
  today — the user requests one at https://embersovereignty.com/briefing/
  and sets `custody.trainer_url` to the address they receive. (Orgs
  operating their own GCP can instead set `custody.project_dir` — see below —
  and still use hosted release custody.)

One constraint to relay to the user: the on-chain policy pins the exact
trainer recipe **including the base model and epochs**. A hosted run trains
what the approved recipe says; changing the base model is a policy change
Ember has to approve, not a config knob.

### `password` — self-hosted, no Ember account (for experiments)

You hold the key custody: the data key is wrapped with a passphrase
(`export PL_PASSPHRASE=...`), and release verification (attestation quote →
Intel root, workload measurement, freshness) runs on the user's machine
before the key is handed to the enclave. Any base model, any params — edit
the trainer compose freely; the pipeline pins whatever it deploys.

Requirements: a GCP project with Intel TDX quota (H100 spot:
`a3-highgpu-1g` in `us-central1-a`), gcloud auth, and the `dstack-cloud`
CLI. Extract the install tarball (`mkdir private-learning && curl -fsSL
https://embersovereignty.com/train/private-learning.tar.gz | tar xz -C
private-learning`) and point `custody.project_dir` at
`private-learning/profiles/job-axolotl` (H100 QLoRA) or
`…/profiles/dev-cpu` (cheap smoke test) — set `gcp_config.project` in its
`app.json` first. The pipeline deploys the CVM, trains, and **destroys it
after a verified receipt** (`custody.destroy_on_success`, default true). If
a run fails, the CVM is left up for debugging — remind the user it bills
hourly until destroyed (`pl job destroy --project-dir …`).

## What a run looks like (set expectations)

- Prep: hours for a real corpus (hundreds of `claude` calls at `parallel: 8`).
- Trainer boot: ~10–15 minutes for the CVM, plus tens of GB of base-model
  download on a cold cache before training starts.
- Training: the reference run (27B base, QLoRA, ~1,300 examples, 2 epochs)
  took ~37 minutes on one spot H100 — a couple of dollars of compute.
- `collect` polls until done, then downloads `adapter.tar.gz`, `receipt.json`,
  `evidence.json` into `<workdir>/result/`.
- `verify` checks the receipt signature against the quote-bound in-enclave
  key, the attestation chain, and the adapter digest, then prints
  `baseline → final` exact-match on the filtered held-out eval.

Interpreting the score: chance is 25% (4-choice MCQs the filter already
proved unguessable from general knowledge). The reference run moved
30.4% → 39.1% — on questions like that, every point above baseline is
knowledge that exists only in the user's data. For a qualitative check, the
source tarball's `poc/gen_answers.py` + `poc/judge.py` run a blind pairwise
base-vs-tuned comparison on held-out open questions (needs a GPU box; the
reference run scored tuned 40 / base 3 / tie 6).

## Serving the trained model (`pl serve`)

The pipeline's output is a LoRA adapter + receipt in `<workdir>/result/`.
To turn it into a usable endpoint with the same custody guarantees:

```sh
pl job deploy --project-dir private-learning/profiles/serve-gpu   # serving CVM (TDX H100)
pl serve upload --url https://TRAINER_IP:8000 \
  --adapter <workdir>/result/adapter.tar.gz --served-name my-model \
  --expect-compose <compose_hash printed by deploy>    # seal + upload; mints API keys
pl serve release --expect-compose <same hash>          # gates verified → key released
pl serve status                                        # preparing → serving (~25 min first time)
pl serve chat "test question"                          # pinned one-shot chat
```

What happens in the enclave: the sealed adapter is decrypted only after the
attestation gates pass, merged into the public base model in-enclave, and
served by vLLM behind a custody supervisor. Only the adapter travels (the
base is pulled from HF inside); the merged weights never leave.

- **RA-TLS, not a CA:** the endpoint's self-signed cert is signed by the
  attested in-enclave key; `pl serve` clients verify the quote then pin the
  cert. Never bypass certificate errors manually — the CLI handles it.
- **Metered API keys:** `upload --keys N` mints N bearer keys and shows them
  ONCE (stored in `serve-owner.json` — protect that file). Per-key token
  usage is counted in-enclave; `pl serve usage` returns the counts signed by
  the enclave key — a billing record neither side can forge.
- **GUI chat:** `pl serve proxy --no-thinking --port 8080` runs a localhost
  bridge; point any OpenAI-compatible app (Chatbox, Jan, Open WebUI) at
  `http://localhost:8080/v1`, any api key value, model = the served name.
- First serve takes ~25 min (base download + in-enclave merge + vLLM load);
  the merged model is cached on the CVM volume for restarts. The CVM bills
  hourly — `pl job destroy --project-dir …/profiles/serve-gpu` when done.
- Thinking-mode bases (Qwen3.8) emit reasoning traces by default; the CLI's
  chat and the proxy's `--no-thinking` disable that. API callers can pass
  `"chat_template_kwargs": {"enable_thinking": false}`.
- 27B-class models need the H100 profile. Runtime multi-LoRA is NOT
  supported for hybrid-attention bases (Qwen3.8) — serving is always the
  merged model; that's why the merge happens in-enclave.

## Troubleshooting

- **Interrupted run** → just re-run `pl pipeline run`; state resumes. Bad
  stage output → `--redo <stage>`.
- **Few eval survivors** → more held-out documents (raise `holdout_every`
  density by adding data, not by weakening the filter).
- **Release refused** → that's the custody gate working, not an outage. The
  verdict names the failed gate: a compose-hash mismatch means the trainer
  isn't running the approved recipe (hosted) or your pinned hash (password);
  freshness failures mean stale evidence — re-run submit.
- **`claude` errors during prep** → per-chunk failures are logged and
  skipped; re-running the stage retries only the missing chunks.
- **Trainer unreachable during collect** → the CVM may still be booting or
  downloading weights; `collect` keeps polling. Spot VMs can be preempted:
  re-run from `--redo submit` if the trainer vanished.