> ## Documentation Index
> Fetch the complete documentation index at: https://dcpma.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Run Lifecycle: From YAML Config to Completed Trials

> How OASIS-LLM processes a run from a YAML config through trial enqueueing, async worker execution, and final result storage in DuckDB.

When you run `oasis-llm run configs/runs/my-run.yaml`, a precise sequence of steps unfolds: the CLI reads your config, creates or resumes a run record in DuckDB, enqueues every `(image, dimension, sample)` combination that hasn't been attempted yet, then drains those trials through an async worker pool that calls your LLM provider and writes results back to the database. Understanding this lifecycle helps you reason about resumption, cost, and failure recovery.

## End-to-end sequence

The diagram below traces a full run from your terminal command through to the final status update.

```mermaid theme={null}
sequenceDiagram
    autonumber
    participant U as User
    participant CLI as cli.run
    participant DB as DuckDB
    participant Q as enqueue_trials
    participant W as asyncio worker pool
    participant LLM as LiteLLM provider

    U->>CLI: oasis-llm run my-run.yaml
    CLI->>DB: connect() — creates schema if absent
    CLI->>DB: upsert_run(cfg)
    DB-->>CLI: (run_id, created?)
    CLI->>Q: enqueue_trials(cfg)
    Q->>DB: anti-join INSERT into trials
    Q-->>CLI: rows_inserted
    CLI->>W: asyncio.gather(N workers)

    loop for each pending trial
        W->>DB: _claim_one (under asyncio.Lock)
        DB-->>W: trial row (status='running')
        W->>LLM: _call_model (image data URL + prompts)
        LLM-->>W: rating + reasoning + usage
        W->>W: _parse_rating (JSON or regex fallback)
        W->>DB: _record_result (status='done' | 'failed')
    end

    W-->>CLI: pool drained
    CLI->>DB: UPDATE runs SET status='done'
```

### Key invariants

Three guarantees hold throughout every run:

* **Atomic claim.** `_claim_one` executes under a single `asyncio.Lock`, so no two workers in the same process can grab the same trial. The runner is single-process by design — there is no cross-process locking mechanism.
* **Stale-claim recovery.** Before claiming a new trial, `_claim_one` resets any trial whose `status='running'` and `claimed_at < now() - 10 min` back to `pending`. If a worker is killed mid-flight, the next invocation of the runner automatically reclaims those trials after at most 10 minutes.
* **Retry budget.** A trial is eligible to be claimed again while `status='pending'` OR (`status='failed'` AND `attempts < 3`). Every call to `_record_result` increments `attempts` regardless of whether the trial succeeded or failed.

<Note>
  The ordering inside `_claim_one` is `ORDER BY attempts, sample_idx, image_id, dimension LIMIT 1`, so fresh trials (attempts = 0) drain before retries. Within an attempt tier, lower `sample_idx` values go first.
</Note>

## Trial lifecycle

Every trial moves through a defined set of states. The diagram below shows all valid transitions, including the stale-recovery path and the two terminal states.

```mermaid theme={null}
stateDiagram-v2
    [*] --> pending: enqueue_trials inserts row
    pending --> running: _claim_one (under Lock)
    running --> done: rating parsed AND no error
    running --> failed: error OR rating unparseable
    running --> pending: stale (claimed_at > 10 min ago)
    failed --> running: attempts < 3 AND _claim_one
    done --> [*]
    failed --> [*]: attempts >= 3
```

A trial reaches a **terminal state** when it is either `done`, or `failed` with `attempts >= 3`. There is no automatic promotion out of a terminal state — to re-run a finished trial you must start a new run with a different name.

## Trial record

Every row in the `trials` table is identified by the composite key `(run_id, image_id, dimension, sample_idx)`. The table below documents every column you may encounter when querying results directly.

| Column          | Type      | Meaning                                                                                      |
| --------------- | --------- | -------------------------------------------------------------------------------------------- |
| `status`        | TEXT      | `pending` / `running` / `done` / `failed`                                                    |
| `rating`        | INTEGER   | Parsed 1–7 rating; NULL on failure                                                           |
| `raw_response`  | TEXT      | Verbatim model output                                                                        |
| `reasoning`     | TEXT      | Parsed JSON `reasoning` field; NULL if absent or not captured                                |
| `prompt_hash`   | TEXT      | `sha256(model + system + user)[:16]` — lets you detect prompt-version changes across runs    |
| `latency_ms`    | INTEGER   | Wall-clock time from the start of `_call_model`                                              |
| `input_tokens`  | INTEGER   | Token count from the provider usage block                                                    |
| `output_tokens` | INTEGER   | Token count from the provider usage block                                                    |
| `cost_usd`      | DOUBLE    | LiteLLM cost estimate; falls back to OpenRouter native `usage.cost`                          |
| `error`         | TEXT      | Error message; NULL on success                                                               |
| `finish_reason` | TEXT      | Provider's finish reason (e.g. `stop`, `length`)                                             |
| `response_id`   | TEXT      | Provider's response ID — useful for correlating with provider-side trace logs and dashboards |
| `attempts`      | INTEGER   | Incremented by every call to `_record_result`, whether success or failure                    |
| `claimed_at`    | TIMESTAMP | Set when a worker claims the trial; used for stale-recovery cutoff                           |
| `completed_at`  | TIMESTAMP | Set when `_record_result` writes the outcome; useful for latency analysis                    |

<Tip>
  Query `cost_usd` and `latency_ms` across a finished run to estimate per-model pricing and throughput before scaling up `samples_per_image`.
</Tip>

## Resumption semantics

`upsert_run` stores a **canonical config hash** alongside every run record and checks it on every subsequent invocation of the same `run_id`. If the hash doesn't match, the runner raises an error rather than silently mixing results from two different experiment configurations:

```python theme={null}
if existing[0] != cfg_hash:
    raise RuntimeError(
        f"Run '{run_id}' exists with different config hash "
        f"(stored={existing[0]}, new={cfg_hash}). Use a new --name or --new-run."
    )
```

The hash is computed as `sha256(model_dump(exclude={name, max_concurrency, request_timeout_s, max_retries, samples_per_image}))[:16]`. Because those five fields are excluded, you can safely change them between invocations of the same run without invalidating earlier results.

**What you CAN change between resumes:**

* `max_concurrency` — tune parallelism up or down freely.
* `request_timeout_s` — extend the per-call timeout if you're seeing timeouts on large images.
* `samples_per_image` — increase from 5 to 20 and only the new samples will be enqueued; already-completed samples are skipped by the anti-join in `enqueue_trials`.

**What you CANNOT silently change:**

* `model` — changing the model changes what the experiment measures.
* Prompt overrides (`system_prompt_override`, `format_hint_suffix`) — these affect what the model sees.
* `dimensions` — adding or removing valence/arousal changes the scope of the experiment.
* `image_set` — changing the image set changes the population being rated.

<Warning>
  Attempting to resume a run after changing any of the above fields will raise a `RuntimeError`. Create a new run with a different `name` instead.
</Warning>

## Exporting results

Once a run is complete, you have three export options from the CLI.

### Export raw trials

Write all completed trials for a run to a CSV file:

```bash theme={null}
oasis-llm export <run_id> outputs/<run_id>.csv
```

The output contains every row from the `trials` table where `status='done'`, including `rating`, `reasoning`, `cost_usd`, `latency_ms`, `input_tokens`, and `output_tokens`.

### Generate paper-style plots

Produce a summary PDF and plots in the style of a research paper:

```bash theme={null}
oasis-llm paper-plots <run_id>
oasis-llm paper-plots <run_id> --out-dir outputs/paper_plots
```

The output directory (default `outputs/paper_plots/<run_id>`) receives:

* Valence and arousal distribution plots
* LLM vs human scatter plots
* A summary JSON with image count, samples per image, and human-overlap correlations (if human norms are available)

### Generate participant-style dataset

Reconstruct a wide participant-style CSV that mirrors the original OASIS data format, where each row represents one "pseudo-participant" that rated a fixed number of images:

```bash theme={null}
oasis-llm participant-dataset <run_id>
oasis-llm participant-dataset <run_id> --out-dir outputs/participant_dataset --images-per-participant 20
```

| Flag                       | Default                                | Description                                               |
| -------------------------- | -------------------------------------- | --------------------------------------------------------- |
| `--out-dir`                | `outputs/participant_dataset/<run_id>` | Directory to write the CSV and per-image plots.           |
| `--images-per-participant` | `20`                                   | Number of images assigned to each pseudo-participant row. |

<Note>
  Rows in the participant-style dataset are reconstructed from `sample_idx` values, not real participant sessions. They are structurally compatible with analysis scripts written for the original OASIS format but do not represent independent participants.
</Note>

## Dataset management

Datasets are named, versioned subsets of the 900 OASIS images. They let you define a reusable stimulus pool and attach it to multiple experiment configs without re-specifying the image list each time. Datasets progress through a lifecycle: `draft` → `approved` → `archived`.

```bash theme={null}
oasis-llm dataset list                        # list all datasets with status and image counts
oasis-llm dataset generate "my-set" --n 30   # create a 30-image stratified draft
oasis-llm dataset show <dataset_id>           # inspect images and metadata
oasis-llm dataset approve <dataset_id>        # lock as immutable (approved)
oasis-llm dataset archive <dataset_id>        # mark as archived (no longer active)
```

### Editing a draft dataset

Before approving, you can refine the image list:

```bash theme={null}
oasis-llm dataset exclude <dataset_id> <image_id> --note "reason"  # exclude an image
oasis-llm dataset include <dataset_id> <image_id>                   # re-include excluded image
oasis-llm dataset add <dataset_id> <image_id>                       # add a new image
oasis-llm dataset duplicate <dataset_id> "new-name"                 # clone to a new draft
oasis-llm dataset delete <dataset_id>                               # delete (built-ins protected)
```

The `generate` command supports three sampling strategies:

| Strategy               | Behaviour                                                                                |
| ---------------------- | ---------------------------------------------------------------------------------------- |
| `stratified` (default) | Proportionally allocates images across OASIS categories (Animal, Scene, Person, Object). |
| `uniform`              | Draws images uniformly at random from the full pool.                                     |
| `all`                  | Selects all 900 images (equivalent to `full_900`).                                       |

```bash theme={null}
oasis-llm dataset generate "arousal-focus" --n 50 --strategy uniform --seed 99
```

<Note>
  Once a dataset is approved, you cannot add or exclude images. Duplicate the dataset to a new draft if you need to modify it.
</Note>

## Experiment management

An experiment groups multiple run configs against a shared dataset, letting you compare several models or parameter combinations in one organised unit. Each config in the experiment gets its own `run_id` and runs sequentially.

```bash theme={null}
oasis-llm experiment list                      # list all experiments with status
oasis-llm experiment create experiment.yaml    # create from a YAML definition
oasis-llm experiment show <experiment_id>      # per-config progress and cost
oasis-llm experiment run <experiment_id>       # execute all configs sequentially
oasis-llm experiment delete <experiment_id>    # delete experiment and all its trials
```

### Experiment YAML format

```yaml theme={null}
name: compare-models-pilot30
dataset: my-dataset-id
description: "Compare Gemma 4 vs Qwen3 on pilot_30"
configs:
  - name: gemma4-pilot30
    provider: openrouter
    model: google/gemma-4-31b-it
    modality: vision
    dimensions: [valence, arousal]
    samples_per_image: 5
    max_concurrency: 4
    capture_reasoning: true

  - name: qwen3-pilot30
    provider: openrouter
    model: qwen/qwen3-vl-7b
    modality: vision
    dimensions: [valence, arousal]
    samples_per_image: 5
    max_concurrency: 4
    capture_reasoning: true
```

`oasis-llm experiment run` executes each config sequentially. If one config is already complete (all trials `done`), it is skipped automatically. You can re-run the command after a failure and it resumes from where it left off, because each underlying run is idempotent.

<Tip>
  Use `oasis-llm experiment show <id>` to check per-config progress and cost mid-run. The table shows `done/total` and cumulative `cost_usd` for each config.
</Tip>