> ## Documentation Index > Fetch the complete documentation index at: https://dcpma.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Compare LLM Ratings Against Human Norms > Use the OASIS-LLM Analysis dashboard page to compare model valence and arousal ratings against the Kurdi et al. 2017 human norms with eight statistical views. The **Analysis** page is where you compare completed run ratings to the OASIS human norms from Kurdi et al. (2017). You select a scope — a set of runs and models — and the page computes eight statistical views live from your local DuckDB store. Every tab shares the same filtered data; only the lens changes. ## Access modes The Analysis page operates in two modes. | Mode | How you scope the data | | ----------- | ------------------------------------------------------------------------------------------------------------------------------- | | **Ad-hoc** | Use the sidebar filters to pick one or more image sets and models directly. Best for quick, exploratory comparisons. | | **Curated** | Select a saved Analysis bundle. The bundle pins a specific list of `run_id` values so your scope is reproducible and shareable. | Both modes feed into the same analytics body below the filter bar. Switch between them using the mode selector at the top of the sidebar. An Analysis bundle requires that all pinned runs were executed against the same `dataset_id`. If you try to add a run from a different image set, OASIS-LLM will reject it with a validation error. ## Pipeline overview The following diagram shows the full path from raw images to the comparison statistics you see on screen. ```mermaid theme={null} flowchart LR A["OASIS images
(900)"] --> B["Image set
(N images, balanced
across categories)"] B --> C["Trial generator
(K samples × 2 dimensions)"] C --> D["LLM rater
(provider / model / temperature)"] D --> E["Trials table
(rating, reasoning, latency)"] H["OASIS human norms
(Valence_mean, Arousal_mean)"] --> F E --> F["Per-image aggregation
(LLM trials → image mean)"] F --> G["LLM vs Human comparison
paired t · Cohen's d · CCC ·
KS/Wasserstein · ICC · ANOVA"] G --> I["Explorer tabs / Download CSV"] ``` The aggregation step collapses all trials for a given `(run_id, image_id, dimension)` into a single mean before any comparison is made. Human norms are the published `Valence_mean` and `Arousal_mean` columns from `OASIS.csv`. ## Filter controls Use the filter bar to narrow the data before any tab renders. | Control | What it does | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------- | | **Model** | Multi-select. Choose one or more models. "All" pools every model in scope into a single LLM mean. | | **Category** | Filter to Animal, Scene, Person, or Object. Categories are colour-coded throughout the UI. | | **Dimension** | Toggle between valence and arousal, or view both. | | **Image** | Substring search on `image_id`. Useful for drilling into a specific stimulus. | | **Aggregation scope** | Choose one of: Pooled all-LLMs · By model · By category · Model × Category. Controls how the comparison statistics are grouped. | ## The eight analysis tabs Shows N, mean, SD, median, and range for humans, each model individually, and the pooled-LLM mean. Use this tab first to check whether your LLM means sit inside the human range before interpreting any inferential statistics. Runs a paired t-test of LLM per-image means against human per-image means for each run × dimension combination. Reports *t*, degrees of freedom, *p*, Cohen's *d* (paired), and Pearson *r*. Optional bootstrap percentile CIs (2 000 resamples) are available when you enable the bootstrap toggle. Fits OLS `LLM = a + b · Human` for each scope. A perfectly calibrated model produces `b = 1, a = 0`. A slope greater than 1 indicates scale stretch; a positive intercept indicates an upward shift. Reports slope, intercept, R², and residual SD. Plots one point per image with LLM mean on one axis and human mean on the other. Points are coloured by model or by OASIS category. The identity line is drawn for reference. Overlays the histogram of raw LLM trial ratings against the distribution of human image means. Reports the two-sample Kolmogorov–Smirnov statistic and *p*-value, plus Wasserstein distance (Earth mover's distance) between the two distributions. Lists the top-K images with the largest absolute difference between LLM mean and human mean. Scope defaults to pooled (averaged across runs), but you can switch to per-run mode. Useful for identifying stimuli where the model's affect representation diverges most from human raters. Computes pairwise Pearson *r* between every pair of selected runs (models), plus per-image SD across runs. Reports mean and median pairwise *r*, mean per-image SD, ICC(2,1) (absolute agreement), and ICC(3,1) (consistency). Use this tab when you want to know whether different models agree with each other, not just with humans. Two-way ANOVA on the per-image absolute error `|LLM − human|` with Category and Model as factors, including their interaction term. Uses Type II sums of squares (statsmodels). Reports *F*, *p*, and η² for each factor. Use this tab to check whether model bias is uniform across image categories or concentrated in specific ones. ## Statistics reference All metrics are computed live from the DuckDB store. No pre-aggregated caches are used. | Statistic | Formula / definition | | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Paired t** | Per-image paired *t* on N images per (model × dimension). Pooled t collapses across models first. | | **Cohen's *d* (paired)** | `mean(diff) / SD(diff)` where `diff = LLM_image_mean − human_image_mean`. | | **Lin's CCC** | `2 · cov(x, y) / (var(x) + var(y) + (mean_x − mean_y)²)`. Combines precision (correlation) and accuracy (mean agreement). | | **KS / Wasserstein** | Two-sample KS test and Wasserstein distance on raw LLM trial ratings vs human image means. | | **ICC(2,1)** | Two-way random effects, single rater, absolute agreement (Shrout & Fleiss, 1979). Use when treating the runs as a sample from a population of possible models. | | **ICC(3,1)** | Two-way mixed effects, single rater, consistency (Shrout & Fleiss, 1979). Use when the specific models in scope are the only ones of interest. | If you are comparing Lin's CCC values across dimensions, note that CCC is sensitive to both correlation and mean-level agreement. A model can have a high Pearson *r* but a low CCC if it applies a systematic scale shift. ## How to access Start the dashboard from your terminal: ```bash theme={null} oasis-llm dashboard ``` Then select **Analysis** in the sidebar. All statistics are computed on demand from your local database. A read-only deployment is available via the link in the navbar. The hosted version supports the same eight tabs but cannot write new analyses to the database.