Skip to main content
OASIS-LLM is a research harness that takes the Open Affective Standardized Image Set (OASIS) — 900 color images rated by 822 MTurk workers on 7-point valence and arousal scales — and replays the same rating procedure against modern vision-language models. You get a reproducible pipeline for running LLM rating experiments and an interactive dashboard to compare model output against established human norms.

Quick Start

Install OASIS-LLM and launch your first pilot run in minutes.

Experiment Design

Understand the OASIS procedure, rating scales, and how trials are structured.

Configuration

Every RunConfig field: providers, models, sampling, prompts, and run identity.

Analysis

Compare LLM ratings to human norms using the dashboard’s statistical tools.

Cost & Latency

Real pilot numbers and full-set extrapolations to budget your runs.

Glossary

Definitions for valence, arousal, ICC, canonical hash, and more.

​
What OASIS-LLM does

OASIS-LLM lets you ask: how do contemporary LLMs rate the emotional content of images? You configure a run with a YAML file, point it at a model, and the harness submits each image to the model with the paper-verbatim valence or arousal instructions. Results are stored in a local DuckDB database. A Streamlit dashboard lets you browse runs, explore individual images, and run statistical comparisons against the human norms from Kurdi, Lozano, & Banaji (2017).
1

Install and configure

Install with uv sync, copy .env.example to .env, and add your provider API key.
2

Launch the dashboard

Run uv run oasis-llm dashboard to open the Streamlit UI at http://localhost:8501.
3

Define an experiment

Create a YAML config selecting your model, image set, and sampling parameters.
4

Run and analyze

Execute the run via CLI or dashboard, then open the Analysis page to compare against human norms.

​
Supported providers

OASIS-LLM routes all model calls through LiteLLM, giving you a single interface across providers:
  • OpenRouter — access to hundreds of open and closed models with cost tracking
  • OpenAI — GPT-4o and other vision models
  • Anthropic — Claude vision models
  • Google — Gemini vision models
  • Ollama — local models running on your own hardware
The 900 OASIS images are licensed under CC BY-NC-SA 4.0 by the original authors and are not bundled with this tool. Download them from osf.io/6pnd7 and unpack into OASIS/images/.