OASIS-LLM: Rate Images with Vision-Language Models

OASIS-LLM replays the OASIS affective rating procedure using LLMs and compares model valence and arousal ratings to 822 human raters across 900 OASIS images.

OASIS-LLM is a research harness that takes the Open Affective Standardized Image Set (OASIS) — 900 color images rated by 822 MTurk workers on 7-point valence and arousal scales — and replays the same rating procedure against modern vision-language models. You get a reproducible pipeline for running LLM rating experiments and an interactive dashboard to compare model output against established human norms.

Quick Start

Install OASIS-LLM and launch your first pilot run in minutes.

Experiment Design

Understand the OASIS procedure, rating scales, and how trials are structured.

Configuration

Every RunConfig field: providers, models, sampling, prompts, and run identity.

Analysis

Compare LLM ratings to human norms using the dashboard’s statistical tools.

Cost & Latency

Real pilot numbers and full-set extrapolations to budget your runs.

Glossary

Definitions for valence, arousal, ICC, canonical hash, and more.

What OASIS-LLM does

OASIS-LLM lets you ask: how do contemporary LLMs rate the emotional content of images? You configure a run with a YAML file, point it at a model, and the harness submits each image to the model with the paper-verbatim valence or arousal instructions. Results are stored in a local DuckDB database. A Streamlit dashboard lets you browse runs, explore individual images, and run statistical comparisons against the human norms from Kurdi, Lozano, & Banaji (2017).

1. Install and configure

   Install with uv sync, copy .env.example to .env, and add your provider API key.

2. Launch the dashboard

   Run uv run oasis-llm dashboard to open the Streamlit UI at http://localhost:8501.

3. Define an experiment

   Create a YAML config selecting your model, image set, and sampling parameters.

4. Run and analyze

   Execute the run via CLI or dashboard, then open the Analysis page to compare against human norms.

Supported providers

OASIS-LLM routes all model calls through LiteLLM, giving you a single interface across providers:

• OpenRouter — access to hundreds of open and closed models with cost tracking
• OpenAI — GPT-4o and other vision models
• Anthropic — Claude vision models
• Google — Gemini vision models
• Ollama — local models running on your own hardware

The 900 OASIS images are licensed under CC BY-NC-SA 4.0 by the original authors and are not bundled with this tool. Download them from osf.io/6pnd7 and unpack into OASIS/images/.