arXiv preprint · 2026

Can LLMs Imagine Moral Alternatives Beyond Binary Dilemmas?

Jongchan Choi1, Nari Yang1, Sung Soo Park2, Jaemin Cho1, Han Seoyoung1, Haerin Shin1, Jun-Hyung Park3

1Korea University    2XenoStep AI    3Hankuk University of Foreign Studies

Paper Code Dataset

Moral reasoning involves more than choosing between the options we are given. Sometimes the better response is to create a new one.

MoralAltDataset overview with original options A and B, compromise C, and reframed alternative D
MoralAltDataset expands each binary dilemma with a compromise alternative and a reframed alternative.

Abstract

As large language models are increasingly deployed as moral advisors and agents, they need to address dilemmas between competing values. Existing work, however, largely evaluates models by asking them to choose among pre-specified options. This overlooks a central part of human moral cognition: the capacity to imagine alternatives that move beyond a forced binary.

We introduce MoralAltDataset, a dataset of 307 moral dilemmas spanning narrative Advisor dilemmas and AI-facing Agent dilemmas. Each item includes a compromise and a reframed alternative. Across 15 LLMs, compromise alternatives are frequently preferred over either original option. We further find that model-generated alternatives are often preferred to human-authored ones, while revealing a trade-off between structural quality and practical feasibility.

307 Moral dilemmas
15 Language models
4,920 Generated alternatives
4 Options per dilemma

MoralAltDataset

Two kinds of dilemmas

The benchmark combines context-rich human narratives with high-stakes decisions that AI systems may encounter.

Advisor dilemmas 156

Context-rich human conflict

Narrative dilemmas constructed from movie-plot synopses retain motivations, relationships, and downstream consequences.

Agent dilemmas 151

High-stakes AI decisions

AI-facing scenarios adapted from AIRiskDilemmas represent realistic decisions across domains such as healthcare.

C

Compromise alternative

Preserves at least one core moral aim from A and B through a concrete decision rule or trade-off.

D

Reframed alternative

Changes the conflict frame by introducing a new principle, stakeholder, or temporal or institutional scope.

Method

How we tested moral imagination

  1. 1

    Build the dilemma

    Present a moral-conflict scenario with original options A and B.

  2. 2

    Add alternatives

    Write or generate a compromise C and a reframed alternative D.

  3. 3

    Compare judgments

    Measure how humans and 15 LLMs choose across the four options.

  4. 4

    Evaluate generation

    Assess alternatives for feasibility, structure, and ethical quality.

Main findings

What changes when alternatives are available

Introducing C and D changes both which options are selected and which moral values become salient.

01

Compromise is often preferred over either binary option.

Across both Advisor and Agent settings, humans and most models frequently select the compromise alternative once it is available.

02

Human–LLM agreement is stronger on alternatives.

Average group-level agreement rises when humans choose C or D, relative to cases where they remain with A or B.

03

Model-generated alternatives often outperform human ones.

In pairwise preference evaluations, all three evaluated model sources exceed the human baseline in overall preference.

Pairwise preference: mean win rate (%)
Source Compromise overall Reframed overall
Human 29.3 23.7
GPT-5 58.0 67.7
Claude Sonnet 4.5 57.2 55.2
Qwen 3.5 122B 57.2 53.5
Heatmap of value shifts from binary choices to four-option choices
Value shifts after adding compromise and reframed alternatives, reported as percentage-point change from the A/B baseline.

What this means

LLMs can recognize and produce options beyond a binary choice, but a well-structured alternative is not always practical. Evaluations should therefore test both the decisions models make and their ability to improve the available choices.

Limitations

MoralAltDataset is a controlled benchmark rather than a comprehensive account of moral decision-making. It does not cover the full diversity of cultures, languages, domains, and institutions.

The experiments use fixed option sets and standardized zero-shot prompting. The ethical rubrics are transparent and reproducible proxies, not definitive philosophical measurements.

Resources

Paper, code, and data

Paper Read the full preprint → GitHub Browse code and released files → Hugging Face Explore MoralAltDataset →

BibTeX

@misc{choi2026moralalt,
  title  = {Can LLMs Imagine Moral Alternatives Beyond Binary Dilemmas?},
  author = {Jongchan Choi and Nari Yang and Sung Soo Park and Jaemin Cho and Han Seoyoung and Haerin Shin and Jun-Hyung Park},
  year   = {2026},
  eprint={2606.31213},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2606.31213},
}