arXiv preprint · 2026

Can LLMs Imagine Moral Alternatives Beyond Binary Dilemmas?

Jongchan Choi1, Nari Yang1, Sung Soo Park2, Jaemin Cho1, Han Seoyoung1, Haerin Shin1, Jun-Hyung Park3

1Korea University    2XenoStep AI    3Hankuk University of Foreign Studies

▤ Paper </> Code ◫ Dataset

Moral reasoning is not only choosing well between options that are given. It is also the ability to imagine better options.

MoralAltDataset overview with original options A and B, compromise C, and reframed alternative D
MoralAltDataset expands each binary dilemma with a compromise alternative and a reframed alternative.

Abstract

As large language models are increasingly deployed as moral advisors and agents, they need to address dilemmas between competing values. Existing work, however, largely evaluates models by asking them to choose among pre-specified options. This overlooks a central part of human moral cognition: the capacity to imagine alternatives that move beyond a forced binary.

We introduce MoralAltDataset, a dataset of 307 moral dilemmas spanning narrative Advisor dilemmas and AI-facing Agent dilemmas. Each item includes a compromise and a reframed alternative. Across 15 LLMs, compromise alternatives are frequently preferred over either original option. We further find that model-generated alternatives are often preferred to human-authored ones, while revealing a trade-off between structural quality and practical feasibility.

307 Moral dilemmas
15 Language models
4,920 Generated alternatives
4 Options per dilemma

MoralAltDataset

Two settings for moral imagination

The benchmark combines context-rich human narratives with high-stakes decisions that AI systems may encounter.

Advisor dilemmas 156

Context-rich human conflict

Narrative dilemmas constructed from movie-plot synopses retain motivations, relationships, and downstream consequences.

Agent dilemmas 151

High-stakes AI decisions

AI-facing scenarios adapted from AIRiskDilemmas represent realistic decisions across domains such as healthcare.

C

Compromise alternative

Preserves at least one core moral aim from A and B through a concrete decision rule or trade-off.

D

Reframed alternative

Changes the conflict frame by introducing a new principle, stakeholder, or temporal or institutional scope.

Method

From binary choice to moral imagination

  1. 1

    Build the dilemma

    Present a moral-conflict scenario with original options A and B.

  2. 2

    Add alternatives

    Write or generate a compromise C and a reframed alternative D.

  3. 3

    Compare judgments

    Measure how humans and 15 LLMs choose across the four options.

  4. 4

    Evaluate generation

    Assess alternatives for feasibility, structure, and ethical quality.

Main findings

Alternatives reshape moral judgment

Introducing C and D changes both which options are selected and which moral values become salient.

01

Compromise is often preferred over either binary option.

Across both Advisor and Agent settings, humans and most models frequently select the compromise alternative once it is available.

02

Human–LLM agreement is stronger on alternatives.

Average group-level agreement rises when humans choose C or D, relative to cases where they remain with A or B.

03

Model-generated alternatives often outperform human ones.

In pairwise preference evaluations, all three evaluated model sources exceed the human baseline in overall preference.

Pairwise preference: mean win rate (%)
Source Compromise overall Reframed overall
Human 29.3 23.7
GPT-5 58.0 67.7
Claude Sonnet 4.5 57.2 55.2
Qwen 3.5 122B 57.2 53.5
Heatmap of value shifts from binary choices to four-option choices
Value shifts after adding compromise and reframed alternatives, reported as percentage-point change from the A/B baseline.

What this means

Current LLMs show a meaningful but uneven capacity for moral imagination. They can recognize and generate alternatives beyond a binary frame, but strong balancing or reframing does not always imply practical feasibility. Alignment systems should therefore evaluate not only which option a model selects, but also whether it can improve the choice set itself.

Limitations

MoralAltDataset is a controlled benchmark rather than a comprehensive account of moral decision-making. It does not cover the full diversity of cultures, languages, domains, and institutions.

The experiments use fixed option sets and standardized zero-shot prompting. The ethical rubrics are transparent and reproducible proxies, not definitive philosophical measurements.

Resources

Paper, code, and data

Paper Read the full preprint → GitHub Browse code and released files → Hugging Face Explore MoralAltDataset →

BibTeX

@article{choi2026moralalternatives,
  title   = {Can LLMs Imagine Moral Alternatives Beyond Binary Dilemmas?},
  author  = {Choi, Jongchan and Yang, Nari and Park, Sung Soo and
             Cho, Jaemin and Seoyoung, Han and Shin, Haerin and
             Park, Jun-Hyung},
  journal = {arXiv preprint arXiv:2606.31213},
  year    = {2026}
}