The Structure of Disagreement in Emotion Annotation

概要

Calculating inter-annotator agreement is an established practice in NLP, and disagreement-aware work increasingly retains the distribution of human judgments rather than reducing annotations to a single gold label. Keeping the distribution, however, does not explain why annotators disagree. This paper examines the types of disagreement that occur in emotion annotation, starting from multilabel annotations of 1,677 sentence-level units from Ernest Hemingway's The Old Man and the Sea and testing the patterns they reveal against individual-level annotations from BRIGHTER and GoEmotions. Across the three datasets, we find three main recurring disagreement structures. Emotion categories overlap systematically; annotators differ both in how broadly they apply labels and in how they treat particular category boundaries, while sparse participation and explicit abstention can look like disagreement. Where intensity is rated, disagreement over whether an emotion is present can be separated from disagreement over its intensity once recognized. Similar aggregate disagreement can therefore arise from qualitatively different annotation processes. Hard aggregation removes most of this structure, while annotator vote-share representations keep marginal label frequencies but not the co-occurrence, annotator-specific, or procedural patterns that produced them. We argue that disagreement should be understood in terms of its underlying patterns before it is summarized, aggregated, or used as a modeling target, and propose a vocabulary for disagreement structure.

参考リンク

• Paper ↗

リンク先のライセンス・再現性・データ条件をご確認ください。自動分類や実装の目安は保証ではありません。

実装を始めるプロンプト

参考リンクを確認したうえで、次の内容をコピーして利用できます。

# 論文・記事
The Structure of Disagreement in Emotion Annotation

# 概要
Calculating inter-annotator agreement is an established practice in NLP, and disagreement-aware work increasingly retains the distribution of human judgments rather than reducing annotations to a…

# 原典
https://arxiv.org/abs/2611.01539

上記の手法を実装してください。実行前に依存関係とライセンスを確認してください。