This is the original paper on collective annotation in which we present the fundamental idea of harnessing ideas from social choice theory to aggregate noisy data collected from individual annotators by means of crowdsourcing. The paper proposes novel aggregation rules, particularly bias-correcting rules and greedy consensus rules, and presents experimental results for crowdsourced data on textual entailment.

This paper introduces a powerful formal framework for modelling aggregation rules for collective annotation. It presents theoretical results regarding the axiomatic characterisation of such rules, with a special focus on the family of bias-correcting rules.

This paper presents experimental results for collective annotation in three different domains: textual entailment, preposition sense disambiguation, and question dialogue act classification. For the latter two domains, we present new crowdsourced datasets of linguistic judgments collected specifically for this paper. The paper also introduces one further aggregation rule, which is based on a simple probabilistic generative model tracking the accuracy of annotators.