T2: Aggregating and Learning from Multiple Annotators

Silviu Paun and Edwin Simpson

Live Session 1: Apr 19, Live Session 1: Apr 19 (08:00-09:00 UTC) [Join Zoom Meeting]
Live Session 2: Apr 19, Live Session 2: Apr 19 (14:00-15:00 UTC) [Join Zoom Meeting]
Abstract: The success of NLP research is founded on high-quality annotated datasets, which are usually obtained from multiple expert annotators or crowd workers. The standard practice to training machine learning models is to first adjudicate the disagreements and then perform the training. To this end, there has been a lot of work on aggregating annotations, particularly for classification tasks. However, many other tasks, particularly in NLP, have unique characteristics not considered by standard models of annotation, e.g., label interdependencies in sequence labelling tasks, unrestricted labels for anaphoric annotation, or preference labels for ranking texts. In recent years, researchers have picked up on this and are covering the gap. A first objective of this tutorial is to connect NLP researchers with state-of-the-art aggregation models for a diverse set of canonical language annotation tasks. There is also a growing body of recent work arguing that following the convention and training with adjudicated labels ignores any uncertainty the labelers had in their classifications, which results in models with poorer generalisation capabilities. Therefore, a second objective of this tutorial is to teach NLP workers how they can augment their (deep) neural models to learn from data with multiple interpretations.

Time Event Hosts
Apr 19, (08:00-09:00 UTC) Part 1 Silviu Paun and Edwin Simpson
Apr 19, (14:00-15:00 UTC) Part 2
Information about the virtual format of this tutorial: This tutorial has a prerecorded talk on this page (see below) that you can watch anytime during the conference. It also has two live sessions that will be conducted on Zoom and will be livestreamed on this page. Additionally, it has a chat window that you can use to have discussions with the tutorial teachers and other attendees anytime during the conference.