Benchmarking Machine Reading Comprehension: A Psychological Perspective

Saku Sugawara, Pontus Stenetorp, Akiko Aizawa

Information Retrieval, Search and Question Answering Long paper Paper

Gather-3A: Apr 23, Gather-3A: Apr 23 (13:00-15:00 UTC) [Join Gather Meeting]

You can open the pre-recorded video in separate windows.

Abstract: Machine reading comprehension (MRC) has received considerable attention as a benchmark for natural language understanding. However, the conventional task design of MRC lacks explainability beyond the model interpretation, i.e., reading comprehension by a model cannot be explained in human terms. To this end, this position paper provides a theoretical basis for the design of MRC datasets based on psychology as well as psychometrics, and summarizes it in terms of the prerequisites for benchmarking MRC. We conclude that future datasets should (i) evaluate the capability of the model for constructing a coherent and grounded representation to understand context-dependent situations and (ii) ensure substantive validity by shortcut-proof questions and explanation as a part of the task design.
NOTE: Video may display a random order of authors. Correct author list is at the top of this page.

Connected Papers in EACL2021

Similar Papers

Discrete Reasoning Templates for Natural Language Understanding
Hadeel Al-Negheimish, Pranava Madhyastha, Alessandra Russo,
Language Modelling as a Multi-Task Problem
Lucas Weber, Jaap Jumelet, Elia Bruni, Dieuwke Hupkes,