Knowledge-Enhanced Neural Networks for Machine Reading Comprehension [Source Code and Additional Material]

Machine Reading Comprehension is a language understanding task where a systemis expected to read a given passage of text and typically answer questionsabout it. When humans assess the task of reading comprehension, in addition to thepresented text, they usually use the knowledge that they already know, such ascommonsense and world knowledge, or language skills that they previouslyacquired - understanding the events and arguments in a text (who did what towhom), their participants and the relation in discourse. In contrast, neural network approaches for machine reading comprehensionfocused on training end-to-end systems that rely only on annotatedtask-specific data.In this thesis, we explore approaches for tackling the reading comprehensionproblem, motivated by how a human would solve the task, using existingbackground and commonsense knowledge or knowledge from various linguistictasks.First, we develop a neural reading comprehension model that integratesexternal commonsense knowledge encoded as a key-value memory. Instead ofrelying only on document-to-question interaction or discrete features, ourmodel attends to relevant external knowledge and combines this knowledge withthe context representation before inferring the answer. This allows the modelto attract and imply knowledge from an external knowledge source that is notexplicitly stated in the text but is relevant for inferring the answer. Wedemonstrated that the proposed approach improves the performance of verystrong base models for cloze-style reading comprehension and open-bookquestion answering. By including knowledge explicitly, our model can also provide evidence aboutthe background knowledge used in the reasoning process.Further, we examined the impact of transferring linguistic knowledge fromlow-level linguistic tasks into a reading comprehension system using neuralrepresentations. Our experiments show that the knowledge transferred from theneural representations trained on these linguistic tasks can be adapted andcombined together to improve the reading comprehension task early in trainingand when trained with small portions of the data. Last, we propose to use structured linguistic annotations as a basis for aDiscourse-Aware Semantic Self-Attention encoder that we employ for readingcomprehension of narrative texts. We extract relations between discourseunits, events, and their arguments, as well as co-referring mentions, usingavailable annotation tools. The empirical evaluation shows that theinvestigated structures improve the overall performance (up to +3.4 Rouge-L),especially intra-sentential and cross-sentential discourse relations,sentence-internal semantic role relations, and long-distance coreferencerelations. We also show that dedicating self-attention heads tointra-sentential relations and relations connecting neighboring sentences isbeneficial for finding answers to questions in longer contexts. These findingsencourage the use of discourse-semantic annotations to enhance thegeneralization capacity of self-attention models for machine readingcomprehension.