key: cord-0508494-dod49kpp
authors: Jaradeh, Mohamad Yaser; Singh, Kuldeep; Stocker, Markus; Both, Andreas; Auer, Soren
title: Better Call the Plumber: Orchestrating Dynamic Information Extraction Pipelines
date: 2021-02-22
journal: nan
DOI: nan
sha: 066223db90f4b5229820cc374c1b177f091765e9
doc_id: 508494
cord_uid: dod49kpp

In the last decade, a large number of Knowledge Graph (KG) information extraction approaches were proposed. Albeit effective, these efforts are disjoint, and their collective strengths and weaknesses in effective KG information extraction (IE) have not been studied in the literature. We propose Plumber, the first framework that brings together the research community's disjoint IE efforts. The Plumber architecture comprises 33 reusable components for various KG information extraction subtasks, such as coreference resolution, entity linking, and relation extraction. Using these components,Plumber dynamically generates suitable information extraction pipelines and offers overall 264 distinct pipelines.We study the optimization problem of choosing suitable pipelines based on input sentences. To do so, we train a transformer-based classification model that extracts contextual embeddings from the input and finds an appropriate pipeline. We study the efficacy of Plumber for extracting the KG triples using standard datasets over two KGs: DBpedia, and Open Research Knowledge Graph (ORKG). Our results demonstrate the effectiveness of Plumber in dynamically generating KG information extraction pipelines,outperforming all baselines agnostics of the underlying KG. Furthermore,we provide an analysis of collective failure cases, study the similarities and synergies among integrated components, and discuss their limitations.

In last one decade, publicly available KGs (DBpedia [2] and Wikidata [42] ) have become rich sources of structured content used in various applications, including Question Answering (QA), fact checking, and dialog systems [39, 4] . The research community developed numerous approaches to extract triple statements [44] , keywords/topics [9] , tables [45, 23, 22] , or entities [35, 36] from unstructured text to complement KGs. Despite extensive research, public KGs are not exhaustive and require continuous effort to align newly emerging unstructured information to the concepts of the KGs.

Research Problem: This work was motivated by an observation of recent approaches [35, 45, 15] that automatically align unstructured text to structured data on the Web. Such approaches are not viable in practice for extracting and structuring information because they only address very specific subtasks of the overall KG information extraction problem. If we consider the exemplary sentence Rembrandt painted The Storm on the Sea of Galilee. It was painted in 1633. (cf. Figure 1 ). To extract statements aligned with the DBpedia KG from the given sentences, a system must first recognize the entities and relation surface forms in the first sentence. The second sentence requires an additional step of the coreference resolution, where It must be mapped to the correct entity surface form (namely, The Storm on the Sea of Galilee). The last step requires the mapping of entity and relation surface forms to the respective DBpedia entities and predicates. There has been extensive research in aligning concepts in unstructured text to KG, including entity linking [15, 18] , relation linking [36, 38, 4] , and triple classification [14] . However, these efforts are disjoint, and little has been done to align unstructured text to the complete KG triples (i.e., represented as subject, predicate, object) [25] . Furthermore, many entity and relation linking tools have been reused in pipelines of QA systems [39, 26] . The literature suggests that once different approaches put forward by the research community are combined, the resulting pipeline-oriented integrated systems can outperform monolithic end-to-end systems [27] . For the KG information extraction task, however, to the best of our knowledge, approaches aiming at dynamically integrating and orchestrating various existing components do not exist. Objective and Contributions: Based on these observations, we build a framework that enables the integration of previously disjoint efforts on the KG-IE task under a single umbrella. We present the Plumber framework (cf. Figure 2 ) for creating Information Extraction pipelines. Plumber integrates 33 reusable components released by the research community for the subtasks entity linking (EL), relation linking (RL), text triple extraction (TE) (subject, predicate, object), and coreference resolution (CR). Overall, there are 264 different composable KG information extraction pipelines (generated by the possible combination of the available 33 components, i.e., for DBpedia 3 CRs, 8 TEs, 10 EL/RLs gives 3*8*10=240, and 4*3*2=24 for the ORKG. Hence, 240+24=264 pipelines). Plumber implements a transformer-based classification algorithm that intelligently chooses the best pipeline based on the unstructured input text.

We perform an exhaustive evaluation of Plumber on the two large-scale KGs DBpedia, and Open Research Knowledge Graph (ORKG) [24] to investigate the efficacy of Plumber in creating KG triples from unstructured text. We demonstrate that independent of the underlying KG; Plumber can find and assemble different extraction components to produce better suited KG triple extraction pipelines, significantly outperforming existing baselines. In summary, we provide the following novel contributions: i) The Plumber framework is the first of its kind for dynamically assembling and evaluating information extraction pipelines based on sequence classification techniques and for a given input text. Plumber is easily extensible and configurable, thus enabling the rapid creation and adjustment of new information extraction components and pipelines. Researchers can also use the framework for running IE components independently for specific subtasks such as triple extraction and entity linking. ii) A collection of 33 reusable IE components that can be combined to create 264 distinct IE pipelines. iii) The exhaustive evaluation and our detailed ablation study of the integrated components and composed pipelines on various input text will guide future research for collaborative KG information extraction.

We motivate our work with a running example; the sentence Rembrandt painted The Storm on the Sea of Galilee. It was painted in 1633. Multiple steps are required to extract these formally represented statements from the given text. First, the pronoun it in the second sentence should be replaced by The Storm on the Sea of Galilee using a coreference resolver. Next, a triple extractor should extract the correct text triples from the natural language text, i.e., <Rembrandt, painted, The Storm on the Sea of Galilee>, and <The Storm on the Sea of Galilee, painted in, 1633>. In the next step, the entity and relation linking component aligns the entity and relation surface forms extracted in the previous step to the DBpedia entities: dbr:Rembrandt for Rembrandt van Rijn, and dbr:The Storm on the Sea of Galilee for The Storm on the Sea of Galilee, and for relations: dbo:Artist for painted, and dbp:year for painted in. Figure 1 illustrates our running example and shows three Plumber IE pipelines with different results. In Pipeline 1, the coreference resolver is unable to map the pronoun it to the respective entity in the previous sentence. Moreover, the triple extractor generates incomplete triples, which also hinders the task of the entity and relation linker in the last step. Pipeline 2 uses a different set of components, and its output differs from the first pipeline. Here, the coreference resolution component is able to correctly co-relate the pronoun it to The Storm on the Sea of Galilee, and extract the text triple correctly. However, the overall result is only partially correct because the second triple is not extracted. Also, the linking component is not able to spot the second entity. Pipeline 3 correctly extracts both triples. This pipeline employs the same component as the second pipeline for coreference resolution but also includes an additional information extraction component (i.e., ReVerb [16] ) and a joint entity and relation linking component, namely Falcon [35] . With this combination of components, the text triple extractors were able to compensate for the loss of information in the second pipeline by adding one more component. Using the extracted text triples, the last component of the pipeline, a joint entity and relation linking tool, can map both triple components correctly to the corresponding KG entities. The reminder of this article is organized as follows. Related work is reviewed in Section 2. Section 3 presents Plumber, which is extensively evaluated in Section 4. Section 5 discusses the results, and Section 6 concludes and outlines directions for future research and work.

In the last decade, many open source tools have been released by the research community to tackle IE tasks for KGs. These IE components are not only used for end-to-end KG triple extraction but also for various other tasks, such as: Text Triple Extraction: The task of open information extraction is a well studied researched task in the NLP community [1] . It relies on NER (Named Entity Recognition) and RE (Relation Extraction). SalIE [33] uses MinIE [21] in combination with PageRank and clustering to find facts in the input text. Furthermore, OpenIE [1] leverages linguistic structures to extract self-contained clauses from the text. A comprehensive survey by Niklaus et al. [32] provides detailed about such techniques. Entity and Relation Linking: Entity and relation linking is a widely studied researched topic in the NLP, Web, and Information Retrieval research communities [3, 4, 11] . Often, entity and relation linking is performed independently. DBpedia Spotlight [10] is one of the first approaches for entity recognition and disambiguation over DBpedia. TagMe [18] links entities to DBpedia using in-link matching to disambiguate candidates entities. Others tools such as RelMatch [38] do not perform entity linking and only focus on linking the relation in the text to the corresponding KG relation. Recon [4] uses graph neural networks to to map relations between the entities with the assumption that entities are already linked in the text. EARL [15] is a joint linking tool over DBpedia and models the task as a generalized traveling salesperson problem. Sakor et al. [35] proposed Falcon, a linguistic rules based tool for joint entity and relation linking over DBpedia. Coreference Resolution: This task is used in conjunction with other tasks in NLP pipelines to disambiguate text and resolve syntactic complexities. The Stanford Coreference Resolver [34] uses a multi pass sieve of deterministic coreference models. Clark and Manning [8] use reinforcement learning to fine-tune a neural mention-ranking model for coreference resolution. And more recently [37] . Frameworks and Dynamic Pipelines: There have been few attempts in various domains aiming to consolidate the disjoint efforts of the research community under a single umbrella for solving a particular task. The Gerbil platform [41] provides an easy-to-use web-based platform for the agile comparison of entity linking tools using multiple datasets and uniform measuring approaches. OKBQA [26] is a community effort for the development of multilingual open knowledge base and QA systems. Frankenstein integrates 24 QA components to build QA systems collaboratively on-top of the Qanary integration framework [6] . Other ETL pipelines system exists such as Apache NiFi. Semantic Web Pipes [31] and LarKC [17] are other prominent examples. End-to-End Extraction Systems: More recently, end-to-end systems are gaining more attention due to the boom of deep learning techniques. Such systems draw on the strengths of deep models and transformers [13, 29] . Kertkeidkachorn and Ichise [25] present an end-to-end system to extract triples and link them to DBpedia. Other attempts such as KG-Bert [44] leverage deep transformers (i.e., BERT [13] ) for the triple classification task, given the entity and relation descriptions of a triple. KG-Bert does not attempt end-to-end alignment of KG triples from a given input text. Liu et al. [28] design an encoder-decoder framework with an attention mechanism to extract and align triples to a KG.

Plumber has a modular design (see Figure 2 ) where each component is integrated as a microservice. To ensure a consistent data exchange between components, the framework maps the output of each component to a homogeneous data representation using the Qanary [6] methodology. Plumber follows three design principles of i) Isolation, ii) Reusability, and iii) Extensibility inspired by [39, 41] .

Dynamic pipeline selection: Plumber uses a RoBERTa [29] based classifier that given a text and a set of requirements, Plumber predicts the best pipeline to extract KG triples. The RoBERTa model acts as intermediary that classifies the contextual embeddings extracted from the input text into a class which represents one of the possible pipelines. Regarding RoBERTa's training, we run each input sequence on all possible pipelines and compute the evaluation metrics F1-score (i.e., estimated performance). RoBERTa is fed with the sentence and the sentence-level performance with the best value among all pipelines as the target class. Hence, in practice, the user points Plumber to a piece of text and internally it uses RoBERTa to classify the text to a class (i.e., the pipeline) to execute against the input text.

Architecture: Plumber includes the following modules: i) IE Compo- Fig. 2 . Overview of Plumber's architecture highlighting the components for pipeline generation, selection, and execution. Plumber receives an input sentence and requirement (underlying KG) from the user. The framework intelligently selects a suitable pipeline based on the contextual features captured from the input sentence.

underlying KG). Users can manually select the underlying KG and, using the metadata associated with each component, Plumber aggregates the components for the concerned KG. iii) IE Pipelines Pool: Plumber stores the configurations of the possible pipelines in the pool of pipelines for faster retrieval and easier interaction with other modules. iv) Pipeline Selector: Based on the requirements (i.e., underlying KG) and the input text, a RoBERTa based model extracts contextual embeddings from the text and classifies the input into one of the possible classes. Each class corresponds to one pipeline configuration that is held in the IE pipelines pool. v) Pipeline Runner: Given the input text, and the generated pipeline configuration, the module executes the pipeline and produce the final KG triples.

In this section, we detail the empirical evaluation of the framework in comparison to baselines on different datasets and knowledge graphs. As such, we study the following research question: How does the dynamic selection of pipelines based on the input text affect the end-to-end information extraction task?

Knowledge Graphs To study the effectiveness of Plumber in building dynamic KG information extraction pipelines, we use the following KGs during our evaluation: DBpedia [2] is containing information extracted automatically from Wikipedia info boxes. DBpedia consists of approximately 11.5B triples [35] . Open Research Knowledge Graph [24] (ORKG) collects structured scholarly knowledge published in research articles, using crowd sourcing and automated techniques. In total, ORKG consists of approximately 984K triples.

Datasets Throughout our evaluation, we employed a set of existing and newly created datasets for structured triple extraction and alignment to knowledge graphs: the WebNLG [20] dataset for DBpedia, and COV-triples for ORKG. WebNLG is the Web Natural Language Generation Challenge. The challenge introduced the task of aligning unstructured text to DBpedia. In total, the dataset contains 46K triples with 9K triples in the testing and 37K in the training set. COV-triples is a handcrafted dataset that focuses on COVID-19 related scholarly articles. The COV-triples dataset consists of 21 abstracts from peer-reviewed articles and aligns the natural language text to the corresponding KG triples into the ORKG. Three Semantic Web researchers verified annotation quality, and triples approved by all three researchers are part of the dataset. The dataset contains only 75 triples. Hence, we use the WebNLG dataset for training, and 75 triples are used as a test set. Components and Implementation The Plumber framework integrates 33 components, the components span different IE tasks from Triple Extraction, Entity and Relation Linking, and Coreference Resolution. Most of the components used are open-sourced and they have been evaluated and used by the community in their respective publications. Plumber's code and all related resources are publicly available online at https://git.io/JtT1s. Baselines We include the following baselines: T2KG [25] is an end-to-end static system aligns a given natural language text to DBpedia KG triples. Frankenstein [39] dynamically composes Question Answering pipelines over the DBpedia KG. It employs logistic regression based classifiers for each component for predicting the accuracy and greedily composes a dynamic pipeline of the best components per task. We adapted Frankenstein for the KG information extraction over DBpedia.

The section summarizes a variety of experiments to compare the Plumber framework against other baselines. Note, that evaluating the performance of individual components or their combination is out of this evaluation's scope, since they were already used, benchmarked, and evaluated in the respective publications. We report values of the standard metrics Precision (P), Recall (R), and F1 score (F1). In all experiments, end-to-end components (e.g., T2KG) are not part of Plumber.

In this experiment, we report results of the static pipelines, i.e., no dynamic selection of a pipeline based on the input text is considered. We ran all 264 pipelines and Table 2 (T2KG & Static noted rows) reports the performance of the best Plumber pipeline against the baselines. Plumber static pipeline for DBpedia comprises of NeuralCoref [8] for coreference resolution, OpenIE [1] for text triple extraction, TagMe [18] for EL, and Falcon [35] for RL tasks. Also, in case of Frankenstein, we choose its best performing static pipeline. Results illustrated in the Table 2 confirm that the static pipeline composed by the components integrated in Plumber outperforms all baselines on DBpedia. We observe that the performance of pipeline approaches is better than an end-to-end monolithic information extraction approaches. Although the Plumber pipeline outperforms the baselines, the overall performance is relatively low. All our components have been trained on distinct corpora in their respective publications and our aim was to put them together to understand their collective strengths and weaknesses. Note, Frankenstein addresses the QA pipeline problem and not all components are comparable and can be applied in the context of information extraction. Thus, we integrated NeuralCoref coreference resolution component and OpenIE triple extraction component used in Plumber static pipeline into Frankenstein for providing the same experimental settings.

Static Pipeline for Scholarly KG In order to assess how Plumber performs on domain-specific use cases, we evaluate the static pipelines' performance on a scholarly knowledge graph. We use the COV-triples dataset for ORKG. To the best of our knowledge, no baseline exists on information extractions of research contribution descriptions over ORKG. Hence, we execute all static pipelines in Plumber tailored to ORKG to select the best one as shown in Table 2 (COVtriples rows). Plumber pipelines over ORKG extract statements determining the reproductive number estimates for the COVID-19 infectious disease from scientific articles as shown below. @prefix orkg : <\ protect \ vrule width0pt \ protect \ href { http :// orkg . org / orkg / resource /}{ http :// orkg . org / orkg / res @prefix orkgp : <\ protect \ vrule width0pt \ protect \ href { http :// orkg . org / orkg / property /}{ http :// orkg . org / orkg / pr orkg : R48100 orkgp : P16022 "2.68" .

In this example, orkg:R48100 refers to the city of Wuhan in China in the ORKG and orkgp:P16022 is the property "has R0 estimate (average)". The number "2.68" is the reproductive number estimate.

In this experiment, we study the effect of the transformer-based pipeline selection approach implemented in Plumber against the pipeline selection approach of Frankenstein. For a comparable experimental setting, we re-use Frankenstein's classification approach in Plumber, keeping the underlying components precisely the same. We perform a 10-fold cross-validation for the classification performance of the employed approach. Table 1 indicates that the Plumber pipeline selection significantly outperforms baselines across the board.

Our third experiment focuses on comparing the performance of Plumber against previous baselines for an end-to-end information extraction task. The results in Table 2 illustrate that the dynamic pipelines built using Plumber for KG information extraction outperform the best static pipelines of Plumber as well as the dynamically selected pipelines by Frankenstein (rows noted with dynamic).

The end-to-end baselines, such as Kertkeidka-chorn and Ichise [25] . We also observe that in cross-domain experiments for COV-triples datasets, dynamically selected pipelines perform better than the static pipeline. In the cross-domain experiment, the static and dynamic Plumber pipelines are relatively better performing than the other two KGs. Unlike components for DBpedia, components integrated into Plumber for ORKG are customized for KG triple extraction. We conclude that when components are integrated into a framework such as Plumber aiming for the KG information extraction task, it is crucial to select the pipeline based on the input text dynamically. The superior performance of Plumber shows that the dynamic pipeline selection has a positive impact agnostic of the underlying KG and dataset. This also answers our overall research question.

Plumber and baselines render relatively low performance on all the employed datasets. Hence, in the ablation studies our aim is to provide a holistic picture of underlying errors, collective success, and failures of the integrated components.

In the first study, we calculate the proportion of errors in Plumber. The modular architecture of the proposed framework allows us to benchmark each component independently. We consider the erroneous cases of Plumber on the We conclude that the text triple extractor components contribute to the largest chunk of the errors over DBpedia. One possible reason for their limited performance is that open-domain information extracting components were not initially released for the KG information extraction task. Also, these components do not incorporate any schema or prior knowledge to guide the extraction. We observe that the errors mainly occur when the sentence is complex (with more than one entity and predicate), or relations are not explicitly mentioned in the sentence. We further analyze the text triple extractor errors. The error analysis at the level of the triple subject, predicate, and object showed that most errors are in predicates (40.17%) followed by objects (35.98%) and subjects (23.85%).

Further Analysis Aiming to understand why IE pipelines perform with low accuracy, we conduct a more in-depth analysis per IE task. In the first analysis, we evaluated each component independently on the WebNLG dataset. Researchers [12, 40] proposed several criterion for micro-benchmarking tools/components for KG tasks (entity linking, relation linking, etc.) based on the linguistic features of a sentence. We motivate our analysis based on the following: I) Text Triple Extraction: We consider the number of words (wc) in the input sentence (a sentence is termed by "simple" with average word length of 7.41 [39] . Sentences with higher number of words than seven are complex sentences). Furthermore, having a comma in a sentence (sub-clause) to separate clauses is another factor. Atomic sentences (e.g., "cats have tails") are a type of sentence that also affects triples extractors' behavior. Moreover, nominal relation as in "Durin, son of Thorin" is another impacting factor on the performance. Uppercase and lowercase mentions of the words (i.e., correct capitalization of the first character and not the entire word) in a sentence are standard errors for entity linking components. We consider this as a micro-benchmarking criteria. II) Coreference Resolution: We focus on the length of the coreference chain (i.e., the number of aliases for a single mention). Additionally, the number of clusters is another criterion in the analysis. A cluster refers to the groups of mentions that require disambiguation (e.g., "mother bought a new phone, she is so happy about it" where the first cluster is mother → she and the second is phone → it). The presence of proper nouns in the sentence is studied as well as acronyms. Furthermore, the demonstrative nature of the sentence is also observed as a factor. Demonstrative sentences are the ones that contain demonstrative pronouns (this, that, etc.).

III) Entity Linking: The number of entities in a sentence (e=1,2) is a crucial observation for the entity linking task. Capitalization of the surface form is another criterion for micro-benchmarking entity linking tools. An entity is termed as an explicit entity when the entity's surface form in a sentence matches the KG label. An entity is implicit when there is a vocabulary mismatch. For example, in the sentence "The wife of Obama is Michelle Obama.", the surface form Obama is expected to be linked to dbr:Barack Obama and considered as an implicit entity [40] . The last linguistic feature is the number of words (w) in an entity label (e.g., The Storm on the Sea of Galilee has seven words).

IV) Relation Linking: Similar to the entity linking criteria, we focus on the number of relations in a sentence (rel=1,2). The type of relation (i.e., explicit, or implicit) is another parameter. Covered relation (sentences without a predicate surface form) is also used as a feature for micro-benchmarking: "Which companies have launched a rocket from Cape Canaveral Air Force station?" where the dbo:manufacturing relation is not mentioned in the sentence. Covered relations highly depend on common sense knowledge (i.e., reasoning) and the structure of the KG [40] . Lastly, the number of words (w<=N) in a predicate surface form is also considered. Figure 3 illustrates micro-benchmarking of various Plumber components per task. We observe that across IE tasks, the F1 score of the components varies significantly based on the sentence's linguistic features. In fact, there exist no single component which performs equally well on all the micro-benchmarking criteria. This observation further validates our hypothesis to design Plumber for building dynamic information extraction pipelines based on the strengths and weaknesses of the integrated components. We also note in Figure 3 that all the CR components report limited performance for the demonstrative sentences (demonstratives). When there is more than one coreference cluster in a sentence, all other CR components observe a discernible drop in F1 score. The NeuralCoref [8] component performs best for proper nouns, whereas PyCobalt [19] performs best for the acronyms feature (almost being tied by NeuralCoref). In the TE task, Graphene [7] shows the most stable performance across all categories. However, the performance of all components (except Dependency Parser) drops significantly when the number of words in a sentence exceeds seven (wc>7). Case sensitivity also affects the performance and all components observe a noticeable drop in F1 score for lowercase entity mentions in the sentence. Similar behavior is observed for entity linking components where case sensitivity is a significant cause of poor performance. When the sentence has one entity and it is implicit (e=1, implicit); all entity linking components face challenges in correctly linking the entities to the underlying KG. Relation linking components also report lower performance for implicit relations.

Even though the dynamic pipelines of Plumber outperforms static pipelines, the overall performance of Plumber and baselines for the KG information extraction task remains low. Our detailed and exhaustive ablation studies suggest that when individual components are plugged together, their individual performance is a major error source. However, this behavior is expected, considering earlier research works in other domains also observe a similar trend. As in 2015 Gerbil framework [41] and in 2018 Frankenstein [39] . Within two years, the community has released several components dedicated to solving entity linking and relation linking [35, 15, 30] , which were two loopholes identified by [39] for the QA task. We observe that state of the art components for information extraction still have much potential to improve their performance (both in terms of runtime and F1 score). It is essential to highlight that some of the issues observed in our ablation study are very basic and repeatedly pointed out by researchers in the community. For instance, Derczynski et al. [12] in 2015, followed by Singh et al. [39] in 2018, showed that case sensitivity is a main challenge for EL tools. Our observation in Figure 3 again confirms that case sensitivity of entity surface forms remains an open issue even for newly released components. In contrast, on specific datasets such as CoNLL-AIDA, several EL approaches reported F1 scores higher than 0.90 [43] , showing that EL tools are highly customized to particular datasets. In a real-world scenario like ours, the underlying limitations of approaches are uncovered.

In this paper, we presented the Plumber approach and framework for information extraction. Plumber effectively selects the best possible pipeline for a given input sentence using the sentential contextual features and a state-of-the-art transformer-based classification model. Plumber has a service-oriented architecture which is scalable, extensible, reusable, and agnostic of the underlying KG. The core idea of Plumber is to combine the strengths of already existing disjoint research for KG information extraction and build a foundation for a platform to promote reusability for the construction of large-scale and semantically structured KGs. Our empirical results suggest that the performance of the individual components directly impacts the end-to-end information extraction accuracy.

This article does not focus on internal system architecture or employed algorithms in a particular IE component to analyze the failures. The focus of the ablation studies is to holistically study the collective success and failure cases for the various tasks. Our studies provide the research community with insightful results over two knowledge graphs, 33 components, 264 pipelines. Our work is a step in the larger research agenda of offering the research community an effective way for synergistically combining and orchestrating various focused IE approaches balancing their strengths and weaknesses taking different application domains into account. We plan to extend our work in the following directions: i) extending Plumber to other KGs such as UMLS [5] and Wikidata [42] . ii) addressing multilinguality with Plumber, and iii) creating high performing RL components.

Leveraging linguistic structure for open domain information extraction

Dbpedia: A nucleus for a web of open data

Entity linking

Recon: Relation extraction using knowledge graph context in a graph neural network

The unified medical language system (umls): integrating biomedical terminology

Qanary -A methodology for vocabulary-driven open question answering systems

Graphene: Semantically-linked propositions in open information extraction

Deep reinforcement learning for mention-ranking coreference models

How hierarchical topics evolve in large text corpora

Improving efficiency and accuracy in multilingual entity extraction

Opentapioca: Lightweight entity linking for wikidata

Analysis of named entity recognition and linking for tweets

BERT: Pre-training of deep bidirectional transformers for language understanding

Triple classification using regions and fine-grained entity typing

EARL: Joint entity and relation linking for question answering over knowledge graphs

Identifying relations for open information extraction

Towards larkc: A platform for web-scale reasoning

TAGME: on-the-fly annotation of short text fragments (by wikipedia entities)

Lambda-3/pycobalt: Coreference resolution in python

Creating training corpora for NLG micro-planners

MinIE: Minimizing facts in open information extraction

Identification of tasks, datasets, evaluation metrics, and numeric scores for scientific leaderboards construction

Bridging quantities in tables and text

Open Research Knowledge Graph: Next Generation Infrastructure for Semantic Scholarly Knowledge

T2kg: An end-to-end system for creating knowledge graph from unstructured text

OKBQA Framework for collaboration on developing natural language question answering systems

Querying knowledge graphs in natural language

Seq2rdf: An end-to-end application for deriving triples from natural language text

A robustly optimized bert pretraining approach

Leveraging semantic parsing for relation linking over knowledge bases

Semantic web pipes

A survey on open information extraction

Facts that matter

A multi-pass sieve for coreference resolution

Old is gold: Linguistic driven approach for entity and relation linking of short text

Falcon 2.0: An entity and relation linking tool over wikidata

A hierarchical multi-task approach for learning embeddings from semantic tasks

Capturing knowledge in semantically-typed relational patterns to enhance relation linking

Why reinvent the wheel: Let's build question answering systems together

Qaldgen: Towards microbenchmarking of question answering systems over knowledge graphs

Gerbil: general entity annotator benchmarking framework

Wikidata: A Free Collaborative Knowledgebase

Learning dynamic context augmentation for global entity linking

Kg-bert: Bert for knowledge graph completion

Tablepedia: Automating pdf table reading in an experimental evidence exploration and analytic system