Main references
- Machine Knowledge: Creation and Curation of Comprehensive Knowledge BasesFoundations and Trends in Databases, 10(2--4), pp. 108–490, 2021.
Equipping machines with comprehensive knowledge of the world's entities and their relationships has been a long-standing goal of AI. Over the last decade, large-scale knowledge bases, also known as knowledge graphs, have been automatically constructed from web contents and text sources, and have become a key asset for search engines. This machine knowledge can be harnessed to semantically interpret textual phrases in news, social media and web tables, and contributes to question answering, natural language processing and data analytics. This article surveys fundamental concepts and practical methods for creating and curating large knowledge bases. It covers models and methods for discovering and canonicalizing entities and their semantic types and organizing them into clean taxonomies. On top of this, the article discusses the automatic extraction of entity-centric properties. To support the long-term life-cycle and the quality assurance of machine knowledge, the article presents methods for constructing open schemas and for knowledge curation. Case studies on academic projects and industrial knowledge graphs complement the survey of concepts and methods.
@article{weikum-etal-2021-machine, author = {Weikum, Gerhard and Dong, Xin Luna and Razniewski, Simon and Suchanek, Fabian}, title = {Machine Knowledge: Creation and Curation of Comprehensive Knowledge Bases}, journal = {Foundations and Trends in Databases}, year = {2021}, archiveprefix = {arXiv}, doi = {10.1561/1900000064}, eprint = {2009.11564}, number = {2--4}, pages = {108--490}, publisher = {Now Publishers}, volume = {10} } - Knowledge Graphs: Fundamentals, Techniques, and ApplicationsMIT Press, 2021.
@book{kejriwal-etal-2021-knowledge, author = {Kejriwal, Mayank and Knoblock, Craig A. and Szekely, Pedro}, title = {Knowledge Graphs: Fundamentals, Techniques, and Applications}, year = {2021}, address = {Cambridge, MA}, isbn = {9780262045094}, publisher = {MIT Press}, url = {https://mitpress.mit.edu/9780262045094/knowledge-graphs/} } - Knowledge Graphs and LLMs in ActionManning, 2025.
@book{negro-etal-2025-knowledge, author = {Negro, Alessandro and Kus, Vlastimil and Futia, Giuseppe and Montagna, Fabio}, title = {Knowledge Graphs and {LLM}s in Action}, year = {2025}, address = {Shelter Island, NY}, isbn = {9781633439894}, publisher = {Manning}, url = {https://www.manning.com/books/knowledge-graphs-and-llms-in-action} }
Entity linking
- Shorten the Long Tail for Rare Entity and Event ExtractionIn Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 1339–1350, 2023.
The distribution of knowledge elements such as entity types and event types is long-tailed in natural language. Hence information extraction datasets naturally conform long-tailed distribution. Although imbalanced datasets can teach the model about the useful real-world bias, deep learning models may learn features not generalizable to rare or unseen expressions of entities or events during evaluation, especially for rare types without sufficient training instances. Existing approaches for the long-tailed learning problem seek to manipulate the training data by re-balancing, augmentation or introducing extra prior knowledge. In comparison, we propose to handle the generalization challenge by making the evaluation instances closer to the frequent training cases. We design a new transformation module that transforms infrequent candidate mention representation during evaluation with the average mention representation in the training dataset. Experimental results on classic benchmarks on three entity or event extraction datasets demonstrates the effectiveness of our framework.
@inproceedings{yu-ji-2023-shorten, author = {Yu, Pengfei and Ji, Heng}, title = {Shorten the Long Tail for Rare Entity and Event Extraction}, booktitle = {Proceedings of the 17th Conference of the {E}uropean Chapter of the {A}ssociation for {C}omputational {L}inguistics}, year = {2023}, address = {Dubrovnik, Croatia}, doi = {10.18653/v1/2023.eacl-main.97}, editor = {Vlachos, Andreas and Augenstein, Isabelle}, month = {May}, pages = {1339--1350}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2023.eacl-main.97/} } - Entity Linking Meets Deep Learning: Techniques and SolutionsIEEE Transactions on Knowledge and Data Engineering, 2021.
Entity linking (EL) is the process of linking entity mentions appearing in web text with their corresponding entities in a knowledge base. EL plays an important role in the fields of knowledge engineering and data mining, underlying a variety of downstream applications such as knowledge base population, content analysis, relation extraction, and question answering. In recent years, deep learning (DL), which has achieved tremendous success in various domains, has also been leveraged in EL methods to surpass traditional machine learning based methods and yield the state-of-the-art performance. In this survey, we present a comprehensive review and analysis of existing DL based EL methods. First of all, we propose a new taxonomy, which organizes existing DL based EL methods using three axes: embedding, feature, and algorithm. Then we systematically survey the representative EL methods along the three axes of the taxonomy. Later, we introduce ten commonly used EL data sets and give a quantitative performance analysis of DL based EL methods over these data sets. Finally, we discuss the remaining limitations of existing methods and highlight some promising future directions.
@article{shen-etal-2021-entity, author = {Shen, Wei and Li, Yuhan and Liu, Yinan and Han, Jiawei and Wang, Jianyong and Yuan, Xiaojie}, title = {Entity Linking Meets Deep Learning: Techniques and Solutions}, journal = {IEEE Transactions on Knowledge and Data Engineering}, year = {2021}, archiveprefix = {arXiv}, doi = {10.1109/TKDE.2021.3117715}, eprint = {2109.12520}, publisher = {IEEE} } - Deep Entity Matching with Pre-Trained Language ModelsProceedings of the VLDB Endowment, 14(1), pp. 50–60, 2020.
We present Ditto, a novel entity matching system based on pre-trained Transformer-based language models. We fine-tune and cast EM as a sequence-pair classification problem to leverage such models with a simple architecture. Our experiments show that a straightforward application of language models such as BERT, DistilBERT, or RoBERTa pre-trained on large text corpora already significantly improves the matching quality and outperforms previous state-of-the-art (SOTA), by up to 29% of F1 score on benchmark datasets. We also developed three optimization techniques to further improve Ditto's matching capability. Ditto allows domain knowledge to be injected by highlighting important pieces of input information that may be of interest when making matching decisions. Ditto also summarizes strings that are too long so that only the essential information is retained and used for EM. Finally, Ditto adapts a SOTA technique on data augmentation for text to EM to augment the training data with (difficult) examples. This way, Ditto is forced to learn "harder" to improve the model's matching capability. The optimizations we developed further boost the performance of Ditto by up to 9.8%. Perhaps more surprisingly, we establish that Ditto can achieve the previous SOTA results with at most half the number of labeled data. Finally, we demonstrate Ditto's effectiveness on a real-world large-scale EM task. On matching two company datasets consisting of 789K and 412K records, Ditto achieves a high F1 score of 96.5%.
@article{li-etal-2020-deep, author = {Li, Yuliang and Li, Jinfeng and Suhara, Yoshihiko and Doan, AnHai and Tan, Wang-Chiew}, title = {Deep Entity Matching with Pre-Trained Language Models}, journal = {Proceedings of the VLDB Endowment}, year = {2020}, archiveprefix = {arXiv}, doi = {10.14778/3421424.3421431}, eprint = {2004.00584}, number = {1}, pages = {50--60}, publisher = {VLDB Endowment}, volume = {14} } - Scalable Zero-shot Entity Linking with Dense Entity RetrievalIn Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6397–6407, 2020.
This paper introduces a conceptually simple, scalable, and highly effective BERT-based entity linking model, along with an extensive evaluation of its accuracy-speed trade-off. We present a two-stage zero-shot linking algorithm, where each entity is defined only by a short textual description. The first stage does retrieval in a dense space defined by a bi-encoder that independently embeds the mention context and the entity descriptions. Each candidate is then re-ranked with a cross-encoder, that concatenates the mention and entity text. Experiments demonstrate that this approach is state of the art on recent zero-shot benchmarks (6 point absolute gains) and also on more established non-zero-shot evaluations (e.g. TACKBP-2010), despite its relative simplicity (e.g. no explicit entity embeddings or manually engineered mention tables). We also show that bi-encoder linking is very fast with nearest neighbor search (e.g. linking with 5.9 million candidates in 2 milliseconds), and that much of the accuracy gain from the more expensive cross-encoder can be transferred to the bi-encoder via knowledge distillation. Our code and models are available at https://github.com/facebookresearch/BLINK.
@inproceedings{wu-etal-2020-scalable, author = {Wu, Ledell and Petroni, Fabio and Josifoski, Martin and Riedel, Sebastian and Zettlemoyer, Luke}, title = {Scalable Zero-shot Entity Linking with Dense Entity Retrieval}, booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing ({EMNLP})}, year = {2020}, address = {Online}, doi = {10.18653/v1/2020.emnlp-main.519}, editor = {Webber, Bonnie and Cohn, Trevor and He, Yulan and Liu, Yang}, month = {November}, pages = {6397--6407}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2020.emnlp-main.519/} } - Entity Linking in 100 LanguagesIn Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7833–7845, 2020.
We propose a new formulation for multilingual entity linking, where language-specific mentions resolve to a language-agnostic Knowledge Base. We train a dual encoder in this new setting, building on prior work with improved feature representation, negative mining, and an auxiliary entity-pairing task, to obtain a single entity retrieval model that covers 100+ languages and 20 million entities. The model outperforms state-of-the-art results from a far more limited cross-lingual linking task. Rare entities and low-resource languages pose challenges at this large-scale, so we advocate for an increased focus on zero- and few-shot evaluation. To this end, we provide Mewsli-9, a large new multilingual dataset matched to our setting, and show how frequency-based analysis provided key insights for our model and training enhancements.
@inproceedings{botha-etal-2020-entity, author = {Botha, Jan A. and Shan, Zifei and Gillick, Daniel}, title = {{E}ntity {L}inking in 100 {L}anguages}, booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing ({EMNLP})}, year = {2020}, address = {Online}, doi = {10.18653/v1/2020.emnlp-main.630}, editor = {Webber, Bonnie and Cohn, Trevor and He, Yulan and Liu, Yang}, month = {November}, pages = {7833--7845}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2020.emnlp-main.630/} } - Zero-Shot Entity Linking by Reading Entity DescriptionsIn Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3449–3460, 2019.
We present the zero-shot entity linking task, where mentions must be linked to unseen entities without in-domain labeled data. The goal is to enable robust transfer to highly specialized domains, and so no metadata or alias tables are assumed. In this setting, entities are only identified by text descriptions, and models must rely strictly on language understanding to resolve the new entities. First, we show that strong reading comprehension models pre-trained on large unlabeled data can be used to generalize to unseen entities. Second, we propose a simple and effective adaptive pre-training strategy, which we term domain-adaptive pre-training (DAP), to address the domain shift problem associated with linking unseen entities in a new domain. We present experiments on a new dataset that we construct for this task and show that DAP improves over strong pre-training baselines, including BERT. The data and code are available at https://github.com/lajanugen/zeshel.
@inproceedings{logeswaran-etal-2019-zero, author = {Logeswaran, Lajanugen and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina and Devlin, Jacob and Lee, Honglak}, title = {Zero-Shot Entity Linking by Reading Entity Descriptions}, booktitle = {Proceedings of the 57th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics}, year = {2019}, address = {Florence, Italy}, doi = {10.18653/v1/P19-1335}, editor = {Korhonen, Anna and Traum, David and M{\`a}rquez, Llu{\'i}s}, month = {July}, pages = {3449--3460}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/P19-1335/} } - DeepType: Multilingual Entity Linking by Neural Type System EvolutionIn Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18), 2018.
The wealth of structured (e.g. Wikidata) and unstructured data about the world available today presents an incredible opportunity for tomorrow's Artificial Intelligence. So far, integration of these two different modalities is a difficult process, involving many decisions concerning how best to represent the information so that it will be captured or useful, and hand-labeling large amounts of data. DeepType overcomes this challenge by explicitly integrating symbolic information into the reasoning process of a neural network with a type system. First we construct a type system, and second, we use it to constrain the outputs of a neural network to respect the symbolic structure. We achieve this by reformulating the design problem into a mixed integer problem: create a type system and subsequently train a neural network with it. In this reformulation discrete variables select which parent-child relations from an ontology are types within the type system, while continuous variables control a classifier fit to the type system. The original problem cannot be solved exactly, so we propose a 2-step algorithm: 1) heuristic search or stochastic optimization over discrete variables that define a type system informed by an Oracle and a Learnability heuristic, 2) gradient descent to fit classifier parameters. We apply DeepType to the problem of Entity Linking on three standard datasets (i.e. WikiDisamb30, CoNLL (YAGO), TAC KBP 2010) and find that it outperforms all existing solutions by a wide margin, including approaches that rely on a human-designed type system or recent deep learning-based entity embeddings, while explicitly using symbolic information lets it integrate new entities without retraining.
@inproceedings{raiman-raiman-2018-deeptype, author = {Raiman, Jonathan and Raiman, Olivier}, title = {{D}eep{T}ype: Multilingual Entity Linking by Neural Type System Evolution}, booktitle = {Proceedings of the Thirty-Second {AAAI} Conference on Artificial Intelligence ({AAAI}-18)}, year = {2018}, archiveprefix = {arXiv}, doi = {10.1609/aaai.v32i1.12008}, eprint = {1802.01021}, publisher = {AAAI Press}, url = {https://ojs.aaai.org/index.php/AAAI/article/view/12008} }
Relation classification
- DEER: Descriptive Knowledge Graph for Explaining Entity RelationshipsIn Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6686–6698, 2022.
We propose DEER (Descriptive Knowledge Graph for Explaining Entity Relationships) - an open and informative form of modeling entity relationships. In DEER, relationships between entities are represented by free-text relation descriptions. For instance, the relationship between entities of machine learning and algorithm can be represented as “Machine learning explores the study and construction of algorithms that can learn from and make predictions on data.” To construct DEER, we propose a self-supervised learning method to extract relation descriptions with the analysis of dependency patterns and generate relation descriptions with a transformer-based relation description synthesizing model, where no human labeling is required. Experiments demonstrate that our system can extract and generate high-quality relation descriptions for explaining entity relationships. The results suggest that we can build an open and informative knowledge graph without human annotation.
@inproceedings{huang-etal-2022-deer, author = {Huang, Jie and Zhu, Kerui and Chang, Kevin Chen-Chuan and Xiong, Jinjun and Hwu, Wen-mei}, title = {{DEER}: Descriptive Knowledge Graph for Explaining Entity Relationships}, booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing}, year = {2022}, address = {Abu Dhabi, United Arab Emirates}, doi = {10.18653/v1/2022.emnlp-main.448}, editor = {Goldberg, Yoav and Kozareva, Zornitsa and Zhang, Yue}, month = {December}, pages = {6686--6698}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2022.emnlp-main.448/} } - KnowledgeNet: A Benchmark Dataset for Knowledge Base PopulationIn Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 749–758, 2019.
KnowledgeNet is a benchmark dataset for the task of automatically populating a knowledge base (Wikidata) with facts expressed in natural language text on the web. KnowledgeNet provides text exhaustively annotated with facts, thus enabling the holistic end-to-end evaluation of knowledge base population systems as a whole, unlike previous benchmarks that are more suitable for the evaluation of individual subcomponents (e.g., entity linking, relation extraction). We discuss five baseline approaches, where the best approach achieves an F1 score of 0.50, significantly outperforming a traditional approach by 79% (0.28). However, our best baseline is far from reaching human performance (0.82), indicating our dataset is challenging. The KnowledgeNet dataset and baselines are available at https://github.com/diffbot/knowledge-net
@inproceedings{mesquita-etal-2019-knowledgenet, author = {Mesquita, Filipe and Cannaviccio, Matteo and Schmidek, Jordan and Mirza, Paramita and Barbosa, Denilson}, title = {{K}nowledge{N}et: A Benchmark Dataset for Knowledge Base Population}, booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing ({EMNLP}-{IJCNLP})}, year = {2019}, address = {Hong Kong, China}, doi = {10.18653/v1/D19-1069}, editor = {Inui, Kentaro and Jiang, Jing and Ng, Vincent and Wan, Xiaojun}, month = {November}, pages = {749--758}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/D19-1069/} } - A Novel Cascade Binary Tagging Framework for Relational Triple ExtractionIn Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1476–1488, 2020.
Extracting relational triples from unstructured text is crucial for large-scale knowledge graph construction. However, few existing works excel in solving the overlapping triple problem where multiple relational triples in the same sentence share the same entities. In this work, we introduce a fresh perspective to revisit the relational triple extraction task and propose a novel cascade binary tagging framework (CasRel) derived from a principled problem formulation. Instead of treating relations as discrete labels as in previous works, our new framework models relations as functions that map subjects to objects in a sentence, which naturally handles the overlapping problem. Experiments show that the CasRel framework already outperforms state-of-the-art methods even when its encoder module uses a randomly initialized BERT encoder, showing the power of the new tagging framework. It enjoys further performance boost when employing a pre-trained BERT encoder, outperforming the strongest baseline by 17.5 and 30.2 absolute gain in F1-score on two public datasets NYT and WebNLG, respectively. In-depth analysis on different scenarios of overlapping triples shows that the method delivers consistent performance gain across all these scenarios. The source code and data are released online.
@inproceedings{wei-etal-2020-novel, author = {Wei, Zhepei and Su, Jianlin and Wang, Yue and Tian, Yuan and Chang, Yi}, title = {A Novel Cascade Binary Tagging Framework for Relational Triple Extraction}, booktitle = {Proceedings of the 58th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics}, year = {2020}, address = {Online}, doi = {10.18653/v1/2020.acl-main.136}, editor = {Jurafsky, Dan and Chai, Joyce and Schluter, Natalie and Tetreault, Joel}, month = {July}, pages = {1476--1488}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2020.acl-main.136/} } - Attention Guided Graph Convolutional Networks for Relation ExtractionIn Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 241–251, 2019.
Dependency trees convey rich structural information that is proven useful for extracting relations among entities in text. However, how to effectively make use of relevant information while ignoring irrelevant information from the dependency trees remains a challenging research question. Existing approaches employing rule based hard-pruning strategies for selecting relevant partial dependency structures may not always yield optimal results. In this work, we propose Attention Guided Graph Convolutional Networks (AGGCNs), a novel model which directly takes full dependency trees as inputs. Our model can be understood as a soft-pruning approach that automatically learns how to selectively attend to the relevant sub-structures useful for the relation extraction task. Extensive results on various tasks including cross-sentence n-ary relation extraction and large-scale sentence-level relation extraction show that our model is able to better leverage the structural information of the full dependency trees, giving significantly better results than previous approaches.
@inproceedings{guo-etal-2019-attention, author = {Guo, Zhijiang and Zhang, Yan and Lu, Wei}, title = {Attention Guided Graph Convolutional Networks for Relation Extraction}, booktitle = {Proceedings of the 57th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics}, year = {2019}, address = {Florence, Italy}, doi = {10.18653/v1/P19-1024}, editor = {Korhonen, Anna and Traum, David and M{\`a}rquez, Llu{\'i}s}, month = {July}, pages = {241--251}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/P19-1024/} } - Graph Convolution over Pruned Dependency Trees Improves Relation ExtractionIn Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2205–2215, 2018.
Dependency trees help relation extraction models capture long-range relations between words. However, existing dependency-based models either neglect crucial information (e.g., negation) by pruning the dependency trees too aggressively, or are computationally inefficient because it is difficult to parallelize over different tree structures. We propose an extension of graph convolutional networks that is tailored for relation extraction, which pools information over arbitrary dependency structures efficiently in parallel. To incorporate relevant information while maximally removing irrelevant content, we further apply a novel pruning strategy to the input trees by keeping words immediately around the shortest path between the two entities among which a relation might hold. The resulting model achieves state-of-the-art performance on the large-scale TACRED dataset, outperforming existing sequence and dependency-based neural models. We also show through detailed analysis that this model has complementary strengths to sequence models, and combining them further improves the state of the art.
@inproceedings{zhang-etal-2018-graph, author = {Zhang, Yuhao and Qi, Peng and Manning, Christopher D.}, title = {Graph Convolution over Pruned Dependency Trees Improves Relation Extraction}, booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing}, year = {2018}, address = {Brussels, Belgium}, doi = {10.18653/v1/D18-1244}, editor = {Riloff, Ellen and Chiang, David and Hockenmaier, Julia and Tsujii, Jun{'}ichi}, month = {October-November}, pages = {2205--2215}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/D18-1244/} }
Open relation extraction
- OpenFact: Factuality Enhanced Open Knowledge ExtractionTransactions of the Association for Computational Linguistics, 11, pp. 686–702, 2023.
We focus on the factuality property during the extraction of an OpenIE corpus named OpenFact, which contains more than 12 million high-quality knowledge triplets. We break down the factuality property into two important aspects—expressiveness and groundedness—and we propose a comprehensive framework to handle both aspects. To enhance expressiveness, we formulate each knowledge piece in OpenFact based on a semantic frame. We also design templates, extra constraints, and adopt human efforts so that most OpenFact triplets contain enough details. For groundedness, we require the main arguments of each triplet to contain linked Wikidata1 entities. A human evaluation suggests that the OpenFact triplets are much more accurate and contain denser information compared to OPIEC-Linked (Gashteovski et al., 2019), one recent high-quality OpenIE corpus grounded to Wikidata. Further experiments on knowledge base completion and knowledge base question answering show the effectiveness of OpenFact over OPIEC-Linked as supplementary knowledge to Wikidata as the major KG.
@article{song-etal-2023-openfact, author = {Song, Linfeng and Wang, Ante and Pan, Xiaoman and Zhang, Hongming and Yu, Dian and Jin, Lifeng and Mi, Haitao and Su, Jinsong and Zhang, Yue and Yu, Dong}, title = {{O}pen{F}act: Factuality Enhanced Open Knowledge Extraction}, journal = {Transactions of the Association for Computational Linguistics}, year = {2023}, address = {Cambridge, MA}, doi = {10.1162/tacl_a_00569}, pages = {686--702}, publisher = {MIT Press}, url = {https://aclanthology.org/2023.tacl-1.40/}, volume = {11} } - Unsupervised Relation Extraction from Language Models using Constrained Cloze CompletionIn Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1263–1276, 2020.
We show that state-of-the-art self-supervised language models can be readily used to extract relations from a corpus without the need to train a fine-tuned extractive head. We introduce RE-Flex, a simple framework that performs constrained cloze completion over pretrained language models to perform unsupervised relation extraction. RE-Flex uses contextual matching to ensure that language model predictions matches supporting evidence from the input corpus that is relevant to a target relation. We perform an extensive experimental study over multiple relation extraction benchmarks and demonstrate that RE-Flex outperforms competing unsupervised relation extraction methods based on pretrained language models by up to 27.8 F1 points compared to the next-best method. Our results show that constrained inference queries against a language model can enable accurate unsupervised relation extraction.
@inproceedings{goswami-etal-2020-unsupervised-relation, author = {Goswami, Ankur and Bhat, Akshata and Ohana, Hadar and Rekatsinas, Theodoros}, title = {Unsupervised Relation Extraction from Language Models using Constrained Cloze Completion}, booktitle = {Findings of the {A}ssociation for {C}omputational {L}inguistics: {EMNLP} 2020}, year = {2020}, address = {Online}, doi = {10.18653/v1/2020.findings-emnlp.113}, editor = {Cohn, Trevor and He, Yulan and Liu, Yang}, month = {November}, pages = {1263--1276}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2020.findings-emnlp.113/} } - Open Relation Extraction: Relational Knowledge Transfer from Supervised Data to Unsupervised DataIn Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 219–228, 2019.
Open relation extraction (OpenRE) aims to extract relational facts from the open-domain corpus. To this end, it discovers relation patterns between named entities and then clusters those semantically equivalent patterns into a united relation cluster. Most OpenRE methods typically confine themselves to unsupervised paradigms, without taking advantage of existing relational facts in knowledge bases (KBs) and their high-quality labeled instances. To address this issue, we propose Relational Siamese Networks (RSNs) to learn similarity metrics of relations from labeled data of pre-defined relations, and then transfer the relational knowledge to identify novel relations in unlabeled data. Experiment results on two real-world datasets show that our framework can achieve significant improvements as compared with other state-of-the-art methods. Our code is available at https://github.com/thunlp/RSN.
@inproceedings{wu-etal-2019-open, author = {Wu, Ruidong and Yao, Yuan and Han, Xu and Xie, Ruobing and Liu, Zhiyuan and Lin, Fen and Lin, Leyu and Sun, Maosong}, title = {Open Relation Extraction: Relational Knowledge Transfer from Supervised Data to Unsupervised Data}, booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing ({EMNLP}-{IJCNLP})}, year = {2019}, address = {Hong Kong, China}, doi = {10.18653/v1/D19-1021}, editor = {Inui, Kentaro and Jiang, Jing and Ng, Vincent and Wan, Xiaojun}, month = {November}, pages = {219--228}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/D19-1021/} }
Question answering
- QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question AnsweringIn Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 535–546, 2021.
The problem of answering questions using knowledge from pre-trained language models (LMs) and knowledge graphs (KGs) presents two challenges: given a QA context (question and answer choice), methods need to (i) identify relevant knowledge from large KGs, and (ii) perform joint reasoning over the QA context and KG. Here we propose a new model, QA-GNN, which addresses the above challenges through two key innovations: (i) relevance scoring, where we use LMs to estimate the importance of KG nodes relative to the given QA context, and (ii) joint reasoning, where we connect the QA context and KG to form a joint graph, and mutually update their representations through graph-based message passing. We evaluate QA-GNN on the CommonsenseQA and OpenBookQA datasets, and show its improvement over existing LM and LM+KG models, as well as its capability to perform interpretable and structured reasoning, e.g., correctly handling negation in questions.
@inproceedings{yasunaga-etal-2021-qa, author = {Yasunaga, Michihiro and Ren, Hongyu and Bosselut, Antoine and Liang, Percy and Leskovec, Jure}, title = {{QA}-{GNN}: Reasoning with Language Models and Knowledge Graphs for Question Answering}, booktitle = {Proceedings of the 2021 Conference of the {N}orth {A}merican Chapter of the {A}ssociation for {C}omputational {L}inguistics: Human Language Technologies}, year = {2021}, address = {Online}, doi = {10.18653/v1/2021.naacl-main.45}, editor = {Toutanova, Kristina and Rumshisky, Anna and Zettlemoyer, Luke and Hakkani-Tur, Dilek and Beltagy, Iz and Bethard, Steven and Cotterell, Ryan and Chakraborty, Tanmoy and Zhou, Yichao}, month = {June}, pages = {535--546}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2021.naacl-main.45/} } - Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question AnsweringIn Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1295–1309, 2020.
Existing work on augmenting question answering (QA) models with external knowledge (e.g., knowledge graphs) either struggle to model multi-hop relations efficiently, or lack transparency into the model’s prediction rationale. In this paper, we propose a novel knowledge-aware approach that equips pre-trained language models (PTLMs) has with a multi-hop relational reasoning module, named multi-hop graph relation network (MHGRN). It performs multi-hop, multi-relational reasoning over subgraphs extracted from external knowledge graphs. The proposed reasoning module unifies path-based reasoning methods and graph neural networks to achieve better interpretability and scalability. We also empirically show its effectiveness and scalability on CommonsenseQA and OpenbookQA datasets, and interpret its behaviors with case studies, with the code for experiments released.
@inproceedings{feng-etal-2020-scalable, author = {Feng, Yanlin and Chen, Xinyue and Lin, Bill Yuchen and Wang, Peifeng and Yan, Jun and Ren, Xiang}, title = {Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answering}, booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing ({EMNLP})}, year = {2020}, address = {Online}, doi = {10.18653/v1/2020.emnlp-main.99}, editor = {Webber, Bonnie and Cohn, Trevor and He, Yulan and Liu, Yang}, month = {November}, pages = {1295--1309}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2020.emnlp-main.99/} } - What Question Answering can Learn from Trivia NerdsIn Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7422–7435, 2020.
In addition to the traditional task of machines answering questions, question answering (QA) research creates interesting, challenging questions that help systems how to answer questions and reveal the best systems. We argue that creating a QA dataset—and the ubiquitous leaderboard that goes with it—closely resembles running a trivia tournament: you write questions, have agents (either humans or machines) answer the questions, and declare a winner. However, the research community has ignored the hard-learned lessons from decades of the trivia community creating vibrant, fair, and effective question answering competitions. After detailing problems with existing QA datasets, we outline the key lessons—removing ambiguity, discriminating skill, and adjudicating disputes—that can transfer to QA research and how they might be implemented.
@inproceedings{boyd-graber-borschinger-2020-question, author = {Boyd-Graber, Jordan and B{\"o}rschinger, Benjamin}, title = {What Question Answering can Learn from Trivia Nerds}, booktitle = {Proceedings of the 58th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics}, year = {2020}, address = {Online}, doi = {10.18653/v1/2020.acl-main.662}, editor = {Jurafsky, Dan and Chai, Joyce and Schluter, Natalie and Tetreault, Joel}, month = {July}, pages = {7422--7435}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2020.acl-main.662/} } - Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question AnsweringIn International Conference on Learning Representations (ICLR), 2020.
Answering questions that require multi-hop reasoning at web-scale necessitates retrieving multiple evidence documents, one of which often has little lexical or semantic relationship to the question. This paper introduces a new graph-based recurrent retrieval approach that learns to retrieve reasoning paths over the Wikipedia graph to answer multi-hop open-domain questions. Our retriever model trains a recurrent neural network that learns to sequentially retrieve evidence paragraphs in the reasoning path by conditioning on the previously retrieved documents. Our reader model ranks the reasoning paths and extracts the answer span included in the best reasoning path. Experimental results show state-of-the-art results in three open-domain QA datasets, showcasing the effectiveness and robustness of our method. Notably, our method achieves significant improvement in HotpotQA, outperforming the previous best model by more than 14 points.
@inproceedings{asai-etal-2020-learning, author = {Asai, Akari and Hashimoto, Kazuma and Hajishirzi, Hannaneh and Socher, Richard and Xiong, Caiming}, title = {Learning to Retrieve Reasoning Paths over {W}ikipedia Graph for Question Answering}, booktitle = {International Conference on Learning Representations ({ICLR})}, year = {2020}, archiveprefix = {arXiv}, eprint = {1911.10470}, url = {https://openreview.net/forum?id=SJgVHkrYDH} }
End-to-end methods
- COMET: Commonsense Transformers for Automatic Knowledge Graph ConstructionIn Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4762–4779, 2019.
We present the first comprehensive study on automatic knowledge base construction for two prevalent commonsense knowledge graphs: ATOMIC (Sap et al., 2019) and ConceptNet (Speer et al., 2017). Contrary to many conventional KBs that store knowledge with canonical templates, commonsense KBs only store loosely structured open-text descriptions of knowledge. We posit that an important step toward automatic commonsense completion is the development of generative models of commonsense knowledge, and propose COMmonsEnse Transformers (COMET) that learn to generate rich and diverse commonsense descriptions in natural language. Despite the challenges of commonsense modeling, our investigation reveals promising results when implicit knowledge from deep pre-trained language models is transferred to generate explicit knowledge in commonsense knowledge graphs. Empirical results demonstrate that COMET is able to generate novel knowledge that humans rate as high quality, with up to 77.5% (ATOMIC) and 91.7% (ConceptNet) precision at top 1, which approaches human performance for these resources. Our findings suggest that using generative commonsense models for automatic commonsense KB completion could soon be a plausible alternative to extractive methods.
@inproceedings{bosselut-etal-2019-comet, author = {Bosselut, Antoine and Rashkin, Hannah and Sap, Maarten and Malaviya, Chaitanya and Celikyilmaz, Asli and Choi, Yejin}, title = {{COMET}: Commonsense Transformers for Automatic Knowledge Graph Construction}, booktitle = {Proceedings of the 57th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics}, year = {2019}, address = {Florence, Italy}, doi = {10.18653/v1/P19-1470}, editor = {Korhonen, Anna and Traum, David and M{\`a}rquez, Llu{\'i}s}, month = {July}, pages = {4762--4779}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/P19-1470/} }
KG completion
- StATIK: Structure and Text for Inductive Knowledge Graph CompletionIn Findings of the Association for Computational Linguistics: NAACL 2022, pp. 604–615, 2022.
Knowledge graphs (KGs) often represent knowledge bases that are incomplete. Machine learning models can alleviate this by helping automate graph completion. Recently, there has been growing interest in completing knowledge bases that are dynamic, where previously unseen entities may be added to the KG with many missing links. In this paper, we present StATIK–Structure And Text for Inductive Knowledge Completion. StATIK uses Language Models to extract the semantic information from text descriptions, while using Message Passing Neural Networks to capture the structural information. StATIK achieves state of the art results on three challenging inductive baselines. We further analyze our hybrid model through detailed ablation studies.
@inproceedings{markowitz-etal-2022-statik, author = {Markowitz, Elan and Balasubramanian, Keshav and Mirtaheri, Mehrnoosh and Annavaram, Murali and Galstyan, Aram and Ver Steeg, Greg}, title = {{S}t{ATIK}: Structure and Text for Inductive Knowledge Graph Completion}, booktitle = {Findings of the {A}ssociation for {C}omputational {L}inguistics: {NAACL} 2022}, year = {2022}, address = {Seattle, United States}, doi = {10.18653/v1/2022.findings-naacl.46}, editor = {Carpuat, Marine and de Marneffe, Marie-Catherine and Meza Ruiz, Ivan Vladimir}, month = {July}, pages = {604--615}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2022.findings-naacl.46/} } - Knowledge Graph Embedding for Link Prediction: A Comparative AnalysisACM Transactions on Knowledge Discovery from Data, 15(2), pp. 14:1–14:49, 2021.
Knowledge Graphs (KGs) have found many applications in industry and academic settings, which in turn, have motivated considerable research efforts towards large-scale information extraction from a variety of sources. Despite such efforts, it is well known that even state-of-the-art KGs suffer from incompleteness. Link Prediction (LP), the task of predicting missing facts among entities already a KG, is a promising and widely studied task aimed at addressing KG incompleteness. Among the recent LP techniques, those based on KG embeddings have achieved very promising performances in some benchmarks. Despite the fast growing literature in the subject, insufficient attention has been paid to the effect of the various design choices in those methods. Moreover, the standard practice in this area is to report accuracy by aggregating over a large number of test facts in which some entities are over-represented; this allows LP methods to exhibit good performance by just attending to structural properties that include such entities, while ignoring the remaining majority of the KG. This analysis provides a comprehensive comparison of embedding-based LP methods, extending the dimensions of analysis beyond what is commonly available in the literature. We experimentally compare effectiveness and efficiency of 16 state-of-the-art methods, consider a rule-based baseline, and report detailed analysis over the most popular benchmarks in the literature.
@article{rossi-etal-2021-knowledge, author = {Rossi, Andrea and Barbosa, Denilson and Firmani, Donatella and Matinata, Antonio and Merialdo, Paolo}, title = {Knowledge Graph Embedding for Link Prediction: A Comparative Analysis}, journal = {ACM Transactions on Knowledge Discovery from Data}, year = {2021}, archiveprefix = {arXiv}, doi = {10.1145/3424672}, eprint = {2002.00819}, number = {2}, pages = {14:1--14:49}, publisher = {Association for Computing Machinery}, volume = {15} } - Open-World Knowledge Graph CompletionIn Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18), 2018.
Knowledge Graphs (KGs) have been applied to many tasks including Web search, link prediction, recommendation, natural language processing, and entity linking. However, most KGs are far from complete and are growing at a rapid pace. To address these problems, Knowledge Graph Completion (KGC) has been proposed to improve KGs by filling in its missing connections. Unlike existing methods which hold a closed-world assumption, i.e., where KGs are fixed and new entities cannot be easily added, in the present work we relax this assumption and propose a new open-world KGC task. As a first attempt to solve this task we introduce an open-world KGC model called ConMask. This model learns embeddings of the entity's name and parts of its text-description to connect unseen entities to the KG. To mitigate the presence of noisy text descriptions, ConMask uses a relationship-dependent content masking to extract relevant snippets and then trains a fully convolutional neural network to fuse the extracted snippets with entities in the KG. Experiments on large data sets, both old and new, show that ConMask performs well in the open-world KGC task and even outperforms existing KGC models on the standard closed-world KGC task.
@inproceedings{shi-weninger-2018-open, author = {Shi, Baoxu and Weninger, Tim}, title = {Open-World Knowledge Graph Completion}, booktitle = {Proceedings of the Thirty-Second {AAAI} Conference on Artificial Intelligence ({AAAI}-18)}, year = {2018}, archiveprefix = {arXiv}, doi = {10.1609/aaai.v32i1.11535}, eprint = {1711.03438}, publisher = {AAAI Press}, url = {https://ojs.aaai.org/index.php/AAAI/article/view/11535} }
Large Language Models
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsIn Advances in Neural Information Processing Systems 35 (NeurIPS 2022), pp. 24824–24837, 2022.
We explore how generating a chain of thought -- a series of intermediate reasoning steps -- significantly improves the ability of large language models to perform complex reasoning. In particular, we show how such reasoning abilities emerge naturally in sufficiently large language models via a simple method called chain of thought prompting, where a few chain of thought demonstrations are provided as exemplars in prompting. Experiments on three large language models show that chain of thought prompting improves performance on a range of arithmetic, commonsense, and symbolic reasoning tasks. The empirical gains can be striking. For instance, prompting a 540B-parameter language model with just eight chain of thought exemplars achieves state of the art accuracy on the GSM8K benchmark of math word problems, surpassing even finetuned GPT-3 with a verifier.
@inproceedings{wei-etal-2022-chain, author = {Wei, Jason and Wang, Xuezhi and Schuurmans, Dale and Bosma, Maarten and Ichter, Brian and Xia, Fei and Chi, Ed and Le, Quoc V. and Zhou, Denny}, title = {Chain-of-Thought Prompting Elicits Reasoning in Large Language Models}, booktitle = {Advances in Neural Information Processing Systems 35 ({NeurIPS} 2022)}, year = {2022}, archiveprefix = {arXiv}, doi = {10.52202/068431-1800}, eprint = {2201.11903}, pages = {24824--24837}, url = {https://proceedings.neurips.cc/paper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html} } - Large Language Models Struggle to Learn Long-Tail KnowledgeIn Proceedings of the 40th International Conference on Machine Learning (ICML), pp. 15696–15707, 2023.
The Internet contains a wealth of knowledge -- from the birthdays of historical figures to tutorials on how to code -- all of which may be learned by language models. However, while certain pieces of information are ubiquitous on the web, others appear extremely rarely. In this paper, we study the relationship between the knowledge memorized by large language models and the information in pre-training datasets scraped from the web. In particular, we show that a language model's ability to answer a fact-based question relates to how many documents associated with that question were seen during pre-training. We identify these relevant documents by entity linking pre-training datasets and counting documents that contain the same entities as a given question-answer pair. Our results demonstrate strong correlational and causal relationships between accuracy and relevant document count for numerous question answering datasets (e.g., TriviaQA), pre-training corpora (e.g., ROOTS), and model sizes (e.g., 176B parameters). Moreover, while larger models are better at learning long-tail knowledge, we estimate that today's models must be scaled by many orders of magnitude to reach competitive QA performance on questions with little support in the pre-training data. Finally, we show that retrieval-augmentation can reduce the dependence on relevant pre-training information, presenting a promising approach for capturing the long-tail.
@inproceedings{kandpal-etal-2023-large, author = {Kandpal, Nikhil and Deng, Haikang and Roberts, Adam and Wallace, Eric and Raffel, Colin}, title = {Large Language Models Struggle to Learn Long-Tail Knowledge}, booktitle = {Proceedings of the 40th International Conference on Machine Learning ({ICML})}, year = {2023}, archiveprefix = {arXiv}, eprint = {2211.08411}, pages = {15696--15707}, publisher = {PMLR}, series = {Proceedings of Machine Learning Research}, url = {https://proceedings.mlr.press/v202/kandpal23a.html}, volume = {202} } - FactKG: Fact Verification via Reasoning on Knowledge GraphsIn Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16190–16206, 2023.
In real world applications, knowledge graphs (KG) are widely used in various domains (e.g. medical applications and dialogue agents). However, for fact verification, KGs have not been adequately utilized as a knowledge source. KGs can be a valuable knowledge source in fact verification due to their reliability and broad applicability. A KG consists of nodes and edges which makes it clear how concepts are linked together, allowing machines to reason over chains of topics. However, there are many challenges in understanding how these machine-readable concepts map to information in text. To enable the community to better use KGs, we introduce a new dataset, FactKG: Fact Verificationvia Reasoning on Knowledge Graphs. It consists of 108k natural language claims with five types of reasoning: One-hop, Conjunction, Existence, Multi-hop, and Negation. Furthermore, FactKG contains various linguistic patterns, including colloquial style claims as well as written style claims to increase practicality. Lastly, we develop a baseline approach and analyze FactKG over these reasoning types. We believe FactKG can advance both reliability and practicality in KG-based fact verification.
@inproceedings{kim-etal-2023-factkg, author = {Kim, Jiho and Park, Sungjin and Kwon, Yeonsu and Jo, Yohan and Thorne, James and Choi, Edward}, title = {{F}act{KG}: Fact Verification via Reasoning on Knowledge Graphs}, booktitle = {Proceedings of the 61st Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)}, year = {2023}, address = {Toronto, Canada}, doi = {10.18653/v1/2023.acl-long.895}, editor = {Rogers, Anna and Boyd-Graber, Jordan and Okazaki, Naoaki}, month = {July}, pages = {16190--16206}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2023.acl-long.895/} } - Reasoning on Graphs: Faithful and Interpretable Large Language Model ReasoningIn International Conference on Learning Representations (ICLR), 2024.
Large language models (LLMs) have demonstrated impressive reasoning abilities in complex tasks. However, they lack up-to-date knowledge and experience hallucinations during reasoning, which can lead to incorrect reasoning processes and diminish their performance and trustworthiness. Knowledge graphs (KGs), which capture vast amounts of facts in a structured format, offer a reliable source of knowledge for reasoning. Nevertheless, existing KG-based LLM reasoning methods only treat KGs as factual knowledge bases and overlook the importance of their structural information for reasoning. In this paper, we propose a novel method called reasoning on graphs (RoG) that synergizes LLMs with KGs to enable faithful and interpretable reasoning. Specifically, we present a planning-retrieval-reasoning framework, where RoG first generates relation paths grounded by KGs as faithful plans. These plans are then used to retrieve valid reasoning paths from the KGs for LLMs to conduct faithful reasoning. Furthermore, RoG not only distills knowledge from KGs to improve the reasoning ability of LLMs through training but also allows seamless integration with any arbitrary LLMs during inference. Extensive experiments on two benchmark KGQA datasets demonstrate that RoG achieves state-of-the-art performance on KG reasoning tasks and generates faithful and interpretable reasoning results.
@inproceedings{luo-etal-2024-reasoning, author = {Luo, Linhao and Li, Yuan-Fang and Haffari, Gholamreza and Pan, Shirui}, title = {Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning}, booktitle = {International Conference on Learning Representations ({ICLR})}, year = {2024}, archiveprefix = {arXiv}, eprint = {2310.01061}, url = {https://openreview.net/forum?id=ZGNWW7xZ6Q} }