KGCQual: An Interpretable Framework for Evaluating the Knowledge Graph Construction Quality from Text
2026-07-11 • Artificial Intelligence
Artificial IntelligenceComputation and Language
AI summaryⓘ
The authors created a new way to check how good automatically built knowledge graphs are by comparing them to an ideal version based on the original text. Their method looks at both the key entities and the relationships between them, including handling some language details like negation. They tested this on several existing systems and found that their metric catches errors and missing information better than current methods. They also showed that their scores relate to how well the graphs work in practical tasks like link prediction.
Knowledge GraphTriple ExtractionEntity ResolutionPredicate RelationsLexical SimilarityDependency ParsingNegation HandlingLink PredictionEvaluation MetricSemantic Faithfulness
Authors
Nipun Misra, Vikranth Udandarao, Aanchal Gupta, Yogender Kumar, Manuj Mukherjee, Raghava Mutharaju
Abstract
Knowledge Graphs (KGs) are increasingly constructed through automated extraction pipelines; however, such systems often introduce spurious or incomplete triples, which degrade downstream performance. Existing evaluation practices rely heavily on task-specific metrics or small-scale manual verification, offering limited insight into the structural and semantic fidelity of extracted graphs. We propose a novel, interpretable metric for intrinsic KG quality assessment that measures how closely an automatically extracted graph approximates an "ideal" graph capturing the key noun phrases, predicate relations, and basic linguistic phenomena such as negation expressed in the source text. Our framework integrates two complementary components: (1) an entity-level assessment that evaluates completeness, resolution quality, and connectivity, and (2) a relation-level assessment that judges predicate preservation and multiplicity using lexical similarity, dependency-parse alignment, and light-weight negation handling to ensure semantic faithfulness. We evaluate our metric across multiple state-of-the-art triple extraction systems and datasets, including WebNLG, TinyButMighty, and BenchIE, demonstrating that it reliably identifies omissions, redundancy, and structural deviations that existing metrics overlook. Our work offers a scalable, model-agnostic, and interpretable framework for comparing automated KG construction methods and provides a foundation for standardised evaluation. We further validate the metric through an ablation study isolating noun and verb components, and a downstream evaluation showing that KGCQual scores correlate significantly with link prediction performance on the same extracted KGs. The code repository is available at https://github.com/kracr/kg-quality-metric.