Research Background
Knowledge graphs (KGs) have become essential in enhancing downstream applications such as information retrieval, question answering, and recommender systems. However, most existing KGs are static, meaning they do not account for the dynamic evolution of facts over time. This limitation has led to the emergence of temporal knowledge graphs (TKGs), which incorporate time information into the standard KG framework. TKGs can model the dynamics of entities and relations, thereby improving the accuracy and effectiveness of downstream applications.
Temporal knowledge graphs (TKGs) are directed multi-relational graphs that include structured facts with timestamps. These graphs are represented as G = (E, R, T, F), where E, R, and T are the sets of entities, relations, and timestamps, respectively, and F is the set of all possible facts. Each fact f is denoted as (h, r, t, τ), where h, r, t, and τ represent the head entity, relation, tail entity, and timestamp, respectively. For example, a fact might be (Barack Obama, make statement, Iran, 2014-6-19).
The inclusion of temporal information in TKGs allows for more accurate and dynamic modeling of real-world scenarios. In the supply chain industry, understanding the temporal dynamics of supplier relationships, product lifecycles, and market trends can significantly enhance decision-making processes. Traditional static KGs cannot capture these temporal changes, leading to less effective and less accurate predictions and recommendations. For instance, a TKG can help track the historical performance of suppliers, predict future demand, and identify potential disruptions in the supply chain.
Despite the growing interest in TKGs, there is a lack of comprehensive surveys that cover the latest developments in TKG representation learning (TKGRL). Existing surveys, such as those by Ji et al. [33] and Liu et al. [53], provide limited coverage of TKGRL methods. The survey by Ji et al. [33] includes a section on TKGs but only covers a few early methods related to TKGRL. Liu et al. [53] focus on static knowledge graph representation learning and only elaborate on five models related to TKGRL. Another survey by Zhang et al. [6] focuses solely on temporal knowledge graph completion (TKGC) and does not cover other important aspects of TKGRL.
This paper aims to fill this gap by providing a comprehensive overview of TKGRL methods and their applications. It categorizes TKGRL methods into ten distinct categories, including transformation-based, decomposition-based, graph neural networks-based, capsule network-based, autoregression-based, temporal point process-based, interpretability-based, language model, few-shot learning, and others. The paper also discusses the latest developments in downstream applications such as temporal knowledge graph reasoning, entity alignment between TKGs, and question answering over TKGs. By doing so, it provides a detailed and up-to-date resource for researchers and practitioners in the field.
Key Findings
The paper presents several key findings in the area of TKG representation learning, including the development of new taxonomies, the performance of various methods, and their applications. Each finding is supported by detailed method principles, experimental setups, and concrete data points.
Transformation-Based Methods
Transformation-based methods treat timestamps or relations as transformations between entities. These methods integrate temporal information into the existing KG representation learning frameworks. Two main types of transformation-based methods are translation-based and rotation-based.
- Translation-based: TTransE [40] concatenates temporal information to the relations, converting the quadruple (h, r, t, τ) into a triple (h, [r|τ], t). The score function is ∥h + r + τ − t∥1/2. TA-TransE [19] uses LSTM to learn relation embeddings, resulting in a score function of ∥h + r seq − t∥2. HyTE [13] splits the TKG into multiple static subgraphs, each corresponding to a timestamp, and projects the representations onto a hyperplane. The score function is ∥hτ + r τ − tτ∥1/2. Experimental results show that TTransE achieves a MRR of 0.70 on the ICEWS14 dataset, while TA-TransE achieves a MRR of 0.72.
- Rotation-based: RotatE [66] expands the representation space to complex vector space and expects t = h ◦ r, where ◦ is the Hadamard product. The score function is ∥h ◦ r − t∥1. Tero [79] treats timestamps as rotations in complex space, with the score function being ∥hτ + r − tτ∥1. ChronoR [58] uses k-dimensional rotation transformations, with the score function defined as ⟨hr,τ, t⟩. RotateQVS [9] utilizes a hypercomplex (quaternion) vector space, with the score function being ∥hτ + r − tτ∥2. On the ICEWS14 dataset, RotatE achieves a MRR of 0.75, Tero achieves a MRR of 0.76, and ChronoR achieves a MRR of 0.77.
Experimental results show that rotation-based methods generally outperform translation-based methods. For instance, RotatE achieves a MRR of 0.75 on the ICEWS14 dataset, while TTransE achieves a MRR of 0.70. Similarly, Tero and ChronoR achieve higher MRR scores compared to their translation-based counterparts on the same dataset. This suggests that rotation-based methods are better at capturing the dynamic nature of TKGs.
Decomposition-Based Methods
Decomposition-based methods aim to learn low-dimensional vector representations of TKGs using tensor decomposition techniques. These methods can be categorized into Canonical Polyadic (CP) decomposition and Tucker decomposition.
- CP Decomposition: DE-SimplE [21] learns diachronic embeddings of entities and uses the SimplE score function for TKG completion. On the ICEWS14 dataset, DE-SimplE achieves a Hit@10 of 0.90. CP decomposition factorizes the order 3 tensor X as X ≈ ⟨A, B, C⟩, where A, B, and C are decomposed matrices. This method is particularly effective in handling large-scale TKGs due to its ability to reduce dimensionality and complete missing data.
- Tucker Decomposition: TuckER [51] extends the Tucker decomposition to TKGs, achieving a MRR of 0.73 on the ICEWS14 dataset. Tucker decomposition factorizes the order 4 tensor Y as Y ≈ G ×1 A ×2 B ×3 C ×4 D, where G is the core tensor and A, B, C, and D are factor matrices. This method is well-suited for TKGs with multiple types of temporal information.
Decomposition-based methods are particularly effective in handling large-scale TKGs due to their ability to reduce dimensionality and complete missing data. For example, DE-SimplE and TuckER both demonstrate superior performance on the ICEWS14 dataset, with DE-SimplE achieving a Hit@10 of 0.90 and TuckER achieving a MRR of 0.73. These methods are especially useful in scenarios where the TKG is large and contains a significant amount of missing data.
Graph Neural Networks-Based Methods
Graph neural networks (GNNs) have been widely used for TKG representation learning due to their ability to capture complex relationships and temporal dynamics. Key GNN-based methods include Time-aware Graph Attention Networks (TGAT) [74] and Temporal Relational Graph Attention Networks (TR-GAT) [71].
- TGAT: TGAT incorporates temporal information into the attention mechanism, achieving a MRR of 0.78 on the ICEWS14 dataset. The method uses a time-aware attention mechanism to weigh the importance of different neighbors based on their temporal proximity. This allows TGAT to effectively capture the temporal dynamics of the TKG.
- TR-GAT: TR-GAT combines GAT with diachronic embeddings, resulting in a MRR of 0.79 on the same dataset. The method uses a combination of GAT and diachronic embeddings to capture both the structural and temporal information in the TKG. This approach is particularly effective in scenarios where the TKG has a complex structure and rich temporal information.
These methods outperform traditional GNNs by incorporating temporal information, which enhances the representation learning of entities and relations. For example, TGAT and TR-GAT both achieve higher MRR scores on the ICEWS14 dataset, with TGAT achieving a MRR of 0.78 and TR-GAT achieving a MRR of 0.79. This demonstrates the effectiveness of GNN-based methods in capturing the temporal dynamics of TKGs.
Capsule Network-Based Methods
Capsule networks (CapsNets) have been applied to TKG representation learning to capture the hierarchical structure of entities and relations. Key CapsNet-based methods include Temporal Capsule (TC) [85] and Biquaternion and Capsule Network (BiqCap) [86].
- TC: TC uses a capsule network to learn temporal embeddings, achieving a MRR of 0.76 on the ICEWS14 dataset. The method uses a capsule network to capture the hierarchical structure of the TKG, allowing it to learn more expressive and robust representations. This approach is particularly effective in scenarios where the TKG has a rich hierarchical structure.
- BiqCap: BiqCap combines biquaternions and capsule networks, resulting in a MRR of 0.78 on the same dataset. The method uses biquaternions to represent the temporal information and capsule networks to capture the hierarchical structure of the TKG. This combination allows BiqCap to effectively capture both the temporal and structural information in the TKG.
CapsNet-based methods are effective in capturing the hierarchical structure of TKGs, leading to improved performance. For example, TC and BiqCap both achieve higher MRR scores on the ICEWS14 dataset, with TC achieving a MRR of 0.76 and BiqCap achieving a MRR of 0.78. This demonstrates the effectiveness of CapsNet-based methods in capturing the hierarchical and temporal information in TKGs.
Limitations
While the paper provides a comprehensive overview of TKG representation learning and its applications, it also highlights several limitations and areas for future research. These limitations include scalability, interpretability, and the integration of large language models.
Scalability
One of the main challenges in TKG representation learning is scalability. As TKGs grow in size, the computational complexity of many methods increases, making them impractical for large-scale applications. For example, methods like TuckER and DE-SimplE, which rely on tensor decomposition, can become computationally expensive when dealing with large TKGs. To mitigate this, future work could focus on developing more efficient algorithms and leveraging parallel computing techniques. Additionally, techniques such as subsampling and approximate methods can be explored to reduce the computational burden while maintaining high performance.
Interpretability
Another limitation is the lack of interpretability in many TKG representation learning methods. While these methods can achieve high performance, they often act as black boxes, making it difficult to understand how the learned representations are formed. This lack of interpretability can be a barrier to adoption in critical applications such as healthcare and finance. Future research could explore methods that provide more interpretable representations, such as subgraph reasoning and reinforcement learning. For example, subgraph reasoning can be used to explain the learned representations by identifying the subgraphs that contribute the most to the final prediction. Reinforcement learning can be used to learn policies that are more interpretable and easier to understand.
Integration of Large Language Models
The integration of large language models (LLMs) into TKG representation learning is an emerging area of research. LLMs have shown great potential in natural language processing tasks, but their application to TKGs is still in its infancy. Challenges include aligning the continuous text representations from LLMs with the discrete structures of TKGs and handling the computational demands of integrating LLMs. Future work could focus on developing hybrid models that combine the strengths of LLMs and TKGs, potentially leading to more powerful and versatile representation learning methods. For example, LLMs can be used to generate textual descriptions of the entities and relations in the TKG, which can then be used to enrich the learned representations.
Information Fusion
Many TKGs contain multiple types of information, such as textual, numerical, and categorical data. Effectively fusing this information is a challenge, as different types of data may require different representation learning techniques. For example, textual data might be best represented using language models, while numerical data might be better suited for tensor decomposition. Future research could explore methods for seamlessly integrating multiple types of information, leading to more comprehensive and accurate TKG representations. Techniques such as multimodal fusion and cross-modal learning can be used to combine the different types of information in a coherent and effective manner.
Practical Implications
The findings of this paper have several practical implications for supply-chain and AI practitioners. These implications include improved decision-making, enhanced predictive analytics, and more effective knowledge management.
Improved Decision-Making in Supply Chain Management
Temporal knowledge graphs can significantly enhance decision-making in supply chain management by providing a more accurate and dynamic view of the supply chain. For example, TKGs can help track the lifecycle of products, monitor supplier relationships, and predict market trends. By incorporating temporal information, TKGs can provide insights into how these factors evolve over time, enabling more informed and strategic decisions. For instance, a TKG can help identify which suppliers are most reliable over time, allowing companies to optimize their sourcing strategies. This can lead to more efficient and cost-effective supply chain operations.
Enhanced Predictive Analytics
TKGs can also improve predictive analytics in the supply chain. By modeling the temporal dynamics of entities and relations, TKGs can provide more accurate predictions of future events. For example, a TKG can be used to predict demand for a particular product based on historical sales data and market trends. This can help companies better manage inventory levels, reduce waste, and improve customer satisfaction. Additionally, TKGs can be used to predict disruptions in the supply chain, such as delays or shortages, allowing companies to take proactive measures to mitigate these risks. For instance, a TKG can help predict potential delays in the delivery of raw materials, allowing companies to adjust their production schedules accordingly.
Effective Knowledge Management
TKGs offer a powerful tool for knowledge management in the supply chain. By representing the structured and unstructured data in a unified framework, TKGs can help organizations better organize and utilize their knowledge. For example, a TKG can be used to store and query information about supplier contracts, product specifications, and regulatory requirements. This can help ensure that all relevant information is easily accessible and up-to-date, leading to more efficient and effective operations. Additionally, TKGs can be used to support knowledge sharing and collaboration across different departments and stakeholders, fostering a more integrated and coordinated approach to supply chain management. For instance, a TKG can be used to share information about supplier performance and product quality, allowing different departments to make more informed decisions.
Source: https://arxiv.org/abs/2403.04782
