Australian National University

Vaibhav Khobragade

#### **Exploring the Potential of Temporal Feature-Logic Embedding (TFLEX) in Complex Query Resolution**

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*H6ME71WASxb20ZcS" />
<figcaption>Weaving Time in Knowledge Graphs by ChatGPT
(DALLE2).</figcaption>
</figure>

### Author

· Vaibhav Khobragade (**ORCID:**
[0009--0009--8807--5982](https://orcid.org/0009-0009-8807-5982))

### **Introduction**

Artificial intelligence (AI) and knowledge representation in the field
of temporal knowledge graphs are rapidly gaining interest. Current
challenges include developing models and frameworks that can effectively
form reasoning mechanisms. **Temporal knowledge graphs (TKGs)** refer to
a specific group of knowledge representation systems that besides
concepts, entities, events, and relationships, capture time information
(Li et al., 2021). These graphs give so much useful material in the
processes of forecasting, organizing plans, and decision-making.

TKGs are active databases that not only drive the information but also,
the context of true incidents when they happen and that makes them
crucial in the prediction of forthcoming trends and the understanding of
complex sequences of events.

Innovative techniques termed **Temporal Feature-Logic Embedding
framework (TFLEX)** were introduced, which is purposefully built to deal
with temporal and logical issues especially to handle queries that have
elements of time information, that classic static KGs would fail to
deal with.

***Let's build a fundamental concept before we delve into TFLEX:***

### **What is a Knowledge Graph?**

A knowledge graph is a way of storing information that uses a
graph-structured model. In this model, entities are represented as nodes
and the relationship between them are represented as edges, connected to
the nodes. This structure depicts complex interrelationships
within data.

For example: An entity "Inception" is connected to an entity "Leonardo
DiCaprio" with the relationship "acted in" and to "Christopher Nolan"
entity with the relationship "directed by".

<figure>
<img src="https://cdn-images-1.medium.com/max/940/0*Xa_zd0gENuy-GKlf" />
<figcaption>Figure1: Knowledge graph (entities and relationship
example)</figcaption>
</figure>

This structure makes it easy to understand the connections between
pieces of information and to use them for reasoning, searching, and
knowledge discovery.

### What is a Temporal Knowledge Graph?

A Temporal Knowledge Graph (TKG) enhances traditional static knowledge
graphs by incorporating time data into the structure. This is achieved
by adding a timestamp to the usual triple format, resulting in elements
described as \<head, relation, tail, timestamp\>.

For instance, consider the data point \<Bill Gates, makes a visit,
India, 2024--04--23\>; these records not only the event and the entities
involved but also the exact date it occurred.

### **TFLEX**

Recent research on **Temporal Knowledge Graph Completion (TKGC)** aims
to infer new facts in TKGs that focus on temporal link prediction, which
is simply a single hop. It tries to predict if two nodes that are not
currently connected will form a single direct link in the future based
on the history of the network and its evolution over time. Some TKGC
methods like **timestamp-based transformation, dynamic embedding, and
Markov process** **model** work by performing multi-hop reasoning.
However, none of them could answer logical questions involving
conjunction, negation, and disjunction operation

For instance, find all scientists who have published their research in
the field of Artificial Intelligence (Conjunction), have never received
an award (Negation), or have collaborated with certain universities
before 2020 (Before and Disjunction). In this article, we focused on
multi-hop logical reasoning to answer the temporal complex queries. To
understand this method, we should know what temporal complex
queries are.

### **Temporal Complex Queries**

This type of query for Temporal Knowledge Graphs (TKGs) specifically
includes both entity and temporal elements. These queries are complex
because they require reasoning across multiple connections (multi-hop)
and involve both entities and the times associated with their
relationships. For example, a query might ask which events a person
attended before a certain year and who else was present at those events.
This type of query goes beyond simple fact retrieval, incorporating
logical and temporal operators to extract structured information across
time and entities as discussed above.

In a mathematical way, a temporal complex query can be represented as:

<figure>
<img src="https://cdn-images-1.medium.com/max/940/0*L6jPwbDy21yoABqZ" />
</figure>

The above query will be discussed in more detail in the paragraphs
that follow.

### **Data Generation Process**

Researchers formulated the above temporal complex queries that involve
both entities and their temporal relationships and then applied these
queries to three different popular TKGs to generate three datasets
(ICEWS14, ICEWS05--15, and GDELT-500) for further training and testing
this model.

Multi-hop logical reasoning over knowledge graphs is a complex process
that involves answering queries by traversing multiple relationships (or
"hops") between entities in a knowledge graph. In simpler terms, it's
like solving a puzzle where each piece (or hop) brings you closer to the
complete answer by following a trail of interconnected data points.

<figure>
<img src="https://cdn-images-1.medium.com/max/940/0*BCcPSrLzstsI3hVF" />
<figcaption>Figure 2: An example of a multi-hop temporal complex query
and its processing network is as follows: “Which countries did Xi
Jinping visit but Barack Obama did not visit during François Hollande’s
presidency of France?” Entity sets (blue circles), timestamp sets (green
triangles), entity set projections (blue arrows), and logical operators
(red rectangles) are represented in the computation graph. (Lin et al.,
2022) Article link: <a
href="https://arxiv.org/abs/2205.14307">https://arxiv.org/abs/2205.14307</a></figcaption>
</figure>

1. **Entity and Timestamp Representation**:

- **Entity Nodes (Circles)**: Entities such as "Xi Jinping" and "Barack
 Obama" are represented as blue circles.
- **Timestamp Nodes (Green Triangles)**: The period of François
 Hollande's presidency is represented as a green triangle, indicating
 the specific timeframe relevant to the query.

**2. Entity and Time Set Projections (Arrows)**:

- **Blue Arrows**: These represent entity set projections. For instance,
 an arrow from the entity node "Xi Jinping" to a logical operation
 indicates that the action of visiting countries is being queried
 specifically for Xi Jinping.
- **Green Arrows**: These represent time set projections. They connect
 the timestamp node of Hollande's presidency to other operations,
 applying the time filter to the activities of the entities involved.

**3. Logical Operations (Red Rectangles)**:

- **Visit Operations**: Separate logical operations check which
 countries each leader visited during Hollande's presidency. These
 operations are linked from the entity nodes (Xi Jinping, Barack Obama)
 through blue arrows, indicating entity-focused actions within the
 specified time, linked through green arrows.
- **AND NOT Operation**: This logical operator compares the outputs from
 the Xi and Obama visit operations. It filters out any overlaps, thus
 excluding countries visited by Obama from Xi's list.

**4. Final Output (Node)**:

- The output node at the end of the computation graph provides the
 answer to the query. It lists countries visited by Xi Jinping but not
 by Barack Obama during the defined timeframe. This is achieved after
 processing through the logical AND and NOT operation, which ensures
 only Xi's unique visits are counted.

<figure>
<img src="https://cdn-images-1.medium.com/max/940/0*xEF6vn1R_85HA6xH" />
<figcaption>Figure 3: The computation of time part in temporal operators
Before and After. (Lin et al., 2022) Article link: <a
href="https://arxiv.org/abs/2205.14307">https://arxiv.org/abs/2205.14307</a></figcaption>
</figure>

The computation graph in Figure 3 represents how these time adjustments
are made within the embeddings. It shows that only the time components
are modified in the highlighted colors leaving the entity-related parts
unchanged in faded gray color. It focuses solely on the temporal shift
implied by the query which is important for reasoning over knowledge
that changes over time.

### **Model Prediction in temporal context before and after the event**

<figure>
<img src="https://cdn-images-1.medium.com/max/940/0*gCoU-bQvkz2ubPny" />
<figcaption>Figure 4: Score distributions of Pt,
bPt and aPt, where s is
Citizen (India), r is Criticize or
denounce, o is Sadhu (India),
ground-truth answer of Pt is
2014–03–27. (Lin et al., 2022) Article link: <a
href="https://arxiv.org/abs/2205.14307">https://arxiv.org/abs/2205.14307</a></figcaption>
</figure>

- Pt is projection in time that shows the most likely time for the event
 to have occurred, which is aligned with ground truth date.
- bPt (Before Projection Time): bPt indicates time earlier than the peak
 for "Pt," as highlighted by the 'before' region in the graph.
- aPt(After Projection Time): aPt indicates time earlier than the peak
 for "Pt," as highlighted by the 'after' region in the graph.

<figure>
<img src="https://cdn-images-1.medium.com/max/940/0*UZSHCPLELDoiXMIs" />
<figcaption>Figure 5: Score distributions of Pt,
bPt and aPt, where s is
Citizen (India), r is Use
unconventional violence, o is Chief Secretary
Chandra, ground-truth answer of Pt is
2014–05–11. (Lin et al., 2022) Article link: <a
href="https://arxiv.org/abs/2205.14307">https://arxiv.org/abs/2205.14307</a></figcaption>
</figure>

Figure 4 & 5 uses the x-axis to detail the model's performance across
different types of queries, demonstrating how predictions align with
ground truth data over time. In simple words, the purpose of the above
visualization is to show the model's ability to predict when events
occur relative to a known point in time. ***"Pt"*** is the model's
direct prediction, while ***"bPt" and "aPt"*** show the model's
understanding of the temporal context before and after the event,
respectively.

### **Result**

Take this query example: Philippines denounced or criticized who on
2014--04--01?

<figure>
<img
src="https://cdn-images-1.medium.com/max/1004/0*Ja6buG4h-uwSHuA0" />
<figcaption>Table 1: Intermediate variable assignments and ranks for
example query. Correctness indicates whether the answer belongs to the
ground-truth set of answers.</figcaption>
</figure>

Taking another query example: On 2014--04--04, who consulted the man who
was appealed to or requested by the Head of Government (Latvia) on
2014--08--01?

<figure>
<img src="https://cdn-images-1.medium.com/max/977/0*SCTi1GRsq-phc64R" />
<figcaption>Table 2: Intermediate variable assignments and ranks for
example second query. Correctness indicates whether the answer belongs
to the ground-truth set of answers.</figcaption>
</figure>

In Table 2, the query "On 2014--04--04, who consulted the man who was
appealed to or requested by the Head of Government (Latvia) on
2014--08--01?" was taken and then TFLEX was used to execute a query and
get an answer. The model successfully classified the level of
difficulties of the answer type into Easy and Hard. It is also able to
classify which answer is correct and wrong. The table shows that TFLEX
gives simple answers "François Hollande", "Taavi Rõivas", and hard
answers "Andris Berzins" a higher ranking than incorrect answers "Angela
Merkel" and "Head of Government (Latvia)". This demonstrates how TFLEX
correctly determines the hard answer through sophisticated reasoning and
separates the correct responses from incorrect ones.

### **Industry Applications**

Temporal knowledge graphs (TKGs) have diverse applications:

1. **Recommendation Systems**: TKGs are used to model recommendation
 systems by analyzing users' past behaviours such as liking, buying,
 watching, reviewing, and so on. It can also offer more personalized
 and timely recommendations, such as suggesting products based on
 recent browsing and purchasing trends.
2. **Financial Application** In financial markets, TKGs can help in
 holding behaviours, trading behaviours, and financial events that
 aid in better investment decisions.
3. **Transportation**: In the field of transportation, temporal
 knowledge graphs can be used to improve traffic prediction models
 and enable intelligent transportation systems. (Yuan & Li, 2021)

### **Conclusion**

This post looks at how artificial intelligence is getting better by
using Temporal Knowledge Graphs (TKGs) and a special method called
Temporal Feature-Logic Embedding (TFLEX). TKGs are advanced versions of
knowledge graphs that include time information, which helps in
understanding and predicting complex patterns. TFLEX is designed to deal
with these time-related challenges and improves how we can answer
detailed questions about past and future events. This is useful in areas
like finance, transportation, and personalized recommendations, where
understanding time can make a big difference.

### **References**

- Lin, X., Xu, C., E, H., Su, F., Zhou, G., Hu, T., Li, N., Sun, M., &
 Luo, H. (2022, May 28). TFLEX: Temporal Feature-Logic Embedding
 Framework for Complex Reasoning over Temporal Knowledge Graph.
 arXiv.org. <https://doi.org/10.48550/arXiv.2205.14307>
- Li, M., Tong, P., Li, M., Jin, Z., Huang, J., & Hua, X. S. (2021, May
 18). Traffic Flow Prediction with Vehicle Trajectories. Proceedings of
 the . . . AAAI Conference on Artificial Intelligence.
 <https://doi.org/10.1609/aaai.v35i1.16104>
- Yuan, H., & Li, G. (2021, January 23). A Survey of Traffic Prediction:
 from Spatio-Temporal Data to Intelligent Transportation. Data Science
 and Engineering. <https://doi.org/10.1007/s41019-020-00151-z>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0da8483262f9){width="1"
height="1"}

Exploring the Potential of Temporal Feature-Logic Embedding (TFLEX) in Complex Query Resolution

Author · Vaibhav Khobragade (

 ORCID:

0009–0009–8807–5982)

 Introduction

Artificial intelligence (AI) and knowledge representation in the field of temporal knowledge graphs are rapidly gaining interest.

Harnessing Temporal Dynamics: Advanced Reasoning using Temporal Knowledge Graphs

Xuzeng He

Latest findings in multiple research directions for tackling reasoning
and common sense challenges

<figure>
<img src="https://cdn-images-1.medium.com/max/894/0*2_PaHcp10KVBvRZq" />
<figcaption>Source: Generated using DeviantArt’s DreamUp</figcaption>
</figure>

### Author:

- Xuzeng He (**ORCID:**
 [0009--0005--7317--7426](https://orcid.org/0009-0005-7317-7426))

Knowledge Graphs, such as Wikidata, contain rich relational information
between entities and have been widely used as a structured format for
storing and representing relational information. With the surge of Large
Language Models (LLMs) nowadays (e.g. ChatGPT), there is a rising trend
in the research of combining Knowledge Graphs with Large Language Models
in multiple ways to solve various kinds of problems.

In this post, we introduce some of the latest findings where the use of
Knowledge Graph is combined with Large Language Models.

### Chain of History (CoH) reasoning

Temporal Knowledge Graph (**TKG**) is a subset of Knowledge Graphs
having some directions and timestamps. This allows capturing how
relationships evolve over time or the order in which they occur. It
holds significant practical value across various applications. For
example, a TKG might show that "Albert Einstein" (entity) "published"
(relationship with time) the theory of relativity in "1905" (timestamp).
Most advanced research on TKGs mainly focuses on predicting future facts
based on given historical facts.

Recent supervised methods primarily rely on Graph Neural Networks
(**GNNs**) to capture structural information (e.g. entities and
relationships). They can learn representations for entities and
relationships by aggregating information from neighboring nodes in the
graph. This helps identify patterns and connections within the TKG.
While GNNs capture structure, they might not fully grasp the semantic
meaning of entities, relationships, and timestamps in a TKG. They focus
on learning patterns from labeled data, which might not translate well
to understanding the underlying meaning of the data. Fortunately, with
the assistance of LLMs, their capability in this direction has been
gradually explored. However, issues still remain. For example,
over-complicated historical information may confuse LLMs and make it
hard for them to give accurate answers. When you input a large paragraph
to LLMs, models may struggle to select useful information and could
possibly generate an answer irrelevant to the question.

To address the existing issues, Chain-of-History (**CoH**) reasoning
method for TKG prediction is proposed. Instead of providing LLMs with
all histories at once in the traditional approach, CoH provides LLMs
with high-order histories (e.g. information not directly related to the
query) step-by-step. Specifically, CoH adopts LLMs to explore important
high-order history chains step-by-step and reason the answers to the
query only based on inferred history chains in the last step.

**First-Order Histories**

- These are the **individual events** or pieces of information
 considered most relevant to answering the query.
- The LLM analyzes the provided information and identifies these key
 elements.

**Second-Order Histories**

- These are formed by **combining** the first-order histories **in a
 specific order**.
- They represent **sequences of events** that could have led to the
 current situation or answer the query. Second-order histories go
 beyond isolated events and capture the **causal relationships**
 between them.

This process continues where chains of first-order histories, forming
more intricate event sequences are formed for third-order, fourth-order
etc histories.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*ae8Y-1ocdu-eSB95" />
<figcaption>An illustration of a two-step CoH reasoning procedure. In
the first step, LLMs are provided with only first-order histories and
asked to infer the most important histories. In the second step, LLMs
are provided with second-order history chains based on the inferred
first-order histories and asked to infer possible answers to the given
query. Then the answers inferred by LLMs and graph-based models are
adaptively fused to make the final prediction. Note that this only
serves as a two-step reasoning example, more steps can be executed with
CoH. (Yuwei Xia et al., 2024) Article Link: <a
href="https://arxiv.org/abs/2402.14382">https://arxiv.org/abs/2402.14382</a></figcaption>
</figure>

Extensive experimental results demonstrate the superiority of CoH, and
its effectiveness in enhancing the performance of graph-based models for
TKG prediction.

### Right for Right Response

Knowledge Graph Question Answering (**KGQA**) methods seek to answer
Natural Language questions using the relational information stored in
Knowledge Graphs (KGs). With the recent advancements of Large Language
Models (LLMs) and their remarkable reasoning abilities, there is a
growing trend to leverage them for KGQA. However, existing methodologies
have only focused on answering factual questions, such as "In which city
was Simone Biles born". In this case, questions involving commonsense
reasoning that real-world users may pose more often, such as "Do I need
separate visas to see the 'Venus of Willendorf' and attend the Olympics
this summer", are left out.

A verifiable methodology for performing KGQA using LLMs called ***Right
for Right Reasons* (R3)** is proposed to solve this issue. R3 casts the
problem of common sense KGQA as a tree-structured search, in which every
reasoning step is either grounded on KG facts, or based on surfaced
commonsense axioms, a key property that makes the reasoning procedure
completely verifiable.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*0NK-rUxHLtFPlTjR" />
<figcaption>An example workflow of common sense KGQA procedure using R3.
(Armin Toroghi et al., 2024) Article Link: <a
href="https://arxiv.org/abs/2403.01390">https://arxiv.org/abs/2403.01390</a></figcaption>
</figure>

In this example workflow, R3 first identifies the anchor entities(The
central entities mentioned in the query that are most relevant to
finding the answer) of a query and obtains the relevant sub-graph for
these entities. Next, it surfaces a commonsense axiom from the LLM that
will guide the reasoning steps in that branch of the search tree. Then,
at each depth level of the tree, it checks whether the common- sense
axiom can be satisfied with the available KG facts, and if possible,
provides an answer grounded on a subset of them. If the available KG
triples are insufficient, by backward-chaining from the axiom, it
selects the next entity to obtain its relevant KG sub-graph to continue
the search. Each branch can continue up to a maximum depth, and if an
answer is not obtained at its bottom, a new commonsense axiom will be
surfaced which will guide search in a new branch until the search tree
reaches its maximum breadth.

Experimental evaluations across three different tasks --- question
answering, claim verification, and preference matching --- showcase R3
as a superior approach, outperforming existing methodologies and notably
reducing instances of hallucination and reasoning errors.

### KnowPhish

Phishing attacks are one of the most impactful types of scams, harming
both individuals and businesses. Many efforts have been made to counter
phishing attacks. Among these, Reference-Based Phishing Detectors
(**RBPDs**), which work by comparing the logos on a target webpage to a
known set of logos, have been established as the state-of-the-art
phishing detection paradigm, garnering considerable research attention.

Specifically, an RBPD consists of a Brand Knowledge Base (**BKB**)
containing brand information (e.g., the logos and legitimate domains of
brands) and a detector backbone that uses information from this BKB for
phishing detection. To detect if a web page is a phishing page, RBPDs
first identify the webpage's brand intention, that is, the brand that
the webpage presents itself as (e.g., a webpage with a Adobe logo and
appearance has the brand intention of Adobe). Then, if the webpage is
detected to have an intention of a certain brand, but its domain does
not match the legitimate domains of that brand, the webpage is
classified as a phishing attempt.

However, RBPD has its limitations as well, such as not being able to
identify textual Brand Intention. Therefore, a large-scale multimodal
BKB named KnowPhish containing rich logo variants, aliases, and domain
information of each brand was recently constructed and publicly
released, along with a Large Language Model (LLM)-based approach to
identify the brand intention of webpages in conjunction with the alias
information in KnowPhish. This approach can be directly integrated with
any standard visual RBPD, enabling it to detect brand intention through
both visual and textual modalities. The resulting multimodal phishing
detection approach, named KnowPhish Detector (**KPD**), is able to
detect phishing webpages with or without logos.

KnowPhish has been shown to dramatically improve the performance of
RBPDs. It is also significantly faster than DynaPhish, another detection
framework by 30 or more times, especially when using image-based RBPDs.
Moreover, incorporating the multimodal approach(incorporating various
data sources such as images, text, audio, etc) KPD, can substantially
boost the number of detected phishing webpages.

### Conclusion

This post explored how Knowledge Graphs (KGs) and Large Language Models
(LLMs) are being combined to address challenges in various tasks. The
key takeaways are:

- **Temporal Knowledge Graphs (TKGs) with LLMs:** The Chain-of-History
 (CoH) reasoning method offers a promising approach for TKG prediction.
 By breaking down complex historical information into steps for LLMs,
 CoH improves reasoning accuracy, especially when combined with
 graph-based models.
- **KG Question Answering (KGQA) with LLMs:** The Right for Right
 Reasons (R3) methodology tackles common-sense KGQA. R3 leverages LLMs
 to surface commonsense knowledge and guide the reasoning process. This
 verifiable approach reduces errors and hallucinations compared to
 existing methods.
- **Phishing Detection with LLMs:** The KnowPhish approach demonstrates
 the power of LLMs in enhancing Reference-Based Phishing Detectors
 (RBPDs). By incorporating a multimodal Brand Knowledge Base (BKB) and
 LLM-based brand intention identification, KnowPhish significantly
 improves phishing detection rates and speeds compared to traditional
 methods.

Overall, the combination of KGs and LLMs opens exciting possibilities
for tackling complex tasks that require reasoning and knowledge
representation. As research in this area progresses, we can expect even
more innovative applications that leverage the strengths of both these
technologies.

### References

- Xia, Y., Wang, D., Liu, Q., Wang, L., Wu, S., & Zhang, X. (2024).
 Enhancing Temporal Knowledge Graph Forecasting with Large Language
 Models via Chain-of-History Reasoning.
- Toroghi, A., Guo, W., Pour, M. M. A., & Sanner, S. (2024). Right for
 Right Reasons: Large Language Models for Verifiable Commonsense
 Knowledge Graph Question Answering.
- Li, Y., Huang, C., Deng, S., Lock, M. L., Cao, T., Oo, N., Hooi, B., &
 Lim, H. W. (2024). KnowPhish: Large Language Models Meet Multimodal
 Knowledge Graphs for Enhancing Reference-Based Phishing Detection.

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=660588951c16){width="1"
height="1"}

Latest findings in multiple research directions for tackling reasoning and common sense challenges Author: Xuzeng He (

 ORCID:

0009–0005–7317–7426) Knowledge Graphs, such as Wikidata, contain rich relational information between entities and have been widely used as a structured format for storing and representing relational information.

Rogue Scholar Beiträge

Harnessing Temporal Dynamics: Advanced Reasoning using Temporal Knowledge Graphs

Large Language Models and Knowledge Graphs: Ways to combine them