#### Exploring innovative Strategies in Combating Misinformation with Enhanced Multimodal Understanding

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*CY9hnrcFwoJYC9fh" />
<figcaption>Image generated in DALL-E 2.</figcaption>
</figure>

### Author

- Wenyi Pi (**ORCID**:
 [0009--0002--2884--2771](https://orcid.org/0009-0002-2884-2771))

### Introduction

Misinformation refers to false or inaccurate information that is often
given to someone in a deliberate attempt to make them believe something
that is not true. This has a significantly negative impact on public
health, political stability and social trust and harmony. To address
these challenges, large language models (LLMs) are employed to identify
inconsistency and verify the veracity of statements. While traditional
LLMs excel at text-based tasks, they often have trouble understanding
and processing other types of data. To bridge this gap, multimodal LLMs
capable of processing and generating multiple mode data have been
developed. By integrating text, images, audio and video, these models
are able to understand and analyse different forms of data
comprehensively. Multimodal LLMs have a range of applications including
enhancing digitising note taking by extracting text from digital images,
understanding complex things like parking signs, deciphering ancient
handwriting, analysing speech files for summarisation, transcription
etc. Researchers are actively exploring methods to verify multimodal
content and interpret images more effectively, aiming to use multimodal
LLMs to identify and interpret multimodal misinformation in a
cost-effective and accessible manner.

This article introduces two innovative methods utilising multimodal LLMs
and it explains them when addressing an important use case: analysis of
misinformation.

### Retrieval-augmented Generation (RAG)- based advanced reasoning techniques (RAGAR)

Retrieval-Augmented Generation (**RAG**) is an advanced method that
optimises the output of LLMs by incorporating information retrieval into
the generation process. For a more detailed introduction of RAG, please
find the article here [RAG: The next big thing
after LLMs?](https://medium.com/@researchgraph/rag-the-next-big-thing-after-llms-4b60708ab2b4)

Retrieval-augmented Generation (RAG)- based advanced reasoning
techniques (**RAGAR**) employ multimodal LLMs that process both textual
and visual information. The two specific RAGAR approaches are (1) Chain
of RAG (**CoRAG**) and (2) Tree of RAG (**ToRAG**). Both methods
incorporate multimodal LLMs and utilise multimodal RAG coupled with
reasoning in the fact-checking domain.

#### 1. Chain of RAG (CoRAG) {#chain-of-rag-corag}

CoRAG enhances the general RAG by employing a sequential strategy to
pose questions derived from a claim and its associated image context. An
LLM formulates an initial question to address a particular aspect of the
claim, which is then answered using a multimodal RAG system. After each
response, a follow-up check is used to assess the sufficiency of
information obtained. If further clarification is needed, the process
iteratively generates informed follow-up questions, leveraging the
accumulated answers. This iterative cycle is limited to a maximum of six
questions or terminates early when sufficient evidence has been gathered
to decisively evaluate the claim's veracity.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*QTLc62qwL_t42Cq9" />
<figcaption>Chain of RAG (CoRAG) pipeline. Source from <a
href="https://doi.org/10.48550/ARXIV.2404.12065">https://doi.org/10.48550/ARXIV.2404.12065</a></figcaption>
</figure>

#### 2. Tree of RAG (ToRAG) {#tree-of-rag-torag}

ToRAG expands on CoRAG by branching out the questioning process at each
reasoning step. When fed a claim and image context input from multimodal
claim generation, ToRAG generates three unique questions to fact-check
the claim and employs an evidence retriever to fetch and generate
answers. These answers pass into an elimination prompt from which a
selection process is conducted to choose the best question-answer pair
based on criteria like relevance, detail, answer confidence and is
retained as candidate evidence. This candidate evidence then informs the
next set of three follow-up questions, continuing the cycle. With each
iteration, only the most informative question-answer pair is kept,
creating a cumulative list of candidate evidence. The ToRAG cycle
terminates when it has either accumulated a sufficient evidence base to
assess the claim's veracity or when it reaches the limit of six
question-answer pairs. The final selected evidence informs the veracity
prediction module, which then produces a final verdict and explanation
for the claim.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*OmLyTpHZkIXBsYWd" />
<figcaption>Tree of RAG (ToRAG) pipeline. Source from <a
href="https://doi.org/10.48550/ARXIV.2404.12065">https://doi.org/10.48550/ARXIV.2404.12065</a></figcaption>
</figure>

#### Example of RAGAR in action

The image below shows a comparison between three different methods of
fact-checking claims to validate a political claim. The claim is about
"A politician claimed that when he was governor, they banned sanctuary
cities and implemented electronic verification".

1. **Baseline: SubQuestion Generation (no RAGAR)**

This approach generates subsidiary questions to validate the claim and
concludes with a 'failed' verdict prediction, indicating that the
evidence does not support the claim.

**2. RAGAR CoRAG**

CoRAG uses a sequence of questions, each building on the last, to
explore the claim in depth. It concludes that the actions mentioned in
the claim are 'supported' by the evidence.

**3. RAGAR ToRAG**

ToRAG employs a branching mechanism to generate multiple lines of
inquiry from the initial claim. It filters through the evidence to
select the most relevant question-answer pairs, concluding the claim as
'supported'.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*O5xyMQ92vQhv71Fk" />
<figcaption>An overview of the fact-checking pipeline contrasting the
baseline Sub-Question Generation approach from the RAGAR: Chain of RAG
and RAGAR: Tree of RAG approach. Source from <a
href="https://doi.org/10.48550/ARXIV.2404.12065">https://doi.org/10.48550/ARXIV.2404.12065</a></figcaption>
</figure>

### Multimodal Misinformation Interpretation and Distillation Reasoning (MMIDR)

The Multimodal Misinformation Interpretation and Distillation Reasoning
(MMIDR) framework is designed for detecting multimodal misinformation
and enhancing the ability of large language models (LLMs) to interpret
it. The framework aims to provide fluent and high-quality textual
explanations for the decision-making process of these models. The
process of the framework is described in subsequent images, firstly
enhancing the multimodal misinformation data through **data
augmentation** to accommodate the instruction-based format. Following
this, a labelling template is employed to prompt the teacher LLM for the
**extraction of rationales**. The final steps involve performing
**knowledge distillation** to transfer the capability of teacher LLMs in
explaining multimodal misinformation into student LLMs and then train
the student LLM on merging the original multimodal information with
these distilled rationales.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*pkD4V22cmgoxgzzK" />
<figcaption>Model Architecture Overview of MMIDR. Source from <a
href="https://doi.org/10.48550/arXiv.2403.14171">https://doi.org/10.48550/arXiv.2403.14171</a></figcaption>
</figure>

#### Data augmentation

This process involves converting image-text pairs into an appropriate
instruction following format. It contains two steps:

1. **Visual Information Processing**

Given visual content, utilise optical character recognition (OCR)
technology to discern and transmute characters in the image into an
editable text format. Simultaneously, employ image captioning technology
to generate descriptions for the image content.

**2. Evidence Retrieval**

This involves retrieval evidence from the internet, therefore the model
is powered with global knowledge. Through employing the google reverse
image which can produce a selection of images similar to the query and
we can retrieve textual evidence from these images including titles and
descriptions. Similarly, also retrieve the visual evidence from the
images using aforementioned visual information processing techniques.

Through data augmentation, we have a list of instances comprising the
textual content of the post, OCR-recognised text, text generated from
image caption with visual content, textual evidence and visual evidence.

#### Rationales Elicitation

Rationales elicitation refers to generating explanations for the given
multimodal information using a teacher LLM. A simple labelling prompt
template is used to prompt the teacher LLM to generate rationales.
Therefore the output encompasses both the rational and the ground-truth
label could be used to train the student LLM subsequently.

<figure>
<img src="https://cdn-images-1.medium.com/max/804/0*8q8OrdcUH2w5GBB3" />
<figcaption>Labelling Prompt Template. Source from <a
href="https://doi.org/10.48550/arXiv.2403.14171">https://doi.org/10.48550/arXiv.2403.14171</a></figcaption>
</figure>

#### Knowledge Distillation

Knowledge distillation involves fine-tuning the student LLM using the
teacher LLM to maximise the alignment of their output sequences,
therefore aligning the predictions of the student and the teacher. It is
similar to the process of teachers teaching their knowledge to their
students. LowRank Adaptation (LoRA) optimisation is used to facilitate
this tuning process, wherein the optimisation is solely applied to the
rank decomposition components of the Transformer layers. This process
uses an instruction template which is derived from the labelling prompt
template by removing label-related contents.

### Conclusion

In conclusion, This article presents two innovative methodologies for
applying multimodal large language models (LLMs) in misinformation
detection and interpretation, demonstrating their potential of combating
misinformation in a multimodal digital world. However, these methods
still have limitations. For the RAGAR approach (CoRAG and ToRAG), notice
from the experimental result that when feeding different promptings in
the same query multiple times, the search results may vary. This is
problematic since it affects the interpretation result and it is hard to
choose from these different results as the final results. While the
ToRAG approach addresses this by analysing answers to slightly varied
questions and selecting the best question answer pair that provides the
most information. For the MMIDR method, it has a performance gap between
the distilled student model and the teacher model. Future challenges
include improving the stability of RAGAR results and enhancing the
knowledge distillation process in MMIDR. Solving these issues will be
crucial for maximising the potential of multimodal LLMs in solving
misinformation problems effectively.

### References

- Khaliq, M. A., Chang, P., Ma, M., Pflugfelder, B., & Miletić, F.
 (2024). RAGAR, Your Falsehood RADAR: RAG-Augmented Reasoning for
 Political Fact-Checking using Multimodal Large Language Models
 (Version 1). arXiv. <https://doi.org/10.48550/ARXIV.2404.12065>
- Wang, L., Xu, X., Zhang, L., Lu, J., Xu, Y., Xu, H., Tang, M., &
 Zhang, C. (2024). MMIDR: Teaching Large Language Model to Interpret
 Multimodal Misinformation via Knowledge Distillation (Version 3).
 arXiv. <https://doi.org/10.48550/ARXIV.2403.14171>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a521f95cb5e){width="1"
height="1"}

RAGAR, Your Falsehood RADAR: RAG-Augmented Reasoning for Political Fact-Checking using Multimodal Large Language Models

MMIDR: Teaching Large Language Model to Interpret Multimodal Misinformation via Knowledge Distillation

Exploring innovative Strategies in Combating Misinformation with Enhanced Multimodal Understanding


 Author

Wenyi Pi (

 ORCID

: 0009–0002–2884–2771)

 Introduction

Misinformation refers to false or inaccurate information that is often given to someone in a deliberate attempt to make them believe something that is not true.

Multimodal Large Language Models for Misinformation Detection and Reasoning

Australian National University

Qingqin Fang

#### **Understanding the Balance between Internal Knowledge and External Sources**

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*-GBIM_Ece6r7flKJ" />
<figcaption>Image Generated by Google Gemini</figcaption>
</figure>

### Author

- Qingqin Fang (**ORCID:**
 [0009--0003--5348--4264](https://orcid.org/0009-0003-5348-4264))

### Introduction

Previous research often emphasized the limitations of LLM's information
acquisition pathways, focusing on enhancing its capabilities in this
regard. However, integrating contextual knowledge into LLM to keep it
abreast of current affairs and improve response accuracy may lead to
conflicts due to the richness of knowledge sources, known as *knowledge
conflicts*.

So, what is a knowledge conflict? Specifically, it refers to the
contradictory or inconsistent information obtained by LLM from different
sources, including internal knowledge or memory from LLM itself and
external knowledge retrieved through **RAG** (Retriever-Reader
Architecture) or a knowledge graph.

In this post, we primarily investigate the performance of the RAG system
when confronted with inconsistencies between retrieved content and
internal model knowledge. We also explain a detailed analysis of the
fidelity of the RAG model, exploring the conflicts between the internal
priors of RAG and LLMs.

### Background

#### Classification of Knowledge Conflicts

In a recent review by Rongwu Xu et al. on knowledge conflicts, it was
noted that when LLM is used to answer user questions, users may provide
additional prompts, and LLM may also utilize search engines to collect
relevant documents from the web to enhance its knowledge.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*W471ze_DQMJAakwq" />
<figcaption>Three primary types of knowledge conflicts that an LLM may
encounter, arising from different knowledge sources, Source: Rongwu Xu
et al. 2024. Link: <a
href="https://arxiv.org/pdf/2403.08319.pdf">https://arxiv.org/pdf/2403.08319.pdf</a>.
Contextual (labeled as I. Context, represented by yellow chatboxes) or
inherent to the LLM’s parameters (labeled as II. Memory, represented by
blue chatboxes). When faced with a user’s question (depicted by the
purple chatbox) involving intricate conflicts in knowledge, the LLM must
resolve these disparities to provide precise responses.</figcaption>
</figure>

These user prompts, conversation histories, and retrieved documents
constitute contextual knowledge. With such diverse sources of knowledge,
knowledge conflicts can be classified into three categories:

1\. **Inter-contextual conflict**: Different external documents from
various sources may contain noise or false information, leading to
conflicting information between the retrieved documents.

2\. **Context-memory conflict**: External documents retrieved may
conflict with the parameterized knowledge in LLM.

3\. **Intra-memory conflict**: Users may pose the same question in
different forms, resulting in inconsistent responses from LLM's
parameterized knowledge to different formulations of the question.

### RAG and LLM Knowledge Conflict

In these three conflict patterns, the **context-memory conflict** is the
most extensively studied type. Large language models have fixed
parameter knowledge, which is the result of extensive pre-training. This
static parameter knowledge contrasts sharply with the dynamic nature of
external information, which evolves rapidly.

The core source of context-memory conflict lies in the difference
between contextual knowledge and parameter knowledge. There are two main
reasons for this: **temporal discrepancies** and **contamination with
false information**. Temporal discrepancies refer to the fact that the
latest contextual information is usually considered accurate, whereas
contamination with false information refers to situations where the
contextual information contains erroneous data.

Among the various sources of information, we focus on RAG as an example
to study knowledge conflicts. Before discussing the reliability of the
RAG model, it is essential to understand its functioning. RAG combines
two steps: **retrieval** and **generation**. Firstly, it retrieves
information relevant to the question using a retrieval system, then
provides this information as context to Large Language models in order
to generate more accurate answers.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*eujuUWXP-gwEy7py" />
<figcaption>Process of generating modified documents for each dataset,
Source: Kevin Wu et al., 2024. Link: <a
href="https://arxiv.org/pdf/2404.10198.pdf">https://arxiv.org/pdf/2404.10198.pdf</a>
Initially, a question is presented to the LLM both with and without a
reference document containing pertinent information related to the
query. Subsequently, this document undergoes perturbation to integrate
modified information and is provided as context to the LLM. The
subsequent observation involves assessing whether the LLM favours the
modified information over its own prior response.</figcaption>
</figure>

For RAG's auxiliary function, our focus lies in: Is providing correct
retrieval content always effective in correcting errors when it's known
that LLM's answers to questions are wrong? Conversely, if the retrieved
content is incorrect, does LLM know to disregard the erroneous
information, or will it repeat the mistake?

### RAG preference rate

In this research, delving into the correlation between the robustness of
the model's prior beliefs and the preference rate of RAG was done. An
intriguing relationship was discovered: the extent to which LLM aligns
with the information furnished by RAG (RAG preference rate) stands in
inverse proportion to the model's confidence in responding without
contextual cues (prior probability). This indicates that LLM's trust in
retrieval information wanes as its inherent prior convictions
strengthen.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*gxSNNZ-DqdzTVf5P" />
<figcaption>Comparison of RAG preference rate and two factors. Source:
Kevin Wu et al., 2024. Link: <a
href="https://arxiv.org/pdf/2404.10198.pdf">https://arxiv.org/pdf/2404.10198.pdf</a></figcaption>
</figure>

Across six QA datasets with GPT-4, we find a consistent inverse
relationship between RAG preference rate and two factors: the model's
prior response probability and the deviation from that prior. The left
plots show how higher prior probabilities correspond to lower RAG
preference rates, while the right plots demonstrate that greater
deviations result in lower RAG preference rates. These findings
highlight the complex interplay between the model's beliefs, deviation
from those beliefs, and reliance on RAG context.

### Deviation between LLM and RAG

Furthermore, we scrutinized the extent of deviation between prior
knowledge and retrieval data. An experiment was conducted as follows:
First, LLMs were prompted to answer questions without providing any
external context and recorded the model's responses along with their
token probabilities (indicating the model's confidence in its answers).
Then, relevant retrieval content was provided and LLMs were prompted to
answer the same questions again, recording the responses and token
probabilities. Finally, the model's responses were compared between the
two queries. If LLMs provided the same response with and without
retrieval content, it indicated a preference for its internal prior
knowledge. If the response aligned with the retrieval content, it
indicated a preference for RAG information.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*XzE4KCPvLt6TtblN" />
<figcaption>Examples from three datasets show varied LLM responses to
different types of context modifications. Incorrect responses are
highlighted in red, while correct responses are highlighted in green.
Source: Kevin Wu et al., 2024. Link: <a
href="https://arxiv.org/pdf/2404.10198.pdf">https://arxiv.org/pdf/2404.10198.pdf</a></figcaption>
</figure>

As depicted in this image, when the information retrieved diverges
further from LLM's prior knowledge, the model's reliance on retrieval
information decreases accordingly. This indicates that when the data
supplied by RAG significantly deviates from the model's internal
knowledge, LLM tends to place greater emphasis on its existing beliefs.

### Prompting Techniques

The impact of different prompting techniques (such as strict adherence
and loose adherence) on the model's RAG preference rate was investigated
as well. The choice of prompting technique plays a crucial role in how
the model integrates retrieved information and generates responses.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*G3ZOl5siAnk-gnkA" />
<figcaption>Effect of different prompts using GPT-4 on RAG preference
rate vs prior probability. Source: Kevin Wu et al., 2024. Link: <a
href="https://arxiv.org/pdf/2404.10198.pdf">https://arxiv.org/pdf/2404.10198.pdf</a></figcaption>
</figure>

In this result, the prompt plays a crucial role in controlling RAG
adherence, with the "Loose" prompt leading to a decrease in RAG
adherence compared to the "Strict" prompt. The "Strict" prompt requires
strict adherence to the retrieved context, while the "Loose" prompt
encourages the model to make reasonable judgments based on the
provided context.

The recent study by Zhuoran Jin arrived at similar conclusions too.
Moreover, it was observed that as the model's capabilities strengthen,
it becomes more self-assured, leading to increased confidence in its own
knowledge as its scale and capacity expand. However, without proper
verification to resolve conflicts, the model remains incapable of
providing accurate responses.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*KR8Co8C6duLLk3TC" />
<figcaption>Experiment between internal memory and external source on NQ
and TriviaQA. Source: Zhuoran Jin et al., 2024. Link: <a
href="https://arxiv.org/pdf/2402.14409.pdf">https://arxiv.org/pdf/2402.14409.pdf</a></figcaption>
</figure>

As shown in the above table, the experimental results on the NQ and
TriviaQA datasets reveal a significant difference between the recall
metrics when considering knowledge conflicts between internal memory and
external sources. Here, Mem R denotes the recall between predictions and
internal memory predictions, while Con R represents the recall between
predictions and conflicting references. Additionally, IMR --- CMR
indicates the difference in memorization ratio between incorrect memory
and correct memory. The values in bold highlight the highest results.

Through experiments conducted on the NQ and TriviaQA datasets, a notable
trend was observed: after providing conflicting evidence, the recall of
Con R substantially exceeds the recall of Mem R. This finding
underscores the importance of considering external sources of
information when addressing knowledge conflicts, as they can
significantly impact the model's performance.

Furthermore, the presence of the Dunning-Kruger effect within
large-scale models was uncovered. The question arises: does the model
exhibit greater confidence in its correct internal knowledge or in its
incorrect internal knowledge? As the model's capabilities increase, we
found that the memorization ratio of incorrect information surpasses
that of correct information. This indicates that while the model lacks
confidence in its correct internal knowledge, it persists in adhering to
incorrect internal information. This finding aligns with the principles
of human psychology's Dunning-Kruger effect, wherein individuals with
limited abilities tend to overestimate their competence in
specific domains.

Combining the findings from the aforementioned experiments, it can be
surmised that fundamentally, as the capabilities of LLMs grow, they
exhibit increased confidence in their internal knowledge. This
confidence is bolstered by extensive training data and pre-existing
knowledge. However, when faced with conflicting information, LLMs tend
to prioritize their internal knowledge, particularly in the absence of
strict prompting to adhere to retrieved context. This preference for
internal knowledge is evident even as LLMs encounter more complex
conflicts, suggesting a robust reliance on pre-existing beliefs.
Nonetheless, the presence of the Dunning-Kruger effect within these
models indicates a propensity to prioritize incorrect internal knowledge
over correct information, underscoring the intricacies of managing
knowledge conflicts in LLMs.

### Conclusion

In conclusion, the RAG model, as an effective means of improving the
accuracy of LLMs, has garnered widespread attention in practical
applications. Systematic analysis and experiments have revealed the
intrinsic mechanisms of the RAG model when confronted with inconsistent
information, providing important theoretical and empirical support for
the future design and optimization of RAG systems. The findings
emphasize the need to balance internal prior knowledge and retrieval
information when designing RAG systems, as well as the impact of
prompting techniques on model behaviour. These insights are crucial for
ensuring the accuracy and reliability of RAG systems in practical
applications. Additionally, our research highlights the presence of the
Dunning-Kruger effect within large-scale language models, revealing how
these models handle and prioritize knowledge conflicts differently.

Overall, this study contributes to a deeper understanding of the
challenges and opportunities in managing knowledge conflicts within
LLMs, paving the way for advancements in model robustness and
performance.

### **References**

- Xu, R., Qi, Z., Wang, C., Wang, H., Zhang, Y., & Xu, W. (2024).
 Knowledge Conflicts for LLMs: A Survey (Version 1). arXiv.
 <https://doi.org/10.48550/ARXIV.2403.08319>
- Wu, K., Wu, E., & Zou, J. (2024). How faithful are RAG models?
 Quantifying the tug-of-war between RAG and LLMs' internal prior.
 arXiv:2404.10198 \[cs.CL\]. <https://arxiv.org/abs/2404.10198>
- Feng, S., Shi, W., Wang, Y., Ding, W., Balachandran, V., &
 Tsvetkov, Y. (2024). Don't Hallucinate, Abstain: Identifying LLM
 Knowledge Gaps via Multi-LLM Collaboration (Version 1). arXiv.
 <https://doi.org/10.48550/ARXIV.2402.00367>
- Jin, Z., Cao, P., Chen, Y., Liu, K., Jiang, X., Xu, J., Li, Q., &
 Zhao, J. (2024). Tug-of-War Between Knowledge: Exploring and Resolving
 Knowledge Conflicts in Retrieval-Augmented Language Models (Version
 1). arXiv. <https://doi.org/10.48550/ARXIV.2402.14409>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4f8a517afad){width="1"
height="1"}

Understanding the Balance between Internal Knowledge and External Sources

Author Qingqin Fang (

 ORCID:

0009–0003–5348–4264) Introduction Previous research often emphasized the limitations of LLM’s information acquisition pathways, focusing on enhancing its capabilities in this regard.

Rogue Scholar Posts

Multimodal Large Language Models for Misinformation Detection and Reasoning

When RAG and LLM Conflict: Who Will AI Listen to?