Australian National University

Qingqin Fang

#### **Understanding the Balance between Internal Knowledge and External Sources**

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*-GBIM_Ece6r7flKJ" />
<figcaption>Image Generated by Google Gemini</figcaption>
</figure>

### Author

- Qingqin Fang (**ORCID:**
 [0009--0003--5348--4264](https://orcid.org/0009-0003-5348-4264))

### Introduction

Previous research often emphasized the limitations of LLM's information
acquisition pathways, focusing on enhancing its capabilities in this
regard. However, integrating contextual knowledge into LLM to keep it
abreast of current affairs and improve response accuracy may lead to
conflicts due to the richness of knowledge sources, known as *knowledge
conflicts*.

So, what is a knowledge conflict? Specifically, it refers to the
contradictory or inconsistent information obtained by LLM from different
sources, including internal knowledge or memory from LLM itself and
external knowledge retrieved through **RAG** (Retriever-Reader
Architecture) or a knowledge graph.

In this post, we primarily investigate the performance of the RAG system
when confronted with inconsistencies between retrieved content and
internal model knowledge. We also explain a detailed analysis of the
fidelity of the RAG model, exploring the conflicts between the internal
priors of RAG and LLMs.

### Background

#### Classification of Knowledge Conflicts

In a recent review by Rongwu Xu et al. on knowledge conflicts, it was
noted that when LLM is used to answer user questions, users may provide
additional prompts, and LLM may also utilize search engines to collect
relevant documents from the web to enhance its knowledge.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*W471ze_DQMJAakwq" />
<figcaption>Three primary types of knowledge conflicts that an LLM may
encounter, arising from different knowledge sources, Source: Rongwu Xu
et al. 2024. Link: <a
href="https://arxiv.org/pdf/2403.08319.pdf">https://arxiv.org/pdf/2403.08319.pdf</a>.
Contextual (labeled as I. Context, represented by yellow chatboxes) or
inherent to the LLM’s parameters (labeled as II. Memory, represented by
blue chatboxes). When faced with a user’s question (depicted by the
purple chatbox) involving intricate conflicts in knowledge, the LLM must
resolve these disparities to provide precise responses.</figcaption>
</figure>

These user prompts, conversation histories, and retrieved documents
constitute contextual knowledge. With such diverse sources of knowledge,
knowledge conflicts can be classified into three categories:

1\. **Inter-contextual conflict**: Different external documents from
various sources may contain noise or false information, leading to
conflicting information between the retrieved documents.

2\. **Context-memory conflict**: External documents retrieved may
conflict with the parameterized knowledge in LLM.

3\. **Intra-memory conflict**: Users may pose the same question in
different forms, resulting in inconsistent responses from LLM's
parameterized knowledge to different formulations of the question.

### RAG and LLM Knowledge Conflict

In these three conflict patterns, the **context-memory conflict** is the
most extensively studied type. Large language models have fixed
parameter knowledge, which is the result of extensive pre-training. This
static parameter knowledge contrasts sharply with the dynamic nature of
external information, which evolves rapidly.

The core source of context-memory conflict lies in the difference
between contextual knowledge and parameter knowledge. There are two main
reasons for this: **temporal discrepancies** and **contamination with
false information**. Temporal discrepancies refer to the fact that the
latest contextual information is usually considered accurate, whereas
contamination with false information refers to situations where the
contextual information contains erroneous data.

Among the various sources of information, we focus on RAG as an example
to study knowledge conflicts. Before discussing the reliability of the
RAG model, it is essential to understand its functioning. RAG combines
two steps: **retrieval** and **generation**. Firstly, it retrieves
information relevant to the question using a retrieval system, then
provides this information as context to Large Language models in order
to generate more accurate answers.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*eujuUWXP-gwEy7py" />
<figcaption>Process of generating modified documents for each dataset,
Source: Kevin Wu et al., 2024. Link: <a
href="https://arxiv.org/pdf/2404.10198.pdf">https://arxiv.org/pdf/2404.10198.pdf</a>
Initially, a question is presented to the LLM both with and without a
reference document containing pertinent information related to the
query. Subsequently, this document undergoes perturbation to integrate
modified information and is provided as context to the LLM. The
subsequent observation involves assessing whether the LLM favours the
modified information over its own prior response.</figcaption>
</figure>

For RAG's auxiliary function, our focus lies in: Is providing correct
retrieval content always effective in correcting errors when it's known
that LLM's answers to questions are wrong? Conversely, if the retrieved
content is incorrect, does LLM know to disregard the erroneous
information, or will it repeat the mistake?

### RAG preference rate

In this research, delving into the correlation between the robustness of
the model's prior beliefs and the preference rate of RAG was done. An
intriguing relationship was discovered: the extent to which LLM aligns
with the information furnished by RAG (RAG preference rate) stands in
inverse proportion to the model's confidence in responding without
contextual cues (prior probability). This indicates that LLM's trust in
retrieval information wanes as its inherent prior convictions
strengthen.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*gxSNNZ-DqdzTVf5P" />
<figcaption>Comparison of RAG preference rate and two factors. Source:
Kevin Wu et al., 2024. Link: <a
href="https://arxiv.org/pdf/2404.10198.pdf">https://arxiv.org/pdf/2404.10198.pdf</a></figcaption>
</figure>

Across six QA datasets with GPT-4, we find a consistent inverse
relationship between RAG preference rate and two factors: the model's
prior response probability and the deviation from that prior. The left
plots show how higher prior probabilities correspond to lower RAG
preference rates, while the right plots demonstrate that greater
deviations result in lower RAG preference rates. These findings
highlight the complex interplay between the model's beliefs, deviation
from those beliefs, and reliance on RAG context.

### Deviation between LLM and RAG

Furthermore, we scrutinized the extent of deviation between prior
knowledge and retrieval data. An experiment was conducted as follows:
First, LLMs were prompted to answer questions without providing any
external context and recorded the model's responses along with their
token probabilities (indicating the model's confidence in its answers).
Then, relevant retrieval content was provided and LLMs were prompted to
answer the same questions again, recording the responses and token
probabilities. Finally, the model's responses were compared between the
two queries. If LLMs provided the same response with and without
retrieval content, it indicated a preference for its internal prior
knowledge. If the response aligned with the retrieval content, it
indicated a preference for RAG information.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*XzE4KCPvLt6TtblN" />
<figcaption>Examples from three datasets show varied LLM responses to
different types of context modifications. Incorrect responses are
highlighted in red, while correct responses are highlighted in green.
Source: Kevin Wu et al., 2024. Link: <a
href="https://arxiv.org/pdf/2404.10198.pdf">https://arxiv.org/pdf/2404.10198.pdf</a></figcaption>
</figure>

As depicted in this image, when the information retrieved diverges
further from LLM's prior knowledge, the model's reliance on retrieval
information decreases accordingly. This indicates that when the data
supplied by RAG significantly deviates from the model's internal
knowledge, LLM tends to place greater emphasis on its existing beliefs.

### Prompting Techniques

The impact of different prompting techniques (such as strict adherence
and loose adherence) on the model's RAG preference rate was investigated
as well. The choice of prompting technique plays a crucial role in how
the model integrates retrieved information and generates responses.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*G3ZOl5siAnk-gnkA" />
<figcaption>Effect of different prompts using GPT-4 on RAG preference
rate vs prior probability. Source: Kevin Wu et al., 2024. Link: <a
href="https://arxiv.org/pdf/2404.10198.pdf">https://arxiv.org/pdf/2404.10198.pdf</a></figcaption>
</figure>

In this result, the prompt plays a crucial role in controlling RAG
adherence, with the "Loose" prompt leading to a decrease in RAG
adherence compared to the "Strict" prompt. The "Strict" prompt requires
strict adherence to the retrieved context, while the "Loose" prompt
encourages the model to make reasonable judgments based on the
provided context.

The recent study by Zhuoran Jin arrived at similar conclusions too.
Moreover, it was observed that as the model's capabilities strengthen,
it becomes more self-assured, leading to increased confidence in its own
knowledge as its scale and capacity expand. However, without proper
verification to resolve conflicts, the model remains incapable of
providing accurate responses.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*KR8Co8C6duLLk3TC" />
<figcaption>Experiment between internal memory and external source on NQ
and TriviaQA. Source: Zhuoran Jin et al., 2024. Link: <a
href="https://arxiv.org/pdf/2402.14409.pdf">https://arxiv.org/pdf/2402.14409.pdf</a></figcaption>
</figure>

As shown in the above table, the experimental results on the NQ and
TriviaQA datasets reveal a significant difference between the recall
metrics when considering knowledge conflicts between internal memory and
external sources. Here, Mem R denotes the recall between predictions and
internal memory predictions, while Con R represents the recall between
predictions and conflicting references. Additionally, IMR --- CMR
indicates the difference in memorization ratio between incorrect memory
and correct memory. The values in bold highlight the highest results.

Through experiments conducted on the NQ and TriviaQA datasets, a notable
trend was observed: after providing conflicting evidence, the recall of
Con R substantially exceeds the recall of Mem R. This finding
underscores the importance of considering external sources of
information when addressing knowledge conflicts, as they can
significantly impact the model's performance.

Furthermore, the presence of the Dunning-Kruger effect within
large-scale models was uncovered. The question arises: does the model
exhibit greater confidence in its correct internal knowledge or in its
incorrect internal knowledge? As the model's capabilities increase, we
found that the memorization ratio of incorrect information surpasses
that of correct information. This indicates that while the model lacks
confidence in its correct internal knowledge, it persists in adhering to
incorrect internal information. This finding aligns with the principles
of human psychology's Dunning-Kruger effect, wherein individuals with
limited abilities tend to overestimate their competence in
specific domains.

Combining the findings from the aforementioned experiments, it can be
surmised that fundamentally, as the capabilities of LLMs grow, they
exhibit increased confidence in their internal knowledge. This
confidence is bolstered by extensive training data and pre-existing
knowledge. However, when faced with conflicting information, LLMs tend
to prioritize their internal knowledge, particularly in the absence of
strict prompting to adhere to retrieved context. This preference for
internal knowledge is evident even as LLMs encounter more complex
conflicts, suggesting a robust reliance on pre-existing beliefs.
Nonetheless, the presence of the Dunning-Kruger effect within these
models indicates a propensity to prioritize incorrect internal knowledge
over correct information, underscoring the intricacies of managing
knowledge conflicts in LLMs.

### Conclusion

In conclusion, the RAG model, as an effective means of improving the
accuracy of LLMs, has garnered widespread attention in practical
applications. Systematic analysis and experiments have revealed the
intrinsic mechanisms of the RAG model when confronted with inconsistent
information, providing important theoretical and empirical support for
the future design and optimization of RAG systems. The findings
emphasize the need to balance internal prior knowledge and retrieval
information when designing RAG systems, as well as the impact of
prompting techniques on model behaviour. These insights are crucial for
ensuring the accuracy and reliability of RAG systems in practical
applications. Additionally, our research highlights the presence of the
Dunning-Kruger effect within large-scale language models, revealing how
these models handle and prioritize knowledge conflicts differently.

Overall, this study contributes to a deeper understanding of the
challenges and opportunities in managing knowledge conflicts within
LLMs, paving the way for advancements in model robustness and
performance.

### **References**

- Xu, R., Qi, Z., Wang, C., Wang, H., Zhang, Y., & Xu, W. (2024).
 Knowledge Conflicts for LLMs: A Survey (Version 1). arXiv.
 <https://doi.org/10.48550/ARXIV.2403.08319>
- Wu, K., Wu, E., & Zou, J. (2024). How faithful are RAG models?
 Quantifying the tug-of-war between RAG and LLMs' internal prior.
 arXiv:2404.10198 \[cs.CL\]. <https://arxiv.org/abs/2404.10198>
- Feng, S., Shi, W., Wang, Y., Ding, W., Balachandran, V., &
 Tsvetkov, Y. (2024). Don't Hallucinate, Abstain: Identifying LLM
 Knowledge Gaps via Multi-LLM Collaboration (Version 1). arXiv.
 <https://doi.org/10.48550/ARXIV.2402.00367>
- Jin, Z., Cao, P., Chen, Y., Liu, K., Jiang, X., Xu, J., Li, Q., &
 Zhao, J. (2024). Tug-of-War Between Knowledge: Exploring and Resolving
 Knowledge Conflicts in Retrieval-Augmented Language Models (Version
 1). arXiv. <https://doi.org/10.48550/ARXIV.2402.14409>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4f8a517afad){width="1"
height="1"}

Understanding the Balance between Internal Knowledge and External Sources

Author Qingqin Fang (

 ORCID:

0009–0003–5348–4264) Introduction Previous research often emphasized the limitations of LLM’s information acquisition pathways, focusing on enhancing its capabilities in this regard.

When RAG and LLM Conflict: Who Will AI Listen to?

CSTonline

**Crossings: Stories of Migration** (June-September 2017) is an ICA-led
UK-wide film and events programme supported by the BFI using National
Lottery funding, in partnership with the Goethe-Institut and the School
of Film & Television, Falmouth University.

We live in an age of migration. Driven by political and military
conflicts, the pressures and possibilities of increasingly globalised
economies, climate change and the desire for a better life, more people
than ever move and live across borders. These are diverse trajectories;
each journey has its own story and each story its own
journey, traversing different landscapes and carrying with it its own
past.

Through a programme of screenings and events, *Crossings* explores the
stories of modern-day migration and reflects on cinema's potential to
interrogate the damaging ways in which migration is often framed.
Including a variety of documentary and fiction features, the programme
engages with transnational crossings from the last five decades through
different forms of aesthetically innovative storytelling.

As part of the *Crossings* project, the ICA intends to curate scholarly
and creative responses to the issues raised by the season, which will be
hosted on their dedicated website <https://www.crossings2017.co.uk/>

The website is intended to function as an ongoing repository for a range
of essays, short films, film essays, reviews, and scholarly endeavour
which will live on past the screening season itself.

[Please note: *Crossings* will run throughout the summer, and as such,
we are as interested in responses to the screenings as we are in
pre-existing or 'anticipatory' work; therefore, a further call for
submissions will be made in early Autumn.]{.underline}

We welcome any proposals, and wish to take a flexible and imaginative
approach to format and length, but academic and creative work from
undergraduate and postgraduate students and ECRs are particularly
welcome.

Topics may include, but are not limited to:

- Transnational communication
- Migration and diasporic community
- Borders and transgressions
- The physical and metaphorical journey of migration
- Migration, power and agency
- The political contexts of migration
- Repression and 'Othering'
- The reproduction or breaching of stereotypes around migration

Please send a brief (500-1000 word) abstract with short bio to
[laura.canning@falmouth.ac.uk](mailto:laura.canning@falmouth.ac.uk){target="_blank"}
including the term "Crossings" in your subject heading.   Deadline for
initial submissions: Friday August 4^th^ 2017.

*Crossings* brings together leading commentators, critics and
contributors from across various cultural and political contexts to
create a rigorous yet diverse platform for discussion to explore
migration from both global and local perspectives. Engaging with ethical
and political questions raised by representing these narratives on
screen, the season aims to open space for dialogue at a time when this
is more urgent than ever. For further details on the schedule of films
and venues, please see
[https://www.crossings2017.co.uk/](https://www.crossings2017.co.uk/){target="_blank"}

Crossings: Stories of Migration

(June-September 2017) is an ICA-led UK-wide film and events programme supported by the BFI using National Lottery funding, in partnership with the Goethe-Institut and the School of Film &amp; Television, Falmouth University. We live in an age of migration.

Messaggi di Rogue Scholar

When RAG and LLM Conflict: Who Will AI Listen to?

Call for Papers, Film Essays, Short Films: “Crossings: Stories of Migration”. Deadline: Aug 4, 2017.