Australian National University

Dhruv Gupta

Improving the performance of Large Language Models

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*U7xu3eHakXP1gTYc" />
<figcaption>Image from Unsplash.</figcaption>
</figure>

### Author

* Dhruv Gupta (ORCID: [0009-0004-7109-5403](https://orcid.org/0009-0004-7109-5403))

ChatGPT, which first came out in late 2022, took the world by storm.
Since then, various LLM models and LLM based products such as Meta's
Llama and Google's Gemini have emerged, demonstrating the power of LLMs.
With applications ranging from chatbots to content generation to
sentiment analysis, one always wonders what is next after LLMs. Even
though LLMs appear to solve many problems, they do have certain
limitations, including outdated data and lack of credibility. This is
where Retrieval-Augmented Generation (RAG) comes into the picture.

In this article we will look at what RAG is and how it works. The
article also focuses on various aspects of RAG and the improvement shown
by RAG models over the other state of the art models.

### **What is RAG?**

RAG stands for Retrieval-Augmented Generation. The idea behind RAG is to
augment LLMs with a latent knowledge retriever, allowing the model to
retrieve knowledge from a vast corpus of data. This corpus could include
sources such as Wikipedia or data from news websites. By incorporating
this large corpus of data, RAG addresses both the issues with LLMs,
capturing the latest data available and providing users with a reliable
data source.

### **Deep Dive into RAG**

A simple RAG model is typically composed of two parts: a retriever and a
generator.

<figure>
<img src="https://cdn-images-1.medium.com/max/743/0*aro1YIcXm3XYyv7G" />
<figcaption>Original RAG model scheme. Source: Lewis et al., 2021. Link:
<a
href="https://arxiv.org/abs/2005.11401">https://arxiv.org/abs/2005.11401</a>.</figcaption>
</figure>

#### **Retriever**

The retriever is arguably the most crucial aspect of a RAG models.
Essentially, a retriever, as the name suggests, is tasked with
retrieving the top k documents, which the generator then uses to produce
a response. However, the process is more intricate than simply obtaining
documents.

The first step of the retriever is to encode the question using a query
encoder, typically achieved using a BERT encoder. The next step is to
fetch the top k related documents from the corpus. This is accomplished
using techniques such as Maximum Inner Product Search (MIPS) and Dense
Passage Retrieval (DPR). The DPR consists of two BERT models: one that
encodes the question and another that encodes the passages. In the case
of RAG, a pre-trained DPR is utilised to aid the MIPS. The MIPS
generates a document index containing the k most relevant documents,
which are then fed into the generator.

#### **Generator**

The operation of the generator component of the RAG model is quite
similar to an encoder-decoder model. The first step of the generator
model is to augment the set of retrieved documents to the query vector.
The augmented query vector is then inputted into a model such as BART or
T5, which generates an output response. Unlike LLMs, RAG can also cite
the source used to generate the response.

### **Results**

RAG has demonstrated immense potential compared to other
state-of-the-art models. RAG has been proven to be more powerful in
terms of both qualitative and quantitative results. Qualitatively, there
have been instances of fewer hallucinations in RAG model outputs.
Additionally, the generated outputs are consistent, knowledgeable, and
accurate compared to its counterpart models. Furthermore, RAG models
have shown an improvement of about 4--16% in terms of absolute accuracy.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*Fp-S-pevHUnZmAYg" />
<figcaption>Some RAG generated outputs highlighting the performance
improvement over other state of the art models. Source: Shuster et al.,
2021. Link: <a
href="https://doi.org/10.48550/arXiv.2104.07567">https://doi.org/10.48550/arXiv.2104.07567</a></figcaption>
</figure>

### **Current Domain Tasks**

Currently, RAG has become an area of extensive research. While
researchers continue to explore RAG's potential, it has already proved
its mettle in some areas. These areas include open-domain question
answering, abstractive question answering, jeopardy question answering,
and fact verification. The context answers generated by RAG have been
accurate and consistent in all of these tasks.

### **Conclusion**

RAG has proven to be very successful in QA-based tasks, and researchers
are confident it can be applied to other Natural Language Processing
(NLP) tasks. The researchers are confident about RAG's potential in
other NLP tasks as well. These tasks include machine translation,
content generation, information retrieval, paragraph paraphrase etc.

In conclusion, RAG has shown great potential in becoming the next big
advancement after LLMs, potentially even surpassing them in solving
complex NLP tasks.

#### Sources

- Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal,
 N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., &
 Kiela, D. (2020). Retrieval-Augmented Generation for
 Knowledge-Intensive NLP Tasks (Version 4). arXiv.
 <https://doi.org/10.48550/ARXIV.2005.11401>
- Mao, Y., He, P., Liu, X., Shen, Y., Gao, J., Han, J., & Chen, W.
 (2020). Generation-Augmented Retrieval for Open-domain Question
 Answering (Version 4). arXiv.
 <https://doi.org/10.48550/ARXIV.2009.08553>
- Siriwardhana 2021 How to Finetune the entire RAG Architecture
 (including DPR retriever). Medium Article May 2021.
 <https://shamanesiri.medium.com/how-to-finetune-the-entire-rag-architecture-including-dpr-retriever-4b4385322552>
- Shuster, K., Poff, S., Chen, M., Kiela, D., & Weston, J. (2021).
 Retrieval Augmentation Reduces Hallucination in Conversation (Version
 1). arXiv. <https://doi.org/10.48550/ARXIV.2104.07567>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b60708ab2b4){width="1"
height="1"}

Improving the performance of Large Language Models Author  Dhruv Gupta (ORCID: 0009-0004-7109-5403)  ChatGPT, which first came out in late 2022, took the world by storm. Since then, various LLM models and LLM based products such as Meta’s Llama and Google’s Gemini have emerged, demonstrating the power of LLMs.

RAG: The next big thing after LLMs?

Amanda Kau

**Improving the performance and application of Large Language Models**

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*Jb1q1366lzVqD2JU" />
<figcaption>Image generated with Google’s Gemini, 24
February 2024.</figcaption>
</figure>

### Author

* Amanda Kau (ORCID: [0009-0004-4949-9284](https://orcid.org/0009-0004-4949-9284))

Large language models (LLMs) like GPT-4, the engine of products like
ChatGPT, have taken centre stage in recent years due to their
astonishing capabilities. Yet, they are far from perfect. Many of us
have since learnt --- perhaps when asking ChatGPT a question or
employing it to write our reports --- that LLMs can hallucinate. This
happens when the LLM so eloquently expresses false knowledge that we
might be fooled by it. This major flaw has spurred the popularity of
Retrieval-Augmented Generation (RAG) techniques as a way to optimise an
LLM's responses.

To start off, this article will cover a brief overview of the key
concept behind RAG. Subsequently, a review on several issues behind the
retrieval step of RAG will be presented. In particular, this article
will review ideas on when and what should be retrieved, the quantity of
retrieved documents, effects of data quality, and RAG applied to
different domains. Strategies proposed by the research community will
also be briefly introduced for each challenge.

### **A Brief Introduction To RAG**

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*T6Zr0wCmrTOe83xp" />
<figcaption>Simple diagram about RAG.</figcaption>
</figure>

Using RAG is very much like performing Google searches while writing a
report. When the LLM attempts to generate a sentence without the
requisite knowledge, it needs to reference an external source to gain
that knowledge. Particularly prevalent in tasks like open domain
question answering (ODQA), the LLM may be enquired about a specific
topic which requires in-depth knowledge. Although LLMs are trained with
extensive amounts of diverse data, it is unreasonable to expect the LLM
to have internalised every potential answer and to deliver a response
that an expert might provide.

This is where RAG comes in handy. RAG comprises two components: the
retriever and the generator. The retriever gathers a set of supporting
documents from a given source based on an input query and passes them on
to the generator or LLM. This allows the LLM to avoid hallucinations,
thereby crafting more well-informed and relevant responses.

### **Addressing Retrieval Problems and Strategies**

#### 1. When and What To Retrieve {#when-and-what-to-retrieve}

Many retrieval-augmented language models utilise a single retrieval
process, which is particularly restrictive if the model is tasked to
generate a long passage of text. It is akin to writing a report whilst
only being permitted to reference documents once at the start.
Conversely, models that attempt to do multiple retrievals might do so at
fixed intervals. This seems illogical when we draw parallels to our
report writing scenario. It is like doing an online search every two
sentences we write. When the LLM is routinely bombarded by additional
information it does not require, the LLM might be disoriented, resulting
in it returning incoherent or irrelevant responses.

Some attempts to address this problem have been made, such as the
Forward-Looking Active REtrieval (FLARE) augmented generation method.
The premise of FLARE is simple: retrieve supporting information only
when the LLM signals low confidence indicating a lack of knowledge.

The subsequent challenge revolves around what information to retrieve
for the upcoming sentence. Imagine if you were tasked to find evidence
for a friend who's formulating an argument, but you must provide the
evidence without knowing what their next sentence will be. FLARE targets
this by prompting the LLM to generate a temporary potential sentence and
using this to retrieve documents --- essentially, asking the friend what
they might want to say next before doing your search.

#### **2. Quantity of Retrieved Documents** {#quantity-of-retrieved-documents}

The architectures of some RAG models confine them to retrieving a
certain number of supporting documents. Picture writing a report whilst
only having access to a fixed number of documents --- your viewpoints
and content would be largely based on that limited pool of knowledge. If
that set of documents were replaced, your argument might undergo a
drastic shift as your supporting information might change, resulting in
a lack of consistency in responses. LLMs are vulnerable in a similar way
and are sensitive to the quality of retrieved documents. That is, if
retrieved documents prove irrelevant to the context, the LLM's
output suffers.

One attempt to mitigate this involves clustering the training data as in
the case of MemGM, a memory-augmented generative model. This notion of
memory-augmentation means the model internalises the characteristics of
clusters of responses and uses them to aid response generation. To do
this, the data, which consists of query and response example pairs, is
grouped according to the similarity of queries. Subsequently, when MemGM
retrieves information from this database, the cluster average
representing the cluster's characteristics is returned instead of a set
of documents. This approach dilutes individual responses, hence
decreasing the LLM's sensitivity to individual documents. This also
serves as a means to provide generalised support from a large number of
documents to the LLM.

#### 3. Quality of Retrieved Documents {#quality-of-retrieved-documents}

**Data Recency:** The knowledge encapsulated within LLMs is frozen in
time from when they were trained, preventing them from being able to
provide up-to-the-minute information about the world. In fact, without
taking corrective measures, LLMs become increasingly outdated and
irrelevant over time. Nonetheless, they are still expected to interact
with the ever changing world, so data recency is of critical importance.

One proposed solution is to harvest the power of the Internet. Not only
does it contain the most current information, but years of refinement
have enabled Internet search engines to properly rank results, navigate
online safety and privacy concerns, and more. By allowing RAG models to
access the Internet dynamically, LLM models would always have access to
up-to-date information to craft accurate responses.

**Data Relevance:** Another factor influencing quality is the relevance
of the retrieved results to the context of the conversation or question
posed to the LLM. Ideally, irrelevant retrieved data should not harm the
LLM's performance, but this is not the case. Instead, irrelevant data
tends to mislead the LLM to either provide inaccurate answers or lose
sight of the initial context. This error compounds in multi-hop question
answering scenarios where successive questions cause the LLM to be
misled further with each question.

A simple natural language inference (NLI) model can be employed on top
of the LLM to circumvent this issue. Simply put, the NLI model
identifies the relevance of the retrieved information to the
conversational context. If the retrieved documents are deemed as
irrelevant, the prompt is given directly to the LLM without additional
documents to avoid confusing it. Additionally, if training the LLM is an
available option, studies have shown that even with a relatively small
dataset of 1,000 examples, LLMs can be trained to disregard irrelevant
documents.

#### 4. Problems Generalising to Specific Domains {#problems-generalising-to-specific-domains}

The final challenge discussed in this article occurs when we want the
LLM to have expertise in specific domains, like healthcare or finance.
It was briefly mentioned in the introduction to RAG that it contains a
retriever component. The retriever in the original RAG model was trained
on Wikipedia-based datasets. Therefore, RAG performs well when provided
with domain-specific data that adheres to the Wikipedia article format,
but fails with other data formats. A simple example is financial news,
which is usually succinct in the form of news flashes or tweets, and
does not provide additional context as it is assumed that the reader
possesses the necessary background knowledge. This adds another
disadvantage to the LLM as it will not have the necessary context to
understand the content presented to it.

The RAG model can be further fine-tuned to be domain-specific to target
this specific issue. In fact, a group of researchers proposed
RAG-end2end which extends the original RAG by jointly training all of
its components for a specific domain. However, this is very
computationally expensive and unfeasible for someone who lacks access to
multiple GPUs. Moreover, for broad topics like finance, it is key to
incorporate macroeconomic information and other contextual information
to give the LLM a well-rounded view of the situation, which remains an
area of research.

### Conclusion

In summary, this article has offered insight into some challenges that
may arise when employing RAG and some solutions proposed by the research
community. It highlights the importance of understanding data sources
and retrieval steps performed in RAG such that the retrieval segment may
benefit the LLM instead of harm its performance. In particular, one
should pay attention to when and what documents are retrieved, as well
as the quantity and quality of retrieved documents. Most importantly,
retrieved information should be relevant to the context and desired
domain for the generation of optimal responses.

#### Sources

Jiang, Z., Xu, F. F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang,
Y., Callan, J., & Neubig, G. (2023). Active Retrieval Augmented
Generation (Version 2). arXiv.
<https://doi.org/10.48550/ARXIV.2305.06983>

Tian, Z., Bi, W., Li, X., & Zhang, N. L. (2019). Learning to Abstract
for Memory-augmented Conversational Response Generation. In Proceedings
of the 57th Annual Meeting of the Association for Computational
Linguistics. Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics. Association for Computational
Linguistics. <https://doi.org/10.18653/v1/p19-1371>

Komeili, M., Shuster, K., & Weston, J. (2021). Internet-Augmented
Dialogue Generation (Version 1). arXiv.
<https://doi.org/10.48550/ARXIV.2107.07566>

Yoran, O., Wolfson, T., Ram, O., & Berant, J. (2023). Making
Retrieval-Augmented Language Models Robust to Irrelevant Context
(Version 1). arXiv. <https://doi.org/10.48550/ARXIV.2310.01558>

Siriwardhana, S., Weerasekera, R., Wen, E., Kaluarachchi, T., Rana, R.,
& Nanayakkara, S. (2023). Improving the Domain Adaptation of Retrieval
Augmented Generation (RAG) Models for Open Domain Question Answering. In
Transactions of the Association for Computational Linguistics (Vol. 11,
pp. 1--17). MIT Press. <https://doi.org/10.1162/tacl_a_00530>

Zhang, B., Yang, H., Zhou, T., Ali Babar, M., & Liu, X.-Y. (2023).
Enhancing Financial Sentiment Analysis via Retrieval Augmented Large
Language Models. In 4th ACM International Conference on AI in Finance.
ICAIF '23: 4th ACM International Conference on AI in Finance. ACM.
<https://doi.org/10.1145/3604237.3626866>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5bcc28a03842){width="1"
height="1"}

Improving the performance and application of Large Language Models

Author Amanda Kau (ORCID: 0009-0004-4949-9284) Large language models (LLMs) like GPT-4, the engine of products like ChatGPT, have taken centre stage in recent years due to their astonishing capabilities. Yet, they are far from perfect.

Understanding Retrieval Pitfalls: Challenges Faced by Retrieval Augmented Generation (RAG) models

Wenyi Pi

**How to efficiently retrieve information for different applications**

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*Aqa9vymMS0uP9eiqqDyE1g.png" />
<figcaption>Image created in <a
href="https://www.canva.com/">https://www.canva.com/</a></figcaption>
</figure>

### Author

* Wenyi Pi (ORCID: [0009-0002-2884-2771](https://orcid.org/0009-0002-2884-2771))

This article aims to explore various ways in which Retrieval-Augmented
Generation (RAG) can be utilised to retrieve information and generate
responses effectively within the dialogue system. The rationale behind
utilising RAG as well as potential ways in which it can be employed
effectively will be covered. Furthermore, the article will delve into
how these methods work within dialogue systems.

With the rise of large language models (LLMs), the task of generating
responses for users in task-oriented dialogues has gained increasing
popularity. This process typically involves several steps.

<figure>
<img src="https://cdn-images-1.medium.com/max/792/0*gZgeHhzv73Czx5dQ" />
<figcaption>Response Generation Process.</figcaption>
</figure>

- **Turn detection**: This step includes determining whether a user turn
 (query) should be managed by an existing API or an unstructured
 knowledge document.
- **Knowledge selection**: Once a turn is identified, the system needs
 to retrieve relevant knowledge snippets that align with the user's
 query and dialogue context.
- **Response generation:** Finally, the system generates appropriate
 responses to users based on the retrieved knowledge and query context.

Retrieval-Augmented Generation (RAG) offers promise in providing more
relevant response to user turns in LLMs. However, it also presents
challenges, such as unreliable generation, incorrect contextual
information and the possibility of hallucination during the response
generation process.

There are several potential factors that contribute to these limitations
when applying RAG in LLMs. Firstly, LLMs struggle to maintain accuracy
in the presence of increased noise levels, leading to a decline in
accuracy with higher noise ratios. Secondly, LLMs struggle to identify
and reject misinformation, especially in noisy documents, affecting
identifying accurate information leading to unpredictable responses.
Thirdly, LLMs face challenges in effectively integrating information,
especially with complex questions and noise, influencing their ability
to provide accurate and comprehensive responses. Finally, LLMs find it
challenging to identify and correct factual errors in documents,
affecting their reliability.

Retrieval-generation interactions consist of two key steps: generation
augmented retrieval (GAR) and retrieval augmented generation (RAG). In
the GAR step, the generated results are used to augment the retrieval
process. It involves retrieving relevant information and then augments
this retrieved information to produce an enriched output. In the RAG
step, the retrieval results are used to augment the generation process.
It involves retrieving relevant information, and then uses this
retrieved information, along with the query to generate responses.

### **Purpose**

The purpose of utilising Retrieval-Augmented Generation(RAG) in natural
language processing (NLP) is to enhance the effectiveness and efficiency
of information retrieval and response generation. Through integrating
both retrieval and generation components, RAG aims to produce more
relevant and accurate information to user turns. Key purpose of using
RAG includes:

1. **Improved Relevance and Accuracy**: By integrating retrieval
 results into augment generation process, LLMs can access context
 information to better answer user queries, leading to more relevant
 and accurate response generation.
2. **Up-to-date information**: RAG facilitates the augmentation of data
 storage with new information and enables the dynamical retrieval of
 the most recent information from external sources, providing users
 with up-to-date information in real-time interactions.
3. **Adaptability to Various Tasks**: RAG is applicable to a wide range
 of NLP tasks, including question answering, dialogue systems,
 content creation, and summarisation.

### **Potential ways for utilising RAG**

1. **Enhancing the retrieval process**

To ensure the effective retrieval of relevant information from knowledge
sources, one approach is known as "retrieve-then-read." This method
entails employing a retriever to access and retrieve relevant documents.
Retrieval-Based Augmentation performs well with similar examples from
retrieval, but may falter with dissimilar examples.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*AqzQG-uBhdpLXWcD" />
<figcaption>Retriever centred process.</figcaption>
</figure>

**2. Improving the generation process**

To ensure the generation of contextually relevant and coherent
responses, the "generate-then-read" approach leverages LLMs to generate
relevant documents before formulating responses, enhancing the
generation process. Also a method called
[**FilCo**](https://github.com/zorazrw/filco) (Filter Context) can be
employed. FilCo enhances the context for the generator through two key
components: identifying useful context using lexical and
information-theoretic approaches, and training context filtering models
to filter retrieved contexts during testing. Generation-based methods
have better generalisation performance but may struggle with similar
retrieval examples.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*8tElJMLKWVDlBgD4" />
<figcaption>Generation centred process</figcaption>
</figure>

**3. Integrating Retrieval and Generation Processes**

To address the isolation and lack of coordination between retrieval and
generation methods, a novel retrieval-augmented mechanism can be
employed. This approach combines the benefits of both retrieval-based
and generation-based methods to enhance the overall performance of the
system. By integrating these processes, the system can efficiently
retrieve relevant information and generate contextually relevant
responses, ensuring better coordination and integration between
retrieval and generation components.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*FLzIks9pDHByx6FJ" />
<figcaption>Retrieve and Generation integrate
process.</figcaption>
</figure>

### **Conclusion**

In summary, RAG offers a promising solution for enhancing the retrieval
and generation process in NLP tasks. However, challenges such as
unreliable generation and the risk of hallucination must be addressed to
fully utilise RAG. With continued research and innovation, RAG can
significantly improve the effectiveness and efficiency of information
retrieval and response generation, paving the way for more intelligent
and intuitive dialogue systems.

#### **Sources**

Chen, J., Lin, H., Han, X., & Sun, L. (2023). Benchmarking Large
Language Models in Retrieval-Augmented Generation (Version 2). arXiv.
<https://doi.org/10.48550/ARXIV.2309.01431>

Feng, Z., Feng, X., Zhao, D., Yang, M., & Qin, B. (2023).
Retrieval-Generation Synergy Augmented Large Language Models (Version
1). arXiv. <https://doi.org/10.48550/ARXIV.2310.05149>

Liu, S., Chen, Y., Xie, X., Siow, J., & Liu, Y. (2020).
Retrieval-Augmented Generation for Code Summarization via Hybrid GNN
(Version 5). arXiv. <https://doi.org/10.48550/ARXIV.2006.05405>

Thulke, D., Daheim, N., Dugast, C., & Ney, H. (2021). Efficient
Retrieval Augmented Generation from Unstructured Knowledge for
Task-Oriented Dialog (Version 1). arXiv.
<https://doi.org/10.48550/ARXIV.2102.04643>

Wang, Z., Araki, J., Jiang, Z., Parvez, M. R., & Neubig, G. (2023).
Learning to Filter Context for Retrieval-Augmented Generation (Version
1). arXiv. <https://doi.org/10.48550/ARXIV.2311.08377>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e46c2282b7c2){width="1"
height="1"}

How to efficiently retrieve information for different applications

Author Wenyi Pi (ORCID: 0009-0002-2884-2771) This article aims to explore various ways in which Retrieval-Augmented Generation (RAG) can be utilised to retrieve information and generate responses effectively within the dialogue system. The rationale behind utilising RAG as well as potential ways in which it can be employed effectively will be covered.

Rogue Scholar Beiträge

RAG: The next big thing after LLMs?

Understanding Retrieval Pitfalls: Challenges Faced by Retrieval Augmented Generation (RAG) models

Efficient Information Retrieval and Response Generation with Retrieval-Augmented Generation (RAG)