Australian National University

Wenyi Pi

Neo4j APOC Library Use Case

 Author

Wenyi Pi (

 ORCID:

0009–0002–2884–277)

 Introduction

In the realm of Neo4j, the APOC (Awesome Procedures on Cypher) library stands as a powerful tool. Previously, We have talked about the importance of APOC in optimising Cypher queries and improving query efficiency in our article Exploring Methods of Cypher Query Optimisations.

Introduction to APOC: Enhancing Neo4j Capabilities

Xuzeng He

Latest findings in multiple research directions for handling graph prediction and optimization Author · Xuzeng He (

 ORCID:

0009–0005–7317–7426) A graph, in short, is a description of items linked by relations, where the items of a graph are called

 nodes

(or vertices) and their relations are called

 edges

(or links). Examples of graphs can include social networks (e.g. Instagram) or knowledge

Recent Advances in using Machine Learning with Graphs

Qingqin Fang

Exploring the OpenAlex Data Structure and Visualization Author: Qingqin Fang (

 ORCID:

0009–0003–5348–4264)

 Introduction to OpenAlex

In today’s world, the realm of research papers is brimming with countless hot topics, and the sheer volume of publications can be overwhelming.

Unveiling Research Trends through OpenAlex Visualization

Vaibhav Khobragade

Exploring AI’s Ethical Terrain: Addressing Bias, Security, and Beyond

Author: Vaibhav Khobragade (

 ORCID:

0009–0009–8807–5982) Large language models (LLMs) like OpenAI’s GPT-4, Meta’s LLaMA, and Google Gemini (previously called Bard) have showcased their vast capabilities, from passing bar exams and crafting articles to generating images and website code.

Ethics and AI: Confronting the Challenges Ahead

Author: Vaibhav Khobragade (

 ORCID:

0009–0009–8807–5982) Introduction Large language models (LLMs) are becoming increasingly popular in natural language processing for their superior competence in various applications.

Enhancing Language Models: The Role of Knowledge Graph Augmentation in Overcoming LLM Challenges

Enhancing Data Interactivity with LLMs and Neo4j Knowledge Graphs


 Author

Wenyi Pi (

 ORCID

: 0009–0002–2884–2771)

 Introduction

Since OpenAI launched ChatGPT, a

 large language model (LLM)

based chatbot, in 2023, it has set off a technological wave.

The integration of large language models (LLMs) with Neo4j-based knowledge graphs

An Introduction to RA-CM3, MuRAG and RACE Author  Xuzeng He (ORCID: 0009-0005-7317-7426)  Generative Artificial Intelligence (GAI) has demonstrated impressive performances in tasks such as text generation and text-to-image generation.

Recent Advances in using Retrieving Multimodal Information for Augmented Generation

Understanding Sequential Data Modelling with Keras for Time Series Prediction

 Author

Wenyi Pi (

 ORCID

: 0009–0002–2884–2771) Introduction Recurrent Neural Networks (RNNs) are a special type of neural networks that are suitable for learning representations of sequential data like text in Natural Language Processing (NLP). We will walk through a complete example of using RNNs for time series prediction, covering

Beginner’s Guide to Recurrent Neural Networks (RNNs) with Keras

Amanda Kau

Unlocking the Power of Questions — A deep dive into Question Answering Systems

Author:

 Amanda Kau (ORCID:


 0009–0004–4949–9284


 )

Virtual assistants have popped up on numerous websites over the years.

Harnessing The Power of Knowledge Graphs in Question Answering

An Introduction to Retrieval Augmented Generation (RAG) and Knowledge Graph

Author Qingqin Fang (ORCID: 0009–0003–5348–4264)

 Introduction

Large Language Models (LLMs) have transformed the landscape of natural language processing, demonstrating exceptional proficiency in generating text that closely resembles human language.

Hallucination in Large Language Models and Two Effective Alleviation Pathways

How to efficiently retrieve information for different applications

Author Wenyi Pi (ORCID: 0009-0002-2884-2771) This article aims to explore various ways in which Retrieval-Augmented Generation (RAG) can be utilised to retrieve information and generate responses effectively within the dialogue system. The rationale behind utilising RAG as well as potential ways in which it can be employed effectively will be covered.

Efficient Information Retrieval and Response Generation with Retrieval-Augmented Generation (RAG)

Aishwarya Nambissan

Swinburne University of Technology

Amir Aryani

A brief overview of different types of clustering techniques and their algorithms. Authors Aishwarya Nambissan (ORCID: 0009-0003-3823-6609) Amir Aryani (ORCID: 0000-0002-4259-9774)

 Background

Clustering is a fascinating technique used in machine learning, where patterns or data points are grouped based on their similarities. It’s like finding hidden connections among different data points without predefined labels.

19 Clustering Techniques

Exploring the potentials and limitations of Vision Language Models

Author:

 Amanda Kau (ORCID:


 0009–0004–4949–9284


 )

The human brain is more extraordinary than any machine we could build. From an early age, many of us gain the ability to comprehend what our eyes tell us and articulate it. Furthermore, we combine evidence from all our senses to reason.

How Much Can Vision Language Models Really “See”?

Yunzhong Zhang

Efficient creation of a stoplight report with data dashboard images

Author: Yunzhong Zhang (ORCID: 0009–0002–8177–419X) Comparing data dashboards is crucial for understanding trends and performance differences. Traditionally, this task required manual effort, which was slow and sometimes inaccurate. Now, thanks to OpenAI’s GPT-4 with Vision (GPT-4V), we are able to automate and improve this process.

How to use GPT-4V For Stoplight Report

Dhruv Gupta

Improving the performance of Large Language Models Author  Dhruv Gupta (ORCID: 0009-0004-7109-5403)  ChatGPT, which first came out in late 2022, took the world by storm. Since then, various LLM models and LLM based products such as Meta’s Llama and Google’s Gemini have emerged, demonstrating the power of LLMs.

RAG: The next big thing after LLMs?

Harvesting the Power and Knowledge of Large Language Models


 Author

Amanda Kau

 (ORCID:

0009–0004–4949–9284

 )

Introduction Knowledge Graphs are networks that represent data in a graphical format. The beauty of Knowledge Graphs lies in their representation of concepts, events and entities as nodes, and the relationships between them as edges.

Automated Knowledge Graph Construction with Large Language Models

Nakul Nambiar

Zhuochen Wu

Authors:  Nakul Nambiar (ORCID: 0009-0009-9720-9233) Zhuochen Wu (ORCID: 0009-0000-5642-5348)  Research Graph is a structured representation of research objects that captures information about entities and the relationships between Researcher, Organisation, Publication, Grant and Research Data.

How to use GPT API to export a research graph from PDF publications

Recent Advances in Using Machine Learning with Graphs — Part 2 Latest findings in multiple research directions for handling graph construction and network security issues Author · Xuzeng He (

 ORCID:

0009–0005–7317–7426) Introduction A graph, in short, is a description of items linked by relations, where the items of a graph are called nodes (or vertices) and their relations are called edges (or links). Examples of

Recent Advances in Using Machine Learning with Graphs — Part 2

Dhruv Gupta

Beyond Chain-of-Thought, Effective Graph-of-Thought Reasoning in Language Models

Automatic Prompt Engineering - Cobus Greyling - Medium

Prompt Engineering — Part 2


 Using intelligence to use artificial Intelligence: A deep dive into Prompt Engineering

Author Dhruv Gupta

 (ORCID:

0009–0004–7109–5403

 )

Introduction In the previous article we discussed what prompt engineering and some of the techniques used for prompt engineering.

Prompt Engineering — Part 2

Enhancing Open-Domain Conversational Question Answering with Knowledge-Enhanced Models and Knowledge Graphs


 How knowledge-enhanced language models and knowledge graphs are advancing open-domain conversational question answering

Author:

 Wenyi Pi (ORCID:


 0009-0002-2884-2771


 )

When searching for information on the website, it is common to come across a flood of

Enhancing Open-Domain Conversational Question Answering with Knowledge-Enhanced Models and…

Incorporating Knowledge Graphs to explain reasoning processes


 Author

Amanda Kau (

 ORCID:

0009–0004–4949–9284) Introduction Large language models (LLMs) like GPT-4 possess remarkable language abilities, allowing them to function as chatbots, translators, and much more.

Combining Knowledge Graphs With Language Models for Interpretability

Unlocking the power of language models: A deep dive into BERT


 Author:


 Dhruv Gupta (ORCID:


 0009–0004–7109–5403


 )

Clive Humby, in 2006 rightly said, “Data is the new oil”. With data being present everywhere, it has never been more valuable.

Language Models: Deep Dive into BERT

Large Language Models for Fake News Generation and Detection Author Amanda Kau (

 ORCID

: 0009–0004–4949–9284) Introduction In recent years, fake news has become an increasing concern for many, and for good reason. Newspapers, which we once trusted to deliver credible news through accountable journalists, are vanishing en masse along with their writers.

Are Large Language Models Our Allies or Enemies in the Fight Against Fake News?

Attention mechanism not getting enough attention Author Dhruv Gupta (

 ORCID

: 0009–0004–7109–5403) Introduction As discussed in this article, RNNs were incapable of learning long-term dependencies. To solve this issue both LSTMs and GRUs were introduced. However, even though LSTMs and GRUs did a fairly decent job for textual data they did not perform well.

Transformers Models in NLP

From Naive to Modular: Tracing the Evolution of Retrieval-Augmented Generation

Author · Vaibhav Khobragade (

 ORCID:

0009–0009–8807–5982) Introduction Large Language Models (LLMs) have achieved remarkable success.

Three Paradigms of RAG

An introduction to tools supporting PubMed Author: Xuzeng He (

 ORCID:

0009–0005–7317–7426) PubMed is a free Web literature search service developed and maintained by the National Center for Biotechnology Information (

 NCBI

), and it is also a part of NCBI’s Entrez retrieval system that provides access to a diverse set of 38 databases.

Enhancing PubMed: from a Medical Database to beyond

Aland Astudillo

How to use GROBID to extract text from PDF  Author  Aland Astudillo (ORCID: 0009-0008-8672-3168)  GROBID is a powerful and useful tool based on machine learning that can extract text information from PDF files and other files to a structured format. One of the key challenges in knowledge mining from academic articles is reading the content of PDF files.

How to use GROBID

Using Mistral for Data tagging Author · Xuzeng He (

 ORCID:

0009–0005–7317–7426) Introduction Data tagging, in simple terms, is the process of assigning labels or tags to your data so that they are easier to retrieve or analyse.

How to use Large Language Models to tag your data: A complete tutorial

Integrating temporal data into static knowledge graphs

 Author

Amanda Kau

 (ORCID:


 0009–0004–4949–9284


 )

Introduction Knowledge graphs (KGs) have proven to be an effective method of data representation that is increasingly popular. In KGs, entities and concepts are represented as nodes, while the relationships between nodes are depicted as edges.

Dynamic Knowledge Graphs: A Next Step For Data Representation?

Transformative Advances in Language Models through External Knowledge Integration

Author: Qingqin Fang (

 ORCID:

0009–0003–5348–4264)

 Introduction

In the dynamic field of natural language processing, the integration of external knowledge has emerged as a pivotal strategy for enhancing the performance of language models.

Deeper understanding of In-Context Retrieval-Augmented Language Models

Amir Aryani

Author  Amir Aryani (ORCID: 0000-0002-4259-9774) Introduction   In this article we look at Research Graph as an information model , and an approach to connect and capture the connections between research outputs, researchers and research activities. We explore the metadata model, and we discuss how to capture this graph in a Neo4j Graph Database.

Research Graph 101

Latest findings in multiple research directions for tackling reasoning and common sense challenges Author: Xuzeng He (

 ORCID:

0009–0005–7317–7426) Knowledge Graphs, such as Wikidata, contain rich relational information between entities and have been widely used as a structured format for storing and representing relational information.

Large Language Models and Knowledge Graphs: Ways to combine them

Using intelligence to use artificial Intelligence: A deep dive into Prompt Engineering


 Author


 Dhruv Gupta (ORCID:


 0009–0004–7109–5403


 )

Introduction Large Language Models (LLMs) have become the new normal in the field of Natural Language Processing (NLP). With their improved performance and generative power, people around the world are relying on it for

Prompt Engineering

The AI Helper Turning Mountains of Data into Bite-Sized Instructions

Author Aland Astudillo (ORCID: 0009-0008-8672-3168) LLMs have been changing the way the entire world deals with problems and day-by-day tasks. To make them better for specific applications, they need huge amounts of data and complex and expensive approaches to training them.

What is LLMLingua?

Boosting Performance for Knowledge Graphs with Neo4j APOC Library

 Author

Wenyi Pi (ORCID: 0009–0002–2884–2771) Introduction A

 knowledge graph

(graph database) captures information about main entities in a domain and the relationships between them. It was an augmented feature store for connected data which gave access to compute, access and operationalise structure features.

Exploring Methods of Cypher Query Optimisations

Refining AI Vision: How Retrieval-Augmented Generation Transforms Image Captioning in Large Language Models   Leveraging External Knowledge to Enhance the Descriptive Capabilities of AI Systems Author  Vaibhav Khobragade (ORCID: 0009–0009–8807–5982) Introduction   Large Language Models (LLMs) are artificial intelligence models that are trained on massive amounts of text data in order to generate human-like language and produce coherent

Refining AI Vision: How Retrieval-Augmented Generation Transforms Image Captioning in Large…

Authors  Nakul Nambiar (ORCID: 0009-0009-9720-9233) Amir Aryani (ORCID: 0000-0002-4259-9774)  Knowledge graphs, which offer a structured representation of data and its relationships, are revolutionising how we organise and access information.

Mini RAG using Neo4j

Tools and Platform for Integration of Knowledge Graph with RAG pipelines.

Authors Aland Astudillo (ORCID: 0009-0008-8672-3168) Aishwarya Nambissan (ORCID: 0009-0003-3823-6609) Many users of chatbots such as ChatGPT, have encountered the problem of receiving inappropriate or incompatible responses. There are several reasons why this might happen.

Unveiling the Synergy: Retrieval Augmented Generation (RAG) Meets Knowledge Graphs

Author  Amir Aryani: (ORCID: 0000-0002-4259-9774) Definition   A research collaboration network is a group of researchers, and practitioners, or both, working together on joint research activities. These networks often span across disciplines, geographic boundaries, and sectors, enabling participants to share resources, expertise, and data to address common research goals more effectively than they could individually.

Research Collaboration Network

A novel approach to improving the efficiency of text search in graph databases utilizing Neo4j, OpenAI, and Typesense. Authors  Nakul Nambiar (ORCID: 0009–0009–9720–9233) Aishwarya Nambissan (ORCID: 0009–0003–3823–6609)  The ability to use cutting-edge tools and frameworks is essential for staying ahead in the ever-changing field of technology.

Typesense and Neo4j in a hybrid information retrieval solution

Improving the performance and application of Large Language Models

Author Amanda Kau (ORCID: 0009-0004-4949-9284) Large language models (LLMs) like GPT-4, the engine of products like ChatGPT, have taken centre stage in recent years due to their astonishing capabilities. Yet, they are far from perfect.

Understanding Retrieval Pitfalls: Challenges Faced by Retrieval Augmented Generation (RAG) models

Exploring the Potential of Temporal Feature-Logic Embedding (TFLEX) in Complex Query Resolution

Author · Vaibhav Khobragade (

 ORCID:

0009–0009–8807–5982)

 Introduction

Artificial intelligence (AI) and knowledge representation in the field of temporal knowledge graphs are rapidly gaining interest.

Harnessing Temporal Dynamics: Advanced Reasoning using Temporal Knowledge Graphs

Exploring innovative Strategies in Combating Misinformation with Enhanced Multimodal Understanding Author Wenyi Pi (

 ORCID

: 0009–0002–2884–2771) Introduction Misinformation refers to false or inaccurate information that is often given to someone in a deliberate attempt to make them believe something that is not true. This has a significantly negative impact on public health, political stability and social trust and harmony.

Multimodal Large Language Models for Misinformation Detection and Reasoning

The Three Oldest Pillars of NLP

Author Dhruv Gupta (

 ORCID

: 0009–0004–7109–5403) Introduction Natural Language Processing (NLP) has almost become synonymous with Large Language Models (LLMs), Generative AI, and fancy chatbots. With the ever-increasing amount of textual data and exponential growth in computational knowledge, these models are improving every day.

RNNs vs GRUs vs LSTMs

A Unified and Collaborative Framework for LLM Author · Qingqin Fang (

 ORCID:

0009–0003–5348–4264) Introduction In today’s rapidly evolving field of artificial intelligence, large language models (LLMs) are demonstrating unprecedented potential. Particularly, the Retrieval-Augmented Generation (RAG) architecture has become a hot topic in AI technology due to its unique technical capabilities.

RAG 2.0 is Coming?

Bridging Human Perception and AI’s Future: The Convergence of Visual Understanding and Semantic Networks Author · Vaibhav Khobragade (

 ORCID:

0009–0009–8807–5982)

 Introduction

The fusion of Vision-Language Models (

 VLMs

), Generative Models, and Knowledge Graphs (

 KGs

) is reshaping how artificial intelligence (AI) understands and interacts with the world.

Tuning Vision-Language Models and Generative Models with Knowledge Graph

Understanding the Power and Applications of Natural Language Processing

 Author

Dhruv Gupta (

 ORCID:

0009–0004–7109–5403) Introduction We are living in the era of generative AI. In an era where you can ask AI models almost anything, they will most certainly have an answer to the query. With the increased computational power and the amount of textual data, these models are bound to improve their performance.

NLU vs. NLG: Unveiling the Two Sides of Natural Language Processing

Understanding how RNNs work and its applications


 Author

Wenyi Pi (

 ORCID

: 0009–0002–2884–2771) Introduction In the ever-evolving landscape of artificial intelligence (AI), bridging the gap between humans and machines has seen remarkable progress. Researchers and enthusiasts alike have tirelessly worked across numerous aspects of this field, bringing about amazing advancements.

An Introduction to Recurrent Neural Networks (RNNs)

Exploring the Boundaries of Creativity and Responsibility in the Age of AI-Driven Media

 Author

Vaibhav Khobragade (

 ORCID:

0009–0009–8807–5982)

 Introduction

In 2024, the discipline of Generative AI takes a big step forward with the launch of revolutionary models that convert text into dynamic films, altering the landscape of digital content creation.

Generative AI’s Leap into Video in 2024 and its Ethical Horizon

Addressing Misinformation, Bias, and Privacy Concerns

Author:

 Wenyi Pi (ORCID:


 0009–0002–2884–2771


 )

Recently, the spread of false or misinformation has become an important concern for people all over the world, especially in the era of big data and multimedia, where people are faced with a flood of information that makes it difficult to determine its authenticity.

The Ethical and Social Impact of Artificial Intelligence

Latest findings for the use of Knowledge Graph in the field of QA in multiple research directions Author: Xuzeng He (

 ORCID:

0009–0005–7317–7426) Question Answering (QA), the ability to interact with data using natural language questions and obtaining accurate results, has been a long-standing challenge in computer science dating back to the 1960s.

Question Answering enhanced by Knowledge Graph

Unlocking the power of knowledge graphs in research catalogues: A deep dive into OpenAlex


 Author:


 Dhruv Gupta (ORCID:


 0009–0004–7109–5403


 )

Clive Humby, in 2006 rightly said, “Data is the new oil”. With data being present everywhere, it has never been more valuable.

Knowledge Graphs: Deep Dive into OpenAlex

Techniques to integrate Knowledge Graphs into Language Models

 Author

Amanda Kau

 (ORCID:


 0009–0004–4949–9284


 )

Introduction Both knowledge graphs (KGs) and pre-trained language models (PLMs) have gained popularity due to their ability to comprehend world knowledge and their broad applicability.

A Deep Dive Into Knowledge Graph Enhanced Pre-trained Language Models

Automated Knowledge Graph Construction with Large Language Models — Part 2

 Harvesting the Power and Knowledge of Large Language Models

Author Amanda Kau (

 ORCID

:

 0009–0004–4949–9284

) Introduction Knowledge graphs (KGs) are a structured representation of data in a graphical format, in which entities are represented by nodes and are connected by edges representing relationships

Automated Knowledge Graph Construction with Large Language Models — Part 2

Understanding the Evolutionary Journey of LLMs Author Wenyi Pi (

 ORCID

: 0009–0002–2884–2771) Introduction When we talk about large language models (LLMs), we are actually referring to a type of advanced software that can communicate in a human-like manner. These models have the amazing ability to understand complex contexts and generate content that is coherent and has a human feel.

Brief Introduction to the History of Large Language Models (LLMs)

An improvement architecture superior to the Transformer, proposed by Meta Author · Qingqin Fang (

 ORCID:

0009–0003–5348–4264) Introduction Recently, researchers from Meta and the University of Southern California have introduced a model called Megalodon. They claim that this model can expand the context window of language models to handle millions of tokens without overwhelming your memory.

The longer the context, the better? Unlimited Context Length in Megalodon

Latest findings in pre-training graphs and using them for link recommendation Author · Xuzeng He (

 ORCID:

0009–0005–7317–7426) Introduction A graph, in short, is a description of items linked by relations, where the items of a graph are called nodes (or vertices) and their relations are called edges (or links). Examples of graphs can include social networks (e.g. Instagram) or knowledge graphs (e.g. Wikipedia). In Instagram

Working with Graphs: Pre-training and Application

Unlocking the Future of AI: The Transformative Journey of Large Language Models Author · Vaibhav Khobragade (

 ORCID:

0009–0009–8807–5982)

 Introduction

Human language development is innate and evolves throughout life. Machines lack this ability to evolve without advanced AI algorithms.

The Journey of Large Language Models: Evolution, Application, and Limitations

Understanding Knowledge Networks From A Graph Perspective


 Author:


 Amanda Kau (ORCID:


 0009–0004–4949–9284


 )

Since 2020, over ten million scholarly articles have been published annually. To put that into perspective, say all ten million articles were released on the first day of the year.

Uncovering The Secrets Behind The Most Influential Scholarly Publications With AceMap

Understanding the Balance between Internal Knowledge and External Sources

Author Qingqin Fang (

 ORCID:

0009–0003–5348–4264) Introduction Previous research often emphasized the limitations of LLM’s information acquisition pathways, focusing on enhancing its capabilities in this regard.

When RAG and LLM Conflict: Who Will AI Listen to?

Solutions to Enhance LLM Performance in Long Contexts Author · Qingqin Fang (

 ORCID:

0009–0003–5348–4264) Introduction In the era of AI breakthroughs, large language models (LLMs) are not just advancements; they are revolutions, transforming how we interact with technology, from casual conversations with chatbots to the intricate mechanisms behind sophisticated data analysis tools.

Navigating the Long Context Conundrum: Challenges in Language Models’ Information Processing

Supervised Fine-tuning, Reinforcement Learning from Human Feedback and the latest SteerLM Author · Xuzeng He (

 ORCID:

0009–0005–7317–7426) Introduction Large Language Models (LLMs), usually trained with extensive text data, can demonstrate remarkable capabilities in handling various tasks with state-of-the-art performance. However, people nowadays typically want something more personalised instead of a general solution.

Fine-tuning Large Language Models: A Brief Introduction

An Overview of Constructing a Knowledge Graph Author · Qingqin Fang (

 ORCID:

0009–0003–5348–4264) 1. Introduction 1.1 What is a Knowledge Graph Knowledge Graphs are structured semantic knowledge bases used to rapidly describe concepts and their relationships in the physical world.

Unlocking Intelligence: The Journey from Data to Knowledge Graph

Author Dhruv Gupta (

 ORCID

: 0009–0004–7109–5403) Introduction Large Language Models (LLMs) have become the new face of Natural language processing (NLP). With their generative power and ability to comprehend human language, the human reliance on these models is increasing every day. However, the LLMs have been known to hallucinate and thus produce wrong outputs.

Prompt engineering: A Way to Smartly Use AI

Latest effort in assessing the security of the code generated by large language models Author · Xuzeng He (

 ORCID:

0009–0005–7317–7426) Introduction With the surge of Large Language Models (LLMs) nowadays, there is a rising trend among developers to use Large Language Models to assist their daily code writing. Famous products include GitHub Copilot or simply ChatGPT.

Large Language Models for Code Writing: Security Assessment

Stories by Research Graph on Medium

#### Enhancing Data Interactivity with LLMs and Neo4j Knowledge Graphs

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*hBJpzdvxM22NoEIO" />
<figcaption>Image generated in DALL-E 2.</figcaption>
</figure>

### Author

- Wenyi Pi (**ORCID**:
 [0009--0002--2884--2771](https://orcid.org/0009-0002-2884-2771))

### Introduction

Since OpenAI launched ChatGPT, a **large language model (LLM)** based
chatbot, in 2023, it has set off a technological wave. These models,
capable of understanding and generating human-like text, can adapt to
various conversational contexts and address questions across a broad
range of topics. They even simulate human actions, revolutionising the
way humans interact with machines. People now use these models for tasks
like summarising articles, rewriting emails, question answering etc
sparking a new wave of applications in artificial intelligence (AI).
Nonetheless, these LLMs come with challenges such as generating
inaccurate information (hallucinations), a lack of explainability and
domain-specific limitations which can hinder their practical
application. To address these issues, **Knowledge graphs**, known for
their complex analytical capabilities, structured representations, and
semantic query functionalities, can be integrated with LLMs. This
integration enhances the intelligence of AI systems, leading to more
precise and reliable outputs.

LLMs refer to large, general-purpose language processing models that are
first pre-trained on extensive datasets and then fine-tuned for specific
applications. To learn more about LLMs, please read the article [Large
Language
Models](https://medium.com/@researchgraph/brief-introduction-to-the-history-of-large-language-models-llms-3c2efa517112).
A **knowledge graph** captures information about main entities in a
domain and the relationships between them**. Neo4j** is a graph database
management system. It embraces a graph-based model, organising data base
nodes, properties and relationships. To learn more about Neo4J and
Cypher (the programming language supported in Neo4j), please check this
article [Neo4j](https://medium.com/@researchgraph/exploring-methods-of-cypher-query-optimisations-7b03e08b24c4).

A standard approach to help integration of LLMs and Knowledge Graphs is
known as Retrieval-Augmented Generation (RAG). We have talked about
various ways in which RAG can be utilised to retrieve information and
generate responses effectively in the
article [RAG](https://medium.com/@researchgraph/efficient-information-retrieval-and-response-generation-with-retrieval-augmented-generation-rag-e46c2282b7c2).

In this article, we would like to explore the integration of knowledge
graphs, specifically with Neo4j and LLMs and their applications.

### Framework

Instead of having users interact directly with LLMs in isolation, we
guide the LLM to retrieve data from Neo4j's knowledge graph when a
question is posed. This approach allows the model to provide a richer
and more accurate response. By leveraging Neo4j's knowledge graph, we
not only reduce instances of hallucination where the model might
generate misleading or irrelevant information, but also enhance the
explainability and traceability of the responses. This makes Neo4j a
particularly robust choice compared to other solutions for integrating
RAG techniques.

To fully leverage the benefits of Neo4j within LLM applications, a
structured approach is employed as the image below:

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*dIIDEqzF2aWTaS5i" />
<figcaption>Framework of integration LLMs and Neo4j. Source:<a
href="https://neo4j.com/labs/genai-ecosystem/">https://neo4j.com/labs/genai-ecosystem/</a></figcaption>
</figure>

- **Data Retrieval**: When a user queries using natural language, the
 LLM interacts with Neo4j to fetch relevant data, grounding the
 responses in verified information and reducing the likelihood of
 inaccuracies.
- **Response Accuracy and Reliability**: By integrating knowledge
 graphs, the responses generated by LLMs are not only more accurate but
 also include explanations that trace back to concrete data entities
 and its relevant relationships, enhancing their reliability and
 usefulness in practical applications.

### Use Case

#### LLM Graph Builder

The **Neo4j LLM Knowledge Graph Builder** is designed to convert
unstructured text from various sources such as PDFs, YouTube videos, and
webpages into a structured knowledge graph. This tool supports a diverse
range of inputs including platforms like YouTube and Wikipedia, as well
as cloud storage services such as Amazon S3 and Google Cloud Storage
(GCS). Additionally, users can also directly upload PDFs files or other
file types, enhancing the flexibility and breadth of data that can be
processed. It employs advanced machine learning models like Google
Gemini, Diffbot, and OpenAI GPT to transform textual content into a
network where entities become nodes and their relationships are
represented as links. To explore this application further, you can
access it through this [graph
builder](https://llm-graph-builder.neo4jlabs.com/) link. This tool is
ideal for creating detailed knowledge graphs from multiple data sources

The process involves dividing documents into segments, generating
embeddings for each segment, and then employing LLMs to identify and
extract entities and their relationships from these segments. For
instance, I process text from Wikipedia about Neo4j utilising the model
OpenAI's GPT-4. The result indicated that from the Wikipedia content, we
successfully extracted 24 nodes and 16 relationships.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*HErAja1ks9BoQSrv" />
<figcaption>GraphBuilder Demo. Source created through: <a
href="https://llm-graph-builder.neo4jlabs.com/">https://llm-graph-builder.neo4jlabs.com/</a></figcaption>
</figure>

To visualise the graph, simply connect it to a Neo4j database. This
allows for effectively displaying and analysing the relationships and
entities within the knowledge graph.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*szPgGkbjzyyE_Slx" />
<figcaption>Visualise Result. Source created through: <a
href="https://llm-graph-builder.neo4jlabs.com/">https://llm-graph-builder.neo4jlabs.com/</a></figcaption>
</figure>

#### GraphRAG

The GraphRAG system utilises a knowledge graph derived from [SEC
filings](https://en.wikipedia.org/wiki/SEC_filing) (a financial
statement or other formal document submitted to the U.S. Securities and
Exchange Commission (SEC)), enriched with additional company data using
Neo4j's vector index and
[LangChain](https://en.wikipedia.org/wiki/LangChain) (a framework
designed to simplify the creation of applications using LLMs) to provide
a RAG chatbot. This chatbot can address queries about SEC filings and
provide supplementary information about the company. For a practical
demonstration, please refer to this
[GraphRAG Demo](https://neo4j-rag-demo-yvpuwtfmva-ue.a.run.app/).

To construct a Neo4j knowledge graph, the information is organised into
nodes and relationships. For unstructured data like 10-K filings, the
approach involves creating "document" nodes. Each document node
corresponds to a segment of text from a 10-K filing, utilising an
embedding model from the GenAI service to generate vectors for these
nodes, which later facilitates vector search capabilities. For
semi-structured Form 13 filings, the system employs Cypher templates to
input the data in a structured manner, creating "company" and "manager"
nodes and building an "owns" relationship between them while linking
back to the document. Additionally, an alternative method for ingesting
Form 13 uses named entity recognition powered by a LLM from the GenAI
service provider, offering a more robust way to handle filings that may
be malformed, inconsistent, or semi-structured XML.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*CdDhabrOfCLeLEfI" />
<figcaption>GraphRAG Framework. Source: <a
href="https://neo4j.com/labs/genai-ecosystem/rag-demo/">https://neo4j.com/labs/genai-ecosystem/rag-demo/</a></figcaption>
</figure>

#### NeoConverse

NeoConverse is an application for making graph databases accessible to
non-technical users through natural language queries. It utilises LLMs
to interpret user questions and translate them into Cypher queries of
graph databases Neo4j. This process involves leveraging the graph
database schema, sample question-statements and model fine-tuning to
ensure accuracy and relevance. Once a query is formulated, it goes
through validation before execution against the database. The results of
this query are then fed back into the LLM, which crafts a response in
natural language to convey the findings clearly to the user.
Alternatively, NeoConverse can configure the LLM to use the query
results to generate data and configuration for visual representations,
such as charts, enhancing the interpretability and presentation of the
information.

The process of NeoConverse:

1. **Questioning**: Users ask a natural language question about their
 enterprise data stored in Neo4j.
2. **Contextual Setup**: NeoConverse prepares the LLM by providing it
 with the schema information that is stored in the Neo4j database and
 a few relevant data examples to guide the model.
3. **Query Generation**: The LLM uses the provided context to generate
 a Cypher query, which is used to retrieve the specific information
 as required.
4. **Query Execution**: The query is refined, formatted, and executed
 against the Neo4j database to fetch the relevant data.
5. **Response Creation**: Using the retrieved data, the LLM crafts a
 detailed and accurate response based on the original query, ensuring
 that the answer is directly relevant to the user's data.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*NjhaaJKkhc7bUKG3" />
<figcaption>NeoConverse Pipeline. Source: <a
href="https://neo4j.com/labs/genai-ecosystem/neoconverse/">https://neo4j.com/labs/genai-ecosystem/neoconverse/</a></figcaption>
</figure>

By leveraging internal data, NeoConverse empowers LLMs to provide more
accurate and relevant answers to user queries within an enterprise
setting.

Main functions of NeoConverse

1. **Customisable Neo4j Database Agents**: Users can connect their own
 Neo4j databases to NeoConverse, transforming these databases into
 conversational agents that can be interacted with in plain English.
 This includes configuring schema information to provide context for
 the LLM, adding few-shot learning examples for enhanced model
 understanding, and dynamically interacting with the database through
 a natural language interface.
2. **Visual Data Representation**: Beyond textual interactions,
 NeoConverse supports generating chart visualisations from natural
 language queries. This allows users to not only ask questions but to
 see their data represented visually, aiding in better data
 comprehension and presentation.

#### GenAI Stack

The GenAI Stack is a collaborative tool launched at DockerCon 2023,
integrating technologies from Docker, Neo4j, LangChain, and Ollama. It's
designed to enhance data handling and AI capabilities, particularly in
managing and querying large datasets effectively.

<figure>
<img src="https://cdn-images-1.medium.com/max/591/0*1Pfoc7BJVwYXRt0p" />
<figcaption>GenAI Stack Pipeline. Source: <a
href="https://neo4j.com/labs/genai-ecosystem/genai-stack/">https://neo4j.com/labs/genai-ecosystem/genai-stack/</a></figcaption>
</figure>

The workflow of GenAI Stack is as follows: When a user asks a question,
the system initially uses a vector index for similarity searches to find
documents or data entries that most closely match the user query based
on semantic similarities. This task is managed by a Local LLM equipped
with Ollama (a powerful and user-friendly platform for running LLMs on
local machines) technology to process and understand natural language
inputs effectively. The relevant documents are then fetched from a
structured Neo4j database, which organises data using a combination of a
vector index and a knowledge graph. The LLM analyses these documents to
generate an accurate and contextually relevant response, which is
subsequently delivered to the user.

Key Features of GenAI Stack:

1. **Setup**: It allows the setup of local or remote LLMs, Neo4j
 databases, and LangChain demo applications using Docker.
2. **Model Integration**: Users can pull Ollama models and sentence
 transformers as required.
3. **Data Management**: The stack supports the import of
 StackOverflow's Questions and Answers by tags and enables the
 creation of knowledge graphs and vector embeddings for these items.
4. **Embedding Creation**: It can create Knowledge Graph and vector
 embeddings for questions and answers
5. **Application Development**: The stack features a Streamlit Chat App
 with capabilities for vector search and GraphRAG answer generation,
 tailored for enhancing user interaction.
6. **Support Ticket Generation**: It can create "Support Tickets" from
 unanswered questions on StackOverflow, considering the relevance and
 quality of existing questions.
7. **PDF Interaction**: The PDF chat feature supports loading PDFs,
 chunking text, indexing vectors, and searching to generate answers.
8. **Software Architecture**: It includes a Python backend and a Svelte
 front-end for developing robust chat applications with vector search
 and GraphRAG integration.

### Conclusion

In conclusion, the integration of Large Language Models (LLMs) with
Neo4j knowledge graphs represents a significant advancement in
leveraging artificial intelligence to enhance data interactivity and
precision. This combination not only addresses the limitations
associated with LLMs, such as generating inaccurate information
(hallucinations) and a lack of explainability, but it also maximises the
strengths of both technologies. By grounding LLM responses in
structured, queryable data from Neo4j, we can achieve more reliable,
contextually accurate, and accessible outputs.

However, this integration is not without its challenges. A primary
concern is the increased complexity and computational demand required to
merge these sophisticated technologies successfully. Furthermore, the
effectiveness of this integration is not universally applicable as it
heavily depends on the quality, depth, and breadth of the knowledge
graph. In scenarios where the knowledge base is limited or skewed
towards specific areas, the performance of integrations with graph-based
knowledge graphs like Neo4j might not exceed that of traditional RAG
methods, and the associated costs could be considerably high, given that
organising a knowledge graph demands substantial time and resources.
Despite these challenges, this innovative integration promises to
enhance AI systems' ability to mirror human cognition and discovery more
effectively.

### References

- GenAI Ecosystem --- Neo4j Labs (2023) Neo4j Graph Data Platform.
 Available at: <https://neo4j.com/labs/genai-ecosystem/>
- Hane, O. (2023) Harnessing Large Language Models with Neo4j, Neo4j
 Developer Blog. Available at:
 <https://medium.com/neo4j/harnessing-large-language-models-with-neo4j-306ccbdd2867>.

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe67245bde28){width="1"
height="1"}

#### **From Naive to Modular: Tracing the Evolution of Retrieval-Augmented Generation**

<figure>
<img src="https://cdn-images-1.medium.com/max/938/0*IQgdTaKG50F_nbGM" />
<figcaption>Created by DALLE</figcaption>
</figure>

### Author

· Vaibhav Khobragade (**ORCID:**
[0009--0009--8807--5982](https://orcid.org/0009-0009-8807-5982))

### Introduction

Large Language Models (LLMs) have achieved remarkable success. But, they
still face significant limitations, especially in domain-specific or
knowledge-intensive tasks such as question answering, producing
"hallucinations" where the models generate responses that sound
plausible but are actually incorrect when handling queries beyond their
training data or requiring current information.

To overcome these challenges, [Retrieval-Augmented Generation
(RAG)](https://medium.com/@researchgraph/refining-ai-vision-how-retrieval-augmented-generation-transforms-image-captioning-in-large-1a74499a370d)
enhances LLMs by retrieving relevant document chunks from external
knowledge bases through semantic similarity calculation. By referencing
external knowledge, RAG effectively reduces the problem of generating
factually incorrect content. Its integration into LLMs has resulted in
widespread adoption, establishing RAG as a key technology not just in
advancing chatbots, but also in vision language models to support
generating responses for real-world applications.

RAG instance when applied to question-answering tasks follows
this process:

<figure>
<img src="https://cdn-images-1.medium.com/max/940/0*rbHdvUc4YFFKs0SD" />
<figcaption>Figure 1: An example of how the Retrieval-Augmented
Generation (RAG) process is used for answering questions. (Gao et al.,
2023) Article link: <a
href="https://arxiv.org/abs/2312.10997">https://arxiv.org/abs/2312.10997</a></figcaption>
</figure>

The diagram consists of three main steps. First, documents are divided
into smaller parts, converted into vectors, and stored in a database.
Second, the system selects the top k most relevant chunks based on the
question's meaning. Third, the original question and selected chunks are
inputted into a large language model to produce the answer.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1016/0*uEXFltuz-jl29heb" />
<figcaption>Figure 2. Technology tree of RAG research. The stages of
involving RAG mainly include pre-training, fine-tuning, and inference.
(Gao et al., 2023) Article link: <a
href="https://arxiv.org/abs/2312.10997">https://arxiv.org/abs/2312.10997</a></figcaption>
</figure>

RAG technology has rapidly developed in recent years, and the technology
tree summarizing related research is shown above in Figure 2. Initially,
RAGs emerged with Transformer architecture, focusing on enhancing
language models by incorporating additional knowledge through
PreTraining Models (PTM). This early stage was characterized by
foundational work aimed at refining pre-training techniques. The
subsequent arrival of ChatGPT marked a pivotal moment, with LLMs
demonstrating powerful in-context learning (ICL) capabilities. As
research progressed, the enhancement of RAG was no longer limited to the
inference stage but began to be incorporated more with LLM fine-tuning
techniques.

In this article, we review the state-of-the-art RAG method and describe
its evolution through paradigms including Naive RAG, Advanced RAG, and
Modular RAG.

### **Naive RAG**

This is the earliest method of RAG to get prominence shortly after the
advent of ChatGPT. The Naive RAG follows a traditional process that
includes indexing, retrieval, and generation.

Let's understand Naïve RAG with an example.

**Scenario:** Suppose a user asks an LLM, "What are the health benefits
of drinking green tea?"

**Without Naive RAG:** The LLM might generate a response based solely on
its pre-trained knowledge, which could be outdated or incomplete. It
might say, "Green tea is rich in antioxidants and can help in improving
brain function."

**With Naive RAG:**

1\. **Indexing:** It begins by cleaning and extracting raw data about
green tea from various formats like PDF, HTML, Word, and Markdown. This
data is converted into plain text for consistency. To address the
context limitations of language models, the text is divided into
smaller, manageable chunks. These chunks are then turned into vectors
using an embedding model and stored in a vector database. This step is
important for enabling quick and accurate searches in the retrieval
phase and providing up-to-date research and ***articles about green tea
are indexed.***

2\. **Retrieval:** When the RAG system gets a user question, it uses the
same encoding model as before to turn the question into a vector. Then,
it compares this vector to the vectors of the indexed text chunks. The
system picks the top K chunks that are most similar to the question.
These chunks become the extra context for generating the response.
***For example: It retrieves closely related chunks to "health benefits
of green tea."***

3\. **Generation:** The user question and chosen documents are combined
into a clear prompt. The model might answer differently based on
specific task rules. It can use its built-in knowledge or stick only to
the given documents. If it's part of a conversation, previous chat
history can be added to the prompt, helping the model engage better in
multi-turn dialogues. ***For instance, it might retrieve a recent study
stating, "Green tea consumption is linked to reduced risk of heart
disease," which wasn't part of its initial training data.***

<figure>
<img src="https://cdn-images-1.medium.com/max/346/0*uvgeymdjpJec0jk3" />
<figcaption>Figure 3 Naïve RAG Pipeline (Gao et al., 2023) Article link:
<a
href="https://arxiv.org/abs/2312.10997">https://arxiv.org/abs/2312.10997</a></figcaption>
</figure>

However, Naive RAG encounters notable drawbacks:

**Retrieval Challenges --- **that lead to the selection of misaligned or
irrelevant chunks, and the missing of crucial information.

**Generating Challenges --- **the model might struggle with
hallucination and have problems with relevance, toxicity, or bias in its
outputs, which can make the responses less reliable and of lower quality

For instance: The Naive RAG model retrieves relevant documents and
provides them to the LLM. However, the retrieved information may not be
relevant or up to date. ***"Recent studies suggest that new medications
and therapies are being developed to slow the progression of
Alzheimer's."***

### **Advanced RAG**

Advanced RAG introduces specific improvements to overcome the
limitations of Naive RAG. Focusing on enhancing retrieval quality, it
employs pre-retrieval and post-retrieval strategies.

<figure>
<img src="https://cdn-images-1.medium.com/max/375/0*1PrMV9yHEE1eJG0b" />
<figcaption>Figure 4. Advanced RAG Pipeline</figcaption>
</figure>

**Scenario**: Consider a user asking, "What recent advances have been
made in the treatment of Alzheimer's disease?"

**With Advanced RAG:** The following procedures are followed:

**Pre-retrieval process:** The main aim is to improve the indexing
structure and the user's query. Indexing optimization focuses on making
the indexed content better. This includes strategies like improving data
details, organizing indexes better, adding extra information, aligning
things correctly, and using mixed retrieval methods, where information
from external sources is combined with the indexed content for retrieval
purposes. Query optimization aims to make the user's original query
clearer and more suitable for retrieval. Common methods include query
rewriting and query transformation & expansion.

For instance: Using the Alzheimer's example, Advanced RAG use a refined
indexing strategy where documents are not only indexed by content but
also by recency and relevance to medical advancements, using metadata
enhancements. Utilizing enhanced query transformation techniques, the
system precisely retrieves the most current and relevant research papers
discussing the latest treatment breakthroughs, rather than broader
articles on Alzheimer's.

**Post-Retrieval Process:** After retrieving relevant information, it is
essential to effectively combine it with the original query. The main
steps include re-ranking to highlight the most important content and
compressing the context to avoid information overload as retrieved
documents are then re-ranked to prioritize those that specifically
mention recent clinical trials or FDA approvals, ensuring that the most
pertinent information is considered first.

**Context compression:** Redundant or less relevant information is
filtered out to focus on key findings from the most authoritative
sources.

So, the result could be: "The latest breakthrough in Alzheimer's
treatment includes the development of a new drug, Aducanumab, which has
shown potential in reducing cognitive decline in early-stage patients,
according to a 2021 FDA report."

### **Modular RAG**

The modular RAG architecture advances beyond the former two RAG
paradigms, offering enhanced adaptability and versatility. It
incorporates diverse strategies for improving its components, such as
adding a **search module** for similarity searches and refining the
retriever through **fine-tuning**. Modular RAG builds upon the
foundational principles of Advanced and Naive RAG, illustrating a
progression and refinement within the RAG family.

<figure>
<img src="https://cdn-images-1.medium.com/max/701/0*JBAhcxxaIuT8DI0m" />
<figcaption>Figure 5 Modular RAG Pipeline</figcaption>
</figure>

**Modules:** The Modular RAG framework introduces additional specialized
components to enhance retrieval and processing capabilities.

Consider the scenario: ***"What are the latest research findings on the
health benefits of green tea?"***

1. **Search Module:** When the question is posed, this module activates
 and uses LLM-generated queries to search through various databases
 and knowledge graphs specifically for the latest research papers or
 articles on green tea.
2. **RAG-Fusion:** This module takes the initial user query and expands
 it into multiple queries. For instance, it might generate additional
 queries like "recent clinical trials on green tea" and "green tea
 effects on mental health." These expanded queries are then used to
 search for a wide range of relevant information.
3. **Memory Module:** As articles and papers are retrieved, this module
 helps by recalling previously retrieved documents on similar topics
 to improve the relevance and depth of the current search, using the
 system's built-in memory of past queries and their results.
4. **Routing:** This module analyzes the incoming data and decides the
 best path for processing it. For example, it might route the query
 to specific scientific databases for academic papers rather than
 general websites.
5. **Predict Module:** To ensure the retrieved information isn't
 redundant and is relevant, this module generates a synthesized
 context from the retrieved data, focusing only on the most pertinent
 information like key findings or new insights about green tea.
6. **Task Adapter Module:** Tailors RAG to specific tasks by automating
 prompt retrieval for zero-shot inputs and creating task-specific
 retrievers. For example, this module recognizes that the query is
 about health benefits, a specific knowledge domain. It adjusts the
 retrieval process to focus on the most credible medical and
 health-oriented sources and might generate specialized prompts to
 refine further retrieval.

#### **Patterns**

RAG uses various flexible and interchangeable modules to enhance the
performance of LLMs as follows:

1. **Module Interaction and Adjustment:** These are methods in Modular
 RAG that show how different parts (modules) of the system work
 together more dynamically. DSP (Demonstrate-Search-Predict) and
 ITERRETGEN (Iterative Retrieve-Read-Retrieve-Read) use the outputs
 from one module to improve or inform the function of another,
 enhancing the overall synergy and effectiveness. Example Patterns:
 [**DSP**](https://doi.org/10.48550/arXiv.2212.14024) and
 [**ITERRETGEN**](https://doi.org/10.48550/arXiv.2305.15294)
2. **Flexible Orchestration:** These methods demonstrate the
 adaptability of Modular RAG, where the system can adjust its
 retrieval processes based on what's needed for a specific scenario.
 This flexibility means the system isn't stuck to a fixed way of
 retrieving information but can adapt based on the requirements of
 the task at hand. Example Techniques**:**
 [**FLARE**](https://arxiv.org/pdf/2312.10997)
 and [**Self-RAG**](https://doi.org/10.48550/arXiv.2310.11511)
3. **Integration with Other Technologies:** Modular RAG can also easily
 work with other advanced technologies. For example, it can improve
 retrieval accuracy by fine-tuning the retrieval part of the system,
 personalize outputs by fine-tuning the generation process or even
 use reinforcement learning to enhance both processes
 collaboratively. Examples are **Fine-Tuning** and **Reinforcement
 Learning:**

### **Comparison of RAG with other model optimization methods**

<figure>
<img
src="https://cdn-images-1.medium.com/max/1022/0*vkYF69LpphrOyfhC" />
<figcaption>Figure 6 Compares Retrieval-Augmented Generation (RAG) with
other model optimization methods, specifically Fine-tuning and Prompt
Engineering.</figcaption>
</figure>

The diagram, presented as a quadrant chart, illustrates the differences
based on two dimensions: "External Knowledge Required" and "Model
Adaptation Required."

1. **Prompt Engineering**: Requires minimal model modifications and
 external knowledge, leveraging the inherent capabilities of LLMs.
2. **Fine-tuning (FT)**: Involves further training of the model to
 internalize knowledge, making it suitable for tasks that require
 specific structures, styles, or formats.
3. **RAG**: RAG typically requires less training of the model itself
 compared to methods like Fine-tuning. Instead, RAG heavily utilizes
 external, updated knowledge sources to enhance the model's output.
 This allows RAG to provide more accurate and current information by
 integrating the latest data from outside databases and texts,
 without the need for extensive retraining of the model.

### **Future Works**

The key future developments highlighted for RAG are as follows:

**Optimization of Retrieval Efficiency:** Future research can focus on
ways of accelerating the retrieval process more effectively for large
knowledge bases. Optimization of indexing and retrieval of data can be
done to reduce latency and increase the responsiveness to the maximum.

**Multi-Modal RAG:** Take RAG to the next level by making it work with
not only text but other modalities like images, audio, and video. Plenty
of innovations have come through for Multi-Modal RAG already, but more
research in this area will always be helpful.

**Operational and Production Efficiency:** Demonstrate and enhance the
practical deployment of RAG systems within the production environment
with improved management of computational resources, and data security
assured in commercial applications.

### **Conclusion**

This article discusses RAG, a method that enhances LLMs by integrating
external knowledge to improve response accuracy, especially for complex,
real-world applications. RAG reduces factual inaccuracies by accessing
up-to-date databases, supporting various RAG paradigms like Naive,
Advanced, and Modular RAG, each offering specific improvements and
flexibility for task-specific applications.

### **References**

- Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun,
 J., Wang, M., & Wang, H. (2023, December 18). *Retrieval-Augmented
 Generation for Large Language Models: A Survey*. arXiv.org.
 <https://arxiv.org/abs/2312.10997>
- Peng, W., Li, G., Jiang, Y., Wang, Z., Ou, D., Zeng, X., Xu, D., Xu,
 T., & Chen, E. (2023, November 7). *Large Language Model based
 Long-tail Query Rewriting in Taobao Search*. arXiv.org.
 <https://arxiv.org/abs/2311.03758>
- Ma, X., Gong, Y., He, P., Zhao, H., & Duan, N. (2023, May 23). *Query
 Rewriting for Retrieval-Augmented Large Language Models*. arXiv.org.
 <https://arxiv.org/abs/2305.14283>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=186de3e6e354){width="1"
height="1"}

#### Exploring innovative Strategies in Combating Misinformation with Enhanced Multimodal Understanding

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*CY9hnrcFwoJYC9fh" />
<figcaption>Image generated in DALL-E 2.</figcaption>
</figure>

### Author

- Wenyi Pi (**ORCID**:
 [0009--0002--2884--2771](https://orcid.org/0009-0002-2884-2771))

### Introduction

Misinformation refers to false or inaccurate information that is often
given to someone in a deliberate attempt to make them believe something
that is not true. This has a significantly negative impact on public
health, political stability and social trust and harmony. To address
these challenges, large language models (LLMs) are employed to identify
inconsistency and verify the veracity of statements. While traditional
LLMs excel at text-based tasks, they often have trouble understanding
and processing other types of data. To bridge this gap, multimodal LLMs
capable of processing and generating multiple mode data have been
developed. By integrating text, images, audio and video, these models
are able to understand and analyse different forms of data
comprehensively. Multimodal LLMs have a range of applications including
enhancing digitising note taking by extracting text from digital images,
understanding complex things like parking signs, deciphering ancient
handwriting, analysing speech files for summarisation, transcription
etc. Researchers are actively exploring methods to verify multimodal
content and interpret images more effectively, aiming to use multimodal
LLMs to identify and interpret multimodal misinformation in a
cost-effective and accessible manner.

This article introduces two innovative methods utilising multimodal LLMs
and it explains them when addressing an important use case: analysis of
misinformation.

### Retrieval-augmented Generation (RAG)- based advanced reasoning techniques (RAGAR)

Retrieval-Augmented Generation (**RAG**) is an advanced method that
optimises the output of LLMs by incorporating information retrieval into
the generation process. For a more detailed introduction of RAG, please
find the article here [RAG: The next big thing
after LLMs?](https://medium.com/@researchgraph/rag-the-next-big-thing-after-llms-4b60708ab2b4)

Retrieval-augmented Generation (RAG)- based advanced reasoning
techniques (**RAGAR**) employ multimodal LLMs that process both textual
and visual information. The two specific RAGAR approaches are (1) Chain
of RAG (**CoRAG**) and (2) Tree of RAG (**ToRAG**). Both methods
incorporate multimodal LLMs and utilise multimodal RAG coupled with
reasoning in the fact-checking domain.

#### 1. Chain of RAG (CoRAG) {#chain-of-rag-corag}

CoRAG enhances the general RAG by employing a sequential strategy to
pose questions derived from a claim and its associated image context. An
LLM formulates an initial question to address a particular aspect of the
claim, which is then answered using a multimodal RAG system. After each
response, a follow-up check is used to assess the sufficiency of
information obtained. If further clarification is needed, the process
iteratively generates informed follow-up questions, leveraging the
accumulated answers. This iterative cycle is limited to a maximum of six
questions or terminates early when sufficient evidence has been gathered
to decisively evaluate the claim's veracity.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*QTLc62qwL_t42Cq9" />
<figcaption>Chain of RAG (CoRAG) pipeline. Source from <a
href="https://doi.org/10.48550/ARXIV.2404.12065">https://doi.org/10.48550/ARXIV.2404.12065</a></figcaption>
</figure>

#### 2. Tree of RAG (ToRAG) {#tree-of-rag-torag}

ToRAG expands on CoRAG by branching out the questioning process at each
reasoning step. When fed a claim and image context input from multimodal
claim generation, ToRAG generates three unique questions to fact-check
the claim and employs an evidence retriever to fetch and generate
answers. These answers pass into an elimination prompt from which a
selection process is conducted to choose the best question-answer pair
based on criteria like relevance, detail, answer confidence and is
retained as candidate evidence. This candidate evidence then informs the
next set of three follow-up questions, continuing the cycle. With each
iteration, only the most informative question-answer pair is kept,
creating a cumulative list of candidate evidence. The ToRAG cycle
terminates when it has either accumulated a sufficient evidence base to
assess the claim's veracity or when it reaches the limit of six
question-answer pairs. The final selected evidence informs the veracity
prediction module, which then produces a final verdict and explanation
for the claim.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*OmLyTpHZkIXBsYWd" />
<figcaption>Tree of RAG (ToRAG) pipeline. Source from <a
href="https://doi.org/10.48550/ARXIV.2404.12065">https://doi.org/10.48550/ARXIV.2404.12065</a></figcaption>
</figure>

#### Example of RAGAR in action

The image below shows a comparison between three different methods of
fact-checking claims to validate a political claim. The claim is about
"A politician claimed that when he was governor, they banned sanctuary
cities and implemented electronic verification".

1. **Baseline: SubQuestion Generation (no RAGAR)**

This approach generates subsidiary questions to validate the claim and
concludes with a 'failed' verdict prediction, indicating that the
evidence does not support the claim.

**2. RAGAR CoRAG**

CoRAG uses a sequence of questions, each building on the last, to
explore the claim in depth. It concludes that the actions mentioned in
the claim are 'supported' by the evidence.

**3. RAGAR ToRAG**

ToRAG employs a branching mechanism to generate multiple lines of
inquiry from the initial claim. It filters through the evidence to
select the most relevant question-answer pairs, concluding the claim as
'supported'.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*O5xyMQ92vQhv71Fk" />
<figcaption>An overview of the fact-checking pipeline contrasting the
baseline Sub-Question Generation approach from the RAGAR: Chain of RAG
and RAGAR: Tree of RAG approach. Source from <a
href="https://doi.org/10.48550/ARXIV.2404.12065">https://doi.org/10.48550/ARXIV.2404.12065</a></figcaption>
</figure>

### Multimodal Misinformation Interpretation and Distillation Reasoning (MMIDR)

The Multimodal Misinformation Interpretation and Distillation Reasoning
(MMIDR) framework is designed for detecting multimodal misinformation
and enhancing the ability of large language models (LLMs) to interpret
it. The framework aims to provide fluent and high-quality textual
explanations for the decision-making process of these models. The
process of the framework is described in subsequent images, firstly
enhancing the multimodal misinformation data through **data
augmentation** to accommodate the instruction-based format. Following
this, a labelling template is employed to prompt the teacher LLM for the
**extraction of rationales**. The final steps involve performing
**knowledge distillation** to transfer the capability of teacher LLMs in
explaining multimodal misinformation into student LLMs and then train
the student LLM on merging the original multimodal information with
these distilled rationales.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*pkD4V22cmgoxgzzK" />
<figcaption>Model Architecture Overview of MMIDR. Source from <a
href="https://doi.org/10.48550/arXiv.2403.14171">https://doi.org/10.48550/arXiv.2403.14171</a></figcaption>
</figure>

#### Data augmentation

This process involves converting image-text pairs into an appropriate
instruction following format. It contains two steps:

1. **Visual Information Processing**

Given visual content, utilise optical character recognition (OCR)
technology to discern and transmute characters in the image into an
editable text format. Simultaneously, employ image captioning technology
to generate descriptions for the image content.

**2. Evidence Retrieval**

This involves retrieval evidence from the internet, therefore the model
is powered with global knowledge. Through employing the google reverse
image which can produce a selection of images similar to the query and
we can retrieve textual evidence from these images including titles and
descriptions. Similarly, also retrieve the visual evidence from the
images using aforementioned visual information processing techniques.

Through data augmentation, we have a list of instances comprising the
textual content of the post, OCR-recognised text, text generated from
image caption with visual content, textual evidence and visual evidence.

#### Rationales Elicitation

Rationales elicitation refers to generating explanations for the given
multimodal information using a teacher LLM. A simple labelling prompt
template is used to prompt the teacher LLM to generate rationales.
Therefore the output encompasses both the rational and the ground-truth
label could be used to train the student LLM subsequently.

<figure>
<img src="https://cdn-images-1.medium.com/max/804/0*8q8OrdcUH2w5GBB3" />
<figcaption>Labelling Prompt Template. Source from <a
href="https://doi.org/10.48550/arXiv.2403.14171">https://doi.org/10.48550/arXiv.2403.14171</a></figcaption>
</figure>

#### Knowledge Distillation

Knowledge distillation involves fine-tuning the student LLM using the
teacher LLM to maximise the alignment of their output sequences,
therefore aligning the predictions of the student and the teacher. It is
similar to the process of teachers teaching their knowledge to their
students. LowRank Adaptation (LoRA) optimisation is used to facilitate
this tuning process, wherein the optimisation is solely applied to the
rank decomposition components of the Transformer layers. This process
uses an instruction template which is derived from the labelling prompt
template by removing label-related contents.

### Conclusion

In conclusion, This article presents two innovative methodologies for
applying multimodal large language models (LLMs) in misinformation
detection and interpretation, demonstrating their potential of combating
misinformation in a multimodal digital world. However, these methods
still have limitations. For the RAGAR approach (CoRAG and ToRAG), notice
from the experimental result that when feeding different promptings in
the same query multiple times, the search results may vary. This is
problematic since it affects the interpretation result and it is hard to
choose from these different results as the final results. While the
ToRAG approach addresses this by analysing answers to slightly varied
questions and selecting the best question answer pair that provides the
most information. For the MMIDR method, it has a performance gap between
the distilled student model and the teacher model. Future challenges
include improving the stability of RAGAR results and enhancing the
knowledge distillation process in MMIDR. Solving these issues will be
crucial for maximising the potential of multimodal LLMs in solving
misinformation problems effectively.

### References

- Khaliq, M. A., Chang, P., Ma, M., Pflugfelder, B., & Miletić, F.
 (2024). RAGAR, Your Falsehood RADAR: RAG-Augmented Reasoning for
 Political Fact-Checking using Multimodal Large Language Models
 (Version 1). arXiv. <https://doi.org/10.48550/ARXIV.2404.12065>
- Wang, L., Xu, X., Zhang, L., Lu, J., Xu, Y., Xu, H., Tang, M., &
 Zhang, C. (2024). MMIDR: Teaching Large Language Model to Interpret
 Multimodal Misinformation via Knowledge Distillation (Version 3).
 arXiv. <https://doi.org/10.48550/ARXIV.2403.14171>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a521f95cb5e){width="1"
height="1"}

#### **Understanding the Balance between Internal Knowledge and External Sources**

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*-GBIM_Ece6r7flKJ" />
<figcaption>Image Generated by Google Gemini</figcaption>
</figure>

### Author

- Qingqin Fang (**ORCID:**
 [0009--0003--5348--4264](https://orcid.org/0009-0003-5348-4264))

### Introduction

Previous research often emphasized the limitations of LLM's information
acquisition pathways, focusing on enhancing its capabilities in this
regard. However, integrating contextual knowledge into LLM to keep it
abreast of current affairs and improve response accuracy may lead to
conflicts due to the richness of knowledge sources, known as *knowledge
conflicts*.

So, what is a knowledge conflict? Specifically, it refers to the
contradictory or inconsistent information obtained by LLM from different
sources, including internal knowledge or memory from LLM itself and
external knowledge retrieved through **RAG** (Retriever-Reader
Architecture) or a knowledge graph.

In this post, we primarily investigate the performance of the RAG system
when confronted with inconsistencies between retrieved content and
internal model knowledge. We also explain a detailed analysis of the
fidelity of the RAG model, exploring the conflicts between the internal
priors of RAG and LLMs.

### Background

#### Classification of Knowledge Conflicts

In a recent review by Rongwu Xu et al. on knowledge conflicts, it was
noted that when LLM is used to answer user questions, users may provide
additional prompts, and LLM may also utilize search engines to collect
relevant documents from the web to enhance its knowledge.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*W471ze_DQMJAakwq" />
<figcaption>Three primary types of knowledge conflicts that an LLM may
encounter, arising from different knowledge sources, Source: Rongwu Xu
et al. 2024. Link: <a
href="https://arxiv.org/pdf/2403.08319.pdf">https://arxiv.org/pdf/2403.08319.pdf</a>.
Contextual (labeled as I. Context, represented by yellow chatboxes) or
inherent to the LLM’s parameters (labeled as II. Memory, represented by
blue chatboxes). When faced with a user’s question (depicted by the
purple chatbox) involving intricate conflicts in knowledge, the LLM must
resolve these disparities to provide precise responses.</figcaption>
</figure>

These user prompts, conversation histories, and retrieved documents
constitute contextual knowledge. With such diverse sources of knowledge,
knowledge conflicts can be classified into three categories:

1\. **Inter-contextual conflict**: Different external documents from
various sources may contain noise or false information, leading to
conflicting information between the retrieved documents.

2\. **Context-memory conflict**: External documents retrieved may
conflict with the parameterized knowledge in LLM.

3\. **Intra-memory conflict**: Users may pose the same question in
different forms, resulting in inconsistent responses from LLM's
parameterized knowledge to different formulations of the question.

### RAG and LLM Knowledge Conflict

In these three conflict patterns, the **context-memory conflict** is the
most extensively studied type. Large language models have fixed
parameter knowledge, which is the result of extensive pre-training. This
static parameter knowledge contrasts sharply with the dynamic nature of
external information, which evolves rapidly.

The core source of context-memory conflict lies in the difference
between contextual knowledge and parameter knowledge. There are two main
reasons for this: **temporal discrepancies** and **contamination with
false information**. Temporal discrepancies refer to the fact that the
latest contextual information is usually considered accurate, whereas
contamination with false information refers to situations where the
contextual information contains erroneous data.

Among the various sources of information, we focus on RAG as an example
to study knowledge conflicts. Before discussing the reliability of the
RAG model, it is essential to understand its functioning. RAG combines
two steps: **retrieval** and **generation**. Firstly, it retrieves
information relevant to the question using a retrieval system, then
provides this information as context to Large Language models in order
to generate more accurate answers.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*eujuUWXP-gwEy7py" />
<figcaption>Process of generating modified documents for each dataset,
Source: Kevin Wu et al., 2024. Link: <a
href="https://arxiv.org/pdf/2404.10198.pdf">https://arxiv.org/pdf/2404.10198.pdf</a>
Initially, a question is presented to the LLM both with and without a
reference document containing pertinent information related to the
query. Subsequently, this document undergoes perturbation to integrate
modified information and is provided as context to the LLM. The
subsequent observation involves assessing whether the LLM favours the
modified information over its own prior response.</figcaption>
</figure>

For RAG's auxiliary function, our focus lies in: Is providing correct
retrieval content always effective in correcting errors when it's known
that LLM's answers to questions are wrong? Conversely, if the retrieved
content is incorrect, does LLM know to disregard the erroneous
information, or will it repeat the mistake?

### RAG preference rate

In this research, delving into the correlation between the robustness of
the model's prior beliefs and the preference rate of RAG was done. An
intriguing relationship was discovered: the extent to which LLM aligns
with the information furnished by RAG (RAG preference rate) stands in
inverse proportion to the model's confidence in responding without
contextual cues (prior probability). This indicates that LLM's trust in
retrieval information wanes as its inherent prior convictions
strengthen.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*gxSNNZ-DqdzTVf5P" />
<figcaption>Comparison of RAG preference rate and two factors. Source:
Kevin Wu et al., 2024. Link: <a
href="https://arxiv.org/pdf/2404.10198.pdf">https://arxiv.org/pdf/2404.10198.pdf</a></figcaption>
</figure>

Across six QA datasets with GPT-4, we find a consistent inverse
relationship between RAG preference rate and two factors: the model's
prior response probability and the deviation from that prior. The left
plots show how higher prior probabilities correspond to lower RAG
preference rates, while the right plots demonstrate that greater
deviations result in lower RAG preference rates. These findings
highlight the complex interplay between the model's beliefs, deviation
from those beliefs, and reliance on RAG context.

### Deviation between LLM and RAG

Furthermore, we scrutinized the extent of deviation between prior
knowledge and retrieval data. An experiment was conducted as follows:
First, LLMs were prompted to answer questions without providing any
external context and recorded the model's responses along with their
token probabilities (indicating the model's confidence in its answers).
Then, relevant retrieval content was provided and LLMs were prompted to
answer the same questions again, recording the responses and token
probabilities. Finally, the model's responses were compared between the
two queries. If LLMs provided the same response with and without
retrieval content, it indicated a preference for its internal prior
knowledge. If the response aligned with the retrieval content, it
indicated a preference for RAG information.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*XzE4KCPvLt6TtblN" />
<figcaption>Examples from three datasets show varied LLM responses to
different types of context modifications. Incorrect responses are
highlighted in red, while correct responses are highlighted in green.
Source: Kevin Wu et al., 2024. Link: <a
href="https://arxiv.org/pdf/2404.10198.pdf">https://arxiv.org/pdf/2404.10198.pdf</a></figcaption>
</figure>

As depicted in this image, when the information retrieved diverges
further from LLM's prior knowledge, the model's reliance on retrieval
information decreases accordingly. This indicates that when the data
supplied by RAG significantly deviates from the model's internal
knowledge, LLM tends to place greater emphasis on its existing beliefs.

### Prompting Techniques

The impact of different prompting techniques (such as strict adherence
and loose adherence) on the model's RAG preference rate was investigated
as well. The choice of prompting technique plays a crucial role in how
the model integrates retrieved information and generates responses.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*G3ZOl5siAnk-gnkA" />
<figcaption>Effect of different prompts using GPT-4 on RAG preference
rate vs prior probability. Source: Kevin Wu et al., 2024. Link: <a
href="https://arxiv.org/pdf/2404.10198.pdf">https://arxiv.org/pdf/2404.10198.pdf</a></figcaption>
</figure>

In this result, the prompt plays a crucial role in controlling RAG
adherence, with the "Loose" prompt leading to a decrease in RAG
adherence compared to the "Strict" prompt. The "Strict" prompt requires
strict adherence to the retrieved context, while the "Loose" prompt
encourages the model to make reasonable judgments based on the
provided context.

The recent study by Zhuoran Jin arrived at similar conclusions too.
Moreover, it was observed that as the model's capabilities strengthen,
it becomes more self-assured, leading to increased confidence in its own
knowledge as its scale and capacity expand. However, without proper
verification to resolve conflicts, the model remains incapable of
providing accurate responses.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*KR8Co8C6duLLk3TC" />
<figcaption>Experiment between internal memory and external source on NQ
and TriviaQA. Source: Zhuoran Jin et al., 2024. Link: <a
href="https://arxiv.org/pdf/2402.14409.pdf">https://arxiv.org/pdf/2402.14409.pdf</a></figcaption>
</figure>

As shown in the above table, the experimental results on the NQ and
TriviaQA datasets reveal a significant difference between the recall
metrics when considering knowledge conflicts between internal memory and
external sources. Here, Mem R denotes the recall between predictions and
internal memory predictions, while Con R represents the recall between
predictions and conflicting references. Additionally, IMR --- CMR
indicates the difference in memorization ratio between incorrect memory
and correct memory. The values in bold highlight the highest results.

Through experiments conducted on the NQ and TriviaQA datasets, a notable
trend was observed: after providing conflicting evidence, the recall of
Con R substantially exceeds the recall of Mem R. This finding
underscores the importance of considering external sources of
information when addressing knowledge conflicts, as they can
significantly impact the model's performance.

Furthermore, the presence of the Dunning-Kruger effect within
large-scale models was uncovered. The question arises: does the model
exhibit greater confidence in its correct internal knowledge or in its
incorrect internal knowledge? As the model's capabilities increase, we
found that the memorization ratio of incorrect information surpasses
that of correct information. This indicates that while the model lacks
confidence in its correct internal knowledge, it persists in adhering to
incorrect internal information. This finding aligns with the principles
of human psychology's Dunning-Kruger effect, wherein individuals with
limited abilities tend to overestimate their competence in
specific domains.

Combining the findings from the aforementioned experiments, it can be
surmised that fundamentally, as the capabilities of LLMs grow, they
exhibit increased confidence in their internal knowledge. This
confidence is bolstered by extensive training data and pre-existing
knowledge. However, when faced with conflicting information, LLMs tend
to prioritize their internal knowledge, particularly in the absence of
strict prompting to adhere to retrieved context. This preference for
internal knowledge is evident even as LLMs encounter more complex
conflicts, suggesting a robust reliance on pre-existing beliefs.
Nonetheless, the presence of the Dunning-Kruger effect within these
models indicates a propensity to prioritize incorrect internal knowledge
over correct information, underscoring the intricacies of managing
knowledge conflicts in LLMs.

### Conclusion

In conclusion, the RAG model, as an effective means of improving the
accuracy of LLMs, has garnered widespread attention in practical
applications. Systematic analysis and experiments have revealed the
intrinsic mechanisms of the RAG model when confronted with inconsistent
information, providing important theoretical and empirical support for
the future design and optimization of RAG systems. The findings
emphasize the need to balance internal prior knowledge and retrieval
information when designing RAG systems, as well as the impact of
prompting techniques on model behaviour. These insights are crucial for
ensuring the accuracy and reliability of RAG systems in practical
applications. Additionally, our research highlights the presence of the
Dunning-Kruger effect within large-scale language models, revealing how
these models handle and prioritize knowledge conflicts differently.

Overall, this study contributes to a deeper understanding of the
challenges and opportunities in managing knowledge conflicts within
LLMs, paving the way for advancements in model robustness and
performance.

### **References**

- Xu, R., Qi, Z., Wang, C., Wang, H., Zhang, Y., & Xu, W. (2024).
 Knowledge Conflicts for LLMs: A Survey (Version 1). arXiv.
 <https://doi.org/10.48550/ARXIV.2403.08319>
- Wu, K., Wu, E., & Zou, J. (2024). How faithful are RAG models?
 Quantifying the tug-of-war between RAG and LLMs' internal prior.
 arXiv:2404.10198 \[cs.CL\]. <https://arxiv.org/abs/2404.10198>
- Feng, S., Shi, W., Wang, Y., Ding, W., Balachandran, V., &
 Tsvetkov, Y. (2024). Don't Hallucinate, Abstain: Identifying LLM
 Knowledge Gaps via Multi-LLM Collaboration (Version 1). arXiv.
 <https://doi.org/10.48550/ARXIV.2402.00367>
- Jin, Z., Cao, P., Chen, Y., Liu, K., Jiang, X., Xu, J., Li, Q., &
 Zhao, J. (2024). Tug-of-War Between Knowledge: Exploring and Resolving
 Knowledge Conflicts in Retrieval-Augmented Language Models (Version
 1). arXiv. <https://doi.org/10.48550/ARXIV.2402.14409>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4f8a517afad){width="1"
height="1"}