Swinburne University of Technology

Amir Aryani

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*VnD8bwqPJmEMEoqQ" />
<figcaption>Photo by <a
href="https://unsplash.com/@iammottakin?utm_source=medium&amp;utm_medium=referral">Mojahid
Mottakin</a> on <a
href="https://unsplash.com?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption>
</figure>

Integrating Large Language Models (LLMs) such as GPT into organizations'
data workflows is a complex process with various challenges. These
obstacles include but are not limited to technical, operational,
ethical, and legal dimensions, each presenting hurdles that
organisations must navigate to harness the full potential of LLMs
effectively. Here's an expanded categorisation:

### Technical Challenges

1.  **Data Privacy and Security:** Ensuring the security and privacy of
    data when using LLMs is a primary concern. Sensitive information
    might be exposed to unauthorized parties if proper safeguards are
    not in place.
2.  I**ntegration Complexity:** Integrating LLMs with existing data
    systems and workflows can be complex, requiring significant
    customization and configuration to ensure seamless operation.
3.  **Scalability and Performance:** Ensuring the LLM can handle the
    scale of data and provide responses promptly can be challenging,
    especially for large organizations with vast amounts of data.
4.  **Data Quality and Bias**: LLMs can perpetuate or even amplify
    biases present in their training data. Ensuring the model generates
    accurate, unbiased outputs requires constant monitoring and
    adjustment.
5.  **Model Updating and Maintenance**: Keeping the LLM updated with the
    latest information and ensuring it evolves with the organization's
    needs requires ongoing effort and resources.

### Operational Challenges

1.  **Cost**: Deploying and maintaining LLMs can be costly, involving
    expenses related to computing resources, licensing fees, and
    personnel for model management and integration.
2.  **Skill Gaps**: There might be a lack of expertise within the
    organization to effectively deploy, manage, and leverage LLMs,
    requiring significant training or hiring of specialized personnel.
3.  **Change Management**: Integrating LLMs into workflows can require
    significant changes in processes and possibly the organizational
    culture, leading to resistance among staff.

### Ethical and Legal Challenges

1.  **Accountability**: Determining accountability for decisions made or
    assisted by LLMs can be difficult, raising questions about liability
    and ethical responsibility.
2.  **Compliance**: Ensuring that the use of LLMs complies with relevant
    laws and regulations, including those related to data protection
    (e.g., GDPR), is essential.
3.  **Transparency and Explainability**: LLMs are often seen as "black
    boxes," making it challenging to understand how they arrive at
    certain conclusions or decisions, which can be problematic for
    regulatory compliance and trust.

### Mitigation

Organizations can address these challenges through a combination of
technical solutions, policy development, and training. This includes
implementing robust data governance policies, investing in secure and
scalable infrastructure, continuously monitoring for bias and errors,
ensuring compliance with legal standards, and fostering a culture of
ethical AI use.

Integrating LLMs into organisational workflows holds immense potential
for efficiency gains and innovation, but navigating these challenges is
crucial for successful and responsible implementation.

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b5e8e2a95bfe){width="1"
height="1"}


Integrating Large Language Models (LLMs) such as GPT into organizations’ data workflows is a complex process with various challenges. These obstacles include but are not limited to technical, operational, ethical, and legal dimensions, each presenting hurdles that organisations must navigate to harness the full potential of LLMs effectively.

What are the challenges in integrating LLMs into organisations' data workflows?

Aland Astudillo

**The AI Helper Turning Mountains of Data into Bite-Sized Instructions**

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*XwvfCyjF7dUWQ7Wj" />
<figcaption>Image generated with Google’s Gemini
12–02–2024.</figcaption>
</figure>

### Author

* Aland Astudillo (ORCID: [0009-0008-8672-3168](https://orcid.org/0009-0008-8672-3168))

LLMs have been changing the way the entire world deals with problems and
day-by-day tasks. To make them better for specific applications, they
need huge amounts of data and complex and expensive approaches to
training them. However, there are some challenges such as limited prompt
size, and limited context windows that make LLM not suitable for some.
This is a major issue for tasks that require huge amounts of
information. LLMLingua has been developed as a framework that helps LLMs
in addressing these limitations. In this article, we will review what
LLMLingua is, what it does, how it does it and what we can expect in the
near future.

Picture this scenario, you're tasked with teaching a group of aspiring
chefs how to prepare a complex gourmet meal. You could throw every
recipe detail and culinary term at them, hoping they absorb it all. But
wouldn't it be more effective to break down the instructions into clear,
concise steps, focusing only on the crucial techniques and ingredients?
This, in essence, is the magic of LLMLingua.

Instead of aspiring chefs, imagine training a team of AI apprentices to
tackle complex tasks. Like those aspiring chefs who thrive in learning
to prepare new dishes, LLMs thrive on information. But feeding LLMs with
mountains of raw data can be overwhelming, leading to slow processing
and limited performance. Enter LLMLingua, the innovative tool acting as
the AI chef, meticulously preparing information into bite-sized
instructions that these AI apprentices can easily digest and master.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*WauCih49veOTPy9V" />
<figcaption>Image generated with Google’s Gemini
12–02–2024.</figcaption>
</figure>

So, how does LLMLingua craft this culinary magic?

It employs two key techniques, similar to how a skilled chef optimises
a recipe:

1.  **Ingredient Trimming:** Imagine each piece of information as an
    ingredient in the recipe. LLMLingua, akin to a seasoned chef,
    identifies and discards unnecessary elements like filler words or
    irrelevant details. This "trimming" streamlines instructions,
    keeping only the essential components, just like using only the
    right spices to enhance a dish's flavour.
2.  **Recipe Refining**: Even carefully chosen ingredients need precise
    instructions. LLMLingua refines the wording and phrasing of
    instructions, ensuring the AI apprentices grasp the core meaning
    with perfect clarity. It's like the chef rewriting the recipe in
    clear, concise steps, leaving no room for confusion or
    misinterpretation.

Currently, several variations of LLMLingua exist, each tailored to
specific tasks and LLM architectures. One version optimises prompts for
question-answering models, while another focuses on improving
summarization tasks. These variations highlight the adaptability and
evolving nature of this technology.

But LLMLingua isn't alone in the quest for efficient AI. Techniques like
knowledge distillation and parameter reduction also aim to streamline
models. What sets LLMLingua apart is its focus on prompt manipulation,
offering a unique and flexible approach.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*hAjLbEGPdLOJyziQ" />
<figcaption>Scheme of the LLMLingua framework. Image created by Aland
Astudillo.</figcaption>
</figure>

Imagine teaching a friend a complex recipe. You wouldn't overwhelm them
with every detail; instead, you'd break it down, focusing on key steps
and ingredients. LLMLingua operates similarly, streamlining information
for large language models (LLMs) to achieve peak performance. Here's a
deeper dive into its technical pipeline:

1.  **Input Preprocessing**: The journey begins with the "raw"
    information intended for the LLM. LLMLingua analyses it, identifying
    irrelevant elements like filler words or redundant instructions.
    Think of skimming unnecessary details from a recipe book, keeping
    only the crucial steps. This initial "trimming" helps reduce the
    information load on the LLM.
2.  **Tokenization**: Next, LLMLingua breaks down the preprocessed text
    into individual units called "tokens," similar to words in a
    sentence. This facilitates further analysis and manipulation.
    Imagine separating the ingredients listed in your recipe into
    individual units --- flour, eggs, milk, etc.
3.  **Budget Control**: LLMLingua employs a "budget controller" to
    ensure compression doesn't compromise information integrity. By
    setting a desired compression ratio, it controls how much
    information can be discarded while maintaining optimal performance
    for the LLM. This is like deciding how much of each ingredient to
    use without altering the final dish drastically.
4.  **Iterative Compression**: Now comes the magic. LLMLingua utilises
    an iterative algorithm to refine the tokenized information. Each
    iteration analyses the remaining tokens, identifying opportunities
    for further compression while considering their importance for the
    LLM's task. Think of repeatedly reviewing your recipe steps,
    replacing complex techniques with simpler alternatives whenever
    possible.
5.  **Instruction Tuning**: Beyond simply removing elements, LLMLingua
    also refines the wording and phrasing of the remaining instructions.
    Imagine rewriting your recipe steps for clarity and accuracy,
    ensuring your friend understands them perfectly. This ensures the
    LLM receives clear and concise instructions, minimising room for
    misinterpretation.
6.  **Output Generation**: Finally, the compressed and refined
    information is presented to the LLM as a "streamlined recipe." The
    LLM processes it efficiently, achieving higher performance with
    faster response times and reduced resource consumption. It's like
    your friend effortlessly executing the optimised recipe, producing a
    delicious dish efficiently.

This is just a simplified overview and the actual LLMLingua pipeline
involves complex algorithms and deep learning techniques that can be
reviewed in the research article (See the references).

**Key concepts: Entropy and perplexity**

Additionally, there are two key concepts that allow the control engine
in the LLMLingua framework: entropy and perplexity. These metrics act as
indicators of information content and difficulty, allowing researchers
to gauge the effectiveness of LLMLingua's compression techniques.

**1. Entropy**

Imagine tossing a coin. With one possible outcome (heads or tails), the
entropy is low. Now, consider a hundred-sided die --- with many more
possibilities --- the entropy is higher. Similarly, entropy in language
measures the unpredictability of the next word given the previous ones.
Longer, more complex sentences typically have higher entropy than
shorter, simpler ones.

LLMLingua aims to reduce the entropy of instructions fed to LLMs. By
removing redundant information and focusing on key elements, it
essentially makes the next word more predictable, like simplifying the
die to fewer sides. This lowers the information load on the LLM, leading
to more efficient processing.

**2. Perplexity**

Perplexity builds upon entropy but is presented as an inverse
probability. A lower perplexity value signifies higher predictability,
indicating that the model can more easily anticipate the next word.
Conversely, a high perplexity suggests complex, unpredictable language,
making it harder for the model to process.

In the context of LLMLingua, a decrease in perplexity after compression
reflects improved efficiency. It means the LLM can understand the
compressed instructions with the same accuracy as the original but with
less effort. This translates to faster response times and lower
computational costs.

It's important to note that:

- Both entropy and perplexity are complex measures with nuances beyond
  this simplified explanation.
- LLMLingua's goal isn't to achieve the absolute lowest entropy or
  perplexity, but to find the optimal balance between compression and
  information fidelity.

So, while entropy and perplexity might seem like abstract concepts, they
play a crucial role in understanding how LLMLingua achieves its
efficiency gains.

**Why is LLMLingua the secret ingredient for AI success?**

LLMs are changing the world, translating languages on the fly, composing
personalised poems, and even answering your questions with remarkable
accuracy. But just like our aspiring chefs, their hunger for information
can create limitations. Imagine translating entire novels
word-for-word --- it's slow, resource-intensive, and ultimately
unsatisfying. LLMLingua tackles this head-on, paving the way for faster,
more efficient, and impactful AI applications:

- **Speedy Chatbots**: Ever feel like waiting forever for a chatbot
  response? LLMLingua compresses your questions, enabling chatbots to
  understand you instantly and respond at lightning speed. It's like
  having a personal AI assistant who's always on top of their game,
  eliminating frustrating wait times.
- **Translation on the Go**: Imagine translating entire documents in
  seconds! LLMLingua empowers translation tools to process information
  with laser precision, breaking down language barriers faster than
  ever. Think of it as a universal translator, seamlessly bridging
  communication gaps across cultures and languages like a skilled
  multilingual chef seamlessly navigating different cuisines.
- **Research Revolution**: Picture sifting through mountains of
  scientific data in minutes. LLMLingua empowers AI assistants to
  analyse complex research papers with unmatched efficiency,
  accelerating scientific breakthroughs and discoveries. It's like
  having a tireless research partner who can summarise vast amounts of
  information, highlighting key findings and saving researchers
  countless hours.
- **AI for Everyone**: From personalised learning experiences to
  efficient task management, LLMLingua can revolutionise how AI
  assistants interact with us. Imagine an AI tutor who personalises
  learning modules based on your individual needs or an assistant who
  manages your schedule with laser-focus, all thanks to the power of
  concise and optimised instructions.

But, wait...how do we solve the problem of the limited context size? Let
me introduce you to **LongLLMLingua**!

### What is LongLLMLingua?

Think of LongLLMLingua as the master chef, overseeing the entire
culinary experience. It builds upon LLMLingua's foundation, applying its
compression techniques not just to single instructions, but to entire
sequences of information. This empowers LLMs to process long
contexts --- think research papers, dialogue history, or complex
narratives --- with impressive efficiency and improved performance.

**How does it work?**

LongLLMLingua operates in three key phases:

- **Coarse Compression**: First, it performs a "rough cut," analysing
  the entire context and identifying large sections it can safely
  discard. Imagine skimming extraneous details from a recipe book,
  focusing only on the core steps for each dish.
- **Fine Compression**: Next, it dives deeper, applying LLMLingua's
  techniques to refine remaining information. Think of meticulously
  preparing each dish, ensuring optimal ingredient selection and precise
  instructions.
- **Reordering for Coherence**: Finally, LongLLMLingua ensures the
  compressed information maintains its original meaning and flow.
  Imagine plating each dish in a visually appealing and sequential
  manner, ensuring a cohesive dining experience.

**Why is LongLLMLingua important?**

Long contexts are crucial in various AI applications:

- **Question Answering**: Imagine needing context from several articles
  to answer a complex question. LongLLMLingua helps LLMs retain key
  information across documents, leading to more accurate answers.
- **Machine Translation**: Imagine translating entire books instead of
  single sentences. LongLLMLingua ensures coherence and preserves
  meaning despite lengthy input.
- **Dialogue Systems**: Imagine chatbots understanding intricate
  conversation history. LongLLMLingua enables them to maintain context,
  leading to more natural and engaging interactions.

Real-world examples of LongLLMLingua in action:

- **LongBench Benchmark**: This benchmark measures LLM performance in
  long context scenarios. When applied to GPT-3.5-Turbo, LongLLMLingua
  achieved a 17.1% performance boost with 4x fewer tokens.
- **ZeroScrolls Benchmark**: This benchmark focuses on reading
  comprehension in long contexts. LongLLingua reduced costs by \$27.4
  per 1,000 samples while maintaining performance.

Remember, LongLLMLingua isn't limited to cooking analogies. It's a
powerful tool revolutionising how LLMs handle long contexts,
accelerating progress in various AI domains. By improving efficiency and
maintaining information fidelity, LongLLMLingua opens doors to a future
where AI interactions are more contextual, seamless, and impactful.

**Conclusion**

In conclusion, both LLMLingua and LongLLMLingua are not just
technological advancements; they're culinary metaphors come to life.
LLMLingua acts as the skilled chef, meticulously preparing information
into bite-sized instructions for AI apprentices. LongLLMLingua takes the
baton, transforming into the master chef, seamlessly handling entire
sequences of information like preparing a multi-course feast. Together,
they're revolutionising how LLMs interact with information, paving the
way for a future where:

- AI responses are faster, more accurate, and contextually relevant.
- Machine translation transcends sentence-by-sentence limitations,
  unlocking global communication on a grand scale.
- AI assistants understand our interactions in their entirety, leading
  to more natural and engaging dialogues.

As these technologies continue to evolve, the possibilities are endless.
Imagine AI tutors adapting to your learning style based on years of
educational data, or chatbots understanding months of conversation
history to anticipate your needs with uncanny accuracy.

The future of AI is hungry for efficiency and understanding, and
LLMLingua and LongLLMLingua are serving up the perfect recipe for
success. By empowering LLMs to process information like culinary
masters, they're opening doors to a new era of seamless interaction,
accelerated discovery, and boundless communication.

**Sources**

- LLMLingua research paper: Jiang et al., 2023. LLMLingua: Compressing
  prompts for accelerated inference of Large Language Models. arXiv
  2310.05736v2 <https://arxiv.org/abs/2310.05736>
- LongLLMLingua research paper: Jiang et al., 2023. LongLLMLingua:
  accelerating and enhancing LLMs in long context scenarios via prompt
  compression. arXiv 2310.06839v1 <https://arxiv.org/abs/2310.06839>
- Microsoft LLMLingua page: LLMLingua: Compressing Prompts for
  Accelerated Inference of Large Language Models --- Microsoft Research
  <https://www.microsoft.com/en-us/research/project/llmlingua/>
- Hugging Face LLMLingua Space: LLMLingua --- a Hugging Face
  <https://huggingface.co/spaces/microsoft/LLMLingua>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a632385b206){width="1"
height="1"}


<strong>
 The AI Helper Turning Mountains of Data into Bite-Sized Instructions
</strong>
Author  Aland Astudillo (ORCID: 0009-0008-8672-3168)  LLMs have been changing the way the entire world deals with problems and day-by-day tasks. To make them better for specific applications, they need huge amounts of data and complex and expensive approaches to training them.

What is LLMLingua?

Nakul Nambiar

Zhuochen Wu

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*7Fy1vUav7gAiy9CAWanAvQ.png" />
<figcaption>Created using DALLE on 6 Feb 2024</figcaption>
</figure>

#### Authors:

* Nakul Nambiar (ORCID: [0009-0009-9720-9233](https://orcid.org/0009-0009-9720-9233))
* Zhuochen Wu (ORCID: [0009-0000-5642-5348](https://orcid.org/0009-0000-5642-5348))

Research Graph is a structured representation of research objects that
captures information about entities and the relationships between
Researcher, Organisation, Publication, Grant and Research Data.
Currently, the publications are available as PDF files, and due to the
free-form text, it is difficult to parse a PDF file to extract
structured information. In this article, we will try to create a
research graph from a PDF of a publication by extracting relevant
information from the text and organising it into a graph structure
using OpenAI.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*40j-_agywNgUsJ_h_kXslA.jpeg" />
<figcaption>Pipeline to create the graph from PDF</figcaption>
</figure>

### OpenAI

In this work, we use OpenAI API and the new Assistant functionality of
GPT (Currently in Beta) to transform PDF documents to a set of
structured JSON files based on Research Graph Schema.

#### Assistants API

The Assistants API allows you to build AI assistants within your
applications. An assistant can answer user questions by using models,
tools, and information per predetermined guidelines. It is a beta API
that is actively under development. Using the assistants API, we can use
the OpenAI-hosted tools like code interpreter and knowledge retrieval.
For this blog post, we will focus on knowledge retrieval.

#### Knowledge Retrieval

Sometimes, we need the AI model to answer queries based on unknown
knowledge, like user-provided documents or sensitive information. We can
augment the model with this information using the assistant APIs
knowledge retrieval tool. We can upload files to the assistant, and it
automatically chunks the documents and creates and stores embeddings to
implement vector search on the data.

#### Example

For our example, we will upload PDF files of publications to the OpenAI
assistant and the knowledge retrieval tool to get a JSON output of the
graph schema for the given publications. The publication we used for
this example can be accessed from the following
[link](https://doi.org/10.5166/jroi-2-1-6).

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*CrAVHSNrRmQiBay81_Bi0Q.png" />
<figcaption>Screenshot from the publication (<a
href="https://doi.org/10.5166/jroi-2-1-6">https://doi.org/10.5166/jroi-2-1-6</a>)</figcaption>
</figure>

#### Step 1

Read the input path where the Publications PDFs are stored and the
output path where the JSON output will be stored.

    import configparser
    config = configparser.ConfigParser()
    config.read('{}/config.ini'.format(current_path))
    input_path = config['DEFAULT']['Input-Path']
    output_path = config['DEFAULT']['Output-Path']
    debug = config['DEFAULT']['Debug']

#### Step 2

Get all the PDF files from the Input path.

    onlyfiles = [f for f in os.listdir(input_path) if os.path.isfile(os.path.join(input_path, f))]

#### Step 3

Then, we need to initialise the assistant to use the Knowledge retrieval
tool. To do this, we need to specify the type of tools in the API for
"retrieval". We also specify the instructions for the assistant and the
OpenAI model to be used.

    my_file_ids = []
    if client.files.list().data==[]:
      for f in onlyfiles:
        file = client.files.create(
          file=open(input_path + f, "rb"),
          purpose='assistants'
        )
        my_file_ids.append(file.id)

    # Add the file to the assistant
    assistant = client.beta.assistants.create(
      instructions = "You are a publication database support chatbot. Use pdf files uploaded to best respond to user queries in JSON.",
      model = "gpt-4-1106-preview",
      tools = [{"type": "retrieval"}],
      # Do not attach all files to the assistant, otherwise, it will mismatch the answers even though specify file ID in query messages.
      # We will attach to each message instead
    )

#### Step 4

Then, we specify the information we need to extract from the
publications files and pass them as user queries to the assistant. After
experimenting with the assistant instructions, we found that requesting
JSON format in each user message generates the most consistent output.

    user_msgs = ["Print the title of this paper in JSON",
                 "Print the authors of this paper in JSON",
                 "Print the abstract section of the paper in JSON",
                 "Print the keywords of the paper in JSON",
                 "Print the DOI number of the paper in JSON",
                 "Print the author affiliations of the paper in JSON",
                 "Print the reference section of the paper in JSON"]

#### Step 5

The next is to pass the queries to the assistant to generate the output.
We need to create an individual thread object for each user query, which
contains the query as a user message. Then, we run the thread and
retrieve the assistant's answer.

    all_results = []

    for i in my_file_ids:
        print('\n#####')
        # the JSON result it can extract and parse, hopefully
        file_result = {}
       
        for q in user_msgs:
            # create thread, user message and run objects for each query
            thread = client.beta.threads.create()
            msg = client.beta.threads.messages.create(
                    thread_id=thread.id,
                    role="user",
                    content=q,
                    file_ids=[i] # specify the file/publication we want to extract from
                )
            print('\n',q)
            run = client.beta.threads.runs.create(
                thread_id=thread.id,
                assistant_id=assistant.id,
                additional_instructions="If answer cannot be found, print 'False'" # not very useful at the time of this presentation
                )
            # checking run status by retrieving updated object each time
            while run.status in ["queued",'in_progress']:
                print(run.status) 
                time.sleep(5)
                run = client.beta.threads.runs.retrieve(
                    thread_id=thread.id,
                    run_id=run.id
                    )
            # usually a rate limit error
            if run.status=='failed':  logging.info("Run failed: ", run)
            if run.status=='completed':
                print("<Complete>")
                # extract updated message object, this includes user messages
                messages = client.beta.threads.messages.list(
                    thread_id=thread.id
                )
                for m in messages:
                    if m.role=='assistant':
                        value = m.content[0].text.value # get the text response
                        if "json" not in value:
                            if value=='False':logging.info("No answer found for ", str(q))
                            else:
                                logging.info("Not JSON output, maybe no answer found in the file or model is outdated: ", str(value))
                        else:
                            # clean the response and try to parse as json
                            value = value.split("```")[1].split('json')[-1].strip()
                            try: 
                                d = json.loads(value)
                                file_result.update(d)
                                print(d)
                            except Exception as e:
                                logging.info(f"Query {q} \nFailed to parse string to JSON: ", str(e))
                                print(f"Query {q} \nFailed to parse string to JSON: ", str(e))
                
        all_results.append(file_result)

The JSON output generated for the above publication file is:

    [{'title': 'Dodes (diagnostic nodes) for Guideline Manipulation',
      'authors': [{'name': 'PM Putora',
        'affiliation': 'Department of Radiation-Oncology, Kantonsspital St. Gallen, St. Gallen, Switzerland'},
       {'name': 'M Blattner',
        'affiliation': 'Laboratory for Web Science, Zürich, Switzerland'},
       {'name': 'A Papachristofilou',
        'affiliation': 'Department of Radiation Oncology, University Hospital Basel, Basel, Switzerland'},
       {'name': 'F Mariotti',
        'affiliation': 'Laboratory for Web Science, Zürich, Switzerland'},
       {'name': 'B Paoli',
        'affiliation': 'Laboratory for Web Science, Zürich, Switzerland'},
       {'name': 'L Plasswilma',
        'affiliation': 'Department of Radiation-Oncology, Kantonsspital St. Gallen, St. Gallen, Switzerland'}],
      'Abstract': {'Background': 'Treatment recommendations (guidelines) are commonly represented in text form. Based on parameters (questions) recommendations are defined (answers).',
       'Objectives': 'To improve handling, alternative forms of representation are required.',
       'Methods': 'The concept of Dodes (diagnostic nodes) has been developed. Dodes contain answers and questions. Dodes are based on linked nodes and additionally contain descriptive information and recommendations. Dodes are organized hierarchically into Dode trees. Dode categories must be defined to prevent redundancy.',
       'Results': 'A centralized and neutral Dode database can provide standardization which is a requirement for the comparison of recommendations. Centralized administration of Dode categories can provide information about diagnostic criteria (Dode categories) underutilized in existing recommendations (Dode trees).',
       'Conclusions': 'Representing clinical recommendations in Dode trees improves their manageability handling and updateability.'},
      'Keywords': ['dodes',
       'ontology',
       'semantic web',
       'guidelines',
       'recommendations',
       'linked nodes'],
      'DOI': '10.5166/jroi-2-1-6',
      'references': [{'ref_number': '[1]',
        'authors': 'Mohler J Bahnson RR Boston B et al.',
        'title': 'NCCN clinical practice guidelines in oncology: prostate cancer.',
        'source': 'J Natl Compr Canc Netw.',
        'year': '2010 Feb',
        'volume_issue_pages': '8(2):162-200'},
       {'ref_number': '[2]',
        'authors': 'Heidenreich A Aus G Bolla M et al.',
        'title': 'EAU guidelines on prostate cancer.',
        'source': 'Eur Urol.',
        'year': '2008 Jan',
        'volume_issue_pages': '53(1):68-80',
        'notes': 'Epub 2007 Sep 19. Review.'},
       {'ref_number': '[3]',
        'authors': 'Fairchild A Barnes E Ghosh S et al.',
        'title': 'International patterns of practice in palliative radiotherapy for painful bone metastases: evidence-based practice?',
        'source': 'Int J Radiat Oncol Biol Phys.',
        'year': '2009 Dec 1',
        'volume_issue_pages': '75(5):1501-10',
        'notes': 'Epub 2009 May 21.'},
       {'ref_number': '[4]',
        'authors': 'Lawrentschuk N Daljeet N Ma C et al.',
        'title': "Prostate-specific antigen test result interpretation when combined with risk factors for recommendation of biopsy: a survey of urologist's practice patterns.",
        'source': 'Int Urol Nephrol.',
        'year': '2010 Jun 12',
        'notes': 'Epub ahead of print'},
       {'ref_number': '[5]',
        'authors': 'Parmelli E Papini D Moja L et al.',
        'title': 'Updating clinical recommendations for breast colorectal and lung cancer treatments: an opportunity to improve methodology and clinical relevance.',
        'source': 'Ann Oncol.',
        'year': '2010 Jul 19',
        'notes': 'Epub ahead of print'},
       {'ref_number': '[6]',
        'authors': 'Ahn HS Lee HJ Hahn S et al.',
        'title': 'Evaluation of the Seventh American Joint Committee on Cancer/International Union Against Cancer Classification of gastric adenocarcinoma in comparison with the sixth classification.',
        'source': 'Cancer.',
        'year': '2010 Aug 24',
        'notes': 'Epub ahead of print'},
       {'ref_number': '[7]',
        'authors': 'Rami-Porta R Goldstraw P.',
        'title': 'Strength and weakness of the new TNM classification for lung cancer.',
        'source': 'Eur Respir J.',
        'year': '2010 Aug',
        'volume_issue_pages': '36(2):237-9'},
       {'ref_number': '[8]',
        'authors': 'Sinn HP Helmchen B Wittekind CH.',
        'title': 'TNM classification of breast cancer: Changes and comments on the 7th edition.',
        'source': 'Pathologe.',
        'year': '2010 Aug 15',
        'notes': 'Epub ahead of print'},
       {'ref_number': '[9]',
        'authors': 'Paleri V Mehanna H Wight RG.',
        'title': "TNM classification of malignant tumours 7th edition: what's new for head and neck?",
        'source': 'Clin Otolaryngol.',
        'year': '2010 Aug',
        'volume_issue_pages': '35(4):270-2'},
       {'ref_number': '[10]',
        'authors': 'Guarino N.',
        'title': 'Formal Ontology and Information Systems',
        'source': '1998 IOS Press'},
       {'ref_number': '[11]',
        'authors': 'Uschold M Gruniger M.',
        'title': 'Ontologies: Principles Methods and Applications.',
        'source': 'Knowledge Engineering Review',
        'year': '1996',
        'volume_issue_pages': '11(2)'},
       {'ref_number': '[12]',
        'authors': 'Aho A Garey M Ullman J.',
        'title': 'The Transitive Reduction of a Directed Graph.',
        'source': 'SIAM Journal on Computing',
        'year': '1972',
        'volume_issue_pages': '1(2): 131–137'},
       {'ref_number': '[13]',
        'authors': 'Tai K',
        'title': 'The tree-to-tree correction problem.',
        'source': 'Journal of the Association for Computing Machinery (JACM)',
        'year': '1979',
        'volume_issue_pages': '26(3):422-433'}]}]

#### Step 6

File objects and assistant object needs to be cleaned up because they
cost money in 'retrieval' mode. Also, it is a good coding practice.

    for f in client.files.list().data:
        client.files.delete(f.id)

    # Retrieve and delete running assistants
    my_assistants = client.beta.assistants.list(
        order="desc",
    )
    for a in my_assistants.data:    
        response = client.beta.assistants.delete(a.id)
        print(response)

#### Step 7

The next step is to generate a graph visualisation using the Python
[Networkx](https://networkx.org/documentation/stable/tutorial.html) package.

    import networkx as nx
    import matplotlib.pyplot as plt

    G = nx.DiGraph()
    node_colors = []

    key = "jroi/" + all_results[0]['title']
    G.add_nodes_from([(all_results[0]['title'], {'doi': all_results[0]['DOI'], 'title': all_results[0]['title'], 'source': 'jroi', 'key': key})])
    node_colors.append('#4ba9dc')

    for author in all_results[0]['authors']:
        key = "jroi/" + author['name']
        G.add_nodes_from([(author['name'], {'key': key, 'local_id': author['name'], 'full_name': author['name'], 'source': 'jroi'})])
        G.add_edge(all_results[0]['title'], author['name'])
        node_colors.append('#63cc9e')


    for reference in all_results[0]['references']:
        key = "jroi/" + reference['title']
        G.add_nodes_from([(reference['title'].split('.')[0][:25] + '...', {'title': reference['title'], 'source': 'jroi', 'key': key})])
        G.add_edge(all_results[0]['title'], reference['title'].split('.')[0][:25] + '...')
        node_colors.append('#4ba9dc')

    pos = nx.spring_layout(G)
    labels = nx.get_edge_attributes(G, 'label')
    nx.draw(G, pos, with_labels=True, node_size=1000, node_color=node_colors, font_size=7, font_color='black')
    nx.draw_networkx_edge_labels(G, pos, edge_labels=labels)
    plt.savefig("graph_image.png")
    plt.show()

The graph visualisation is as below:

<figure>
<img
src="https://cdn-images-1.medium.com/max/640/1*-a5y19GUSyQZv63KMXKnZA.png" />
<figcaption>Graph Visualisation generated using Networkx</figcaption>
</figure>

**Note:** Please note that the output structure generated by OpenAI may
be different for different executions. So you may need to update the
above code according to that structure.

### Conclusion

In conclusion, leveraging the GPT API to extract a Research Graph from
PDF publications offers a powerful and efficient solution for
researchers and data analysts. This workflow streamlines the process of
converting PDF publications into structured and accessible Research
Graphs. But we must also pay careful attention to the inconsistencies of
the responses generated by LLM. Over time, accuracy and relevance may be
further enhanced by routinely updating and improving the extraction
model.

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4655f64337db){width="1"
height="1"}


Authors:  Nakul Nambiar (ORCID: 0009-0009-9720-9233) Zhuochen Wu (ORCID: 0009-0000-5642-5348)  Research Graph is a structured representation of research objects that captures information about entities and the relationships between Researcher, Organisation, Publication, Grant and Research Data.

How to use GPT API to export a research graph from PDF publications

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*Gzmbpq5qJUz2W1zpHd1XGA.png" />
<figcaption>Created using DALL-E on 28 Jan 2024</figcaption>
</figure>

### Authors

* Nakul Nambiar (ORCID: [0009-0009-9720-9233](https://orcid.org/0009-0009-9720-9233))
* Amir Aryani (ORCID: [0000-0002-4259-9774](https://orcid.org/0000-0002-4259-9774))

Knowledge graphs, which offer a structured representation of data and
its relationships, are revolutionising how we organise and access
information. With large amounts of data, it sometimes becomes difficult
to draw insights from it. This blog article examines how to combine
Neo4j, a graph database, with OpenAI's Retrieval-Augmented Generation
(RAG) model to build a robust knowledge management system.

LLMs are pre-trained models and generate outputs based on the prompts we
pass them. Sometimes, the LLM results are precisely what you need, but
other times, they just return random facts from their training data.
Retrieval-augmented generation (RAG) is an AI framework that tries to
improve the response of the LLM models by providing context knowledge
from external sources. Additional benefits of RAG include:

- Enables getting insights into private data without sharing or using it
  for training.
- Reduces the chances of 'Hallucination' and incorrect responses.
- Reduces the need to train the model on new data continuously.

### Sample Data

For our exercise, we will use OpenAI, combined with the Australian
National Graph Database, to create an RAG pipeline.

The [Australian National Graph
Database](https://researchgraph.org/national-graph/) is a vast knowledge
graph with around 11 million nodes. We can't possibly fit all the data
into the OpenAI context. So, in the first phase, we will retrieve a
subset of the graph and then use this graph as a context for the OpenAI
and to answer prompts.

We use a Virtual Machine (VM) with a Neo4j server running the graph
database as a Neo4j server. The [graph
schema](https://researchgraph.org/schema/) includes five primary node
types that describe the research ecosystem: Researchers, Publications,
Datasets, Grants and Organisations.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*UocexTQHeuX3_VAwSJ03CQ.png"
alt="Research Graph Schema: https://researchgraph.org/schema/" />
<figcaption>Research Graph Schema. <a
href="https://researchgraph.org/schema/">https://researchgraph.org/schema/</a></figcaption>
</figure>

### Step 1

Firstly, we must establish the connection to the Neo4j database. Also,
you might notice that we use an OpenAI API key. This key can be
generated at the OpenAI portal. A tutorial is available
[here](https://elephas.app/blog/how-to-create-openai-api-keys-cl5c4f21d281431po7k8fgyol0).

    import os
    import json
    import pandas as pd
    import configparser
    from neo4j import GraphDatabase

    config = configparser.ConfigParser()
    config.read('config.ini')

    # Set the OpenAI API key env variable manually
    os.environ["OPENAI_API_KEY"] = config['DEFAULT']['openai-key']

    # DB credentials
    url = config['DEFAULT']['url']
    username = config['DEFAULT']['username']
    password = config['DEFAULT']['password']

### Step 2

We get a subset of the graph from the larger dataset by querying Neo4j.
The output of the graph is retrieved in JSON format.

    def run_query(cypher_query):
        with GraphDatabase.driver(url, auth=(username, password)) as driver:
            with driver.session() as session:
                result = session.run(cypher_query)
                return result.data()

    cypher_query = """
    MATCH (n:organisation)-[r]-(g:grant)
    where g.source='arc.gov.au' RETURN * LIMIT 100
    """

    result = run_query(cypher_query)

Note: We have used a limit of 100 nodes in the response to limit the
size of the returning JSON file. This would impact the LLM functionality
if it exceeds the context window size.

We create a system prompt by providing background knowledge about the
graph database and the task. Along with it, we pass the JSON subgraph
as context.

    system_prompt = f'''
    You are a helpful agent designed to fetch information from a graph database.
    Data
    {json.dumps(result)}

    This is the output of the Neo4j cypher which returns nodes and relationships. n is the organisations, g is the grants
    and r is the relationships between the grants and organisations.

    Based on this data answer the user prompts and return the output as json.

    '''

Users can then enter the prompt to get insights from the subgraph. For
example, in the prompt below, we are trying to find the grant with the
most relationship to organisations.

    prompt = "Help me find the grants which is related to the most organisations"

### Step 3

We pass the system and user prompts to the OpenAI model. In this case,
we use "gpt-4--1106-preview". More information is available on the
[OpenAI page on Models](https://platform.openai.com/docs/models).

    from openai import OpenAI
    client = OpenAI()

    def define_query(prompt, model="gpt-4-1106-preview"):
        completion = client.chat.completions.create(
        model=model,
        temperature=0,
        response_format= {
            "type": "json_object"
        },
        messages=[
            {
                "role": "system",
                "content": system_prompt
            },
            {
                "role": "user",
                "content": prompt
            }
            ]
        )
        return completion.choices[0].message.content

OpenAI provides the grant details with the most connections to
organisations as a JSON output.

    print(define_query(prompt))

### Output

This is the output provided by OpenAI RAG.

    {
      "grants": [
        {
          "arc_for_primary": "1117",
          "local_id": "LP200301123",
          "start_year": "2023",
          "purl": "http://purl.org/au-research/grants/arc/LP200301123",
          "source": "arc.gov.au",
          "type": "grant",
          "arc_scheme_name": "Linkage Projects",
          "title": "Making Australia resilient to airborne infection transmission.",
          "arc_announcement_administering_organisation": "Queensland University of Technology",
          "url": "http://dataportal.arc.gov.au/NCGP/API/grants/LP200301123",
          "arc_funding_at_announcement": "898013",
          "arc_grant_status": "Active",
          "arc_for": "040101,1117,111705,120404",
          "r_number": "158880661",
          "end_year": "2026",
          "participant_list": "Lidia Morawska,Duzgun Agdas,Laurie Buys,Keith Grimwood,Graham Johnson,Guy Marks,Alexander Paz,Thomas Rainey,Zoran Ristovski,Kirsten Spann,Bo Xia,Patrick Chambers,Shaun Clough,Stephen Corbett,Charles dePuthod,Neal Durant,Travis Kline,Peter McGarry,Ross Mensforth,Clive Paige,John Penny,Brad Prezant,Uma Rajappa,David Ward,Scott Bell,Giorgio Buonanno,Mark Jermy,Jose Jimenez,Claire Wainwright,Sandra Glaister,greg Bell,Louisa O'Toole",
          "key": "arc/LP200301123",
          "related_organisations_count": 24
        }
      ]
    }

We can compare it visually with the graph from Neo4j and see that RAG
provided the correct output.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*PkIQl8fupR-TdiwppqdJkg.png" />
</figure>

We have tested this mini pipeline with other examples, and the
simplicity of the pipeline makes it functional as long as the size of
the subgraph produced in Step 2 does not exceed the context window of
the OpenAI model.

### Conclusion

The integration of OpenAI RAG and Neo4j paves the way for innovative
solutions in knowledge graph applications, and recommendation systems.
RAG is efficient in grounding the LLM on the most recent data, and
verified information and reduces the expense of retraining the model
when new data is available.

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a183df816ce){width="1"
height="1"}


Authors  Nakul Nambiar (ORCID: 0009-0009-9720-9233) Amir Aryani (ORCID: 0000-0002-4259-9774)  Knowledge graphs, which offer a structured representation of data and its relationships, are revolutionising how we organise and access information.

Mini RAG using Neo4j

Authors: [Hui Yin](https://www.linkedin.com/in/hui-yin-784538213/),
[Amir Aryani](https://www.linkedin.com/in/amiraryani/)

> RAG is an AI framework enhancing pre-trained language models with
> up-to-date, reliable external knowledge.

As we discussed in our previous article "[A Brief Introduction to
Retrieval Augmented Generation
(RAG)](https://medium.com/@amiraryani/a-brief-introduction-to-retrieval-augmented-generation-rag-4bd6e50da532)",
RAG is an artificial intelligence framework that incorporates the latest
reliable external knowledge and aims to improve the quality of responses
generated by pre-trained language models (PLM). Initially, it was
designed to improve the performance of knowledge-intensive NLP tasks
([Lewis et al., 2020](https://arxiv.org/abs/2005.11401)). As more and
more people notice the advantages of RAG, it has also been applied to a
broader range of scenarios. We conducted a rapid review of RAG-related
publications and found there are mainly five categories of use cases:
Medical, Knowledge Graph, Code Comment, Chatbot, Review, and Image
processing.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*HGpI_s3nvDIyHRE2_yE5kQ.png" />
<figcaption>Created by Dr Hui Yin, Jan 2024. CC-BY License
for reuse.</figcaption>
</figure>

### **Chatbot**

Chatbots have gained widespread recognition and use by the public. It
acts as an intermediary, translating user queries or requests written in
natural language and communicating with the underlying LLM to generate
responses.

The researchers incorporated RAG into the following tasks to improve the
performance of the chatbot, allowing it to provide more accurate
responses.

Question Answering ([Mao et al.,
2020](https://arxiv.org/abs/2009.08553), [Ju et al.,
2022](https://doi.org/10.48550/arXiv.2210.02933), [Shamane et al.,
2023](https://doi.org/10.1162/tacl_a_00530)) requires access to external
knowledge bases, documents, or structured data to provide accurate and
informative responses.

Text summarization ([Cai et al.,
2022](https://doi.org/10.1145/3477495.3532682), [Hofstätter et al.,
2023](https://doi.org/10.1145/3539618.3591687)), which requires an
understanding of the content and context with the assistance of external
knowledge to generate a concise and informative summary of longer text.

In certain scenarios, dialogue systems can also benefit from RAG
technology, allowing them to handle specific tasks more efficiently
([Shuster et al., 2021](https://doi.org/10.48550/arXiv.2104.07567),
[Thule et al., 2021](https://doi.org/10.48550/arXiv.2102.04643)).

### **Code Comment and Summarization**

Code comments are crucial as they provide explanatory information about
the purpose, functionality, and usage of the code, clear and informative
comments help improve code understanding.

[Yu et al., 2022](https://doi.org/10.48550/arXiv.2206.13325) proposed
BashExplainer which aims to automatically generate code comments for
Bash scripts to enhance code comprehension, especially for developers
who may struggle with understanding Bash code due to its unique
characteristics.

[Liu et al., 2021](https://doi.org/10.48550/arXiv.2006.05405) introduced
a novel retrieval-augmented mechanism for code summarization, aiming to
bridge the language gap between source code and natural language
summaries.

### **Image Processing**

Image processing covers three topics, text-to-image, image-to-text and
image synthesis.

Text-to-image is the generation of visual content (images) from textual
descriptions.

[Chen et al., 2022](https://doi.org/10.48550/arXiv.2209.14491) designed
Re-Image to address challenges in generating images for uncommon
entities. By leveraging a multi-modal knowledge base and a retrieval
step, the model significantly improves fidelity, particularly for rare
entities.

Image-to-text is the reverse of text-to-image, it converts visual
information(images) into textual descriptions, such as generating a
caption or description.

[Sarto et al., 2022](https://doi.org/10.48550/arXiv.2207.13162) proposed
a Retrieval-Augmented Transformer for Image Captioning, which combines a
knowledge retriever, differentiable encoder, and kNN-augmented attention
layer, and enhances caption quality by using explicit external memory.

Image synthesis is very different from the above two items. It creates
new images or modifies existing images, such as generating realistic
images of non-existent objects or scenes, often using generative models
such as generative adversarial networks (GAN). You can notice that some
medium posts contain very stunning images at the top of the post page
and are labelled "Generated by AI". So far, we know of only one work
that uses RAG for image captioning. Blattmann et al. proposed a
Retrieval-Augmented Diffusion Model as a semi-parametric approach for
generative image synthesis.

### **Review**

RAG enhances review generation by incorporating external knowledge, such
as product specifications and customer feedback. The retrieval mechanism
ensures that generated reviews are more informative, contextually
relevant, and aligned with the product or service characteristics,
resulting in nuanced and helpful content compared to traditional
generative models. Several works were found in this research topic, such
as [Kim et al., 2020](https://aclanthology.org/2020.coling-main.207).

### **Medical**

RAG technology has been widely used in Medical area, such as the
Summarisation of electronic health records (EHRs) ([Saba et al.,
2024](https://doi.org/10.48550/arXiv.2401.01469), [Thompson et al.,
2023](https://doi.org/10.48550/arXiv.2312.06457)), Evidence-based
medicine (EBM) ([Vaid et al.,
2024](https://doi.org/10.48550/arXiv.2401.02851)). In fact, it is
obvious that the data processed by the models in these works is text
data related to medical content, which is still an NLP task in nature,
and the external data used in this example specifically belongs to the
medical field.

### **Knowledge Graph**

Integration of Knowledge Graph in RAG is mostly done to improve the
accuracy and performance of the RAG Pipelines. This is mainly a
methodology rather than a use case. However, given the substantial
community support, it has transformed to be an important characteristic
of RAG pipelines.

The knowledge graph is a structured representation of knowledge that
captures entities, their attributes, and their relationships in a
graph-like format. It is structured data that is easy to retrieve. In
Retrieval-Augmented Generation (RAG), leveraging information from
knowledge graphs like Wikipedia can assist in generating more
contextually relevant and accurate natural language text, especially for
open-domain tasks. In addition, the RAG structure uses other external
resources depending on the specific task, such as web scraping,
retrieval of databases, existing text corpora, and external APIS. Below
are some survey papers on RAG with in-depth discussions on external
sources and knowledge graphs, [Wei et al.,
2021](https://doi.org/10.48550/arXiv.2110.08455), [Agrawal et al.,
2023](https://doi.org/10.48550/arXiv.2311.07914), [Pan et
al., 2023](https://doi.org/10.48550/arXiv.2306.08302).

### **Acknowledgement**

This research was supported by the Australian Government through the
Australian Research Council's Industrial Transformation Training Centre
for Information Resilience (CIRES) project number IC200100022.

### References

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N.,
Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., Riedel, S., &
Kiela, D. (2020b). Retrieval-Augmented Generation for
Knowledge-Intensive NLP Tasks. Neural Information Processing Systems 33
(NeurIPS 2020). <https://arxiv.org/abs/2005.11401>

Mao, Y., He, P., Liu, X., Shen, Y., Gao, J., Han, J., & Chen, W. (2020).
Generation-Augmented Retrieval for Open-domain Question Answering.
*CoRR*, abs/2009.08553. <https://arxiv.org/abs/2009.08553>

Ju, M., Yu, W., Zhao, T., Zhang, C., & Ye, Y. (2022). Grape: Knowledge
Graph Enhanced Passage Reader for Open-domain Question
Answering.<https://doi.org/10.48550/arXiv.2210.02933>

Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen, Tharindu
Kaluarachchi, Rajib Rana, Suranga Nanayakkara; Improving the Domain
Adaptation of Retrieval Augmented Generation (RAG) Models for Open
Domain Question Answering. *Transactions of the Association for
Computational Linguistics* 2023.11
1--17.<https://doi.org/10.1162/tacl_a_00530>

Cai, D., Wang, Y., Liu, L., & Shi, S. (2022). Recent advances in
retrieval-augmented text generation. In \*Proceedings of the 45th
International ACM SIGIR Conference on Research and Development in
Information Retrieval\* (pp. 3417--3419)
[https://doi.org/10.1145/3477495.3532682](https://doi.org/10.1145/3477495.3532682%5D(https://doi.org/10.1145/3477495.3532682))

Hofstätter, S., Chen, J., Raman, K., & Zamani, H. (2023). Fid-light:
Efficient and effective retrieval-augmented text generation. In
\*Proceedings of the 46th International ACM SIGIR Conference on Research
and Development in Information Retrieval\* (pp. 1437--1447)
[https://doi.org/10.1145/3539618.3591687](https://doi.org/10.1145/3539618.3591687%5D(https://doi.org/10.1145/3539618.3591687))

Shuster, K., Poff, S., Chen, M., Kiela, D., & Weston, J. (2021).
Retrieval augmentation reduces hallucination in conversation. \*arXiv
preprint arXiv:2104.07567\*. <https://doi.org/10.48550/arXiv.2104.07567>

Thulke, D., Daheim, N., Dugast, C., & Ney, H. (2021). Efficient
Retrieval Augmented Generation from Unstructured Knowledge for
Task-Oriented Dialog. \*arXiv preprint\* arXiv:2102.04643.
[https://doi.org/10.48550/arXiv.2102.04643](https://doi.org/10.48550/arXiv.2102.04643%5D(https://doi.org/10.48550/arXiv.2102.04643))

Yu, C., Yang, G., Chen, X., Liu, K., & Zhou, Y. (2022). BashExplainer:
Retrieval-Augmented Bash Code Comment Generation based on Fine-tuned
CodeBERT. <https://doi.org/10.48550/arXiv.2206.13325>

Liu, S., Chen, Y., Xie, X., Siow, J., & Liu, Y. (2021).
Retrieval-Augmented Generation for Code Summarization via Hybrid GNN.
<https://doi.org/10.48550/arXiv.2006.05405>

Chen, W., Hu, H., Saharia, C., & Cohen, W. W. (2022). Re-Imagen:
Retrieval-Augmented Text-to-Image
Generator.<https://doi.org/10.48550/arXiv.2209.14491>

Sarto, S., Cornia, M., Baraldi, L., & Cucchiara, R. (2022).
Retrieval-Augmented Transformer for Image
Captioning.<https://doi.org/10.48550/arXiv.2207.13162>

Blattmann, A., Rombach, R., Oktay, K., Müller, J., & Ommer, B. (2022).
Retrieval-augmented diffusion models. \*Advances in Neural Information
Processing Systems, 35\*, 15309--15324.

Kim, J., Choi, S., Amplayo, R. K., & Hwang, S. (2020).
Retrieval-augmented controllable review generation. In Proceedings of
the 28th International Conference on Computational Linguistics (pp.
2284--2295).<https://aclanthology.org/2020.coling-main.207>

Saba, W., Wendelken, S., & Shanahan, J. (2024). Question-Answering Based
Summarization of Electronic Health Records using Retrieval Augmented
Generation.<https://doi.org/10.48550/arXiv.2401.01469>

Vaid, A., Lampert, J., Lee, J., Sawant, A., Apakama, D., Sakhuja,
A., ... Nadkarni, G. (2024). Generative Large Language Models are
autonomous practitioners of evidence-based medicine.
<https://doi.org/10.48550/arXiv.2401.02851>

Thompson, W. E., Vidmar, D. M., De Freitas, J. K., Pfeifer, J. M.,
Fornwalt, B. K., Chen, R., ... Miotto, R. (2023). Large Language Models
with Retrieval-Augmented Generation for Zero-Shot Disease
Phenotyping.<https://doi.org/10.48550/arXiv.2312.06457>

Wei, X., Wang, S., Zhang, D., Bhatia, P., & Arnold, A. (2021). Knowledge
Enhanced Pretrained Language Models: A Comprehensive
Survey.<https://doi.org/10.48550/arXiv.2110.08455>

Agrawal, G., Kumarage, T., Alghami, Z., & Liu, H. (2023). Can Knowledge
Graphs Reduce Hallucinations in LLMs?: A Survey.
<https://doi.org/10.48550/arXiv.2311.07914>

Pan, S., Luo, L., Wang, Y., Chen, C., Wang, J., & Wu, X. (2023).
Unifying Large Language Models and Knowledge Graphs: A
Roadmap.<https://doi.org/10.48550/arXiv.2306.08302>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=111ad032653d){width="1"
height="1"}


Authors: Hui Yin, Amir Aryani  As we discussed in our previous article “A Brief Introduction to Retrieval Augmented Generation (RAG)”, RAG is an artificial intelligence framework that incorporates the latest reliable external knowledge and aims to improve the quality of responses generated by pre-trained language models (PLM). Initially, it was designed to improve the performance of knowledge-intensive NLP tasks (Lewis et al., 2020). As

Messaggi di Rogue Scholar

What are the challenges in integrating LLMs into organisations' data workflows?

What is LLMLingua?

How to use GPT API to export a research graph from PDF publications

Mini RAG using Neo4j

Navigating the Present: Exploring Practical Horizons of Retrieval-Augmented Generation (RAG)