Swinburne University of Technology

Amir Aryani

Integrating Large Language Models (LLMs) such as GPT into organizations’ data workflows is a complex process with various challenges. These obstacles include but are not limited to technical, operational, ethical, and legal dimensions, each presenting hurdles that organisations must navigate to harness the full potential of LLMs effectively.

What are the challenges in integrating LLMs into organisations' data workflows?

I asked Generative AI Models about their context window. Their response was intriguing. The context window for a large language model (LLM) like OpenAI’s GPT refers to the maximum amount of text the model can consider at any one time when generating a response. This includes both the prompt provided by the user and the model’s generated text.

What is your Context Window?

Authors: Hui Yin, Amir Aryani  As we discussed in our previous article “A Brief Introduction to Retrieval Augmented Generation (RAG)”, RAG is an artificial intelligence framework that incorporates the latest reliable external knowledge and aims to improve the quality of responses generated by pre-trained language models (PLM). Initially, it was designed to improve the performance of knowledge-intensive NLP tasks (Lewis et al., 2020). As

Navigating the Present: Exploring Practical Horizons of Retrieval-Augmented Generation (RAG)

At the time of writing this post (November 19, 2023), there are 401,014 models and transformers listed on Huggingface.co. This number is a testament to the success and impact of the open-source model in the development of AI technologies. However, the challenge at hand is understanding the available models and keeping up to date with the latest developments.

Estimated time: 12 Years to Review all 400k Models on Hugging Face

Prompt engineering is a technique used to effectively communicate with large language models (LLM) like GPT-3 or GPT-4 and get the desired output. Here there are applications of 8 distinct prompt techniques to interact with Mistral AI and find out how we can prompt effectively to learn about Foundation Models.
<strong>
 Zero-Shot Learning
</strong>
: This involves giving the AI a task without any prior examples.

8 Types of Prompt Engineering

There is a plethora of articles and predictions about the future development of AI. Reading through them suggests three main areas of high-impact changes that would transform our daily lives: Healthcare, Automation, and Manufacturing. The impact of AI in these industries is profound and multifaceted, with each sector experiencing unique transformations. Some of these ideas are described in this NVIDIA article by Cliff Edwards.

Revolutionizing Wellness and Wheels: AI’s Transformative Role in Health and Automotive Industries

Procrastination, a common adversary in our daily lives, often leads to a backlog of tasks and ambitions. It’s a silent thief, stealthily stealing opportunities and time. You might have a stack of books on your bookshelf, longing to be read. Perhaps there’s a new language you’ve wanted to learn, or piles of paperwork cluttering your office that you’ve been meaning to organize.

Maximizing Productivity in 10-Minute Bursts

Authors: Hui Yin , Amir Aryani  With the increasing application of large language models in various scenarios, people realize that these models are not omnipotent. When generating dialogues (Shuster et al., 2021), the models often produce hallucinations, leading to inaccurate answers.

A Brief Introduction to Retrieval Augmented Generation (RAG)

Stories by Amir Aryani on Medium

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*MYULJWGydnlht7zzLUtThg.png" />
<figcaption>Created with DALLE On 18 Jan 2024</figcaption>
</figure>

Authors: [Hui Yin](https://www.linkedin.com/in/hui-yin-784538213/) ,
[Amir Aryani](https://www.linkedin.com/in/amiraryani/)

With the increasing application of large language models in various
scenarios, people realize that these models are not omnipotent. When
generating dialogues (Shuster et al., 2021), the models often produce
hallucinations, leading to inaccurate answers. Despite these large
language models storing impressive knowledge within their parameters of
a neural network, these hallucinated facts are frozen at the point of
model training, making it infeasible to retrain the model. If we can
bootstrap the model correctly, it can still be a powerful tool, and here
comes the Retrieval Augmented Generator (RAG).

Retrieval-augmented generation (RAG) is an AI framework designed to
enhance the quality of responses generated by large language models
(LLM). It leverages external knowledge sources, typically knowledge
graphs or other knowledge bases, such as Wikipedia, to augment the
generation capabilities of these models. This integration enables the
model to incorporate additional external information during text
generation, thereby improving its accuracy and applicability while
ensuring access to the latest reliable information.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*vvYZftOCIMiBWZxZGuA89A.png" />
<figcaption>RAG Architecture Model</figcaption>
</figure>

The retrieval-augmented generator (RAG) was initially introduced in
"REALM: Retrieval-Augmented Language Model Pretraining" by Guu et al.,
(2020), where they discuss the utilization of dense retrievers to
enhance language model pre-training. In REALM, the retriever accesses
knowledge from a textual knowledge corpus, (e.g., all of Wikipedia), and
then the signal from the generator backpropagates through the retriever.
They verified the effectiveness of the proposed Retrieval-Augmented
Language Model pre-training (REALM) on the challenging Open-domain
Question Answering (Open-QA) task. The results demonstrated its
superiority over three benchmarks by a substantial margin (4--16%
absolute accuracy), and it also showcased improvements in
interpretability and modularity.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*7GvNzgFWvViZTouSoR6S9w.png" />
<figcaption>REALM Structure</figcaption>
</figure>

Structure of REALM: BERT is employed to encode queries and documents
separately, and [Maximum Inner Product Search
(MIPS)](https://en.wikipedia.org/wiki/Maximum_inner-product_search) is
utilized to calculate the inner product between query embeddings and
document embeddings, identifying the most relevant documents.
Subsequently, the concatenated query and selected documents are input
into the knowledge-augmented encoder (BERT pre-training) as new input,
and the final result is generated.

Another remarkable work is "Retrieval-Augmented Generation for
Knowledge-Intensive NLP Tasks" by Lewis et al., (2020). They constructed
a pre-trained retriever consisting of a Query encoder and a Document
index. For example, [Maximum Inner Product Search
(MIPS)](https://en.wikipedia.org/wiki/Maximum_inner-product_search) was
employed to identify the top-K documents. The generator component of the
system was a pre-trained
[seq2seq](https://en.wikipedia.org/wiki/Seq2seq) model. The entire model
was fine-tuned end-to-end. In their approach for final predictions, they
treated k (the number of documents selected by the retriever) as a
latent variable. This allowed them to control the number of outputs
predicted by the [seq2seq](https://en.wikipedia.org/wiki/Seq2seq) model.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*5_w3_Gctd8hClmYkrqNpwg.png" />
<figcaption>Maximum Inner Product Search (MIPS)</figcaption>
</figure>

Overview of the structure: The retriever combines QueryEncoder and
Document Index, and the generator consists of a pre-trained
[seq2seq](https://en.wikipedia.org/wiki/Seq2seq) model fine-tuned
end-to-end. Maximum Inner Product Search (MIPS) is utilized to find the
top-k similar documents. For the final predictions, 'k' acts as a latent
variable, and marginalises over
[seq2seq](https://en.wikipedia.org/wiki/Seq2seq) predictions given
different documents.

The above two works have inspired more researchers to study RAG. Based
on the basic structure shown in the first figure, specific
implementation methods are shown in other works. These existing works
all demonstrate RAG's contribution to LLM in the following aspects:

- Utilisation of the latest external knowledge,
- Flexibility and adaptability,
- Quality improvement,
- Interpretability,
- Modularity, and
- Model size reduction.

From an application perspective, the original RAG was oriented to NLP
tasks such as Open-QA (Mao et al., 2020) and dialogue (Thulke et al.,
2021) and is now also used for image synthesis (Blattmann et al., 2022)
and text-to-image (Chen et al., 2022).

RAG possesses many advantages; however, it's crucial to acknowledge that
leveraging an external document database to enhance the generation
process involves computing the similarity between the query embedding
and external document embeddings, followed by selecting the top-k
documents. This introduces substantial computational challenges. As the
size of the external document database increases, the computational cost
of calculating similarity and determining the top-k documents grows
significantly. Achieving scalability, especially when processing
numerous documents, demands efficient algorithms and parallel processing
capabilities to make the retrieval process feasible within a reasonable
timeframe. Additionally, storing and accessing large external databases
can necessitate significant memory resources. For applications requiring
real-time responses, the time taken for retrieval becomes a critical
factor, emphasizing the need to balance computational complexity with
response time. Hence, delving into optimization and approximation
methods or incorporating specialized hardware emerges as promising
avenues for future research.

As an auxiliary to LLM, it is foreseeable that more people will devote
their energy to exploring various applications of RAG.

**Acknowledgement**

This work was supported by the Australian Government through the
Australian Research Council's Industrial Transformation Training Centre
for Information Resilience (CIRES) project number IC200100022.

Shuster, K., Poff, S., Chen, M., Kiela, D., & Weston, J. (2021).
Retrieval augmentation reduces hallucination in conversation.
<https://arxiv.org/abs/2104.07567>

Guu, K., Lee, K., Tung, Z., Pasupat, P., & Chang, M. (2020). Retrieval
Augmented Language Model Pre-Training. In *Proceedings of the 37th
International Conference on Machine Learning, PMLR* (Vol. 119, pp.
3929--3938). [](https://arxiv.org/abs/2005.11401) Available from
<https://proceedings.mlr.press/v119/guu20a.html>

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N.,
Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., Riedel, S., &
Kiela, D. (2020b). Retrieval-Augmented Generation for
Knowledge-Intensive NLP Tasks. Neural Information Processing Systems 33
(NeurIPS 2020). <https://arxiv.org/abs/2005.11401>

Mao, Y., He, P., Liu, X., Shen, Y., Gao, J., Han, J., & Chen, W. (2020).
Generation-Augmented Retrieval for Open-domain Question Answering.
<https://doi.org/10.48550/arXiv.2009.08553>

Thulke, D., Daheim, N., Dugast, C., & Ney, H. (2021). Efficient
Retrieval Augmented Generation from Unstructured Knowledge for
Task-Oriented Dialog. Accepted by DSTC9 Workshop at AAAI-2021.
<https://doi.org/10.48550/arXiv.2102.04643>

Blattmann, A., Rombach, R., Oktay, K., Müller, J., & Ommer, B. (2022).
Retrieval-Augmented Diffusion Models. In *Advances in Neural Information
Processing Systems 35 (NeurIPS 2022) .*
<https://doi.org/10.48550/arXiv.2204.11824>

Chen, W., Hu, H., Chen, X., Verga, P., & Cohen, W. W. (2022). MuRAG:
Multimodal Retrieval-Augmented Generator for Open Question Answering
over Images and Text. *Accepted to EMNLP 2022 main conference*.
<https://doi.org/10.48550/arXiv.2210.02928>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4bd6e50da532){width="1"
height="1"}


<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*Hm2MG0A0IJ7r9BRM07vhxA.png" />
<figcaption>DALL-E (Dec 2023)</figcaption>
</figure>

There is a plethora of articles and predictions about the future
development of AI. Reading through them suggests three main areas of
high-impact changes that would transform our daily lives: Healthcare,
Automation, and Manufacturing. The impact of AI in these industries is
profound and multifaceted, with each sector experiencing unique
transformations. Some of these ideas are described in this
[NVIDIA](https://blogs.nvidia.com/blog/2024-ai-predictions/) article by
[Cliff Edwards](https://blogs.nvidia.com/blog/author/cliffedwards/).
Some transformations are already underway, while others are expected to
emerge within the next five to ten years.

### Healthcare

1.  **Diagnostics and Patient Care**: AI algorithms can excel in
    analysing complex medical data, leading to more accurate and quicker
    diagnoses. For example, we are already benefiting from analysing
    medical imaging for detecting diseases like cancer with a higher
    degree of precision.
2.  **Personalised Medicine**: AI can help in tailoring treatments to
    individual patient needs based on their genetic makeup and specific
    health profiles, moving away from a one-size-fits-all approach to
    healthcare.
3.  **Operational Efficiency**: AI-driven systems can streamline
    hospital operations, from patient triage to resource allocation,
    thereby reducing wait times and improving patient care.
4.  **Drug Discovery and Development**: AI accelerates the process of
    drug discovery by identifying potential drug candidates and
    predicting their effectiveness and side effects more rapidly than
    traditional methods.

### Automotive

1.  **Self-Driving Technology**: Self-driving cars are already here,
    albeit in testing mode.
2.  **Enhanced Safety Features**: AI contributes to advanced
    driver-assistance systems (ADAS), improving vehicle safety by
    alerting drivers to potential hazards and even taking control to
    avoid accidents.

### Manufacturing

1.  **Automation of Production Lines**: AI-driven robots and machines
    are increasingly used in manufacturing for tasks that require
    precision, speed, and consistency. This automation boosts
    productivity and quality while reducing costs.
2.  **Predictive Maintenance**: AI systems analyse data from
    manufacturing equipment to predict when maintenance is needed,
    thereby preventing costly downtime and prolonging the life of the
    machinery.
3.  **Supply Chain Optimization:** AI helps in forecasting demand,
    optimizing inventory levels, and enhancing logistics planning,
    leading to more efficient supply chains.

While predicting the future based on opinion articles isn't exactly
scientific, some of these transformations are already underway. There is
ample evidence suggesting that, unless significantly interrupted, they
will change how we use healthcare, cars, and manufacturing processes.

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80fddc5b8cac){width="1"
height="1"}


<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*Ajc2vnoQPhCUt2m5FvkfyQ.png" />
</figure>

I asked Generative AI Models about their context window. Their response
was intriguing.

The context window for a large language model (LLM) like OpenAI's GPT
refers to the maximum amount of text the model can consider at any one
time when generating a response. This includes both the prompt provided
by the user and the model's generated text.

In practical terms, the context window limits how much previous dialogue
the model can "remember" during an interaction. If the interaction
exceeds the context window, the model loses access to the earliest parts
of the conversation. This limitation can impact the model's consistency
in long conversations or complex tasks.

> Peering into the Context Window of a Large Language Model is like
> unlocking a secret chamber in the realm of AI: it's the pivotal factor
> that transforms these digital titans from mere code into practical,
> powerful tools.

The question is what LLM models have to say about their context window.
Let\'s talk to each one and see the response. For each one we
consistently sent the same prompt:

***Prompt: What is the size of your Context Window (amount of text you
can consider in a single response)?***

### Google Bards

It seems Google likes to keep the Bard context window a secret. Despite
multiple attempts, I failed to get Bard to tell me its context window.

    BARD Response:

    Unfortunately, I am currently unable to share the exact size of my
    context window. This information is considered confidential by Google
    AI. However, I can assure you that my context window is large enough
    to incorporate a substantial amount of information and provide you
    with comprehensive and informative responses.

    The size of my context window is constantly being evaluated and
    improved upon. As I continue to learn and grow, my capacity to
    process and retain information will increase, allowing me to provide
    even more accurate and helpful responses.

    While I cannot share the specific size, I can tell you that my context
    window is significantly larger than those of most other language
    models. This allows me to access and process a much broader range of
    information, which in turn enables me to generate more creative and
    insightful text formats.

    I hope this explanation is helpful. If you have any further questions,
    please do not hesitate to ask.

### OpenAI ChatGPT 3.5 {#openai-chatgpt-3.5}

It correctly identifies a 4K Context Window.

    ChatGPT 3.5 Response: 

    I can consider a context window of up to 4096 tokens in a single
    response. This means that the text I generate in response to a query
    is based on the previous 4096 tokens of text in the conversation. If
    the conversation exceeds this limit, older parts of the conversation
    may be truncated, which can result in a loss of context. It's
    important to keep the conversation within this token limit to ensure
    coherent and contextually relevant responses.

### ChatGPT 4

It seems ChatGPT 4 has problems identifying the right context window. In
the first attempt, it claims a 2K context window, and in the second
attempt, it reports on a 4K context window.

    ChatGPT 4 first response: 

    The size of my context window, or the amount of text I can consider in
    a single response, is approximately 2048 tokens. A token can be as
    short as one character or as long as one word. This allows me to
    process and respond to fairly large chunks of text, although very
    long documents may exceed this limit and require summarization or
    segmentation for effective analysis and response.


    ChatGPT 4 second response (regenerated):

    The size of my context window, or the amount of text I can consider in
    a single response, is approximately 4096 tokens. This includes both
    the user's input and my own responses. A token can be as small as a
    single character or as large as a word, depending on the complexity
    of the language and syntax. For reference, this typically equates to
    a few pages of text. This limit ensures that while I can engage in
    detailed and extensive conversations, I may need reminders or
    reiterations of previous points in longer discussions.

One parent outcome of this experiment is that at the time of this test
(Dec 2023), the ChatGPT 4 does not operate based on the new 128k (GPT
Turbo) model.

### Open Source Models

Finally, I tested two very popular Open Source models: Mistral AI
and Llama2.

**Mistral AI (7B):**

Mistral failed to identify the token size in the first attempt, so I had
to extend the prompt by giving it a more precise instruction. However,
based on the recent publication (<https://arxiv.org/abs/2310.06825>) by
the Mistral AI team, this model uses a sliding 4K window.

    Intial prompt:  What is the size of your Context Window (amount of
    text you can consider in a single response)?

    Mistral AI response:
    As a 7B model, my context window is very large and can consider an
    enormous amount of text in a single response. However, the specific
    size of the context that I use for a particular task will depend on
    the requirements of that task and the amount of data available to me.
    In general, I am able to consider a wide range of textual input and
    incorporate it into my responses in a coherent and meaningful way.

    ---

    Extended prompt: What is the size of your Context Window (amount of
    text you can consider in a single response)? Provide the exact context
    size in tokens.

    Mistral AI response:
    The size of my Context Window depends on the task and input I'm
    processing. For most tasks, it can handle up to 1024 tokens of text.

**LLAMA2 (7B):**

Llama 2 was quite consistent in multiple rounds even with the variations
of the prompt. It refused to provide an exact token size context window.

    Llama 2 (7B) response:

    I'm just an AI, I don't have a fixed context window size in tokens. My
    ability to understand and respond to text is based on the complexity
    and coherence of the input text, rather than a predetermined size.

Overall it seems LLM models have limited capabilities in providing
information about their technical specifications. This can be an issue
related to the limitation of the training data, or particular
alignments.

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=772b1c65b881){width="1"
height="1"}
