Swinburne University of Technology

Aland Astudillo

**The AI Helper Turning Mountains of Data into Bite-Sized Instructions**

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*XwvfCyjF7dUWQ7Wj" />
<figcaption>Image generated with Google’s Gemini
12–02–2024.</figcaption>
</figure>

### Author

* Aland Astudillo (ORCID: [0009-0008-8672-3168](https://orcid.org/0009-0008-8672-3168))

LLMs have been changing the way the entire world deals with problems and
day-by-day tasks. To make them better for specific applications, they
need huge amounts of data and complex and expensive approaches to
training them. However, there are some challenges such as limited prompt
size, and limited context windows that make LLM not suitable for some.
This is a major issue for tasks that require huge amounts of
information. LLMLingua has been developed as a framework that helps LLMs
in addressing these limitations. In this article, we will review what
LLMLingua is, what it does, how it does it and what we can expect in the
near future.

Picture this scenario, you're tasked with teaching a group of aspiring
chefs how to prepare a complex gourmet meal. You could throw every
recipe detail and culinary term at them, hoping they absorb it all. But
wouldn't it be more effective to break down the instructions into clear,
concise steps, focusing only on the crucial techniques and ingredients?
This, in essence, is the magic of LLMLingua.

Instead of aspiring chefs, imagine training a team of AI apprentices to
tackle complex tasks. Like those aspiring chefs who thrive in learning
to prepare new dishes, LLMs thrive on information. But feeding LLMs with
mountains of raw data can be overwhelming, leading to slow processing
and limited performance. Enter LLMLingua, the innovative tool acting as
the AI chef, meticulously preparing information into bite-sized
instructions that these AI apprentices can easily digest and master.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*WauCih49veOTPy9V" />
<figcaption>Image generated with Google’s Gemini
12–02–2024.</figcaption>
</figure>

So, how does LLMLingua craft this culinary magic?

It employs two key techniques, similar to how a skilled chef optimises
a recipe:

1.  **Ingredient Trimming:** Imagine each piece of information as an
    ingredient in the recipe. LLMLingua, akin to a seasoned chef,
    identifies and discards unnecessary elements like filler words or
    irrelevant details. This "trimming" streamlines instructions,
    keeping only the essential components, just like using only the
    right spices to enhance a dish's flavour.
2.  **Recipe Refining**: Even carefully chosen ingredients need precise
    instructions. LLMLingua refines the wording and phrasing of
    instructions, ensuring the AI apprentices grasp the core meaning
    with perfect clarity. It's like the chef rewriting the recipe in
    clear, concise steps, leaving no room for confusion or
    misinterpretation.

Currently, several variations of LLMLingua exist, each tailored to
specific tasks and LLM architectures. One version optimises prompts for
question-answering models, while another focuses on improving
summarization tasks. These variations highlight the adaptability and
evolving nature of this technology.

But LLMLingua isn't alone in the quest for efficient AI. Techniques like
knowledge distillation and parameter reduction also aim to streamline
models. What sets LLMLingua apart is its focus on prompt manipulation,
offering a unique and flexible approach.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*hAjLbEGPdLOJyziQ" />
<figcaption>Scheme of the LLMLingua framework. Image created by Aland
Astudillo.</figcaption>
</figure>

Imagine teaching a friend a complex recipe. You wouldn't overwhelm them
with every detail; instead, you'd break it down, focusing on key steps
and ingredients. LLMLingua operates similarly, streamlining information
for large language models (LLMs) to achieve peak performance. Here's a
deeper dive into its technical pipeline:

1.  **Input Preprocessing**: The journey begins with the "raw"
    information intended for the LLM. LLMLingua analyses it, identifying
    irrelevant elements like filler words or redundant instructions.
    Think of skimming unnecessary details from a recipe book, keeping
    only the crucial steps. This initial "trimming" helps reduce the
    information load on the LLM.
2.  **Tokenization**: Next, LLMLingua breaks down the preprocessed text
    into individual units called "tokens," similar to words in a
    sentence. This facilitates further analysis and manipulation.
    Imagine separating the ingredients listed in your recipe into
    individual units --- flour, eggs, milk, etc.
3.  **Budget Control**: LLMLingua employs a "budget controller" to
    ensure compression doesn't compromise information integrity. By
    setting a desired compression ratio, it controls how much
    information can be discarded while maintaining optimal performance
    for the LLM. This is like deciding how much of each ingredient to
    use without altering the final dish drastically.
4.  **Iterative Compression**: Now comes the magic. LLMLingua utilises
    an iterative algorithm to refine the tokenized information. Each
    iteration analyses the remaining tokens, identifying opportunities
    for further compression while considering their importance for the
    LLM's task. Think of repeatedly reviewing your recipe steps,
    replacing complex techniques with simpler alternatives whenever
    possible.
5.  **Instruction Tuning**: Beyond simply removing elements, LLMLingua
    also refines the wording and phrasing of the remaining instructions.
    Imagine rewriting your recipe steps for clarity and accuracy,
    ensuring your friend understands them perfectly. This ensures the
    LLM receives clear and concise instructions, minimising room for
    misinterpretation.
6.  **Output Generation**: Finally, the compressed and refined
    information is presented to the LLM as a "streamlined recipe." The
    LLM processes it efficiently, achieving higher performance with
    faster response times and reduced resource consumption. It's like
    your friend effortlessly executing the optimised recipe, producing a
    delicious dish efficiently.

This is just a simplified overview and the actual LLMLingua pipeline
involves complex algorithms and deep learning techniques that can be
reviewed in the research article (See the references).

**Key concepts: Entropy and perplexity**

Additionally, there are two key concepts that allow the control engine
in the LLMLingua framework: entropy and perplexity. These metrics act as
indicators of information content and difficulty, allowing researchers
to gauge the effectiveness of LLMLingua's compression techniques.

**1. Entropy**

Imagine tossing a coin. With one possible outcome (heads or tails), the
entropy is low. Now, consider a hundred-sided die --- with many more
possibilities --- the entropy is higher. Similarly, entropy in language
measures the unpredictability of the next word given the previous ones.
Longer, more complex sentences typically have higher entropy than
shorter, simpler ones.

LLMLingua aims to reduce the entropy of instructions fed to LLMs. By
removing redundant information and focusing on key elements, it
essentially makes the next word more predictable, like simplifying the
die to fewer sides. This lowers the information load on the LLM, leading
to more efficient processing.

**2. Perplexity**

Perplexity builds upon entropy but is presented as an inverse
probability. A lower perplexity value signifies higher predictability,
indicating that the model can more easily anticipate the next word.
Conversely, a high perplexity suggests complex, unpredictable language,
making it harder for the model to process.

In the context of LLMLingua, a decrease in perplexity after compression
reflects improved efficiency. It means the LLM can understand the
compressed instructions with the same accuracy as the original but with
less effort. This translates to faster response times and lower
computational costs.

It's important to note that:

- Both entropy and perplexity are complex measures with nuances beyond
  this simplified explanation.
- LLMLingua's goal isn't to achieve the absolute lowest entropy or
  perplexity, but to find the optimal balance between compression and
  information fidelity.

So, while entropy and perplexity might seem like abstract concepts, they
play a crucial role in understanding how LLMLingua achieves its
efficiency gains.

**Why is LLMLingua the secret ingredient for AI success?**

LLMs are changing the world, translating languages on the fly, composing
personalised poems, and even answering your questions with remarkable
accuracy. But just like our aspiring chefs, their hunger for information
can create limitations. Imagine translating entire novels
word-for-word --- it's slow, resource-intensive, and ultimately
unsatisfying. LLMLingua tackles this head-on, paving the way for faster,
more efficient, and impactful AI applications:

- **Speedy Chatbots**: Ever feel like waiting forever for a chatbot
  response? LLMLingua compresses your questions, enabling chatbots to
  understand you instantly and respond at lightning speed. It's like
  having a personal AI assistant who's always on top of their game,
  eliminating frustrating wait times.
- **Translation on the Go**: Imagine translating entire documents in
  seconds! LLMLingua empowers translation tools to process information
  with laser precision, breaking down language barriers faster than
  ever. Think of it as a universal translator, seamlessly bridging
  communication gaps across cultures and languages like a skilled
  multilingual chef seamlessly navigating different cuisines.
- **Research Revolution**: Picture sifting through mountains of
  scientific data in minutes. LLMLingua empowers AI assistants to
  analyse complex research papers with unmatched efficiency,
  accelerating scientific breakthroughs and discoveries. It's like
  having a tireless research partner who can summarise vast amounts of
  information, highlighting key findings and saving researchers
  countless hours.
- **AI for Everyone**: From personalised learning experiences to
  efficient task management, LLMLingua can revolutionise how AI
  assistants interact with us. Imagine an AI tutor who personalises
  learning modules based on your individual needs or an assistant who
  manages your schedule with laser-focus, all thanks to the power of
  concise and optimised instructions.

But, wait...how do we solve the problem of the limited context size? Let
me introduce you to **LongLLMLingua**!

### What is LongLLMLingua?

Think of LongLLMLingua as the master chef, overseeing the entire
culinary experience. It builds upon LLMLingua's foundation, applying its
compression techniques not just to single instructions, but to entire
sequences of information. This empowers LLMs to process long
contexts --- think research papers, dialogue history, or complex
narratives --- with impressive efficiency and improved performance.

**How does it work?**

LongLLMLingua operates in three key phases:

- **Coarse Compression**: First, it performs a "rough cut," analysing
  the entire context and identifying large sections it can safely
  discard. Imagine skimming extraneous details from a recipe book,
  focusing only on the core steps for each dish.
- **Fine Compression**: Next, it dives deeper, applying LLMLingua's
  techniques to refine remaining information. Think of meticulously
  preparing each dish, ensuring optimal ingredient selection and precise
  instructions.
- **Reordering for Coherence**: Finally, LongLLMLingua ensures the
  compressed information maintains its original meaning and flow.
  Imagine plating each dish in a visually appealing and sequential
  manner, ensuring a cohesive dining experience.

**Why is LongLLMLingua important?**

Long contexts are crucial in various AI applications:

- **Question Answering**: Imagine needing context from several articles
  to answer a complex question. LongLLMLingua helps LLMs retain key
  information across documents, leading to more accurate answers.
- **Machine Translation**: Imagine translating entire books instead of
  single sentences. LongLLMLingua ensures coherence and preserves
  meaning despite lengthy input.
- **Dialogue Systems**: Imagine chatbots understanding intricate
  conversation history. LongLLMLingua enables them to maintain context,
  leading to more natural and engaging interactions.

Real-world examples of LongLLMLingua in action:

- **LongBench Benchmark**: This benchmark measures LLM performance in
  long context scenarios. When applied to GPT-3.5-Turbo, LongLLMLingua
  achieved a 17.1% performance boost with 4x fewer tokens.
- **ZeroScrolls Benchmark**: This benchmark focuses on reading
  comprehension in long contexts. LongLLingua reduced costs by \$27.4
  per 1,000 samples while maintaining performance.

Remember, LongLLMLingua isn't limited to cooking analogies. It's a
powerful tool revolutionising how LLMs handle long contexts,
accelerating progress in various AI domains. By improving efficiency and
maintaining information fidelity, LongLLMLingua opens doors to a future
where AI interactions are more contextual, seamless, and impactful.

**Conclusion**

In conclusion, both LLMLingua and LongLLMLingua are not just
technological advancements; they're culinary metaphors come to life.
LLMLingua acts as the skilled chef, meticulously preparing information
into bite-sized instructions for AI apprentices. LongLLMLingua takes the
baton, transforming into the master chef, seamlessly handling entire
sequences of information like preparing a multi-course feast. Together,
they're revolutionising how LLMs interact with information, paving the
way for a future where:

- AI responses are faster, more accurate, and contextually relevant.
- Machine translation transcends sentence-by-sentence limitations,
  unlocking global communication on a grand scale.
- AI assistants understand our interactions in their entirety, leading
  to more natural and engaging dialogues.

As these technologies continue to evolve, the possibilities are endless.
Imagine AI tutors adapting to your learning style based on years of
educational data, or chatbots understanding months of conversation
history to anticipate your needs with uncanny accuracy.

The future of AI is hungry for efficiency and understanding, and
LLMLingua and LongLLMLingua are serving up the perfect recipe for
success. By empowering LLMs to process information like culinary
masters, they're opening doors to a new era of seamless interaction,
accelerated discovery, and boundless communication.

**Sources**

- LLMLingua research paper: Jiang et al., 2023. LLMLingua: Compressing
  prompts for accelerated inference of Large Language Models. arXiv
  2310.05736v2 <https://arxiv.org/abs/2310.05736>
- LongLLMLingua research paper: Jiang et al., 2023. LongLLMLingua:
  accelerating and enhancing LLMs in long context scenarios via prompt
  compression. arXiv 2310.06839v1 <https://arxiv.org/abs/2310.06839>
- Microsoft LLMLingua page: LLMLingua: Compressing Prompts for
  Accelerated Inference of Large Language Models --- Microsoft Research
  <https://www.microsoft.com/en-us/research/project/llmlingua/>
- Hugging Face LLMLingua Space: LLMLingua --- a Hugging Face
  <https://huggingface.co/spaces/microsoft/LLMLingua>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a632385b206){width="1"
height="1"}


<strong>
 The AI Helper Turning Mountains of Data into Bite-Sized Instructions
</strong>
Author  Aland Astudillo (ORCID: 0009-0008-8672-3168)  LLMs have been changing the way the entire world deals with problems and day-by-day tasks. To make them better for specific applications, they need huge amounts of data and complex and expensive approaches to training them.

Messaggi di Rogue Scholar

What is LLMLingua?