Bastian Greshake Tzovaras

[![a black and white photo of a fence overlooking some train
tracks](/assets/images/2024-07-04-fence.jpg)](/assets/images/2024-07-04-fence.jpg)

Or: Why 2024 is my personal *year of Linux on the desktop*

According to Doctorow, [*enshittification* might be coming *for
absolutely everything*](https://en.wikipedia.org/wiki/Enshittification).
I've more and more felt this was for my own digital devices and means of
digital production: Be that Apple's just increasing love for walled
gardens - especially since having the opaque app review processes (c.f.
[cpython's `--with-app-store-compliance`{.language-plaintext
.highlighter-rouge}](https://github.com/python/cpython/pull/120984)),
Microsoft's (successful?) attempts to [monopolize open source through
GitHub & VS code](https://ghuntley.com/fracture/), or Google's enclosure
of email, maps, and even the browser market. And then there's Adobe's
lock-in through its subscription model and maybe the [final loss of
trust over whether and how they use their customer data to train
'AI'](https://www.theverge.com/2024/6/7/24173838/adobe-tos-update-firefly-generative-ai-trust).

While none of these troublesome developments are really *new*, I feel
like they now collectively have reached my personal tipping point --- at
which it felt like too much of cognitive dissonance between my own
interests in contributing to an open and collectively owned digital
infrastructure, and the closed tools I used for building those. Which is
why I've recently started to divest my personal computing environments
from closed systems, and started to move towards free/open software
alternatives where possible. Luckily, thanks to efforts by folks like
Robert W. Gehl (of [*FOSS Academic*](https://fossacademic.tech/)),
[David Llwellyn-Jones](https://www.flypig.co.uk/blog), and projects like
the [*Opt-Out Project*](https://www.optoutproject.net/) or [*Switching
Software*](https://switching.software/), there is also plenty of
documentation how to do this.

To dip my toes (back or deeper) into it, I started out by replacing *VS
Code* with good old `vim`{.language-plaintext .highlighter-rouge} a few
weeks ago, and have been refreshing my muscle memory for all the
shortcuts ever since. Similarly, I've started to replace *Google Maps*
with [*Organic Maps*](https://organicmaps.app/), which uses
*OpenStreetMap* for its maps (and which also provides an easy way to
make minor contributions to it). The change I was most worried about was
how I would replace Adobe's *Lightroom* as my main photo editing suite
(as that was always one of the main drivers to stick with Mac OS for
me). But, after [seeing the great work done with *Darktable* on
Mastodon](https://mastodon.social/@darktable) I gave that a go and was
a) impressed with the quality of the output and b) more than happy with
using that for my photo workflows.

And so a couple of weeks ago I ordered a machine to replace my aging
MacBook - the DIY version of the 13\" Framework Laptop, to also benefit
from the modular repairability going forward. The assembly and
installing Ubuntu were both without any issues, and since then I\'ve
been using it as my main (non-paid work) machine! And (as I had hoped),
my day to day kept being nearly^[\[1\]](#fn1)^ drama free! I\'ve already
successfully scanned & edited the first \~10 rolls of film using VueScan
& DarkTable (some of which [are already on
Flickr](https://www.flickr.com/photos/gedankenstuecke/)). If you\'re
looking for a free & open alternative to Lightroom I\'d strongly suggest
giving it a try. The paradigms to editing are a bit different to
Adobe\'s, but nothing one wouldn\'t get quite quickly used to --- I had
used Lightroom for 15+ years and still managed to make the switch.

I'm quite happy with the changes made so far, but of course I know that
there's a lot of more things I could (and hopefully will) replace with
more open alternatives in the future. Which I plan on, once I've made
myself comfortable with the replacements done so far. Here's some rough
next steps I already have on my roadmap:

1.  Leave Google Mail for something non-Google
2.  Move my own projects away from GitHub to a free alternative
3.  In the very long run: replace iOS as my mobile OS

If you have recommendations for any of those, [drop me a line on
Mastodon](https://scholar.social/@gedankenstuecke)!

[[\[1\]](#fn1-anchor) This was actually the one slightly less trivial
thing, so this at least partially serves as my own reminder: My 35mm
negative scanner - a Pacific Image Electronics PrimeFilm XA - didn\'t
allow scanning as non-root with plug & play. I fixed this by adding
`SUBSYSTEMS=="usb", ATTRS{idVendor}=="ID", ATTRS{idProduct}=="ID", MODE="0666"`
into a freshly made `/lib/udev/rules.d/40-scanner.rules` file. The `ID`
can be found via `lsusb`.]{.small}


Or: Why 2024 is my personal

 year of Linux on the desktop

According to Doctorow,

 enshittification

might be coming

 for absolutely everything

. I’ve more and more felt this was for my own digital devices and means of digital production: Be that Apple’s just increasing love for walled gardens - especially since having the opaque app review processes (c.f. cpython’s --with-app-store-compliance), Microsoft’s (successful?)

Seizing my means of computation

Australian National University

Qingqin Fang

#### A Unified and Collaborative Framework for LLM

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*ZEyY1nv-Cnfs4BZg" />
<figcaption>Image Generated by Fotor</figcaption>
</figure>

### Author

· Qingqin Fang (**ORCID:**
[0009--0003--5348--4264](https://orcid.org/0009-0003-5348-4264))

### Introduction

In today's rapidly evolving field of artificial intelligence, large
language models (LLMs) are demonstrating unprecedented potential.
Particularly, the Retrieval-Augmented Generation (RAG) architecture has
become a hot topic in AI technology due to its unique technical
capabilities. RAG can effectively reduce errors in generated content by
utilising vast knowledge bases to bring an unprecedented level of
auditability and transparency to the application of LLMs, which is
crucial in modern AI technology.

### Background

The RAG architecture, as one of the most popular and relatively mature
LLM application frameworks currently, has attracted widespread attention
from developers. In today's LLM technology stack, RAG leverages
extensive local knowledge bases for information retrieval, effectively
reducing hallucinations in generated content and enhancing the model's
auditability, thereby supporting more transparent and reliable
application development.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*4OrCYQOX3K6X3SiC" />
<figcaption>Simplified RAG architecture. Draw on Canvas.</figcaption>
</figure>

However, reality is often more complex than theory. Despite the
theoretical advantages of RAG, practical application faces numerous
challenges. In typical RAG systems, although data is processed using
ready-made embedding models, information is efficiently retrieved
through vector databases, and content is generated using complex
black-box language models, these technological components, when
integrated, often exhibit what is termed the "Frankenstein's monster
effect," where each part may function independently but the combined
performance falls far short of expectations. Such systems' fragility,
high dependence on extensive prompts, and lack of specialized
adjustments for specific deployment domains make it difficult for these
systems to pass production tests and are rarely implemented in actual
production.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*9AKGXqnJTF77hCXc" />
<figcaption>RAG 2.0 performs significantly better than existing RAG
systems. Souce: <a
href="https://contextual.ai/introducing-rag2/">https://contextual.ai/introducing-rag2/</a></figcaption>
</figure>

Faced with these challenges, Contextual AI has introduced a new
generation of enterprise-level AI systems --- **RAG 2.0**, which
resolves the limitations of traditional RAG systems through end-to-end
optimization of language models and retrievers. RAG 2.0 is not just a
technological upgrade but a comprehensive response to existing issues,
aimed at achieving more powerful, stable, and highly specialized AI
applications.

### Update of RAG 2.0 {#update-of-rag-2.0}

Building on the traditional RAG system, RAG 2.0 represents a significant
transformation. Previous RAG systems relied on ready-made models and
retrievers that, while effective independently, were simply combined
without deep integration or optimization, often resulting in suboptimal
overall performance.

RAG 2.0 adopts a completely new strategy, *integrating the language
model and retriever into a unified system with end-to-end optimization*.
This means that from pre-training to fine-tuning, and finally to system
alignment, all steps are carried out in coordination. This type of
end-to-end optimization not only significantly improves system
performance but also enhances the model's stability and reliability when
dealing with complex tasks. With this strategy, RAG 2.0 ensures
simultaneous optimization of both the language model and the retriever,
achieving deep collaboration and precise adjustment within the system.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*WTFo6GkpkBiFXF1f" />
<figcaption>Comparison between RAG and RAG 2. Souce: <a
href="https://contextual.ai/introducing-rag2/">https://contextual.ai/introducing-rag2/</a></figcaption>
</figure>

Furthermore, RAG 2.0 utilizes the **backward propagation
algorithm** --- a fundamental and powerful technique in deep learning.
This method allows the system to adjust model parameters in reverse,
based on the discrepancies between predicted outcomes and actual
results, progressively refining the model's performance. This not only
enables the model to better adapt to specific tasks but also enhances
its generalization capabilities for new tasks. In tasks that require
deep language understanding and precise information retrieval, RAG 2.0
can use retrieved information more effectively, producing more accurate
and relevant responses.

### RAG 2.0 in Performance {#rag-2.0-in-performance}

Interestingly, in the tests conducted by Contextual AI on RAG 2.0, not
only were benchmark data provided for open-domain question answering,
fidelity, and novelty on test datasets, but results from real-world
datasets were also shared, showing equally impressive performance.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*CkmCqUO8m7pKntdv" />
<figcaption>Results across knowledge-intensive benchmarks. Souce: <a
href="https://contextual.ai/introducing-rag2/">https://contextual.ai/introducing-rag2/</a></figcaption>
</figure>

Specifically, in tasks such as finance-specific open-book question
answering, RAG 2.0 outperformed the original RAG systems. Similar
enhancements were observed in other specialized fields, including law
and hardware engineering.

<figure>
<img src="https://cdn-images-1.medium.com/max/928/0*gIf9B7EfClEhqmd0" />
<figcaption>CLMs (RAG 2.0) outperform frozen RAG systems on
finance-specific open book question answering. Souce: <a
href="https://contextual.ai/introducing-rag2/">https://contextual.ai/introducing-rag2/</a></figcaption>
</figure>

Meanwhile, for long texts, the benchmark test involves hiding a small
fact within a vast text corpus and evaluating the model's effectiveness
through specific queries. As the length of context increases, it can be
observed that RAG 2.0 achieves higher accuracy and better performance in
handling long contexts while requiring fewer computational resources
than other models. This difference becomes significant in real-world
production.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*fHuKPAKLPrYO43hK" />
<figcaption>Long context models are typically evaluated with <a
href="https://github.com/gkamradt/LLMTest_NeedleInAHaystack">“Needle in
a Haystack”</a> benchmarks wherein a “needle” (i.e., a fact) is hidden
within a large “haystack” (i.e., a corpus of text), and models are
evaluated with a query that aims to elicit the particular needle. RAG
2.0 outperforms. Souce: <a
href="https://contextual.ai/introducing-rag2/">https://contextual.ai/introducing-rag2/</a></figcaption>
</figure>

### RAG in Future Application

Although end-to-end RAG systems have shown potential in handling complex
tasks such as processing lengthy dialogues, there are still many areas
that have not been fully explored. In-depth research into these areas is
crucial for optimizing the performance and practicality of RAG 2.0.
While successful use cases in practical applications are encouraging,
extensive experimentation and improvements are still needed before the
system can be deployed widely and effectively.

#### **Complexity and Cost**

Although the report from [contextual.ai](https://contextual.ai/) does
not provide detailed data, it is conceivable that as RAG integrates all
components into a single unified system through pre-training,
fine-tuning, and alignment, the overall structure's unification will
inevitably increase the model's complexity and training costs, and may
reduce its generalization ability. This is a factor that may need to be
considered when implementing the model in practical applications.

With this comes a sharp increase in costs. However, the current trend
for LLMs is towards lightweight and inexpensive models. While RAG 2.0
has become a trend, the reliance on large amounts of data and complex
models may deter many users and developers. Whether the increase in
these massive models is beneficial or detrimental remains to be
considered.

#### **Response and Delay**

On the other hand, the integration and complexity of RAG 2.0 may not
only impact costs but also affect the model's response times. In
applications where real-time interaction is crucial, the increased
computational demands of a unified, complex system could lead to
noticeable delays. These delays can be problematic in dynamic
environments where timely data processing is essential for
decision-making or user engagement. As the system scales up to handle
more requests or larger datasets, maintaining a balance between accuracy
and speed becomes increasingly challenging. This aspect of RAG 2.0 might
necessitate further optimization strategies to ensure that enhancements
in capabilities do not compromise the responsiveness required by
real-world applications.

### Conclusion

As we continue to explore the capabilities and applications of RAG 2.0,
the technology presents both immense possibilities and notable
challenges. It remains to be seen how its advanced features will be
integrated into everyday technology solutions and what new innovations
will emerge from its use. The ongoing development of RAG 2.0 will
undoubtedly influence the future trajectory of AI, prompting both
academic and industry professionals to rethink how we build, integrate,
and deploy intelligent systems. How will these changes shape the next
generation of AI applications? Only time will tell.

### **References**

- Team, Contextual AI. "Introducing RAG 2.0." *Contextual AI*, 19 Mar.
 2024, contextual.ai/introducing-rag2/. Accessed 30 Apr. 2024.
- Rajput, Vishal. "RAG 2.0: Retrieval Augmented Language Models."
 *AIGuys*, 16 Apr. 2024,
 medium.com/aiguys/rag-2--0-retrieval-augmented-language-models-3762f3047256.
 Accessed 30 Apr. 2024.
- Gregorio, Ignacio de. "RAG 2.0, Finally Getting RAG Right!" *Medium*,
 10 Apr. 2024,
 pub.towardsai.net/rag-2--0-finally-getting-rag-right-f74d0194a720.
 Accessed 30 Apr. 2024.

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9dd3a5b1986a){width="1"
height="1"}

A Unified and Collaborative Framework for LLM Author · Qingqin Fang (

 ORCID:

0009–0003–5348–4264) Introduction In today’s rapidly evolving field of artificial intelligence, large language models (LLMs) are demonstrating unprecedented potential. Particularly, the Retrieval-Augmented Generation (RAG) architecture has become a hot topic in AI technology due to its unique technical capabilities.

Publicaciones de Rogue Scholar

Seizing my means of computation

RAG 2.0 is Coming?