Australian National University

Dhruv Gupta

#### Attention mechanism not getting enough attention

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*rNlIXsxZSIs6p63O" />
<figcaption>Image from Unsplash.</figcaption>
</figure>

### Author

- Dhruv Gupta (**ORCID**:
 [0009--0004--7109--5403](https://orcid.org/0009-0004-7109-5403))

### Introduction

As discussed in this
[article](https://medium.com/@researchgraph/rnns-vs-grus-vs-lstms-d69fd3b3f455),
RNNs were incapable of learning long-term dependencies. To solve this
issue both LSTMs and GRUs were introduced. However, even though LSTMs
and GRUs did a fairly decent job for textual data they did not perform
well. Transformer-based models which first came out in 2017 took the
Natural Language Processing (NLP) world by storm. They were initially
introduced to solve the problem of sequence-to-sequence translation.
However, they have now become the backbone of almost all the generative
AI models. Models like GPT-3 and BERT, use large transformer-based
models for training on huge amounts of data. In this article, we will
discuss the architecture of transformer-based models and how the
attention mechanism works.

### Transformers

Transformer-based models introduced two new concepts which changed the
field of NLP forever. They were attention-based models and
encoder-decoder models. The concept of attention-based models allowed
the language models to focus on only important parts of the text. This
gave the model the ability to comprehend the long-term dependencies. In
addition to this, the attention mechanism along with the encoder-decoder
architecture of the transformer, empowered the model to comprehend the
subtle nuances in the text and get the textual data, like how we humans
do it.

### What is the Attention mechanism?

The attention mechanism allows a machine learning model to emphasise
certain aspects of the input data and this forms the heart of a
transformer model. However, the transformer model uses a set of
self-attention blocks. Self-attention blocks allow the model to focus on
different positions of a single sequence thereby computing a
representation of the sequence. So the important question is how does
this attention mechanism work?

#### **Keys, Values, and Queries?**

Keys, values, and queries are the heart of the self-attention mechanism.
They are used to calculate the self-attention weights using the input
**X**. The input **X** is multiplied by weight matrices that are learnt
during training.

- Query Vector **(Q) = X Wq**. It can be thought of as the current word.
- Key Vector **(K) = X Wk**. This acts as the index value of the
 value vector.
- Value Vector **(V) = X Wv**. Can be considered as the information
 stored in the input word.

What self-attention does is that for every query (**Q**) the most
similar key **K**, is calculated using the dot product between **Q** and
**K**. The dot product is then multiplied with a v vector to get the
information stored in the input word.

<figure>
<img src="https://cdn-images-1.medium.com/max/628/0*viA1Ma9jV8Jen1IW" />
<figcaption>Attention equation being used in transformers. Source:
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez,
A.N., Kaiser, L. and Polosukhin, I. (2017). Attention Is All You
Need. Link:<a
href="https://doi.org/10.48550/arXiv.1706.03762">https://doi.org/10.48550/arXiv.1706.03762</a></figcaption>
</figure>

### Architecture

The transformer uses an encoder-decoder architecture. The idea behind
such an architecture is that the encoder processes the input data and
transforms it into a different representation, which is then
subsequently decoded by the decoder to produce the desired output.
Transformers which were originally designed for translation use this
architecture a bit differently. The encoder is given the input sentence
and the decoder is given the same sentence in the target language.
However, the decoder only gets the words that have been translated. For
example: if there are 5 words in a sentence and three have been
translated. The decoder will get the already translated words and the
original input sentence and will try to predict the fourth word.

<figure>
<img src="https://cdn-images-1.medium.com/max/693/0*HyLiYiY_6k4XOy09" />
<figcaption>Encoder Decoder cell in transformer</figcaption>
</figure>

#### **Diving into the Architecture**

As discussed in the above section the transformer model consists of an
encoder-decoder model.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*KuZhQSjOQVkOx27s" />
<figcaption>Complete Architecture of transformer block. Source: Vaswani,
A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N.,
Kaiser, L. and Polosukhin, I. (2017). Attention Is All You
Need. Link:<a
href="https://doi.org/10.48550/arXiv.1706.03762">https://doi.org/10.48550/arXiv.1706.03762</a></figcaption>
</figure>

#### Encoder

The input data is first converted into word embeddings which are then
concatenated with the positional embeddings. The positional embeddings
play an important role because the word embeddings themselves lack the
positional information which plays a crucial role in textual data. The
encoder model has two sub-layers: multi-head self-attention and the
feed-forward neural network. The multi-head self-attention model
consists of a self-attention mechanism which is applied parallelly. This
output is then passed through to the feed-forward neural network which
then learns from the output of the attention model.

#### Decoder

The decoder also consists of a similar sub-layer as the encoder. There
is a masked multi-head attention block which works on the output vectors
of the previous iteration. The multi-head attention block works on the
output of the encoder and masked multi-head attention in the decoder.
The output of the attention block is then passed through to a fully
connected feed-forward neural network which then produces the output
probability for the next words.

The output of the decoder model is fed into the decoder again just like
a typical RNN model. The masked multi-head attention block is used to
mask the future words so that the decoder only generates the output
using the previously seen outputs and previously seen input from the
encoder model.

### Are they still worthy?

Transformer models which first came out in 2017 for language translation
have changed the face of NLP. While NLP tasks are where transformers are
at their peak, they have a lot of other applications outside of text
processing. They can be used for tasks such as speech recognition, image
captioning, text classification, among others.

Apart from solving NLP-related tasks, transformer models have also
formed the backbone of the new-age generative AI models. Most of the
generative AI models such as GPT, GPT-3, and BERT have stacks of
transformer models in them because of their excellent feature extraction
capability. Additionally, the newer version of generative AI models such
as RAG also has transformer-based encoder-decoder models working in
the backend.

Additionally, big companies such as Google, Facebook, Vault, and
Grammarly which are heavily focused on NLP-based applications still use
transformer-based models in the backend. Their ability to accurately
recognise patterns and context in data makes them an invaluable tool in
the field of NLP.

Therefore, in conclusion, with continued advancements in the field of
LLM, transformers have a big role to play. Hence, with a vast array of
applications and even more to follow, transformers are still worthy of
solving almost all complex NLP-based tasks.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*lAJNOh-EtkhBAAeQ" />
<figcaption>Source: <a
href="https://en.meming.world/wiki/I%27m_Still_Worthy">https://en.meming.world/wiki/I%27m_Still_Worthy</a></figcaption>
</figure>

### **References**

- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez,
 A.N., Kaiser, L. and Polosukhin, I. (2017). *Attention Is All You
 Need*. \[online\] arXiv.org.
 doi:https://doi.org/10.48550/arXiv.1706.03762.
- Kulshrestha, R. (2020). *Transformers*. Medium. Available at:
 <https://towardsdatascience.com/transformers-89034557de14.>
- huggingface.co *How do Transformers work? --- Hugging Face NLP
 Course*. \[online\] Available at:
 <https://huggingface.co/learn/nlp-course/en/chapter1/4>
- Kulshrestha, R. (2020b). *Understanding Attention In Deep Learning*.
 Medium. Available at:
 <https://towardsdatascience.com/attaining-attention-in-deep-learning-a712f93bdb1e.>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=322fb0e912cf){width="1"
height="1"}

Attention mechanism not getting enough attention Author Dhruv Gupta (

 ORCID

: 0009–0004–7109–5403) Introduction As discussed in this article, RNNs were incapable of learning long-term dependencies. To solve this issue both LSTMs and GRUs were introduced. However, even though LSTMs and GRUs did a fairly decent job for textual data they did not perform well.

Messaggi di Rogue Scholar

Transformers Models in NLP