Australian National University

Dhruv Gupta

#### **The Three Oldest Pillars of NLP**

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*VwllsakCa6-DJeD_" />
<figcaption>Image from Unsplash.</figcaption>
</figure>

### Author

- Dhruv Gupta (**ORCID**:
  [0009--0004--7109--5403](https://orcid.org/0009-0004-7109-5403))

### Introduction

Natural Language Processing (NLP) has almost become synonymous with
Large Language Models (LLMs), Generative AI, and fancy chatbots. With
the ever-increasing amount of textual data and exponential growth in
computational knowledge, these models are improving every day. However,
people often forget about the initial pillars of deep neural networks
which were used for processing data. In this article, we will talk about
the three oldest pillars of NLP, i.e., Recurrent Neural Networks (RNNs),
Long Short term memory (LSTM) networks and Gated Recurrent Unit (GRU).
Additionally, we will also discuss their limitations and disadvantages.

### Recurrent Neural Networks (RNNs)

The recurrent neural network (RNNs) is a type of neural network where
the output of the previous is fed as an input to the current state. This
was considered to be a breakthrough in the field of NLP because in
traditional neural networks the inputs and outputs were independent. The
idea behind this approach was to mimic the way humans work with
dependencies in the data. Since text data is highly dependent on each
other, RNNs came out to be the perfect solution.

#### Architecture

RNN has a recurrent unit, which has the unique ability to maintain a
hidden state. This hidden state enables the RNN to maintain sequential
dependencies by remembering the previous inputs.

<figure>
<img src="https://cdn-images-1.medium.com/max/177/0*FlIcmULy53oqH6QE" />
<figcaption>Architecture of a RNN . Source: <a
href="https://www.geeksforgeeks.org/introduction-to-recurrent-neural-network/">https://www.geeksforgeeks.org/introduction-to-recurrent-neural-network/</a></figcaption>
</figure>

The hidden state allows the unit to hold the previous knowledge from the
already-seen data. This hidden state is updated at every time step to
acknowledge the change in the knowledge.

<figure>
<img
src="https://cdn-images-1.medium.com/max/792/1*PX_yD2_HyiFUEplqjvzcXw.png" />
<figcaption>Equation of a new state.</figcaption>
</figure>

In this equation, *h\[t\]* denotes the current state, *h\[t-1\]* denotes
the previous state and *x\[t\]* denotes the current input of the
recurrent cell. This ability to remember each piece of information
enabled RNNs to process the sequential data in a much better way.

#### Disadvantages

Even though RNNs provided a crucial breakthrough in working with
sequential data, they had some shortcomings which hindered their ability
to be applied and used for complex NLP tasks. One of the biggest
disadvantages of the RNNs was the vanishing gradient problem. Due to
this issue, RNN could not process longer sequences of text. The RNN
architectures remembered all the information and therefore, were not
able to differentiate between long term and short term dependencies.

### Long Short Term Memory (LSTM)

RNNs were incapable of learning long-term dependencies due to the
presence of a single hidden state. However, LSTMs were able to address
this problem by introducing a memory block to store information. This
memory block allowed LSTM to store information for extended periods
which allowed it to be a perfect solution for NLP tasks such as language
translation.

#### Architecture

The heart of an LSTM lies in the memory cell. The memory cell contains
three different gates, namely, the **input gate**, the **forget gate,**
and the **output gate**. These gates essentially decide what information
needs to be added, removed, and passed onto the next state.

<figure>
<img src="https://cdn-images-1.medium.com/max/964/0*2F5kx_zAjsenn4NQ" />
<figcaption>Architecture of a LSTM cell. Source: <a
href="https://www.linkedin.com/pulse/understanding-lstm-in-depth-look-its-architecture-pros-babu-thomas/">https://www.linkedin.com/pulse/understanding-lstm-in-depth-look-its-architecture-pros-babu-thomas/</a></figcaption>
</figure>

- **Forget gate:** This gate allows LSTM to learn long-term
  dependencies. This gate allows the model to remove all the unnecessary
  information from the previous hidden state, thereby allowing it to
  learn the long-term dependency.
- **Input gate:** The input gate is responsible for the addition of
  useful information into the current cell state. This gate filters down
  the values coming from the previous state and then passes onto the
  filtered value.
- **Output gate:** The output gate is responsible for extracting useful
  information from the current cell state. After extracting the
  necessary information from the current cell state the output is added
  to the output of both forget and input gate and passed onto the
  next cell.

This change in architecture enables the LSTM to store long-term
dependencies and enables it to get rid of the vanishing and exploding
gradients issue of RNNs. It enables LSTMs to capture the important
contexts even if there is a significant gap between the events.

#### Disadvantages

Even though LSTMs capture the context and long-term dependencies they
are computationally expensive. This leads to them needing to be more
scalable. In addition, LSTMs require larger datasets to learn the
dependencies, which was not possible back in the day.

### Gated Recurrent Neural Networks (GRUs)

GRUs as the name suggests use the gating mechanism which is very similar
to LSTMs. However, they are slightly faster due to fewer parameters
to train.

#### Architecture

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*vat-eIu0dddGaUWO" />
<figcaption>Architecture of a GRU cell. Source: <a
href="https://towardsdatascience.com/illustrated-guide-to-lstms-and-gru-s-a-step-by-step-explanation-44e9eb85bf21">https://towardsdatascience.com/illustrated-guide-to-lstms-and-gru-s-a-step-by-step-explanation-44e9eb85bf21</a></figcaption>
</figure>

In comparison to the three gates in LSTM, the GRU only has two gates.

- **Update gate**: This gate is responsible for determining the amount
  of past information that needs to be passed along to the future
  states. This gate prevents all the information from going to the
  future and thereby prevents the model from both vanishing and
  exploding gradients.
- **Reset Gate**: This gate is responsible for maintaining the
  short-term memory of the cell. The gate decides upon the amount of
  past information that needs to be forgotten.

Therefore, the previous hidden state is first passed through the reset
gate. Once the reset gate decides the amount of influence the previous
state will have on the current state it is then passed onto the update
gate. The update gate decides the amount of information from current and
past states that must be passed forward.

#### Disadvantages

Even though GRUs are much lighter than LSTMs they are still
computationally expensive and require huge amounts of data to be trained
upon. Additionally, the GRU-based models are not as interpretable as
other neural networks which makes it difficult to understand how the
predictions are being made.

### Conclusion

Recurrent Neural Network (RNN) was the first of its kind neural network.
It provided the pathway for other neural networks such as LSTMs and GRUs
to come to the surface and improve the work on textual data. However,
due to a lack of computational power, these neural networks were not
scalable. Additionally, after the introduction of transformer-based
neural network architectures, these models went off the radar.
Therefore, even though these models are widely used in the industry
these days, no one talks about their contribution to the advancement in
NLP tasks.

### **References**

- Chugh, A. (2019). *Deep Learning \| Introduction to Long Short Term
  Memory*. \[online\] GeeksforGeeks. Available at:
  <https://www.geeksforgeeks.org/deep-learning-introduction-to-long-short-term-memory/.>
- Analytics Vidhya. (2021). *Gated Recurrent Unit \| Introduction to
  Gated Recurrent Unit(GRU)*. \[online\] Available at:
  <https://www.analyticsvidhya.com/blog/2021/03/introduction-to-gated-recurrent-unit-gru/.>
- GeeksforGeeks (2018). *Introduction to Recurrent Neural
  Network --- GeeksforGeeks*. \[online\] GeeksforGeeks. Available at:
  <https://www.geeksforgeeks.org/introduction-to-recurrent-neural-network/.>
- Staudemeyer, Ralf C and Morris, E.R. (2019). Understanding LSTM --- a
  tutorial into Long Short-Term Memory Recurrent Neural Networks. *arXiv
  (Cornell University)*. doi:https://doi.org/10.48550/arxiv.1909.09586.
- Chung, J., Gulcehre, C., Cho, K. and Bengio, Y. (2014). *Empirical
  Evaluation of Gated Recurrent Neural Networks on Sequence Modeling*.
  \[online\] arXiv.org. doi:https://doi.org/10.48550/arXiv.1412.3555.

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d69fd3b3f455){width="1"
height="1"}


<strong>
 The Three Oldest Pillars of NLP
</strong>
Author  Dhruv Gupta (
<strong>
 ORCID
</strong>
: 0009–0004–7109–5403) Introduction   Natural Language Processing (NLP) has almost become synonymous with Large Language Models (LLMs), Generative AI, and fancy chatbots. With the ever-increasing amount of textual data and exponential growth in computational knowledge, these models are improving every day.

Messaggi di Rogue Scholar

RNNs vs GRUs vs LSTMs