Australian National University

Dhruv Gupta

#### **The Three Oldest Pillars of NLP**

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*VwllsakCa6-DJeD_" />
<figcaption>Image from Unsplash.</figcaption>
</figure>

### Author

- Dhruv Gupta (**ORCID**:
 [0009--0004--7109--5403](https://orcid.org/0009-0004-7109-5403))

### Introduction

Natural Language Processing (NLP) has almost become synonymous with
Large Language Models (LLMs), Generative AI, and fancy chatbots. With
the ever-increasing amount of textual data and exponential growth in
computational knowledge, these models are improving every day. However,
people often forget about the initial pillars of deep neural networks
which were used for processing data. In this article, we will talk about
the three oldest pillars of NLP, i.e., Recurrent Neural Networks (RNNs),
Long Short term memory (LSTM) networks and Gated Recurrent Unit (GRU).
Additionally, we will also discuss their limitations and disadvantages.

### Recurrent Neural Networks (RNNs)

The recurrent neural network (RNNs) is a type of neural network where
the output of the previous is fed as an input to the current state. This
was considered to be a breakthrough in the field of NLP because in
traditional neural networks the inputs and outputs were independent. The
idea behind this approach was to mimic the way humans work with
dependencies in the data. Since text data is highly dependent on each
other, RNNs came out to be the perfect solution.

#### Architecture

RNN has a recurrent unit, which has the unique ability to maintain a
hidden state. This hidden state enables the RNN to maintain sequential
dependencies by remembering the previous inputs.

<figure>
<img src="https://cdn-images-1.medium.com/max/177/0*FlIcmULy53oqH6QE" />
<figcaption>Architecture of a RNN . Source: <a
href="https://www.geeksforgeeks.org/introduction-to-recurrent-neural-network/">https://www.geeksforgeeks.org/introduction-to-recurrent-neural-network/</a></figcaption>
</figure>

The hidden state allows the unit to hold the previous knowledge from the
already-seen data. This hidden state is updated at every time step to
acknowledge the change in the knowledge.

<figure>
<img
src="https://cdn-images-1.medium.com/max/792/1*PX_yD2_HyiFUEplqjvzcXw.png" />
<figcaption>Equation of a new state.</figcaption>
</figure>

In this equation, *h\[t\]* denotes the current state, *h\[t-1\]* denotes
the previous state and *x\[t\]* denotes the current input of the
recurrent cell. This ability to remember each piece of information
enabled RNNs to process the sequential data in a much better way.

#### Disadvantages

Even though RNNs provided a crucial breakthrough in working with
sequential data, they had some shortcomings which hindered their ability
to be applied and used for complex NLP tasks. One of the biggest
disadvantages of the RNNs was the vanishing gradient problem. Due to
this issue, RNN could not process longer sequences of text. The RNN
architectures remembered all the information and therefore, were not
able to differentiate between long term and short term dependencies.

### Long Short Term Memory (LSTM)

RNNs were incapable of learning long-term dependencies due to the
presence of a single hidden state. However, LSTMs were able to address
this problem by introducing a memory block to store information. This
memory block allowed LSTM to store information for extended periods
which allowed it to be a perfect solution for NLP tasks such as language
translation.

#### Architecture

The heart of an LSTM lies in the memory cell. The memory cell contains
three different gates, namely, the **input gate**, the **forget gate,**
and the **output gate**. These gates essentially decide what information
needs to be added, removed, and passed onto the next state.

<figure>
<img src="https://cdn-images-1.medium.com/max/964/0*2F5kx_zAjsenn4NQ" />
<figcaption>Architecture of a LSTM cell. Source: <a
href="https://www.linkedin.com/pulse/understanding-lstm-in-depth-look-its-architecture-pros-babu-thomas/">https://www.linkedin.com/pulse/understanding-lstm-in-depth-look-its-architecture-pros-babu-thomas/</a></figcaption>
</figure>

- **Forget gate:** This gate allows LSTM to learn long-term
 dependencies. This gate allows the model to remove all the unnecessary
 information from the previous hidden state, thereby allowing it to
 learn the long-term dependency.
- **Input gate:** The input gate is responsible for the addition of
 useful information into the current cell state. This gate filters down
 the values coming from the previous state and then passes onto the
 filtered value.
- **Output gate:** The output gate is responsible for extracting useful
 information from the current cell state. After extracting the
 necessary information from the current cell state the output is added
 to the output of both forget and input gate and passed onto the
 next cell.

This change in architecture enables the LSTM to store long-term
dependencies and enables it to get rid of the vanishing and exploding
gradients issue of RNNs. It enables LSTMs to capture the important
contexts even if there is a significant gap between the events.

#### Disadvantages

Even though LSTMs capture the context and long-term dependencies they
are computationally expensive. This leads to them needing to be more
scalable. In addition, LSTMs require larger datasets to learn the
dependencies, which was not possible back in the day.

### Gated Recurrent Neural Networks (GRUs)

GRUs as the name suggests use the gating mechanism which is very similar
to LSTMs. However, they are slightly faster due to fewer parameters
to train.

#### Architecture

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*vat-eIu0dddGaUWO" />
<figcaption>Architecture of a GRU cell. Source: <a
href="https://towardsdatascience.com/illustrated-guide-to-lstms-and-gru-s-a-step-by-step-explanation-44e9eb85bf21">https://towardsdatascience.com/illustrated-guide-to-lstms-and-gru-s-a-step-by-step-explanation-44e9eb85bf21</a></figcaption>
</figure>

In comparison to the three gates in LSTM, the GRU only has two gates.

- **Update gate**: This gate is responsible for determining the amount
 of past information that needs to be passed along to the future
 states. This gate prevents all the information from going to the
 future and thereby prevents the model from both vanishing and
 exploding gradients.
- **Reset Gate**: This gate is responsible for maintaining the
 short-term memory of the cell. The gate decides upon the amount of
 past information that needs to be forgotten.

Therefore, the previous hidden state is first passed through the reset
gate. Once the reset gate decides the amount of influence the previous
state will have on the current state it is then passed onto the update
gate. The update gate decides the amount of information from current and
past states that must be passed forward.

#### Disadvantages

Even though GRUs are much lighter than LSTMs they are still
computationally expensive and require huge amounts of data to be trained
upon. Additionally, the GRU-based models are not as interpretable as
other neural networks which makes it difficult to understand how the
predictions are being made.

### Conclusion

Recurrent Neural Network (RNN) was the first of its kind neural network.
It provided the pathway for other neural networks such as LSTMs and GRUs
to come to the surface and improve the work on textual data. However,
due to a lack of computational power, these neural networks were not
scalable. Additionally, after the introduction of transformer-based
neural network architectures, these models went off the radar.
Therefore, even though these models are widely used in the industry
these days, no one talks about their contribution to the advancement in
NLP tasks.

### **References**

- Chugh, A. (2019). *Deep Learning \| Introduction to Long Short Term
 Memory*. \[online\] GeeksforGeeks. Available at:
 <https://www.geeksforgeeks.org/deep-learning-introduction-to-long-short-term-memory/.>
- Analytics Vidhya. (2021). *Gated Recurrent Unit \| Introduction to
 Gated Recurrent Unit(GRU)*. \[online\] Available at:
 <https://www.analyticsvidhya.com/blog/2021/03/introduction-to-gated-recurrent-unit-gru/.>
- GeeksforGeeks (2018). *Introduction to Recurrent Neural
 Network --- GeeksforGeeks*. \[online\] GeeksforGeeks. Available at:
 <https://www.geeksforgeeks.org/introduction-to-recurrent-neural-network/.>
- Staudemeyer, Ralf C and Morris, E.R. (2019). Understanding LSTM --- a
 tutorial into Long Short-Term Memory Recurrent Neural Networks. *arXiv
 (Cornell University)*. doi:https://doi.org/10.48550/arxiv.1909.09586.
- Chung, J., Gulcehre, C., Cho, K. and Bengio, Y. (2014). *Empirical
 Evaluation of Gated Recurrent Neural Networks on Sequence Modeling*.
 \[online\] arXiv.org. doi:https://doi.org/10.48550/arXiv.1412.3555.

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d69fd3b3f455){width="1"
height="1"}

The Three Oldest Pillars of NLP

Author Dhruv Gupta (

 ORCID

: 0009–0004–7109–5403) Introduction Natural Language Processing (NLP) has almost become synonymous with Large Language Models (LLMs), Generative AI, and fancy chatbots. With the ever-increasing amount of textual data and exponential growth in computational knowledge, these models are improving every day.

RNNs vs GRUs vs LSTMs

Wenyi Pi

#### Understanding Sequential Data Modelling with Keras for Time Series Prediction

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*Ob-EpFxiacU7rIQ2" />
<figcaption>Image created in <a
href="https://www.canva.com/">https://www.canva.com/</a></figcaption>
</figure>

### **Author**

- Wenyi Pi (**ORCID**:
 [0009--0002--2884--2771](https://orcid.org/0009-0002-2884-2771))

### Introduction

Recurrent Neural Networks (RNNs) are a special type of neural networks
that are suitable for learning representations of sequential data like
text in Natural Language Processing (NLP). We will walk through a
complete example of using RNNs for time series prediction, covering data
preprocessing, model building, training, evaluation, and visualisation.
This article will introduce Keras for RNN and provide an end-to-end
system using RNN for time series prediction.

#### **Introduction to Keras**

Unlike traditional neural networks which assume that all inputs and
outputs are independent of each other, RNNs make use of sequential
information with the output dependent on the previous computations. For
a more detailed introduction of RNN, please find this article [RNN
Introduction](https://medium.com/@researchgraph/an-introduction-to-recurrent-neural-networks-rnns-802fcfee3098).
Keras, developed by Google, is a high-level deep learning API known for
its modular, user-friendly, and extensible features, facilitating the
implementation of neural networks. In contrast, TensorFlow is an
end-to-end open-source deep learning framework that provides
comprehensive support for various machine learning tasks. Acknowledging
the benefits of both platforms, Google merged Keras into TensorFlow as
its official high-level API, enhancing accessibility and enabling
support for multiple backend neural network computations. This
integration streamlines the implementation of neural networks, providing
users with simplified model development processes.

Below is a basic pipeline when using Keras for building a model:

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*HH0q0-y3JqMeNHB3" />
<figcaption>Building a model in Keras. Image from: <a
href="https://www.simplilearn.com/tutorials/deep-learning-tutorial/what-is-keras#what_is_keras">What
is Keras</a></figcaption>
</figure>

1. **Define a network**: Define different layers in the model and the
 connections between them. Keras has two main model types: Sequential
 and Functional.
2. **Compile a network**: Convert code in a machine understandable
 format. To compile code, we define the **loss function** to
 calculate losses in the model, **optimiser** to reduce the loss, and
 **metrics** to find the model accuracy.
3. **Fit the network**: After compiling the model, train and fit model
 on the data.
4. **Evaluate the network**: After fitting the model, evaluate the
 error in the model.
5. **Make predictions**: Make predictions on new data.

### A complete application for time series prediction

#### **Set up the environment**

We assume that you have Python installed (version 3.11.3). We need to
have the TensorFlow and the TensorFlow Datasets packages installed on
the system. In this tutorial we used the TensorFlow version 2.13.0.

Use:

 pip install tensorflow
 pip install matplotlib

#### **Import the necessary packages**

 from pandas import read_csv
 import numpy as np
 import math
 from keras.models import Sequential
 from keras.layers import Dense, SimpleRNN
 from sklearn.preprocessing import MinMaxScaler
 from sklearn.metrics import mean_squared_error
 import matplotlib.pyplot as plt

#### **Read data from url**

 #Read data from given url and extract the second column
 def read_data(url):
 df = read_csv(url, usecols=[1], engine='python')
 data = np.array(df.values.astype('float32'))
 #Normalise data into (0,1) range 
 scaler = MinMaxScaler(feature_range=(0, 1))
 data = scaler.fit_transform(data).flatten()
 n = len(data)
 return data, n

 sunspots_url = 'https://raw.githubusercontent.com/jbrownlee/Datasets/master/monthly-sunspots.csv'
 data, n = read_data(sunspots_url)

#### **Split data into training and test sets**

 #Splitting data into train and test based on split ratio
 def get_train_test(split_percent, data):
 n = len(data)
 split = int(n * split_percent)
 train_data = data[:split]
 test_data = data[split:]
 return train_data, test_data

 split_percent = 0.8
 train_data, test_data = get_train_test(split_percent, data)

The
[MinMaxScaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html)
scales and translates each feature individually so that it is in the
given range such as \[0, 1\]. The data are split into 80% training set
and 20% testing set.

#### **Reshaping data for Keras**

This step is to prepare data for Keras model training. It is important
to reshape input data in preparing data for use with a neural network.
This helps to ensure that the network can make use of the data. For
sequence prediction tasks like time series forecasting, the input shape
typically is (**total_samples** x **time_steps** x **features**).
**total_samples** is the number of data points in the dataset.
**time_steps** refers to the number of historical observations used to
predict the next time step. **features** denote the number of variables
used for prediction.

 #Reshape data into input-output pairs with specified time steps
 def get_XY(dat, time_steps):
 Y_ind = np.arange(time_steps, len(dat), time_steps)
 Y = dat[Y_ind]
 rows_x = len(Y)
 #Prepare Training and testing data
 X = dat[range(time_steps*rows_x)]
 X = np.reshape(X, (rows_x, time_steps, 1)) 
 return X, Y

 time_steps = 12
 trainX, trainY = get_XY(train_data, time_steps)
 testX, testY = get_XY(test_data, time_steps)

#### **Create the RNN Model and Train**

 #Define the RNN model
 def create_RNN(units, dense_units, input_shape, activation):
 model = Sequential()
 model.add(SimpleRNN(units, input_shape=input_shape, 
 activation=activation[0], return_sequences=True))
 model.add(Dense(dense_units, activation=activation[1]))
 #Compile the model
 model.compile(loss='mean_squared_error', optimizer='adam')
 return model

 model = create_RNN(units=3, dense_units=1, input_shape=(time_steps,1), 
 activation=['tanh', 'tanh'])
 model.fit(trainX, trainY, epochs=10, batch_size=1, verbose=2)

The function above returns a sequential model with two layers, a
**SimpleRNN** layer with three units and a **Dense** layer with one
unit. It defines the input shape at 12×1 and *tanh activation* function
is used. The
[SimpleRNN](https://sztensorflow.org/api_docs/python/tf/keras/layers/SimpleRNN)
layer utilises the TensorFlow Keras API's implementation, ensuring that
the hidden state is recurrently fed back as input to the next state.The
[Dense](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense)
layer is a standard fully connected layer which performs a linear
transformation of the input data followed by an activation function.

A **loss function** measures the difference between the predicted output
of a model and the actual output, while an **optimizer** adjusts the
model's parameters to minimise the loss function. The model is compiled
with the *mean squared error loss* and *Adam gradient descent
optimiser*.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*3Iiyl1nN47inPdCl" />
<figcaption>Model training log output.</figcaption>
</figure>

The model performed as expected, with losses continuing to decline in
each period. This reduction in losses means that the model is
effectively learning from the training data and making progress in
optimising its parameters to better fit the training data. This is a
positive sign that the model is moving towards a solution and improving
its predictive capabilities over time.

#### **Compute and print the Root Mean Square Error (RMSE)**

 #Get error of predictions to evaluate it
 def print_error(trainY, testY, train_predict, test_predict): 
 train_predict = train_predict.reshape(-1)
 test_predict = test_predict.reshape(-1)
 train_rmse = math.sqrt(mean_squared_error(trainY,train_predict))
 test_rmse = math.sqrt(mean_squared_error(testY, test_predict))
 print('Train RMSE: %.3f RMSE' % (train_rmse))
 print('Test RMSE: %.3f RMSE' % (test_rmse)) 

 #Make predictions
 train_predict = model.predict(trainX)
 test_predict = model.predict(testX)
 #Mean square error
 print_error(trainY, testY, train_predict, test_predict)

<figure>
<img src="https://cdn-images-1.medium.com/max/896/0*XIZtghZl8moWnsiz" />
<figcaption>Root Mean Squared Error.</figcaption>
</figure>

RMSE is a measure of the differences between values predicted by the
model and the actual observed values. Lower RMSE values typically
indicate better performance of the model. According to a commonly used
rule of thumb, RMSE values falling between 0.2 and 0.5 indicate that the
model can relatively predict the data accurately. We can find from the
result above, the test RMSE is 0.087 RMSE. Given this value falls lower
than 0.2, it can be inferred that the RMSE values are relatively low.
Consequently, it suggests that the model is performing well on the
testing datasets.

#### **Visualise the Results**

 #Plot the result
 def plot_result(trainY, testY, train_predict, test_predict):
 actual = np.append(trainY, testY)
 predictions = np.append(train_predict, test_predict)
 rows = len(actual)
 plt.figure(figsize=(15, 6), dpi=80)
 plt.plot(range(rows), actual)
 plt.plot(range(rows), predictions)
 plt.axvline(x=len(trainY), color='r')
 plt.legend(['Actual', 'Predictions'])
 plt.xlabel('Observation number after given time steps')
 plt.ylabel('Sunspots scaled')
 plt.title('Actual and Predicted Values. The Red Line Separates The Training And Test Examples')

 plot_result(trainY, testY, train_predict, test_predict)

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*5-0aATxZV4Sk1bFw" />
<figcaption>Visualising the original data and resulting
predictions.</figcaption>
</figure>

According to the graph, we can see that the predicted values closely
match the actual values even in the test set portion, which indicates
that the RNN model has learned the underlying patterns in the data well
and it can effectively make predictions.

### **Conclusion**

In this article, we have shown how to implement a simple Recurrent
Neural Network model for time series prediction using Keras with the
TensorFlow Python package. By following the step-by-step process,
beginners can gain a foundational understanding of RNNs and their
application in sequential data modelling.

### **References**

- Saeed, M. (2023) Understanding simple recurrent neural networks in
 Keras, MachineLearningMastery.com. Available at:
 <https://machinelearningmastery.com/understanding-simple-recurrent-neural-networks-in-keras>.
- Working with RNNS: Tensorflow Core (2023) TensorFlow. Available at:
 <https://www.tensorflow.org/guide/keras/working_with_rnns>.
- Zhou, V. (2020) *Keras for beginners: Implementing a recurrent neural
 network*, *Victor Zhou*. Available at:
 <https://victorzhou.com/blog/keras-rnn-tutorial>.

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b8eb408caa1){width="1"
height="1"}

Understanding Sequential Data Modelling with Keras for Time Series Prediction

 Author

Wenyi Pi (

 ORCID

: 0009–0002–2884–2771) Introduction Recurrent Neural Networks (RNNs) are a special type of neural networks that are suitable for learning representations of sequential data like text in Natural Language Processing (NLP). We will walk through a complete example of using RNNs for time series prediction, covering

Beginner’s Guide to Recurrent Neural Networks (RNNs) with Keras

#### **Understanding how RNNs work and its applications**

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*_d__R3-HMoRfm8Je" />
<figcaption>Image created in <a
href="https://unsplash.com/">https://unsplash.com/</a></figcaption>
</figure>

### **Author**

- Wenyi Pi (**ORCID**:
 [0009--0002--2884--2771](https://orcid.org/0009-0002-2884-2771))

### Introduction

In the ever-evolving landscape of artificial intelligence (AI), bridging
the gap between humans and machines has seen remarkable progress.
Researchers and enthusiasts alike have tirelessly worked across numerous
aspects of this field, bringing about amazing advancements. Among these
domains, machine learning stands out as a pivotal area of exploration
and innovation.

The agenda in the field of machine learning is multifaceted and aims to
improve the ability of computers to learn from data and make predictions
or decisions without explicit programming. This includes a wide array of
tasks, including: pattern recognition, predictive analytics, natural
language processing (NLP), deep learning, etc. The evolution of
applications in NLP, time series analysis, and sequential data
processing has been significantly shaped by the refinement of
**Recurrent Neural Networks (RNNs)**.

This article aims to provide a comprehensive introduction for
understanding of how RNNs work and its applications.

### **What Is a Recurrent Neural Network (RNN)?**

Artificial Neural Networks (ANNs), inspired by the human brain, aim to
teach computers to process data. This involves a machine learning
process (deep learning) which uses interconnected nodes, or neurons, in
a hierarchical structure similar to the human brain. It creates an
adaptive system that computers use to learn from mistakes and constantly
improve. As a result, ANNs attempt to solve complex problems, such as
summarising documents or recognising faces, with greater precision.

A RNN is a special type of ANN adapted to work for time series data or
data that involves sequences. It is trained to process and convert a
sequential data input into a specific sequential data output. Sequential
data is data such as words, sentences, time-series data where sequential
components interrelate based on complex semantic and syntax rules.

### **Unfolding a Recurrent Neural Network**

A simple RNN has a feedback loop, as shown in the figure below. The
feedback loop shown in the grey rectangle can be unrolled in three time
steps to produce the second network below. We can also vary the
architecture so that the network unroll *k-*time steps.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*Kd6TGewYoXMfugw-" />
<figcaption>Compressed representation (top), unfolded network
(bottom). Image from: <a
href="https://machinelearningmastery.com/an-introduction-to-recurrent-neural-networks-and-the-math-that-powers-them/">RNN
Introduction</a></figcaption>
</figure>

At every time step, we can unfold the network for k time steps to get
the output at time *k+1*. The unfolded network is similar to the
feedforward neural network. In the feedforward pass of an RNN, the
network computes the values of the hidden units and the output after
*k-*time steps. The weights associated with the network are shared
temporally. Each recurrent layer has two sets of weights: one for the
input and the second for the hidden unit. The last feedforward layer,
which computes the final output for the kth time step, is just like an
ordinary layer of a traditional feedforward network.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*1a1Dw-44w7D4v8p7" />
<figcaption>An Example of RNN. Image from:<a
href="https://www.youtube.com/watch?v=Y2wfIKQyd1I&amp;t=526s">https://www.youtube.com/watch?v=Y2wfIKQyd1I&amp;t=526s</a></figcaption>
</figure>

These are four single same layers but show the status of different time
steps. Supply the output of the previous word as an input to the second
word to generate text in sequence.

### **Decoding RNN's Operational Dynamics**

RNNs are made of neurons which are data-processing nodes that work
together to perform complex tasks. There are typically four layers in
RNN, the input layer, output layer, hidden layer and loss layer. The
input layer receives information to process, the output layer provides
the result. Positioned between the input and output layers, the hidden
layer can remember and use previous inputs for future predictions based
on the stored memory. The iterative processing unfolds as sequential
data traverses through hidden layers, with each step bringing about
incremental insights and computations.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*JjR9vn-9o9-5ElLZ" />
<figcaption>RNN network. Image from: <a
href="https://aws.amazon.com/what-is/recurrent-neural-network/">https://aws.amazon.com/what-is/recurrent-neural-network/</a></figcaption>
</figure>

### **The Activation Function**

Activation functions determine whether a neuron should be activated or
not by calculating the weighted sum and further adding bias to it. It
can introduce non-linearity that typically converts the output of a
neuron to a value between 0 and 1 or -1 and 1.

Common activation functions that can be used are:

- Sigmoid function
- Tanh function
- ReLu function

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*9YMvxd7VxGXHlJvS" />
<figcaption>Activation functions. Image from Kazi, <a
href="https://www.researchgate.net/figure/This-figure-shows-three-different-activation-functions-graphs-and-its-equations-From_fig4_322537764">activation
function</a></figcaption>
</figure>

### **Training a Recurrent Neural Network**

RNNs are trained by feeding with training data and refining its
performance. Neurons have weights which are used to signal the
importance of information when predicting the result during training.
Weights are adjusted to improve prediction accuracy. A method called
backpropagation through time (BPTT) can calculate model error and adjust
weight relatively. BPTT moves from right to left through the graph that
rolls back output to previous time step and recalculates the error rate
to identify which hidden state is causing errors and adjust this weight
to reduce the error, therefore improving prediction accuracy.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*YwKaA39qaLTh09SE" />
<figcaption>Training RNN. Image from: <a
href="https://www.youtube.com/watch?v=Y2wfIKQyd1I&amp;t=526s">What
is Recurrent Neural Network (RNN)?, codebasics</a></figcaption>
</figure>

The general steps are as follow:

1. Initialise the parameters of the neural network such as weight
 matrices *(U, V, W)* and bias vector *(b, c)*.
2. Forward propagation to compute predictions. This involves computing
 the output of each layer in the network sequentially, using the
 current parameter values.
3. Compute the loss to measure the difference between the predicted
 output and the actual target value.
4. Back-propagation to compute gradients of the loss function. This
 involves propagating the error backward layer by layer through the
 network, and using the calculus chain rule to calculate the
 gradients.
5. Following gradient computation, adjust weights and biases using
 optimisation methods like **SGD** or **Adam** to minimise the loss
 function.
6. Repeat steps 2--5 until criterion is met.

Training result:

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*RQJvHZbVBNg36XE5" />
<figcaption>Training result example. Image from: <a
href="https://www.youtube.com/watch?v=6niqTuYFZLQ">Lecture 10 |
Recurrent Neural Networks, Stanford University School of Engineering
Youtube channel (12 ago 2017)</a></figcaption>
</figure>

Through the training process, the model gradually learns to make better
predictions by adjusting its parameters based on the observed data and
the computed gradients. As you can see from the image above, through
gradual training, the result generated sentences that are more coherent,
structured, and meaningful.

### **Types of RNNs**

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*DDnRh8vQZM7ux8OS" />
<figcaption>RNN Types. Source: <a
href="https://www.youtube.com/watch?v=6niqTuYFZLQ">Lecture 10 |
Recurrent Neural Networks, Stanford University School of Engineering
Youtube channel (12 ago 2017)</a></figcaption>
</figure>

Each rectangle is a vector and arrows represent functions. Input vectors
are in red, output vectors are in blue and green vectors hold the RNN's
state. From left to right:

1. Traditional neural networks employ a **one-to-one** architecture.
 Vanilla neural networks, from fixed-sized input to fixed-sized
 output. e.g. image classification takes an image and outputs a
 single classification word.
2. **One-to-many** networks, a single input can produce multiple
 outputs. Sequence output, from fixed-sized input to sequence of
 variable length. e.g. image captioning takes an image and outputs a
 sentence of words.
3. **Many-to-one** networks, many inputs from different time steps
 produce a single output. Sequence input, from variable-sized input
 to fixed-sized output. e.g. sentiment analysis where a given
 sentence is classified as expressing positive or negative sentiment.
4. **Many-to-many** networks: Sequence input and sequence output, from
 variable-sized input to variable-sized output. e.g. Machine
 Translation, an RNN reads a sentence in English and then outputs a
 sentence in French.
5. **Many-to-many** networks with **synced** sequence: Synced sequence
 input and output, from variable-sized input to variable-sized
 output. e.g. video classification where we wish to label each frame
 of the video.

### **Real Life RNN Use Case**

The following are four useful sequence model applications.

#### 1. Google Gmail {#google-gmail}

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*zmJ_zHDIZEipviy4" />
<figcaption>Google Gmail.</figcaption>
</figure>

When you type a sentence, it will automatically complete it. Google has
this RNN embedded in it.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*m7RrKzZAdgLkeqDH" />
<figcaption>Autogenerate result.</figcaption>
</figure>

#### 2. Google Translate {#google-translate}

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*p5lxmr7LT5M1vwpU" />
<figcaption>Google Translate.</figcaption>
</figure>

Translate sentence from one to another language

#### 3. Named entity recognition (NER) {#named-entity-recognition-ner}

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*3HX8mAkN6pAUDA_9" />
<figcaption>NER. Image from: <a
href="https://www.youtube.com/watch?v=Y2wfIKQyd1I&amp;t=526s">What
is Recurrent Neural Network (RNN)?, codebasics</a></figcaption>
</figure>

Given a statement, it will analyse the text to detect and classify
entities.

#### 4. Sentiment Analysis {#sentiment-analysis}

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*1Ca7YMabX_kssxOm" />
<figcaption>Sentiment Analysis. Image from: <a
href="https://www.youtube.com/watch?v=Y2wfIKQyd1I&amp;t=526s">What
is Recurrent Neural Network (RNN)?, codebasics</a></figcaption>
</figure>

Given a statement, it will analyse text to determine the sentiment or
emotional tone expressed within it.

### **Two RNN Main Limitations**

- [**Exploding
 gradient**](https://deepai.org/machine-learning-glossary-and-terms/exploding-gradient-problem#:~:text=The%20exploding%20gradient%20problem%20is,(weights)%20become%20excessively%20large):
 When individual derivatives are large, the final derivative will also
 become huge and weights would change drastically.
- [**Vanishing
 gradient**](https://en.wikipedia.org/wiki/Vanishing_gradient_problem):
 As the number of hidden layers grows, the gradient becomes very small
 and weights will hardly change. This will hamper the learning process.

### **RNN Variants**

#### **Bidirectional recurrent neural networks (BRNN)**

In
[BRNN](https://en.wikipedia.org/wiki/Bidirectional_recurrent_neural_networks),
data is processed in two directions with both forward and backward
layers to consider past and future contexts. Combining both layers
enables BRNN to have improved prediction accuracy compared to RNN which
only has forward layers.

#### **Long short-term memory (LSTM)**

In [LSTM](https://en.wikipedia.org/wiki/Long_short-term_memory), a model
can expand its memory capacity to accommodate a longer timeline. It has
a special memory block (cells) which is controlled by input gate, output
gate and forget gate, therefore LSTM can remember more helpful
information than RNN.

#### **Gated recurrent units (GRU)**

In [GRU](https://en.wikipedia.org/wiki/Gated_recurrent_unit), it enables
selective memory retention. The model has an update and forget gate
which can store or remove information in the memory.

### **Conclusion**

In conclusion, Recurrent Neural Networks (RNNs) is a powerful and useful
neural network for processing sequential data. With the ability to
process sequence variables, RNN has a wide range of applications in text
generation, text translation, speech recognition, sentiment analysis
etc. Overall, RNNs continue to be a vital tool in the machine learning
and natural language processing field.

### **References**

- Chen, Q. (2020) *Language models and RNN*, *Medium*. Available at:
 <https://medium.com/@rachel_95942/language-models-and-rnn-c516fab9545b> .
- Nabi, J. (2019) *Recurrent neural networks (RNNs)*, *Medium*.
 Available at:
 <https://towardsdatascience.com/recurrent-neural-networks-rnns-3f06d7653a85> .
- Saeed, M. (2023) An introduction to recurrent neural networks and the
 math that powers them, MachineLearningMastery.com. Available at:
 <https://machinelearningmastery.com/an-introduction-to-recurrent-neural-networks-and-the-math-that-powers-them>.
- *What is RNN? --- Recurrent Neural Networks Explained --- AWS*.
 Available at:
 <https://aws.amazon.com/what-is/recurrent-neural-network>.

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=802fcfee3098){width="1"
height="1"}

Understanding how RNNs work and its applications


 Author

Wenyi Pi (

 ORCID

: 0009–0002–2884–2771) Introduction In the ever-evolving landscape of artificial intelligence (AI), bridging the gap between humans and machines has seen remarkable progress. Researchers and enthusiasts alike have tirelessly worked across numerous aspects of this field, bringing about amazing advancements.

Messaggi di Rogue Scholar

RNNs vs GRUs vs LSTMs

Beginner’s Guide to Recurrent Neural Networks (RNNs) with Keras

An Introduction to Recurrent Neural Networks (RNNs)