Australian National University

Wenyi Pi

#### **Understanding how RNNs work and its applications**

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*_d__R3-HMoRfm8Je" />
<figcaption>Image created in <a
href="https://unsplash.com/">https://unsplash.com/</a></figcaption>
</figure>

### **Author**

- Wenyi Pi (**ORCID**:
 [0009--0002--2884--2771](https://orcid.org/0009-0002-2884-2771))

### Introduction

In the ever-evolving landscape of artificial intelligence (AI), bridging
the gap between humans and machines has seen remarkable progress.
Researchers and enthusiasts alike have tirelessly worked across numerous
aspects of this field, bringing about amazing advancements. Among these
domains, machine learning stands out as a pivotal area of exploration
and innovation.

The agenda in the field of machine learning is multifaceted and aims to
improve the ability of computers to learn from data and make predictions
or decisions without explicit programming. This includes a wide array of
tasks, including: pattern recognition, predictive analytics, natural
language processing (NLP), deep learning, etc. The evolution of
applications in NLP, time series analysis, and sequential data
processing has been significantly shaped by the refinement of
**Recurrent Neural Networks (RNNs)**.

This article aims to provide a comprehensive introduction for
understanding of how RNNs work and its applications.

### **What Is a Recurrent Neural Network (RNN)?**

Artificial Neural Networks (ANNs), inspired by the human brain, aim to
teach computers to process data. This involves a machine learning
process (deep learning) which uses interconnected nodes, or neurons, in
a hierarchical structure similar to the human brain. It creates an
adaptive system that computers use to learn from mistakes and constantly
improve. As a result, ANNs attempt to solve complex problems, such as
summarising documents or recognising faces, with greater precision.

A RNN is a special type of ANN adapted to work for time series data or
data that involves sequences. It is trained to process and convert a
sequential data input into a specific sequential data output. Sequential
data is data such as words, sentences, time-series data where sequential
components interrelate based on complex semantic and syntax rules.

### **Unfolding a Recurrent Neural Network**

A simple RNN has a feedback loop, as shown in the figure below. The
feedback loop shown in the grey rectangle can be unrolled in three time
steps to produce the second network below. We can also vary the
architecture so that the network unroll *k-*time steps.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*Kd6TGewYoXMfugw-" />
<figcaption>Compressed representation (top), unfolded network
(bottom). Image from: <a
href="https://machinelearningmastery.com/an-introduction-to-recurrent-neural-networks-and-the-math-that-powers-them/">RNN
Introduction</a></figcaption>
</figure>

At every time step, we can unfold the network for k time steps to get
the output at time *k+1*. The unfolded network is similar to the
feedforward neural network. In the feedforward pass of an RNN, the
network computes the values of the hidden units and the output after
*k-*time steps. The weights associated with the network are shared
temporally. Each recurrent layer has two sets of weights: one for the
input and the second for the hidden unit. The last feedforward layer,
which computes the final output for the kth time step, is just like an
ordinary layer of a traditional feedforward network.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*1a1Dw-44w7D4v8p7" />
<figcaption>An Example of RNN. Image from:<a
href="https://www.youtube.com/watch?v=Y2wfIKQyd1I&amp;t=526s">https://www.youtube.com/watch?v=Y2wfIKQyd1I&amp;t=526s</a></figcaption>
</figure>

These are four single same layers but show the status of different time
steps. Supply the output of the previous word as an input to the second
word to generate text in sequence.

### **Decoding RNN's Operational Dynamics**

RNNs are made of neurons which are data-processing nodes that work
together to perform complex tasks. There are typically four layers in
RNN, the input layer, output layer, hidden layer and loss layer. The
input layer receives information to process, the output layer provides
the result. Positioned between the input and output layers, the hidden
layer can remember and use previous inputs for future predictions based
on the stored memory. The iterative processing unfolds as sequential
data traverses through hidden layers, with each step bringing about
incremental insights and computations.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*JjR9vn-9o9-5ElLZ" />
<figcaption>RNN network. Image from: <a
href="https://aws.amazon.com/what-is/recurrent-neural-network/">https://aws.amazon.com/what-is/recurrent-neural-network/</a></figcaption>
</figure>

### **The Activation Function**

Activation functions determine whether a neuron should be activated or
not by calculating the weighted sum and further adding bias to it. It
can introduce non-linearity that typically converts the output of a
neuron to a value between 0 and 1 or -1 and 1.

Common activation functions that can be used are:

- Sigmoid function
- Tanh function
- ReLu function

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*9YMvxd7VxGXHlJvS" />
<figcaption>Activation functions. Image from Kazi, <a
href="https://www.researchgate.net/figure/This-figure-shows-three-different-activation-functions-graphs-and-its-equations-From_fig4_322537764">activation
function</a></figcaption>
</figure>

### **Training a Recurrent Neural Network**

RNNs are trained by feeding with training data and refining its
performance. Neurons have weights which are used to signal the
importance of information when predicting the result during training.
Weights are adjusted to improve prediction accuracy. A method called
backpropagation through time (BPTT) can calculate model error and adjust
weight relatively. BPTT moves from right to left through the graph that
rolls back output to previous time step and recalculates the error rate
to identify which hidden state is causing errors and adjust this weight
to reduce the error, therefore improving prediction accuracy.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*YwKaA39qaLTh09SE" />
<figcaption>Training RNN. Image from: <a
href="https://www.youtube.com/watch?v=Y2wfIKQyd1I&amp;t=526s">What
is Recurrent Neural Network (RNN)?, codebasics</a></figcaption>
</figure>

The general steps are as follow:

1. Initialise the parameters of the neural network such as weight
 matrices *(U, V, W)* and bias vector *(b, c)*.
2. Forward propagation to compute predictions. This involves computing
 the output of each layer in the network sequentially, using the
 current parameter values.
3. Compute the loss to measure the difference between the predicted
 output and the actual target value.
4. Back-propagation to compute gradients of the loss function. This
 involves propagating the error backward layer by layer through the
 network, and using the calculus chain rule to calculate the
 gradients.
5. Following gradient computation, adjust weights and biases using
 optimisation methods like **SGD** or **Adam** to minimise the loss
 function.
6. Repeat steps 2--5 until criterion is met.

Training result:

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*RQJvHZbVBNg36XE5" />
<figcaption>Training result example. Image from: <a
href="https://www.youtube.com/watch?v=6niqTuYFZLQ">Lecture 10 |
Recurrent Neural Networks, Stanford University School of Engineering
Youtube channel (12 ago 2017)</a></figcaption>
</figure>

Through the training process, the model gradually learns to make better
predictions by adjusting its parameters based on the observed data and
the computed gradients. As you can see from the image above, through
gradual training, the result generated sentences that are more coherent,
structured, and meaningful.

### **Types of RNNs**

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*DDnRh8vQZM7ux8OS" />
<figcaption>RNN Types. Source: <a
href="https://www.youtube.com/watch?v=6niqTuYFZLQ">Lecture 10 |
Recurrent Neural Networks, Stanford University School of Engineering
Youtube channel (12 ago 2017)</a></figcaption>
</figure>

Each rectangle is a vector and arrows represent functions. Input vectors
are in red, output vectors are in blue and green vectors hold the RNN's
state. From left to right:

1. Traditional neural networks employ a **one-to-one** architecture.
 Vanilla neural networks, from fixed-sized input to fixed-sized
 output. e.g. image classification takes an image and outputs a
 single classification word.
2. **One-to-many** networks, a single input can produce multiple
 outputs. Sequence output, from fixed-sized input to sequence of
 variable length. e.g. image captioning takes an image and outputs a
 sentence of words.
3. **Many-to-one** networks, many inputs from different time steps
 produce a single output. Sequence input, from variable-sized input
 to fixed-sized output. e.g. sentiment analysis where a given
 sentence is classified as expressing positive or negative sentiment.
4. **Many-to-many** networks: Sequence input and sequence output, from
 variable-sized input to variable-sized output. e.g. Machine
 Translation, an RNN reads a sentence in English and then outputs a
 sentence in French.
5. **Many-to-many** networks with **synced** sequence: Synced sequence
 input and output, from variable-sized input to variable-sized
 output. e.g. video classification where we wish to label each frame
 of the video.

### **Real Life RNN Use Case**

The following are four useful sequence model applications.

#### 1. Google Gmail {#google-gmail}

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*zmJ_zHDIZEipviy4" />
<figcaption>Google Gmail.</figcaption>
</figure>

When you type a sentence, it will automatically complete it. Google has
this RNN embedded in it.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*m7RrKzZAdgLkeqDH" />
<figcaption>Autogenerate result.</figcaption>
</figure>

#### 2. Google Translate {#google-translate}

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*p5lxmr7LT5M1vwpU" />
<figcaption>Google Translate.</figcaption>
</figure>

Translate sentence from one to another language

#### 3. Named entity recognition (NER) {#named-entity-recognition-ner}

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*3HX8mAkN6pAUDA_9" />
<figcaption>NER. Image from: <a
href="https://www.youtube.com/watch?v=Y2wfIKQyd1I&amp;t=526s">What
is Recurrent Neural Network (RNN)?, codebasics</a></figcaption>
</figure>

Given a statement, it will analyse the text to detect and classify
entities.

#### 4. Sentiment Analysis {#sentiment-analysis}

<figure>
<img
src="https://cdn-images-1.medium.com/max/1000/0*1Ca7YMabX_kssxOm" />
<figcaption>Sentiment Analysis. Image from: <a
href="https://www.youtube.com/watch?v=Y2wfIKQyd1I&amp;t=526s">What
is Recurrent Neural Network (RNN)?, codebasics</a></figcaption>
</figure>

Given a statement, it will analyse text to determine the sentiment or
emotional tone expressed within it.

### **Two RNN Main Limitations**

- [**Exploding
 gradient**](https://deepai.org/machine-learning-glossary-and-terms/exploding-gradient-problem#:~:text=The%20exploding%20gradient%20problem%20is,(weights)%20become%20excessively%20large):
 When individual derivatives are large, the final derivative will also
 become huge and weights would change drastically.
- [**Vanishing
 gradient**](https://en.wikipedia.org/wiki/Vanishing_gradient_problem):
 As the number of hidden layers grows, the gradient becomes very small
 and weights will hardly change. This will hamper the learning process.

### **RNN Variants**

#### **Bidirectional recurrent neural networks (BRNN)**

In
[BRNN](https://en.wikipedia.org/wiki/Bidirectional_recurrent_neural_networks),
data is processed in two directions with both forward and backward
layers to consider past and future contexts. Combining both layers
enables BRNN to have improved prediction accuracy compared to RNN which
only has forward layers.

#### **Long short-term memory (LSTM)**

In [LSTM](https://en.wikipedia.org/wiki/Long_short-term_memory), a model
can expand its memory capacity to accommodate a longer timeline. It has
a special memory block (cells) which is controlled by input gate, output
gate and forget gate, therefore LSTM can remember more helpful
information than RNN.

#### **Gated recurrent units (GRU)**

In [GRU](https://en.wikipedia.org/wiki/Gated_recurrent_unit), it enables
selective memory retention. The model has an update and forget gate
which can store or remove information in the memory.

### **Conclusion**

In conclusion, Recurrent Neural Networks (RNNs) is a powerful and useful
neural network for processing sequential data. With the ability to
process sequence variables, RNN has a wide range of applications in text
generation, text translation, speech recognition, sentiment analysis
etc. Overall, RNNs continue to be a vital tool in the machine learning
and natural language processing field.

### **References**

- Chen, Q. (2020) *Language models and RNN*, *Medium*. Available at:
 <https://medium.com/@rachel_95942/language-models-and-rnn-c516fab9545b> .
- Nabi, J. (2019) *Recurrent neural networks (RNNs)*, *Medium*.
 Available at:
 <https://towardsdatascience.com/recurrent-neural-networks-rnns-3f06d7653a85> .
- Saeed, M. (2023) An introduction to recurrent neural networks and the
 math that powers them, MachineLearningMastery.com. Available at:
 <https://machinelearningmastery.com/an-introduction-to-recurrent-neural-networks-and-the-math-that-powers-them>.
- *What is RNN? --- Recurrent Neural Networks Explained --- AWS*.
 Available at:
 <https://aws.amazon.com/what-is/recurrent-neural-network>.

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=802fcfee3098){width="1"
height="1"}

Understanding how RNNs work and its applications


 Author

Wenyi Pi (

 ORCID

: 0009–0002–2884–2771) Introduction In the ever-evolving landscape of artificial intelligence (AI), bridging the gap between humans and machines has seen remarkable progress. Researchers and enthusiasts alike have tirelessly worked across numerous aspects of this field, bringing about amazing advancements.

An Introduction to Recurrent Neural Networks (RNNs)

Chris Armit

[![](http://gigasciencejournal.com/blog/wp-content/uploads/2020/07/MIDL2020-300x100.png){.alignnone
.wp-image-3369 loading="lazy" decoding="async" width="600" height="200"
srcset="http://gigasciencejournal.com/blog/wp-content/uploads/2020/07/MIDL2020-300x100.png 300w, http://gigasciencejournal.com/blog/wp-content/uploads/2020/07/MIDL2020-768x256.png 768w, http://gigasciencejournal.com/blog/wp-content/uploads/2020/07/MIDL2020-1024x341.png 1024w, http://gigasciencejournal.com/blog/wp-content/uploads/2020/07/MIDL2020.png 1136w"
sizes="(max-width: 600px) 100vw, 600px"}](Medical%20Imaging%20with%20Deep%20Learning%202020)

Montréal was the venue for Medical Imaging with Deep Learning 2020
([MIDL 2020](https://2020.midl.io/)) that took place on 6-9 July 2020.
Due to the COVID-19 crisis, like many meetings this year (such as the
ISMB conference we attend every year), the new normal has been to make
it a virtual conference. *GigaScience* Data Scientist Chris Armit sat
through the talks as they were streamed, and was deeply impressed at how
Deep Learning is being used to support analysis of clinical imaging
data. If you don't have time to watch the archived content yourselves
Chris has provided his write-up (following his many other [imaging
conference reports](http://gigasciencejournal.com/blog/author/chrisa/))
and curated some videos of his favorite talks here.

**[Deep Learning: Predicting Short-Term Outcome of COVID-19  
]{.underline}**As a keynote speaker, Nikos Paragios of CentraleSupélec
delivered a highly topical and fascinating talk on "AI-driven
quantification, staging outcome prediction of COVID-19 pneumonia". Nikos
explained that there do exist AI-based tools to quantify interstitial
lung diseases that share common CT features with COVID-19, but that
there is a need for a validated model that is able to predict short-term
outcome for COVID-19 lung disease including the need for intubation. The
high dimensional radiomic features utilised by Nikos in his analysis
included: shape features, first-order statistics that describe
individual voxel values, and second-order statistics that describe
textural features calculated between neighbouring voxels. Nikos compared
the ability of 2D AtlasNet, which involves spatially mapping CT image
data of the lung to a template, with 3D U-Net that captures the 3D
texture of the diseased lung and reported that, in this comparison of
Deep Learning ensembles, 2D AtlasNet performed slightly better for
predicting COVID-19 outcome. In addition, Nikos detailed the Holistic
Multi-Omics Selection that was used to determine the most informative
features in the clinical setting, and a highly significant finding was
the importance of biological features, such as age, sex, blood pressure,
CRP, diabetes, and fat index, in predicting COVID-19 outcome. Indeed, by
using an ablation study, whereby biological features and/or imaging data
of the heart and lung were removed from the analysis to find if they are
necessary, Nikos highlighted that imaging was not the most important
feature. This highlights the need for clinical metadata to be included
in the analysis of COVID-19 pneumonia. As we've encouraged [community
annotation of COVID-19
research](http://gigasciencejournal.com/blog/hypothes-is-integration/),
imaging is another area where the richness of curation and metadata is
extremely important.

**[Machine Learning: The Dangers of Multisite Analysis  
]{.underline}**In her keynote talk entitled "Machine Learning: A New
Approach to Drug Discovery", Daphne Koller -- the CEO and Founder of
[insitro](https://insitro.com/) -- highlighted the importance of data
quality by using a highly illustrative example of machine learning
applied to X-ray images and that was originally reported by [Badgeley et
al](https://doi.org/10.1038/s41746-019-0105-1). On initial inspection,
this Machine Learning model offered a clean separation of X-ray images
with fracture and those without fracture. However, on closer inspection
the underlying predictive model had picked up on differences between the
various scanners used in different hospitals, and the Machine Learning
algorithm was simply finding the hospital scanner that had been used to
scan a higher incidence of fractures. Consequently, the device used to
capture the radiograph was, in this instance, the main source of
variation.

As a means of addressing this issue, Daniel Moyer of USC presented an
insightful "Overview of Scanner Invariant Representations". As Daniel
explained, whereas supervised problems are often of the form data x
(image) predicts label y (diagnoses, outcomes), multi-site analyses can
be a problem as "multi-site analyses have varying site signals". The
task then becomes how to remove scanner bias prior to performing the
supervised problem. At the conference, Daniel reported that there is
value in using a Deep Learning method based on variational
[auto-encoders](https://en.wikipedia.org/wiki/Autoencoder) (VAE) to
construct scanner invariant encodings of the imaging data. A key feature
of this approach is that it uses a conditional auto-encoder that
utilises adversarial image-space losses to remove scanner bias. In a
poster session, Daniel showcased how this approach has been successfully
applied to the [2018 MICCAI Computational Diffusion MRI (CDMRI)
Challenge Harmonization](http://cmic.cs.ucl.ac.uk/cdmri18/) dataset.

**[Predicting Age-Related Changes in the Brain  
]{.underline}**Marianne Rakic of MIT and ETH Zürich showcased
"Anatomical Predictions using Subject-Specific Medical Data". Given a
baseline MRI brain scan and subject specific information, Marianne
questioned whether it is possible to predict the follow-up MRI brain
scan at some later time? This would be incredibly useful as it could
help guide clinical treatment and may also provide scientific insight.
To test this hypothesis, Marianne captured the difference between a
baseline MRI brain scan and a follow-up MRI brain scan by using a
deformation field. A [Convolutional Neural Network
(CNN)](https://en.wikipedia.org/wiki/Convolutional_neural_network) was
then used to predict the deformation field. Marianne reports that this
approach performs better on brain ventricles than on smaller structures,
but that it is further improved by the inclusion of non-image
information, such as knowing the age of the patient. As with the
aforementioned keynote lecture by Nikos Paragios on COVID-19 pneumonia,
a key message was that there is great value in clinical metadata in
these Deep Learning approaches.

**[Digital Pathology -- When CNNs Fail  
]{.underline}**Jasper Linmans of Radboud UMC, working with Jeroen van
der Laak and Geert Litjens, reported on "Efficient Out-of-Distribution
Detection in Digital Pathology Using Multi-Head Convolutional Neural
Networks". Jeroen van der Laak and Geert Litjens had previously
published the 1399 H&E-stained sentinel lymph node sections of breast
cancer patients used in the CAMELYON study in
[*GigaScience*](https://doi.org/10.1093/gigascience/giy065), and so I
was intrigued to find out about their recent research on Convolutional
Neural Networks (CNNs). As Jasper explained, "CNNs fail silently" and
there is a need for Out-of-Distribution (OOD) detection as a means of
delivering an uncertainty score. For detecting OOD, most popular
approaches measure entropy value using Deep Ensembles or Mc-Dropout.
However, Jasper instead proposes using Multi-Head CNNs that are both
computationally and memory efficient. As Jasper explained, "the biggest
advantage of being more memory efficient is the ability to train the
entire model at once and to promote diversity during training". Jasper
showcased this method on the CAMELYON lymphoma data, and highlighted how
Multi-Head CNNs offer efficient OOD detection, and can be used to
correct Machine Learning model false-positive predictions.

**[Generative Adversarial Networks and Tensor Networks in Digital
Pathology  
]{.underline}**In the field of Digital Pathology, there is a continuing
focus on Machine Learning as a tool for automated diagnosis. However, I
was deeply impressed by the presentation by Adalberto Claudio Quiros of
the University of Glasgow who reported on "Pathology GAN: Learning deep
representations of cancer tissue". Adalberto explained that there are
limitations on supervised learning as "it cannot provide unknown
information about the data" and that consequently there is need for a
generative model that can find fundamental morphological characteristics
of tissue and that can be used to "identify and reproduce the different
types of tissue". Adalberto utilised [Generative Adversarial Networks
(GAN)](https://en.wikipedia.org/wiki/Generative_adversarial_network),
whereby two neural networks contest with each other to deliver
high-level features of histology image data, and applied this approach
to deep representation of the entire tissue architecture of cancer
tissue. Whereas other existing GAN-based tools -- such as
[BigGAN](https://arxiv.org/abs/1809.11096) -- randomly place vector
samples in latent space, a key advantage of Pathology GAN is that it
shows structure in the latent space thus making the image that is
generated interpretable. Adalberto anticipates that the disentangled
representations delivered by Pathology GAN will provide "further
understanding on phenotype diversity between and within tumours."

I was additionally impressed by the presentation by Raghavendra Selvan,
who reported on "Tensor Networks for Medical Image Classification".
Raghavendra suggested that, rather than flatten images into a
one-dimensional vector, an alternative approach is to create a Locally
orderless Tensor Network that can be shown as a series of layers. A
tensor is a multidimensional array of numerical values that can be used
to describe an image, and in tensor notation a zero edge tensor is a
scalar, a one-edge tensor is a 1D vector, a two-edge tensor is a 2D
matrix, and a three-edge tensor is a 3D matrix. Raghavendra's approach
is to first flatten a 2D image into a vector, but then to use tensor
products to aggregate at multiple resolutions to create a high
dimensional feature map. Raghavendra reported that this "squeeze
operation" helps retain structure. The tensor network approach has the
added value that it leads to a massive reduction in GPU utilisation,
which is a known issue when processing very large images.

Medical Imaging with Deep Learning 2020 was a virtual conference and as
we are rapidly learning these can be something of a challenge. However,
the organisers are to be commended for delivering an exceptional online
experience, and for providing a much-needed forum for deep learning
researchers and clinicians at the intersection of machine learning and
medical image analysis. One advantage of online conferences is that it's
easier to archive the video content, and the replay videos of [day
1](https://youtu.be/wNJQScMgpsI), [day 2](https://youtu.be/qWTgWgurlR0)
and [day3](https://youtu.be/fCjtNyUkji0) are all available on youtube.

MIDL 2020 additionally highlighted the need for high-level curation of
medical imaging data that is used for Machine Learning and Deep
Learning. The [International Society of
Biocuration](https://www.biocuration.org/) has touched on this subject
previously, and have explicitly stated how important it is to annotate
imaging data and ["to properly attribute and track data
provenance"](https://doi.org/10.1371/journal.pcbi.1006906).
Collaboration between ISB and MISL 2020 may help to further address this
need, and to raise awareness of the needs of the Deep Learning community
amongst biocurators.

The post [Medical Imaging with Deep Learning in the Time of
Coronavirus](http://gigasciencejournal.com/blog/medical-imaging-with-deep-learning/){rel="nofollow"}
appeared first on
[GigaBlog](http://gigasciencejournal.com/blog){rel="nofollow"}.


Montréal was the venue for Medical Imaging with Deep Learning 2020 (MIDL 2020) that took place on 6-9 July 2020. Due to the COVID-19 crisis, like many meetings this year (such as the ISMB conference we attend every year), the new normal has been to make it a virtual conference.

Rogue Scholar Posts

An Introduction to Recurrent Neural Networks (RNNs)

Medical Imaging with Deep Learning in the Time of Coronavirus