Australian National University

Dhruv Gupta

**Unlocking the power of language models: A deep dive into BERT**

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*B8Q2EUsqwL_rNpW_" />
<figcaption>Image from Unsplash.</figcaption>
</figure>

# **Author:**

- **Dhruv Gupta (ORCID:**
 [**0009--0004--7109--5403**](https://orcid.org/0009-0004-7109-5403)**)**

Clive Humby, in 2006 rightly said, "Data is the new oil". With data
being present everywhere, it has never been more valuable. Almost all
companies have large amounts of textual data and are looking at ways to
draw sense out of it. With the amount of data increasing exponentially,
researchers are moving towards large language models to help them better
utilise the data. In this article, we will discuss one of the first
large language models, BERT which came out in 2018.

### **What is a Large Language Model?**

A large language model (LLM) can be defined as a deep learning algorithm
that can perform a variety of NLP tasks. These models use transformers
and are trained on massive amounts of data. This enables an LLM to
perform tasks such as translation, question-answering, and name entity
recognition on textual data. These models are trained on huge corpora of
data such as Wikipedia, GitHub, etc which allows them to adapt to
different NLP tasks.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*zkaQx2PUe1Wp7RF0" />
<figcaption>An example of a Language Model. Source: Attri.ai Link: <a
href="https://attri.ai/blog/introduction-to-large-language-models">https://attri.ai/blog/introduction-to-large-language-models</a></figcaption>
</figure>

### **What is BERT?**

**BERT** which stands for Bidirectional Encoder Representations for
Transformers first came out in 2018 and took the NLP world by storm. On
a high level, BERT consists of three modules. These modules are:

- Embedding: converts one hot encoding vector to an array
 representation. These array representations of each word represent
 certain features of the word. These features include the context of
 the word and the general meaning of the word. For example: the word
 "horror" will get assigned an array of integers which is very similar
 to the word "scary".
- Encoders: These are transformer encoders which convert embeddings into
 transformer representations. The transformer representations are made
 by augmenting the embedding layer with positional encodings. These
 positional encodings contain the position of different words in the
 input sentence.
- Un-embedding: converts the output generated by the transformer back
 into one hot vector so that the loss can be generated.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*15WpnTSkg_ELNSp9" />
<figcaption>Figure Showing the BERT training process. Source: Devlin,
J., Chang, M.-W., Lee, K., &amp; Toutanova, K. (2018). BERT:
Pre-training of Deep Bidirectional Transformers for Language
Understanding (Version 2). Link: <a
href="https://doi.org/10.48550/ARXIV.1810.04805">https://doi.org/10.48550/ARXIV.1810.04805</a></figcaption>
</figure>

### **BERT Training**

The BERT model was trained in two phases: pre-training and fine-tuning.

**Pre-Training BERT**

BERT is trained using two unsupervised tasks which are described below

1. **Masked LM:** In this task, some part of the sentence is covered
 via a \[MASK\] and the model is asked to predict the masked part of
 the sentence given the context of the words. The training data
 generator chooses 15% of the tokens to be masked at random. The ith
 token chosen for masking is replaced with \[MASK\] 80% of the time
 or a random token 10% of the time or the original token the
 remaining 10% of the time.

**2. Next Sentence Prediction (NSP):** In this task, the model is fed
with a pair of sentences (A, B) and the model is asked to predict
whether the two sentences occurred sequentially or not in the training
corpus. For training purposes, 50% of the time sentences A and B do
occur sequentially in the training corpus. Whereas, the remaining 50% of
the time sentence A is followed by a random sentence.

Pre-training allows BERT to learn the latent representation of words and
sentences in context.

**Fine-Tuning BERT**

Fine-tuning BERT is straightforward since the self-attention mechanism
in the transformer allows BERT to downstream tasks. For each task, the
task-specific inputs and outputs are plugged into BERT and all the
parameters are fine-tuned end to end.

### **BERT Architecture**

BERT model has two sizes: BERT base with total parameters of 110M and
Bert large with total parameters of 340M. The first layer is the
embedding layer, which has three components: word embedding, positional
embedding and token type embedding. The token type embedding helps the
model know what type of token it is. The different types of tokens used
in the BERT model are:

- **CLS**: Represents the classification task for the NSP task.
- **SEP**: Represents the separation between two sentences for the
 NSP task.
- **MASK**: Represents the masked word.
- **UNK**: Represents the words that are out of the vocabulary.

The three outputs are added and then pushed into a normalisation layer.
The normalisation layer outputs an array of vectors of dimension 768.
These are then pushed into 12 Transformer encoders.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*gCvNNkU2FwVrZIfn" />
<figcaption>Figure Showing the Masking process. Source: Devlin, J.,
Chang, M.-W., Lee, K., &amp; Toutanova, K. (2018). BERT: Pre-training of
Deep Bidirectional Transformers for Language Understanding (Version 2).
Article Link: <a
href="https://doi.org/10.48550/ARXIV.1810.04805">https://doi.org/10.48550/ARXIV.1810.04805</a></figcaption>
</figure>

### **BERT Experiments**

BERT achieved state-of-the-art performance on eleven different NLP
tasks. It also pushed the GLUE score (General Language understanding
evaluation) to 80.5%.

For MultiNLI (Multi-Genre Natural Language Interface) the score was
pushed to 86.7% which is a 4.6% absolute improvement.

For SQuAD (Stanford Question answering dataset) v1.1 Test F1 the score
went to 93.2 and 83.1 for v2.0. The dataset contains 100k
question-answer pairs. Given a question, the answer is in the Wikipedia
text. The task was to span the answer. There was an improvement in both
EM (exact matching) and F1(precision and recall of chosen word) scores.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*64Il5eeOZcsD4eKz" />
</figure>

<figure>
<img src="https://cdn-images-1.medium.com/max/724/0*eVvCavzdjisrbdMW" />
</figure>

<figure>
<img src="https://cdn-images-1.medium.com/max/724/0*WzFGjCqjUA6oqU3c" />
<figcaption>Figures showing the results for various datasets. Source:
Devlin, J., Chang, M.-W., Lee, K., &amp; Toutanova, K. (2018). BERT:
Pre-training of Deep Bidirectional Transformers for Language
Understanding (Version 2). Link: <a
href="https://doi.org/10.48550/ARXIV.1810.04805">https://doi.org/10.48550/ARXIV.1810.04805</a></figcaption>
</figure>

### **Conclusion**

BERT when it first came out in 2018 took the NLP world by storm. In
contrast to other deep learning networks, BERT has already been
pre-trained. Therefore, it requires very little data and can be used for
an array of NLP tasks. These properties prevent it from being obsolete
and even today in LLMs like GPT and RAG models we see BERT being used.

### Sources

- Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). BERT:
 Pre-training of Deep Bidirectional Transformers for Language
 Understanding (Version 2). Link:
 <https://doi.org/10.48550/ARXIV.1810.04805>
- "BERT (Language Model)." *Wikipedia*, 24 Sept. 2020,
 en.wikipedia.org/wiki/BERT\_(language_model).
- "What Is a Large Language Model? \| a Comprehensive LLMs Guide."
 [*Www.elastic.co*,](http://Www.elastic.co,)
 [www.elastic.co/what-is/large-language-models#.](http://www.elastic.co/what-is/large-language-models#.)
- "Large Language Models: Basics, Working & Examples \| Attri AI Blog \|
 Attri.ai Blog." *Attri.ai*,
 attri.ai/blog/introduction-to-large-language-models. Accessed 12
 Mar. 2024.
- Rani Horev. "BERT Explained: State of the Art Language Model for NLP."
 *Medium*, Towards Data Science, 10 Nov. 2018,
 towardsdatascience.com/bert-explained-state-of-the-art-language-model-for-nlp-f8b21a9b6270.

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=739f886caacc){width="1"
height="1"}

Unlocking the power of language models: A deep dive into BERT


 Author:


 Dhruv Gupta (ORCID:


 0009–0004–7109–5403


 )

Clive Humby, in 2006 rightly said, “Data is the new oil”. With data being present everywhere, it has never been more valuable.

Messaggi di Rogue Scholar

Language Models: Deep Dive into BERT