Australian National University

Dhruv Gupta

**Unlocking the power of language models: A deep dive into BERT**

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*B8Q2EUsqwL_rNpW_" />
<figcaption>Image from Unsplash.</figcaption>
</figure>

# **Author:**

- **Dhruv Gupta (ORCID:**
 [**0009--0004--7109--5403**](https://orcid.org/0009-0004-7109-5403)**)**

Clive Humby, in 2006 rightly said, "Data is the new oil". With data
being present everywhere, it has never been more valuable. Almost all
companies have large amounts of textual data and are looking at ways to
draw sense out of it. With the amount of data increasing exponentially,
researchers are moving towards large language models to help them better
utilise the data. In this article, we will discuss one of the first
large language models, BERT which came out in 2018.

### **What is a Large Language Model?**

A large language model (LLM) can be defined as a deep learning algorithm
that can perform a variety of NLP tasks. These models use transformers
and are trained on massive amounts of data. This enables an LLM to
perform tasks such as translation, question-answering, and name entity
recognition on textual data. These models are trained on huge corpora of
data such as Wikipedia, GitHub, etc which allows them to adapt to
different NLP tasks.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*zkaQx2PUe1Wp7RF0" />
<figcaption>An example of a Language Model. Source: Attri.ai Link: <a
href="https://attri.ai/blog/introduction-to-large-language-models">https://attri.ai/blog/introduction-to-large-language-models</a></figcaption>
</figure>

### **What is BERT?**

**BERT** which stands for Bidirectional Encoder Representations for
Transformers first came out in 2018 and took the NLP world by storm. On
a high level, BERT consists of three modules. These modules are:

- Embedding: converts one hot encoding vector to an array
 representation. These array representations of each word represent
 certain features of the word. These features include the context of
 the word and the general meaning of the word. For example: the word
 "horror" will get assigned an array of integers which is very similar
 to the word "scary".
- Encoders: These are transformer encoders which convert embeddings into
 transformer representations. The transformer representations are made
 by augmenting the embedding layer with positional encodings. These
 positional encodings contain the position of different words in the
 input sentence.
- Un-embedding: converts the output generated by the transformer back
 into one hot vector so that the loss can be generated.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*15WpnTSkg_ELNSp9" />
<figcaption>Figure Showing the BERT training process. Source: Devlin,
J., Chang, M.-W., Lee, K., &amp; Toutanova, K. (2018). BERT:
Pre-training of Deep Bidirectional Transformers for Language
Understanding (Version 2). Link: <a
href="https://doi.org/10.48550/ARXIV.1810.04805">https://doi.org/10.48550/ARXIV.1810.04805</a></figcaption>
</figure>

### **BERT Training**

The BERT model was trained in two phases: pre-training and fine-tuning.

**Pre-Training BERT**

BERT is trained using two unsupervised tasks which are described below

1. **Masked LM:** In this task, some part of the sentence is covered
 via a \[MASK\] and the model is asked to predict the masked part of
 the sentence given the context of the words. The training data
 generator chooses 15% of the tokens to be masked at random. The ith
 token chosen for masking is replaced with \[MASK\] 80% of the time
 or a random token 10% of the time or the original token the
 remaining 10% of the time.

**2. Next Sentence Prediction (NSP):** In this task, the model is fed
with a pair of sentences (A, B) and the model is asked to predict
whether the two sentences occurred sequentially or not in the training
corpus. For training purposes, 50% of the time sentences A and B do
occur sequentially in the training corpus. Whereas, the remaining 50% of
the time sentence A is followed by a random sentence.

Pre-training allows BERT to learn the latent representation of words and
sentences in context.

**Fine-Tuning BERT**

Fine-tuning BERT is straightforward since the self-attention mechanism
in the transformer allows BERT to downstream tasks. For each task, the
task-specific inputs and outputs are plugged into BERT and all the
parameters are fine-tuned end to end.

### **BERT Architecture**

BERT model has two sizes: BERT base with total parameters of 110M and
Bert large with total parameters of 340M. The first layer is the
embedding layer, which has three components: word embedding, positional
embedding and token type embedding. The token type embedding helps the
model know what type of token it is. The different types of tokens used
in the BERT model are:

- **CLS**: Represents the classification task for the NSP task.
- **SEP**: Represents the separation between two sentences for the
 NSP task.
- **MASK**: Represents the masked word.
- **UNK**: Represents the words that are out of the vocabulary.

The three outputs are added and then pushed into a normalisation layer.
The normalisation layer outputs an array of vectors of dimension 768.
These are then pushed into 12 Transformer encoders.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*gCvNNkU2FwVrZIfn" />
<figcaption>Figure Showing the Masking process. Source: Devlin, J.,
Chang, M.-W., Lee, K., &amp; Toutanova, K. (2018). BERT: Pre-training of
Deep Bidirectional Transformers for Language Understanding (Version 2).
Article Link: <a
href="https://doi.org/10.48550/ARXIV.1810.04805">https://doi.org/10.48550/ARXIV.1810.04805</a></figcaption>
</figure>

### **BERT Experiments**

BERT achieved state-of-the-art performance on eleven different NLP
tasks. It also pushed the GLUE score (General Language understanding
evaluation) to 80.5%.

For MultiNLI (Multi-Genre Natural Language Interface) the score was
pushed to 86.7% which is a 4.6% absolute improvement.

For SQuAD (Stanford Question answering dataset) v1.1 Test F1 the score
went to 93.2 and 83.1 for v2.0. The dataset contains 100k
question-answer pairs. Given a question, the answer is in the Wikipedia
text. The task was to span the answer. There was an improvement in both
EM (exact matching) and F1(precision and recall of chosen word) scores.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*64Il5eeOZcsD4eKz" />
</figure>

<figure>
<img src="https://cdn-images-1.medium.com/max/724/0*eVvCavzdjisrbdMW" />
</figure>

<figure>
<img src="https://cdn-images-1.medium.com/max/724/0*WzFGjCqjUA6oqU3c" />
<figcaption>Figures showing the results for various datasets. Source:
Devlin, J., Chang, M.-W., Lee, K., &amp; Toutanova, K. (2018). BERT:
Pre-training of Deep Bidirectional Transformers for Language
Understanding (Version 2). Link: <a
href="https://doi.org/10.48550/ARXIV.1810.04805">https://doi.org/10.48550/ARXIV.1810.04805</a></figcaption>
</figure>

### **Conclusion**

BERT when it first came out in 2018 took the NLP world by storm. In
contrast to other deep learning networks, BERT has already been
pre-trained. Therefore, it requires very little data and can be used for
an array of NLP tasks. These properties prevent it from being obsolete
and even today in LLMs like GPT and RAG models we see BERT being used.

### Sources

- Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). BERT:
 Pre-training of Deep Bidirectional Transformers for Language
 Understanding (Version 2). Link:
 <https://doi.org/10.48550/ARXIV.1810.04805>
- "BERT (Language Model)." *Wikipedia*, 24 Sept. 2020,
 en.wikipedia.org/wiki/BERT\_(language_model).
- "What Is a Large Language Model? \| a Comprehensive LLMs Guide."
 [*Www.elastic.co*,](http://Www.elastic.co,)
 [www.elastic.co/what-is/large-language-models#.](http://www.elastic.co/what-is/large-language-models#.)
- "Large Language Models: Basics, Working & Examples \| Attri AI Blog \|
 Attri.ai Blog." *Attri.ai*,
 attri.ai/blog/introduction-to-large-language-models. Accessed 12
 Mar. 2024.
- Rani Horev. "BERT Explained: State of the Art Language Model for NLP."
 *Medium*, Towards Data Science, 10 Nov. 2018,
 towardsdatascience.com/bert-explained-state-of-the-art-language-model-for-nlp-f8b21a9b6270.

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=739f886caacc){width="1"
height="1"}

Unlocking the power of language models: A deep dive into BERT


 Author:


 Dhruv Gupta (ORCID:


 0009–0004–7109–5403


 )

Clive Humby, in 2006 rightly said, “Data is the new oil”. With data being present everywhere, it has never been more valuable.

Language Models: Deep Dive into BERT

Yanina Bellini Saibene

Camille Santistevan

Maya Sanghvi

Lou Woodley

Maëlle Salmon

Mauro Lepore

Heather Turner

Pao Corrales

Our first cohort of the [rOpenSci Champions Program](/champions/) has
now completed the first phase of their training.One of the goals of this
program was to ensure that all participants gained new skills and
understanding.We wanted to support Champions and Mentors, but also those
who applied to the program but were not selected.Therefore we ensured
that all groups had access to different training opportunities.

In this article, we share the curriculum created for the Champions,
Mentors, and non-selected applicants, as well as what we learned during
this pilot.

## Training for the Champions

We planned six weeks of training that covered the following topics:

- how to plan and facilitate engaging and inclusive workshops to support
 participants' success,
- knowledge of various channels through which new members can engage in
 and contribute to rOpenSci and R projects, and
- technical skills in software development and review.

The final curriculum for this pilot consisted of 5 courses.In addition,
Champions were invited to the [training session for non-selected
applicants](#training-for-non-selected-applicants) as an optional
activity.

### Making a PACT for engaging virtual meetings and events

In this 90-minute mini-workshop hosted by the [Center for Scientific
Collaboration and Community
Engagement](https://www.cscce.org/),participants explored designing and
facilitating engaging and productive virtual meetings and events.

Participants discussed the opportunities and challenges of working
together online and outlined the "Making a PACT" framework for effective
meeting design. In addition to reviewing examples of the framework in
action, attendees practiced applying it to some example scenarios to
empower them to try using the framework in their own work.Participants
were welcome to review the CSCCE's free guidebook on Zenodo ahead of the
session: [A guide to using virtual events to facilitation community
building: Making a PACT for more engaging virtual meetings and
events](https://zenodo.org/record/4987666#.Y-1Y4nbMK5c)

This training was mandatory for all Champions, and was taught by
[Camille Santistevan](/author/camille-santistevan/) and [Maya
Sanghvi](/author/maya-sanghvi/).

### [Package Development: The Mechanics](https://rpkgdev-mechanics.netlify.app/) {#package-development-the-mechanicshttpsrpkgdev-mechanicsnetlifyapp}

In this two-hour session with a live demo, an rOpenSci trainer
demystified the creation of R packages. R packages are mostly
well-organized folders, with automatic tools to help, such as
usethis.This workshop was intended for Champions planning to develop a
package who had not previously developed or contributed to packages.

This training was developed for the Champions Program and taught by
[Maëlle Salmon](/author/ma%C3%ABlle-salmon/).

### [Package Development: Not Rocket Science](https://rpkgdev-rocket.netlify.app/) {#package-development-not-rocket-sciencehttpsrpkgdev-rocketnetlifyapp}

This two-hour workshop was for R users who had already developed or
contributed to packages or had taken the "Package Development: The
Mechanics workshop". It was intended for Champions planning to develop a
package or submit it for review and who wanted to learn several tips and
tricks on package development.Ideally, participants attended with a
package of their own to use as a playground.

This training was developed for the Champion Program and taught by
[Maëlle Salmon](/author/ma%C3%ABlle-salmon/).

### [How rOpenSci Performs Peer Review](https://github.com/maurolepore/ropensci-review) {#how-ropensci-performs-peer-reviewhttpsgithubcommauroleporeropensci-review}

This one-hour workshop was for people wanting to try the rOpenSci review
process, particularly for Champions planning to send a package to review
or become reviewers. We reviewed the way we communicate, build, and
review software with selected examples and resources, with some time for
questions and comments.

This training was developed for the Champion Program and taught by
[Mauro Lepore](/author/mauro-lepore/).

### [How to contribute to base R](/events/2023-05-18-how-to-contribute-base-r/) {#how-to-contribute-to-base-revents2023-05-18-how-to-contribute-base-r-}

This workshop was a 60-minute walk-through of how to contribute to base
R. It included how to contribute to a bug, confirm a bug, diagnose a
bug, and propose a fix to code/documentation.

The workshop presented tools and strategies to find a bug to work on and
discussed which features make a bug good for a first-timer. Using real
bug reports, participants practiced identifying good first issues. Then
the process of debugging code/documentation was introduced in a
collaborative exercise. Participants were briefly shown how to propose a
bug fix by commenting on Bugzilla or creating a patch using GitHub.
Finally, pointers and resources were shared.

This training was developed and taught by [Heather
Turner](/author/heather-turner/).

## Training for the Mentors

In addition to the following workshop, Mentors were invited to all the
training sessions offered to Champions and non-selected applicants as an
optional activity.

### [rOpenSci Champions Program. Mentorship Training and Orientation](https://ropensci-training.github.io/ropensci-mentors/) {#ropensci-champions-program-mentorship-training-and-orientationhttpsropensci-traininggithubioropensci-mentors}

Mentors developed their mentoring skills during this training and
orientation, which covered active listening, effective questioning, and
giving feedback. They received resources for their mentoring roles, such
as the Mentors' Guidelines, Meeting Templates, and details of the
logistics of the Champions Program.

This training was mandatory for all Mentors. It was developed for the
Champions Program and taught by [Yanina Bellini
Saibene](/author/yanina-bellini-saibene/).

## Training for non-selected applicants

We also conducted a workshop for those not selected to be Champions. The
aim of this workshop was to provide an opportunity for all applicants to
develop skills in open-source software development to support their
professional advancement, which may include future applications to this
or other similar programs.

### [Developing Software Together](https://paocorrales.github.io/git-developing-software-together/) {#developing-software-togetherhttpspaocorralesgithubiogit-developing-software-together}

This was a three-hour course on using Git and GitHub to collaboratively
develop open-source software with the R language. The training presented
how to work with Git, using the command line, RStudio, and GitHub. The
workshop introduced the concept of version control systems, local and
remote repositories, and actions like stage, commit, push, pull, and
merge. It also demonstrated how to collaborate with others through forks
and pull requests. The workshop had hands-on and group exercises.

This training was developed by [Paola Corrales](/author/pao-corrales/)
and taught by Paola and Yanina Bellini Saibene.

## What do our participants think of the training received?

At the end of each workshop, we asked those who participated to fill out
an anonymous survey asking three questions:

- What did you like the most?

- What did you like the least?

- Can you rate this meeting? (on a scale from 1-worst to 5-best).

We plan to use this feedback to improve the next run of the Champions
Program. The average score for all the training sessions was **4.6**,
which shows that the attendees are very satisfied with the workshops.

When we analyze the responses to "What did you like the most?" we found
four positive aspects mentioned: *Interaction with other participants*,
*Structure* of the workshop, *Materials and Resources* shared during the
workshop, and *Exercises* performed during training.

The following plot shows the number of mentions for each aspect and an
example comment explaining why it was perceived positively by the
participants. The comments highlight various aspects of the training,
such as enjoyable interactions, valuable tools provided, opportunities
for reflection and a good structure based on practice after instruction.

<figure class="pull-left">
<img
src="/blog/2023/08/04/champions-program-training/feedback_positive.png"
alt="Ranked quantities of four different positive aspects in a bar plot. The aspects are Material and Resources (15 mentions), Interaction with other participants, (10 mentions), Exercises (8 mentions), and Structure (7 mentions). Each aspect is followed by a comment explaining why it was perceived positively by the participants." />
</figure>

Even though the answers to "What did you like the least?" show that most
attendees think there is *Nothing* they liked the least, we identify
five areas for improvement: *Level* of the training, *More time* for all
the content or exercises, *Tools* used during the workshops, *Exercises*
during the workshop, and *Pace* of the teaching. In the plot we show the
number of mentions and each aspect is followed by a comment explaining
the specific improvement, suggestion, or feedback.

The comments highlight the need for more time to cover content, the
desire for better collaboration tools, the request for additional
exercises, and the suggestion to offer different levels of training, one
for beginners and one for competent practitioners. The comments also
include expressions of overall satisfaction while mentioning challenges,
like time zones.

<figure class="pull-right">
<img
src="/blog/2023/08/04/champions-program-training/feedback_toimprove.png"
alt="Ranked quantities of five areas of improvement in a bar plot. The aspect are Nothing (12 mentions), Level (5 mentions), More time (4 mentions), Tools (4 mentions), Exercises (3 mentions), and Pace (3 mentions). Each aspect has a comment explaining the specific improvement suggestion or feedback." />
</figure>

The survey data provided us with valuable insights into the
participants' perspectives on the strengths and areas for improvement of
our workshops. We're looking forward to taking these comments into
account when we develop the next set of training sessions.

## Next Steps

With a new cohort starting this year ([you can apply
now](/blog/2023/07/14/call-champions-program/)!) we plan to improve the
schedule for each training session, allowing participants to choose
between two runs of the session that cover different time zones, and
recording the sessions for participants to watch after the meeting. We
will develop guidelines to increase accessibility of each session and
will change the order of the workshops. Specifically, training on how to
organize events will be conducted closer to the time when Champions
should start planning their outreach activities. In this way they will
be able to use the workshop to work on their concrete proposals and
receive feedback from the instructors and the rest of their peers.

We also plan to complete another assessment at the end of the program,
including a final survey and interviews with Mentors and Champions with
the support of CSCCE team.

## Achievements

We developed a new curriculum involving not only rOpenSci staff but also
Champions, Mentors, and individuals from other communities, which has
strengthened connections among rOpenSci and others in our shared
ecosystem.

We also published all the training material with open licenses. This has
enabled other organizations, such as [The Netherlands eScience
Center](https://www.esciencecenter.nl/), to adapt and reuse our
materials for training Mentors for their own [Fellowship
program](https://www.esciencecenter.nl/fellowship-programme/). If you
use our training materials, please let us know [by sharing your use
case](/usecases/).

Overall, the rOpenSci Champions Program successfully trained
participants in essential skills for their role as Champions and Mentors
while fostering engagement and collaboration in open-source research
software development.

## Acknowledgements

The rOpenSci Champions Program is funded by [The Chan Zuckerberg
Initiative](https://chanzuckerberg.com/). This phase of the Champions
Program was made possible by the work and feedback of many people.

We want to thank the [CSCCE](https://www.cscce.org/) team, especially
*Lou Woodley*, *Camille Santistevan*, and *Maya Sanghvi*, for their
support in designing the overall program and for their "Making a PACT"
workshop for the Champions.

We also want to thank *Maëlle Salmon*, *Mauro Lepore*, *Paola Corrales*,
*Heather Turner*, and *Yanina Bellini Saibene* for developing and
teaching the rest of the workshops in the program and to *Yanina* for
supporting the design of the new workshops, the support infrastructure
and organizing all the training.

We also thank *Batool Almarzouq* for her feedback on the idea of Mentor
training and *Abigail Cabunoc Mayes* for her advice and material in
developing the Mentor's workshop.

Finally, thanks to our [Champions and Mentors](/champions/) for actively
participating and sharing their recommendations to improve the training
sessions in this program.

Our first cohort of the rOpenSci Champions Program has now completed the first phase of their training.One of the goals of this program was to ensure that all participants gained new skills and understanding.We wanted to support Champions and Mentors, but also those who applied to the program but were not selected.Therefore we ensured that all groups had access to different training opportunities.

rOpenSci Champions Pilot Year: Training Wrap-Up

Andreas Herten

[![Group of Hackathon participants jumping in front of JUWELS Cluster in
the machine
hall](/assets/img/posts/2023-07-03-gpu-hackathon-23/machine-hall-hackathon23.tn.4x.jpg "Group picture of some GPU Hackathon 2023 participants in JSC machine hall; link to FZJ Blogpost about Hackathon"){srcset="/assets/img/posts/2023-07-03-gpu-hackathon-23/machine-hall-hackathon23.tn.4x.jpg 1x, /assets/img/posts/2023-07-03-gpu-hackathon-23/machine-hall-hackathon23.tn.8x.jpg 2x"}](https://blogs.fz-juelich.de/zweikommazwei/2023/06/30/2023-helmholtz-gpu-hackathon/)
Together with our colleagues from Helmholtz-Zentrum Dresden-Rossendorf
(HZDR) and in collaboration with HIDA and OpenHackathons, we hosted the
*Helmholtz GPU Hackathon 2023* in Jülich in May.

**I've blogged about the event for the [*Zweikommazwei* blog of
Forschungszentrum](https://blogs.fz-juelich.de/zweikommazwei/2023/06/30/2023-helmholtz-gpu-hackathon/);
it even includes some video statements we filmed with a few
participants!**

It was the first Hackathon we held in meat space again after the start
of the Pandemic. It was great to see so many excited scientists digging
through their code, the LAN party feeling was strong! This year, we
couldn't use our precious *Rotunda* room at JSC and needed to go to
another building on the campus. That meant hauling tables and chairs,
not only between rooms, but also between
buildings.^[1](#fn:otti){.footnote rel="footnote"}^ But: That allowed us
to accept a few more people!  
Through the support by HIDA, we were able to provide catering to the
event. The attendees appreciated that this allowed them to focus even
more on their GPU hacking. Great!

Despite some Wi-Fi struggles at the beginning of the first day, the
teams were able to make great progress on their respective applications.
Of course, *progress* is very relative. For some, it meant finding a
place to hook in OpenACC statements or even compile with the NVHPC
compiler for a first time. For others, it meant refactoring their
already full-fledged GPU application and squeezing just a little more
juice out of the GPU. In any case, the team of HPC and GPU experts
available helped as much as they could and guided the teams
individually.

We already started planning the Helmholtz Hackathon 2024, most likely
for April. This time, HZDR is responsible for hosting!

::: {.footnotes role="doc-endnotes"}
1.  ::: {#fn:otti}
    Thanks to everyone helping out with logistics and especially, thank
    you, Otti! [↩](#fnref:otti){.reversefootnote role="doc-backlink"}
    :::
:::


Together with our colleagues from Helmholtz-Zentrum Dresden-Rossendorf (HZDR) and in collaboration with HIDA and OpenHackathons, we hosted the

 Helmholtz GPU Hackathon 2023

in Jülich in May.

 I’ve blogged about the event for the
 
 Zweikommazwei
 
 blog of Forschungszentrum;

Messages de Rogue Scholar

Language Models: Deep Dive into BERT

rOpenSci Champions Pilot Year: Training Wrap-Up

Helmholtz GPU Hackathon 2023