Australian National University

Dhruv Gupta

**Unlocking the power of language models: A deep dive into BERT**

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*B8Q2EUsqwL_rNpW_" />
<figcaption>Image from Unsplash.</figcaption>
</figure>

# **Author:**

- **Dhruv Gupta (ORCID:**
 [**0009--0004--7109--5403**](https://orcid.org/0009-0004-7109-5403)**)**

Clive Humby, in 2006 rightly said, "Data is the new oil". With data
being present everywhere, it has never been more valuable. Almost all
companies have large amounts of textual data and are looking at ways to
draw sense out of it. With the amount of data increasing exponentially,
researchers are moving towards large language models to help them better
utilise the data. In this article, we will discuss one of the first
large language models, BERT which came out in 2018.

### **What is a Large Language Model?**

A large language model (LLM) can be defined as a deep learning algorithm
that can perform a variety of NLP tasks. These models use transformers
and are trained on massive amounts of data. This enables an LLM to
perform tasks such as translation, question-answering, and name entity
recognition on textual data. These models are trained on huge corpora of
data such as Wikipedia, GitHub, etc which allows them to adapt to
different NLP tasks.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*zkaQx2PUe1Wp7RF0" />
<figcaption>An example of a Language Model. Source: Attri.ai Link: <a
href="https://attri.ai/blog/introduction-to-large-language-models">https://attri.ai/blog/introduction-to-large-language-models</a></figcaption>
</figure>

### **What is BERT?**

**BERT** which stands for Bidirectional Encoder Representations for
Transformers first came out in 2018 and took the NLP world by storm. On
a high level, BERT consists of three modules. These modules are:

- Embedding: converts one hot encoding vector to an array
 representation. These array representations of each word represent
 certain features of the word. These features include the context of
 the word and the general meaning of the word. For example: the word
 "horror" will get assigned an array of integers which is very similar
 to the word "scary".
- Encoders: These are transformer encoders which convert embeddings into
 transformer representations. The transformer representations are made
 by augmenting the embedding layer with positional encodings. These
 positional encodings contain the position of different words in the
 input sentence.
- Un-embedding: converts the output generated by the transformer back
 into one hot vector so that the loss can be generated.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*15WpnTSkg_ELNSp9" />
<figcaption>Figure Showing the BERT training process. Source: Devlin,
J., Chang, M.-W., Lee, K., &amp; Toutanova, K. (2018). BERT:
Pre-training of Deep Bidirectional Transformers for Language
Understanding (Version 2). Link: <a
href="https://doi.org/10.48550/ARXIV.1810.04805">https://doi.org/10.48550/ARXIV.1810.04805</a></figcaption>
</figure>

### **BERT Training**

The BERT model was trained in two phases: pre-training and fine-tuning.

**Pre-Training BERT**

BERT is trained using two unsupervised tasks which are described below

1. **Masked LM:** In this task, some part of the sentence is covered
 via a \[MASK\] and the model is asked to predict the masked part of
 the sentence given the context of the words. The training data
 generator chooses 15% of the tokens to be masked at random. The ith
 token chosen for masking is replaced with \[MASK\] 80% of the time
 or a random token 10% of the time or the original token the
 remaining 10% of the time.

**2. Next Sentence Prediction (NSP):** In this task, the model is fed
with a pair of sentences (A, B) and the model is asked to predict
whether the two sentences occurred sequentially or not in the training
corpus. For training purposes, 50% of the time sentences A and B do
occur sequentially in the training corpus. Whereas, the remaining 50% of
the time sentence A is followed by a random sentence.

Pre-training allows BERT to learn the latent representation of words and
sentences in context.

**Fine-Tuning BERT**

Fine-tuning BERT is straightforward since the self-attention mechanism
in the transformer allows BERT to downstream tasks. For each task, the
task-specific inputs and outputs are plugged into BERT and all the
parameters are fine-tuned end to end.

### **BERT Architecture**

BERT model has two sizes: BERT base with total parameters of 110M and
Bert large with total parameters of 340M. The first layer is the
embedding layer, which has three components: word embedding, positional
embedding and token type embedding. The token type embedding helps the
model know what type of token it is. The different types of tokens used
in the BERT model are:

- **CLS**: Represents the classification task for the NSP task.
- **SEP**: Represents the separation between two sentences for the
 NSP task.
- **MASK**: Represents the masked word.
- **UNK**: Represents the words that are out of the vocabulary.

The three outputs are added and then pushed into a normalisation layer.
The normalisation layer outputs an array of vectors of dimension 768.
These are then pushed into 12 Transformer encoders.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*gCvNNkU2FwVrZIfn" />
<figcaption>Figure Showing the Masking process. Source: Devlin, J.,
Chang, M.-W., Lee, K., &amp; Toutanova, K. (2018). BERT: Pre-training of
Deep Bidirectional Transformers for Language Understanding (Version 2).
Article Link: <a
href="https://doi.org/10.48550/ARXIV.1810.04805">https://doi.org/10.48550/ARXIV.1810.04805</a></figcaption>
</figure>

### **BERT Experiments**

BERT achieved state-of-the-art performance on eleven different NLP
tasks. It also pushed the GLUE score (General Language understanding
evaluation) to 80.5%.

For MultiNLI (Multi-Genre Natural Language Interface) the score was
pushed to 86.7% which is a 4.6% absolute improvement.

For SQuAD (Stanford Question answering dataset) v1.1 Test F1 the score
went to 93.2 and 83.1 for v2.0. The dataset contains 100k
question-answer pairs. Given a question, the answer is in the Wikipedia
text. The task was to span the answer. There was an improvement in both
EM (exact matching) and F1(precision and recall of chosen word) scores.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*64Il5eeOZcsD4eKz" />
</figure>

<figure>
<img src="https://cdn-images-1.medium.com/max/724/0*eVvCavzdjisrbdMW" />
</figure>

<figure>
<img src="https://cdn-images-1.medium.com/max/724/0*WzFGjCqjUA6oqU3c" />
<figcaption>Figures showing the results for various datasets. Source:
Devlin, J., Chang, M.-W., Lee, K., &amp; Toutanova, K. (2018). BERT:
Pre-training of Deep Bidirectional Transformers for Language
Understanding (Version 2). Link: <a
href="https://doi.org/10.48550/ARXIV.1810.04805">https://doi.org/10.48550/ARXIV.1810.04805</a></figcaption>
</figure>

### **Conclusion**

BERT when it first came out in 2018 took the NLP world by storm. In
contrast to other deep learning networks, BERT has already been
pre-trained. Therefore, it requires very little data and can be used for
an array of NLP tasks. These properties prevent it from being obsolete
and even today in LLMs like GPT and RAG models we see BERT being used.

### Sources

- Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). BERT:
 Pre-training of Deep Bidirectional Transformers for Language
 Understanding (Version 2). Link:
 <https://doi.org/10.48550/ARXIV.1810.04805>
- "BERT (Language Model)." *Wikipedia*, 24 Sept. 2020,
 en.wikipedia.org/wiki/BERT\_(language_model).
- "What Is a Large Language Model? \| a Comprehensive LLMs Guide."
 [*Www.elastic.co*,](http://Www.elastic.co,)
 [www.elastic.co/what-is/large-language-models#.](http://www.elastic.co/what-is/large-language-models#.)
- "Large Language Models: Basics, Working & Examples \| Attri AI Blog \|
 Attri.ai Blog." *Attri.ai*,
 attri.ai/blog/introduction-to-large-language-models. Accessed 12
 Mar. 2024.
- Rani Horev. "BERT Explained: State of the Art Language Model for NLP."
 *Medium*, Towards Data Science, 10 Nov. 2018,
 towardsdatascience.com/bert-explained-state-of-the-art-language-model-for-nlp-f8b21a9b6270.

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=739f886caacc){width="1"
height="1"}

Unlocking the power of language models: A deep dive into BERT


 Author:


 Dhruv Gupta (ORCID:


 0009–0004–7109–5403


 )

Clive Humby, in 2006 rightly said, “Data is the new oil”. With data being present everywhere, it has never been more valuable.

Language Models: Deep Dive into BERT

We ducked a fundamental question raised by our proposal for infrastructure principles: â€œwhat exactly counts as infrastructure?â€ Of course this is not a straightforward question and part of the reason for leaving it in untouched in the introductory post. We believe that any definition must entail a much broader discussion from the community.

Geoff Bilder

::: {.wp-caption .alignleft .zemanta-img style="width: 360px"}
[![This is a photo of a black leopard from the
Ou\...](http://cameronneylon.net/wp-content/uploads/2015/08/350px-blackleopard.jpg "This is a photo of a black leopard from the Ou..."){.zemanta-img-inserted
.zemanta-img-configured decoding="async" loading="lazy" width="350"
height="207"}](http://commons.wikipedia.org/wiki/File:Blackleopard.JPG){target="_blank"}

Black leopard from Out of Africa Wildlife Park in Arizona (Wikipedia)
:::

*Cite as: What exactly is infrastructure? Seeing the leopard's spots.
Geoffrey Bilder, Jennifer Lin, Cameron Neylon.
[fig**share**](http://figshare.com).
<http://dx.doi.org/10.6084/m9.figshare.1520432>*

We ducked a fundamental question raised by our proposal for
infrastructure principles: *â€œwhat exactly counts as
infrastructure?â€?* This question matters. If our claim is that
infrastructures should operate according to a set of principles, we need
to be able to identify the thing of which we speak. Call the leopard, a
leopard. Of course this is not a straightforward question and part of
the reason for leaving it in untouched in the introductory post. We
believe that any definition must entail a much broader discussion from
the community. But we wanted to kick this off with a discussion of an
important part of the infrastructure puzzle that we think is often
missed.

In our conversations with scholars and others in the research ecosystem,
people frequently speak of â€œinfrastructuresâ€? when what they mean are
services on top of deeper infrastructures. Or indeed infrastructures
that sit on deeper infrastructures. Most people think of the web as an
essential piece of infrastructures and it is the platform that makes
much of what we are talking about possible. But the web is built on
deeper layers of infrastructure: the internet, MAC addresses, IP, and
TCP/IP. Things that many readers will never even have heard of because
they have disappeared from view. . Similarly in academia, a researcher
will point to â€œCERNâ€? or â€œGenbankâ€? or â€œPubmedâ€? or â€œThe
[Perseus
Project](http://en.wikipedia.org/wiki/Perseus_Project "Perseus Project"){.zem_slink
target="_blank" rel="wikipedia"}â€? when asked about critical
infrastructure. The Physicists at CERN have long since taken the
plumbing, electricity, roads, tracks, airports etc. that make CERN
possible for granted.

All these examples involve services operating a layer above ones we have
been considering. Infrastructure is not commonly seen. That lower level
infrastructure has become invisible just as the underlying network
protocols, which make Genbank,
[PubMed](http://www.ncbi.nlm.nih.gov/pubmed/ "PubMed"){.zem_slink
target="_blank" rel="homepage"} and the Perseus Project possible have
also long since become invisible and taken for granted. To put a prosaic
point on it, what is not commonly seen is also not commonly noticed. And
that makes it even more important for us to focus on getting these
deeper layers of infrastructure right.

If doing research entails the search for new discoveries, those involved
are more inclined to focus on what is *different* about their research.
Every sub-community within academia tends to think at a level of
abstraction that is typically one layer above the truly essential â€"
and shared â€" infrastructure. We hear physicists, chemists, biologists,
humanists at meetings and conferences assume that the problems that they
are trying to solve in online scholarly communication are specific to
their particular discipline. They say â€œwe need to uniquely identify
antibodiesâ€? or â€œwe need storage for astronomy dataâ€? or â€œwe need
to know which journals are open accessâ€? or â€œhow many times has this
article been downloadedâ€?. Then they build the thing that they (think
they) need.

This then leads to another layer of invisibility â€" the infrastructures
that we were concerned with in the [*Principles for Open Scholarly
Infrastructure*](http://cameronneylon.net/blog/principles-for-open-scholarly-infrastructures/)
are about what is *the same* across disciplines, not what is different.
It is precisely the fact that these common needs are boring that means
they starts to disappear from view, in some cases before they even get
built. For us it is almost a law: people tend to identify infrastructure
one layer too high. We want to refocus attention on the layer below, the
one that is disappearing from view. It turns out that a black
leopardâ€™s spots can be seen -- [once theyâ€™re viewed under infrared
light](http://www-public.jcu.edu.au/news/JCU_148708).

So where does that leave us? What *we* hear in these conversations
across disciplines are not the things that are different (ie., what is
special about antibodies or astronomy data or journal classifications or
usage counts) but what is in common across all of these problems. This
class of common problems need shared solutions. â€œWe need
identifiersâ€? and â€œwe need storageâ€? and â€?we need to assign
metadataâ€? and â€œwe need to record relationshipsâ€?. These
infrastructures are ones that will allow us to *identify* objects of
interest (ex: identify new kinds of research objects), *store* resources
where more specialised storage doesnâ€™t already exists (ex: validate a
data analysis pipeline), and record *metadata* and *relationships*
between resources, objects and ideas (ex: describing the relationships
between funders and datasets). For example, the [Directory of Open
Access
Journals](http://www.doaj.org/ "Directory of Open Access Journals"){.zem_slink
target="_blank" rel="homepage"} provides identifiers for Open Access
journals, claims about such journals, and *relationships* with other
resources and objects (such as article level metadata, like
[Crossref](http://crossref.org "CrossRef"){.zem_slink target="_blank"
rel="homepage"} DOIs and Creative Commons license URLs).

But what has generally happened in the past Â is that each group
re-invents the wheel for its own particular niche. Specialist resources
build a whole stack of tools rather than layering on the one specific
piece that they need on an existing set of infrastructures. There is an
important counter-example: the ability to easily cross-reference
articles and datasets as well as connect these to the people who created
them. This is made possible by Crossref and Datacite DOIs with ORCID
IDs. ORCID is an infrastructure that provides *identifiers* for people
as well as metadata and claims about *relationships* between people and
other resources (e.g., articles, funders, and institutions) which are in
turn described by identifiers from other infrastructures (Crossref,
FundRef, ISNI). The need to identify objects is something that we have
recognised as common across the research enterprise. And the common
infrastructures that have been built are amongst the most powerful that
we have at our disposal. But yet most of us donâ€™t even notice that we
are using them.

Infrastructures for *identification, storage, metadata* and
*relationships* enable scholarship. We need to extend the base platform
of identifiers into those new spaces, beyond identification to include
storage and references. If we can harness the benefits on the same scale
that have arisen from the provision of identifiers like Crossref DOIs
then the building of new services that are specific to given disciplines
will become so much easier. Â In particular, we need to address the gap
in providing a way to describe *relationships* between objects and
resources in general. This base layer may be â€œboringâ€? and it may be
invisible to the view of most researchers. But thatâ€™s the way it
should be. *Thatâ€™s* what makes it infrastructure.

It isnâ€™t what is immediately visible on the surface that makes a
leopard a leopard, otherwise the black leopard wouldnâ€™t be, it is what
is buried beneath.

\[pdf-lite\]

Black leopard from Out of Africa Wildlife Park in Arizona (Wikipedia)

 Cite as: What exactly is infrastructure? Seeing the leopard’s spots. Geoffrey Bilder, Jennifer Lin, Cameron Neylon. fig
 
 share
 
 . http://dx.doi.org/10.6084/m9.figshare.1520432

We ducked a fundamental question raised by our proposal for infrastructure principles:

 â€œwhat exactly counts as infrastructure?â€?

This question matters.

What exactly is infrastructure? Seeing the leopard’s spots

Cameron Neylon

ChemSpidey lives! Even in the face of [Karen James' heavy
irony](http://twitter.com/kejames/status/3474150785) I am still amazed
that someone like me with very little programming experience was able to
pull together something that actually worked effectively in a live demo.
As long as you're not actively scared of trying to put things together
it is becoming relatively straightforward to build tools that do useful
things. Building ChemSpidey relied heavily on existing services and
other people's code but pulling that together was a relatively
straightforward process. The biggest problems were fixing the strange
and in most cases undocumented behaviour of some of the pieces I used.
So what is ChemSpidey?

ChemSpidey is a [Wave
robot](http://code.google.com/apis/wave/extensions/robots/index.html)
that can be found at chemspidey@appspot.com. The [code
repository](http://github.com/cameronneylon/ChemSpidey) is available at
[Github](http://github.com) and you should feel free to re-use it in
anyway you see fit, although I wouldn't really recommend it at the
moment, it isn't exactly the highest quality. One of the first
applications I see for Wave is to make it easy to author (semi-)semantic
documents which link objects within the document to records on the web.
In chemistry it would be helpful to link the names of compounds through
to records about those compounds on the relevant databases.

If ChemSpidey is added to a wave it watches for text of the form
"chem\[ChemicalName{;weight {m}g}\]" where the curly bracketed parts are
optional. When a blip is submitted by hitting the "done" button
ChemSpidey searches through the blip looking for this text and if it
finds it, strips out the name and sends it) to the [ChemSpider
SimpleSearch
service](http://www.chemspider.com/Search.asmx?op=SimpleSearch).
ChemSpider returns a list of database ids and the robot currently just
pulls the top one off the list and adds the text ChemicalName
(csid:####) to the wave, where the id is linked back to ChemSpider. If
there is a weight present it asks the [ChemSpider MassSpec
API](http://www.chemspider.com/MassSpecAPI.asmx?op=GetExtendedCompoundInfo)
for the nominal molecular weight calculates the number of moles and
inserts that. You can see [video of it working
here](http://www.viddler.com/explore/CameronNeylon/videos/8/) (look
along the timeline for the ChemSpidey tag).

![ChemSpidey](http://farm4.static.flickr.com/3453/3860977361_88220c2555_o.jpg){decoding="async"
loading="lazy" width="570" height="180"}

What have I learned? Well some stuff that is probably obvious to anyone
who is a proper developer. Use the current version of the API. Google
AppEngine pushes strings around as unicode which broke my tests because
I had developed things using standard Python strings. But I think it
might be useful to start drawing some more general lessons about how
best to design robots for research, so to kick of the discussion here
are my thoughts, many of which came out of discussions with Ian Mulvany
as we prepared for last weeks demo.

1.  **Always add a Welcome Blip when the Robot is added to a wave**.
    This makes the user confident that something has happened. Lets you
    notify users if a new version has been released, which might change
    the way the robot works, and lets you provide some short
    instructions.It's good to include a version number here as well.
2.  **Have some help available.** Ian's [Janey
    robot](http://github.com/IanMulvany/janey-robot) responds to the
    request (janey help) in a blip with an extended help blip explaining
    context. Blips are easily deleted later if the user wants to get rid
    of them and putting these in separate blips keeps them out of the
    main document.
3.  **Where you modify text leave an annotation.** I've only just
    started to play with annotations but it seems immensely useful to at
    least attempt to leave a trace here of what you've done that makes
    it easy for either your own Robot or others, or just human users to
    see who did what. I would suggest leaving annotations that identfy
    the robot, include any text that was parsed, and ideally provide
    some domain information. We need to discuss how to setup some name
    spaces for this.
4.  **Try to isolate the "science handling" from the "wave handling".**
    ChemSpidey mixes up a lot of things into one Python script. Looking
    back at it now it makes much more sense to isolate the interaction
    with the wave from the routines that parse text, or do mole
    calculations. This means both that the different levels of code
    should become easier for others to re-use and also if Wave doesn't
    turn out to be the one system to rule them all that we can re-use
    the code. I am no architecture expert and it would be good to get
    some clues from some good ones about how best to separate things
    out.

These are just some initial thoughts from a very novice Python
programmer. My code satisfies essentially none of these suggestions but
I will make a concerted attempt to improve on that. What I really want
to do is kick the conversation on from where we are at the moment, which
is basically playing around, into how we design an architecture that
allows rapid development of useful and powerful functionality.


ChemSpidey lives! Even in the face of Karen James’ heavy irony I am still amazed that someone like me with very little programming experience was able to pull together something that actually worked effectively in a live demo. As long as you’re not actively scared of trying to put things together it is becoming relatively straightforward to build tools that do useful things.

Writing a Wave Robot – Some thoughts on good practice for Research Robots

Interesting conversation yesterday on [Twitter](http://twitter.com) with
[Evgeniy Meyke](http://twitter.com/emeyke) of
[EarthCape](http://www.earthcape.com/) prompted in part by my [last
post](http://blog.openwetware.org/scienceintheopen/2009/08/11/the-trouble-with-business-models-facebook-buys-friendfeed/).
We started talking about what a [Friendfeed](http://friendfeed.com)
replacement might look like and how it might integrate more directly
into scientific data. Is it possible to build something general or will
it always need to be domain specific. Might this in fact be an
advantage? Evgeniy
[asked](http://twitter.com/emeyke/statuses/3243368657):

> @CameronNeylon do you think that "something new" could be more
> vertically oriented rather then for "research community" in general?

His thinking being, as I understand it that to get at domain specific
underlying data is always likely to take local knowledge. As he said in
his [next tweet](http://twitter.com/emeyke/statuses/3243639868):

> @CameronNeylon It might be that the broader the coverage the shallower
> is integration with underlining research data, unless api is good

This lead me to thinking about integration layers between data and
people and recalled something that I said in jest to someone some time
ago;

> "If you're using a human as your API then you need to work on your
> user interface."

Thinking about the way Friendfeed works there is a real sense in which
the system talks to a wide range of automated APIs but at the core there
is a human layer that firstly selects feeds of interest and then when
presented with other feeds selects from them specific items. What
Friendfeed does very well in some senses is provide a flexible API
between feeds and the human brain. But Evegeniy [made the
point](http://twitter.com/emeyke/statuses/3243726876) that this "works
only 4 'discussion based' collaboration (as in FF), not 4 e.g. collab.
taxonomic research that needs specific data inegration with taxonomic
databases".

Following from this was an [interesting
conversation](http://search.twitter.com/search?q=emeyke+cameronneylon)
\[[Webcite Archived Version](http://www.webcitation.org/5j06npMEY)\]
about how we might best integrate the "human API" for some imaginary
"Science Stream" with domain specific machine APIs that work at the data
level. In a sense this is the core problem of scientific informatics.
How do you optimise the ability of machines to abstract and use data and
meaning while at the same time fully exploiting the ability of the human
scientist to contribute their own unique skills, pattern recognition,
insight, lateral thinking. And how do you keep these in step with each
other so both are optimally utilised? Thinking in computational terms
about the human as a layer in the system with its own APIs could be a
useful way to design systems.

Friendfeed in this view is a peer to peer system for pushing curated and
annotated data streams. It mediates interactions with the underlying
stream but also with other known and unknown users. Friendfeed seems to
get three things very right: 1) Optimising the interaction with the
incoming data stream; 2) Facilitating the curation and republication of
data into a new stream for consumption by others, creating a virtuous
feedback look in fact; and 3) Facilitating discovery of new peers.
Friendfeed is actually a bittorrent for sharing conversational objects.

This conversational layer, a research discourse layer if you like, is at
the very top of the stack, keeping the humans to a high level abstracted
level of conversation, where we are probably still at our best. And my
guess is that something rather like Friendfeed is pretty good at being
the next layer down, the API to feeds of interesting items.Â  But
Evgeniy's question was more about the bottom of the stack, where the
data is being generated and needs to be turned into a useful and
meaningful feed, ready to be consumed. The devil is always in the
details and vertical integration is likely to help her. So what do these
vertical segments look like?

In some domains these might be lab notebooks, in some they might be
specific databases, or they might be a mixture of both and of other
things. At the coal face it is likely to be difficult to find a way of
describing the detail in a way that is both generic enough to be
comprehensible and detailed enough to be useful. The needs of the data
generator are likely to be very different to those of a generic data
consumer. But if there is a curation layer, perhaps human or machine
mediated, that partly abstracts this then we may be on the way to
generating the generic feeds that will be finally consumed at the top
layer.Â  This curation layer would enable semantic markup, ideally
automatically, would require domain specific tooling to translate from
the specific to the generic, and provide a publishing mechanism. In
short it sounds (again) quite a bit like [Wave](http://wave.google.com).
Actually it might just as easily be
[Chem4Word](http://research.microsoft.com/en-us/projects/chem4word/default.aspx)
or any other domain specific semantic authoring tool, or just a
translation engine that takes in detailed domain specific info and
correlates it with a wider vocabulary.

One of the things that appeals to me about Wave, and Chem4Word, is that
they can (or at least have the potential to) hide the complexities of
the semantics within a straightforward and comprehensible authoring
environment. Wave can be integrated into domain specific systems via
purpose built Robots making it highly extensible. Both are capable of
"speaking web" and generating feeds that can be consumed and processed
in other places and by other services. At the bottom layer we can chew
the problem off one piece at a tim, including human processing where it
is appropriate and avoiding it where we can.

The middleware is of coures, as always, the problem. The middleware is
agreed and standardised vocabularies and data formats. While in the past
I have thought this near intractable actually it seems as though many of
the pieces are actually falling into place. There is still a great need
for standardisation and perhaps a need for more meta-standards but it
seems like a lot of this is in fact on the way. I'm still not convinced
that we have a useful vocabulary for actually describing experiments but
enough smart people disagree with me that I'm going to shut up on that
one until I've found the time to have a closer look at the various
things out there in more detail.

These are half baked thoughts -- but I think the idea of where we
optimally place the human in the system is a useful question. It also
hasn't escaped my notice that I'm talking about something very similar
to the architecture that [Simon Coles](http://elnblog.com/) of [Amphora
Research Systems](http://www.amphora-research.com/) always puts up in
his [presentations on Electronic Lab
Notebook](http://www.amphora-research.com/resources/index.html)[s](http://www.amphora-research.com/resources/index.html).
Fundamentally because the same core drivers are there.


Interesting conversation yesterday on Twitter with Evgeniy Meyke of EarthCape prompted in part by my last post. We started talking about what a Friendfeed replacement might look like and how it might integrate more directly into scientific data. Is it possible to build something general or will it always need to be domain specific. Might this in fact be an advantage?

Messaggi di Rogue Scholar

Language Models: Deep Dive into BERT

What exactly is infrastructure? Seeing the leopard’s spots

Writing a Wave Robot – Some thoughts on good practice for Research Robots

Where is the best place in the Research Stack for the human API?