DataCite

Kristian Garza

![Converting DOI metadata between metadata schemas using ChatGPT](https://cdn-images-1.medium.com/max/1024/0*YZJlee9xdhaDG3zD.png)

Large language models have been making waves in artificial intelligence for a good reason. These models have the ability to understand and generate human language, which has far-reaching implications for the future of technology. One area where these models have a significant impact is in the realm of software development. Specifically, with [Github Copilot](https://github.com/features/copilot) and [ChatGPT](https://openai.com/blog/chatgpt/), large language models might possibly transform how we write and understand code. In this post, I'd like to discuss the potential impact that language models could have on metadata mapping.

On the one hand, it is easy to see how large language models could greatly simplify the process of writing and understanding code. Currently, software developers must spend a significant amount of time learning the intricacies of different programming languages, and even then, they often struggle to understand code written by others. With large language models, like those in Github Copilot, developers can simply describe their desired program in plain language, and the model would convert that description into working code. These benefits would save developers time and make it easier for non-experts to understand and modify code.

Another potential use for large language models is their ability to transform data from one schema to another without the need for manual coding. Metadata transformation AI-powered tools could be especially useful for organizations that need to integrate data from multiple sources, such as from different institutions, metadata models, or from external partners.

Currently, the metadata transformation process can be tedious and error-prone, as developers must write code to extract, transform, and load the data. In the case of **DOI metadata**, there exist libraries and services that provide such features, such as [Bolognese](https://github.com/datacite/bolognese) and [Crosscite Content Negotiation](https://support.datacite.org/docs/datacite-content-resolver0). However, even these tools require constant updating when new versions of metadata schemas are released. The coding is time-consuming, and the generation of mappings between multiple schemas also takes significant effort; the [Research Metadata Schemas Working Group](https://www.rd-alliance.org/groups/research-metadata-schemas-wg) of the **Research Data Alliance** dedicates much time to generating such mappings and crosswalks.

With large language models, however, this process could be significantly simplified. Developers could simply describe the desired schema, and the model automatically generates the necessary code to transform the data. Using LLMs would not only save time but also reduce the risk of errors, which could lead to data loss or corruption.

In my experiments, I have been using a large language model and a Chain-of-Thought (CoT) approach to generate software for creating schema mappings. The results have been interesting. To demonstrate this, we can look at a few experiments.

-   The first experiment shows the capabilities of ChatGPT to generate code for schema transformation.
-   The second experiment shows how the GPT-3.5 model can be used to convert data from one schema to another, removing the need to write code completely.

### Generating Code for schema transformation

To generate a prompt for ChatGPT and include a set of constraints. These were:

Prompt:

> Act as a Senior kotlin Developer. I will give you a set of Project and functional Requirements and you will write the code to accomplish that project using the kotlin language.

> Project Requirements:

> \- Write the code for a metadata converter.

> \- Use schema.org as exchange format.

> \- Use a facade design pattern to create the converter.

> Functional requirements:\
> Given that program receives a XML file that contains metadata in DataCite Schema. Then the program should take the file and map it to the appropriate Crossref type. And The program should output the result of the mapping into a XML file. Let's think step by step.

**The results (below) are not perfect, but they are promising and extremely helpful.** ChatGPT was able to create the needed data model and main functions for the facade pattern but omitted the parsing logic for each function.

Following a Chain-of-Thought approach, I used the results of the first prompt to generate the code for the parsing and serializing logic. Furthermore, I asked ChaGPT to include all mandatory and recommended fields of the schemas. The results have been impressive; ChatGPT was able to generate the logic and the metadata model without inputting any mapping. All the files generated can be found in [GitHub](https://gist.github.com/kjgarza/2b1be71e3d2a2a4e700b546af794960b)

While it is clear that large language models and the Chain-of-Thought approach have the potential to reduce the manual labor involved in metadata mapping significantly, it is evident that they will not replace software developers completely. Developers are still needed to architect the software and ensure that it is designed in a way that makes the best use of these language models.

### Forget about Code; just transfrom the data

![Metadata conversion side-by-side comparison. Left: Original Crossref Metadata. Middle: ChatGPT converted Metadata. Right: Crosscite CN converted Metadata [*https://gist.github.com/kjgarza/343e8c44b3ed018eaa9d216b479145f9*](https://gist.github.com/kjgarza/343e8c44b3ed018eaa9d216b479145f9)](https://cdn-images-1.medium.com/max/1024/0*2mLeofIeHYz2Dn93.png)

The second experiment is different; why bother transforming the code when the GPT-3.5 model already knows about Datacite Schema and Crossref Schema? I can ask the GPT-3.5 model, via ChatGPT, to covert data from one schema to another. The example below shows that. I took the metadata of a DOI in Crossref schema (i.e., [10.1002/EJIC.201900339](https://doi.org/10.1002/EJIC.201900339)) and used it as a prompt to convert to DataCite Schema. I had to trim the metadata a bit as it has too many relationships, but it is good enough to check the conversation. Then compared, the results with the metadata conversion using Crosscite Content Negotiation (a service maintained by Crossref and DataCite). Finally, I validated the resulting metadata against the DataCite API validation.

![Sucessulful DOI metadata Validation from [*https://gist.github.com/kjgarza/343e8c44b3ed018eaa9d216b479145f9*](https://gist.github.com/kjgarza/343e8c44b3ed018eaa9d216b479145f9)](https://cdn-images-1.medium.com/max/1024/0*kh1qx8Jqj7bQL5MV.png)

Overall the results are excellent. First, the conversion into valid DataCite metadata indicates that it understands the DataCite Metadata. Second, in terms of the comparison, the results are very positive but not perfect. Mandatory fields mapped corrected. ChatGPT also made some interpretations to enrich the metadata with "subjects" and a "description." Optional fields like dates and funding had mixed results, but they are not inaccurate. The metadata [file outputs can be found on GitHub](https://gist.github.com/kjgarza/343e8c44b3ed018eaa9d216b479145f9).

The results from both experiments are promising. Both examples show that the LLM model is aware of the different schemas, their properties, formats, software patterns, languages, and syntax. **Basically, we are** [**outsourcing domain knowledge**](https://www.youtube.com/watch?v=Qt3eb33296A&ab_channel=BriinkIntelligenceGmbH) **to the LLM.**

Nevertheless, it is important to consider the potential for bias in large language models. Language is inherently subjective, and if models are trained on biased data, they may perpetuate that bias in the code or mappings they generate. For example, what happens when the model chooses to translate to an older version of the schema instead of the newest one? Such biases could lead to unintended consequences and discrimination in the programs and systems we build.

In short, **I believe that the use of large language models can potentially significantly transform the way we map metadata schema models**. By automating the process and reducing the need for manual labor, we can improve the efficiency and accuracy of metadata mapping. As language models continue to improve, I would not be surprised if we see even more drastic transformations in the future.

So, there you have it. The potential of large language models to transform metadata mapping is enormous. It is an exciting time to work in this field, and I cannot wait to see what the future holds.

*Originally published at* [*https://kjgarza.github.io*](https://kjgarza.github.io/pages/2023/01/16/coversion_software.html)*.*

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8df54e41e24){width="1" height="1"}


Large language models have been making waves in artificial intelligence for a good reason. These models have the ability to understand and generate human language, which has far-reaching implications for the future of technology. One area where these models have a significant impact is in the realm of software development.

Revolutionizing Metadata Schema Mapping with ChatGPT

### Academic Publishing web forms meet your demise: The unstoppable rise of large language models (ChatGPT)

![Prompt to create DOI metadata using ChatGPT.](https://cdn-images-1.medium.com/max/1024/0*VddPt0uQe4DvX791.png)

As we enter the age of artificial intelligence, it's worth considering how large language models will revolutionize how we interact with websites and applications. Web forms have been the dominant method for users to input data and complete tasks online for decades. But as anyone who has struggled to fill out a form on the web can attest, these forms can be clunky and frustrating. Large language models offer a more intuitive and human-like alternative. Imagine being able to speak or type your request to a website or application, rather than having to remember which information goes in which field and navigate through multiple pages of a form. This level of convenience doesn't exist, but chatGPT shows signs of the possibilities.

An area of application in which I'm genuinely interested is **metadata input in academic publications and data**. The replacement of web forms with large language models for metadata input in academic journals has the potential to streamline the submission process significantly. Traditionally web forms can be time-consuming and laborious to complete, especially for academics who may have to input large amounts of information about their research and publication history. With large language models, academics could type prompts about their metadata into a web application, saving significant time and effort. This could also lead to fewer errors in the submission process, as large language models can understand the context and provide more accurate input. Adopting large language models for metadata input in academic publications could lead to a more efficient and user-friendly submission process.

In the past, one potential criticism of this approach is that LLM may only sometimes be able to understand and respond to user requests accurately. While large language models have made significant progress in recent years, they can still struggle with ambiguous or complex language, particularly in domain-specific contexts such as academic publications. This could lead to confusion or frustration for users who may not get the desired results from their natural language queries. However, [OpenAI announcement](https://openai.com/blog/chatgpt/) of the GPT-3.5 series model and its flagship app chatGPT has potentially shattered that criticism. Since the announcement, the web has been floated with examples of prompts generating code, creating copy, and answering questions.

ChatGPT has shown to excel on the tasks above, but how would it fare creating metadata for academic publications? This metadata is often made with large web forms. The content submitted in these web forms is often serialized and put in a payload submitted in APIs. The payloads follow a particular schema. In the case of academic publications metadata, this is often DOI metadata. These metadata payloads follow a schema that lives on the web; therefore is likely that chatGPT knows about it and, when prompted, can generate valid metadata.

I did a small experiment to test that assumption, and the results are pretty interesting. I asked chatGPT to generate the metadata for a dataset with a specific title. The title is actually from a real resource on the web (i.e., [10.6084/m9.figshare.7588511](https://doi.org/10.6084/m9.figshare.7588511)), but the rest of the metadata requested is all fictional. See the example below:

> **User:** Create the metadata, using datacite schema and xml format, of a dataset on the titled "Predicting Outcomes from First Session Interactions in Remote Cognitive Behavioural Therapy for Severe Health Anxiety" under a cc-by license and authored by barry blitt from NYU

> **ChatGPT:** Here is an example of metadata for the dataset you described, using the DataCite schema and XML format:

    <resource xmlns="http://datacite.org/schema/kernel-4" 
              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
              xsi:schemaLocation="http://datacite.org/schema/kernel-4 http://schema.datacite.org/meta/kernel-4/metadata.xsd">
      <identifier identifierType="DOI">10.5072/example-dataset</identifier>
      <creators>
        <creator>
          <creatorName>Barry Blitt</creatorName>
          <affiliation>NYU</affiliation>
        </creator>
      </creators>
      <titles>
        <title>Predicting Outcomes from First Session Interactions in Remote Cognitive Behavioural Therapy for Severe Health Anxiety</title>
      </titles>
      <publisher>NYU</publisher>
      <publicationYear>2022</publicationYear>
      <subjects>
        <subject>Cognitive Behavioural Therapy</subject>
        <subject>Severe Health Anxiety</subject>
        <subject>Remote Therapy</subject>
      </subjects>
      <rightsList>
        <rights rightsURI="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International</rights>
      </rightsList>
      <descriptions>
        <description descriptionType="Abstract">
          This dataset contains data collected from first session interactions in remote cognitive behavioural therapy for severe health anxiety. It includes information on patient demographics, therapy session details, and prediction outcomes. The data is intended to be used for research purposes.
        </description>
      </descriptions>
    </resource>

Surprisingly the metadata generated is valid Datacite Metadata, which includes all the metadata supplied. The language model also generated a description and subject/keywords to classify the dataset. These subjects, when compared with the real resource on the web (see [metadata](https://api.datacite.org/application/vnd.datacite.datacite+xml/10.6084/m9.figshare.7588511)), are not precisely the same; nevertheless, the generated subject fall within the knowledge domain of the web resource.

Over the last couple of weeks, I have run similar experiments, all giving similar accuracy. The results are a far cry from traditional web forms, which often require selecting specific inputs to use specific fields. Creating the same output in the usual web form would have required a user to enter data in eleven input elements. Using dialogue via chatGPT presents a superior user experience/convenience and decision support as opposed to filling out a time-consuming form. But the benefits of large language models go beyond convenience. As seen in the example, these models can also understand the context and provide richer responses.

As large language models continue to improve, they will likely replace traditional web forms for many tasks. This change could lead to a more seamless and enjoyable user experience online. So the next time you're confronted with a tedious web form, imagine the potential for a more significant and effortless way to interact with the digital world.

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dbb1bd6c2ee8){width="1" height="1"}


Academic Publishing web forms meet your demise: The unstoppable rise of large language models (ChatGPT)     As we enter the age of artificial intelligence, it’s worth considering how large language models will revolutionize how we interact with websites and applications. Web forms have been the dominant method for users to input data and complete tasks online for decades.

Academic Publishing web forms meet your demise: The unstoppable rise of large language models…

Adam Day

There's a quote attributed to Ernest Rutherford: "That which is not
measurable is not science. That which is not physics is stamp
collecting". I think his point was that a lot of scientific work is just
*documenting* things. In Rutherford's day, there was a lot of exciting
new *creative* work happening in physics, so perhaps physics seemed
special to him. On the other hand, it was a mean-spirited thing to say
about other fields of science and I'm sure he felt very silly when he
was handed the Nobel prize in chemistry.

Generative AI is all the rage right now isn't it? Essentially, we can
now use AI models to do *creative* tasks, like painting a picture.

Do you remember that famous series of paintings that Claud Monet did
called "computer nerds hard at work"?

<figure>
<img
src="https://cdn-images-1.medium.com/max/512/1*ag7wUHny722vK6beW0UdfQ.png" />
</figure>

<figure>
<img
src="https://cdn-images-1.medium.com/max/512/1*GVoYZVm-u9uQuxJvjhWKIg.png" />
</figure>

<figure>
<img
src="https://cdn-images-1.medium.com/max/512/1*U9V_-9NXaDl0O3CbDU0dwQ.png" />
</figure>

<figure>
<img
src="https://cdn-images-1.medium.com/max/512/1*T5AMW56GTO2GSe6eW5I1VA.png" />
</figure>

No? Wikipedia says that Monet passed away in 1926. That means he was
dead at the time these computers were invented and can't possibly have
painted them, but they certainly look like Monet's work, don't they? If
I had wanted to fake images like this 12 months ago, I would have needed
canvas, oils and a lot of talent with a brush. Today, I just prompted
Stable Diffusion to do it for me. No coding skills required. It
took seconds.

Speaking of coding, I was looking for a very specific piece of code
recently. After half an hour searching on Google and StackOverflow I was
ready to give up searching and write it myself, but then I asked ChatGPT
to do it for me. ChatGPT wrote exactly what I needed. Again, it
took seconds.

ChatGPT is a chatbot based on a *Large Language Model* called GPT-3. You
can ask ChatGPT to write just about anything. Poems, stories, homework
solutions, research papers... anything!

The value of new AI models for **creative** tasks like this is immense.
Images and text and even video can be generated rapidly with
increasingly authentic quality standards.

I put it to you that when you write a research paper, that *isn't*
creative work. I mean, it obviously is to an extent, but the purpose of
a research paper is to document something. It's the stamp collecting
part of science and that's an important distinction.

If I ask ChatGPT for text for an original research paper, it will give
it to me. But the text won't be based on any real novel piece of
research, it will be made up and it will be either derivative, false, or
both. It will look real and [a recent study by Catherine Gao at
Northwestern University in Chicago,
Illinois](https://www.nature.com/articles/d41586-023-00056-7) showed
that scientists were often unable to distinguish real scientific
abstracts from those written by ChatGPT.

By the way, if I prompt Stable Diffusion for fake scientific images,
it'll cough those up too.

So, AI can already produce believable scientific fakes that are hard to
detect. Worse, the people building these AI applications might not even
be aware of the extent and dangers of [organised
research fraud](https://medium.com/@publisherad/how-papermills-work-generally-c080bb9fff43).

Through my company, Clear Skies, I've already been working on a couple
of methods that should be able to detect fraudulent ChatGPT-generated
content. But I'm in 2 minds as to whether it's a good idea to develop
this capability because there's no good reason that AI should be
producing this kind of content in the first place.

- AI shouldn't be trained to generate documentary material at all.
- APIs for these tools should also classify prompts and output to
  recognise when they are being abused for fraudulent or harmful
  purposes (and then decline to give output to a user).

I hope that doesn't sound like wishful thinking. Controls like these
actually do exist on both ChatGPT and Stable Diffusion. So I am
optimistic that, as the people building these applications receive
feedback, they make adaptations like those above to avoid creating
harmful outputs.

In the meantime, we are already seeing policies on artificially
generated content. StackOverflow [has temporarily banned AI-generated
content until such time as it can determine suitable guidelines
for use](https://meta.stackoverflow.com/questions/421831/temporary-policy-chatgpt-is-banned).

The future of this problem for Schol-Comms is complex. There is a lot to
consider:

- Should publishers ban AI-generated content like StackOverflow
  has done?
- Or should guidelines be more nuanced --- like should AI-generated
  content be treated like quotations where the non-original content in a
  paper is clearly set apart from the rest of the text? What about AI
  autocomplete? Should that be banned or quoted, or is that different?
- Do the copyright licenses on scientific papers allow them to be used
  for training generative AI? (And what should those licenses allow?
  It's not like they were written with this use case in mind.)

It's a tough one and I'm sure it'll take time to figure out the best way
to handle this. What is clear, though, is that there is plenty
to discuss!

There are a couple of addenda worth making here. There are other AI
use-cases that can be trained on research papers. E.g.
question-answering. That's where we can give an open-ended question like
"what's the best treatment for COVID-19" and then an AI model can find
all the best papers on that topic and generate a response to the
question based on those. That's a brilliant thing to do and so there's
definitely a good case for training models for some tasks.

It's also worth mentioning Meta's recently released Galactica model.
Galactica was supposed to enable tasks like those described above.
However: there was clear potential to use the model for research fraud
and it had a few other issues (e.g. it produced nonsense from time to
time --- if we want scientific answers, that's not very helpful). The
dangers were clear and [I was relieved to see Meta kill Galactica just
days after its
release](https://www.technologyreview.com/2022/11/18/1063487/meta-large-language-model-ai-only-survived-three-days-gpt-3-science/).
There's still potential for a project like that, but it's great to see
decisive action from Meta to address the risk.

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=72ab43bd217){width="1"
height="1"}


There’s a quote attributed to Ernest Rutherford: “That which is not measurable is not science. That which is not physics is stamp collecting”. I think his point was that a lot of scientific work is just
<em>
 documenting
</em>
things. In Rutherford’s day, there was a lot of exciting new
<em>
 creative
</em>
work happening in physics, so perhaps physics seemed special to him.

Messaggi di Rogue Scholar

Revolutionizing Metadata Schema Mapping with ChatGPT

Academic Publishing web forms meet your demise: The unstoppable rise of large language models…

We need a chat about ChatGPT