DataCite

Kristian Garza

Academic Publishing web forms meet your demise: The unstoppable rise of large language models (ChatGPT)     As we enter the age of artificial intelligence, it’s worth considering how large language models will revolutionize how we interact with websites and applications. Web forms have been the dominant method for users to input data and complete tasks online for decades.

Academic Publishing web forms meet your demise: The unstoppable rise of large language models…

In the winding world of academic research, the quest for an appropriate repository to store and share scholarly data has been akin to a modern-day odyssey. Traditionally, this journey involved wading through static pages and search indexes, a process both time-consuming and fraught with uncertainty.

The Impact of Language User Interfaces on Finding Scholarly Repositories

Picture, if you will, the labyrinthine world of academic information management, where metadata schema mapping serves as a vital underpinning for the exchange and intermingling of data across diverse platforms and systems. This arena has long been dominated by the venerable metadata schema crosswalk, which, though serviceable, has begun to show its age.

ParrotGPT: On the Advantages of Large Language Models Tools for Academic Metadata Schema Mapping

Large language models have been making waves in artificial intelligence for a good reason. These models have the ability to understand and generate human language, which has far-reaching implications for the future of technology. One area where these models have a significant impact is in the realm of software development.

Revolutionizing Metadata Schema Mapping with ChatGPT

Stories by Kristian Garza on Medium

![Creating Crosswalk with chatGPT Plus. Image partially created with Dall-e](https://cdn-images-1.medium.com/max/1024/0*OycdbBJvXaXdL33P.png)

Picture, if you will, the labyrinthine world of academic information management, where metadata schema mapping serves as a vital underpinning for the exchange and intermingling of data across diverse platforms and systems. This arena has long been dominated by the venerable metadata schema crosswalk, which, though serviceable, has begun to show its age. The [traditional method of creating metadata schema crosswalks](https://medium.com/@kj.garza/revolutionizing-metadata-schema-mapping-with-chatgpt-a8df54e41e24) is a time-consuming and monotonous task that requires significant coordination. Enter the digital behemoths known as large language models (LLMs), whose unparalleled prowess in understanding and generating the complex tapestry of human language has opened up a new frontier in metadata schema mapping. I think LLMs present a far superior alternative to the traditional crosswalks of yore.

The LLMs have undergone significant improvements and can perform scholarly metadata mapping with minimal input. In a [previous blogpost](https://medium.com/@kj.garza/revolutionizing-metadata-schema-mapping-with-chatgpt-a8df54e41e24), I showed that with a short prompt, the LLM behind ChatGPT can convert metadata from Crossref schema to DataCite schema with great accuracy and even metadata enrichment. Among the multiple GPT-3 models available, the "Davinci" model stands out as the most powerful and efficient at this task. Compared with other GPT-3 models such as [ADA](https://platform.openai.com/docs/models/gpt-3) and [Curie](https://platform.openai.com/docs/models/gpt-3) it performs way better in terms of metadata transformation. "Davinci" has been trained on different datasets, which makes it capable of handling various metadata schema mapping. This level of precision is challenging to achieve with crosswalks, which require a considerable amount of manual input to create and maintain.

Furthermore, the [newly released GPT-4 model](https://openai.com/research/gpt-4) is more accurate and can perform tasks that previous GPT-3 family models will give up on performing. For example, while mapping metadata is a task that all GPT-family models can perform with different degrees of accuracy, generating a crosswalk of two schemas is a task that the GPT-3 family models will reject performing. GPT-4 models do perform that task with good results when compared to [expert crosswalks](https://zenodo.org/record/7661399#.ZBcMaOzMI-Q). Below you can see how the GPT-4 generates a crosswalk between DataCite Schema and schema.org (type ResearchProject) schema.

![Crosswalk output using GPT-4 from DataCite schema to schema.rog](https://cdn-images-1.medium.com/max/1024/0*6oC1iV8o6ZOlSD-K.png)

### Tools to interact with Large Language models

In an effort to harness the full potential of all the GPT models for metadata schema mapping, I have taken it upon myself to develop the Python package, [Parrot-GPT](https://github.com/kjgarza/parrot_gpt). This tool utilizes the formidable GPT-3 models to effortlessly translate metadata from one schema to another, all the while obviating the need for the manual creation and upkeep of crosswalks, thus bestowing upon its users the gift of time and resource savings. As of this publication, I am working on [incorporating](https://github.com/kjgarza/parrot_gpt/issues/1) the new [OpenAI Chat API syntax for GPT-3.5 and 4](https://platform.openai.com/docs/guides/chat/introduction). Contributions are [welcomed](https://github.com/kjgarza/parrot_gpt/blob/main/CONTRIBUTING.rst).

### Continuous Adaptation in the LLM's Realm

Consider the crosswalk, which, much like an aging, inflexible cartographer, finds itself unable to evolve with minimal intervention. In stark contrast, the LLM-based mapping is a veritable chameleon, gracefully adapting to new schema properties and maintaining accuracy in its ever-changing environment. Furthermore, Parrot-GPT can even generate metadata for new schema properties, making it a truly versatile tool for metadata management.

Parrot-GPT can also be used to generate metadata from new schema properties. In an experiment, I created metadata with new fields and translated metadata from one schema to another. The goal of this experiment is to demonstrate the ability of the LLM behind Parrot_GPT to correctly map metadata fields from one schema to another, even when new fields are introduced that were not present in the original schema.

To begin the experiment, a metadata file was created with a new field that did not exist in DataCite Schema, but an analogous field is present in the DCAT schema. Then Parrot_GPT was then asked to convert the metadata file from DataCite Schema to DCAT schema, with no prior knowledge of the mapping between the two schemas. For the new field, I chose "distribution", a field that currently does not exist in DataCite schema 4.4, but it will be included soon in [version 4.5](https://docs.google.com/document/d/1UyQQwtjnu-4_4zXE4TFZ74-mjLZI3NkEf8RrF0WeOdI). I took the example from the examples that [DataCite is preparing for the new schema release](https://github.com/datacite/schema/blob/schema-4.5/source/meta/kernel-4.5/example/datacite-distribution1-v4.xml)

Parrot_GPT can be run directly from the CLI specifying the file to transform {-mf}, the input schema {-i}, and the output schema {-t}. I transformed this DataCite Schema 4.5 metadata file in the example below to the DCAT schema. Input and output files can be found in [github](https://gist.github.com/kjgarza/6bbd6b0653d7ade6c7a67c954d926fed).

    python -m parrot_gpt.cli -mf tests/example-distribution.xml -i datacite_xml -t DCAT > output.rdf

<https://medium.com/media/0aba7c99e765a6d471f6ab9ca18d8f73/href>

The results were promising, demonstrating that LLM mapping can handle new schema properties with minimal intervention. The new field "distribution" was correctly mapped to the [DCAT class distribution](https://w3c.github.io/dxwg/dcat/#Class:Distribution); all sub-properties were also correctly mapped. Interestingly the LLM chose to map DataCite contentUrl property to the DCAT accessUrl property, which one could argue is not the most suitable candidate. A few more interactions of the same experiment show that the model would choose the downloadUrl property on certain occasions instead. This raises a question about the model variable decision-making. Nevertheless, it is feasible to fine-tune this kind of issue with [model fine-tuning](https://platform.openai.com/docs/guides/fine-tuning). Overall this capability makes LLM mapping more flexible and adaptable than crosswalks.

I think the experiment supports the effectiveness of using LLMs for metadata schema mapping, as they can handle new and unexpected metadata fields with minimal additional input or intervention. We need to explore whether further fine-tuning can improve this general model. These capabilities make LLM mapping a highly scalable and efficient solution for managing metadata schemas in academic institutions and other organizations that deal with large volumes of data.

### The diminishment of the LLMs drawbacks

Despite its advantages, using LLMs for metadata schema mapping has some drawbacks. One such drawback is the limited accuracy of LLMs when dealing with complex and nuanced metadata schemas. Human intervention may still be necessary to ensure accuracy, as we saw in the experiments. On the other hand, creating metadata schema crosswalks can be a tedious and manual process that requires continuous updates to ensure accuracy. Additionally, crosswalks may not be as scalable as LLMs in handling the proliferation of new schemas and their fast-pacing evolution.

![Source: [**https://xkcd.com/927/**](https://xkcd.com/927)](https://cdn-images-1.medium.com/max/500/0*lbwuhbeAclcpnxw6.png)

Large language models, particularly state-of-the-art models like "GPT-3", require substantial computational resources to run. This may present a barrier for smaller institutions or those with limited budgets. In contrast, traditional crosswalks can be created and maintained with significantly less computational power. However, recent developments have made LLMs-based services truly affordable. While writing this article, Microsoft Azure just announced [Azure OpenAI Service](https://azure.microsoft.com/en-us/pricing/details/cognitive-services/openai-service/) providing REST API access to OpenAI's powerful language models, including the GPT-3, Codex, and Embeddings model series, with very affordable prices. This very month, the [inference of Facebook's LLaMA model](https://news.ycombinator.com/item?id=35100086) was published in pure C/C++, allowing anybody to run a model of similar capabilities to GPT-3 directly from a MacBook.

In conclusion, allow me to reiterate that the utilization of large language models for mapping academic metadata schemas represents a transcendent method when compared to the creation of metadata schema crosswalks. Through the development of the [Parrot-GPT Python package](https://parrot-gpt.readthedocs.io), we have unlocked seamless schema mapping, continuous adaptation to new schema properties, and the generation of metadata for novel fields. The results of this experiment serve as a testimony to the myriad advantages of using LLMs for metadata schema mapping, thus making a compelling case for their adoption in the hallowed halls of academic information management systems.

The dawn of this new metadata mapping era, fueled by large language models, sparks immense enthusiasm. As we explore these innovative technologies more deeply, I'm eager for a future where academic information management is more connected and streamlined.

*Originally published at* [*https://kjgarza.github.io*](https://kjgarza.github.io/pages/2023/03/13/parrot-gpt.html)*.*

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=434cceabc68b){width="1" height="1"}


![Converting DOI metadata between metadata schemas using ChatGPT](https://cdn-images-1.medium.com/max/1024/0*YZJlee9xdhaDG3zD.png)

Large language models have been making waves in artificial intelligence for a good reason. These models have the ability to understand and generate human language, which has far-reaching implications for the future of technology. One area where these models have a significant impact is in the realm of software development. Specifically, with [Github Copilot](https://github.com/features/copilot) and [ChatGPT](https://openai.com/blog/chatgpt/), large language models might possibly transform how we write and understand code. In this post, I'd like to discuss the potential impact that language models could have on metadata mapping.

On the one hand, it is easy to see how large language models could greatly simplify the process of writing and understanding code. Currently, software developers must spend a significant amount of time learning the intricacies of different programming languages, and even then, they often struggle to understand code written by others. With large language models, like those in Github Copilot, developers can simply describe their desired program in plain language, and the model would convert that description into working code. These benefits would save developers time and make it easier for non-experts to understand and modify code.

Another potential use for large language models is their ability to transform data from one schema to another without the need for manual coding. Metadata transformation AI-powered tools could be especially useful for organizations that need to integrate data from multiple sources, such as from different institutions, metadata models, or from external partners.

Currently, the metadata transformation process can be tedious and error-prone, as developers must write code to extract, transform, and load the data. In the case of **DOI metadata**, there exist libraries and services that provide such features, such as [Bolognese](https://github.com/datacite/bolognese) and [Crosscite Content Negotiation](https://support.datacite.org/docs/datacite-content-resolver0). However, even these tools require constant updating when new versions of metadata schemas are released. The coding is time-consuming, and the generation of mappings between multiple schemas also takes significant effort; the [Research Metadata Schemas Working Group](https://www.rd-alliance.org/groups/research-metadata-schemas-wg) of the **Research Data Alliance** dedicates much time to generating such mappings and crosswalks.

With large language models, however, this process could be significantly simplified. Developers could simply describe the desired schema, and the model automatically generates the necessary code to transform the data. Using LLMs would not only save time but also reduce the risk of errors, which could lead to data loss or corruption.

In my experiments, I have been using a large language model and a Chain-of-Thought (CoT) approach to generate software for creating schema mappings. The results have been interesting. To demonstrate this, we can look at a few experiments.

-   The first experiment shows the capabilities of ChatGPT to generate code for schema transformation.
-   The second experiment shows how the GPT-3.5 model can be used to convert data from one schema to another, removing the need to write code completely.

### Generating Code for schema transformation

To generate a prompt for ChatGPT and include a set of constraints. These were:

Prompt:

> Act as a Senior kotlin Developer. I will give you a set of Project and functional Requirements and you will write the code to accomplish that project using the kotlin language.

> Project Requirements:

> \- Write the code for a metadata converter.

> \- Use schema.org as exchange format.

> \- Use a facade design pattern to create the converter.

> Functional requirements:\
> Given that program receives a XML file that contains metadata in DataCite Schema. Then the program should take the file and map it to the appropriate Crossref type. And The program should output the result of the mapping into a XML file. Let's think step by step.

**The results (below) are not perfect, but they are promising and extremely helpful.** ChatGPT was able to create the needed data model and main functions for the facade pattern but omitted the parsing logic for each function.

Following a Chain-of-Thought approach, I used the results of the first prompt to generate the code for the parsing and serializing logic. Furthermore, I asked ChaGPT to include all mandatory and recommended fields of the schemas. The results have been impressive; ChatGPT was able to generate the logic and the metadata model without inputting any mapping. All the files generated can be found in [GitHub](https://gist.github.com/kjgarza/2b1be71e3d2a2a4e700b546af794960b)

While it is clear that large language models and the Chain-of-Thought approach have the potential to reduce the manual labor involved in metadata mapping significantly, it is evident that they will not replace software developers completely. Developers are still needed to architect the software and ensure that it is designed in a way that makes the best use of these language models.

### Forget about Code; just transfrom the data

![Metadata conversion side-by-side comparison. Left: Original Crossref Metadata. Middle: ChatGPT converted Metadata. Right: Crosscite CN converted Metadata [*https://gist.github.com/kjgarza/343e8c44b3ed018eaa9d216b479145f9*](https://gist.github.com/kjgarza/343e8c44b3ed018eaa9d216b479145f9)](https://cdn-images-1.medium.com/max/1024/0*2mLeofIeHYz2Dn93.png)

The second experiment is different; why bother transforming the code when the GPT-3.5 model already knows about Datacite Schema and Crossref Schema? I can ask the GPT-3.5 model, via ChatGPT, to covert data from one schema to another. The example below shows that. I took the metadata of a DOI in Crossref schema (i.e., [10.1002/EJIC.201900339](https://doi.org/10.1002/EJIC.201900339)) and used it as a prompt to convert to DataCite Schema. I had to trim the metadata a bit as it has too many relationships, but it is good enough to check the conversation. Then compared, the results with the metadata conversion using Crosscite Content Negotiation (a service maintained by Crossref and DataCite). Finally, I validated the resulting metadata against the DataCite API validation.

![Sucessulful DOI metadata Validation from [*https://gist.github.com/kjgarza/343e8c44b3ed018eaa9d216b479145f9*](https://gist.github.com/kjgarza/343e8c44b3ed018eaa9d216b479145f9)](https://cdn-images-1.medium.com/max/1024/0*kh1qx8Jqj7bQL5MV.png)

Overall the results are excellent. First, the conversion into valid DataCite metadata indicates that it understands the DataCite Metadata. Second, in terms of the comparison, the results are very positive but not perfect. Mandatory fields mapped corrected. ChatGPT also made some interpretations to enrich the metadata with "subjects" and a "description." Optional fields like dates and funding had mixed results, but they are not inaccurate. The metadata [file outputs can be found on GitHub](https://gist.github.com/kjgarza/343e8c44b3ed018eaa9d216b479145f9).

The results from both experiments are promising. Both examples show that the LLM model is aware of the different schemas, their properties, formats, software patterns, languages, and syntax. **Basically, we are** [**outsourcing domain knowledge**](https://www.youtube.com/watch?v=Qt3eb33296A&ab_channel=BriinkIntelligenceGmbH) **to the LLM.**

Nevertheless, it is important to consider the potential for bias in large language models. Language is inherently subjective, and if models are trained on biased data, they may perpetuate that bias in the code or mappings they generate. For example, what happens when the model chooses to translate to an older version of the schema instead of the newest one? Such biases could lead to unintended consequences and discrimination in the programs and systems we build.

In short, **I believe that the use of large language models can potentially significantly transform the way we map metadata schema models**. By automating the process and reducing the need for manual labor, we can improve the efficiency and accuracy of metadata mapping. As language models continue to improve, I would not be surprised if we see even more drastic transformations in the future.

So, there you have it. The potential of large language models to transform metadata mapping is enormous. It is an exciting time to work in this field, and I cannot wait to see what the future holds.

*Originally published at* [*https://kjgarza.github.io*](https://kjgarza.github.io/pages/2023/01/16/coversion_software.html)*.*

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8df54e41e24){width="1" height="1"}
