Zijian Yang

### OpenAI Canvas and DevDay: Advancing AI Collaboration with Voice Realtime API, Model Distillation, and More

#### Explore OpenAI's latest tools and features --- Canvas, Realtime API, Prompt Cache, Visual Fine-Tuning {#explore-openais-latest-tools-and-features-canvas-realtime-api-prompt-cache-visual-fine-tuning}

<figure>
<img
src="https://cdn-images-1.medium.com/max/800/0*EdCGEumO7YJU66dB.png" />
</figure>

### Author

- [Zijian Yang](https://www.linkedin.com/in/zijian-yang/) (**ORCID:**
  [0009--0006--8301--7634](https://orcid.org/0009-0006-8301-7634))

### Introduction

OpenAI recently introduced a new feature called Canvas, which is
designed to provide ChatGPT users with a more powerful writing and
coding collaboration experience. Canvas is a newly crafted interface
that offers an independent workspace beyond the usual ChatGPT dialogue,
built on the latest GPT-4o model to support more complex writing and
coding projects.

Canvas focuses on two main categories: writing and coding. On the
writing side, Canvas offers editing suggestions, adjustable text length,
readability level options, and the ability to add final touches,
including emojis. On the coding side, Canvas provides code review,
annotation and logging, bug fixing, and the capability to convert code
into other programming languages such as JavaScript or Python. These
features allow users to co-create and refine ideas with AI, making
collaboration more seamless and effective.

In essence, Canvas introduces a novel way for users and AI to work side
by side --- not just through simple dialogue, but as genuine
collaborators. This helps users precisely address and optimise content
within the project's context, marking yet another step by OpenAI towards
enhancing productivity with AI, aiming for a more integrated and smooth
collaborative experience.

At the recent DevDay, OpenAI also announced five major innovations:
Realtime API, Prompt Cache, Model Distillation, Visual Fine-Tuning, and
New Playground Features. These updates signal a shift in OpenAI's
strategic focus towards infrastructure and the developer ecosystem,
rather than solely competing in the end-user AI application market.

Compared to last year's grand reveal, this year's DevDay was more
understated, focusing on incremental improvements to existing AI tools
and API suites while showcasing enhanced developer capabilities and
community stories. Here's a brief overview of the five updates:

- **Realtime API**: Offers developers a near real-time
  "speech-to-speech" experience, with six voice options provided
  by OpenAI.
- **Prompt Cache**: Similar to the feature introduced by Anthropic
  earlier, this allows developers to cache common contexts between API
  calls, reducing costs and improving latency.
- **Model Distillation**: Developers can fine-tune smaller models (such
  as GPT-4o mini) using larger models like o1-preview and GPT-4o.
- **Visual Fine-Tuning**: Combines images and text to fine-tune GPT-4o
  applications, significantly enhancing the model's visual
  understanding.
- **New Playground Features**: Offers a new framework and usage options
  for prompts, along with breakthroughs in structured output, making
  development easier than ever.

### Canvas: Collaborate Side-by-Side with ChatGPT

Canvas is a new feature of ChatGPT that provides an independent
workspace, allowing users to interact with ChatGPT in a more intuitive
and flexible way. Built on OpenAI's latest GPT-4o model, Canvas
overcomes the limitations of traditional chat windows for handling
complex tasks. Whether it's writing or coding, this new collaborative
approach offers an enhanced experience.

Currently, Canvas is in its testing phase and can be manually selected
across all models. Plus users have immediate access without any wait,
while OpenAI plans to eventually roll it out to all free users. With
Canvas, you can do research, write code, draft emails, and most
importantly, brainstorm creatively alongside ChatGPT.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*MUxNob5kVs7SruI6.png" />
</figure>

In Canvas mode, it's no longer just about having a simple conversation.
Instead, users can refine, rework, and organise the generated
content --- essentially transforming the process into something akin to
a content or code editor.

> For instance, if you upload a file and ChatGPT generates some content,
> you'll notice an **"Edit"** button in the **bottom right corner** of
> the interface, allowing you to take full control of the generated
> output.

<figure>
<img
src="https://cdn-images-1.medium.com/max/796/0*hcqH_5mn7LInMrTw.gif" />
</figure>

This button reveals **five features**, which include:

- **Suggest edits**
- **Adjust the length**
- **Reading level**
- **Add final polish**
- **Add emojis**

For example, with the "Suggest edits" feature, you can simply select the
portion of text you want to modify, then click **"Apply"** to have the
content automatically regenerated.

<figure>
<img
src="https://cdn-images-1.medium.com/max/796/0*St5DNZeIrNy0_96c.gif" />
</figure>

You can also "highlight" the article title and have it "reworked" to
your requirements, with fine-tuning options available.

<figure>
<img
src="https://cdn-images-1.medium.com/max/796/0*Hxel9Mr9Kzq9Vg8S.gif" />
</figure>

You can also highlight specific sections to draw ChatGPT's attention,
allowing the model to provide in-line feedback and suggestions while
considering the overall project context.

<figure>
<img
src="https://cdn-images-1.medium.com/max/783/0*gfvVpJ2juJQjkj8T.gif" />
</figure>

So, what else can you do with Canvas? Let's continue exploring.

### Writing with ChatGPT

Earlier, we showcased the "Edit" feature within Canvas writing. Now,
let's take a look at the **"Adjust the Length"** option.

As the name suggests, this feature allows you to adjust the length of
the document, making it either shorter or longer. There are five
settings available: Current Length, Longer, Longest, Shorter, and
Shortest.

<figure>
<img
src="https://cdn-images-1.medium.com/max/798/0*IIDxo9DLmQ0kV4e2.gif" />
</figure>

Once you select the desired length, ChatGPT will adjust the entire text,
modifying it word by word and paragraph by paragraph.

<figure>
<img
src="https://cdn-images-1.medium.com/max/797/0*NIA4KtoIfzhNZSg5.gif" />
</figure>

The **"Change Reading Level"** feature is quite impressive, offering six
levels: **Kindergarten**, **Middle School**, **Current Level**, **High
School**, **College**, and **Graduate**.

Simply select the desired reading level, and ChatGPT will adjust the
entire text accordingly.

<figure>
<img
src="https://cdn-images-1.medium.com/max/796/0*WUqaStUDESwRCVmo.gif" />
</figure>

The **"Add Final Polish"** feature makes comprehensive edits to the
entire text, including checking grammar, clarity, and consistency.

<figure>
<img
src="https://cdn-images-1.medium.com/max/640/0*0W4aleahXJVCiUiX.gif" />
</figure>

The **"Add Emojis"** feature is quite fun --- it lets you add emojis to
your text, making the content more lively and engaging.

For example, one user shared their creative result:

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*qYPgOMC6QCMiSugc.png" />
</figure>

### Coding with ChatGPT

In addition to writing, Canvas also offers a **coding** feature, which
includes the following five capabilities:

- **Review code**
- **Add logs**
- **Add comments**
- **Fix bugs**
- **Port to a language**

<figure>
<img
src="https://cdn-images-1.medium.com/max/640/0*a6pDeE2I_zpiU_oB.gif" />
</figure>

For example, let's give ChatGPT a request:

> *Help me write an API web server in Rust.*

<figure>
<img
src="https://cdn-images-1.medium.com/max/785/0*UaPIZKjAXIIibE7b.gif" />
</figure>

In Canvas, you can highlight a specific code snippet and request
modifications according to your requirements.

<figure>
<img
src="https://cdn-images-1.medium.com/max/785/0*ncuffX5z3PCfLHFy.gif" />
</figure>

For specific details, you also have the option to manually make changes.

<figure>
<img
src="https://cdn-images-1.medium.com/max/785/0*DvGxuBB406PHyaxk.gif" />
</figure>

For features like code review, adding logs, adding comments, and fixing
bugs, a single click allows you to make global changes.

Notably, the **Change Programming Language** feature currently offers
options including JavaScript, TypeScript, Python, Java, C++, and PHP.

<figure>
<img
src="https://cdn-images-1.medium.com/max/802/0*LRUC3mXVyWGUUXZm.gif" />
</figure>

### Making AI the Ultimate Collaborator

For Canvas, OpenAI has implemented over 20 automated internal
evaluations to measure progress, utilising novel synthetic data
generation techniques --- such as extracting outputs from OpenAI
o1-preview --- to conduct post-training on the model for its core
behaviours.

The advantage of this approach is the ability to quickly address writing
quality and new user interaction issues without relying on manually
generated data.

A key challenge is defining when to trigger Canvas. The model needs to
be sensitive enough to identify tasks requiring detailed review or
modification, like "write a blog post about the history of coffee
beans," while avoiding unnecessary activations for more general tasks,
such as "help me create a new dinner recipe," which might not
require Canvas.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*bnJNDOycfTYpq5wo.png" />
</figure>

For writing and coding tasks, we improved correctly triggering the
canvas decision boundary, reaching 83% and 94% respectively compared to
a baseline zero-shot GPT-4o with prompted instructions.

The second challenge involves adjusting the model's editing behaviour
once Canvas is triggered, particularly deciding when to make targeted
edits instead of rewriting the entire content.

The model is trained to favour targeted edits, especially when the user
explicitly selects text. As the model continues to evolve, its ability
to handle these nuanced editing tasks is also improving.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*nEUWtqX3dkqXqF5w.png" />
</figure>

For writing and coding tasks, we prioritised improving canvas-targeted
edits. GPT-4o with canvas performs better than a baseline prompted
GPT-4o by 18%.

Lastly, training the model to generate high-quality code comments also
requires a detailed iterative process. This evaluation differs from the
previous cases. While determining whether Canvas was triggered or
whether targeted edits were made can be easily automated, assessing the
quality of code comments is more challenging to automate, leading the
team to opt for manual evaluation.

Compared to the baseline model, the integration of Canvas improved
GPT-4o's comment accuracy by 30% and overall quality by 16%. This
indicates that training with synthetic data significantly enhances the
model's response quality and behaviour, compared to zero-shot prompts
with detailed instructions.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*2gLX1QEl6PUDaTug.png" />
</figure>

Human evaluations assessed canvas comment quality and accuracy
functionality. Our canvas model outperforms the zero-shot GPT-4o with
prompted instructions by 30% in accuracy and 16% in quality.

A developer described the Canvas interface as a game changer. They
recently used Canvas with ThreeJS to create a tesseract/Hypercube
visualisation tool and mentioned how much they enjoy the unified UX for
chatting, live reviews and watching GPT-4o work its magic on
code --- all in one place, always relevant.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*B8xVKylexLVk6BBV.png" />
</figure>

<figure>
<img
src="https://cdn-images-1.medium.com/max/960/0*9sqqQVDeyDnGc9Vz.gif" />
</figure>

### Realtime Multimodal Conversation API: The Era of AI Voice Applications Has Arrived!

OpenAI's newly launched Realtime API is **currently in public beta**.

This new product allows developers to create low-latency, multimodal
experiences, particularly for voice-to-voice applications.

This means developers can now start adding ChatGPT's "voice controls" to
their apps.

To showcase the API's potential, OpenAI demonstrated an updated version
of Wanderlust --- a "travel planning" app originally presented at last
year's event. With the Realtime API, users can interact with the app
directly, planning trips in a natural, conversational manner. The system
even supports interruptions, mimicking the pauses and nuances of human
conversation.

While the travel planning app is just an example, the Realtime API opens
up a wide range of possibilities for voice applications across various
industries.

From **customer service** to **education** and **assistive tools**,
developers now have a powerful new resource for creating more intuitive,
responsive AI-driven experiences.

The Realtime API essentially simplifies the process of building **voice
assistants** and other **conversational AI tools**, eliminating the need
to piece together multiple models for transcription, inference, and
text-to-speech conversion.

Early adopters of the Realtime API, such as Healthify --- a nutrition
and fitness guidance app --- and Speak, a language learning platform,
have already integrated the API into their products. These
implementations demonstrate the potential of the API to create more
natural, engaging user experiences in fields like healthcare and
education.

Though the Realtime API **is not cheap (\$0.06 per minute for audio
input, \$0.24 per minute for audio output)**, it still represents a
significant value proposition for those looking to develop voice-based
applications.

### Prompt Cache: A Major Budget Saver for Developers

The "Prompt Cache" feature is **designed to reduce costs and latency for
developers**.

The system automatically applies a 50% discount to tokens that the model
has recently processed, which can lead to substantial savings for
applications that frequently reuse context.

The significant reduction in costs opens up major opportunities for
startups and large enterprises to explore new applications that were
previously unattainable due to budget constraints.

At the 2024 OpenAI DevDay, the pricing chart showed a significant
reduction in the cost of using AI models. Cached input tokens can save
up to 50% compared to non-cached tokens across various GPT models. The
new o1 model further highlights its advanced capabilities.

This also involves structured prompts, where caching hits are only
possible when the prefixes of prompts match exactly. To take full
advantage of caching, it's necessary to place static content (such as
instructions and examples) at the beginning of the prompt and put
variable content (such as user-specific information) at the end. This
also applies to images and tools, which must remain consistent between
requests.

### Model Distillation: Making AI Model Training More Efficient

Perhaps the most transformative change at DevDay was the introduction of
"Model Distillation." This integrated workflow allows developers to use
outputs from advanced models like o1-preview and GPT-4o to improve the
performance of more efficient models like GPT-4o mini.

In everyday training, larger AI models (such as o1-preview and GPT-4o)
can be used to fine-tune smaller models (like GPT-4o mini). This
approach enables small companies to utilise capabilities similar to
advanced models without incurring the same computational costs.

It bridges the longstanding gap in the AI industry between cutting-edge,
resource-intensive systems and more accessible but less powerful
systems. For instance, a small health tech startup developing an AI
diagnostic tool for rural clinics can use model distillation to train a
compact model that runs on a standard laptop or tablet, capturing much
of the diagnostic capability of larger models. This brings advanced AI
capabilities to resource-limited settings, potentially improving
healthcare outcomes in underserved areas.

### Visual Fine-Tuning: The New Frontier of Visual AI

Another major update is the introduction of visual fine-tuning for
OpenAI's latest large language model, GPT-4o. This feature allows
developers to customise the model's visual understanding using both
images and text.

The impact of this update is profound, with potential applications in
areas such as autonomous vehicles, medical imaging, and visual search
functionalities. According to OpenAI, Grab, a leading food delivery and
ride-hailing company in Southeast Asia, has utilised this technology to
improve its mapping services. Reportedly, with only 100 examples, Grab
increased lane count accuracy by 20% and speed sign recognition accuracy
by 13%.

This real-world application demonstrates the potential of visual
fine-tuning, showing that using small batches of visual training data
can significantly enhance AI services across industries.

**Gathering information in a "pure visual" manner through "seeing" will
greatly expand future application scenarios.** Combined with "real-time
voice interaction," OpenAI's advances in information processing and
interaction create possibilities for a new wave of applications.

Envisioned scenarios include assisting people with visual or hearing
impairments, recognising emergencies and hazards through visual
identification, and calling for help automatically.

### New Playground Features: Building a Sustainable AI Ecosystem

More explanations have been provided on the structured framework for
prompts and other usages. Parts of this information were allegedly
leaked a day earlier, including the so-called "System Prompts."

The key elements include: "Understand the task: Grasp the main
objectives, goals, requirements, constraints, and expected outputs."

- **Minimal Changes**: If an existing prompt is provided, only make
  simple improvements if possible. For more complex prompts, enhance
  clarity and add missing elements without changing the original
  structure.
- **Reasoning Before Conclusion**: Encourage reasoning steps before
  arriving at any conclusion. Attention! If the user provides an example
  where reasoning follows, REVERSE the order! Never begin examples with
  a conclusion!
- **Reasoning Order**: Highlight the reasoning parts of the prompt and
  conclusion sections (specifically named fields). For each, determine
  the ORDER in which to do so and whether a reversal is needed.
- **Conclusion, Classification, or Outcome**: These should always
  come last.

### Conclusion

OpenAI's Canvas and recent DevDay innovations represent a major step
forward in AI collaboration, offering developers powerful tools like the
Realtime API, Prompt Cache, Model Distillation, Visual Fine-Tuning, and
Playground features. These advancements make AI more accessible, enhance
productivity in writing and coding, and support industries like
healthcare and assistive technology. By bridging human creativity with
AI efficiency, OpenAI aims to foster a sustainable ecosystem that
empowers developers and creates real-world impact.

### References

- "Introducing Canvas." *Openai.com*, 2024,
  openai.com/index/introducing-canvas/.
- "X.com." *X (Formerly Twitter)*, 2024,
  x.com/karinanguyen\_/status/1841889811931791642. Accessed 12
  Oct. 2024.

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a140e4d65486){width="1"
height="1"}


OpenAI Canvas and DevDay: Advancing AI Collaboration with Voice Realtime API, Model Distillation, and More  Explore OpenAI’s latest tools and features — Canvas, Realtime API, Prompt Cache, Visual Fine-Tuning  Author  Zijian Yang (
<strong>
 ORCID:
</strong>
0009–0006–8301–7634) Introduction   OpenAI recently introduced a new feature called Canvas, which is designed to provide ChatGPT users with a more powerful writing and coding collaboration

OpenAI Canvas and DevDay: Advancing AI Collaboration with Voice Realtime API, Model Distillation…

Swinburne University of Technology

Nakul Nambiar

Zhuochen Wu

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*7Fy1vUav7gAiy9CAWanAvQ.png" />
<figcaption>Created using DALLE on 6 Feb 2024</figcaption>
</figure>

#### Authors:

* Nakul Nambiar (ORCID: [0009-0009-9720-9233](https://orcid.org/0009-0009-9720-9233))
* Zhuochen Wu (ORCID: [0009-0000-5642-5348](https://orcid.org/0009-0000-5642-5348))

Research Graph is a structured representation of research objects that
captures information about entities and the relationships between
Researcher, Organisation, Publication, Grant and Research Data.
Currently, the publications are available as PDF files, and due to the
free-form text, it is difficult to parse a PDF file to extract
structured information. In this article, we will try to create a
research graph from a PDF of a publication by extracting relevant
information from the text and organising it into a graph structure
using OpenAI.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*40j-_agywNgUsJ_h_kXslA.jpeg" />
<figcaption>Pipeline to create the graph from PDF</figcaption>
</figure>

### OpenAI

In this work, we use OpenAI API and the new Assistant functionality of
GPT (Currently in Beta) to transform PDF documents to a set of
structured JSON files based on Research Graph Schema.

#### Assistants API

The Assistants API allows you to build AI assistants within your
applications. An assistant can answer user questions by using models,
tools, and information per predetermined guidelines. It is a beta API
that is actively under development. Using the assistants API, we can use
the OpenAI-hosted tools like code interpreter and knowledge retrieval.
For this blog post, we will focus on knowledge retrieval.

#### Knowledge Retrieval

Sometimes, we need the AI model to answer queries based on unknown
knowledge, like user-provided documents or sensitive information. We can
augment the model with this information using the assistant APIs
knowledge retrieval tool. We can upload files to the assistant, and it
automatically chunks the documents and creates and stores embeddings to
implement vector search on the data.

#### Example

For our example, we will upload PDF files of publications to the OpenAI
assistant and the knowledge retrieval tool to get a JSON output of the
graph schema for the given publications. The publication we used for
this example can be accessed from the following
[link](https://doi.org/10.5166/jroi-2-1-6).

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*CrAVHSNrRmQiBay81_Bi0Q.png" />
<figcaption>Screenshot from the publication (<a
href="https://doi.org/10.5166/jroi-2-1-6">https://doi.org/10.5166/jroi-2-1-6</a>)</figcaption>
</figure>

#### Step 1

Read the input path where the Publications PDFs are stored and the
output path where the JSON output will be stored.

    import configparser
    config = configparser.ConfigParser()
    config.read('{}/config.ini'.format(current_path))
    input_path = config['DEFAULT']['Input-Path']
    output_path = config['DEFAULT']['Output-Path']
    debug = config['DEFAULT']['Debug']

#### Step 2

Get all the PDF files from the Input path.

    onlyfiles = [f for f in os.listdir(input_path) if os.path.isfile(os.path.join(input_path, f))]

#### Step 3

Then, we need to initialise the assistant to use the Knowledge retrieval
tool. To do this, we need to specify the type of tools in the API for
"retrieval". We also specify the instructions for the assistant and the
OpenAI model to be used.

    my_file_ids = []
    if client.files.list().data==[]:
      for f in onlyfiles:
        file = client.files.create(
          file=open(input_path + f, "rb"),
          purpose='assistants'
        )
        my_file_ids.append(file.id)

    # Add the file to the assistant
    assistant = client.beta.assistants.create(
      instructions = "You are a publication database support chatbot. Use pdf files uploaded to best respond to user queries in JSON.",
      model = "gpt-4-1106-preview",
      tools = [{"type": "retrieval"}],
      # Do not attach all files to the assistant, otherwise, it will mismatch the answers even though specify file ID in query messages.
      # We will attach to each message instead
    )

#### Step 4

Then, we specify the information we need to extract from the
publications files and pass them as user queries to the assistant. After
experimenting with the assistant instructions, we found that requesting
JSON format in each user message generates the most consistent output.

    user_msgs = ["Print the title of this paper in JSON",
                 "Print the authors of this paper in JSON",
                 "Print the abstract section of the paper in JSON",
                 "Print the keywords of the paper in JSON",
                 "Print the DOI number of the paper in JSON",
                 "Print the author affiliations of the paper in JSON",
                 "Print the reference section of the paper in JSON"]

#### Step 5

The next is to pass the queries to the assistant to generate the output.
We need to create an individual thread object for each user query, which
contains the query as a user message. Then, we run the thread and
retrieve the assistant's answer.

    all_results = []

    for i in my_file_ids:
        print('\n#####')
        # the JSON result it can extract and parse, hopefully
        file_result = {}
       
        for q in user_msgs:
            # create thread, user message and run objects for each query
            thread = client.beta.threads.create()
            msg = client.beta.threads.messages.create(
                    thread_id=thread.id,
                    role="user",
                    content=q,
                    file_ids=[i] # specify the file/publication we want to extract from
                )
            print('\n',q)
            run = client.beta.threads.runs.create(
                thread_id=thread.id,
                assistant_id=assistant.id,
                additional_instructions="If answer cannot be found, print 'False'" # not very useful at the time of this presentation
                )
            # checking run status by retrieving updated object each time
            while run.status in ["queued",'in_progress']:
                print(run.status) 
                time.sleep(5)
                run = client.beta.threads.runs.retrieve(
                    thread_id=thread.id,
                    run_id=run.id
                    )
            # usually a rate limit error
            if run.status=='failed':  logging.info("Run failed: ", run)
            if run.status=='completed':
                print("<Complete>")
                # extract updated message object, this includes user messages
                messages = client.beta.threads.messages.list(
                    thread_id=thread.id
                )
                for m in messages:
                    if m.role=='assistant':
                        value = m.content[0].text.value # get the text response
                        if "json" not in value:
                            if value=='False':logging.info("No answer found for ", str(q))
                            else:
                                logging.info("Not JSON output, maybe no answer found in the file or model is outdated: ", str(value))
                        else:
                            # clean the response and try to parse as json
                            value = value.split("```")[1].split('json')[-1].strip()
                            try: 
                                d = json.loads(value)
                                file_result.update(d)
                                print(d)
                            except Exception as e:
                                logging.info(f"Query {q} \nFailed to parse string to JSON: ", str(e))
                                print(f"Query {q} \nFailed to parse string to JSON: ", str(e))
                
        all_results.append(file_result)

The JSON output generated for the above publication file is:

    [{'title': 'Dodes (diagnostic nodes) for Guideline Manipulation',
      'authors': [{'name': 'PM Putora',
        'affiliation': 'Department of Radiation-Oncology, Kantonsspital St. Gallen, St. Gallen, Switzerland'},
       {'name': 'M Blattner',
        'affiliation': 'Laboratory for Web Science, Zürich, Switzerland'},
       {'name': 'A Papachristofilou',
        'affiliation': 'Department of Radiation Oncology, University Hospital Basel, Basel, Switzerland'},
       {'name': 'F Mariotti',
        'affiliation': 'Laboratory for Web Science, Zürich, Switzerland'},
       {'name': 'B Paoli',
        'affiliation': 'Laboratory for Web Science, Zürich, Switzerland'},
       {'name': 'L Plasswilma',
        'affiliation': 'Department of Radiation-Oncology, Kantonsspital St. Gallen, St. Gallen, Switzerland'}],
      'Abstract': {'Background': 'Treatment recommendations (guidelines) are commonly represented in text form. Based on parameters (questions) recommendations are defined (answers).',
       'Objectives': 'To improve handling, alternative forms of representation are required.',
       'Methods': 'The concept of Dodes (diagnostic nodes) has been developed. Dodes contain answers and questions. Dodes are based on linked nodes and additionally contain descriptive information and recommendations. Dodes are organized hierarchically into Dode trees. Dode categories must be defined to prevent redundancy.',
       'Results': 'A centralized and neutral Dode database can provide standardization which is a requirement for the comparison of recommendations. Centralized administration of Dode categories can provide information about diagnostic criteria (Dode categories) underutilized in existing recommendations (Dode trees).',
       'Conclusions': 'Representing clinical recommendations in Dode trees improves their manageability handling and updateability.'},
      'Keywords': ['dodes',
       'ontology',
       'semantic web',
       'guidelines',
       'recommendations',
       'linked nodes'],
      'DOI': '10.5166/jroi-2-1-6',
      'references': [{'ref_number': '[1]',
        'authors': 'Mohler J Bahnson RR Boston B et al.',
        'title': 'NCCN clinical practice guidelines in oncology: prostate cancer.',
        'source': 'J Natl Compr Canc Netw.',
        'year': '2010 Feb',
        'volume_issue_pages': '8(2):162-200'},
       {'ref_number': '[2]',
        'authors': 'Heidenreich A Aus G Bolla M et al.',
        'title': 'EAU guidelines on prostate cancer.',
        'source': 'Eur Urol.',
        'year': '2008 Jan',
        'volume_issue_pages': '53(1):68-80',
        'notes': 'Epub 2007 Sep 19. Review.'},
       {'ref_number': '[3]',
        'authors': 'Fairchild A Barnes E Ghosh S et al.',
        'title': 'International patterns of practice in palliative radiotherapy for painful bone metastases: evidence-based practice?',
        'source': 'Int J Radiat Oncol Biol Phys.',
        'year': '2009 Dec 1',
        'volume_issue_pages': '75(5):1501-10',
        'notes': 'Epub 2009 May 21.'},
       {'ref_number': '[4]',
        'authors': 'Lawrentschuk N Daljeet N Ma C et al.',
        'title': "Prostate-specific antigen test result interpretation when combined with risk factors for recommendation of biopsy: a survey of urologist's practice patterns.",
        'source': 'Int Urol Nephrol.',
        'year': '2010 Jun 12',
        'notes': 'Epub ahead of print'},
       {'ref_number': '[5]',
        'authors': 'Parmelli E Papini D Moja L et al.',
        'title': 'Updating clinical recommendations for breast colorectal and lung cancer treatments: an opportunity to improve methodology and clinical relevance.',
        'source': 'Ann Oncol.',
        'year': '2010 Jul 19',
        'notes': 'Epub ahead of print'},
       {'ref_number': '[6]',
        'authors': 'Ahn HS Lee HJ Hahn S et al.',
        'title': 'Evaluation of the Seventh American Joint Committee on Cancer/International Union Against Cancer Classification of gastric adenocarcinoma in comparison with the sixth classification.',
        'source': 'Cancer.',
        'year': '2010 Aug 24',
        'notes': 'Epub ahead of print'},
       {'ref_number': '[7]',
        'authors': 'Rami-Porta R Goldstraw P.',
        'title': 'Strength and weakness of the new TNM classification for lung cancer.',
        'source': 'Eur Respir J.',
        'year': '2010 Aug',
        'volume_issue_pages': '36(2):237-9'},
       {'ref_number': '[8]',
        'authors': 'Sinn HP Helmchen B Wittekind CH.',
        'title': 'TNM classification of breast cancer: Changes and comments on the 7th edition.',
        'source': 'Pathologe.',
        'year': '2010 Aug 15',
        'notes': 'Epub ahead of print'},
       {'ref_number': '[9]',
        'authors': 'Paleri V Mehanna H Wight RG.',
        'title': "TNM classification of malignant tumours 7th edition: what's new for head and neck?",
        'source': 'Clin Otolaryngol.',
        'year': '2010 Aug',
        'volume_issue_pages': '35(4):270-2'},
       {'ref_number': '[10]',
        'authors': 'Guarino N.',
        'title': 'Formal Ontology and Information Systems',
        'source': '1998 IOS Press'},
       {'ref_number': '[11]',
        'authors': 'Uschold M Gruniger M.',
        'title': 'Ontologies: Principles Methods and Applications.',
        'source': 'Knowledge Engineering Review',
        'year': '1996',
        'volume_issue_pages': '11(2)'},
       {'ref_number': '[12]',
        'authors': 'Aho A Garey M Ullman J.',
        'title': 'The Transitive Reduction of a Directed Graph.',
        'source': 'SIAM Journal on Computing',
        'year': '1972',
        'volume_issue_pages': '1(2): 131–137'},
       {'ref_number': '[13]',
        'authors': 'Tai K',
        'title': 'The tree-to-tree correction problem.',
        'source': 'Journal of the Association for Computing Machinery (JACM)',
        'year': '1979',
        'volume_issue_pages': '26(3):422-433'}]}]

#### Step 6

File objects and assistant object needs to be cleaned up because they
cost money in 'retrieval' mode. Also, it is a good coding practice.

    for f in client.files.list().data:
        client.files.delete(f.id)

    # Retrieve and delete running assistants
    my_assistants = client.beta.assistants.list(
        order="desc",
    )
    for a in my_assistants.data:    
        response = client.beta.assistants.delete(a.id)
        print(response)

#### Step 7

The next step is to generate a graph visualisation using the Python
[Networkx](https://networkx.org/documentation/stable/tutorial.html) package.

    import networkx as nx
    import matplotlib.pyplot as plt

    G = nx.DiGraph()
    node_colors = []

    key = "jroi/" + all_results[0]['title']
    G.add_nodes_from([(all_results[0]['title'], {'doi': all_results[0]['DOI'], 'title': all_results[0]['title'], 'source': 'jroi', 'key': key})])
    node_colors.append('#4ba9dc')

    for author in all_results[0]['authors']:
        key = "jroi/" + author['name']
        G.add_nodes_from([(author['name'], {'key': key, 'local_id': author['name'], 'full_name': author['name'], 'source': 'jroi'})])
        G.add_edge(all_results[0]['title'], author['name'])
        node_colors.append('#63cc9e')


    for reference in all_results[0]['references']:
        key = "jroi/" + reference['title']
        G.add_nodes_from([(reference['title'].split('.')[0][:25] + '...', {'title': reference['title'], 'source': 'jroi', 'key': key})])
        G.add_edge(all_results[0]['title'], reference['title'].split('.')[0][:25] + '...')
        node_colors.append('#4ba9dc')

    pos = nx.spring_layout(G)
    labels = nx.get_edge_attributes(G, 'label')
    nx.draw(G, pos, with_labels=True, node_size=1000, node_color=node_colors, font_size=7, font_color='black')
    nx.draw_networkx_edge_labels(G, pos, edge_labels=labels)
    plt.savefig("graph_image.png")
    plt.show()

The graph visualisation is as below:

<figure>
<img
src="https://cdn-images-1.medium.com/max/640/1*-a5y19GUSyQZv63KMXKnZA.png" />
<figcaption>Graph Visualisation generated using Networkx</figcaption>
</figure>

**Note:** Please note that the output structure generated by OpenAI may
be different for different executions. So you may need to update the
above code according to that structure.

### Conclusion

In conclusion, leveraging the GPT API to extract a Research Graph from
PDF publications offers a powerful and efficient solution for
researchers and data analysts. This workflow streamlines the process of
converting PDF publications into structured and accessible Research
Graphs. But we must also pay careful attention to the inconsistencies of
the responses generated by LLM. Over time, accuracy and relevance may be
further enhanced by routinely updating and improving the extraction
model.

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4655f64337db){width="1"
height="1"}


Authors:  Nakul Nambiar (ORCID: 0009-0009-9720-9233) Zhuochen Wu (ORCID: 0009-0000-5642-5348)  Research Graph is a structured representation of research objects that captures information about entities and the relationships between Researcher, Organisation, Publication, Grant and Research Data.

How to use GPT API to export a research graph from PDF publications

Aaron Tay

I've been a subscriber of OpenAI's ChatGPT plus for a while though I have been struggling to justify it to myself for a while until recently.

Messaggi di Rogue Scholar

OpenAI Canvas and DevDay: Advancing AI Collaboration with Voice Realtime API, Model Distillation…

How to use GPT API to export a research graph from PDF publications

GPT4+Code interpreter - Playing Electronic Resources Librarian and Research Data Management Librarian roles