Australian National University

Wenyi Pi

Neo4j APOC Library Use Case

 Author

Wenyi Pi (

 ORCID:

0009–0002–2884–277)

 Introduction

In the realm of Neo4j, the APOC (Awesome Procedures on Cypher) library stands as a powerful tool. Previously, We have talked about the importance of APOC in optimising Cypher queries and improving query efficiency in our article Exploring Methods of Cypher Query Optimisations.

Introduction to APOC: Enhancing Neo4j Capabilities

Xuzeng He

Latest findings in multiple research directions for handling graph prediction and optimization Author · Xuzeng He (

 ORCID:

0009–0005–7317–7426) A graph, in short, is a description of items linked by relations, where the items of a graph are called

 nodes

(or vertices) and their relations are called

 edges

(or links). Examples of graphs can include social networks (e.g. Instagram) or knowledge

Recent Advances in using Machine Learning with Graphs

Vaibhav Khobragade

Author: Vaibhav Khobragade (

 ORCID:

0009–0009–8807–5982) Introduction Large language models (LLMs) are becoming increasingly popular in natural language processing for their superior competence in various applications.

Enhancing Language Models: The Role of Knowledge Graph Augmentation in Overcoming LLM Challenges

Enhancing Data Interactivity with LLMs and Neo4j Knowledge Graphs Author Wenyi Pi (

 ORCID

: 0009–0002–2884–2771) Introduction Since OpenAI launched ChatGPT, a

 large language model (LLM)

based chatbot, in 2023, it has set off a technological wave.

The integration of large language models (LLMs) with Neo4j-based knowledge graphs

An Introduction to RA-CM3, MuRAG and RACE Author  Xuzeng He (ORCID: 0009-0005-7317-7426)  Generative Artificial Intelligence (GAI) has demonstrated impressive performances in tasks such as text generation and text-to-image generation.

Recent Advances in using Retrieving Multimodal Information for Augmented Generation

Amanda Kau

Unlocking the Power of Questions — A deep dive into Question Answering Systems

Author:

 Amanda Kau (ORCID:


 0009–0004–4949–9284


 )

Virtual assistants have popped up on numerous websites over the years.

Harnessing The Power of Knowledge Graphs in Question Answering

Qingqin Fang

An Introduction to Retrieval Augmented Generation (RAG) and Knowledge Graph

Author Qingqin Fang (ORCID: 0009–0003–5348–4264)

 Introduction

Large Language Models (LLMs) have transformed the landscape of natural language processing, demonstrating exceptional proficiency in generating text that closely resembles human language.

Hallucination in Large Language Models and Two Effective Alleviation Pathways

Yunzhong Zhang

Efficient creation of a stoplight report with data dashboard images

Author: Yunzhong Zhang (ORCID: 0009–0002–8177–419X) Comparing data dashboards is crucial for understanding trends and performance differences. Traditionally, this task required manual effort, which was slow and sometimes inaccurate. Now, thanks to OpenAI’s GPT-4 with Vision (GPT-4V), we are able to automate and improve this process.

How to use GPT-4V For Stoplight Report

Dhruv Gupta

Improving the performance of Large Language Models Author  Dhruv Gupta (ORCID: 0009-0004-7109-5403)  ChatGPT, which first came out in late 2022, took the world by storm. Since then, various LLM models and LLM based products such as Meta’s Llama and Google’s Gemini have emerged, demonstrating the power of LLMs.

RAG: The next big thing after LLMs?

Harvesting the Power and Knowledge of Large Language Models


 Author

Amanda Kau

 (ORCID:

0009–0004–4949–9284

 )

Introduction Knowledge Graphs are networks that represent data in a graphical format. The beauty of Knowledge Graphs lies in their representation of concepts, events and entities as nodes, and the relationships between them as edges.

Automated Knowledge Graph Construction with Large Language Models

Swinburne University of Technology

Nakul Nambiar

Zhuochen Wu

Authors:  Nakul Nambiar (ORCID: 0009-0009-9720-9233) Zhuochen Wu (ORCID: 0009-0000-5642-5348)  Research Graph is a structured representation of research objects that captures information about entities and the relationships between Researcher, Organisation, Publication, Grant and Research Data.

How to use GPT API to export a research graph from PDF publications

Dhruv Gupta

Prompt Engineering — Part 2


 Using intelligence to use artificial Intelligence: A deep dive into Prompt Engineering

Author Dhruv Gupta

 (ORCID:

0009–0004–7109–5403

 )

Introduction In the previous article we discussed what prompt engineering and some of the techniques used for prompt engineering.

Prompt Engineering — Part 2

Enhancing Open-Domain Conversational Question Answering with Knowledge-Enhanced Models and Knowledge Graphs


 How knowledge-enhanced language models and knowledge graphs are advancing open-domain conversational question answering

Author:

 Wenyi Pi (ORCID:


 0009-0002-2884-2771


 )

When searching for information on the website, it is common to come across a flood of

Enhancing Open-Domain Conversational Question Answering with Knowledge-Enhanced Models and…

Recent Advances in Using Machine Learning with Graphs — Part 2 Latest findings in multiple research directions for handling graph construction and network security issues Author · Xuzeng He (

 ORCID:

0009–0005–7317–7426) Introduction A graph, in short, is a description of items linked by relations, where the items of a graph are called nodes (or vertices) and their relations are called edges (or links). Examples of

Recent Advances in Using Machine Learning with Graphs — Part 2

Exploring AI’s Ethical Terrain: Addressing Bias, Security, and Beyond

Author: Vaibhav Khobragade (

 ORCID:

0009–0009–8807–5982) Large language models (LLMs) like OpenAI’s GPT-4, Meta’s LLaMA, and Google Gemini (previously called Bard) have showcased their vast capabilities, from passing bar exams and crafting articles to generating images and website code.

Ethics and AI: Confronting the Challenges Ahead

How to efficiently retrieve information for different applications

Author Wenyi Pi (ORCID: 0009-0002-2884-2771) This article aims to explore various ways in which Retrieval-Augmented Generation (RAG) can be utilised to retrieve information and generate responses effectively within the dialogue system. The rationale behind utilising RAG as well as potential ways in which it can be employed effectively will be covered.

Efficient Information Retrieval and Response Generation with Retrieval-Augmented Generation (RAG)

Amir Aryani

Authors  Nakul Nambiar (ORCID: 0009-0009-9720-9233) Amir Aryani (ORCID: 0000-0002-4259-9774)  Knowledge graphs, which offer a structured representation of data and its relationships, are revolutionising how we organise and access information.

Mini RAG using Neo4j

Aishwarya Nambissan

A novel approach to improving the efficiency of text search in graph databases utilizing Neo4j, OpenAI, and Typesense. Authors  Nakul Nambiar (ORCID: 0009–0009–9720–9233) Aishwarya Nambissan (ORCID: 0009–0003–3823–6609)  The ability to use cutting-edge tools and frameworks is essential for staying ahead in the ever-changing field of technology.

Typesense and Neo4j in a hybrid information retrieval solution

Amir Aryani

A brief overview of different types of clustering techniques and their algorithms. Authors Aishwarya Nambissan (ORCID: 0009-0003-3823-6609) Amir Aryani (ORCID: 0000-0002-4259-9774)

 Background

Clustering is a fascinating technique used in machine learning, where patterns or data points are grouped based on their similarities. It’s like finding hidden connections among different data points without predefined labels.

19 Clustering Techniques

Understanding the Evolutionary Journey of LLMs Author Wenyi Pi (

 ORCID

: 0009–0002–2884–2771) Introduction When we talk about large language models (LLMs), we are actually referring to a type of advanced software that can communicate in a human-like manner. These models have the amazing ability to understand complex contexts and generate content that is coherent and has a human feel.

Brief Introduction to the History of Large Language Models (LLMs)

Understanding Sequential Data Modelling with Keras for Time Series Prediction

 Author

Wenyi Pi (

 ORCID

: 0009–0002–2884–2771) Introduction Recurrent Neural Networks (RNNs) are a special type of neural networks that are suitable for learning representations of sequential data like text in Natural Language Processing (NLP). We will walk through a complete example of using RNNs for time series prediction, covering

Beginner’s Guide to Recurrent Neural Networks (RNNs) with Keras

Incorporating Knowledge Graphs to explain reasoning processes


 Author

Amanda Kau (

 ORCID:

0009–0004–4949–9284) Introduction Large language models (LLMs) like GPT-4 possess remarkable language abilities, allowing them to function as chatbots, translators, and much more.

Combining Knowledge Graphs With Language Models for Interpretability

Exploring the OpenAlex Data Structure and Visualization Author: Qingqin Fang (

 ORCID:

0009–0003–5348–4264)

 Introduction to OpenAlex

In today’s world, the realm of research papers is brimming with countless hot topics, and the sheer volume of publications can be overwhelming.

Unveiling Research Trends through OpenAlex Visualization

Large Language Models for Fake News Generation and Detection Author Amanda Kau (

 ORCID

: 0009–0004–4949–9284) Introduction In recent years, fake news has become an increasing concern for many, and for good reason. Newspapers, which we once trusted to deliver credible news through accountable journalists, are vanishing en masse along with their writers.

Are Large Language Models Our Allies or Enemies in the Fight Against Fake News?

Unlocking the power of language models: A deep dive into BERT


 Author:


 Dhruv Gupta (ORCID:


 0009–0004–7109–5403


 )

Clive Humby, in 2006 rightly said, “Data is the new oil”. With data being present everywhere, it has never been more valuable.

Language Models: Deep Dive into BERT

Attention mechanism not getting enough attention Author Dhruv Gupta (

 ORCID

: 0009–0004–7109–5403) Introduction As discussed in this article, RNNs were incapable of learning long-term dependencies. To solve this issue both LSTMs and GRUs were introduced. However, even though LSTMs and GRUs did a fairly decent job for textual data they did not perform well.

Transformers Models in NLP

Exploring the potentials and limitations of Vision Language Models

Author:

 Amanda Kau (ORCID:


 0009–0004–4949–9284


 )

The human brain is more extraordinary than any machine we could build. From an early age, many of us gain the ability to comprehend what our eyes tell us and articulate it. Furthermore, we combine evidence from all our senses to reason.

How Much Can Vision Language Models Really “See”?

A novel compression technique ensuring comparable performance with 70% less parameters

Author Amanda Kau (

 ORCID

: 0009–0004–4949–9284) Introduction The sizes of large language models (LLMs) have been steadily increasing over the last few years.

CompactifAI: Large Language Models Don’t Actually Have To Be Large

From Naive to Modular: Tracing the Evolution of Retrieval-Augmented Generation

Author · Vaibhav Khobragade (

 ORCID:

0009–0009–8807–5982) Introduction Large Language Models (LLMs) have achieved remarkable success.

Three Paradigms of RAG

An introduction to tools supporting PubMed Author: Xuzeng He (

 ORCID:

0009–0005–7317–7426) PubMed is a free Web literature search service developed and maintained by the National Center for Biotechnology Information (

 NCBI

), and it is also a part of NCBI’s Entrez retrieval system that provides access to a diverse set of 38 databases.

Enhancing PubMed: from a Medical Database to beyond

Aland Astudillo

How to use GROBID to extract text from PDF  Author  Aland Astudillo (ORCID: 0009-0008-8672-3168)  GROBID is a powerful and useful tool based on machine learning that can extract text information from PDF files and other files to a structured format. One of the key challenges in knowledge mining from academic articles is reading the content of PDF files.

How to use GROBID

Using Mistral for Data tagging Author · Xuzeng He (

 ORCID:

0009–0005–7317–7426) Introduction Data tagging, in simple terms, is the process of assigning labels or tags to your data so that they are easier to retrieve or analyse.

How to use Large Language Models to tag your data: A complete tutorial

Integrating temporal data into static knowledge graphs

 Author

Amanda Kau

 (ORCID:


 0009–0004–4949–9284


 )

Introduction Knowledge graphs (KGs) have proven to be an effective method of data representation that is increasingly popular. In KGs, entities and concepts are represented as nodes, while the relationships between nodes are depicted as edges.

Dynamic Knowledge Graphs: A Next Step For Data Representation?

Transformative Advances in Language Models through External Knowledge Integration

Author: Qingqin Fang (

 ORCID:

0009–0003–5348–4264)

 Introduction

In the dynamic field of natural language processing, the integration of external knowledge has emerged as a pivotal strategy for enhancing the performance of language models.

Deeper understanding of In-Context Retrieval-Augmented Language Models

Author  Amir Aryani (ORCID: 0000-0002-4259-9774) Introduction   In this article we look at Research Graph as an information model , and an approach to connect and capture the connections between research outputs, researchers and research activities. We explore the metadata model, and we discuss how to capture this graph in a Neo4j Graph Database.

Research Graph 101

Latest findings in multiple research directions for tackling reasoning and common sense challenges Author: Xuzeng He (

 ORCID:

0009–0005–7317–7426) Knowledge Graphs, such as Wikidata, contain rich relational information between entities and have been widely used as a structured format for storing and representing relational information.

Large Language Models and Knowledge Graphs: Ways to combine them

Using intelligence to use artificial Intelligence: A deep dive into Prompt Engineering


 Author


 Dhruv Gupta (ORCID:


 0009–0004–7109–5403


 )

Introduction Large Language Models (LLMs) have become the new normal in the field of Natural Language Processing (NLP). With their improved performance and generative power, people around the world are relying on it for

Prompt Engineering

Boosting Performance for Knowledge Graphs with Neo4j APOC Library

 Author

Wenyi Pi (ORCID: 0009–0002–2884–2771) Introduction A

 knowledge graph

(graph database) captures information about main entities in a domain and the relationships between them. It was an augmented feature store for connected data which gave access to compute, access and operationalise structure features.

Exploring Methods of Cypher Query Optimisations

The AI Helper Turning Mountains of Data into Bite-Sized Instructions

Author Aland Astudillo (ORCID: 0009-0008-8672-3168) LLMs have been changing the way the entire world deals with problems and day-by-day tasks. To make them better for specific applications, they need huge amounts of data and complex and expensive approaches to training them.

What is LLMLingua?

Refining AI Vision: How Retrieval-Augmented Generation Transforms Image Captioning in Large Language Models   Leveraging External Knowledge to Enhance the Descriptive Capabilities of AI Systems Author  Vaibhav Khobragade (ORCID: 0009–0009–8807–5982) Introduction   Large Language Models (LLMs) are artificial intelligence models that are trained on massive amounts of text data in order to generate human-like language and produce coherent

Refining AI Vision: How Retrieval-Augmented Generation Transforms Image Captioning in Large…

Tools and Platform for Integration of Knowledge Graph with RAG pipelines.

Authors Aland Astudillo (ORCID: 0009-0008-8672-3168) Aishwarya Nambissan (ORCID: 0009-0003-3823-6609) Many users of chatbots such as ChatGPT, have encountered the problem of receiving inappropriate or incompatible responses. There are several reasons why this might happen.

Unveiling the Synergy: Retrieval Augmented Generation (RAG) Meets Knowledge Graphs

Author  Amir Aryani: (ORCID: 0000-0002-4259-9774) Definition   A research collaboration network is a group of researchers, and practitioners, or both, working together on joint research activities. These networks often span across disciplines, geographic boundaries, and sectors, enabling participants to share resources, expertise, and data to address common research goals more effectively than they could individually.

Research Collaboration Network

Improving the performance and application of Large Language Models

Author Amanda Kau (ORCID: 0009-0004-4949-9284) Large language models (LLMs) like GPT-4, the engine of products like ChatGPT, have taken centre stage in recent years due to their astonishing capabilities. Yet, they are far from perfect.

Understanding Retrieval Pitfalls: Challenges Faced by Retrieval Augmented Generation (RAG) models

Exploring the Potential of Temporal Feature-Logic Embedding (TFLEX) in Complex Query Resolution

Author · Vaibhav Khobragade (

 ORCID:

0009–0009–8807–5982)

 Introduction

Artificial intelligence (AI) and knowledge representation in the field of temporal knowledge graphs are rapidly gaining interest.

Harnessing Temporal Dynamics: Advanced Reasoning using Temporal Knowledge Graphs

Exploring innovative Strategies in Combating Misinformation with Enhanced Multimodal Understanding Author Wenyi Pi (

 ORCID

: 0009–0002–2884–2771) Introduction Misinformation refers to false or inaccurate information that is often given to someone in a deliberate attempt to make them believe something that is not true. This has a significantly negative impact on public health, political stability and social trust and harmony.

Multimodal Large Language Models for Misinformation Detection and Reasoning

The Three Oldest Pillars of NLP

Author Dhruv Gupta (

 ORCID

: 0009–0004–7109–5403) Introduction Natural Language Processing (NLP) has almost become synonymous with Large Language Models (LLMs), Generative AI, and fancy chatbots. With the ever-increasing amount of textual data and exponential growth in computational knowledge, these models are improving every day.

RNNs vs GRUs vs LSTMs

A Unified and Collaborative Framework for LLM Author · Qingqin Fang (

 ORCID:

0009–0003–5348–4264) Introduction In today’s rapidly evolving field of artificial intelligence, large language models (LLMs) are demonstrating unprecedented potential. Particularly, the Retrieval-Augmented Generation (RAG) architecture has become a hot topic in AI technology due to its unique technical capabilities.

RAG 2.0 is Coming?

Bridging Human Perception and AI’s Future: The Convergence of Visual Understanding and Semantic Networks Author · Vaibhav Khobragade (

 ORCID:

0009–0009–8807–5982)

 Introduction

The fusion of Vision-Language Models (

 VLMs

), Generative Models, and Knowledge Graphs (

 KGs

) is reshaping how artificial intelligence (AI) understands and interacts with the world.

Tuning Vision-Language Models and Generative Models with Knowledge Graph

Understanding the Power and Applications of Natural Language Processing

 Author

Dhruv Gupta (

 ORCID:

0009–0004–7109–5403) Introduction We are living in the era of generative AI. In an era where you can ask AI models almost anything, they will most certainly have an answer to the query. With the increased computational power and the amount of textual data, these models are bound to improve their performance.

NLU vs. NLG: Unveiling the Two Sides of Natural Language Processing

Understanding how RNNs work and its applications


 Author

Wenyi Pi (

 ORCID

: 0009–0002–2884–2771) Introduction In the ever-evolving landscape of artificial intelligence (AI), bridging the gap between humans and machines has seen remarkable progress. Researchers and enthusiasts alike have tirelessly worked across numerous aspects of this field, bringing about amazing advancements.

An Introduction to Recurrent Neural Networks (RNNs)

Exploring the Boundaries of Creativity and Responsibility in the Age of AI-Driven Media

 Author

Vaibhav Khobragade (

 ORCID:

0009–0009–8807–5982)

 Introduction

In 2024, the discipline of Generative AI takes a big step forward with the launch of revolutionary models that convert text into dynamic films, altering the landscape of digital content creation.

Generative AI’s Leap into Video in 2024 and its Ethical Horizon

Addressing Misinformation, Bias, and Privacy Concerns

Author:

 Wenyi Pi (ORCID:


 0009–0002–2884–2771


 )

Recently, the spread of false or misinformation has become an important concern for people all over the world, especially in the era of big data and multimedia, where people are faced with a flood of information that makes it difficult to determine its authenticity.

The Ethical and Social Impact of Artificial Intelligence

Latest findings for the use of Knowledge Graph in the field of QA in multiple research directions Author: Xuzeng He (

 ORCID:

0009–0005–7317–7426) Question Answering (QA), the ability to interact with data using natural language questions and obtaining accurate results, has been a long-standing challenge in computer science dating back to the 1960s.

Question Answering enhanced by Knowledge Graph

Unlocking the power of knowledge graphs in research catalogues: A deep dive into OpenAlex


 Author:


 Dhruv Gupta (ORCID:


 0009–0004–7109–5403


 )

Clive Humby, in 2006 rightly said, “Data is the new oil”. With data being present everywhere, it has never been more valuable.

Knowledge Graphs: Deep Dive into OpenAlex

Techniques to integrate Knowledge Graphs into Language Models

 Author

Amanda Kau

 (ORCID:


 0009–0004–4949–9284


 )

Introduction Both knowledge graphs (KGs) and pre-trained language models (PLMs) have gained popularity due to their ability to comprehend world knowledge and their broad applicability.

A Deep Dive Into Knowledge Graph Enhanced Pre-trained Language Models

Automated Knowledge Graph Construction with Large Language Models — Part 2

 Harvesting the Power and Knowledge of Large Language Models

Author Amanda Kau (

 ORCID

:

 0009–0004–4949–9284

) Introduction Knowledge graphs (KGs) are a structured representation of data in a graphical format, in which entities are represented by nodes and are connected by edges representing relationships

Automated Knowledge Graph Construction with Large Language Models — Part 2

An improvement architecture superior to the Transformer, proposed by Meta Author · Qingqin Fang (

 ORCID:

0009–0003–5348–4264) Introduction Recently, researchers from Meta and the University of Southern California have introduced a model called Megalodon. They claim that this model can expand the context window of language models to handle millions of tokens without overwhelming your memory.

The longer the context, the better? Unlimited Context Length in Megalodon

Latest findings in pre-training graphs and using them for link recommendation Author · Xuzeng He (

 ORCID:

0009–0005–7317–7426) Introduction A graph, in short, is a description of items linked by relations, where the items of a graph are called nodes (or vertices) and their relations are called edges (or links). Examples of graphs can include social networks (e.g. Instagram) or knowledge graphs (e.g. Wikipedia). In Instagram

Working with Graphs: Pre-training and Application

Unlocking the Future of AI: The Transformative Journey of Large Language Models Author · Vaibhav Khobragade (

 ORCID:

0009–0009–8807–5982)

 Introduction

Human language development is innate and evolves throughout life. Machines lack this ability to evolve without advanced AI algorithms.

The Journey of Large Language Models: Evolution, Application, and Limitations

Understanding Knowledge Networks From A Graph Perspective


 Author:


 Amanda Kau (ORCID:


 0009–0004–4949–9284


 )

Since 2020, over ten million scholarly articles have been published annually. To put that into perspective, say all ten million articles were released on the first day of the year.

Uncovering The Secrets Behind The Most Influential Scholarly Publications With AceMap

Understanding the Balance between Internal Knowledge and External Sources

Author Qingqin Fang (

 ORCID:

0009–0003–5348–4264) Introduction Previous research often emphasized the limitations of LLM’s information acquisition pathways, focusing on enhancing its capabilities in this regard.

When RAG and LLM Conflict: Who Will AI Listen to?

Solutions to Enhance LLM Performance in Long Contexts Author · Qingqin Fang (

 ORCID:

0009–0003–5348–4264) Introduction In the era of AI breakthroughs, large language models (LLMs) are not just advancements; they are revolutions, transforming how we interact with technology, from casual conversations with chatbots to the intricate mechanisms behind sophisticated data analysis tools.

Navigating the Long Context Conundrum: Challenges in Language Models’ Information Processing

Supervised Fine-tuning, Reinforcement Learning from Human Feedback and the latest SteerLM Author · Xuzeng He (

 ORCID:

0009–0005–7317–7426) Introduction Large Language Models (LLMs), usually trained with extensive text data, can demonstrate remarkable capabilities in handling various tasks with state-of-the-art performance. However, people nowadays typically want something more personalised instead of a general solution.

Fine-tuning Large Language Models: A Brief Introduction

An Overview of Constructing a Knowledge Graph Author · Qingqin Fang (

 ORCID:

0009–0003–5348–4264) 1. Introduction 1.1 What is a Knowledge Graph Knowledge Graphs are structured semantic knowledge bases used to rapidly describe concepts and their relationships in the physical world.

Unlocking Intelligence: The Journey from Data to Knowledge Graph

Author Dhruv Gupta (

 ORCID

: 0009–0004–7109–5403) Introduction Large Language Models (LLMs) have become the new face of Natural language processing (NLP). With their generative power and ability to comprehend human language, the human reliance on these models is increasing every day. However, the LLMs have been known to hallucinate and thus produce wrong outputs.

Prompt engineering: A Way to Smartly Use AI

Latest effort in assessing the security of the code generated by large language models Author · Xuzeng He (

 ORCID:

0009–0005–7317–7426) Introduction With the surge of Large Language Models (LLMs) nowadays, there is a rising trend among developers to use Large Language Models to assist their daily code writing. Famous products include GitHub Copilot or simply ChatGPT.

Large Language Models for Code Writing: Security Assessment

Research Graph

Google’s Groundbreaking AI Model Explained
 


 Introduction

In the rapidly evolving landscape of artificial intelligence, Google continues to stand at the forefront, consistently pushing the boundaries of what these technologies can achieve. One of their latest advancements, GEMMA2, represents a significant leap in AI capabilities.

What is Gemma2?

Stories by Research Graph on Medium

#### Bridging Human Perception and AI's Future: The Convergence of Visual Understanding and Semantic Networks

<figure>
<img src="https://cdn-images-1.medium.com/max/900/0*CKgpNGbkidSp4EsM" />
<figcaption>Integrating Vision, Language, and Knowledge in AI by ChatGPT
(DALL-E 2).</figcaption>
</figure>

### Author

· Vaibhav Khobragade (**ORCID:**
[0009--0009--8807--5982](https://orcid.org/0009-0009-8807-5982))

### **Introduction**

The fusion of Vision-Language Models (**VLMs**), Generative Models, and
Knowledge Graphs (**KGs**) is reshaping how artificial intelligence (AI)
understands and interacts with the world. For example, *Automatic Image
Description for Visually Impaired Users* exists in which AI can generate
accurate and detailed descriptions of images, aiding visually impaired
users in understanding their content.

VLMs integrate visual and textual information, enabling tasks like image
captioning, while Generative Models create new, diverse content such as
generating text, video, and images. KGs, with their structured
representation of real-world entities and relationships, enhance these
models by providing deep, contextual insights. This combination unlocks
more accurate, relevant, and contextually rich AI capabilities, from
improved search engines to creative content generation, making
technology more intuitive and closer to human-like understanding and
creativity.

### **Knowledge Graphs**

A knowledge graph (KG) is a structured representation of real-world
entities and their interrelationships, encapsulating complex information
in a graph-structured form where nodes represent entities, and edges
denote relationships. For instance, 'Leonardo DiCaprio' and 'Inception'
are nodes linked by an edge representing his role in the film. For more
details, refer
to [this](https://medium.com/@researchgraph/enhancing-language-models-the-role-of-knowledge-graph-augmentation-in-overcoming-llm-challenges-c232b5e9328f).

### **Vision-language Models**

Vision-language models (VLMs) are tools that combine what they see in
pictures with written words to both understand and create new content.
They are good at making captions for pictures and answering questions
about what's in an image. DALL-E is a prime example of VLMs. It
showcases the capabilities of VLMs by generating images from textual
descriptions, combining the fields of computer vision and natural
language processing. For instance, if given the prompt "A futuristic
city at sunset,". It generates the following image

<figure>
<img src="https://cdn-images-1.medium.com/max/847/0*QFKhHuhRExW6wOSw" />
<figcaption>A futuristic city at sunset by DALL-E.</figcaption>
</figure>

### **Generative Models**

Generative models, often part of the broader VLM framework, are capable
of creating new content, such as images or text, by learning the
underlying distribution of training data, enabling applications like
text-to-image synthesis (as in the above instance) and style transfer.

### **How knowledge graphs work with LLMs**

#### **Tuning Vision-Language Models With Dual Knowledge Graph**

Previous techniques of tuning Vision-Language models like
[CLIP-Adapter](https://arxiv.org/abs/2110.04544),
[TaskRes](https://arxiv.org/abs/2211.10277), and
[Tip-Adapter](https://arxiv.org/abs/2111.03930) have two main issues
associated with them. First, they focus on adapting models using
knowledge from a single modality (either text or images) which means
that they don't fully utilise the **relationship** between the two
modalities.

For example, consider an image of a cat and a dog playing together.
Focusing solely on the image might lead the model to identify individual
objects (cat, dog) but miss the interaction between them. Likewise,
focusing solely on the text description might not capture the visual
details of the scene.

Second, they do not fully leverage the structured knowledge of
relationships between different concepts, particularly in scenarios with
limited data that can lead to several issues such as suboptimal
solutions, bias towards partial attributes, and inefficient transfer and
generalisation.

For example, if the VLM encounters an image of a chef holding a knife,
it might struggle to understand the specific action (chopping) without
additional knowledge about the relationship between chefs, knives, and
food preparation.

The new method of using Dual Knowledge Graphs addresses the limitations
of current VLM tuning methods, particularly under scenarios with limited
data. The core innovation is the **GraphAdapter**, a strategy that
leverages dual KGs --- separate but interconnected graphs for textual
and visual knowledge --- to enrich the model's understanding and
generation capabilities.

By creating two interconnected KGs --- one for text and another for
visual information --- the GraphAdapter enables VLMs to draw on a richer
set of relationships and semantic understandings. This dual-graph
approach allows the model to better capture the nuances of how objects
and concepts are related across visual and textual data, leading to more
accurate and context-aware outputs.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*VTIP54a3ARo94oQe" />
<figcaption>Comparison between (a) Zero-shot CLIP, (b) CLIP-Adapter (c)
TaskRes, and (d) proposed GraphAdapter by (Li et al.,
2023) Article link: <a
href="https://arxiv.org/abs/2309.13625">https://arxiv.org/abs/2309.13625</a></figcaption>
</figure>

In the above image, while the output classifications might look similar
at a glance, the essence of GraphAdapter's innovation is not just in the
accuracy of classification but in how it achieves this result. By
leveraging dual Knowledge Graphs, GraphAdapter can potentially offer
richer contextual understanding and generalisation capabilities,
especially in "low-data regimes" or when faced with nuanced,
fine-grained distinctions between classes. This approach marks a
significant shift from previous methods, aiming to deeply integrate
cross-modal knowledge and structured relationships into the adaptation
process for Vision-Language Models.

A direct comparison of the results across different methods, including
Zero-shot CLIP, CLIP-Adapter, TaskRes, and their proposed GraphAdapter,
has been shown in the image below. It shows how GraphAdapter
consistently outperforms the baseline methods across different numbers
of shots, underscoring the effectiveness of integrating dual Knowledge
Graphs for structured knowledge exploitation.

#### **Working**

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*DyWA5ETiGg7IC72e" />
<figcaption>The pipeline of GraphAdapter, which is composed of the dual
knowledge graph and CLIP (Contrastive Language-Image Pre-training )based
image classification pipeline by (Li et al., 2023) Article link: <a
href="https://arxiv.org/abs/2309.13625">https://arxiv.org/abs/2309.13625</a></figcaption>
</figure>

1. **Starting point (Input Images and Texts):** The process starts with
 images and their corresponding texts. These could be things like
 photos of animals along with descriptions or labels.
2. **Transformation (Text and Visual Encoders):** The text descriptions
 are processed by a Text Encoder, which transforms the text into a
 vector by word embedding like word2vec with which the computer can
 work more efficiently. Similarly, the Visual Encoder processes the
 images, turning them into matrices.
3. **Mapping relationships (Dual Knowledge Graphs Creation):** *For
 text,* a Textual-Subgraph is created. It's like making a map that
 shows how different words or phrases are related based on their
 meanings. *For visuals*, a visual sub-graph aims to capture and
 model the relationships between different visual elements and
 concepts within the images. This involves understanding what the
 image contains (e.g., objects, scenes, actions) and how these
 elements are related to each other in terms of their visual and
 contextual relationships.
4. **Refining Connections (Convolutional Networks (GCNs)):** These are
 special tools that help to blend and refine the information in our
 text and visual maps (the subgraphs), making sure the connections
 and relationships are as accurate and helpful as possible. K and d
 are the number of classes and the dimension of textual/visual
 features.
5. **Fusion and Adjustment (GraphAdapter):** This is the heart of the
 process. It takes the refined maps of texts and visuals and combines
 them, ensuring that the final output makes sense both visually and
 textually. It's like making sure the description "a big red apple"
 matches with pictures of apples, not bananas.
6. **Final Output:** The final step produces adjusted or enhanced text
 and image features, ensuring that images match their descriptions
 accurately and vice versa. This can be used to make better image
 recognitions, more accurate image descriptions, and so on.

#### **Performance Comparison**

The model evaluates its approach, GraphAdapter, on few-shot learning
tasks using 11 datasets and observes significant improvements over
existing methods, particularly for tasks with limited examples (1- to
16-shot settings). It also examines how well GraphAdapter generalises
unseen data by testing it across four diverse datasets. The findings
reveal that GraphAdapter not only excels in adapting to new tasks with
few examples but also demonstrates strong generalisation capabilities,
outperforming other state-of-the-art methods in most scenarios.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*cJPS40vwRLarkUKU" />
</figure>

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*WHXe9ihe-l-KP_JY" />
<figcaption>The performance comparison of our GraphAdapter with the
state-of-the-art methods on few-shot learning, including
1-/2-/4-/8-/16-shots on 11 benchmark datasets by (Li et al., 2023)
Article link: <a
href="https://arxiv.org/abs/2309.13625">https://arxiv.org/abs/2309.13625</a></figcaption>
</figure>

#### **Knowledge Graph Embeddings (KGEs) can enhance generative models**

Instead of using complex formulas, researchers propose a new way to make
KGEs like [COMPLEX](https://arxiv.org/abs/1606.06357),
[CP](https://arxiv.org/abs/1806.07297),
[RESCAL](https://www.researchgate.net/publication/221345089_A_Three-Way_Model_for_Collective_Learning_on_Multi-Relational_Data),
and [TUCKER](https://arxiv.org/abs/1901.09590) generate new
relationships between concepts. They achieve this by:

1. Transforming existing KGE models into circuits: These circuits
 involve calculations that consider multiple possibilities to arrive
 at different probabilities.
2. Fine-tuning the calculations within the circuits: This ensures the
 final output is always a valid probability (between 0 and 1).

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*zXNiB9141DtgpXcm" />
<figcaption>Explores how certain Knowledge Graph Embedding (KGE) models,
like COMPLEX, CP, RESCAL, and TUCKER, can be adapted into effective
generative models for triples. These models’ scoring functions can be
represented as circuits (highlighted in lilac). To convert these into
valid probabilistic circuits (PC, shown in orange) that model the
probability distribution over triples, we either have to ensure the
activations are non-negative (indicated in blue) or square the
activations (marked in red) by (Niepert et al., 2023) Article link: <a
href="https://arxiv.org/abs/2305.15944">https://arxiv.org/abs/2305.15944</a></figcaption>
</figure>

#### Working

Knowledge Graph Embedding (**KGE**) converts models such as COMPLEX, CP,
RESCAL, and TUCKER into generative models by reinterpreting them as
**circuits**, or structured computational graphs. This reinterpretation
allows for efficient processes like marginalisation, which is crucial
for understanding the distribution of certain variables within a larger
set. Marginalisation is a fundamental concept in probability theory and
statistics. When you have a probability distribution over multiple
variables, marginalisation involves summing (or integrating, in the case
of continuous variables) the probabilities of all possible values of the
variables you are not interested in, to obtain the probability
distribution of the variables you are interested in.

To make these models generative, their outputs are modified through
**non-negative restriction** or **squaring**, ensuring the outputs can
represent probabilities. These adapted circuits, named Generative KGE
Circuits (**GeKCs**), can then generate new triples for knowledge graphs
by efficiently sampling from their modeled probability distributions.
Moreover, GeKCs are designed to integrate logical constraints directly,
ensuring that all generated or predicted triples are logically
consistent, such as adhering to rules that specify how entities can or
cannot relate to each other. This approach not only retains the models'
link prediction capabilities but also enhances their applicability by
enabling them to handle large graphs efficiently and generate new,
plausible triples that respect predetermined logical constraints.

Consider the following scenario as an example:

**Scenario:** Imagine a knowledge graph containing information about
people, movies, and their genres. We want to use a GeKC Model to predict
missing information.

**Logical Constraint:** One logical constraint could be: "A person
cannot act in a movie that is released before their date of birth."

**Existing Triples:**

- (Tom Hanks, acted_in, Forrest Gump)
- (Tom Hanks, date_of_birth, 1956)

**Prediction Task:** Predict the release date of "Forrest Gump".

**Without Logical Constraints:** The model might simply predict any
release date based on statistical patterns in the data. This could lead
to illogical predictions like "Forrest Gump" being released in 1954,
which would contradict Tom Hanks' date of birth.

**With Logical Constraints:** The model with logical constraints would
consider the "date_of_birth" information and the constraint mentioned
earlier. This would eliminate the possible release date predictions of a
date before 1956 (Tom Hanks' date of birth).

#### **Evaluation of the model**

The empirical evaluation demonstrates that GeKCs are competitive with
traditional Knowledge Graph Embeddings (KGEs) for link prediction tasks,
showing that the generative approach does not compromise on accuracy.
Furthermore, incorporating domain constraints directly into the model
significantly enhances predictions by ensuring they adhere to logical
rules, improving reliability and relevance. Lastly, the quality of
triples generated by GeKCs is evaluated through a novel metric,
revealing that GeKCs can efficiently produce high-quality, new triples
that are consistent with the knowledge graph's existing information,
thereby enriching the graph with plausible connections.

#### **Potential Applications**

**Improved Question Answering Systems:** This new method could allow the
virtual assistants to not only find existing answers within the
knowledge graph but also generate new, logical relationships between
concepts. This could lead to more comprehensive and informative answers,
even for complex or open-ended questions.

**Drug Discovery and Material Science:** Researchers could use these
enhanced KGEs to explore potential relationships between different
chemicals or biological entities. By generating new, plausible
connections based on existing knowledge, the system could help identify
promising candidates for new drugs or materials with desired properties.

**Recommendation Systems and Market Analysis:** KGEs are already used by
some recommendation systems to understand user preferences and suggest
relevant products or services. This new approach could allow the system
to go beyond simply recommending existing items. It could potentially
generate new product ideas or identify previously unexplored market
connections based on the knowledge graph.

### **Conclusion**

The integration of Knowledge Graph Embeddings (KGEs) with
Vision-Language Models (VLMs) and Generative Models, particularly
through the innovative GraphAdapter and generative KGE circuits (GeKCs),
represents a significant leap forward in AI's ability to understand and
generate complex content. This approach not only enhances the models'
predictive accuracy and contextual understanding but also ensures
logical consistency in generated content. Moreover, it promises
scalability and efficiency, making it a robust solution for enriching
and expanding knowledge graphs in various applications.

### **References**

- Niepert, M., Garcia-Duran, A., & Onoro-Rubio, D. (2023). How to turn
 your knowledge graph embeddings into generative models via
 probabilistic circuits. arXiv. <https://arxiv.org/abs/2305.15944>
- Li, X., Lian, D., Lu, Z., Bai, J., Chen, Z., & Wang, X. (2023).
 GraphAdapter: Tuning vision-language models with dual knowledge graph.
 arXiv. <https://arxiv.org/abs/2309.13625>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0e2b4af3aff){width="1"
height="1"}