Wendi Fan

#### Let's explore GR-2 together. {#lets-explore-gr-2-together.}

<figure>
<img
src="https://cdn-images-1.medium.com/max/654/0*xxkw6kbWp_pCr3x3.jpg" />
<figcaption>GR-2 logo, downloaded from paper “GR-2: A Generative
Video-Language-Action Model with Web-Scale Knowledge for Robot
Manipulation”</figcaption>
</figure>

### Author

- [Wendi Fan](https://www.linkedin.com/in/wendi-fan-265996310/)
  (**ORCID: 0000--0003--0284--9166**)

### Introduction

GR-2 is a cutting-edge generative model designed for versatile and
generalisable robot manipulation, developed by the Robotics Research
Team at ByteDance Research. It represents a significant step forward in
robotics, enabling robots to execute a wide range of manipulation tasks
in various environments. GR-2 leverages video generation and action
prediction to empower robots with strong generalisation capabilities,
allowing them to perform complex tasks even in unseen scenarios.

At its core, GR-2 aims to solve one of the key challenges in modern
robotics: how to make robots adapt to diverse tasks and environments
with minimal retraining. Traditionally, robots require task-specific
programming, which limits their flexibility in real-world applications.
GR-2 addresses this by training on a massive dataset of internet videos,
capturing the dynamics of the world and translating that knowledge into
robotic actions. By pre-training on 38 million video clips and over 50
billion tokens, followed by fine-tuning on robot trajectories, GR-2 can
generalise across a wide range of tasks and environments, achieving a
high success rate in complex manipulation scenarios.

### Training Process

GR-2's training process is divided into two key stages:

1.  **Pre-training on Video Generation**: The first stage involves
    training GR-2 to predict future states in videos, capturing the
    temporal dynamics and semantics from large-scale video datasets.
    This training is essential for GR-2 to learn how objects interact
    and how environments evolve over time. The dataset used for
    pre-training is highly diverse, consisting of human activities in
    contexts like households, workplaces, and outdoor environments. The
    model learns to generate the next video frame based on a given
    textual description and the current frame, providing it with a
    strong foundation for subsequent robot learning.
2.  **Fine-tuning on Robot Data**: In the second stage, GR-2 is
    fine-tuned using robot trajectory data to predict both action
    trajectories and corresponding video sequences. Unlike the video
    data used in pre-training, robot data often involves multiple camera
    angles and viewpoints, which GR-2 is designed to handle gracefully.
    This fine-tuning stage optimises the model to generate accurate
    action trajectories in Cartesian space, allowing robots to perform
    tasks precisely. The model's Whole-Body Control (WBC) algorithm
    ensures that the robot follows the planned trajectories smoothly,
    incorporating real-time motion tracking and trajectory optimisation.

<figure>
<img
src="https://cdn-images-1.medium.com/max/864/0*XWf3O2QE3XtxuXVI.jpg" />
<figcaption>Pre-training Dataset downloaded from paper “GR-2: A
Generative Video-Language-Action Model with Web-Scale Knowledge for
Robot Manipulation”</figcaption>
</figure>

By first conducting pre-training on video generation, the GR-2 model can
fully leverage the general knowledge obtained from large-scale video
data, significantly reducing the need for extensive robot-specific data.
**After completing video generation pre-training, the model learns how
to predict future visual states. During fine-tuning, these visual
predictions aid the model in more effectively planning the robot's
action sequences.** Since the model already possesses the ability to
anticipate environmental changes, it can swiftly generate accurate
action trajectories during real-world operations, ensuring smoother and
more efficient execution of robotic tasks.

<figure>
<img
src="https://cdn-images-1.medium.com/max/832/0*Tls2RejbBUiVmbfj.jpg" />
<figcaption>GR-2 undergoes two stages of training: video generation
pre-training and robot data fine-tuning, downloaded from paper “GR-2: A
Generative Video-Language-Action Model with Web-Scale Knowledge for
Robot Manipulation”</figcaption>
</figure>

### Benchmark Test Results

GR-2 was evaluated across multiple benchmarks to demonstrate its
capability:

### Multi-task Learning

**Multi-task Learning**: GR-2 was tested on 105 different manipulation
tasks, achieving an average success rate of 97.7%. These tasks involved
a variety of complex operations such as picking, placing, uncapping,
pressing, and more. GR-2 excelled even in challenging settings like
environments with distractors, unseen backgrounds, and unseen tasks.
This high performance highlights its ability to handle real-world
variability and adapt to new situations with minimal additional
training.

<figure>
<img
src="https://cdn-images-1.medium.com/max/894/0*mc9sB64bkApAOd2K.jpg" />
<figcaption>Success rates of multi-task learning, downloaded from paper
“GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge
for Robot Manipulation”</figcaption>
</figure>

<figure>
<img
src="https://cdn-images-1.medium.com/max/600/0*IMnCxOJbCTryy9zc.gif" />
<figcaption>Performance of this model, downloaded from paper “GR-2: A
Generative Video-Language-Action Model with Web-Scale Knowledge for
Robot Manipulation”</figcaption>
</figure>

### End-to-End Bin-Picking

**End-to-End Bin-Picking**: In an industrial context, GR-2 was tested on
a bin-picking task involving 122 objects, 55 of which were seen during
training and 67 unseen. GR-2 outperformed its predecessor GR-1 by a
large margin, maintaining a success rate comparable between seen and
unseen objects. It also managed cluttered environments without
significant degradation in performance, showcasing its potential for
practical applications in industries like logistics and manufacturing.

<figure>
<img
src="https://cdn-images-1.medium.com/max/445/0*qkUI86FcMOnU8LTE.jpg" />
<figcaption>Success rates of end-to-end bin-picking, downloaded from
paper “GR-2: A Generative Video-Language-Action Model with Web-Scale
Knowledge for Robot Manipulation”</figcaption>
</figure>

<figure>
<img
src="https://cdn-images-1.medium.com/max/777/0*T81jWjaPDxfSveQi.jpg" />
<figcaption>Objects used in the bin-picking experiments, downloaded from
paper “GR-2: A Generative Video-Language-Action Model with Web-Scale
Knowledge for Robot Manipulation”</figcaption>
</figure>

### CALVIN Benchmark

**CALVIN Benchmark**: GR-2 established a new state-of-the-art on the
CALVIN benchmark, a simulation environment designed for long-horizon,
language-conditioned robot manipulation tasks. GR-2 outperformed five
leading models, including RT-1 and HULC, in terms of success rates and
average task completion length. It demonstrated the ability to complete
sequences of tasks in a row, further proving its versatility and
efficiency.

<figure>
<img
src="https://cdn-images-1.medium.com/max/832/0*MuYJC_gkyh0PnHS3.jpg" />
<figcaption>CALVIN benchmark results, downloaded from paper “GR-2: A
Generative Video-Language-Action Model with Web-Scale Knowledge for
Robot Manipulation”</figcaption>
</figure>

CALVIN Benchmark: The CALVIN Benchmark consists of 34 different
manipulation tasks, where the robot needs to execute these tasks
sequentially after receiving language instructions. These tasks involve
complex multi-step operations, with the core evaluation focusing on:

- **Language-conditioned manipulation**: Whether the robot can
  understand and execute tasks based on natural language instructions.
- **Long-horizon task sequences**: The robot needs to maintain accurate
  operations across multiple tasks while understanding and handling a
  continuous chain of tasks.
- **Generalisation ability to tasks**: Whether the robot can efficiently
  complete tasks in previously unseen environments and with unfamiliar
  instructions.

Thus, the CALVIN Benchmark is a comprehensive standard used to assess a
robot's ability in task generalisation, long-horizon task handling, and
complex task planning under language-conditioned manipulation.

### Auto-Regressive Video Generation

A unique aspect of GR-2 is its use of auto-regressive video generation
to aid in action prediction. By generating future video frames, the
model can act as a planner, simulating how a task will unfold before
executing it. This capability allows GR-2 to improve action accuracy by
ensuring that predicted trajectories align with expected visual
outcomes. In experiments, the generated videos closely matched
real-world rollouts, indicating that the model can effectively "replay"
the predicted action based on the generated visual trajectory. This
iterative improvement of video generation translates directly into more
accurate and reliable action prediction.

<figure>
<img
src="https://cdn-images-1.medium.com/max/879/0*DY6EJRNSbU_MA6pj.jpg" />
<figcaption>Pred and GT results of testings, downloaded from paper
“GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge
for Robot Manipulation”</figcaption>
</figure>

In this image, **Pred** and **GT** represent the following:

- **Pred**: This stands for **Predicted**, referring to the video
  sequence predicted by the GR-2 model. It shows how the model
  anticipates future actions and corresponding visual outputs based on
  the input. For example, in this figure, the model predicts the robot's
  actions for tasks such as "open the oven" and "press the
  toaster switch."
- **GT**: This stands for **Ground Truth**, representing the actual
  video frames of the robot performing these tasks in real life. This is
  the real data of how the robot executed these tasks in reality.

The purpose is to demonstrate how well the model's predictions (Pred)
align with real-world execution (GT), showcasing GR-2's ability to
predict actions through auto-regressive video generation. This
highlights the model's accuracy in forecasting future visual states.

### Real-World Application and Performance

#### 1. Accurate grasping capability {#accurate-grasping-capability}

The GR-2 model excels in precise object manipulation, including highly
accurate grasping capabilities. It can effortlessly handle tasks such as
grasping cylindrical glass bottles or even very thin items, ensuring
reliable performance even with challenging objects that require careful
handling. This makes it a powerful tool for intricate and precise
robotic tasks in various real-world applications.

<figure>
<img
src="https://cdn-images-1.medium.com/max/500/0*gSwe8rQ3gCzYeknH.gif" />
<figcaption>Accurate grasping capability</figcaption>
</figure>

<figure>
<img
src="https://cdn-images-1.medium.com/max/480/0*HvGdXE6SRguHrbE2.gif" />
<figcaption>Grasping items while giving new items</figcaption>
</figure>

<figure>
<img
src="https://cdn-images-1.medium.com/max/480/0*q44O76KFYMWjzqH8.gif" />
<figcaption>Grasping cylindrical glass bottles</figcaption>
</figure>

<figure>
<img
src="https://cdn-images-1.medium.com/max/500/0*pePdGTdCE4mFlc_e.gif" />
<figcaption>Grasping very thin items</figcaption>
</figure>

### 2. Accurate grasping capability {#accurate-grasping-capability-1}

The GR-2 model is also capable of assisting humans with a wide range of
everyday tasks, from preparing breakfast to making coffee. Its advanced
capabilities allow it to perform complex activities with precision,
making it an ideal solution for automating routine tasks and supporting
daily activities with ease.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*JqUFk4x-hHwjEiAg.jpg" />
<figcaption>Making breakfast for human</figcaption>
</figure>

<figure>
<img
src="https://cdn-images-1.medium.com/max/480/0*WZOV0yjsWrrrRgzv.gif" />
<figcaption>Making breakfast for human</figcaption>
</figure>

### 3. Sorting tasks {#sorting-tasks}

The GR-2 model can also perform sorting tasks with high efficiency. For
example, it can sort fruits on a table and place them into different
bowls as needed. This ability to organise and classify objects makes
GR-2 a versatile tool for managing sorting tasks in both everyday and
industrial scenarios.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*ZSp0LpT_MNlFWPxl.jpg" />
<figcaption>Before sorting</figcaption>
</figure>

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/0*PhqJ61yCuyN9iCrB.jpg" />
<figcaption>After sorting</figcaption>
</figure>

In the past, many AI models primarily focused on large language models
(LLMs) or image generation capabilities, with their influence largely
confined to the digital realm, limited to interactions within computers
or operating systems. They lacked the ability to directly affect the
physical world. However, with this model, they are taking a significant
step forward. GR-2 goes beyond the boundaries of traditional AI by
bridging the gap between digital intelligence and real-world action.
This model not only excels in understanding and generating language, but
also has the capability to manipulate and interact with the physical
environment.

In the future, this could pave the way for AI-driven generalist robots,
capable of not only engaging in conversation with humans but also
solving real-world problems --- whether it's managing tasks in a
household or assisting in industrial operations. This vision, once
confined to science fiction films, could soon become a reality. GR-2
represents a glimpse of what the future of AI holds: a world where
intelligent robots enhance our lives in both virtual and physical
spaces. This is not just an advancement in robotics; it's a fundamental
shift in how AI can impact and transform the world around us.

### Conclusion

GR-2 represents a significant leap forward in the field of robotics,
combining cutting-edge video generation and action prediction to enable
robots to interact with and manipulate the physical world in complex
ways. Its ability to generalise across tasks, environments, and unseen
scenarios --- combined with its precise handling of objects and
real-world tasks --- makes it a powerful and versatile tool for both
industrial and everyday applications. From accurate grasping to sorting
tasks, GR-2 demonstrates remarkable adaptability and efficiency,
bringing us closer to the future of AI-driven robots capable of
performing a wide range of human tasks. This model not only extends the
capabilities of traditional AI models but also bridges the gap between
digital intelligence and physical action, offering a glimpse into a
future where intelligent robots can work alongside humans to solve
real-world problems.

### References

- Cheang, C.L., Chen, G., Jing, Y., Kong, T., Li, H., Li. Y., Liu, Y.,
  Wu, H., Xu, J., Yang, Y., Zhang, H., Zhu, M. (2024). GR-2: A
  Generative Video-Language-Action Model with Web-Scale Knowledge for
  Robot Manipulation. DOI: 10.48550/arXiv.2410.06158.

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d920b36afab3){width="1"
height="1"}


<strong>
 Let’s explore GR-2 together.
</strong>
<strong>
 Author
</strong>
Wendi Fan (
<strong>
 ORCID: 0000–0003–0284–9166
</strong>
)
<strong>
 Introduction
</strong>
GR-2 is a cutting-edge generative model designed for versatile and generalisable robot manipulation, developed by the Robotics Research Team at ByteDance Research.

GR-2: A Generative Video-Language-Action Model for Robot Manipulation

Hans Zauner

**A new "multi-modal" data set on arm motion control, [published
recently in *GigaScience*](https://doi.org/10.1093/gigascience/giab043),
is an important contribution to develop robotic prosthetic devices and
other tools at the interface between human and machine.**

  
Robotic arms and other devices that support limb movements are getting
ever more sophisticated. New materials and new computational approaches
are paving the path for prosthetic devices that "blend in" with the
human organism. However, to navigate the interface between human and
machine, data is needed on all aspects of arm motion control, both in
healthy and disease conditions.

  
Developers of robotic arms and other human-machine interfaces that
support limb movements need quantitative data that describe how human
limbs operate during everyday tasks, such as lifting a cup of tea to
your lips or turning a page in a book.

## One of the biggest datasets on human motor control available {#one-of-the-biggest-datasets-on-human-motor-control-available .has-normal-font-size .wp-block-heading}

Although these tasks look trivial from the outside, the underlying
kinematics and involvement of diverse muscle groups is a complex affair;
even more if interaction with objects -- an apple, a cup, a page of a
book -- is involved. To really understand what is going on when we reach
for an apple and take a bite, for example, researchers need quantitative
information on arm movements and corresponding activity patterns of
muscles. They also need data from brain activity and other physiological
parameters connected to arm motion control.

<figure class="wp-block-image size-large">
<img
src="http://gigasciencejournal.com/blog/wp-content/uploads/2021/07/upperlimp.jpg"
class="wp-image-4052" loading="lazy" decoding="async"
srcset="http://gigasciencejournal.com/blog/wp-content/uploads/2021/07/upperlimp.jpg 720w, http://gigasciencejournal.com/blog/wp-content/uploads/2021/07/upperlimp-300x244.jpg 300w"
sizes="(max-width: 720px) 100vw, 720px" width="720" height="585" />
</figure>

  
A collaboration coordinated by *[Istituto Italiano di
Tecnologia](https://www.iit.it/)*, with the participation of the
University of Pisa as well as of other scientists across Europe, has now
published one of the biggest datasets on human motor control available,
a "multi-modal" data collection of upper limb movements, both in healthy
and post-stroke conditions (read the *GigaScience* data note
[here](https://doi.org/10.1093/gigascience/giab043)). Corresponding
author Guiseppe Averta says,

"*It represents a previously unmatched source of information for the
research community, because it will enable a deeper analysis not only of
how the motor control works looking at a specific functional domains,
but also on the relationship between them.*"

  
The researchers met under the roof of
"[SoftPro](https://www.softpro.eu/)", an initiative with the goal to
"*study and design soft synergy-based robotics technologies to develop
new prostheses, exoskeletons, and assistive devices for upper limb
rehabilitation*". Averta says the aim is not only to develop new
technology and better performing devices, but also to contribute to
making the technology accessible to real people with real needs:

## Technology for real people with real needs {#technology-for-real-people-with-real-needs .has-normal-font-size .wp-block-heading}

*"It is the result of a strong integration and coordination effort.
U-Limb is now publicly available and published in the open access
journal GigaScience, coherently with the open approach to innovation
that characterized the Project*."

  
To go a bit into the details, the newly published article is based on 91
able-bodied and 65 post-stroke participants. The data combines tracking
of movements (kinematics) with data on muscle activity, heart rate and
brain function. This integration of different data sources is one of the
great strengths of the project, Averta explains:

  
*"It will be possible to study how certain cortical patterns reflect in
muscular activation, and how this influences the coordinate movement to
execute a certain task."*

<figure class="wp-block-image size-large">
<img
src="http://gigasciencejournal.com/blog/wp-content/uploads/2021/07/hand-1.jpg"
class="wp-image-4054" loading="lazy" decoding="async"
srcset="http://gigasciencejournal.com/blog/wp-content/uploads/2021/07/hand-1.jpg 478w, http://gigasciencejournal.com/blog/wp-content/uploads/2021/07/hand-1-291x300.jpg 291w"
sizes="(max-width: 478px) 100vw, 478px" width="478" height="493" />
</figure>

  
To achieve such a multi-modal data collection, the researchers used
three protocols:

1\) **Participants performed a range of daily activities, such as
pointing at something**, grasping the handle of a suitcase, or removing
a tea bag from a cup. During these tasks, limb movements were recorded
on camera, based on defined marker positions, while muscle and brain
activity as well as heart rate were measured as well.

  
2) **Hand grasping was also studied in fMRI experiments.**

  
3) In addition, a device called **VPIT ("virtual peg insertion test")
was used to collect data on the coordination of arm and hand.**

To see how the VPIT works, watch this video from ETH RELAB:

<figure
class="wp-block-embed-youtube aligncenter wp-block-embed is-type-video is-provider-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio">
<div class="wp-block-embed__wrapper">
<div class="iframe">
<div id="player">

</div>
<div class="player-unavailable">
<h1 id="ein-fehler-ist-aufgetreten." class="message">Ein Fehler ist
aufgetreten.</h1>
<div class="submessage">
<a href="https://www.youtube.com/watch?v=TyJyd5uVN68"
target="_blank">Sieh dir dieses Video auf www.youtube.com an</a> oder
aktiviere JavaScript, falls es in deinem Browser deaktiviert sein
sollte.
</div>
</div>
</div>
</div>
</figure>

  
For some of the experiments, a subgroup of the participants wore a "full
body sensor suit", equipped with gyroscopes and accelerometers.  
The full data is openly available via the [Harvard dataverse
repository](https://doi.org/10.7910/DVN/FU3QZ9).

## Use cases for humans and robots {#use-cases-for-humans-and-robots .has-normal-font-size .wp-block-heading}

What are potential use cases for this new data? Averta explains more:

*"For example, a study of the coordination across joints could be used
to inform the design of under-actuated prostheses that can replace
missing limbs with a limited number of motors. On the other side, we can
exploit the knowledge of natural behavior to provide a comparison with
stroke patients, helping clinicians in assessing the pathology's
severity."*

There may even be applications in industrial robotics, for example in
robotic approaches where machines are trained to replicate human
behavior:

"*This would be an asset to develop manipulators, and in general
humanoid robots, that behave as a human would actually do in the same
situation."*

**Read the *GigaScience* Data Note:**

Averta G, Barontini F, Catrambone V, Haddadin S, Handjaras G, Held JPO,
Hu T, Jakubowitz E, Kanzler CM, Kühn J, Lambercy O, Leo A, Obermeier A,
Ricciardi E, Schwarz A, Valenza G, Bicchi A, Bianchi M. U-Limb: A
multi-modal, multi-center database on arm motion control in healthy and
post-stroke conditions. *Gigascience*. 2021 Jun 18;**10**(6):giab043.
doi:
[10.1093/gigascience/giab043](https://doi.org/10.1093/gigascience/giab043).

**Data available here:**

Averta G, Barontini F, Catrambone V, Haddadin S, Handjaras G, Held JPO,
Hu T, Jakubowitz E, Kanzler CM, Kühn J, Lambercy O, Leo A, Obermeier A,
Ricciardi E, Schwarz A, Valenza G, Bicchi A, Bianchi M. U-Limb: A
multi-modal, multi-center database on arm motion control in healthy and
post-stroke conditions. Harvard Dataverse, V3. 2020,
<https://doi.org/10.7910/DVN/FU3QZ9>

The post [Data on arm motion control for the human-machine
interface](http://gigasciencejournal.com/blog/arm-motion-control-data/){rel="nofollow"}
appeared first on
[GigaBlog](http://gigasciencejournal.com/blog){rel="nofollow"}.


<strong>
 A new “multi-modal” data set on arm motion control, published recently in
 <em>
  GigaScience
 </em>
 , is an important contribution to develop robotic prosthetic devices and other tools at the interface between human and machine.
</strong>
Robotic arms and other devices that support limb movements are getting ever more sophisticated.

Messaggi di Rogue Scholar

GR-2: A Generative Video-Language-Action Model for Robot Manipulation

Data on arm motion control for the human-machine interface