Various notes from the Accelerating Devices Lab (X-Dev) of Jülich Supercomputing Centre

Chelsea John

The Supercomputing Conference 2023 took place in Denver, Colorado, from November 12th to 17th. For the

 Women in HPC

workshop, we submitted a paper, which focused on benchmarking different accelerators for AI. The paper was accepted and I was invited to hold a lightning talk to show the work, spun off our OpenGPT-X project.

SC23 WHPC Workshop Paper: OpenGPT-X – Novel Architecture Exploration

Stepan Nassyr

** Poster in institute repository:** http://dx.doi.org/10.34734/FZJ-2023-03437 During the RISC-V Summit Europe 2023


in Barcelona we presented our work generating highly optimized RISC-V and ARM GEMM microkernels for BLIS using a custom software tool.

 1

We presented results on the Fujitsu A64FX processor, the in-development RISC-V

 VEC

processor from the EUPILOT project using an FPGA SDV (RVV 0.7.1, later RVV

Poster: GEMMERATOR – GEMM Kernel Generator

Andreas Herten

Together with our colleagues from Helmholtz-Zentrum Dresden-Rossendorf (HZDR) and in collaboration with HIDA and OpenHackathons, we hosted the

 Helmholtz GPU Hackathon 2023

in Jülich in May.

 I’ve blogged about the event for the
 
 Zweikommazwei
 
 blog of Forschungszentrum;

Helmholtz GPU Hackathon 2023

At the end of May, Forschungszentrum Jülich got a new website. Finally, the group also has a dedicated page there, listing the things we do, the project we are part of, and all the members in the group. And thanks to a photo shooting by Sebastian, we have great portraits and an amazing group picture.

X-Dev Website at fz-juelich.de

Jayesh Badwaik

** Poster in institute repository:** http://dx.doi.org/10.34734/FZJ-2023-04519  At the ISC High Performance Conference 2023 a little while ago, we presented a project poster on the SCALABLE project, embedded at the end of this post. The work was also presented as a paper at the the Computing Frontiers 2023 conference previously.

ISC23 Project Poster: SCALABLE: Scalable Lattice-Boltzmann Leaps to Exascale

Sebastian Achilles

MSA Concept MSA Software Building Blocks Workshop Exercises 1: Hello World! 2. GPU Hello World! 3: CPU-GPU Ping Pong Slides  On May 29, we held a workshop about using the Modular Supercomputing Architecture (MSA) together with project partners from ParTec.

MSA Introduction Workshop

** Poster publication:** http://hdl.handle.net/2128/34532  The ISC High Performance Conference 2023 was held at Hamburg, Germany from 21st May to 25th May. At the conference, we presented a project poster on the OpenGPT-X project, outlining the progress and initial exploration results. The poster was even featured in HPCWire’s May 24 recap of ISC within the AI segment!

ISC23 Project Poster: OpenGPT-X – Training Large Language Models on HPC Systems

For a recent talk at DKRZ in the scope of the natESM project, I created a table summarizing the current state of using a certain programming model on a GPU of a certain vendor, for C++ and Fortran. Since it lead to quite a discussion in the session, I made a standalone version of it with some updates and elaborations here and there.

GPU Vendor/Programming Model Compatibility Table

TL;DR: I held a HPC intro talk. Slides are below.

In MAELSTROM, we connect three areas of science:

 🌍Weather and climate simulation

with

 🤖Machine Learning methods and workflows

using

 📈HPC techniques and resources

. Halfway into the project, we held a boot camp at JSC to teach this Venn diagram to a group of students a few days ago. Some were ML experts, but had never used a HPC system.

Talk: Introduction to HPC

Albert Kahira

A few days ago, OPTIMA announced the release of deliverable 3.5, to which I contributed. This deliverable is part of a set of five deliverables under work package 3. But first, let’s talk about

 OPTIMA

. OPTIMA is an EU-funded project whose goal is to prove that several HPC applications can take advantage of the future highly heterogeneous FPGA-populated HPC systems.

OOPS Version 1 Release

On June 21 and 22, we held a workshop looking back on the last ten years of our lab together with NVIDIA – the NVIDIA Application Lab at Jülich (or

 NVLab

, as I sometimes abbreviate it). The material can be found in the agenda at Indico. We invited a set of application owners with which we worked together during that time to present past developments, recent challenges, and future plans.

10 Year Anniversary Workshop of NVIDIA Application Lab at Jülich

In November of 2022, I created a table comparing GPU programming models and their  support on GPUs of the three vendors (AMD, Intel, NVIDIA) for a talk. The audience liked it, so I beefed it up a little and posted it in this very blog.

Paper: Many Cores, Many Models – GPU Programming Model vs. Vendor Compatibility Overview

Environment Setup Enabling UCC in OpenMPI Enabling NCCL in UCC (Team Layer Selection) All The Variables Results 1. Plain OpenMPI 2. OpenMPI with UCC 3. OpenMPI with UCC+NCCL Scaling Plots Average Latency Bus Bandwidth Comparing MPI, UCC, UCC+NCCL Comparing UCC+NCCL, NCCL Summary Technical Details  This post showcases how to use the highly optimized, GPU-based collectives of Nvidia’s (NCCL library) by using it as a UCC backend through MPI.

MPI as API: Using UCC’s NCCL Backend for MPI’s Allreduce

A few months ago, we extended the JURECA Evaluation Platform

 1

at JSC by two nodes with AMD Instinct MI250 GPUs (four GPUs each). The nodes are Gigabyte G262-ZO0 servers, each with a dual socket AMD EPYC 7443 processor (24 cores per socket, SMT-2) and with four MI250 GPUs (128 GB memory). OSU Bandwidth Micro-Benchmark A100 Comparison GPU STREAM Variant Data Size Scan Threads and Data Sizes Conclusion Technical Details OSU

First Benchmarks with AMD Instinct MI250 GPUs at JSC

Carolin Penke

** Poster publication:** http://hdl.handle.net/2128/32006  The 14th JLESC workshop (JLESC: Joint Laboratory for Extreme-Scale Computing) was hosted by the National Center for Supercomputing Applications (NCSA) in Urbana, Illinois from 28th September to 30th September. We had the opportunity to present the OpenGPT-X project in form of a poster.

Poster: OpenGPT-X - Training Large Language Models on HPC Systems

About    This blog post is based on a presentation I held at the “New Trends in Computational Science in Engineering and Industrial Mathematics” workshop in Magdeburg on 01/07/2022. My goal is to give a brief introduction to the state of current large language models, the OpenGPT-X project, and the transformer neural network architecture for people unfamiliar with the subject. About What is a language model?

A mathematician’s introduction to transformers and large language models

1

This blog is an experiment. We want to share bits and pieces of our work; the reports we write, the presentations we hold, or the little discoveries we make, or even some first, water-testing investigations; and all the rest. It’s a

 documentation

of what we do. Little bits of science, collected in the open, and sometimes even not that little.

DOIng it Right! (DOIs for This Blog)

JSC Accelerating Devices Lab

The [Supercomputing Conference 2023](https://sc23.supercomputing.org/)
took place in Denver, Colorado, from November 12th to 17th. For the
*Women in HPC* workshop, we submitted a
[paper](https://zenodo.org/doi/10.5281/zenodo.10116241), which focused
on benchmarking different accelerators for AI. The paper was accepted
and I was invited to hold a [lightning
talk](https://sc23.conference-program.com/presentation/?id=ws_whpc106&sess=sess442)
to show the work, spun off our [OpenGPT-X](https://opengpt-x.de/en/)
project.

Given the multitude of AI accelerators available in the market, our
objective was to start establishing a framework for benchmarking these
accelerators systematically. The currently selected benchmarks enable a
evaluation of performance and energy efficiency in two key AI domains:
Computer Vision and Natural Language Processing (NLP). As an initial
step in assessing hardware capacities, we evaluated two benchmarks using
resources available in the [JURECA
DC](https://doi.org/10.17815/jlsrf-7-182) supercomputer, specifically
the [JURECA DC Evaluation
Platform](https://apps.fz-juelich.de/jsc/hps/jureca/evaluation-platform-overview.html).
We evaluated

- NVIDIA A100 GPUs (40 GB memory, SXM),
- NVIDIA H100 GPUs (80 GB, PCIe),
- AMD MI250 GPUs (64 GB), and
- Graphcore GC200 IPUs (IPU-M2000 POD-4, ≈260 GB).

## ResNet Benchmark

The first benchmark employs TensorFlow for a
[ResNet-50](https://en.wikipedia.org/wiki/Residual_neural_network)
benchmark, providing insights into the overall machine learning capacity
of the hardware. The second benchmark, derived from the OpenGPT-X fork
of [Megatron-LM](https://github.com/NVIDIA/Megatron-LM.git), offers
insights into large language model (LLM) training. A [fork by our
colleagues from Helmholtz
AI](https://github.com/HelmholtzAI-FZJ/tf_cnn_benchmarks.git) of the
ResNet-50 benchmark was executed on the NVIDIA and AMD GPUs. For
Graphcore IPUs, a [device-optimized
version](https://github.com/graphcore/examples.git) provided by the
vendor was used, as a naïve TensorFlow setup is not compatible with the
IPU architecture.

The results for the ResNet-50 benchmark are displayed as heat maps in
the following, with the number of devices on the *y axis* and the global
batch size on the *x axis*. The color map depicts the throughput
measured in images/sec. The throughput scales in relation to the global
batch size and the number of devices used. As the ResNet-50 model fits
into a single device for all the tested hardware, it implies that the
degree of data parallelism matches the number of devices employed.

[![](/assets/img/posts/2023-12-12-sc-novel-architecture-exploration/heatmap-comparison.svg)](/assets/img/posts/2023-12-12-sc-novel-architecture-exploration/heatmap-comparison.svg)

By considering the NVIDIA A100 GPU as a reference point, several
insights can be inferred. Specifically, upon comparing the last row of
the heatmaps for the NVIDIA A100 and H100, a nearly ≈ 1.4 − 2× increase
in throughput is observed. This increase aligns with expectations from
the latest generation of hardware. As seen from the last five columns of
all the heatmaps, NVIDIA GPUs appear to deliver the highest throughput
for larger batch sizes \>128.

Conversely, upon examining Graphcore, the initial four columns,
corresponding to batch sizes below 256, demonstrate superior performance
relative to other accelerators. This observation suggests that the
unique memory architecture of Graphcore IPUs, with SRAM distributed into
an organized set of small independent memory units, contributes to
increased in-processor memory. This enhancement is particularly
beneficial with the MIMD (Multiple Instructions, Multiple Data)
architecture, especially when batches fit into the in-process memory.

Regarding the AMD MI250, we've faced challenges in reaching the GPUs'
peak performance and intend to investigate it further.

## NLP Benchmark

Shifting our focus to the second benchmark, we trained an 800 million
parameter GPT model using the OpenGPT-X fork of Megatron-LM on a single
node (4 GPUs) of NVIDIA A100 and H100 GPUs. The following bar graph
illustrates the compute performance per GPU on the y axis across
different batch sizes for data parallelism of 4. Similar to the ResNet
results, we observed a 1.5× increase in performance.

[![](/assets/img/posts/2023-12-12-sc-novel-architecture-exploration/LLM_tflops_nvidia.svg){width="66%"}](/assets/img/posts/2023-12-12-sc-novel-architecture-exploration/LLM_tflops_nvidia.svg)

We also provide insight into energy efficiency, an aspect of growing
importance in modern computing. To study energy consumption, we train
the model from above for 1 h to achieve representative results. The
total energy consumed per device in a node is calculated using power
values logged with `nvidia-smi`{.language-plaintext .highlighter-rouge}.
H100 GPUs consume less energy than A100 GPUs, with an 18% decrease for a
global batch size of 16. We've added an *energy efficiency* to the
figure, dividing performance (in FLOP/s) by consumed energy (in Wh).

# Conclusion

In conclusion, from our results we observe that the IPU architecture
excels when dealing with small batch sizes that can efficiently fit into
in-processor memory, showcasing remarkable performance gains in these
scenarios. Conversely, the GPU architecture demonstrates its superiority
when it comes to accommodating large batch sizes and scaling, where its
parallel processing capabilities shine. The H100 is expectedly providing
better performance compared to A100, especially if energy is considered
as well.

*OpenGPT-X is funded by the Federal Ministry for Economic Affairs and
Climate Action (BMWK) of Germany for the period 2022-2024.*


::: {.pub}
** Poster publication:**
[http://hdl.handle.net/2128/34532](%20http://hdl.handle.net/2128/34532)
:::

[![ISC23 Poster about OpenGPT-X by Chelsea
John](/assets/img/posts/2023-05-26-isc23-opengptx/poster.tn.png "OpenGPT-X Poster"){style="width: 100px;"
srcset="/assets/img/posts/2023-05-26-isc23-opengptx/poster.tn.png 1x, /assets/img/posts/2023-05-26-isc23-opengptx/poster.tn.2x.png 2x, /assets/img/posts/2023-05-26-isc23-opengptx/poster.tn.4x.png 4x"}](/assets/img/posts/2023-05-26-isc23-opengptx/poster.pdf)The
[ISC High Performance Conference 2023](https://www.isc-hpc.com/) was
held at Hamburg, Germany from 21st May to 25th May. At the conference,
we presented a [project
poster](https://app.swapcard.com/event/isc-high-performance-2023/planning/UGxhbm5pbmdfMTIyMDkwNg==)
on the [OpenGPT-X](https://opengpt-x.de/en/) project, outlining the
progress and initial exploration results. The poster was even featured
in [HPCWire's May 24 recap of
ISC](https://taborevents.com/live-from-isc-2023/) within the AI segment!

The poster gives a high-level outlook of the structure and goals of the
project, along with some background knowledge on training large language
models. It is embedded at the end of the post.

In order to train and scale *large* language models of transformer
architecture, different parallelization schemes such as *Data
Parallelism (DP)*, *Pipeline Parallelism (PP)*, and *Tensor Parallelism*
can be used.  
Data parallelism is when the training data is distributed in batches
across the replicated full models, with timely averaged gradient updates
of the model via *allreduce*. Pipeline Parallelism is when the model
layers are distributed across ranks, with asynchronous pipe scheduling
for gradient accumulation. Further, when tensor operations are
partitioned across ranks, tensor parallelism comes into play.

Finding the correct level of parallelism by combining the three
parallelization schemes is important for efficient training and
utilization of the hardware.

In the poster, an example for a 13.6 Billion parameter model trained on
English-German data is shown. Using a fork of the
[Megatron-DeepSpeed](https://github.com/bigscience-workshop/Megatron-DeepSpeed)
library, the size of the model amounts to 56 GB ( parameters +
gradients + optimizers states + [ZeRO Stage
1](https://arxiv.org/abs/1910.02054)). To fit into the memory of an
NVIDIA A100 (40 GB), it is partitioned using pipeline parallelism of 2.
Now the full model is split across 2 GPUs, occupying 28 GB each.

With a data parallelism level of 80 and global batch size of 960 on 160
GPUs (40 nodes) of JUWELS Booster, we see 96% average GPU utilization
with close to full memory usage. Furthermore, the decreasing model loss
attests to the quality of the model training.

[![](/assets/img/posts/2023-05-26-isc23-opengptx/barplot-comparison.svg){.barplot}](/assets/img/posts/2023-05-26-isc23-opengptx/barplot-comparison.svg)

In the poster, we show results of benchmarking novel hardware
architectures by using a ResNet-50 TensorFlow benchmark. Three different
accelerators are evaluated: NVIDIA A100 GPU, AMD MI250 GPU, and
Graphcore GC200 IPU. All are part of JURECA DC or the JURECA DC
Evaluation Platform.

::: {style="float: right; width: 30%;"}
[![](/assets/img/posts/2023-05-26-isc23-opengptx/heatmap-a100.svg){.heatmap}](/assets/img/posts/2023-05-26-isc23-opengptx/heatmap-a100.svg)
[![](/assets/img/posts/2023-05-26-isc23-opengptx/heatmap-mi250.svg){.heatmap}](/assets/img/posts/2023-05-26-isc23-opengptx/heatmap-mi250.svg)
[![](/assets/img/posts/2023-05-26-isc23-opengptx/heatmap-graphcore.svg){.heatmap}](/assets/img/posts/2023-05-26-isc23-opengptx/heatmap-graphcore.svg)
:::

The column graph shows the performance in *images per sec* for global
batch sizes varying from 16 to 2048 on a single node of the
accelerators. Small batch sizes aids to the hardware architecture and
on-tile memory feature of Graphcore IPUs. For larger batch sizes, the
A100's potential can be seen. Further optimizations and evaluation are
required to address the unexpectedly slow performance of the MI250. The
three heatmaps shown on the poster -- which are representations of the
same data as in the column plot -- are shown to the right.

Recent advancements such as [Sequence
Paralleism](https://arxiv.org/abs/2205.05198) and
[FlashAttention](https://arxiv.org/abs/2205.14135) could further improve
the training performance. We are evaluating them currently. Finally, the
poster also outlines some challenges faced in the regions of data,
hardware robustness, energy and model biases.

*OpenGPT-X is funded by the Federal Ministry for Economic Affairs and
Climate Action (BMWK) of Germany for the period 2022-2024. Compute time
on the GCS Supercomputer JUWELS Booster at JSC is provided through the
Gauss Centre for Supercomputing e.V*

Your browser is unable to display the PDF. Instead, please download it.


::: {.pub}
** Poster publication:**
[http://hdl.handle.net/2128/32006](%20http://hdl.handle.net/2128/32006)
:::

The [14th JLESC
workshop](https://publish.illinois.edu/14th-jlesc-workshop/) (JLESC:
Joint Laboratory for Extreme-Scale Computing) was hosted by the National
Center for Supercomputing Applications (NCSA) in Urbana, Illinois from
28th September to 30th September.

[![JLESC 2022 Poster about OpenGPT-X by Carolin
Penke](/assets/slides/posts/2022-10-06-poster-jlesc-opengptx/opengptx-poster.tn.png "OpenGPT-X Poster"){style="width: 100px;"
srcset="/assets/slides/posts/2022-10-06-poster-jlesc-opengptx/opengptx-poster.tn.png 1x, /assets/slides/posts/2022-10-06-poster-jlesc-opengptx/opengptx-poster.tn.2x.png 2x, /assets/slides/posts/2022-10-06-poster-jlesc-opengptx/opengptx-poster.tn.4x.png 4x"}](/assets/slides/posts/2022-10-06-poster-jlesc-opengptx/opengptx-poster.pdf)We
had the opportunity to present the [OpenGPT-X](https://opengpt-x.de/en/)
project in form of a poster.

On it, you can find information about the project partners within
OpenGPT-X, its goals and the use cases of large language models it
explores. Various available language models are presented.

Recent breakthroughs became possible due to the novel neural network
architecture called [transformer](/2022/07/13/transformers-matmul.html),
based on so-called self-attention layers. These allow for the parallel
processing of input using highly efficient matrix products.

In order to scale to a full supercomputer, three dimensions of
parallelism are intertwined: Data parallelism, pipeline parallelism, and
tensor parallelism. The total number of used GPU devices is given by
multiplying these three parallel degrees.

Novel AI architectures explored at the Jülich Supercomputing Centre
include AMD Instinct MI250 GPUs and Graphcore IPUs.

Using the Megatron-Deepspeed training framework one can easily achieve
about 50% of peak performance on Nvidia A100 GPUs. In our tests,
including 32 GPUs on 8 nodes of JUWELS Booster, the highest throughput
(in terms of TFLOP/s) is achieved when the focus is given to data
parallelism, and pipeline parallelism is used to reduce the memory
footprint of the model.

Project challenges include hardware related spurious errors and energy
consumption.

Our runs in OpenGPT-X are made possible through compute time on JUWELS
Booster, given by the Gauss Centre for Supercomputing e.V.
([www.gauss-centre.eu](http://www.gauss-centre.eu)) through the John von
Neumann Institute for Computing at JSC.

Your browser is unable to display the PDF. Instead, please download it.
