Various notes from the Accelerating Devices Lab (X-Dev) of Jülich Supercomputing Centre

Chelsea John

The Supercomputing Conference 2023 took place in Denver, Colorado, from November 12th to 17th. For the

 Women in HPC

workshop, we submitted a paper, which focused on benchmarking different accelerators for AI. The paper was accepted and I was invited to hold a lightning talk to show the work, spun off our OpenGPT-X project.

SC23 WHPC Workshop Paper: OpenGPT-X – Novel Architecture Exploration

Stepan Nassyr

** Poster in institute repository:** http://dx.doi.org/10.34734/FZJ-2023-03437 During the RISC-V Summit Europe 2023


in Barcelona we presented our work generating highly optimized RISC-V and ARM GEMM microkernels for BLIS using a custom software tool.

 1

We presented results on the Fujitsu A64FX processor, the in-development RISC-V

 VEC

processor from the EUPILOT project using an FPGA SDV (RVV 0.7.1, later RVV

Poster: GEMMERATOR – GEMM Kernel Generator

Andreas Herten

Together with our colleagues from Helmholtz-Zentrum Dresden-Rossendorf (HZDR) and in collaboration with HIDA and OpenHackathons, we hosted the

 Helmholtz GPU Hackathon 2023

in Jülich in May.

 I’ve blogged about the event for the
 
 Zweikommazwei
 
 blog of Forschungszentrum;

Helmholtz GPU Hackathon 2023

At the end of May, Forschungszentrum Jülich got a new website. Finally, the group also has a dedicated page there, listing the things we do, the project we are part of, and all the members in the group. And thanks to a photo shooting by Sebastian, we have great portraits and an amazing group picture.

X-Dev Website at fz-juelich.de

Jayesh Badwaik

** Poster in institute repository:** http://dx.doi.org/10.34734/FZJ-2023-04519  At the ISC High Performance Conference 2023 a little while ago, we presented a project poster on the SCALABLE project, embedded at the end of this post. The work was also presented as a paper at the the Computing Frontiers 2023 conference previously.

ISC23 Project Poster: SCALABLE: Scalable Lattice-Boltzmann Leaps to Exascale

Sebastian Achilles

MSA Concept MSA Software Building Blocks Workshop Exercises 1: Hello World! 2. GPU Hello World! 3: CPU-GPU Ping Pong Slides  On May 29, we held a workshop about using the Modular Supercomputing Architecture (MSA) together with project partners from ParTec.

MSA Introduction Workshop

** Poster publication:** http://hdl.handle.net/2128/34532  The ISC High Performance Conference 2023 was held at Hamburg, Germany from 21st May to 25th May. At the conference, we presented a project poster on the OpenGPT-X project, outlining the progress and initial exploration results. The poster was even featured in HPCWire’s May 24 recap of ISC within the AI segment!

ISC23 Project Poster: OpenGPT-X – Training Large Language Models on HPC Systems

For a recent talk at DKRZ in the scope of the natESM project, I created a table summarizing the current state of using a certain programming model on a GPU of a certain vendor, for C++ and Fortran. Since it lead to quite a discussion in the session, I made a standalone version of it with some updates and elaborations here and there.

GPU Vendor/Programming Model Compatibility Table

TL;DR: I held a HPC intro talk. Slides are below.

In MAELSTROM, we connect three areas of science:

 🌍Weather and climate simulation

with

 🤖Machine Learning methods and workflows

using

 📈HPC techniques and resources

. Halfway into the project, we held a boot camp at JSC to teach this Venn diagram to a group of students a few days ago. Some were ML experts, but had never used a HPC system.

Talk: Introduction to HPC

Albert Kahira

A few days ago, OPTIMA announced the release of deliverable 3.5, to which I contributed. This deliverable is part of a set of five deliverables under work package 3. But first, let’s talk about

 OPTIMA

. OPTIMA is an EU-funded project whose goal is to prove that several HPC applications can take advantage of the future highly heterogeneous FPGA-populated HPC systems.

OOPS Version 1 Release

On June 21 and 22, we held a workshop looking back on the last ten years of our lab together with NVIDIA – the NVIDIA Application Lab at Jülich (or

 NVLab

, as I sometimes abbreviate it). The material can be found in the agenda at Indico. We invited a set of application owners with which we worked together during that time to present past developments, recent challenges, and future plans.

10 Year Anniversary Workshop of NVIDIA Application Lab at Jülich

In November of 2022, I created a table comparing GPU programming models and their  support on GPUs of the three vendors (AMD, Intel, NVIDIA) for a talk. The audience liked it, so I beefed it up a little and posted it in this very blog.

Paper: Many Cores, Many Models – GPU Programming Model vs. Vendor Compatibility Overview

Environment Setup Enabling UCC in OpenMPI Enabling NCCL in UCC (Team Layer Selection) All The Variables Results 1. Plain OpenMPI 2. OpenMPI with UCC 3. OpenMPI with UCC+NCCL Scaling Plots Average Latency Bus Bandwidth Comparing MPI, UCC, UCC+NCCL Comparing UCC+NCCL, NCCL Summary Technical Details  This post showcases how to use the highly optimized, GPU-based collectives of Nvidia’s (NCCL library) by using it as a UCC backend through MPI.

MPI as API: Using UCC’s NCCL Backend for MPI’s Allreduce

A few months ago, we extended the JURECA Evaluation Platform

 1

at JSC by two nodes with AMD Instinct MI250 GPUs (four GPUs each). The nodes are Gigabyte G262-ZO0 servers, each with a dual socket AMD EPYC 7443 processor (24 cores per socket, SMT-2) and with four MI250 GPUs (128 GB memory). OSU Bandwidth Micro-Benchmark A100 Comparison GPU STREAM Variant Data Size Scan Threads and Data Sizes Conclusion Technical Details OSU

First Benchmarks with AMD Instinct MI250 GPUs at JSC

Carolin Penke

** Poster publication:** http://hdl.handle.net/2128/32006  The 14th JLESC workshop (JLESC: Joint Laboratory for Extreme-Scale Computing) was hosted by the National Center for Supercomputing Applications (NCSA) in Urbana, Illinois from 28th September to 30th September. We had the opportunity to present the OpenGPT-X project in form of a poster.

Poster: OpenGPT-X - Training Large Language Models on HPC Systems

About    This blog post is based on a presentation I held at the “New Trends in Computational Science in Engineering and Industrial Mathematics” workshop in Magdeburg on 01/07/2022. My goal is to give a brief introduction to the state of current large language models, the OpenGPT-X project, and the transformer neural network architecture for people unfamiliar with the subject. About What is a language model?

A mathematician’s introduction to transformers and large language models

1

This blog is an experiment. We want to share bits and pieces of our work; the reports we write, the presentations we hold, or the little discoveries we make, or even some first, water-testing investigations; and all the rest. It’s a

 documentation

of what we do. Little bits of science, collected in the open, and sometimes even not that little.

DOIng it Right! (DOIs for This Blog)

JSC Accelerating Devices Lab

[![](/assets/img/posts/2024-06-12-p3hpc-compat/table.png){srcset="/assets/img/posts/2024-06-12-p3hpc-compat/table.png 1x, /assets/img/posts/2024-06-12-p3hpc-compat/table.2x.png 2x"}](/models/)

In [November of 2022](/2022/11/02/gpu-vendor-model-compat.html), I
created a table comparing GPU programming models and their support on
GPUs of the three vendors (AMD, Intel, NVIDIA) for a talk. The audience
liked it, so I beefed it up a little and posted it in this very blog.

People [**still** liked
it](https://scholar.google.de/scholar?oi=bibs&hl=de&cites=8160066363790021249),
so I took opportunity of some downtime to update and significantly
extend it, and make it into a paper for the [SC23 Workshop on
*Performance, Portability and Productivity in
HPC*](https://p3hpc.org/workshop/2023/) ([P3HPC](https://p3hpc.org/)).
In November 2023, [the paper was published as part of the SC23 Workshop
Proceedings](https://dl.acm.org/doi/10.1145/3624062.3624178?cid=99659740494).
Because it's closed access, you can access it freely if you come from
this blog^[1](#fn:authorizer){.footnote rel="footnote"}^; alternatively,
check out [the
preprint](https://arxiv.org/abs/2309.05445)^[2](#fn:perfectionism){.footnote
rel="footnote"}^.  
I also created a dedicated [*Models page* here on the blog](/models/),
which has the content in HTML form -- and also includes the link to the
paper.

I've changed the following things, compared to the original blog post:

1.  Update all entries and check for additions, deprecations, and other
    developments; extend the descriptions
2.  Describe more properly the methods and categories
3.  Add references for each entry, usually referring to the tool or
    specific aspect discussed
4.  Add discussions and a conclusion
5.  Keep the [GitHub repo](https://github.com/AndiH/gpu-lang-compat) as
    the single source of truth, but extend it to generate the PDF paper
    and to include references in the HTML version (through some horrible
    Pandoc trickery^[3](#fn:pandoctrickery){.footnote rel="footnote"}^)

As the paper is by no means *finished* but a mere snapshot of the status
quo in November 2023, my mini-presentation at the P3HPC workshop
(embedded below) asked for help from the community to keep the content
up-to-date; in the form of contributions to the [GitHub
repository](https://github.com/AndiH/gpu-lang-compat).  
So, **please help** [keep the content
up-to-date](https://github.com/AndiH/gpu-lang-compat/issues)! All
included updates will be incorporated immediately into the [*Models*
page](/models/), the new home of the table. In case you ever forget, the
table and repository are also linked [from the P3HPC workshop
website](https://p3hpc.org/resources/).

Your browser is unable to display the PDF. Instead, please [download
it](/assets/slides/posts/2024-06-12-p3hpc-compat/aherten-p3phpc-mcmm.pdf).

::: {.footnotes role="doc-endnotes"}
1.  ::: {#fn:authorizer}
    ACM gives authors the opportunity to add two referring domains to an
    *accept list*, which displays the paper despite closed access. Not
    great, but better than
    nothing. [↩](#fnref:authorizer){.reversefootnote
    role="doc-backlink"}
    :::

2.  ::: {#fn:perfectionism}
    While I have you here, let me talk about TeX. I've created the PDF
    following the SC23 ACM template guidelines. The most elaborate
    TeXing is some TikZ sprinkled here and there. For the literature, I
    used BibLaTeX, as per ACM's guidelines. Now, first, arXiv has issues
    with the two-column layout and stumbles over hyphenating words for
    line breaks. I could fix most, but not all. Then, for the actual
    publishing, turns out thath the publisher can not even accept
    BibLaTeX, but needs BibTeX. Because I use a lot of URLs and the
    appropriate fields in the bib file, I needed to rework this
    extensively -- last-minute and with a less pleasing look. So while
    the arXiv version still contains BibLaTeX (but has hyphenation
    issues), the published version has BibTeX (but no hyphenation
    issues). [A locally generated pre-preprint with proper hyphenation
    and bibliography can be found here for
    reference](/assets/misc/posts/2024-06-12-p3hpc-compat/aherten-p3hpc-gpu-compat.pdf).
    Phew. [↩](#fnref:perfectionism){.reversefootnote
    role="doc-backlink"}
    :::

3.  ::: {#fn:pandoctrickery}
    References can be referred to by multiple entries, of course, so
    just adding a reference behind each explaining entry would create a
    lot of duplications. To resolve that, I basically concatenate all
    references in a long, Markdown-formatted string and convert it to
    HTML with Pandoc, letting Pandoc resolve the dependencies. I then
    parse the HTML string again to extract the references for each table
    entry in HTML form and add them to the
    output. [↩](#fnref:pandoctrickery){.reversefootnote
    role="doc-backlink"}
    :::
:::


The [Supercomputing Conference 2023](https://sc23.supercomputing.org/)
took place in Denver, Colorado, from November 12th to 17th. For the
*Women in HPC* workshop, we submitted a
[paper](https://zenodo.org/doi/10.5281/zenodo.10116241), which focused
on benchmarking different accelerators for AI. The paper was accepted
and I was invited to hold a [lightning
talk](https://sc23.conference-program.com/presentation/?id=ws_whpc106&sess=sess442)
to show the work, spun off our [OpenGPT-X](https://opengpt-x.de/en/)
project.

Given the multitude of AI accelerators available in the market, our
objective was to start establishing a framework for benchmarking these
accelerators systematically. The currently selected benchmarks enable a
evaluation of performance and energy efficiency in two key AI domains:
Computer Vision and Natural Language Processing (NLP). As an initial
step in assessing hardware capacities, we evaluated two benchmarks using
resources available in the [JURECA
DC](https://doi.org/10.17815/jlsrf-7-182) supercomputer, specifically
the [JURECA DC Evaluation
Platform](https://apps.fz-juelich.de/jsc/hps/jureca/evaluation-platform-overview.html).
We evaluated

- NVIDIA A100 GPUs (40 GB memory, SXM),
- NVIDIA H100 GPUs (80 GB, PCIe),
- AMD MI250 GPUs (64 GB), and
- Graphcore GC200 IPUs (IPU-M2000 POD-4, ≈260 GB).

## ResNet Benchmark

The first benchmark employs TensorFlow for a
[ResNet-50](https://en.wikipedia.org/wiki/Residual_neural_network)
benchmark, providing insights into the overall machine learning capacity
of the hardware. The second benchmark, derived from the OpenGPT-X fork
of [Megatron-LM](https://github.com/NVIDIA/Megatron-LM.git), offers
insights into large language model (LLM) training. A [fork by our
colleagues from Helmholtz
AI](https://github.com/HelmholtzAI-FZJ/tf_cnn_benchmarks.git) of the
ResNet-50 benchmark was executed on the NVIDIA and AMD GPUs. For
Graphcore IPUs, a [device-optimized
version](https://github.com/graphcore/examples.git) provided by the
vendor was used, as a naïve TensorFlow setup is not compatible with the
IPU architecture.

The results for the ResNet-50 benchmark are displayed as heat maps in
the following, with the number of devices on the *y axis* and the global
batch size on the *x axis*. The color map depicts the throughput
measured in images/sec. The throughput scales in relation to the global
batch size and the number of devices used. As the ResNet-50 model fits
into a single device for all the tested hardware, it implies that the
degree of data parallelism matches the number of devices employed.

[![](/assets/img/posts/2023-12-12-sc-novel-architecture-exploration/heatmap-comparison.svg)](/assets/img/posts/2023-12-12-sc-novel-architecture-exploration/heatmap-comparison.svg)

By considering the NVIDIA A100 GPU as a reference point, several
insights can be inferred. Specifically, upon comparing the last row of
the heatmaps for the NVIDIA A100 and H100, a nearly ≈ 1.4 − 2× increase
in throughput is observed. This increase aligns with expectations from
the latest generation of hardware. As seen from the last five columns of
all the heatmaps, NVIDIA GPUs appear to deliver the highest throughput
for larger batch sizes \>128.

Conversely, upon examining Graphcore, the initial four columns,
corresponding to batch sizes below 256, demonstrate superior performance
relative to other accelerators. This observation suggests that the
unique memory architecture of Graphcore IPUs, with SRAM distributed into
an organized set of small independent memory units, contributes to
increased in-processor memory. This enhancement is particularly
beneficial with the MIMD (Multiple Instructions, Multiple Data)
architecture, especially when batches fit into the in-process memory.

Regarding the AMD MI250, we've faced challenges in reaching the GPUs'
peak performance and intend to investigate it further.

## NLP Benchmark

Shifting our focus to the second benchmark, we trained an 800 million
parameter GPT model using the OpenGPT-X fork of Megatron-LM on a single
node (4 GPUs) of NVIDIA A100 and H100 GPUs. The following bar graph
illustrates the compute performance per GPU on the y axis across
different batch sizes for data parallelism of 4. Similar to the ResNet
results, we observed a 1.5× increase in performance.

[![](/assets/img/posts/2023-12-12-sc-novel-architecture-exploration/LLM_tflops_nvidia.svg){width="66%"}](/assets/img/posts/2023-12-12-sc-novel-architecture-exploration/LLM_tflops_nvidia.svg)

We also provide insight into energy efficiency, an aspect of growing
importance in modern computing. To study energy consumption, we train
the model from above for 1 h to achieve representative results. The
total energy consumed per device in a node is calculated using power
values logged with `nvidia-smi`{.language-plaintext .highlighter-rouge}.
H100 GPUs consume less energy than A100 GPUs, with an 18% decrease for a
global batch size of 16. We've added an *energy efficiency* to the
figure, dividing performance (in FLOP/s) by consumed energy (in Wh).

# Conclusion

In conclusion, from our results we observe that the IPU architecture
excels when dealing with small batch sizes that can efficiently fit into
in-processor memory, showcasing remarkable performance gains in these
scenarios. Conversely, the GPU architecture demonstrates its superiority
when it comes to accommodating large batch sizes and scaling, where its
parallel processing capabilities shine. The H100 is expectedly providing
better performance compared to A100, especially if energy is considered
as well.

*OpenGPT-X is funded by the Federal Ministry for Economic Affairs and
Climate Action (BMWK) of Germany for the period 2022-2024.*
