Various notes from the Accelerating Devices Lab (X-Dev) of Jülich Supercomputing Centre

Chelsea John

The Supercomputing Conference 2023 took place in Denver, Colorado, from November 12th to 17th. For the

 Women in HPC

workshop, we submitted a paper, which focused on benchmarking different accelerators for AI. The paper was accepted and I was invited to hold a lightning talk to show the work, spun off our OpenGPT-X project.

SC23 WHPC Workshop Paper: OpenGPT-X – Novel Architecture Exploration

Stepan Nassyr

** Poster in institute repository:** http://dx.doi.org/10.34734/FZJ-2023-03437 During the RISC-V Summit Europe 2023


in Barcelona we presented our work generating highly optimized RISC-V and ARM GEMM microkernels for BLIS using a custom software tool.

 1

We presented results on the Fujitsu A64FX processor, the in-development RISC-V

 VEC

processor from the EUPILOT project using an FPGA SDV (RVV 0.7.1, later RVV

Poster: GEMMERATOR – GEMM Kernel Generator

Andreas Herten

Together with our colleagues from Helmholtz-Zentrum Dresden-Rossendorf (HZDR) and in collaboration with HIDA and OpenHackathons, we hosted the

 Helmholtz GPU Hackathon 2023

in Jülich in May.

 I’ve blogged about the event for the
 
 Zweikommazwei
 
 blog of Forschungszentrum;

Helmholtz GPU Hackathon 2023

At the end of May, Forschungszentrum Jülich got a new website. Finally, the group also has a dedicated page there, listing the things we do, the project we are part of, and all the members in the group. And thanks to a photo shooting by Sebastian, we have great portraits and an amazing group picture.

X-Dev Website at fz-juelich.de

Jayesh Badwaik

** Poster in institute repository:** http://dx.doi.org/10.34734/FZJ-2023-04519  At the ISC High Performance Conference 2023 a little while ago, we presented a project poster on the SCALABLE project, embedded at the end of this post. The work was also presented as a paper at the the Computing Frontiers 2023 conference previously.

ISC23 Project Poster: SCALABLE: Scalable Lattice-Boltzmann Leaps to Exascale

Sebastian Achilles

MSA Concept MSA Software Building Blocks Workshop Exercises 1: Hello World! 2. GPU Hello World! 3: CPU-GPU Ping Pong Slides  On May 29, we held a workshop about using the Modular Supercomputing Architecture (MSA) together with project partners from ParTec.

MSA Introduction Workshop

** Poster publication:** http://hdl.handle.net/2128/34532  The ISC High Performance Conference 2023 was held at Hamburg, Germany from 21st May to 25th May. At the conference, we presented a project poster on the OpenGPT-X project, outlining the progress and initial exploration results. The poster was even featured in HPCWire’s May 24 recap of ISC within the AI segment!

ISC23 Project Poster: OpenGPT-X – Training Large Language Models on HPC Systems

For a recent talk at DKRZ in the scope of the natESM project, I created a table summarizing the current state of using a certain programming model on a GPU of a certain vendor, for C++ and Fortran. Since it lead to quite a discussion in the session, I made a standalone version of it with some updates and elaborations here and there.

GPU Vendor/Programming Model Compatibility Table

TL;DR: I held a HPC intro talk. Slides are below.

In MAELSTROM, we connect three areas of science:

 🌍Weather and climate simulation

with

 🤖Machine Learning methods and workflows

using

 📈HPC techniques and resources

. Halfway into the project, we held a boot camp at JSC to teach this Venn diagram to a group of students a few days ago. Some were ML experts, but had never used a HPC system.

Talk: Introduction to HPC

Albert Kahira

A few days ago, OPTIMA announced the release of deliverable 3.5, to which I contributed. This deliverable is part of a set of five deliverables under work package 3. But first, let’s talk about

 OPTIMA

. OPTIMA is an EU-funded project whose goal is to prove that several HPC applications can take advantage of the future highly heterogeneous FPGA-populated HPC systems.

OOPS Version 1 Release

On June 21 and 22, we held a workshop looking back on the last ten years of our lab together with NVIDIA – the NVIDIA Application Lab at Jülich (or

 NVLab

, as I sometimes abbreviate it). The material can be found in the agenda at Indico. We invited a set of application owners with which we worked together during that time to present past developments, recent challenges, and future plans.

10 Year Anniversary Workshop of NVIDIA Application Lab at Jülich

In November of 2022, I created a table comparing GPU programming models and their  support on GPUs of the three vendors (AMD, Intel, NVIDIA) for a talk. The audience liked it, so I beefed it up a little and posted it in this very blog.

Paper: Many Cores, Many Models – GPU Programming Model vs. Vendor Compatibility Overview

Environment Setup Enabling UCC in OpenMPI Enabling NCCL in UCC (Team Layer Selection) All The Variables Results 1. Plain OpenMPI 2. OpenMPI with UCC 3. OpenMPI with UCC+NCCL Scaling Plots Average Latency Bus Bandwidth Comparing MPI, UCC, UCC+NCCL Comparing UCC+NCCL, NCCL Summary Technical Details  This post showcases how to use the highly optimized, GPU-based collectives of Nvidia’s (NCCL library) by using it as a UCC backend through MPI.

MPI as API: Using UCC’s NCCL Backend for MPI’s Allreduce

A few months ago, we extended the JURECA Evaluation Platform

 1

at JSC by two nodes with AMD Instinct MI250 GPUs (four GPUs each). The nodes are Gigabyte G262-ZO0 servers, each with a dual socket AMD EPYC 7443 processor (24 cores per socket, SMT-2) and with four MI250 GPUs (128 GB memory). OSU Bandwidth Micro-Benchmark A100 Comparison GPU STREAM Variant Data Size Scan Threads and Data Sizes Conclusion Technical Details OSU

First Benchmarks with AMD Instinct MI250 GPUs at JSC

Carolin Penke

** Poster publication:** http://hdl.handle.net/2128/32006  The 14th JLESC workshop (JLESC: Joint Laboratory for Extreme-Scale Computing) was hosted by the National Center for Supercomputing Applications (NCSA) in Urbana, Illinois from 28th September to 30th September. We had the opportunity to present the OpenGPT-X project in form of a poster.

Poster: OpenGPT-X - Training Large Language Models on HPC Systems

About    This blog post is based on a presentation I held at the “New Trends in Computational Science in Engineering and Industrial Mathematics” workshop in Magdeburg on 01/07/2022. My goal is to give a brief introduction to the state of current large language models, the OpenGPT-X project, and the transformer neural network architecture for people unfamiliar with the subject. About What is a language model?

A mathematician’s introduction to transformers and large language models

1

This blog is an experiment. We want to share bits and pieces of our work; the reports we write, the presentations we hold, or the little discoveries we make, or even some first, water-testing investigations; and all the rest. It’s a

 documentation

of what we do. Little bits of science, collected in the open, and sometimes even not that little.

DOIng it Right! (DOIs for This Blog)

JSC Accelerating Devices Lab

1. [MSA Concept](#msa-concept){#markdown-toc-msa-concept}
2. [MSA Software Building
 Blocks](#msa-software-building-blocks){#markdown-toc-msa-software-building-blocks}
3. [Workshop](#workshop){#markdown-toc-workshop}
 1. [Exercises](#exercises){#markdown-toc-exercises}
 1. [1: Hello
 World!](#1-hello-world){#markdown-toc-1-hello-world}
 2. [2. GPU Hello
 World!](#2-gpu-hello-world){#markdown-toc-2-gpu-hello-world}
 3. [3: CPU-GPU Ping
 Pong](#3-cpu-gpu-ping-pong){#markdown-toc-3-cpu-gpu-ping-pong}
 2. [Slides](#slides){#markdown-toc-slides}

On May 29, we held a workshop about using the Modular Supercomputing
Architecture (MSA) together with project partners from ParTec. The
audience were collaborators from two projects funded in the same
[SCALEXA funding
call](https://www.bmbf.de/bmbf/shareddocs/bekanntmachungen/de/2021/08/2021-08-26-Bekanntmachung-SCALEXA.html)
by BMBF: [MExMeMo](https://gauss-allianz.de/de/project/title/MExMeMo)
and [IF(CES)^2^](https://gauss-allianz.de/de/project/title/IFCES2).

## MSA Concept

[![Slide from MSA Workshop slidedeck showing JSC\'s Supecomputer
strategy (page
6)](/assets/img/posts/2023-06-28-scal-msa-workshop/slides-p6.tn.png "JSC's supercomputer history; link to PDF page"){srcset="/assets/img/posts/2023-06-28-scal-msa-workshop/slides-p6.tn.png 1x, /assets/img/posts/2023-06-28-scal-msa-workshop/slides-p6.tn.2x.png 2x, /assets/img/posts/2023-06-28-scal-msa-workshop/slides-p6.tn.4x.png 4x"}](/assets/slides/posts/2023-06-28-scal-msa-workshop/aherten-sachilles--scalexa-msa-workshop-slides.pdf#page=6)While
CPU-based system are versatile and can be used for various workloads,
accelerated systems (think GPUs, FPGAs, AI chips) can be quite
efficient, but only suited for a sub-set of workloads. The MSA concept
foresees a combination of HPC systems with distinct hardware (*modules*)
into one combined *super*-system, usable for execution by a program.
That enables heterogeneous workloads, in which parts map to the
best-suited modules. The MSA concept was developed to production-grade
in the [EU-funded DEEP projects](https://www.deep-projects.eu/), of
which the [DEEP
system](https://www.fz-juelich.de/de/ias/jsc/systeme/prototypsysteme/deep-system)
is still being used to develop further features. By now, also many
production systems are built-up in an MSA fashion, like
[JURECA](https://www.fz-juelich.de/en/ias/jsc/systems/supercomputers/jureca)
and
[JUWELS](https://www.fz-juelich.de/en/ias/jsc/systems/supercomputers/juwels)
at JSC, but also for example [Leonardo at
CINECA](https://leonardo-supercomputer.cineca.eu/).

[![Slide from MSA Workshop slidedeck showing, amongst other things, the
JUWELS network
topology.](/assets/img/posts/2023-06-28-scal-msa-workshop/slides-p8.tn.png "JUWELS Modular Systems; link to PDF page"){srcset="/assets/img/posts/2023-06-28-scal-msa-workshop/slides-p8.tn.png 1x, /assets/img/posts/2023-06-28-scal-msa-workshop/slides-p8.tn.2x.png 2x, /assets/img/posts/2023-06-28-scal-msa-workshop/slides-p8.tn.4x.png 4x"}](/assets/slides/posts/2023-06-28-scal-msa-workshop/aherten-sachilles--scalexa-msa-workshop-slides.pdf#page=28)In
MExMeMo, the project we are involved in, we want to use MSA for a
heterogeneous multi-scale physics simulation: two application
cooperatively simulate a soft matter process -- one takes over the large
scales using CPUs, and the [other
simulates](https://gitlab.com/InnocentBug/SOMA) certain parts very
detailed and verifies the large-scale model on GPUs.

## MSA Software Building Blocks

[![Slide from MSA Workshop slidedeck with an example srun
call.](/assets/img/posts/2023-06-28-scal-msa-workshop/slides-p27.tn.png "An example heterogeneous job; link to PDF page"){srcset="/assets/img/posts/2023-06-28-scal-msa-workshop/slides-p27.tn.png 1x, /assets/img/posts/2023-06-28-scal-msa-workshop/slides-p27.tn.2x.png 2x, /assets/img/posts/2023-06-28-scal-msa-workshop/slides-p27.tn.4x.png 4x"}](/assets/slides/posts/2023-06-28-scal-msa-workshop/aherten-sachilles--scalexa-msa-workshop-slides.pdf#page=27)From
a software perspective, MSA utilizes many pre-existing components. We
usually use the heterogeneous job features of Slurm to launch HPC jobs
crossing module boundaries; we use MPI to communicate with the
components of the job. In addition, it can be useful to employ tools
like [`xenv`{.language-plaintext
.highlighter-rouge}](https://apps.fz-juelich.de/jsc/hps/jureca/modular-jobs.html),
[`env`{.language-plaintext
.highlighter-rouge}](https://man7.org/linux/man-pages/man1/env.1.html),
or even [`renv`{.language-plaintext
.highlighter-rouge}](https://github.com/FZJ-JSC/renv) to manipulate the
module-specific environments in an *ad-hoc fashion*. Through the setup,
application parts or even entirely disparate application, can be
executed on the respective modules and exchange messages with each
other, solving a scientific problem collaboratively. Advanced MSA
features are implemented in
[ParaStationMPI](https://github.com/ParaStation/psmpi), like MPI
collectives which take module affinity under consideration.

## Workshop

Just like this blogpost, the workshop started with a theoretical
introduction to the topic. In slides, ParTec shared details about the
MSA concept and usage with Slurm ^[1](#fn:partecslides){.footnote
rel="footnote"}^. Following that, we took over to present our MSA setup
at JSC -- globally and the specific software environment. Find the
slides embedded at the end!

The second part of the workshop were actual hands-on exercises in *fill
in the blank* style. Augmented with `TODO`{.language-plaintext
.highlighter-rouge} comments and descriptions in readmes, the
participants were steered to solving MSA software challenges of
increasing complexity.

### Exercises

We have [released the exercises material on
Zenodo](https://doi.org/10.5281/zenodo.8083130), but you can also find
it in [our JSC Gitlab
repository](https://gitlab.jsc.fz-juelich.de/herten1/msa-hello-world).
In the following, a quick run through the three exercises.

#### 1: Hello World!

The [first
exercises](https://gitlab.jsc.fz-juelich.de/herten1/msa-hello-world/-/tree/main/01-MSA-hello-world)
is a simple MPI *hello world* program which prints the hostname of the
node it is executed on. See the following (incomplete) listing for a
sketch:

::: {.language-c++ .highlighter-rouge}
::: {.highlight}
``` highlight
MPI_Init(NULL, NULL);
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
MPI_Get_processor_name(processor_name, &name_len);
printf("Hello world from processor %s, rank %d out of %d processors\n",
 processor_name, world_rank, world_size);
```
:::
:::

The source code is not different from a standard MPI hello world.
However, this task should teach how to launch an MPI application to two
modules. That is why the focus is on the job script. Using [Slurm's
heterogeneous job
support](https://slurm.schedmd.com/heterogeneous_jobs.html) enables
specifying requirements for each component of a job using the
`:`{.language-plaintext .highlighter-rouge} (*colon*) syntax. In
addition, we are using `xenv`{.language-plaintext .highlighter-rouge} to
load required modules, which might be different for the individual
components.

An example batch jobs definition reads:

::: {.language-bash .highlighter-rouge}
::: {.highlight}
``` highlight
#!/bin/bash -x
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --partition=cpu
#SBATCH hetjob
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --partition=gpu

srun \
 xenv -P -L GCC -L ParaStationMPI ./mpi_hello_world.cpu.out : \
 xenv -P -L GCC -L ParaStationMPI ./mpi_hello_world.gpu.out
```
:::
:::

Executing the script with the example application on the Cluster and
Booster modules of JUWELS, the two different machines can be easily
identified:

::: {.language-bash .highlighter-rouge}
::: {.highlight}
``` highlight
$ cat slurm-out.7577116 
Hello world from processor jwc00n024.juwels, rank 0 out of 2 processors
Hello world from processor jwb0001.juwels, rank 1 out of 2 processors
```
:::
:::

Of course we are not limited to only two job components in MSA, but we
could add as many as needed by the modular application!

Now that we have learned how we can launch an MSA job, we can continue
and sprinkle a little GPU on top.

#### 2. GPU Hello World!

In the [second
exercise](https://gitlab.jsc.fz-juelich.de/herten1/msa-hello-world/-/tree/main/02-MSA-hello-world-gpu),
a message is sent from a CPU node directly to a GPU buffer on a
GPU-equipped node:

::: {.language-c++ .highlighter-rouge}
::: {.highlight}
``` highlight
cudaMalloc((void**)&d_payload, 6);
cudaMemcpy(d_payload, payload, 6, cudaMemcpyHostToDevice);

hello<<<1, 1>>>(d_payload); // contains a `printf()`

if (rank == 1)
 MPI_Recv(d_payload, 6, MPI_CHAR, 0, tag, MPI_COMM_WORLD, &stat);

hello<<<1, 1>>>(d_payload);
```
:::
:::

The job script from task 1 needs to be slightly extended by using
GPU-specific modules for the second job component:

::: {.language-bash .highlighter-rouge}
::: {.highlight}
``` highlight
srun \
 xenv -P -L GCC -L ParaStationMPI ./hello.cpu.out : \
 xenv -P -L GCC -L ParaStationMPI -L MPI-settings/CUDA ./hello.gpu.out
```
:::
:::

The out is then showing the combined message from both parts:

::: {.language-bash .highlighter-rouge}
::: {.highlight}
``` highlight
$ cat slurm-out.7916615 
hello world!
```
:::
:::

#### 3: CPU-GPU Ping Pong

Finally, the [third
exercise](https://gitlab.jsc.fz-juelich.de/herten1/msa-hello-world/-/tree/main/03-MSA-ping-pong)
not only sends a single "world" to a GPU buffer, but a continuous
exchange between CPU buffer and GPU buffer is implemented across the
module boundaries (i.e. JUWELS Cluster ↔ JUWELS Booster). The message
size is increased step-wise and the bandwidth printed:

::: {.language-bash .highlighter-rouge}
::: {.highlight}
``` highlight
Transfer size (B): 8, Time (s): 0.000002580, Bandwidth (GB/s): 0.002888259
Transfer size (B): 16, Time (s): 0.000002594, Bandwidth (GB/s): 0.005745580
...
Transfer size (B): 1073741824, Time (s): 0.086975408, Bandwidth (GB/s): 11.497502804
```
:::
:::

### Slides

The slides were presented at the beginning of the workshop and were
[accompanied by slides from
ParTec](/assets/slides/posts/2023-06-28-scal-msa-workshop/partec-msa-workshop-mexmemo_ifces2.pdf)
focusing on the perspective from a *MSA solutions provider*; the ParTec
team introduced the benefits of the MSA concepts and details about the
MSA features of ParaStationMPI.

[Download our
slides](/assets/slides/posts/2023-06-28-scal-msa-workshop/aherten-sachilles--scalexa-msa-workshop-slides.pdf)
or browse through them here:

Your browser is unable to display the PDF. Instead, please [download
it](/assets/slides/posts/2023-06-28-scal-msa-workshop/aherten-sachilles--scalexa-msa-workshop-slides.pdf).

::: {.footnotes role="doc-endnotes"}
1. ::: {#fn:partecslides}
 ParTec allowed us to upload the slides as well, please [find them
 here](/assets/slides/posts/2023-06-28-scal-msa-workshop/partec-msa-workshop-mexmemo_ifces2.pdf). [↩](#fnref:partecslides){.reversefootnote
 role="doc-backlink"}
 :::
:::