Various notes from the Accelerating Devices Lab (X-Dev) of Jülich Supercomputing Centre

Chelsea John

The Supercomputing Conference 2023 took place in Denver, Colorado, from November 12th to 17th. For the

 Women in HPC

workshop, we submitted a paper, which focused on benchmarking different accelerators for AI. The paper was accepted and I was invited to hold a lightning talk to show the work, spun off our OpenGPT-X project.

SC23 WHPC Workshop Paper: OpenGPT-X – Novel Architecture Exploration

Stepan Nassyr

** Poster in institute repository:** http://dx.doi.org/10.34734/FZJ-2023-03437 During the RISC-V Summit Europe 2023


in Barcelona we presented our work generating highly optimized RISC-V and ARM GEMM microkernels for BLIS using a custom software tool.

 1

We presented results on the Fujitsu A64FX processor, the in-development RISC-V

 VEC

processor from the EUPILOT project using an FPGA SDV (RVV 0.7.1, later RVV

Poster: GEMMERATOR – GEMM Kernel Generator

Andreas Herten

Together with our colleagues from Helmholtz-Zentrum Dresden-Rossendorf (HZDR) and in collaboration with HIDA and OpenHackathons, we hosted the

 Helmholtz GPU Hackathon 2023

in Jülich in May.

 I’ve blogged about the event for the
 
 Zweikommazwei
 
 blog of Forschungszentrum;

Helmholtz GPU Hackathon 2023

At the end of May, Forschungszentrum Jülich got a new website. Finally, the group also has a dedicated page there, listing the things we do, the project we are part of, and all the members in the group. And thanks to a photo shooting by Sebastian, we have great portraits and an amazing group picture.

X-Dev Website at fz-juelich.de

Jayesh Badwaik

** Poster in institute repository:** http://dx.doi.org/10.34734/FZJ-2023-04519  At the ISC High Performance Conference 2023 a little while ago, we presented a project poster on the SCALABLE project, embedded at the end of this post. The work was also presented as a paper at the the Computing Frontiers 2023 conference previously.

ISC23 Project Poster: SCALABLE: Scalable Lattice-Boltzmann Leaps to Exascale

Sebastian Achilles

MSA Concept MSA Software Building Blocks Workshop Exercises 1: Hello World! 2. GPU Hello World! 3: CPU-GPU Ping Pong Slides  On May 29, we held a workshop about using the Modular Supercomputing Architecture (MSA) together with project partners from ParTec.

MSA Introduction Workshop

** Poster publication:** http://hdl.handle.net/2128/34532  The ISC High Performance Conference 2023 was held at Hamburg, Germany from 21st May to 25th May. At the conference, we presented a project poster on the OpenGPT-X project, outlining the progress and initial exploration results. The poster was even featured in HPCWire’s May 24 recap of ISC within the AI segment!

ISC23 Project Poster: OpenGPT-X – Training Large Language Models on HPC Systems

For a recent talk at DKRZ in the scope of the natESM project, I created a table summarizing the current state of using a certain programming model on a GPU of a certain vendor, for C++ and Fortran. Since it lead to quite a discussion in the session, I made a standalone version of it with some updates and elaborations here and there.

GPU Vendor/Programming Model Compatibility Table

TL;DR: I held a HPC intro talk. Slides are below.

In MAELSTROM, we connect three areas of science:

 🌍Weather and climate simulation

with

 🤖Machine Learning methods and workflows

using

 📈HPC techniques and resources

. Halfway into the project, we held a boot camp at JSC to teach this Venn diagram to a group of students a few days ago. Some were ML experts, but had never used a HPC system.

Talk: Introduction to HPC

Albert Kahira

A few days ago, OPTIMA announced the release of deliverable 3.5, to which I contributed. This deliverable is part of a set of five deliverables under work package 3. But first, let’s talk about

 OPTIMA

. OPTIMA is an EU-funded project whose goal is to prove that several HPC applications can take advantage of the future highly heterogeneous FPGA-populated HPC systems.

OOPS Version 1 Release

On June 21 and 22, we held a workshop looking back on the last ten years of our lab together with NVIDIA – the NVIDIA Application Lab at Jülich (or

 NVLab

, as I sometimes abbreviate it). The material can be found in the agenda at Indico. We invited a set of application owners with which we worked together during that time to present past developments, recent challenges, and future plans.

10 Year Anniversary Workshop of NVIDIA Application Lab at Jülich

In November of 2022, I created a table comparing GPU programming models and their  support on GPUs of the three vendors (AMD, Intel, NVIDIA) for a talk. The audience liked it, so I beefed it up a little and posted it in this very blog.

Paper: Many Cores, Many Models – GPU Programming Model vs. Vendor Compatibility Overview

Environment Setup Enabling UCC in OpenMPI Enabling NCCL in UCC (Team Layer Selection) All The Variables Results 1. Plain OpenMPI 2. OpenMPI with UCC 3. OpenMPI with UCC+NCCL Scaling Plots Average Latency Bus Bandwidth Comparing MPI, UCC, UCC+NCCL Comparing UCC+NCCL, NCCL Summary Technical Details  This post showcases how to use the highly optimized, GPU-based collectives of Nvidia’s (NCCL library) by using it as a UCC backend through MPI.

MPI as API: Using UCC’s NCCL Backend for MPI’s Allreduce

A few months ago, we extended the JURECA Evaluation Platform

 1

at JSC by two nodes with AMD Instinct MI250 GPUs (four GPUs each). The nodes are Gigabyte G262-ZO0 servers, each with a dual socket AMD EPYC 7443 processor (24 cores per socket, SMT-2) and with four MI250 GPUs (128 GB memory). OSU Bandwidth Micro-Benchmark A100 Comparison GPU STREAM Variant Data Size Scan Threads and Data Sizes Conclusion Technical Details OSU

First Benchmarks with AMD Instinct MI250 GPUs at JSC

Carolin Penke

** Poster publication:** http://hdl.handle.net/2128/32006  The 14th JLESC workshop (JLESC: Joint Laboratory for Extreme-Scale Computing) was hosted by the National Center for Supercomputing Applications (NCSA) in Urbana, Illinois from 28th September to 30th September. We had the opportunity to present the OpenGPT-X project in form of a poster.

Poster: OpenGPT-X - Training Large Language Models on HPC Systems

About    This blog post is based on a presentation I held at the “New Trends in Computational Science in Engineering and Industrial Mathematics” workshop in Magdeburg on 01/07/2022. My goal is to give a brief introduction to the state of current large language models, the OpenGPT-X project, and the transformer neural network architecture for people unfamiliar with the subject. About What is a language model?

A mathematician’s introduction to transformers and large language models

1

This blog is an experiment. We want to share bits and pieces of our work; the reports we write, the presentations we hold, or the little discoveries we make, or even some first, water-testing investigations; and all the rest. It’s a

 documentation

of what we do. Little bits of science, collected in the open, and sometimes even not that little.

DOIng it Right! (DOIs for This Blog)

JSC Accelerating Devices Lab

::: {.pub}
** Poster in institute repository:**
[http://dx.doi.org/10.34734/FZJ-2023-03437](%20http://dx.doi.org/10.34734/FZJ-2023-03437)
:::

[![RISC-V Summit Europe 2023 Poster about a BLIS Codegenerator for
Vegtor-lenght-agnostic
ISAs](/assets/img/posts/2023-12-05-gemmerator/rv-summit-23-poster.tn.png "GEMMerator Poster at RISC-V Summit Europe 2023"){style="width: 100px;"
srcset="/assets/img/posts/2023-12-05-gemmerator/rv-summit-23-poster.tn.png 1x, /assets/img/posts/2023-12-05-gemmerator/rv-summit-23-poster.tn.2x.png 2x, /assets/img/posts/2023-12-05-gemmerator/rv-summit-23-poster.tn.4x.png 4x"}](/assets/slides/posts/2023-12-05-gemmerator/rv-summit-23-poster.pdf)During
the [RISC-V Summit Europe
2023](https://riscv-europe.org/summit/2023/)^[](http://web.archive.org/web/20231204152026/https://riscv-europe.org/summit/2023/ "The RISC-V Summit website has changed somewhat in the last months, so just to be sure we provide a Wayback Machine link.")^
in Barcelona we [presented our
work](https://riscv-europe.org/summit/2023/posters#stepan-nassyr--programmatically-reaching-the-roof-automated-blis-kernel-generator-for-sve-and-rvv)
generating highly optimized RISC-V and ARM GEMM microkernels for
[BLIS](https://github.com/flame/blis) using a custom software
tool.^[1](#fn:phd){.footnote rel="footnote"}^ We presented results on
the Fujitsu A64FX processor, the in-development RISC-V *VEC* processor
from the [EUPILOT project](https://eupilot.eu/) using an FPGA SDV (RVV
0.7.1, later RVV 1.0.0)^[2](#fn:vecsdv){.footnote rel="footnote"}^, and
a commercially available, non-HPC Allwinner D1 processor (RVV
0.7.1).^[3](#fn:allwinner){.footnote rel="footnote"}^

The presented poster is embedded at the end of this post; it is
accompanied by an [extended
abstract](/assets/slides/posts/2023-12-05-gemmerator/rv-summit-23-abstract.pdf).^[4](#fn:documents){.footnote
rel="footnote"}^

One of the reasons for selecting a code generation approach is that
*[Analytical Modeling is enough for High-Performance
BLIS](https://dl.acm.org/doi/pdf/10.1145/2925987)*. We assume that
parameters that shape a compute kernel can be derived from
micro-architectural parameters of the system it will be running on
resulting in (nearly) optimal performance.

The generator accepts the aforementioned kernel parameters and outputs
the O(n^3^) inner part of the BLIS GEMM microkernel - we chose the term
*nanokernel*. With some boilerplate code on top, a full BLIS microkernel
can be created. Our tool can generate assembly code for multiple ISAs,
like different x86 flavors, ARM NEON or SVE, and RISC-V RVV (0.7.1/1.0).
It also accepts parameters for tuning the implementation, for example
using different instructions as shown below for a RISC-V DGEMM example.

<table id="asm-highlight">
<colgroup>
<col style="width: 50%" />
<col style="width: 50%" />
</colgroup>
<thead>
<tr class="header">
<th colspan="2">Part of RISC-V Assembly from a generated DGEMM
kernel</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<th>RISC-V ASM (vv-FMA form)</th>
<th>RISC-V ASM (vf-FMA form)</th>
</tr>
&#10;<tr class="odd">
<td><figure class="highlight">
<pre class="gas" data-lang="gas"><code>vle64.v v2, (t0)
add t0,t0,t5
vfmacc.vv v24,v0,v4
vle64.v v3, (t0)
add t0,t0,t5
vfmacc.vv v25,v1,v4
vlse64.v v4, (t1), zero
add t1,t1,8
vfmacc.vv v26,v0,v5
vfmacc.vv v27,v1,v5
vlse64.v v5, (t1), zero
add t1,t1,8
vfmacc.vv v28,v0,v6
vfmacc.vv v29,v1,v6
vlse64.v v6, (t1), zero
add t1,t1,8
vfmacc.vv v30,v0,v7
vfmacc.vv v31,v1,v7
vlse64.v v7, (t1), zero
add t1,t1,8</code></pre>
</figure></td>
<td><figure class="highlight">
<pre class="gas" data-lang="gas"><code>vle64.v v2,(t0)
add t0,t0,t3
vfmacc.vf v24,f0,v0
vle64.v v3,(t0)
add t0,t0,t3
vfmacc.vf v25,f0,v1
fld f0,0(t1)
vfmacc.vf v26,f1,v0
vfmacc.vf v27,f1,v1
fld f1,8(t1)
vfmacc.vf v28,f2,v0
vfmacc.vf v29,f2,v1
fld f2,16(t1)
vfmacc.vf v30,f3,v0
vfmacc.vf v31,f3,v1
fld f3,24(t1)
add t1,t1,32
&#10;
</code></pre>
</figure></td>
</tr>
</tbody>
</table>

In the poster, we focused on RISC-V RVV and ARM SVE. Both are newer ISAs
that share the feature of being vector-length-agnostic (VLA), i.e. the
vector size/SIMD size is not known at compile time. And both ISAs
promise exciting applications in future HPC-grade chips.

To assess the quality of the generated code, the tool can also generate
standalone benchmarks that can measure the per-clock-cycle performance.
Of course, the benchmarks are configurable as well, allowing for running
the code entirely in the L1 cache, redirecting the memory access to the
same memory location, or eliminating memory accesses completely to
investigate the compute performance independently of the memory
architecture.

While multiple kernel parameters are supported, for this work we focus
on the microkernel *size* - the size of the microtile of the C-matrix
that the microkernel is working on. Since we focus on VLA ISAs, the size
is given in *numbers of vector registers* times *elements*. For example,
`2Vx10`{.language-plaintext .highlighter-rouge} on A64FX (SIMD size of
512 bit), would mean that the SGEMM microkernel works on a C microtile
32 elements tall and 10 elements wide, as 32 elements with 32 bit per
element (32 × 32 = 1024) account to 2 times the SIMD (2 × 512 = 1024).

You can see the results of our benchmarks in the following plots:

::: {.imagegrid}
[![Generated SGEMM kernels on
A64FX](/assets/img/posts/2023-12-05-gemmerator/a64fx_MVxN_sgemm.svg "Generated SGEMM kernels on A64FX"){style="float: initial"}](/assets/img/posts/2023-12-05-gemmerator/a64fx_MVxN_sgemm.svg)
[![Generated SGEMM kernels Allwinner
D1](/assets/img/posts/2023-12-05-gemmerator/d1_MVxN_sgemm.svg "Generated SGEMM kernels on Allwinner D1"){style="float: initial"}](/assets/img/posts/2023-12-05-gemmerator/d1_MVxN_sgemm.svg)
[![Generated SGEMM kernels on
VEC](/assets/img/posts/2023-12-05-gemmerator/fpga_epac_MVxN_sgemm.svg "Generated SGEMM kernels on VEC"){style="float: initial"}](/assets/img/posts/2023-12-05-gemmerator/fpga_epac_MVxN_sgemm.svg)
[![Generated DGEMM kernels on
VEC](/assets/img/posts/2023-12-05-gemmerator/fpga_epac_MVxN_dgemm.svg "Generated DGEMM kernels on VEC"){style="float: initial"}](/assets/img/posts/2023-12-05-gemmerator/fpga_epac_MVxN_dgemm.svg)
:::

The graphs show the performance of various microkernel sizes in
\#FLOP/cycle. Different line colours stand for different M dimensions of
the microkernel, while the N dimension is on the x axis. The code
generation is excellent for A64FX, reaching over 99% efficiency. For the
VEC processor we get above 90% efficiency and for the D1 above 80%.

Since the Summit, we managed to improve the performance on the VEC
processor by another FLOP/cycle and in collaboration with the chip
developers also helped uncover some hints towards a potential
performance bug.

The work on the tool is ongoing and we are interested in expanding its
scope to other compute kernels and libraries. One of our next software
targets is FFT, as it is also quite important for the work in EUPILOT --
just like BLAS.

We also plan to release a BLIS version with our generated microkernels.
Stay tuned!

*The European PILOT project has received funding from the European
High-Performance Computing Joint Undertaking (JU) under grant agreement
No.101034126. The JU receives support from the European Union's Horizon
2020 research and innovation programme and Spain, Italy, Switzerland,
Germany, France, Greece, Sweden, Croatia and Turkey.*

Your browser is unable to display the PDF. Instead, please [download
it](/assets/slides/posts/2023-12-05-gemmerator/rv-summit-23-poster.pdf).

::: {.footnotes role="doc-endnotes"}
1. ::: {#fn:phd}
 The tool is actual at the core of my
 PhD! [↩](#fnref:phd){.reversefootnote role="doc-backlink"}
 :::

2. ::: {#fn:vecsdv}
 BSC published a related paper on [SDVs for
 co-design](https://doi.org/10.1007/978-3-031-40843-4_39) [↩](#fnref:vecsdv){.reversefootnote
 role="doc-backlink"}
 :::

3. ::: {#fn:allwinner}
 One of the few commercially available RISC-V vector ISA
 implementations, albeit feature a pre-final
 version. [↩](#fnref:allwinner){.reversefootnote role="doc-backlink"}
 :::

4. ::: {#fn:documents}
 [Poster](https://riscv-europe.org/summit/2023/media/proceedings/posters/2023-06-07-Stepan-NASSYR-poster.pdf)
 and
 [abstract](https://riscv-europe.org/summit/2023/media/proceedings/posters/2023-06-07-Stepan-NASSYR-abstract.pdf)
 can also be found on the Summit
 website [↩](#fnref:documents){.reversefootnote role="doc-backlink"}
 :::
:::