Juan Carlos Graciosa

Introduction and benchmarking

The convection of the Earth’s mantle is usually modelled as an incompressible process, referred to as the Boussinesq approximation. However, in the Earth’s mantle, the pressure increase associated with depth also increases the density due to self-compression (King et al. 2010). In some applications, this compressibility may be non-negligible and modelling it may be desirable.

Compressible convection in cartesian coordinates with Underworld3

Romain Beucher

Conda packages are now available for most of the underworldcode suite of software


 (Underworld, UWGeodynamics, Lavavu, Stripy) from the geo-down-under conda channel.

Let’s face it. Users don’t want to spend time compiling code on their machine. It is time consuming, requires some specific skills and creates unnecessary hurdles in a workflow.

Here comes Conda...

Haibin Yang

A Lagrangian integration point finite element method for large deformation modeling of viscoelastic geomaterials

Abstract   Numerical models have become an indispensable tool for understanding and predicting the flow of ice sheets and glaciers. Here we present the full-Stokes software package Underworld to the glaciological community. The code is already well established in simulating complex geodynamic systems. Advantages for glaciology are that it provides a full-Stokes solution for elastic–viscous–plastic materials and includes mechanical anisotropy.

ISMIP-HOM benchmark experiments using Underworld

Louis Moresi

Meghan S Miller, Louis Moresi, Research School of Earth Sciences, Australian National University


 A new study by an international team of scientists has found lockdown measures to stop the spread of COVID-19 led to a 50 per cent reduction in seismic noise observed around the world.

It is the largest reduction in human-generated noise ever observed globally.

COVID-19 lockdown leads to seismic noise “quiet period”

Meghan S. Miller,

 Australian National University

and Louis Moresi,

 Australian National University


 
 How we built a simple dashboard using Github actions with open source software and openly available (FAIR) data.
 

We recently wrote an article in The Conversation that shows how the Australian Seismometers in schools network registers the pulse of Australian life through changes in the seismic noise

Australian Seismometers in Schools - Noise monitoring dashboard

Julian Giordani

The Underworld team would like to congratulate Bénédicte Cenki-Tok and co-authors at the University of Sydney and the University of Montpellier on their recent publication,  B. Cenki-Tok, P.F. Rey, D. Arcay;

Underworld publication news: Crustal thickness anomalies in stable continents.

Meghan S. Miller,

 Australian National University

and Louis Moresi,

 Australian National University

Our responses to the COVID-19 pandemic have dramatically changed human activity all over the world. People are working from home, schools are closed in many places, travel is restricted, and in some cases only essential shops and businesses are open. Scientists see signs of these changes wherever they look.

Australian cities are quiet during lockdown. Earthquake scientists are making the most of it

John Mansour

A new paper by John Mansour and others has just been published in the Journal of Open Source Software. Mansour, J., Giordani, J., Moresi, L., Beucher, R., Kaluza, O.,  Velic,  M., Farrington, R., Quenette, S., Beall, A., 2020, Underworld2:  Python  Geodynamics Modelling for Desktop, HPC and Cloud, Journal of Open

JOSS publication - Underworld 2

Rohan Byrne

As the old truism states, the best way to learn is by doing. This is terribly unhelpful since, really, you can't 'do' anything

 until

you learn. This is a Catch-22 that every grad student encounters; doubly so when it comes to anything involving code, where daily life provides very little in the way of intuition. Unlike other geodynamic modelling software,

 Underworld

is designed to be approached with code.

Getting started: 60 seconds to Underworld

Rebecca Farrington

Viscoelastic materials exhibit the properies of both solids and liquids, with deformation rates dependent on both the viscous stress and elastic stress rate.

Viscoelasticity in Underworld2

Making your research reproducible means that you provide the entire workflow from data, through software and post-processing freely available. Not only can somebody repeat your experiments and verify them, they can build upon them. In lab-based disciplines, there are many further challenges, but in research that is predominantly based on data processing, this ought to be an achievable goal.

Alaska Moho Model (Reproducible research with containers)

Zenodo is a repository for immutable versions of software that are provided with a persistent DOI for the purposes of citation and reproducibility.   Underworld can be cited via a zenodo DOI. There is a master DOI for all releases (10.5281/zenodo.1436039) and releases after 2.6.0 are automatically given a DOI _under the master_.

Cite Underworld from Zenodo

(they don't make them like they used to ... )

Cratons are anomalously-strong regions of the continents that have largely resisted tectonic forces for billions of years. How such strong zones could be forged in a hot, low-viscosity, low stress,  early-Earth has been a long-standing puzzle for geologists.

Craton Formation and the Onset of Plate Tectonics

Using physical units and how to appropriately scale a model is a top question users ask when beginning with Underworld. The equations Underworld solves are stated in a physically correct form, they remain valid as long as every material constant, geometry, time, etc., are expressed in the same system.  There is no explicit concept of units no scaling in Underworld, the choice is entirely left to the user.

Using physical units in Underworld

(aka the Underworld renovation project)

Underworld is our parallel, particle-in-cell, finite element geodynamics code [1]. For the past year or so, the Underworld team has been working on a refurbished user interface. We've known for a long time that it needed to be done but we finally bit the bullet.

Underworld 2

In a recent paper [1], we used Underworld models to examine subduction congestion associated with the ingestion of a continental ribbon. The SE Australian geological record turned out to be a wonderful place to study this process. Here is a short summary of the work for a relatively non-technical audience that we put together and some additional figures which I prepared.

The Dynamics of Continental Accretion

The Underworld Community has been supporting usage of Docker containers for developing and running Underworld powered scientific models.

Developing Underworld using Python Virtual Environments

The new generation of Apple Mac comes with the new Apple Silicon (M1) chip which has an Arm architecture (as opposed to the older generation that had i386 Intel processor). This brings all manner of troubles and requirements for the development of codes. Here I detail how to install Underworld on Mac OSX Big Sur 11.5.2 (Apple Silicon) using Python 3.9  Important Note: The following guide uses the CMake branch of Underworld.

How to install Underworld on Mac OSX (Apple Silicon M1)

How we added an auto-updating set of citations to underworld in our publications webpage.

We need to curate all the publications that we can find that use the underworld geodynamics code and provide this information on our website. To avoid needless repetition, we take advantage of the fact that nearly all the information we require is online and automatically construct the bibliography from an online public zotero library.

An automated (zotero) bibliography in a webpage

By the way, before you read this post, catch up with how we use docker by reading part 1  Docker allows us to distribute pre-built applications which are hosted in a virtual machine and are therefore platform independent.

Underworld and Docker (part 2)

How does Underworld scale on a HPC? In this post we showcase how Underworld 2.11 scales across two of Australia's premiere HPC systems. Gadi - https://nci.org.au/our-systems/hpc-systems Magnus - https://pawsey.org.au/systems/magnus/  The reference model chosen for this scaling showcase is a extended 3D stokes flow: Analytic Solution SolDB3D . Q1P0 elements were used and a fixed solver iteration count for solving the saddle point problem.

Underworld 2.11 Scaling

One of the most popular cloud services for python / jupyter - based codes is binder (www.mybinder.org) which can transform an online repository into a running instance in the cloud with little effort on the part of the owner of that repository to set it up and no effort at all for the end user.

Underworld's  lightweight cloud for online classrooms.

Modelling the relative time-scales of the Rayleigh-Taylor Instability and delamination, using Underworld

With Adam Beall, Cardiff University. Why model sub-continental gravitational instabilities? Within the plate tectonics framework, continents are generally considered to have a much lower density than the asthenosphere below and therefore avoid the kind of recycling that the oceanic crust and lithosphere undergoes.

Modelling Drips and Delamination with Underworld

In honour of Hans Mühlhaus' 70th birthday this month, here are some shear band simulations made with Underworld. We are investigating the role of dilatancy in the geometry of the shear bands for a box of material when a small trapdoor is opened. The extent to which large-scale deformation is needed to release material through the trapdoor depends on how much the material dilates when shear bands form.

Shear Bands with Dilatancy modelled with Underworld

Underworld2 is a python-friendly version of the Underworld geodynamics code which provides a programmable and flexible front end to all the functionality of the code running in a parallel HPC environment. This gives signficant advantages to the user, with access to the power of python libraries for setup of complex problems, analysis at runtime, problem steering, and coupling of multiple problems.

Underworld2 Workshop at CIG 2016 Meeting

While huge improvements in usability have been achieved in Underworld2, the installation process is still unfortunately as painful as ever. This difficulty is in large part due to Underworld's numerous dependencies, and also the multiple platforms we try to support. Compounding this, the legacy of a individual user's machine often conspires against success, rarely in obvious ways. Simply, compiling is not fun. Enter docker.

Underworld and Docker (part 1)

Yeah, one repo, one vision! In an effort to simplify maintenance and compatibility between Underworld and UWGeodynamics, we have decided to merge the codes into a single  repository. Starting with version 2.13, UWGeodynamics will now live under Underworld. All UWGeodynamics functionalities and workflows will remain available to the users.

Folding UWGeodynamics into Underworld

https://doi.org/10.59350/t7ghx-8f823 The following is compatible with Ubuntu 20.04 under

 
 Windows
 

10/11 WSL 2 (

 
 Windows
 

Subsystem for Linux). PETSc, the Portable, Extensible Toolkit for Scientific Computation is the main dependency required for building Underworld. In the following, I will describe my current workflow for configuring and installing PETSc on Linux.

Configuring and Installing PETSc for Underworld

Version control systems and online-communities such as Github have made contributing to the development of opensource software incredibly easy. That is assuming that people get over their initial fear of sharing their code!  In this short blog post I go through how to submit a pull request to one of our repository. Do not fear and join us for some fun!

Getting started with Pull requests

Version 2.8 of Underworld has been released recently. As with all major releases, this release brings numerous new features, enhancements and bug fixes. A summary of changes may be found within the usual CHANGES.md file. As is also usually the case, numerous API changes have been necessary or warranted.

Underworld Release&nbsp;2.8

Computational approaches to studying non-linear dynamics of the crust and mantle

Numerical investigation of 2D convection with extremely large viscosity variations

The accuracy of finite element solutions of Stokes's flow with strongly varying viscosity

Role of temperature‐dependent viscosity and surface plates in spherical shell models of mantle convection

We mark January 17, 1994 (the day better known for the Northridge Earthquake) as the birthday of the CITCOM family of codes. I was shaken awake early in the morning, and spent the rest of the day nervously waiting for the next aftershock while I checked that the code was really solving all the test problems quickly and accurately.

30 Years of Citcom, Ellipsis and Underworld

Neng Lu

Free surface in geodynamics simulations

Geodynamic simulations increasingly rely on models with a true free surface to investigate questions of tectonic deformation, mantle convection, and coupling of surface processes and lithosphere dynamics. Historically, most mantle convection simulations have been performed with free-slip boundary conditions at the surface.

Free surface in Underworld

To date

 weak scaling

tests have been run on two of the largest computers in Australia:

 Gadi

(NCI) and

 Magnus

(Pawsey). Here we present the results of those tests and discuss:

 Gadi: Weak scaling - SolDB3D Q1


 Gadi: Weak scaling - SolDB3D Q2


 Magnus: Weak scaling - SolDB3D Q2 - v2.10 vs v2.9

Underworld's

 Gadi

installation is setup as a

Scaling in Underworld

Issues with intra-element discontinuities in PIC-FEM   The classical finite element method (FEM) has been widely used to simulate diverse problems in engineering. Unlike most engineering problems, geological simulations are dominated by emergence of geometrical structures due to the non-linear processes involved.

Stress recovery in Underworld

As mentioned, packages can be installed directly from our conda channel. You may however want to build your own package at some point in the future or you may want to fix some of ours by submitting a Pull Request on github (You are more welcome to do so...)  The following goes through the steps of building a conda package for Badlands which contains Python and Fortran code. This is relatively advanced but not really difficult.

How to build Conda packages?

Underworld 2.10 has been released! Available via

 docker

(recommended),

 pip

and

 source code

. See here for more information on each install process. For a

 quick

taste of Underworld try our binder cloud resource, a temporary "virtual sandbox" to explore the release and example models. Click the button below. This new release includes the following: Enhancements Using UWGeodynamics-2.10.1, see here.

Underworld 2.10

Ben Mather

We've been busy creating the next major release of Stripy. To refresh your memory, Stripy is a Python tool for triangulating scattered points either in Cartesian coordinates or on the sphere. It wraps a bunch of Fortran codes in a neat, object-oriented Python interface that can be used for many geographical applications. What's new?

 Spline tension

- a lot of data transformations in Stripy are underpinned by cubic splines (e.g.

Stripy 2.0 released

The Underworld 2.9 release is available from Github, as a docker container and via zenodo (doi:10.5281/zenodo.3964957) it is also available through pip install for the first time (see below). Underworld paper in the Journal of Open Source Software   The paper of record for underworld 2 has been published. It does tie to the Underworld 2.9 release but is sufficiently broad to cite for all 2.x releases.

Underworld 2.9

Underworld Geodynamics Community

::: {.kg-card .kg-callout-card .kg-callout-card-blue}
::: kg-callout-text
[https://doi.org/10.59350/t7ghx-8f823](https://doi.org/10.59350/t7ghx-8f823?ref=underworldcode.org)
:::
:::

\
The following is compatible with Ubuntu 20.04 under ***Windows*** 10/11 WSL 2 (***Windows*** Subsystem for Linux).\
\
PETSc, the Portable, Extensible Toolkit for Scientific Computation is the main dependency required for building Underworld. In the following, I will describe my current workflow for configuring and installing PETSc on Linux. The instructions are also valid for the Windows Subsystem for Linux (WSL2) using the Ubuntu 20.04 app.

# Build dependencies

\
The minimum required to build PETSc is a C compiler (typically GCC), and a couple of utilities (Make, Python3). We will also need a C++ and a Fortran compiler to build the External packages that are part of the toolkit used by Underworld.

- wget (needed to download the source code, could also use git)
- gcc (C Compiler, required by PETSc)
- g++ (C++ Compiler, required to compile External Packages)
- gfortran (Fortran Compiler, required to compile External Packages)
- make (needed to build the source)
- python3 (needed to configure the build)

#### Ubuntu Linux (also under WSL2)

\...and other Debian based distribution)

<figure class="kg-card kg-code-card">
<div class="sourceCode" id="cb1"><pre class="sourceCode bash"><code class="sourceCode bash"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a>sudo apt-get update
<a href="#cb1-2" aria-hidden="true" tabindex="-1"></a>sudo apt-get install \
<a href="#cb1-3" aria-hidden="true" tabindex="-1"></a> make \
<a href="#cb1-4" aria-hidden="true" tabindex="-1"></a> gcc \
<a href="#cb1-5" aria-hidden="true" tabindex="-1"></a> python3 \
<a href="#cb1-6" aria-hidden="true" tabindex="-1"></a> gfortran \
<a href="#cb1-7" aria-hidden="true" tabindex="-1"></a> g++ \
<a href="#cb1-8" aria-hidden="true" tabindex="-1"></a> git</code></pre></div>
<figcaption>d</figcaption>
</figure>

### Arch Linux

\... and derived distributions (Manjaro)

``` bash
sudo pacman -Syy
sudo pacman -S \
 make \
 gcc \
 gcc-fortran \
 python3 \
 git
```

## Getting PETSc sources

The PETSc source code can be downloaded directly from the Web using git (Recommended)

``` bash
git clone -b release https://gitlab.com/petsc/petsc.git petsc
```

``` bash
cd petsc # Step in directory
git pull # Do a pull, just in case
git checkout v3.16.1 # Checkout vMAJOR_MINOR_PATCH
```

## Configuring PETSc

### Note on External Packages

PETSc (the Portable, Extensible Toolkit for Scientific Computation) can be configured to work with multiple of tools. Here we detail one possible configuration that should be sufficient for most Underworld users. The PETSc configuration system gives you the option to download most of the external packages. We can thus leverage PETSc to install most of the tools we need\...

#### Optimized BLAS

PETSc has only one hard requirement which is the BLAS (Basic Linear Algebra Subprograms) library. BLAS is available on most systems but some packages provide optimized version of the libraries that are more efficient. In most cases we recommend using the OPENBLAS packages. The Intel Math Library (MKL) also provides optimized BLAS libraries, it is free and commonly available on HPC machines.

Like most external packages you can download OPENBLAS by passing the `--download-openblas` command to the configuration script. At the time of writing this causes some errors and we chose to install OPENBLAS using the package manager of out LINUX distribution.

**Ubuntu**:

Note that Ubuntu provides a `libopenblas-dev` package that depends on `libopenblas-serial-dev` OR `libopenblas-openmp-dev`. Use the serial version when possible.

``` bash
sudo apt-get install \
 libopenblas-serial-dev \
 libopenblas-dev
```

**Arch Linux**

 sudo pacman -S openblas

### MPI (Message Passive Interface)

\
PETSc and Underworld use MPI. There are mainly 2 implementation of MPI:

- MPICH
- OPENMPI

The differences are irrelevant for most users. The important thing to keep in mind is that all packages used with PETSc must be configured to use the same MPI installation.\
We discourage users to build PETSc againts the MPI installation provided by their LINUX distribution. This is because upgrading the package may cause PETSc to complain that it was built with a different version of the one available on the machine.\
We recommend treating MPI as a normal External Package and have it installed via the PETSc configure script (see below).

### Configuring and Installing

We are going to install PETSc in the `/opt` directory which is a rather standard location for built-from-sources packages.

From the PETSc source directory:

``` bash
PETSC_VERSION=3.16.1 # Should Match version downloaded!

sudo mkdir /opt/petsc/${PETSC_VERSION}
sudo chown $USER /opt/petsc/${PETSC_VERSION}
```

A typical toolkit for Underworld includes some general utilities:

- HDF5 (for saving and reading H5 files)
- MPI (see above)
- MUMPS (**MU**ltifrontal **M**assively **P**arallel sparse direct **S**olver)
- METIS and ParMETIS (Parallel Graph Partitioning and Fill-reducing Matrix Ordering**)**
- SuperLU and SuperLU DIST (Supernodal LU Solver for sequential and distributed memory system)
- HYPRE (Scalable Linear Solvers and Multigrid Methods)
- ScaLAPACK (Scalable Linear Algebra PACKage)

``` bash
cd petsc-${PETSC_VERSION}

./configure \
 --prefix=/opt/petsc/${PETSC_VERSION} \
 --with-debugging=yes \
 --COPTFLAGS="-O3" \
 --CXXOPTFLAGS="-O3" \
 --FOPTFLAGS="-O3" \
 --with-shared-libraries \
 --with-cxx-dialect=C++11 \
 --with-make-np=8 \
 --download-mpich=yes \
 --download-hdf5=yes \
 --download-mumps=yes \
 --download-parmetis=yes \
 --download-metis=yes \
 --download-superlu=yes \
 --download-hypre=yes \
 --download-superlu_dist=yes \
 --download-scalapack=yes \
 --download-cmake=yes
```

*Note that in the above command, I am passing some optimization flags to the compilers for completeness (COPTFLAGS, CXXOPTFLAGS, FOPTFLAGS). However, I am also passing the `--with-debugging=yes` which effectively overwrite the flag options.*

The configuration step should download all the required packages.

Once it is done you can build PETSc by doing:

``` bash
make PETSC_DIR=$PWD PETSC_ARCH=arch-linux-c-debug all
```

Then install it to its destination location:

``` bash
make PETSC_DIR=$PWD PETSC_ARCH=arch-linux-c-debug install
```

Do a final check:

``` bash
make PETSC_DIR=/opt/petsc/${PETSC_VERSION} PETSC_ARCH="" check
```

Which should return:

``` bash
Running check examples to verify correct installation
Using PETSC_DIR=/opt/petsc/3.16.1 and PETSC_ARCH=
C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process
C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes
C/C++ example src/snes/tutorials/ex19 run successfully with hypre
C/C++ example src/snes/tutorials/ex19 run successfully with mumps
C/C++ example src/snes/tutorials/ex19 run successfully with superlu_dist
C/C++ example src/vec/vec/tests/ex47 run successfully with hdf5
Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process
Completed test examples
```

## Conclusion

\
We have described one way to have a working PETSc installation to be used with Underworld. Experienced users may choose a different path and have different installations of PETSc pointing to different libraries with optimization options turned on etc\... Users may also want to experiment with different compilers, different implementation of BLAS and/or MPI for example. PETSc is very powerful and flexible so many configurations can (should?) be tested.\
\
Feel free to contact me or the other members of the team on Github (@rbeucher)