DIRSIG5 supports the automated processing of single/multi-frame simulations across multiple processes/nodes using either the OpenMPI (1.x) or MPICH implementation of the Message Passing Interface (MPI).

Important The early releases of DIRSIG5 required the user to download a separate release that had been compiled with MPI. As of the DIRSG 2022.12 release, MPI can be enabled in the main release using a command-line option (see the use of --exec=mpi in the examples below).

Overview

A unified build for MPI and non-MPI Linux environments is available from myDIRSIG. When running in non-MPI mode, DIRSIG uses conventional file I/O to write output files, while the MPI mode leverages MPI-IO. It is recommended that end-users running on a single workstation with a single CPU socket use non-MPI mode to reduce the number of run-time dependencies. Workstations featuring more than one CPU socket can gain a significant run-time performance benefit by using the MPI build with a proper process topology.

MPI execution is supported by the following programs:

Dependencies

MPI poses an challenging dependency problem, as various implementations are not runtime-compatible with each other. As such, DIRSIG has taken the approach of distributing a number of adapter plugins to make DIRSIG compatible with various MPI implementations. These adapters, however, are simply a compatibility layer. The OpenMPI and MPICH run-time libraries are NOT distributed with DIRSIG5 because they are assumed to be installed on the host. It is also assumed that the runtime libraries for these implementations are in your system’s LD_LIBRARY_PATH. Currently, the adapters included are:

  • OpenMPI 1.x: Built with OpenMPI v1.10.7.

  • MPICH: Built with MPICH v3.2.

The build system used to produce the release is Redhat Enterprise Linux (RHEL) 7.

Process Topology

Due to NUMA considerations, it is recommended that one runs one bound process per CPU socket. Each process should then spawn a number of threads equal to the number of CPU cores available on said socket. In general, it is recommended that one disable simultaneous multi-threading (SMT, e.g. Hyper-Threading) to better control NUMA characteristics, but the performance impact of this varies between system configurations and may not be optimal in all situations.

MPI Modes

DIRSIG supports two scheduling modes:

All Processes per-Frame (APF)

This is the default behavior. APF distributes the processing of each frame (capture) in a simulation across all processes. It is used to maximize complete frame throughput.

Unique Frame per-Process (UFP)

UFP assigns one unique frame (capture) to each process. Under certain circumstances it can achieve a lower overall simulation run-time than APF for multi-frame simulations (e.g. poor filesystem/MPI-IO) implementation support for writing simultaneously to the same file and/or if the scene changes significantly between frames). It only is available for multi-frame simulations with a platform truth/image schedule of capture (i.e. one image and/or truth file per frame).

Note Currently, DIRSIG doesn’t transition from UFP to AFP scheduling mode for remainder frames, so efficiency peaks when the number of frames is evenly divisible by the number of processes.

Quick Start

A user can easily load the OpenMPI utilities/libraries into their shell environment using environment modules, the MPI environment environment module is installed with the openmpi package in RHEL-like Linux distributions when deployed via yum.

Load the OpenMPI environment

Load the environment module for OpenMPI into the user’s shell environment:

$ module load mpi/openmpi-x86_64

Run DIRSIG with 1 process per socket (default APF schedule)

Use the --npersocket option to mpirun to have OpenMPI run a single process per socket:

$ mpirun --npersocket 1 dirsig5 --exec=mpi MY_SIM.sim
Important It is critical to include the --exec=mpi flag on any MPI-aware DIRSIG program. DIRSIG has no reliable way of detecting if it was run using mpirun or mpiexec, so it relies on this flag to select the proper scheduler and drivers.

Same as above, but with UFP schedule

Use the DIRSIG5 --mpi_one_event_per_node option to switch the internal scheduler into the unique frame per-process (UFP) mode:

$ mpirun --npersocket 1 dirsig5 --mpi_one_event_per_node --exec=mpi MY_SIM.sim