Performance

To ensure Synthesizer is performant enough to handle the large dynamic range of possible input datasets we have put a lot of effort into optimising the codebase. Needless to say, we are always looking for ways to improve performance further, so if you have any suggestions or find any bottlenecks, please do not hesitate to open an issue on GitHub.

Optimisations

We have implemented a number of performance optimisations, including:

  • Using C++ extensions for computationally intensive tasks.

  • Using OpenMP for shared memory parallelism to avoid the GIL bottleneck in Python.

  • Reducing memory allocations and copies as much as possible (including removing copies inherent during unyt conversion operations).

  • User-controllable floating-point precision for inputs and outputs, halving memory footprints where reduced precision is acceptable (see Controlling Precision).

Build options

By default the extensions are compiled for the baseline target of whatever architecture you are on. On x86-64 that means SSE2 and 128-bit vectors, while every current HPC CPU has AVX2 or wider, so the default build leaves half the vector width unused. Set NATIVE=1 to compile for the instruction set of the machine doing the build:

NATIVE=1 WITH_OPENMP=1 pip install .

This is opt-in rather than the default because the resulting binary will not run on an older CPU. That matters in two common cases: when a cluster’s login node and its compute nodes are different generations, and when building a wheel for distribution. Build on a node of the same generation as the one you will run on, or pass an explicit target such as CFLAGS="-march=znver2" instead. A build left on the baseline says so in build_synth.log.

On an AMD EPYC 7H12 this is worth about 1.2x on the CIC spectra extraction at one thread, and about 2.2x on the separable attenuation kernel, where the architecture flag is what lets the C library replace its scalar exp with a vector one.

NUMA memory placement

Grids and output buffers are allocated and filled by a single Python thread, so every page of them lands on one NUMA domain. Threads running on the other domains then read all of that data remotely, and the total bandwidth is capped by one memory controller no matter how many threads are used. On a COSMA8 node (two EPYC 7H12, eight NUMA domains of sixteen cores) that caps every streaming kernel near 37 GB/s, which shows up as scaling that flattens out once the thread count passes the size of a single domain.

Interleaving the pages across all domains lifts that cap:

SYNTHESIZER_NUMA_INTERLEAVE=1 python my_script.py

The variable is read when synthesizer is imported, before any grid or output array exists. The policy applies to the importing thread and to threads it creates afterwards, which covers the large arrays because they are allocated and filled by the main Python thread, and covers the OpenMP workers because they are normally spawned later and inherit it.

numactl --interleave=all <command> sets the same policy from outside the process and needs no support from Synthesizer; the two measure the same to within a few percent. Prefer numactl whenever threads may already exist when Synthesizer is imported, since those keep the default policy. That includes threads you started yourself, and also the OpenMP pool, which persists once created: if anything ran a parallel region earlier in the same process, a SciPy or scikit-learn call for instance, those workers predate the import and will not pick the policy up.

Interleaving is not free. Below about eight threads everything a thread reads would otherwise have been local, and spreading it costs 5-20%. It is off by default and only worth setting when using more cores than one NUMA domain holds. On a two-socket EPYC 7H12 node (eight domains of sixteen cores) it is worth 1.9x at 32 threads and 2.2x at 64 on particle spectra extraction, and 3.2x and 5.4x respectively on the flux conversion and scaling kernels.

Idle OpenMP threads

When a parallel region finishes, most OpenMP runtimes keep the worker threads spinning for a while so the next region can start without waking them. The serial work between regions (Python bookkeeping, tree construction, unit handling) then shares the node with every spinning thread, and at high thread counts it runs measurably slower than it does on one thread.

If your workload alternates between short parallel kernels and serial Python, let the idle threads sleep instead:

OMP_WAIT_POLICY=passive python my_script.py

The trade-off is that each parallel region then pays to wake its threads, so this helps when the serial gaps dominate and hurts when one parallel kernel follows another back to back. It is a runtime setting, so try both on your own workload.

Profiling Suite

Note

Before running the profiling suite, you will need to download the grids. See the Downloading Grids documentation for details.

To ensure the code remains performant, we maintain a comprehensive profiling suite to test the performance of the codebase. The profiling scripts and documentation can be found in the profiling directory of the repository.

Profiling Categories:

  • Particle and Wavelength Scaling: How individual operations scale with problem size (number of particles or wavelength elements)

  • Pipeline Profiling: Real-world benchmarks with multiple operations performed in sequence

  • Thread Scaling: How performance scales with OpenMP thread count for fixed problem sizes

  • MPI Scaling: How the Pipeline scales across MPI ranks, for fixed total work (strong) and fixed work per rank (weak)

The profiling suite includes scripts to:

  • Run timing benchmarks for various operations and configurations

  • Profile memory usage with configurable sampling frequencies

  • Analyse and visualise profiling results

  • Generate the performance plots shown in this documentation

See the profiling README for details on running the profiling suite and reproducing these benchmarks.

Hardware Specifications

The benchmarks shown in this documentation were run on the Cosma8 HPC at Durham University. The output of the lscpu command is shown below, which gives an idea of the hardware used for these tests:

CPU(s):                   256
Model name:               AMD EPYC 7763 64-Core Processor
Thread(s) per core:       2
Core(s) per socket:       64
Socket(s):                2
NUMA node(s):             8

All the benchmark plots were run on nodes of this type (the cosma8-milan partition), using 32 threads unless otherwise specified. The NATIVE and NUMA interleave figures quoted above were measured separately, on COSMA8 nodes with two AMD EPYC 7H12 processors.

Memory Footprint Note

Note

The memory plots in this documentation measure different aspects of memory usage depending on the benchmark:

  • Pipeline profiling: Shows RSS (Resident Set Size) memory sampled at high frequency during execution. This captures the total memory footprint including transient spikes.

  • Individual operation benchmarks: Show the size of the final objects stored in memory (e.g., the generated spectra or photometry data). While these represent the permanent memory cost added to your session, there may be transient spikes in memory usage during the actual computation that are slightly higher than these values.

Performance Benchmarks