Turbulent Channel Flow with NekRS + Ginkgo

Simulates turbulent channel flow at $Re_\tau = 590$ using NekRS for the spectral element discretization and time stepping, with Ginkgo replacing NekRS’s built-in PCG for the pressure Poisson solve at every time step.

Compatible with NekRS v26+.

Integration Overview

This example extends the NekRS Poisson example to a full time-dependent simulation. The Ginkgo solver is called at every time step within NekRS’s operator-splitting scheme. The same NekRSOperator wrapper from nekrs/nekrs_ginkgo.hpp is reused.

Key Integration Points

Setup

void UDF_Setup()
{
    auto* pSolver = nrs->fluid->ellipticSolverP;
    if (pSolver) {
        std::cout << "[Ginkgo] Turbulent channel flow setup" << std::endl;
        std::cout << "[Ginkgo] Pressure DOFs: " << pSolver->fieldOffset()
                  << std::endl;
    }
}

Per-Timestep Solve

Ginkgo’s CG replaces NekRS’s built-in PCG at every time step:

void UDF_ExecuteStep(double time, int tstep)
{
    auto* pSolver = nrs->fluid->ellipticSolverP;
    if (!pSolver || tstep == 0) return;

    auto o_rhs = nrs->fluid->o_explicitTerms("pressure");
    auto o_sol = nrs->fluid->o_solution("pressure");

    nekrs_ginkgo_solve(pSolver, static_cast<occa::memory&>(o_rhs),
                       static_cast<occa::memory&>(o_sol));
    ginkgo_solve_count++;

    if (tstep % 500 == 0) {
        std::cout << "[Ginkgo] Step " << tstep << ", time = " << time
                  << ", total Ginkgo solves: " << ginkgo_solve_count
                  << std::endl;
    }
}

Building

NekRS UDFs compile as shared libraries (plugins):

cmake -S . -B build --preset default \
  -DNEKRS_DIR=/path/to/nekrs \
  -DGinkgo_DIR=/path/to/ginkgo
cmake --build build

Running

Note: Running has not been tested end-to-end. The code compiles against the NekRS v26 API, validating the integration pattern, but actually executing it requires a full NekRS case setup.

To run, copy the built shared library into a NekRS turbulent channel case directory (e.g., the channel example bundled with NekRS):

cp -r /path/to/nekrs/examples/channel /tmp/test-channel
cp build/libturbulent_channel.so /tmp/test-channel/
cd /tmp/test-channel && mpirun -np 4 nekrs --setup channel

As with the Poisson example, a production integration would need to disable NekRS’s built-in pressure solve and route it through Ginkgo.

Expected Output (approximate)

[Ginkgo] Turbulent channel flow setup
[Ginkgo] Pressure DOFs: 262144
[Ginkgo] Step 500, time = 1.0, total Ginkgo solves: 500
[Ginkgo] Step 1000, time = 2.0, total Ginkgo solves: 1000