LAST SEEN ONLINE:
NOV 22, 2026 11:36 UTC
LOCAL_TIME_SYNC:
--:--:-- | -- _
← Digital notebook
CFD 30 min read First-Principles CFD / Module 4

Module 4: Algebraic Systems and Numerical Solvers

Learn how discretized CFD equations become systems of algebraic equations. Explore matrix representations, boundary conditions, direct and iterative solvers, residuals, convergence, conditioning, and how to implement the foundations of a CFD solver in C++.

Introduction

In Module 3, we learned how to convert continuous differential equations into discrete algebraic equations using finite differences. We approximated derivatives, studied truncation error, and derived a discrete form of the one-dimensional heat-conduction equation.

But obtaining a discrete equation is only part of the problem. A practical CFD simulation must solve thousands, millions, or even billions of coupled algebraic equations to determine the unknown pressure, velocity, temperature, and other physical quantities.

This module explains what happens between discretizing a governing equation and obtaining a numerical solution.

We will start with a simple one-dimensional heat-conduction problem, construct its algebraic equations, express them in matrix form, and examine different numerical methods for solving them. We will also learn how boundary conditions enter the system, what residuals mean, why convergence matters, and how these concepts translate into C++ code.

The objective is to understand the numerical machinery behind a CFD solver rather than treating a commercial solver as a black box.

Learning Objectives

By the end of this module, you should be able to:

  • Explain how discretized differential equations form systems of algebraic equations.
  • Represent a numerical problem using matrix notation, Ax=bA\mathbf{x}=\mathbf{b}.
  • Derive a matrix system for one-dimensional steady heat conduction.
  • Distinguish direct solvers from iterative solvers.
  • Understand Jacobi, Gauss–Seidel, and successive over-relaxation methods.
  • Explain residuals, convergence criteria, and stopping conditions.
  • Recognize the importance of matrix sparsity, diagonal dominance, and conditioning.
  • Implement a basic iterative solver in C++.
  • Understand how numerical linear algebra fits into a larger CFD algorithm.

1. From Discretized Equations to an Algebraic System

Consider the steady, one-dimensional heat-conduction equation in a homogeneous solid with constant thermal conductivity and no internal heat generation:

d2Tdx2=0\frac{d^2T}{dx^2}=0

As discussed in Module 3, using the central finite-difference approximation gives

Ti+1−2Ti+Ti−1Δx2=0\frac{T_{i+1}-2T_i+T_{i-1}}{\Delta x^2}=0

Multiplying both sides by Δx2\Delta x^2 gives

Ti+1−2Ti+Ti−1=0T_{i+1}-2T_i+T_{i-1}=0

Rearranging,

−Ti−1+2Ti−Ti+1=0- T_{i-1}+2T_i-T_{i+1}=0

Equivalently,

Ti=Ti−1+Ti+12T_i=\frac{T_{i-1}+T_{i+1}}{2}

These equations state that the temperature at an interior grid point equals the average of its two neighboring temperatures.

However, each equation contains multiple unknown temperatures. We must solve all the equations together to obtain the temperature distribution.

A simple numerical example

Suppose a one-dimensional wall has:

  • Length: L=1 mL=1\ \mathrm{m}.
  • Left boundary temperature: TL=100∘CT_L=100^\circ\mathrm{C}.
  • Right boundary temperature: TR=0∘CT_R=0^\circ\mathrm{C}.
  • Four equally spaced intervals, giving Δx=0.25 m\Delta x=0.25\ \mathrm{m}.

There are five grid points:

x0=0,x1=0.25,x2=0.50,x3=0.75,x4=1x_0=0,\quad x_1=0.25,\quad x_2=0.50,\quad x_3=0.75,\quad x_4=1

The boundary temperatures are known:

T0=100,T4=0T_0=100,\qquad T_4=0

The unknowns are the three interior temperatures:

T1,T2,T3T_1,\qquad T_2,\qquad T_3

Applying the discrete equation at each interior point gives

2T1−T2=T0=1002T_1-T_2=T_0=100 −T1+2T2−T3=0-T_1+2T_2-T_3=0 −T2+2T3=T4=0-T_2+2T_3=T_4=0

We now have three equations and three unknowns. This is a linear algebra problem.

Solving the equations gives

T1=75∘C,T2=50∘C,T3=25∘CT_1=75^\circ\mathrm{C},\qquad T_2=50^\circ\mathrm{C},\qquad T_3=25^\circ\mathrm{C}

The temperature decreases linearly from the hot wall to the cold wall, as expected from the physics.

The important idea is that a differential equation has been transformed into a finite collection of algebraic equations that a computer can solve.

2. Matrix Representation: The Form Ax = b

Writing individual equations is manageable for three unknowns. For a real CFD mesh, there may be millions of unknowns, so we need a systematic representation.

A linear system can be written as

Ax=bA\mathbf{x}=\mathbf{b}

where:

  • AA is the coefficient matrix.
  • x\mathbf{x} is the vector of unknowns.
  • b\mathbf{b} is the right-hand-side vector.

For our heat-conduction example,

[2−10−12−10−12][T1T2T3]=[10000]\begin{bmatrix} 2 & -1 & 0\\ -1 & 2 & -1\\ 0 & -1 & 2 \end{bmatrix} \begin{bmatrix} T_1\\ T_2\\ T_3 \end{bmatrix} = \begin{bmatrix} 100\\ 0\\ 0 \end{bmatrix}

The matrix contains the coefficients associated with each unknown.

The temperature vector contains the unknown interior temperatures, and the right-hand-side vector contains contributions from the prescribed boundary temperatures.

What does each row represent?

Each row corresponds to one discrete equation.

The first row,

2T1−T2=1002T_1-T_2=100

represents the equation at grid point 1.

The second row,

−T1+2T2−T3=0-T_1+2T_2-T_3=0

represents the equation at grid point 2.

The third row,

−T2+2T3=0-T_2+2T_3=0

represents the equation at grid point 3.

Each row connects one unknown to its neighbors. This local connectivity is fundamental to the structure of CFD matrices.

General matrix notation

For NN unknowns, the system can be expressed as

[a11a12⋯a1Na21a22⋯a2N⋮⋮⋱⋮aN1aN2⋯aNN][x1x2⋮xN]=[b1b2⋮bN]\begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1N}\\ a_{21} & a_{22} & \cdots & a_{2N}\\ \vdots & \vdots & \ddots & \vdots\\ a_{N1} & a_{N2} & \cdots & a_{NN} \end{bmatrix} \begin{bmatrix} x_1\\ x_2\\ \vdots\\ x_N \end{bmatrix} = \begin{bmatrix} b_1\\ b_2\\ \vdots\\ b_N \end{bmatrix}

In a CFD problem, the unknown vector might represent temperature, pressure, one velocity component, or a collection of coupled variables.

For example, a simple thermal problem may have one temperature unknown per cell. A more complex incompressible-flow solver may need to handle pressure and multiple velocity components, with additional equations connecting these quantities.

3. How Boundary Conditions Enter the Matrix

Boundary conditions are not merely settings supplied to the solver. They influence the algebraic equations and therefore the numerical solution.

Consider the first interior equation:

2T1−T2=T02T_1-T_2=T_0

Since the left boundary temperature is known,

T0=TLT_0=T_L

we substitute the boundary value into the equation:

2T1−T2=TL2T_1-T_2=T_L

The boundary temperature becomes a contribution to the right-hand side.

Similarly, at the last interior point,

−TN−1+2TN=TR-T_{N-1}+2T_N=T_R

for a grid whose interior unknowns are indexed from 1 to NN.

This is one common way to incorporate prescribed-temperature, or Dirichlet, boundary conditions into a matrix system.

Dirichlet boundary condition

A Dirichlet condition prescribes the value of the unknown at a boundary.

For temperature,

T=TwT=T_w

where TwT_w is the specified wall temperature.

Examples include a wall maintained at 100∘C100^\circ\mathrm{C} or a cooling surface maintained at 20∘C20^\circ\mathrm{C}.

Neumann boundary condition

A Neumann condition prescribes a derivative or flux.

For example, a prescribed temperature gradient is

∂T∂n=g\frac{\partial T}{\partial n}=g

where nn denotes the direction normal to the boundary.

For a stationary solid with Fourier heat conduction,

q=−k∇T\mathbf{q}=-k\nabla T

so specifying heat flux is closely related to specifying a normal temperature gradient.

An insulated wall is often represented by zero normal heat flux:

q⋅n=0\mathbf{q}\cdot\mathbf{n}=0

For constant, nonzero thermal conductivity, this implies

∂T∂n=0\frac{\partial T}{\partial n}=0

A Neumann condition usually modifies the discrete equation at the boundary rather than simply inserting a known temperature into the right-hand side.

Why this matters in CFD

An incorrect boundary treatment can produce a plausible-looking but physically incorrect result.

For example, accidentally prescribing a temperature where a heat flux was intended changes the physical problem. Similarly, incorrectly handling a velocity boundary can alter the flow rate or disturb mass conservation.

Boundary conditions must therefore be consistent with both the physics and the discrete equations.

4. Understanding Matrix Structure

Not all matrices are equally expensive to solve. The pattern of nonzero coefficients is particularly important in CFD.

Dense matrices

A dense matrix contains nonzero values in most or all positions.

For a system with NN unknowns, a dense matrix stores approximately N2N^2 coefficients.

If N=10,000N=10{,}000, the matrix contains 10810^8 entries. Storing those entries as 64-bit floating-point numbers requires approximately 800 MB for the matrix alone.

For large CFD simulations, this can become prohibitively expensive.

Sparse matrices

A sparse matrix contains mostly zeros.

In the one-dimensional heat-conduction problem, each interior equation connects only to the previous point, the current point, and the next point. The matrix is therefore tridiagonal:

A=[2−100⋯−12−10⋯0−12−1⋯00−12⋯⋮⋮⋮⋮⋱]A= \begin{bmatrix} 2 & -1 & 0 & 0 & \cdots\\ -1 & 2 & -1 & 0 & \cdots\\ 0 & -1 & 2 & -1 & \cdots\\ 0 & 0 & -1 & 2 & \cdots\\ \vdots & \vdots & \vdots & \vdots & \ddots \end{bmatrix}

Each row contains only a small number of nonzero entries.

A three-dimensional finite-volume cell usually interacts directly with a limited number of neighboring cells. For a simple structured mesh, that might mean six face-neighbors plus the cell itself. More general meshes have varying numbers of neighbors.

This local interaction creates sparse systems.

Why sparsity matters

Sparse storage avoids wasting memory on zeros. Sparse algorithms also avoid unnecessary arithmetic involving zero coefficients.

For large CFD problems, the difference between dense and sparse methods can determine whether a simulation is practical.

A production CFD solver therefore usually does not store a large coefficient matrix as a conventional two-dimensional array. It may use compressed sparse row storage, specialized matrix structures, or matrix-free operations.

For our first C++ learning examples, however, we can initially use simple vectors and loops before introducing more advanced sparse data structures.

5. Direct Solvers and Iterative Solvers

Once we have assembled the system

Ax=bA\mathbf{x}=\mathbf{b}

we need a numerical method to solve it.

Two broad approaches are direct solvers and iterative solvers.

Direct solvers

A direct solver uses a sequence of algebraic operations to obtain the solution, subject to floating-point rounding error.

For example, Gaussian elimination transforms a system into an equivalent triangular system that can be solved by back substitution.

For a small system,

2x−y=1−x+2y=0\begin{aligned} 2x-y&=1\\ -x+2y&=0 \end{aligned}

we can solve the second equation for

y=x2y=\frac{x}{2}

Substituting into the first equation,

2x−x2=12x-\frac{x}{2}=1

which gives

x=23,y=13x=\frac{2}{3},\qquad y=\frac{1}{3}

A computer generalizes these operations to much larger systems.

Common direct methods include:

  • Gaussian elimination.
  • LU factorization.
  • Cholesky factorization for suitable matrices.
  • Specialized direct methods for structured systems.

Direct solvers can be very effective for small and medium-sized problems, but their memory and computational costs can become large for huge systems. Sparse direct solvers exploit the zero structure of the matrix, although factorization may introduce additional nonzero entries called fill-in.

Iterative solvers

An iterative solver begins with an initial estimate and repeatedly improves it.

Instead of computing the exact solution through a single finite sequence of elimination operations, it generates a sequence of approximations:

x(0),x(1),x(2),…\mathbf{x}^{(0)},\quad \mathbf{x}^{(1)},\quad \mathbf{x}^{(2)},\quad \ldots

The superscript indicates the iteration number.

The method stops when the solution estimate satisfies a specified convergence criterion or when a maximum iteration limit is reached.

Common iterative methods include:

  • Jacobi iteration.
  • Gauss–Seidel iteration.
  • Successive over-relaxation (SOR).
  • Conjugate gradient for suitable systems.
  • GMRES for more general systems.
  • BiCGSTAB for certain nonsymmetric systems.

The choice depends on the matrix properties, computational resources, and the requirements of the physical problem.

Comparison

FeatureDirect solverIterative solver
Basic approachAlgebraic elimination or factorizationRepeatedly improves an estimate
Stopping ruleFinite factorization and solve operationsResidual or iteration criterion
Memory useCan be high for large sparse systemsOften lower for sparse systems
Large CFD systemsMay be costlyOften preferred
Sensitivity to matrix propertiesDepends on factorization and conditioningDepends strongly on method and matrix properties
Main limitationFactorization cost and memoryMay converge slowly or fail to converge

The distinction is not that direct solvers are always accurate and iterative solvers are inaccurate. Both are subject to numerical errors, and iterative solvers can produce highly accurate solutions when used appropriately.

6. Jacobi Iteration: Solving One Equation at a Time

The Jacobi method is one of the simplest iterative methods. It is useful for learning because each new estimate is calculated using values from the previous iteration.

Consider a linear system with diagonal coefficients aiia_{ii}:

∑j=1Naijxj=bi\sum_{j=1}^{N}a_{ij}x_j=b_i

Separating the diagonal term gives

aiixi+∑j≠iaijxj=bia_{ii}x_i+\sum_{j\ne i}a_{ij}x_j=b_i

Rearranging,

xi=1aii(bi−∑j≠iaijxj)x_i=\frac{1}{a_{ii}} \left(b_i-\sum_{j\ne i}a_{ij}x_j\right)

The Jacobi update formula is

xi(k+1)=1aii(bi−∑j≠iaijxj(k))x_i^{(k+1)} = \frac{1}{a_{ii}} \left( b_i-\sum_{j\ne i}a_{ij}x_j^{(k)} \right)

Every new value uses the previous iteration’s values.

Jacobi iteration for heat conduction

For our interior heat-conduction equations,

−Ti−1+2Ti−Ti+1=bi-T_{i-1}+2T_i-T_{i+1}=b_i

the update becomes

Ti(k+1)=Ti−1(k)+Ti+1(k)+bi2T_i^{(k+1)} = \frac{T_{i-1}^{(k)}+T_{i+1}^{(k)}+b_i}{2}

For the first and last interior points, the prescribed boundary temperatures are included in the right-hand side.

For the example with boundary temperatures 100∘C100^\circ\mathrm{C} and 0∘C0^\circ\mathrm{C}, the system is

2T1−T2=100−T1+2T2−T3=0−T2+2T3=0\begin{aligned} 2T_1-T_2&=100\\ -T_1+2T_2-T_3&=0\\ -T_2+2T_3&=0 \end{aligned}

Starting with

T1(0)=T2(0)=T3(0)=0T_1^{(0)}=T_2^{(0)}=T_3^{(0)}=0

the first iteration gives

T1(1)=100+02=50T_1^{(1)}=\frac{100+0}{2}=50 T2(1)=0+02=0T_2^{(1)}=\frac{0+0}{2}=0 T3(1)=0+02=0T_3^{(1)}=\frac{0+0}{2}=0

Thus,

T(1)=[5000]\mathbf{T}^{(1)}= \begin{bmatrix} 50\\ 0\\ 0 \end{bmatrix}

For the next iteration,

T1(2)=100+T2(1)2=50T_1^{(2)}=\frac{100+T_2^{(1)}}{2}=50 T2(2)=T1(1)+T3(1)2=25T_2^{(2)}=\frac{T_1^{(1)}+T_3^{(1)}}{2}=25 T3(2)=T2(1)2=0T_3^{(2)}=\frac{T_2^{(1)}}{2}=0

giving

T(2)=[50250]\mathbf{T}^{(2)}= \begin{bmatrix} 50\\ 25\\ 0 \end{bmatrix}

The estimate is moving toward the final solution:

T=[755025]\mathbf{T}= \begin{bmatrix} 75\\ 50\\ 25 \end{bmatrix}

It does not reach the exact solution in just two iterations. The process continues until the solution satisfies the stopping criterion.

A key implementation detail

The Jacobi method requires two sets of values:

  • The old solution, x(k)\mathbf{x}^{(k)}.
  • The new solution, x(k+1)\mathbf{x}^{(k+1)}.

All updates must use the old vector. If the new values are written directly into the old vector while traversing the grid, the algorithm is no longer standard Jacobi iteration.

This distinction matters when implementing numerical algorithms in C++.

7. Gauss–Seidel Iteration

Gauss–Seidel uses newly calculated values as soon as they become available.

Its update formula is

xi(k+1)=1aii(bi−∑j<iaijxj(k+1)−∑j>iaijxj(k))x_i^{(k+1)} = \frac{1}{a_{ii}} \left( b_i -\sum_{j<i}a_{ij}x_j^{(k+1)} -\sum_{j>i}a_{ij}x_j^{(k)} \right)

The first summation uses values already updated during the current iteration. The second uses values that have not yet been updated and therefore still belong to the previous iteration.

For our one-dimensional heat-conduction example, sweeping from left to right gives

T1(k+1)=100+T2(k)2T_1^{(k+1)}=\frac{100+T_2^{(k)}}{2} T2(k+1)=T1(k+1)+T3(k)2T_2^{(k+1)}=\frac{T_1^{(k+1)}+T_3^{(k)}}{2} T3(k+1)=T2(k+1)2T_3^{(k+1)}=\frac{T_2^{(k+1)}}{2}

Starting from zero, the first iteration is

T1(1)=50T_1^{(1)}=50 T2(1)=25T_2^{(1)}=25 T3(1)=12.5T_3^{(1)}=12.5

The update has propagated information from the hot boundary through the grid during a single sweep.

For this particular example and update ordering, Gauss–Seidel makes more progress per sweep than Jacobi. This does not mean it is always faster for every matrix or implementation. Convergence depends on the system, and parallel implementations may favor different algorithms.

8. Successive Over-Relaxation (SOR)

Successive over-relaxation extends Gauss–Seidel by multiplying the update correction by a relaxation factor, ω\omega.

Let x^i(k+1)\widehat{x}_i^{(k+1)} be the value obtained from a standard Gauss–Seidel update. Then SOR uses

xi(k+1)=(1−ω)xi(k)+ωx^i(k+1)x_i^{(k+1)} = (1-\omega)x_i^{(k)} + \omega\widehat{x}_i^{(k+1)}

The parameter ω\omega controls the amount of correction applied.

  • If ω=1\omega=1, SOR reduces to Gauss–Seidel.
  • If 0<ω<10<\omega<1, the update is under-relaxed.
  • If 1<ω<21<\omega<2, the update is over-relaxed; for certain matrix classes, this can accelerate convergence.

An unsuitable relaxation factor can slow convergence or cause divergence. The useful range and optimal value depend on the matrix and problem.

In CFD, relaxation is also used to stabilize coupled or nonlinear solution procedures. The relaxation used in an outer CFD iteration is related in spirit to SOR, but it is not necessarily the same algorithm as applying SOR to a linear system.

9. Residuals: How Do We Know Whether the Equations Are Satisfied?

A solver needs a way to measure how well the current estimate satisfies the algebraic system.

For

Ax=bA\mathbf{x}=\mathbf{b}

the residual vector is commonly defined as

r=b−Ax\mathbf{r}=\mathbf{b}-A\mathbf{x}

Some software uses the opposite sign, Ax−bA\mathbf{x}-\mathbf{b}. The sign convention changes the direction of the residual, but not its magnitude.

If the exact algebraic solution is reached, then

r=0\mathbf{r}=\mathbf{0}

For an approximate solution, the residual is generally nonzero.

Residual of a single equation

Consider

2x−y=12x-y=1

If our current estimate is x=0.4x=0.4 and y=0.2y=0.2, the left-hand side is

2(0.4)−0.2=0.62(0.4)-0.2=0.6

The residual using r=b−Axr=b-Ax is

r=1−0.6=0.4r=1-0.6=0.4

The equation is not yet satisfied.

If we improve the estimate to x=0.6x=0.6 and y=0.2y=0.2, then

2(0.6)−0.2=12(0.6)-0.2=1

and the residual becomes zero for this equation.

A system with many equations has a residual for every row. We usually combine them into a single measure.

Common residual norms

The L2L_2 norm is

∥r∥2=∑i=1Nri2\|\mathbf{r}\|_2 = \sqrt{\sum_{i=1}^{N}r_i^2}

The infinity norm is

∥r∥∞=max⁡i∣ri∣\|\mathbf{r}\|_\infty = \max_i|r_i|

The L2L_2 norm measures the overall magnitude of the residual vector, while the infinity norm reports the largest absolute residual component.

A solver may use one of these measures or another appropriate norm to determine convergence.

Absolute and relative residuals

An absolute residual depends on the scale of the equations. A relative residual compares the current residual with a reference scale, for example,

rrel=∥b−Ax∥2∥b∥2r_{\mathrm{rel}} = \frac{\|\mathbf{b}-A\mathbf{x}\|_2} {\|\mathbf{b}\|_2}

when ∥b∥2≠0\|\mathbf{b}\|_2\ne 0.

This helps compare convergence across some differently scaled systems. However, the precise definition of a solver’s reported residual may include normalization based on initial residuals, matrix coefficients, or other quantities. Always check the convention used by the software.

If the right-hand side is zero, the formula above is undefined. In that case, a different normalization or an absolute criterion may be appropriate.

Why residuals matter in CFD

During a CFD simulation, residuals help monitor how well the discrete equations are being satisfied.

A decreasing residual often indicates progress, but a low residual alone does not prove that the solution is physically correct. The solution may still have:

  • Incorrect boundary conditions.
  • An inadequate mesh.
  • Incorrect material properties.
  • Significant discretization error.
  • Poor mass or energy conservation.
  • A physically inappropriate turbulence or multiphase model.

Residuals are one important diagnostic, not a complete verification and validation strategy.

10. Convergence Criteria and Stopping Conditions

An iterative solver should not continue indefinitely. We need a rule for deciding when to stop.

A common criterion is

∥r∥<ε\|\mathbf{r}\| < \varepsilon

where ε\varepsilon is a prescribed tolerance.

For example,

ε=10−8\varepsilon=10^{-8}

means the chosen residual measure must fall below 10−810^{-8} before the residual-based stopping criterion is satisfied.

The meaning of this number depends on the residual definition and the scaling of the problem. A tolerance of 10−810^{-8} is not automatically necessary or appropriate for every CFD equation.

A robust implementation may stop when either:

  1. The residual reaches the specified tolerance.
  2. A maximum number of iterations is reached.

If the maximum iteration count is reached first, the solver should report that it did not meet the convergence criterion.

Monitoring solution changes

Another useful diagnostic is the change between successive estimates:

Δx(k)=x(k+1)−x(k)\Delta\mathbf{x}^{(k)} = \mathbf{x}^{(k+1)}-\mathbf{x}^{(k)}

A small change can indicate that the iteration is stabilizing. However, small changes do not necessarily mean that the equations are satisfied. An iteration can stagnate far from the solution.

Therefore, residuals and solution changes should be interpreted together.

Linear convergence versus CFD convergence

It is important to distinguish two levels of convergence.

Linear solver convergence means that the iterative algorithm has reduced the residual of a particular algebraic system.

Overall CFD convergence concerns the behavior of the coupled governing equations as the simulation progresses.

A CFD solver may repeatedly solve its linear systems accurately while the outer flow solution is still changing. Conversely, an overly loose linear-solver tolerance can make the outer iteration inefficient or unreliable.

A practical simulation should monitor equation residuals and relevant physical quantities, such as mass imbalance, pressure drop, drag, lift, heat transfer, and other quantities of interest.

11. Why Do Some Systems Converge Slowly?

The behavior of an iterative solver depends on the mathematical properties of the matrix.

Diagonal dominance

A matrix is strictly diagonally dominant by rows if

∣aii∣>∑j≠i∣aij∣|a_{ii}|>\sum_{j\ne i}|a_{ij}|

for every row ii.

In simple terms, the magnitude of each diagonal coefficient exceeds the sum of the magnitudes of the other coefficients in that row.

Strict diagonal dominance is a useful sufficient condition for convergence of Jacobi and Gauss–Seidel methods, although it is not a necessary condition. Other matrix properties can also guarantee convergence.

For example, the matrix

A=[4−1−13]A= \begin{bmatrix} 4 & -1\\ -1 & 3 \end{bmatrix}

is strictly diagonally dominant because

4>∣−1∣,3>∣−1∣4>|-1|,\qquad 3>|-1|

This matrix has a strong diagonal structure that is often favorable for basic iterative methods.

Conditioning

Conditioning describes how sensitive a problem’s solution is to perturbations in its input data.

For a nonsingular matrix, the 2-norm condition number is

κ2(A)=∥A∥2∥A−1∥2\kappa_2(A)=\|A\|_2\|A^{-1}\|_2

A large condition number indicates that small perturbations in the matrix or right-hand side may cause relatively large changes in the solution.

A poorly conditioned system can therefore make accurate numerical computation more difficult.

Mesh refinement can affect conditioning. For example, discretizing diffusion equations on very fine grids often produces systems that are more challenging for simple iterative methods. The exact behavior depends on the discretization, boundary conditions, geometry, and scaling.

Why more iterations may not be the answer

If a solver converges slowly, increasing the maximum iteration count may help, but it does not address every underlying cause.

Possible improvements include:

  • Improving the matrix scaling.
  • Choosing a more suitable iterative method.
  • Using a preconditioner.
  • Applying a multigrid method.
  • Reviewing mesh quality and boundary conditions.
  • Checking whether the physical model or discretization produces an ill-conditioned system.

In modern CFD, preconditioning and multigrid methods are particularly important because they can substantially accelerate convergence for large systems.

12. Preconditioning: Making a System Easier to Solve

Suppose we need to solve

Ax=bA\mathbf{x}=\mathbf{b}

A preconditioner is an operator, commonly denoted MM, that approximates some useful properties of AA while being cheaper to apply or invert than the original system.

One form of a preconditioned system is

M−1Ax=M−1bM^{-1}A\mathbf{x}=M^{-1}\mathbf{b}

The solution remains the same when the transformation is valid and applied consistently, but the new system may be easier for an iterative method to solve.

A good preconditioner can reduce the number of iterations needed to reach a target residual.

An intuitive analogy

Imagine walking through a mountain range to reach a destination. A poor route may involve many steep climbs and sharp turns. A better route changes the way you approach the same destination, making the journey easier without changing where you need to end up.

Preconditioning is similar: it changes the numerical problem’s representation to improve the efficiency of the solution process.

Common preconditioning approaches include diagonal (Jacobi) preconditioning, incomplete LU factorization, and more advanced methods used with Krylov-subspace solvers.

The effectiveness of a preconditioner depends on the equation and the solver. It is not a universal shortcut that improves every system equally.

13. From One Equation to Coupled CFD Equations

The heat-conduction example contains one unknown temperature at each interior grid point. A fluid-flow simulation is more complicated because multiple physical quantities are coupled.

For incompressible flow, the principal governing equations include continuity,

∇⋅u=0\nabla\cdot\mathbf{u}=0

and momentum conservation,

ρ(∂u∂t+u⋅∇u)=−∇p+μ∇2u+ρf\rho\left( \frac{\partial\mathbf{u}}{\partial t} + \mathbf{u}\cdot\nabla\mathbf{u} \right) = -\nabla p+\mu\nabla^2\mathbf{u}+\rho\mathbf{f}

Here, velocity influences momentum transport, while pressure helps enforce the incompressibility constraint.

When these equations are discretized, they produce algebraic systems involving the discrete velocity and pressure fields.

In a simplified description, the coupled system can be written as

[AuGD0][up]=[bubp]\begin{bmatrix} A_u & G\\ D & 0 \end{bmatrix} \begin{bmatrix} \mathbf{u}\\ p \end{bmatrix} = \begin{bmatrix} \mathbf{b}_u\\ \mathbf{b}_p \end{bmatrix}

where:

  • AuA_u represents the discretized momentum operator.
  • GG represents a discrete pressure-gradient operator.
  • DD represents a discrete divergence operator.
  • u\mathbf{u} represents the discrete velocity unknowns.
  • pp represents the discrete pressure unknowns.

This is a schematic representation of a coupled incompressible-flow system, not the exact matrix assembled by every CFD solver. The pressure block and right-hand-side structure depend on the formulation and treatment of boundary conditions.

Pressure–velocity coupling

Pressure is not simply another independent transported scalar in incompressible flow. Its role is closely connected to satisfying the continuity equation.

Algorithms such as SIMPLE and PISO address this pressure–velocity coupling through correction or prediction steps. These algorithms are commonly used in finite-volume CFD solvers, although their precise implementation differs across software and problem types.

A practical CFD code may therefore contain several nested numerical procedures:

  1. An outer loop updates the nonlinear flow solution.
  2. Momentum equations are assembled and solved.
  3. A pressure-correction equation is constructed and solved.
  4. Velocity and pressure are corrected.
  5. Additional equations, such as energy or turbulence transport, are solved.
  6. Convergence and physical conservation are monitored.

The linear solvers introduced in this module are building blocks inside this larger procedure.

14. Nonlinear Equations and Linearization

Not every discretized CFD problem leads directly to a single linear system.

The momentum equation contains the convective term

u⋅∇u\mathbf{u}\cdot\nabla\mathbf{u}

Because the velocity appears in both the transported quantity and the transport velocity, this term is nonlinear.

A nonlinear discrete system can be represented abstractly as

F(x)=0\mathbf{F}(\mathbf{x})=\mathbf{0}

where x\mathbf{x} contains the unknown fields and F\mathbf{F} represents the residuals of the discrete governing equations.

A common strategy is to construct a sequence of linear systems that approximate the nonlinear problem around the current estimate.

For example, a Newton-type method uses the linearized system

J(x(k))Δx=−F(x(k))J(\mathbf{x}^{(k)})\Delta\mathbf{x} = -\mathbf{F}(\mathbf{x}^{(k)})

where JJ is the Jacobian matrix of derivatives of the residuals with respect to the unknowns.

The estimate is then updated:

x(k+1)=x(k)+Δx\mathbf{x}^{(k+1)} = \mathbf{x}^{(k)}+\Delta\mathbf{x}

CFD solvers may instead use segregated methods, Picard-type linearization, under-relaxation, or specialized pressure–velocity algorithms.

The important distinction is that solving a linear system is not always the same as solving the complete physical problem. A nonlinear CFD calculation may require repeatedly assembling and solving linearized systems until the coupled equations converge.

15. Implementing Jacobi Iteration in C++

We will now implement a basic Jacobi solver for the one-dimensional heat-conduction example.

The goal is educational: to understand how a numerical algorithm translates into loops, arrays, residual calculations, and a stopping criterion.

We will solve the three-equation system

2T1−T2=100−T1+2T2−T3=0−T2+2T3=0\begin{aligned} 2T_1-T_2&=100\\ -T_1+2T_2-T_3&=0\\ -T_2+2T_3&=0 \end{aligned}

The expected solution is

T=[755025]\mathbf{T}= \begin{bmatrix} 75\\ 50\\ 25 \end{bmatrix}

Create a file named jacobi_heat.cpp.

#include <algorithm>
#include <cmath>
#include <iomanip>
#include <iostream>
#include <vector>

int main()
{
    // Three interior temperature unknowns.
    const int n = 3;

    // Matrix A and right-hand-side vector b.
    const std::vector<std::vector<double>> A = {
        { 2.0, -1.0,  0.0 },
        {-1.0,  2.0, -1.0 },
        { 0.0, -1.0,  2.0 }
    };

    const std::vector<double> b = {
        100.0, 0.0, 0.0
    };

    // Initial guess.
    std::vector<double> x(n, 0.0);

    // Separate vector for the next iteration.
    std::vector<double> xNew(n, 0.0);

    const int maxIterations = 1000;
    const double tolerance = 1.0e-10;

    bool converged = false;

    std::cout << std::scientific
              << std::setprecision(6);

    for (int iteration = 1;
         iteration <= maxIterations;
         ++iteration)
    {
        // Jacobi update: every new value uses only x.
        for (int i = 0; i < n; ++i)
        {
            double sum = 0.0;

            for (int j = 0; j < n; ++j)
            {
                if (j != i)
                {
                    sum += A[i][j] * x[j];
                }
            }

            xNew[i] = (b[i] - sum) / A[i][i];
        }

        // Compute the residual norm using xNew.
        double residualSquared = 0.0;

        for (int i = 0; i < n; ++i)
        {
            double Ax = 0.0;

            for (int j = 0; j < n; ++j)
            {
                Ax += A[i][j] * xNew[j];
            }

            const double residual = b[i] - Ax;
            residualSquared += residual * residual;
        }

        const double residualNorm =
            std::sqrt(residualSquared);

        // Report progress.
        if (iteration == 1 || iteration % 10 == 0)
        {
            std::cout << "Iteration " << iteration
                      << ", residual L2 = "
                      << residualNorm << '\n';
        }

        // Accept the new estimate.
        x = xNew;

        if (residualNorm < tolerance)
        {
            converged = true;
            std::cout << "Converged at iteration "
                      << iteration << '\n';
            break;
        }
    }

    std::cout << "\nTemperature solution:\n";

    for (int i = 0; i < n; ++i)
    {
        std::cout << "T[" << i + 1 << "] = "
                  << x[i] << '\n';
    }

    if (!converged)
    {
        std::cout << "\nWarning: maximum iterations reached "
                     "without meeting the tolerance.\n";
        return 1;
    }

    return 0;
}

Understanding the implementation

Step 1: Store the matrix.

The matrix is represented using a vector of vectors. This is simple and readable for a small example. It is not an efficient sparse-matrix representation for a large CFD simulation.

Step 2: Create an initial guess.

The initial temperature estimate is zero at every interior point. This is not the physical solution; it is merely the starting point for the iteration.

Step 3: Calculate the new values.

The nested loops implement the Jacobi formula. The inner loop sums the contributions from all off-diagonal matrix entries.

Step 4: Calculate the residual.

The code evaluates

r=b−Axnew\mathbf{r}=\mathbf{b}-A\mathbf{x}_{\mathrm{new}}

and calculates the L2L_2 norm:

∥r∥2=∑iri2\|\mathbf{r}\|_2 = \sqrt{\sum_i r_i^2}

Step 5: Test convergence.

The solver stops when the residual norm falls below the specified tolerance. If it reaches the maximum iteration count first, it reports that convergence was not achieved.

Step 6: Report the temperatures.

The final estimates should approach 7575, 5050, and 2525 degrees Celsius for the three interior grid points.

Compile and run

Using a C++ compiler such as GCC:

g++ -std=c++20 -O2 -Wall -Wextra -pedantic jacobi_heat.cpp -o jacobi_heat

On Linux or WSL, run:

./jacobi_heat

On Windows PowerShell, if the executable is named jacobi_heat.exe, run:

.\jacobi_heat.exe

The output should show the residual decreasing and the solution approaching the expected temperatures.

A note about this implementation

This example prioritizes clarity rather than performance. It stores a dense matrix even though the heat-conduction matrix is sparse.

A production solver would usually avoid storing all the zero coefficients and would use a suitable sparse representation. It might also use a more efficient linear solver or preconditioner.

For this small system, Jacobi is sufficient to demonstrate the principles. For large CFD systems, plain Jacobi is often too slow to use as the only solver.

16. Improving the C++ Solver

The educational implementation can be improved in several ways.

Validate the matrix

The Jacobi update divides by each diagonal coefficient:

xi(k+1)=bi−∑j≠iaijxj(k)aiix_i^{(k+1)} = \frac{b_i-\sum_{j\ne i}a_{ij}x_j^{(k)}}{a_{ii}}

Therefore, the implementation should reject a zero or numerically unsuitable diagonal entry instead of dividing by it.

It should also verify that the matrix and right-hand-side dimensions match.

Use sparse storage

For a one-dimensional tridiagonal system, storing three diagonals is enough. There is no need to store the entire N×NN\times N matrix.

For more general sparse systems, compressed sparse row (CSR) storage is a common option. It stores the nonzero coefficients and their column indices alongside row offsets.

Use appropriate stopping criteria

A practical solver may use a relative residual, an absolute residual, or a combination of criteria.

For example, a stopping condition could be based on

∥r(k)∥2≤max⁡(εabs,εrel∥b∥2)\|\mathbf{r}^{(k)}\|_2 \le \max\left( \varepsilon_{\mathrm{abs}}, \varepsilon_{\mathrm{rel}}\|\mathbf{b}\|_2 \right)

This combines absolute and relative tolerances. Special handling is needed when the right-hand-side norm is zero, and the exact criterion should match the solver’s design.

Add tests

A numerical solver should be tested against cases where the correct solution is known.

For the heat-conduction problem, the discrete solution is exactly linear for uniform spacing and constant conductivity when there is no internal heat generation and the boundary temperatures are prescribed.

For a larger grid, compare the numerical values with

T(x)=TL+TR−TLLxT(x)=T_L+\frac{T_R-T_L}{L}x

Then calculate the maximum absolute error:

Emax⁡=max⁡i∣Ti−T(xi)∣E_{\max} = \max_i|T_i-T(x_i)|

For this particular problem, the central-difference discretization reproduces the linear solution to floating-point precision, assuming the equations and boundary conditions are assembled correctly. This is an excellent first verification test, although it does not test every possible behavior of a solver.

17. What This Means for a Real CFD Project

The simple heat-conduction example is a foundation for a more substantial solver.

A sensible development path is to add complexity one step at a time.

Stage 1: Verify the algebraic solver

  • Solve small systems with known analytical solutions.
  • Compare the computed result with the expected result.
  • Test convergence and maximum-iteration handling.
  • Test invalid matrix dimensions and diagonal coefficients.
  • Record residual histories.

Stage 2: Introduce a larger diffusion problem

Discretize a one-dimensional heat equation on increasingly fine grids. Assemble the corresponding algebraic system and solve it with a suitable method.

Compare the result with the analytical temperature distribution.

Stage 3: Use sparse matrix storage

Replace the dense matrix with a representation that stores only nonzero coefficients. Compare memory requirements and computational cost as the number of unknowns increases.

Stage 4: Add finite-volume discretization

Construct equations from flux balances over control volumes. This makes the code more naturally aligned with the conservation-based structure of many CFD solvers.

Stage 5: Introduce transport and fluid flow

Once diffusion and algebraic solution are reliable, add advection–diffusion, transient terms, and then more complex flow equations.

For incompressible flow, pressure–velocity coupling requires additional numerical machinery beyond solving an isolated scalar equation.

Stage 6: Build verification into the project

For every new feature, define a benchmark and measurable acceptance criteria before trusting the results.

Possible checks include:

  • Analytical solutions.
  • Grid-refinement studies.
  • Mass conservation.
  • Energy conservation.
  • Residual convergence.
  • Comparison with trusted reference data.

This development sequence helps separate implementation errors from discretization errors and physical-model errors.

18. Common Mistakes to Avoid

Mistake 1: Assuming a small residual guarantees a correct physical answer.

A small residual indicates that a particular algebraic system is being satisfied according to the selected residual definition. It does not guarantee that the governing equations, boundary conditions, mesh, or physical models are appropriate.

Mistake 2: Updating Jacobi values in place.

Standard Jacobi iteration requires a separate vector for the next estimate. Updating the same vector during the sweep changes the algorithm.

Mistake 3: Ignoring boundary contributions.

Boundary conditions affect the assembled equations. Leaving out a prescribed boundary contribution can change the physical solution.

Mistake 4: Choosing a tolerance without considering scaling.

Residuals depend on how the equations are formulated and normalized. A tolerance that is suitable for one equation may not be suitable for another.

Mistake 5: Using a dense matrix for a large CFD problem.

A dense matrix wastes memory and computation when most coefficients are zero. Sparse storage and algorithms are important for scalability.

Mistake 6: Treating every linear solver as interchangeable.

Different algorithms have different requirements and convergence properties. Conjugate gradient, for example, is designed for symmetric positive-definite systems, while GMRES can handle more general nonsymmetric systems.

Mistake 7: Confusing linear-solver convergence with overall simulation convergence.

A linear system can be solved accurately at every outer iteration even while the overall nonlinear CFD solution continues to change.

Key Takeaways

  • Discretization converts differential equations into algebraic equations.
  • A linear system is commonly represented as Ax=bA\mathbf{x}=\mathbf{b}.
  • Boundary conditions influence the coefficient matrix and right-hand-side vector.
  • CFD matrices are generally sparse because discrete equations couple each unknown to a limited set of neighboring unknowns.
  • Direct solvers use algebraic elimination or factorization, while iterative solvers repeatedly improve an estimate.
  • Jacobi uses values from the previous iteration; Gauss–Seidel immediately reuses updated values.
  • SOR introduces a relaxation factor that may accelerate convergence when chosen appropriately.
  • Residuals measure how closely an estimate satisfies the discrete equations.
  • A low residual does not by itself prove physical accuracy or conservation.
  • Matrix properties, conditioning, preconditioning, and the choice of solver strongly affect computational performance.
  • Real CFD calculations often involve nonlinear equations and coupled linear systems rather than one isolated matrix equation.
  • Reliable solver development requires analytical benchmarks, automated tests, and systematic verification.

What’s Next?

In Module 5: Finite Volume Method from First Principles, we will move from finite-difference approximations toward the conservation-based framework widely used in CFD.

We will derive the finite-volume method by integrating a governing equation over a control volume, examine face fluxes, and construct discrete equations that conserve quantities across neighboring cells.

The key question will be: how can a numerical method preserve the physical conservation laws while representing fluid flow on a finite computational mesh?