Local Minimum: When Training Stops Improving But Is Not Finished

What problem does this solve?

A telecom churn model trains for forty epochs. Loss falls sharply for six epochs, settles at 0.42, and stays there. Validation accuracy sits at 71% when the business case needed 80%. The team doubles the epochs, sees nothing change, and concludes the model has hit its ceiling.

That conclusion is usually premature. A flat curve has several possible causes and they are not interchangeable: the optimiser may be in a genuine basin, it may be at a saddle point where the gradient is near zero but not minimal, the learning rate may have decayed until steps are too small to escape anything, or half the ReLU units may have died and stopped passing gradient at all.

Because these look identical on a loss plot, teams often cycle through optimisers at random. Adam, then SGD with momentum, then back to Adam with a different learning rate — a lot of compute spent without a hypothesis.

How the solution works

Separate the diagnosis from the fix. Log gradient norms per layer, the current learning rate, and the fraction of activations that are exactly zero. Those three numbers distinguish the four causes in a single run.

If gradients are near zero everywhere and the loss surface is flat in all directions, that is a plateau or a saddle. If gradients are healthy but the learning rate has decayed to near-nothing, the schedule is the problem. If a large fraction of ReLU outputs are zero, the units are dead and no optimiser change will help.

The interventions differ accordingly: warm restarts and momentum for saddles, a revised schedule for over-decay, weight re-initialisation or a LeakyReLU for dead units, and a genuinely different initialisation seed when you suspect a real basin.

1. 1
   Gradient descent is local Each step moves parameters against the gradient computed at the current point. Nothing in the update rule can see beyond the immediate neighbourhood.
2. 2
   A minimum is where the gradient vanishes At a stationary point the gradient is zero, so the update becomes zero and training stops moving regardless of how many epochs remain.
3. 3
   Curvature decides the type If every direction curves upward the point is a local minimum. If some directions curve down it is a saddle point, and escape is possible.
4. 4
   High dimensions favour saddles For a point to be a true local minimum, every one of millions of directions must curve upward simultaneously — statistically far less likely than at least one curving down.
5. 5
   Momentum carries through Accumulated velocity lets the optimiser continue past a region of near-zero gradient rather than stopping in it.

Reference architecture

Four independent factors shape whether training escapes a flat region. Changing them together makes the result uninterpretable.

Deployment options: Relevant on any training hardware, but the cost of the mistake scales with it. On a multi-GPU node an unnecessary re-run is measured in hours of compute, which is why diagnosis before intervention pays for itself quickly.

Key capabilities

Integrations

Optimisation diagnostics belong in the training platform rather than in a notebook, because the value comes from comparing runs.

Industry use cases

Implementation approach

1. 1
   Instrument first Log gradient norm, learning rate and dead-unit fraction before changing anything.
2. 2
   Rule out the schedule If the learning rate has decayed to near zero, that alone explains the stall.
3. 3
   Check for dead units A large fraction of exactly-zero ReLU outputs means no optimiser change will help.
4. 4
   Add momentum or restarts For a genuine flat region, momentum and cosine warm restarts are the cheapest effective interventions.
5. 5
   Change one thing at a time Altering the optimiser, schedule and initialisation together makes the result impossible to attribute.

Security & deployment

Training logs record gradient statistics and loss values, not raw records, and are generally safe to retain outside the data boundary. Checkpoints are a different matter: model weights fitted on sensitive data can leak information about that data and should be stored under the same controls as the training set itself.

A worked example

Take the one-dimensional function L(x) = x^4 - 3x^3 + 2, which has a shallow dip and a deeper one, to see how the starting point decides the outcome.

1. The derivative. L'(x) = 4x^3 - 9x^2 = x^2 * (4x - 9), so stationary points sit at x = 0 and x = 2.25.
2. Classify them. L''(x) = 12x^2 - 18x. At x = 2.25, L'' = 20.25 > 0, a genuine minimum. At x = 0, L'' = 0 — a saddle-like inflection, not a minimum at all.
3. Start at x = 0.05. The gradient is 4(0.000125) - 9(0.0025) = -0.0220, tiny. With a learning rate of 0.01 the step is 0.00022. Progress is glacial and the curve looks flat.
4. Add momentum. With momentum 0.9, accumulated velocity builds across steps and carries the parameter through the flat region toward x = 2.25.

Same function, same optimiser, different outcome — decided entirely by whether the update rule could accumulate velocity through a flat region. The loss curve for the first case looks exactly like a model that has 'converged'.

In code

Rather than guess, instrument the run. This logs the three quantities that separate the causes.

import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(32, 128), nn.ReLU(),
    nn.Linear(128, 64), nn.ReLU(),
    nn.Linear(64, 1),
)
optimizer = torch.optim.SGD(model.parameters(), lr=0.05, momentum=0.9)
scheduler = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(
    optimizer, T_0=10, T_mult=2,
)

dead_relu_fraction = {}

def record_dead(name):
    def hook(_module, _inp, out):
        dead_relu_fraction[name] = (out == 0).float().mean().item()
    return hook

for i, layer in enumerate(model):
    if isinstance(layer, nn.ReLU):
        layer.register_forward_hook(record_dead(f"relu_{i}"))

def diagnose(step):
    total = 0.0
    for p in model.parameters():
        if p.grad is not None:
            total += p.grad.detach().norm().item() ** 2
    grad_norm = total ** 0.5
    lr = optimizer.param_groups[0]["lr"]
    print(f"step={step} grad_norm={grad_norm:.6f} lr={lr:.6f} dead={dead_relu_fraction}")
    # grad_norm ~ 0 and lr healthy  -> plateau or saddle: try warm restarts
    # grad_norm healthy and lr ~ 0  -> schedule decayed too far
    # dead fraction > 0.5           -> dead ReLUs: re-init or use LeakyReLU

Per-step gradient norm, current learning rate and the fraction of zero ReLU outputs per layer. The combination identifies which of the three causes you are looking at; the comments map each pattern to its intervention.

Diagnostic checks

• Plot gradient norm alongside loss. A flat loss with healthy gradients is not a minimum — look at the learning rate.
• Print the current learning rate every epoch. Aggressive decay schedules frequently reach values too small to move anything.
• Measure the fraction of ReLU activations that are exactly zero. Above roughly half, dead units are the constraint.
• Restart training from a different random seed. If the plateau lands at a noticeably different loss, the surface has multiple basins.
• Overfit a single batch deliberately. If the model cannot drive loss to near zero on ten samples, the problem is capacity or a bug, not optimisation.

Limitations & prerequisites

• In high-dimensional networks, true local minima are far less common than the term's popularity suggests; most stalls are something else.
• There is no practical way to prove a point is a global minimum for a large network, so 'escaped the local minimum' is always provisional.
• Warm restarts increase total training time and can undo useful convergence if applied too aggressively.
• Momentum can carry the optimiser past a good solution as readily as past a bad one.

Telling the four causes apart

All four produce a flat loss curve. Only the instrumentation distinguishes them.

The instrumentation costs one run. Cycling through optimisers blindly costs several, and still leaves you without a cause.

Key takeaways

• A flat loss curve is a symptom with four common causes that look identical on a plot.
• Gradient norm, learning rate and dead-unit fraction distinguish them in a single instrumented run.
• True local minima are rarer in deep networks than the term implies — suspect saddles and schedules first.
• Momentum and cosine warm restarts are the cheapest interventions for a genuine flat region.
• Change one variable at a time, or the result cannot be attributed to anything.

FAQ

Related topics & solutions

• AI Products Portfolio