Qwen3.8 13 · Gradient Clipping

What produced the improvement

The progress graph should show three distinct stages.

First, ordinary global-norm clipping supplied most of the gain. Capping the global gradient norm at 1.0 lowered loss from 1.247052 to 1.229606. Tightening that fixed cap to 0.5 improved it again to 1.225278. In practical terms, the original training run benefited from suppressing unusually large updates, and a moderately restrictive cap worked better than the looser one.

Second, adapting the cap to the model’s recent gradient scale was more effective than keeping it fixed. An exponential moving average of gradient norms, constrained to a bounded range, reduced loss to 1.218228. A more permissive adaptive range—recorded as a 90th-percentile-style threshold clamped between 0.2 and 1.5—then reached 1.216037. This suggests that clipping needed to follow the changing scale of training rather than impose the same absolute ceiling throughout the five-minute run.

Finally, tightening the adaptive coefficient produced small additional gains. Moving from 0.9 to 0.7 reached 1.215579, and 0.6 produced the best retained result of 1.215235. These last improvements were much smaller than the initial gains from introducing clipping and making it adaptive.

Why the 0.6 adaptive setting is the practical choice

The useful interpretation is not simply “clip harder.” The experiments found a balance: large gradient excursions were constrained, while ordinary updates were allowed to scale with the model’s recent behavior. Relaxing the coefficient above the successful setting generally worsened the benchmark, but tightening it further also failed to improve the best score.

For this exact pinned benchmark, the evidence therefore supports keeping adaptive global-norm clipping with a coefficient of 0.6. It does not support replacing global clipping with element-wise per-parameter clipping: the tested per-parameter value cap scored 1.247220, effectively losing all of the improvement and performing slightly worse than the baseline.

What the rejected runs teach

After reaching 1.215235, many experiments adjusted the coefficient by small amounts in both directions. None produced a strict improvement. Examples include 0.57 at 1.215483, 0.56 at 1.215942, 0.63 at 1.215744, and 0.55 at 1.215808. This cluster supports stopping the fine coefficient search around 0.6: nearby values did not beat it.

Later runs around the same region often scored roughly 1.220–1.221, including both tighter and looser settings. Because the evidence provides no repeated measurements or run-to-run variance, those results cannot establish whether conditions changed, the benchmark is noisy, or the apparent advantage of exactly 0.6 is fully reproducible. The defensible conclusion is narrower: 0.6 is the best observed setting, not a proven universal optimum.

Allowing larger adaptive updates was consistently unhelpful in the reported comparisons. Several attempts near 0.95 scored between 1.216444 and 1.217099 against a then-best 1.216037. More aggressive restriction below 0.6 also failed, with tested values such as 0.58, 0.55, 0.53, 0.52, and 0.5 all missing the final best. Repeating more tiny coefficient changes is therefore unlikely to be the most informative next step.

Recommended next step

Retain the adaptive global-norm mechanism and the 0.6 coefficient for the current benchmark. Before treating it as settled, rerun the retained 0.6 configuration several times under identical conditions and compare its distribution with one nearby alternative, such as 0.57 or 0.63. The present evidence cannot quantify benchmark variance, so reproducibility is the main unanswered question.

The last planned 0.53 experiment was stopped before producing a measurement and adds no result. More importantly, the research log contains 67 experiment records despite the stated ten-record stopping rule. Any follow-up should enforce that limit explicitly so the search remains controlled and each experiment tests one coherent hypothesis.