Qwen3.8 16 · Layer Sharing

What the successful measurements showed

Only two sharing variants completed evaluation, and both shared one MLP module across every transformer block. They produced prediction losses of 1.287785 and 1.294727. Because lower is better, these were approximately 3.1% and 3.7% worse than the baseline.

The graph therefore shows no accepted improvement after the initial measurement. Although global MLP sharing reduced the number of distinct parameters, the evidence does not show that this translated into better sample efficiency within the five-minute RTX 3090 run. In practice, the loss of layer-specific MLP capacity outweighed any benefit the tested sharing scheme may have provided.

Why most attempts were inconclusive

Seven experiments failed before producing a score. Six MLP-sharing attempts stopped because the optimizer’s parameter accounting still assumed an untied model. Reusing a module changes how many unique parameters exist, so optimizer grouping and validation must count parameter identity rather than the number of layer references.

One attention-sharing attempt failed with an indentation error and never reached training. It therefore provides no evidence for or against shared attention projections.

These failures should be treated as implementation failures, not negative benchmark results. Repeating nearly the same MLP-sharing change consumed most of the ten-record budget without testing substantially different hypotheses.

What to do next

Keep the baseline for now. A future run should first add a short preflight check that constructs the model, verifies unique optimizer coverage, and executes one forward and backward pass before spending a benchmark record.

After that, test the requested sharing families separately: repeated blocks, partial sharing, alternating shared layers, and shared cores with lightweight untied adapters. None of those schemes produced a valid measurement here, so their effect on prediction loss remains unknown. Shared attention is also still unknown because its only attempt did not run.