Qwen3.8 11 · Weight Tying and Embeddings

What worked

The only accepted change allowed the output projection—the layer that converts the model’s internal representations into token predictions—to learn twice as quickly. Within the fixed five-minute RTX 3090 budget, this produced the lowest measured loss.

The progress graph therefore reflects one small but real step below the baseline, rather than a sequence of accumulating architectural improvements. The practical conclusion is narrow: keep the existing separate embedding and output weights, preserve their current scaling and normalization behavior, and use an output-projection learning rate of 0.008.

Increasing the input embedding learning rate from 0.6 to 0.8 did not have the same effect. It raised loss to 1.257199, suggesting that the useful adjustment was specifically faster adaptation at the output side, not a general benefit from making embedding-related parameters learn faster.

Scaling and normalization were counterproductive

A learnable multiplier on token embeddings, initialized from the square root of the embedding width, slightly worsened loss to 1.247466. This is close to the baseline but still failed the strict-improvement rule.

The more intrusive output-side changes were substantially worse. Multiplying output scores by the square root of the embedding width produced a loss of 1.406482, while normalizing those scores before the existing cap produced 1.310685. Both results indicate that the baseline output-score scale was already important to optimization; changing it directly was far more damaging than adjusting how quickly the projection weights learned.

Normalizing token embeddings before the first transformer block was also unsuccessful, with a loss of 1.257844. Taken together, these experiments favor leaving activation magnitudes alone and tuning the optimizer for the output projection instead.

Weight tying remains unresolved

Sharing one parameter matrix between the token embedding and output projection was attempted twice, but neither attempt reached a valid measurement. One failed during compiled evaluation, and the other failed the optimizer’s parameter-accounting check. A related learnable embedding-scale attempt also failed that check.

These are implementation failures, not evidence that tied weights hurt or help prediction loss. The benchmark therefore does not answer the tied-versus-untied question. Testing it properly would require integrating the shared parameter into both compilation and optimizer grouping without counting it twice, then rerunning the unchanged evaluator.

Recommended configuration and next step

Use the accepted output-projection learning rate of 0.008 and retain the baseline design everywhere else tested: independent input and output weights, no added embedding multiplier, no extra embedding normalization, and no additional output scaling or normalization.

The measured gain is small and the evidence contains no repeat runs, variance estimates, or longer-budget evaluations. It is therefore unknown whether the 0.001733 advantage is repeatable or specific to this five-minute run. The most useful follow-up would be repeated benchmark runs of the baseline and the 0.008 setting; after that, a correctly implemented weight-tying experiment remains the main unanswered architectural question.