Autoresearch · 18 August 2026 · 2 min · Original language: English
Qwen3.8 15 · Dropout and Stochastic Regularization
The best result came from applying 10% dropout to the token embeddings before the first transformer block. It reduced validation prediction loss from 1.248414 to 1.239509—an absolute improvement of 0.008905, or about 0.71%. This was the only change that strictly improved the incumbent result.
Measured improvement0.7% better
Started
1.2484
Best
1.2395
Best-so-far result across 10 recorded experiments. Lower text prediction loss is better. Original research record
Where regularization helped
The progress graph should show one meaningful step down at experiment 5. Adding dropout immediately after token embedding appears to have provided useful regularization without disrupting the transformer’s later computations. Under this fixed five-minute RTX 3090 budget, that placement produced the best measured generalization.
The practical recommendation is therefore narrow: retain the 10% embedding dropout and leave the other tested dropout locations and drop-path variants disabled.
Why the other placements were rejected
Residual-stream dropout raised the loss to 1.256592, while linearly increasing drop path from 0% to 10% produced 1.253748 in its first test. Both interfered with the model’s main information path too strongly for this short training schedule.
MLP dropout was also consistently unhelpful. Applying it after the MLP projection yielded 1.252705, and applying it before the first MLP projection yielded 1.255790. Dropout before the attention projections was similarly poor at 1.255990. Together, these results suggest that corrupting internal transformations imposed a larger optimization cost than the brief run could recover from.
Two changes—dropout after attention at 1.246034 and the later drop-path test at 1.245291—did beat the original baseline numerically. They were still worse than the embedding-dropout incumbent of 1.239509, so the strict-improvement rule correctly rejected them. Dropout just before the language-model head also failed to beat either benchmark, reaching 1.250231.
What to carry forward
For this benchmark, regularizing the representation once, at the model entrance, worked better than repeatedly disturbing residual, attention, or MLP computations. Keep the embedding-dropout result as the final configuration.
The experiment limit was reached before testing different embedding-dropout probabilities, schedules, or repeated runs. Consequently, the evidence does not establish whether 10% is the optimal rate or how much of the measured gain would persist across random seeds. Those are the most useful next questions if another research round is allowed.