Autoresearch · 18 August 2026 · 2 min · Original language: English
Qwen3.8 02 · Depth Width Tradeoff
The best tested configuration used 6 layers and an aspect ratio of 56, reaching a validation prediction loss of 1.189416. That improved on the 1.251327 starting point by 0.061911, or about 4.95%. Under this five-minute RTX 3090 budget, a moderately smaller model trained more effectively than the deeper or wider alternatives.
Measured improvement4.9% better
Started
1.2513
Best
1.1894
Best-so-far result across 10 recorded experiments. Lower text prediction loss is better. Original research record
Most of the improvement came from reducing depth
The decisive change was reducing the transformer from 8 layers to 6. Loss fell from 1.251327 to 1.190876.
This result shows that, for this short wall-clock benchmark, the extra capacity of the 8-layer model was less valuable than the additional training progress enabled by the cheaper 6-layer model. Restoring 8 layers raised loss to 1.250553, while a 7-layer compromise still performed poorly at 1.243846. A 12-layer run did not finish before the 600-second watchdog.
The large early drop visible in the progress graph therefore represents a real depth–throughput tradeoff: six layers was substantially better suited to the available training time.
A modest width reduction produced the final gain
With depth fixed at 6, reducing the aspect ratio from 64 to 56 improved loss slightly further, from 1.190876 to 1.189416. This was a much smaller gain than the depth change, but it was a strict improvement and became the final retained configuration.
Moving farther in either direction was counterproductive. Widening the model from aspect ratio 64 to 96 increased its hidden width from 384 to 576 and worsened loss to 1.256385. Narrowing from 56 to 52 also lost ground, reaching 1.196828. The evidence therefore points to a local sweet spot near an aspect ratio of 56: wide enough to preserve useful capacity, but inexpensive enough to make good progress within five minutes.
An aspect ratio of 48 was started but never completed because the backend stopped, so its performance remains unknown.
What to keep and what to test next
For this exact benchmark, keep depth 6 and aspect ratio 56. The completed neighboring tests make depth 7 or 8, aspect ratio 52, and aspect ratio 96 unattractive under the same time limit.
The search did not produce measured comparisons for different attention head dimensions, nor did it isolate parameter count, tokens processed, step count, or throughput. Consequently, it cannot establish whether the winning configuration is generally more parameter-efficient or whether the result transfers to longer training budgets. The strongest next experiment would be to hold depth 6 and aspect ratio 56 fixed while varying head dimension one step at a time, recording throughput and parameter count alongside validation loss. Re-running the unfinished aspect-ratio-48 case would also close the remaining width gap.