Most new Transformer tweaks still fail to improve real tasks

Original: Most Transformer Modifications Still Do Not Transfer at 1-3B: A 2020-2026 Update to Narang et al. (2021) with Downstream Evaluation and a Noise Floor

Source: arxiv.org ↗