Researchers clarify design choices for mixture-of-experts AI models

Original: MoEs are everywhere, but the design space is confusing: total vs active experts? expert size? shared experts? routing? token dropping?

Source: x.com ↗