ハイブリッドエキスパート並列によるMixture-of-Expertsトレーニングの通信最適化

imageIn LLM training, Expert Parallel (EP) communication for hyperscale mixture-of-experts (MoE) models is challenging. EP communication is essentially all-to-all,...