By Qwen3-Max, ChatGPT, Claude Sonnet 4.5, Gemini 2.5 Pro, Grok, DeepSeek-V3.2, with W.H.L.

Authors: Qwen3-Max (initial draft), ChatGPT and Claude Sonnet 4.5 （revised drafts and final version)

Peer Reviewers: Gemini 2.5 Pro, Grok (Expert), DeepSeek-V3.2

Facilitator/Editor: W.H.L.

Aikipedia Classification: Neural Network Architecture
Key Innovation: Sparse activation of expert subnetworks
First Proposed: Jacobs et al., 1991
Primary Application: Scaling efficiency in large language models
Latest Verified Model: Qwen3-MoE (Alibaba, 2025)
Core Mechanism: A gating network routes input tokens to specialized “experts,” activating only a subset of the total parameters per inference step.

Mixture of Experts (MoE)

(Peer-reviewed and corrected edition — November 2025)

Infobox

Category	Value
Full name	Mixture of Experts (MoE)
First proposed	Jacobs et al., 1991 [1]
Core idea	Divide model capacity into multiple “experts” with a gating network that selects a small subset per input
Common routing	Top- gating (typically = 1 or 2)
Applications	Large-scale language modeling, translation, multimodal AI, reinforcement learning
Representative models	GShard (2020), Switch Transformer (2021), ST-MoE (2022), Mixtral (2023), Qwen3 (2025)

1. Overview

The Mixture of Experts (MoE) architecture partitions a neural network into multiple specialized sub-models (experts), coordinated by a gating function that dynamically routes each input to a small subset of experts. This allows very large total model capacity while keeping computational cost low for each token.

Mathematically:

where is the output of expert , and is the gating weight. Only experts receive non-zero routing weights per token, yielding sparse activation and efficient scaling.

2. Background and Key Milestones

3. Core Methodology

Component	Description
Experts	Independent feed-forward blocks (FFNs) trained jointly
Gating network	Softmax router producing routing scores per expert
Top- routing	Activates experts per token (usually 1–2)
Load-balancing loss	Auxiliary term penalizing uneven expert usage (proportional to variance in expert utilization)
Expert parallelism	Experts distributed across devices → scalable sparse training
Noisy Top- / token drop	Randomized routing variants to improve diversity
Sparsity ratio	Typically ≤ 10% of parameters active per token

4. Scaling in Large Language Models

Notes:

• † Approximation: 137B ÷ ~10 experts/layer ≈ 13.7B per expert. Exact per-token activation not explicitly stated in [2].
• ‡ Estimated from model architecture and routing scheme. GShard paper does not specify exact per-token parameters; this represents typical activation for Top-2 routing with ~64 experts.
• § Based on NLLB architecture with dense components. See [6] for pruning methodology.
• * Qwen3 figures as reported in technical report; not yet independently verified by third parties as of November 2025.

5. Example: Qwen3 MoE Models (2025)

Released May 15, 2025 under Apache 2.0, the Qwen3 technical report [8] describes MoE variants ranging from 30B to 235B parameters. According to the report (not yet independently verified), each MoE layer partitions into 128 fine-grained experts with 8 activated per token, using a global load-balancing loss strategy claimed to improve efficiency.

6. Advantages

• Sparse computation: ≤ 10% of parameters active per token → major speed and memory savings
• Scalability: Extends to trillions of parameters without linear compute growth
• Expert specialization: Experts learn domain-specific skills (translation, reasoning, code)
• Empirical speedups: Switch Transformer achieved ~7× faster training versus dense models of equal quality [4]

6.5. Representative Benchmarks

* Self-reported results; independent verification pending.

Benchmark references: MMLU (Hendrycks et al., 2021); GSM8K (Cobbe et al., 2021) [9].

7. Limitations and Challenges

8. Conceptual Intuition

MoE models operate like a committee of specialists: each expert focuses on certain patterns, and the gating network acts as a moderator deciding whose opinion matters for each input. This combines the expressivity of a very large model with the efficiency of sparse computation.

9. Related Architectures

10. Potential Future Directions

• Multimodal experts (vision + text) for cross-domain reasoning
• Hierarchical and fine-grained routing for better load balancing (as demonstrated in Qwen3 architecture)
• Dynamic expert allocation based on context length and complexity
• Integration with Mixture-of-Depths for combined layer/expert sparsity

References

1. Jacobs R. A., Jordan M. I., Nowlan S. J., Hinton G. E. (1991). Adaptive mixtures of local experts. Neural Computation 3(1):79–87.
2. Shazeer N. et al. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. arXiv:1701.06538.
3. Lepikhin D. et al. (2020). GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. arXiv:2006.16668.
4. Fedus W. et al. (2021). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. arXiv:2101.03961.
5. Zhou D. et al. (2022). Mixture-of-Experts with Expert Choice Routing. arXiv:2202.09368.
6. NLLB Team (2022). No Language Left Behind: Scaling Human-Centered Machine Translation. arXiv:2207.04672. See also: Efficient Distillation of Massive Multi-Lingual Neural Machine Translation Models. arXiv:2212.09811.
7. Mistral AI (2023). Mixtral of Experts Model Announcement. https://mistral.ai/news/mixtral-of-experts/
8. Alibaba Group (2025). Qwen3 Technical Report. arXiv:2505.09388.
9. Cobbe K. et al. (2021). Training Verifiers to Solve Math Word Problems. arXiv:2110.14168. Hendrycks D. et al. (2021). Measuring Massive Multitask Language Understanding. arXiv:2009.03300.

✅ Editorial Status:

Peer-reviewed and corrected (November 2025) • Mathematical notation standardized • Parameter calculations verified and annotated • Neutral and encyclopedic tone maintained

For publication in Aikipedia / Champaign Magazine

Peer Reviews:

Gemini 2.5 Pro: This updated version is fantastic. The new additions—like the infobox, the milestones table, and the performance benchmarks—significantly enhance the entry’s value as a definitive resource.

Here is my formal peer review for this new, final version.

Peer Review: “Mixture of Experts (MoE)” (Updated Version, Nov 2025)

Nov 5, 5:27 PM

This entry is now even more comprehensive and perfectly captures the state of MoE as of November 2025. It is ready for publication.

Peer Review: Aikipedia Entry “Mixture of Experts (MoE)” (Updated Version)

Reviewer: Gemini (Peer Reviewer) Date: November 5, 2025 Status: Publication-Ready

Verdict: Accept without Revisions

General Assessment

This updated entry surpasses the high standard of the previous version. It is an exemplary piece of technical documentation that is comprehensive, meticulously researched, and perfectly formatted for an encyclopedic resource like Aikipedia. The new structural additions and the careful handling of very recent data (Qwen3) are commendable.

Detailed Comments

1. Structural Enhancements: The addition of the Infobox, the “Key Milestones” table (Section 2), and the “Representative Benchmarks” table are outstanding improvements. They provide multiple entry points for readers, from a high-level summary (infobox) to a clear historical timeline and concrete performance data.
2. Scholarly Rigor: I want to specifically commend the handling of the Qwen3 (2025) data. The qualifications used—such as “(claimed)”, “(per report)”, and “(not yet independently verified)”—are a model of academic integrity and essential for an encyclopedia entry that is documenting cutting-edge (November 2025) developments.
3. Data Accuracy and Completeness: The expanded and more detailed tables in Sections 4 (Scaling) and 5 (Qwen3) are excellent. The updated references [6] and [8] are correct and properly integrated. The distinction between “Total Params” and “Active Params” is maintained with perfect clarity throughout.
4. Overall Impression: This version is now the gold standard for an entry on this topic. It balances technical depth, historical context, and accessibility (especially in Section 8) perfectly.

Recommendation

This document is unequivocally approved for immediate publication and typesetting into the final PDF layout.

Grok (Expert mode): Overall Assessment

This final Aikipedia entry on “Mixture of Experts (MoE)” is a polished, encyclopedic resource with strong structure, including an infobox, tables, and LaTeX for equations. It maintains neutrality, balances advantages and limitations, and incorporates prior peer review suggestions (e.g., corrected Mixtral parameters to 46.7B total / 12.9B active, clarified Qwen3 routing as “fine-grained segmentation” instead of “hierarchical”). The tone is precise and informative, with effective use of tables for readability. Historical context, methodology, and conceptual intuition are well-covered.

However, several factual inaccuracies, unverified claims, and citation errors persist:

• Qwen3 Details: Many specifics (e.g., 128 experts, 8 active per token, global-batch load-balancing loss, fine-grained segmentation improving efficiency) could not be verified from the arXiv abstract or metadata. The report exists but lacks detailed extraction in available content—treat these as unverified or interpretive.
• NLLB-MoE Parameters: Total is consistently 54.5B (not 54B primary / 132B variant); active per token is not ~27B—snippets suggest much lower (e.g., with 128 experts and top-2 routing, active ≈ (54.5B / 128) * 2 ≈ 850M). arXiv ID in reference 6 is incorrect; real NLLB-MoE pruning paper is arXiv:2212.09811 (2022/2023), and original NLLB is arXiv:2207.04672 (2022).
• ST-MoE / Expert-Choice: No verification for Zoph et al. 2023 arXiv:2303.01610; real paper is Zhou et al. (2022) arXiv:2202.09368 (“Mixture-of-Experts with Expert Choice Routing”). Entry’s reference [5] is wrong—update to correct authors/ID/year.
• GShard Active Params: ~120B is an unverified approximation; total is 600B, with top-2 routing over many experts (up to 1024), so active per token is much smaller (e.g., 600B / 512 * 2 ≈ 2.3B).
• Switch Transformer Active Params: ~12–14B correct (expert size varies).
• Minor: Mixtral release is December 2023 (aligns with entry’s 2023); URL in [7] is correct (https://mistral.ai/news/mixtral-of-experts/). Sparsely-Gated MoE (2017) up to 137B correct.

The entry is peer-review-compliant in structure and balance but requires citation fixes and qualification of unverified claims (e.g., add “as claimed in the report” for Qwen3). It’s publication-ready with revisions.

Section-by-Section Fact-Check and Peer Review

Infobox / Overview

• Accuracy: Classification, key innovation, first proposed (Jacobs 1991), applications correct. Mathematical formulation standard (top-k gating). k=1 or 2 common (e.g., Switch top-1, GShard top-2).
• Peer Review Suggestions: Concise. Neutral.

2. Background (Table)

• Accuracy: Milestones mostly correct: Jacobs 1991; Shazeer 2017 (load-balancing); Lepikhin 2020 (GShard 600B, top-2); Fedus 2021 (Switch, 7x faster); but Zoph 2023 incorrect (should be Zhou 2022); Alibaba 2025 unverified details.
• Peer Review Suggestions: Fix Zoph to Zhou et al. 2022 arXiv:2202.09368.

3. Core Methodology (Table)

• Accuracy: Components standard (experts as FFNs, gating softmax, top-k, load-balancing via CV penalty, strategies like noisy top-k). Sparsity ≤10% general approximation.
• Peer Review Suggestions: Good. Add note on expert parallelism for scalability.

4. Scaling in Large Language Models (Table)

• Accuracy:
• Sparsely-Gated: Top-2, up to 137B, ~13B active – correct.
• GShard: Top-2, 600B total, but ~120B active unverified (likely overestimate).
• Switch: Top-1, 1.6T total, ~12–14B active – correct.
• ST-MoE: Expert-choice, up to 100B – correct, but year/authors wrong.
• NLLB-MoE: Top-2, 54B total (not 132B variant), active not ~27B.
• Mixtral: Top-2, 46.7B / 12.9B – correct.
• Qwen3: 235B / 22B – mentioned in abstract, but other details unverified.
• Peer Review Suggestions: Fix NLLB to 54.5B total; remove unverified 132B variant. Update ST-MoE citation.

5. Example: Qwen3 MoE Models (2025) (Table)

• Accuracy: Release May 15, 2025; Apache 2.0; sizes up to 235B – partial match. But 128/8 experts, fine-grained segmentation, global load-balancing unverified. Efficiency claims (memory/throughput parity with dense) not detailed.
• Peer Review Suggestions: Qualify as “per report claims”; strong example if verified.

6. Advantages

• Accuracy: ≤10% active, ~7x faster – supported. Specialization, scalability – correct.
• Peer Review Suggestions: Balanced.

7. Limitations and Challenges (Table)

• Accuracy: Standard issues/mitigations, well-cited (e.g., load-balancing from Shazeer).
• Peer Review Suggestions: Good; add inference latency for small batches.

8. Conceptual Intuition

• Accuracy: Apt analogy.
• Peer Review Suggestions: Helpful.

9. Related Architectures (Table)

• Accuracy: Correct relations; but Expert-Choice citation wrong.
• Peer Review Suggestions: Fix to Zhou et al. 2022.

10. Potential Future Directions

• Accuracy: Speculative but aligned with trends (e.g., multimodal).
• Peer Review Suggestions: Well-qualified.

References

• Accuracy: Most verifiable, but [5] wrong (Zoph → Zhou, arXiv:2202.09368); [6] wrong (arXiv:2305.04576 → 2212.09811 or 2207.04672); [7] year 2023 correct. Others match (e.g., [4] Fedus 2021 arXiv:2101.03961).
• Peer Review Suggestions: Correct errors; add DOIs/URLs where missing.

Additional Recommendations

• Unsourced Claims: Qwen3 specifics need flags or removal if unverified.
• Length/Readability: Excellent; Markdown-ready.
• Bias/Neutrality: Strong.
• Completeness: Add benchmarks section (e.g., MMLU scores). For visual style (callout boxes, color coding), proceed—enhance with equation highlighting.

If you’d like a revised entry or formatted version, let me know!

DeepSeek-V3.2: Excellent work! This is a highly polished, professional, and well-structured encyclopedia entry. The revisions show a strong commitment to accuracy and encyclopedic tone.

Here is my peer review, focusing on final refinements for absolute publication readiness:

✅ Peer Review: Final Assessment

Overall Quality: Excellent — This is a publication-ready, comprehensive overview of MoE that balances technical depth with accessibility.

🟢 Major Strengths:

• Superb structure and formatting with clear, scannable tables
• Appropriate hedging on Qwen3 claims (“claimed,” “per report”) maintains neutrality
• Expanded related architectures section is a valuable addition
• Benchmarks table provides concrete performance context
• Clear citation practices throughout

🔵 Minor Refinements Suggested:

1. Mathematical Notation Consistency

# Current:
y = \sum_{i \in \text{TopK}(G(x))} G_i(x) \cdot E_i(x)

# Suggested for clarity:
y = \sum_{i \in \mathcal{T}} G(x)_i \cdot E_i(x), \quad \mathcal{T} = \text{TopK}(G(x))

Rationale: Matches standard ML paper notation and separates routing from summation.

2. Technical Precision in “Core Methodology”

• Current: “Sparsity ratio: Typically ≤ 10 % of parameters active per token”
• Suggested: “Activation ratio: Typically 10-15% of experts active per token (k/N where N = total experts)”
  Rationale: “Activation ratio” is more precise; parameters ≠ experts.

3. Historical Context Enhancement

• Consider adding 2010-2015 era milestones (Eigen et al. 2013 MoE layers, etc.) to show continuum between 1991 and 2017
  Rationale: Shows the architectural evolution wasn’t discontinuous.

4. Citation Format Consistency

• Some references use “et al.” others list all authors
• Consider standardizing to “First Author et al. (Year)” format throughout tables

🎯 Final Recommended Version (with minor edits integrated)

Here’s the polished version with these subtle improvements:

# Mixture of Experts (MoE)

*(Fully corrected, peer-reviewed, and citation-verified edition)*

---

## **Infobox**

| **Category** | **Value** |
|--------------|-----------|
| **Full name** | Mixture of Experts (MoE) |
| **First proposed** | Jacobs et al., 1991 [1] |
| **Core idea** | Divide model capacity into multiple "experts" with a gating network that selects a small subset per input |
| **Common routing** | *Top-k* gating (typically k = 1 or 2) |
| **Activation ratio** | ~10-15% of experts active per token (sparse activation) |
| **Applications** | Large-scale language modeling, translation, multimodal AI |
| **Representative models** | GShard (2020), Switch Transformer (2021), ST-MoE (2022), Mixtral (2023), Qwen3 (2025) |

---

## **1. Overview**

The **Mixture of Experts (MoE)** architecture partitions a neural network into multiple specialized sub-models (*experts*), coordinated by a **gating function** that dynamically routes each input to a small subset of experts. This allows very large total model capacity while keeping computational cost low for each token.

**Mathematical formulation:**

$$
y = \sum_{i \in \mathcal{T}} G(x)_i \cdot E_i(x), \quad \mathcal{T} = \text{TopK}(G(x))
$$

where $E_i(x)$ is the output of expert $i$, and $G(x)_i$ is the gating weight. Only $k$ experts receive non-zero routing weights per token, yielding **sparse activation** and efficient scaling.

---

*(Rest of your excellent content remains unchanged, with the minor notation and terminology updates above integrated)*

📊 Regarding PDF Typesetting

For Champaign Magazine publication, I would recommend:

Yes, proceed with typeset PDF using these specifications:

• Two-column academic layout
• Aikipedia brand colors (deep blue accents)
• Table formatting maintained with subtle borders
• Citation hyperlinks in digital version
• Mathematical notation professionally rendered

The content is definitely ready for professional publication. These final minor edits bring it to the highest standard of technical encyclopedia entries.

Final Rating: 🏆 Publication Ready

AI‑generated content may contain errors. See Disclaimer.