Structured-Then-Unstructured Pruning for Scalable MoE Pruning [Paper Reflection]

Combination-of-Consultants (MoEs) architectures provide a promising answer by sparsely activating particular elements of the mannequin, decreasing the inference overhead. Nevertheless, even with MoEs, the sheer variety of parameters and consultants makes deployment and serving pricey.

Pruning is a longtime methodology to cut back the variety of parameters of a skilled mannequin whereas sustaining its job efficiency. Usually, we distinguish two sorts of approaches. Unstructured pruning removes particular person weights, whereas structured pruning removes total mannequin elements.

Attributable to their clear construction, structured pruning appears to be a perfect match for MoEs. By eradicating redundant consultants, we will shrink the whole mannequin measurement. Nevertheless, present approaches for skilled pruning require many ahead passes, whose quantity grows exponentially with the variety of consultants. Additional, structured pruning doesn’t scale back the variety of energetic weights throughout inference.

In our paper STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning, which was accepted for a presentation at ACL 2025, we mix the 2 courses of pruning strategies and introduce an strategy that works exceptionally properly for MoEs with over 100 consultants. In a nutshell, STUN first removes redundant consultants after which performs unstructured pruning inside particular person consultants.

Scaling limitations for Combination of Knowledgeable fashions

MoEs are an efficient method to extend the whole variety of mannequin parameters whereas protecting computational calls for in verify. By dividing the mannequin into specialised buildings, known as consultants, and selectively activating them primarily based on the enter, MoEs obtain effectivity positive factors in coaching and inference.

Extra consultants permit the mannequin to seize a broader vary of representations and specializations, enhancing efficiency on numerous duties or complicated knowledge. Unsurprisingly, we see a transparent pattern in the direction of an elevated variety of consultants in MoEs. As an example this evolution, Mistral’s Mixtral 8x7B (December 2023) builds on eight consultants, Databricks’ DBRX (March 2024) on 16, and Snowflake’s Arctic (April 2024) makes use of 128 consultants.

Nevertheless, as fashions scale additional, the effectivity positive factors offered by the MoE structure alone are inadequate. Right here, pruning turns into important, refining the structure by eradicating redundant parameters with out compromising total efficiency. Combining MoEs with pruning strategies can optimize inference velocity and reminiscence consumption, making it a promising path for additional scaling fashions.

Fixing the exponential scaling problem in structured MoE pruning

Structured pruning removes particular patterns, akin to rows or total weight tensors. Within the context of MoEs, as skilled buildings from coaching MoEs correspond to such patterns, pruning consultants is a pure match for structured pruning.

Whereas a rise from 8 to 128 consultants could appear modest, it renders present pruning strategies unviable. Roughly talking, they take a “combinatorial” strategy to figuring out which buildings to take away, requiring the enumeration of all doable subsets of consultants to find out the optimum configuration. As an example, when the variety of consultants will increase from 8 to 128, the ahead passes of combinatorial pruning algorithms develop exponentially, from 70 to 2.4 × 10³⁷.

In distinction, STUN leverages the behavioral similarity between consultants to make knowledgeable pruning selections. Particularly, it first identifies clusters of comparable consultants primarily based on their behavioral similarity. We are able to decide the similarity at a minimal price by inspecting the mannequin’s weights. If two rows have related values, this implies a excessive pairwise similarity between the 2 corresponding consultants. Such an skilled pair tends to activate on related inputs and exhibit related outputs, thus forming a cluster.

By pruning all however one consultant skilled from every cluster, STUN successfully reduces the mannequin measurement whereas preserving its total performance. This strategy drastically reduces the exponential complexity of exhaustively enumerating mixtures to fixed O(1), making it extremely scalable for enormous MoEs.

Exploring the potential of a two-phase strategy to MoE pruning

A key query in our analysis was: How a lot can we achieve from a further unstructured pruning section? After we take away all redundant consultants, there is likely to be much less “margin” for additional pruning in comparison with a situation the place we solely apply unstructured pruning.

We are able to quantify this margin because the kurtosis of the mannequin weights’ distribution, colloquially often known as its “tailedness.” As unstructured pruning removes near-zero weights, it reduces the burden distribution’s kurtosis.

Not like unstructured pruning, which selectively targets weights that minimally influence the mannequin’s output, structured pruning removes teams of parameters (in our case, consultants) primarily based on redundancy or low significance. Thus, structured pruning doesn’t considerably lower kurtosis, leaving loads of margin for unstructured pruning. 

For example, if two consultants in an MoE carry out identically, one will be eliminated with out altering the mannequin’s output. Nonetheless, this doesn’t considerably affect the general weight distribution—it solely reduces the mannequin’s measurement.

Since structured pruning primarily reduces architectural redundancy quite than reshaping the underlying weight distribution, our two-phase strategy—leveraging unstructured pruning after structured pruning—outperforms unstructured-only pruning.

Placing STUN to the take a look at

Our evaluations present that STUN achieves excessive sparsity with no loss in efficiency on numerous MoE architectures, together with Snowflake’s Arctic, a 480B-sized MoE with 128 consultants.

We achieved practically no loss in efficiency with 40% sparsity, even on difficult generative duties like GSM8K (Grade College Math 8K), a broadly adopted query answering job testing on mathematical issues that require multi-step reasoning.

In some circumstances, STUN carried out orders of magnitude higher than unstructured pruning strategies. Our O(1) skilled pruning methodology additionally outperformed current, extra computationally costly strategies, akin to Lu et al. (2024), highlighting the effectiveness of our strategy.

What’s subsequent in MoE pruning?

Since STUN doesn’t make any assumption about base MoE fashions, it’s generalizable to different MoE households, akin to Mixtral. Our code is out there on GitHub. We encourage you to learn our paper and adapt it to your MoE fashions.

Past making use of and evaluating STUN, an important subsequent space of optimization is {hardware} acceleration for unstructuredly pruned fashions. Unstructured pruning removes particular person weights with out contemplating their location or association within the mannequin. Due to this, the ensuing mannequin’s sparsity is random and unaligned—some rows, columns, and even small sections might change into very sparse, whereas others stay dense.

This irregularity is difficult as a result of {hardware} like GPUs or TPUs assumes common, contiguous reminiscence layouts. Whereas structured pruning yields a predictable sparsity sample that permits for reminiscence optimization, the irregularly sparse fashions ensuing from unstructured pruning stop environment friendly reminiscence entry and parallel processing.

Specialised {hardware} help can reorganize reminiscence entry patterns to cut back overheads from irregularity. Such co-evolution of {hardware} and software program help will possible additional set up pruning as a cornerstone of scaling and making use of MoE fashions.