Princeton University Researchers Introduce Self-MoA and Self-MoA-Seq: Optimizing LLM Performance with Single-Model Ensembles

Large Language Models (LLMs) such as GPT, Gemini, and Claude utilize vast training datasets and complex architectures to generate high-quality responses. However, optimizing their inference-time computation remains challenging, as increasing model size leads to higher computational costs. Researchers continue to explore strategies that maximize efficiency while maintaining or improving model performance.

One widely adopted approach for improving LLM performance is ensembling, where multiple models are combined to generate a final output. Mixture-of-Agents (MoA) is a popular ensembling method that aggregates responses from different LLMs to synthesize a high-quality response. However, this method introduces a fundamental trade-off between diversity and quality. While combining diverse models may offer advantages, it can also result in suboptimal performance due to the inclusion of lower-quality responses. Researchers aim to balance these factors to ensure optimal performance without compromising response quality.

Traditional MoA frameworks operate by first querying multiple proposer models to generate responses. An aggregator model then synthesizes these responses into a final answer. The effectiveness of this method relies on the assumption that diversity among proposer models leads to better performance. However, this assumption does not account for potential quality degradation caused by weaker models in the mix. Prior research has primarily focused on increasing cross-model diversity rather than optimizing proposer models’ quality, leading to performance inconsistencies.

A research team from Princeton University introduced Self-MoA, a novel ensembling method that eliminates the need for multiple models by aggregating various outputs from a single high-performing model. Unlike traditional MoA, which mixes different LLMs, Self-MoA leverages in-model diversity by repeatedly sampling from the same model. This approach ensures that only high-quality responses contribute to the final output, addressing the quality-diversity trade-off observed in Mixed-MoA configurations.

Self-MoA operates by generating multiple responses from a single top-performing model and synthesizing them into a final output. Doing so eliminates the need to incorporate lower-quality models, thereby improving overall response quality. To further enhance scalability, researchers introduced Self-MoA-Seq, a sequential variation that processes multiple responses iteratively. This allows for efficient aggregation of outputs even in scenarios where computational resources are constrained. Self-MoA-Seq processes outputs using a sliding window approach, ensuring that LLMs with shorter context lengths can still benefit from ensembling without compromising performance.

Experiments demonstrated that Self-MoA significantly outperforms Mixed-MoA across various benchmarks. On the AlpacaEval 2.0 benchmark, Self-MoA achieved a 6.6% improvement over traditional MoA. When tested across multiple datasets, including MMLU, CRUX, and MATH, Self-MoA showed an average improvement of 3.8% over Mixed-MoA approaches. When applied to one of the top-ranking models in AlpacaEval 2.0, Self-MoA set a new state-of-the-art performance record, further validating its effectiveness. Further, Self-MoA-Seq proved to be as effective as aggregating all outputs simultaneously while addressing the limitations imposed by model context length constraints.

The research findings highlight a crucial insight into MoA configurations—performance is highly sensitive to proposer quality. The results confirm that incorporating diverse models does not always lead to superior performance. Instead, ensembling responses from a single high-quality model yields better outcomes. Researchers conducted over 200 experiments to analyze the trade-off between quality and diversity, concluding that Self-MoA consistently outperforms Mixed-MoA when the best-performing model is used exclusively as the proposer.

This study challenges the prevailing assumption that mixing different LLMs leads to better results. By demonstrating the superiority of Self-MoA, it presents a new perspective on optimizing LLM inference-time computation. The findings indicate that focusing on high-quality individual models rather than increasing diversity can improve overall performance. As LLM research continues to evolve, Self-MoA provides a promising alternative to traditional ensembling methods, offering an efficient and scalable approach to enhancing model output quality.

Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and join our Telegram Channel and LinkedIn Group. Don’t Forget to join our 75k+ ML SubReddit.

read more