Apple study finds minimal agent matches or beats multi-agent ML engineering systems

The researchers pitted Malena against four more elaborate systems: MLEvolve, AiScientist, Arbor and ScienceFlow. Each was tested under matched conditions using frontier large language models, so the comparison isolated the harness rather than the underlying model.

The Numbers

On MLE-bench, a benchmark built to measure how well agents handle machine learning engineering work, Malena posted a 62.5% any-medal rate. That figure tracks how often an agent's results were strong enough to earn a medal on a given task.

Comparison to Other Systems

The best-performing external harness, AiScientist, managed 47.1%. That works out to a gap of 15.4 percentage points in favor of the system with fewer moving parts.

Evaluation Settings

The evaluation covered two settings across MLE-bench and NatureBench. One set included 30 tasks run within a 24-hour budget, and another included 40 tasks run over an 8-hour budget.

Limitations of the Study

The researchers also flagged a limitation. Compute budgets were constrained, which kept the number of seeds, or repeated runs with different random starting points, modest. Fewer seeds mean results carry more statistical noise than a lab with unlimited GPUs might accept.

Conclusion of the Study

The team's core conclusion points at the model itself. The strength of the underlying model was the main driver of performance, and additional harness complexity was often futile on current benchmarks.

Implications for AI Builders

For researchers, the study raises a methodological flag. If harness improvements are reported without matched backbone models, gains attributed to clever architecture may actually come from a stronger underlying LLM.

Template for Future Harness Papers

Apple's matched-conditions setup offers a template for separating those effects. Future harness papers may face pressure to show their gains hold when the model is held constant.

Current Benchmarks

The researchers framed their conclusion around current benchmarks, and modest seed counts mean some margins could narrow with more runs. MLE-bench and NatureBench measure specific kinds of ML engineering work.

Limitations of Current Benchmarks

The study's limitations highlight the need for more comprehensive benchmarks that account for the nuances of real-world ML engineering tasks.

Future Research Directions

Future research should focus on developing more robust and generalizable benchmarks that can capture the full range of ML engineering challenges.

Conclusion

The study's findings have significant implications for the AI industry, highlighting the importance of the underlying model's strength and the limitations of harnesses in achieving impressive results.

Future Directions

Future research should aim to develop more comprehensive and robust benchmarks that can capture the full range of ML engineering challenges.

Conclusion

The study's findings raise important questions about the effectiveness of harnesses in AI development and highlight the need for a more nuanced understanding of the relationship between the model and the harness.