A Comprehensive Look at UNO-Bench: An Open Benchmark for Unified Evaluation of Multimodal Understanding and Reasoning

I. Abstract

UNO-Bench is an open-source benchmark for unified evaluation of "single-model/full-model" questions, covering both perception and reasoning dimensions. It provides Chinese real-world scenario questions and multi-step open-ended question answering (MO) questions. The data and tools emphasize high quality and human-led construction, and are equipped with a general scoring model for automated evaluation.

II. Core Features

1. Unified capability framework: 44 types of tasks, 5 modal combinations, with the same indicator caliber for single-modality and full-modality tasks.
2. High quality and solvability: 1250 full-modal data points, human-reviewed construction, 98% solvable across modalities.
3. Efficiency optimization: Automatic compression of 18 public benchmarks speeds up evaluation by approximately 90% and improves consistency by approximately 98%.
4. More realistic question types: Multi-step open-ended questions and answers have been added to cover complex reasoning chains.
5. General scoring: Supports 6 types of questions, with approximately 95% consistency in annotation in OOD scenarios.
6. Key findings: Strong models exhibit "power-law synergy" (capabilities increase multiplicatively with modal combinations).

III. Installation

1. Dataset: datasets.load_dataset("meituan-longcat/UNO-Bench") Retrieves the default shards.

1. Source code and documentation: View the README and evaluation script examples in the cloned GitHub repository.
2. Environment: Python/Transformers/Datasets. A standard environment is sufficient. Install dependencies according to the repository instructions.

IV. Typical Use Cases

1. Model cross-sectional evaluation: Compare the differences between single-model and full-model under a unified scale.
2. Chinese scenario verification: Perception and reasoning ability in real-life/cultural/social contexts.
3. Reasoning chain analysis: Use multi-step open-ended question answering to diagnose weaknesses in long-chain reasoning.
4. RAG/Multimodal System: Validate the overall benefits of audio, image, and video fusion.

V. Ecology and Competitors

1. Ecosystem: Provides datasets, leaderboards, and papers; the toolchain is under development.
2. Competitors: Compared with visual/subject-specific benchmarks such as MMBEC, MMMU, and MathVista, UNO-Bench emphasizes "unified evaluation of single-mode to full-mode" and real-world Chinese scenarios; its compression method facilitates rapid alignment of multiple benchmarks.

VI. Limitations and Precautions

1. The applicability of automatic compression needs to be verified on a task-by-task basis; some sub-tasks may lack sufficient information.
2. The general scoring model may still have biases for long answers/generative outputs, and it is recommended to manually review samples.
3. Currently, the focus is on Chinese-language scenarios, and collaborations for multilingual extensions and English versions are still being solicited.
4. "Power-law synergy" is an empirical discovery, and it needs to be re-verified when transferred to new tasks.

VII. Project Address

https://github.com/meituan-longcat/UNO-Bench

VIII. Frequently Asked Questions

Q: What modalities and tasks does UNO-Bench cover?

A: It covers combinations of audio, images, and video, with a total of 5 modal combinations and 44 task categories, targeting both perception and reasoning dimensions.

Q: How can I quickly run the UNO-Bench benchmark?

A: Load data via Hugging Face, and perform inference and scoring using sample scripts from the repository and a general scoring model.

Q: How much does automatic compression affect the reliability of the results?

A: The ranking consistency is maintained at approximately 98% across 18 publicly available benchmarks, but it is still recommended to combine this with sampling of the original set.

Q: Does it support English or multiple languages?

A: The official focus is currently on the Chinese language version, and we are looking for partners to jointly develop English and multilingual versions.

Q: Does power-law collaboration hold true for all models?

A: It is mainly significant in strong models; for weak models, it is more like a "weakest link effect" and needs to be specifically evaluated and confirmed.