The Meituan LongCat team has launched VitaBench, an intelligent agent benchmark suite. This framework selects three high-frequency life service scenarios: food delivery, restaurant dine-in, and online travel. It abstracts 66 composable tools and constructs 100 cross-scenario and 300 single-scenario tasks. The system systematically measures reasoning, tool usage, and adaptive interaction capabilities. The project page and paper simultaneously publish code, data, and leaderboards for easy replication and comparison.
The latest results show a cross-scenario Avg@4 of 30.0% and a single-scenario Avg@4 of 48.3%. The cross-scenario Pass@4 rate is approximately 60%, and the Pass⁴ (4/4 complete success) rate is close to 0%. These data indicate that while top-tier "thinking/non-thinking" models have potential in certain areas, they lack stability and production-level reliability in complex real-world environments, and still face challenges in long-term planning, cross-domain switching, and multi-tool collaboration.
Frequently Asked Questions
Q: What capabilities and scenarios does VitaBench cover?
A: Covers food delivery, catering, and online travel, and evaluates reasoning, tool calling, and adaptive interaction, including 66 tools and 400 tasks.
Q: How to interpret key indicators?
A: Avg@4 is the average success rate of four attempts; Pass@4 means at least one success out of four attempts; Pass⁴ means all four attempts were successful.
Q: Why is cross-scenario more difficult?
A: It requires long-term planning and switching between a larger tool set and cross-domain rules, which can easily lead to error accumulation and decreased stability.
Q: Is it open source?
A: Yes. We provide a project page, code repository, and Hugging Face dataset, and maintain a public leaderboard.
Q: What insights are there for implementation?
A: The current availability and consistency of intelligent agents in real business are still insufficient, and tool dependency modeling, robustness and error correction strategies need to be strengthened.
