Abstract-Level Summary

The research introduces "Mobile-Agent-E," a hierarchical, multi-agent framework designed to improve mobile assistant functionalities by overcoming current limitations in addressing complex, long-horizon tasks on smartphones. The study employs a multi-agent system featuring roles such as a Manager for planning and various specialized agents for action execution and error handling. Notably, the framework incorporates a unique self-evolution module that utilizes accumulated experience to refine its capabilities over time. Mobile-Agent-E demonstrated a 22% performance increase over existing mobile agents across several foundational model backbones when assessed through the newly developed Mobile-Eval-E benchmark, which simulates real-world multi-app interactions. The study suggests significant enhancements in both mobile efficiency and user experience.

Introduction Highlights

The study addresses the inefficiencies of mobile assistants in managing complex, multi-step tasks that require intensive reasoning and adaptability. Existing agents lack the capability to learn and improve over time, often treating every task as a first-time endeavor, leading to repeated mistakes and inefficient task handling. Mobile-Agent-E is proposed as a solution to bridge these gaps by focusing on hierarchical task decomposition and incorporating a mechanism for self-evolution through accumulated user experience, aiming to improve both efficiency and accuracy in long-term mobile task management.

Methodology

The approach deploys a hierarchical framework where tasks are split between high-level planning by a Manager and execution by subordinate agents such as the Perceptor, Operator, Action Reflector, and Notetaker. Mobile-Agent-E's novel self-evolution component continuously updates its knowledge base with "Tips" and "Shortcuts," derived from previous experiences, to enhance future performance. Additionally, a new benchmark, Mobile-Eval-E, was created to effectively evaluate the system in real-world contexts, accompanied by a Satisfaction Score to gauge user satisfaction against a rubric-based standard instead of binary success metrics.

Key Findings

• Mobile-Agent-E showed a significant 22.1% improvement in satisfaction scores compared to prior models using the Mobile-Eval-E benchmark.
• The self-evolution mechanism yielded a 6.5% performance increase by utilizing learned shortcuts and tips.
• Compared to existing state-of-the-art models, Mobile-Agent-E greatly enhanced long-term planning and accuracy while significantly reducing task termination errors.

Implications and Contributions

The study presents a pivotal advancement in mobile assistant technology by demonstrating the feasibility and effectiveness of hierarchical task management and self-evolution in improving user interaction with mobile devices. This framework's potential application spans various sectors, enhancing productivity, accessibility for users with disabilities, and possibly refining other AI systems predicated on multi-agent collaboration. The findings set a foundation for further research into personalized user experience and optimizing computational efficiency in complex task environments.

Conclusion

Mobile-Agent-E establishes a benchmark for superior mobile agent performance, leveraging hierarchical agent deployment and experiential learning. Initial limitations include a dependency on the accurate prediction of task preconditions and successful execution of newly proposed shortcuts. Future research should focus on enhancing the personalization of shortcuts and strengthening model safety to align with user intent while maintaining privacy and security.

Glossary

1. Mobile-Agent-E: A hierarchical multi-agent mobile assistant framework designed for complex task management and self-evolution.
2. Large Multimodal Model (LMM): Advanced AI frameworks capable of processing multiple forms of data such as text and visual information.
3. Self-Evolution Module: A component allowing an AI system to learn from past experiences by updating its capabilities with tips and shortcuts.
4. Satisfaction Score: A performance metric based on human evaluator judgments against rubric-based task completion criteria.
5. Multimodal: Involving multiple types of data or sensory inputs, common in advanced AI tasks involving images, text, and speech.