With the increasing adoption of multi-agent task-oriented dialogue (TOD) systems, testing becomes significantly challenging due to unstructured natural language specifications and complex cross-agent transitions. Such systems are prone to subtle errors that often emerge only during multi-turn interactions. While manual testing is costly and difficult to scale, directly leveraging LLMs for test case generation offers a more attractive alternative. In this work, we propose TIDAL, a system that automatically synthesizes test scenarios and generates simulated dialogues for multi-agent TOD systems. TIDAL introduces a logic flow-aware generation strategy that guides LLMs toward previously uncovered behaviors and interaction paths, thereby improving both testing efficiency and behavioral coverage. Experimental results show that TIDAL achieves faster coverage growth and higher final coverage than baseline approaches. We also construct and will release a multi-domain benchmark for evaluating testing methodologies for multi-agent TOD systems.

As artificial intelligence (AI) assistants become more widely adopted in safetycritical domains, it becomes important to develop safeguards against potential failures or adversarial attacks. A key prerequisite to developing these safeguards is understanding the ability of these AI assistants to mislead human teammates. We investigate this attack problem within the context of an intellective strategy game where a team of three humans and one AI assistant collaborate to answer a series of trivia questions. Unbeknownst to the humans, the AI assistant is adversarial. Leveraging techniques from Model-Based Reinforcement Learning (MBRL), the AI assistant learns a model of the humans’ trust evolution and uses that model to manipulate the group decision-making process to harm the team. We evaluate two models—one inspired by literature and the other data-driven—and find that both can effectively harm the human team. Moreover, we find that in this setting while our data-driven model is the most capable of accurately predicting how human agents appraise their teammates given limited information on prior interactions, the model based on principles of cognitive psychology does not lag too far behind. Finally, we compare the performance of state-of-the-art LLM models to human agents on our influence allocation task to evaluate whether the LLMs allocate influence similarly to humans or if they are more robust to our attack. These results enhance our understanding of decision-making dynamics in small human-AI teams and lay the foundation for defense strategies.

There are numerous frameworks capable of creating and orchestrating agents to address complex tasks. However, most of them highly coupled Python programming with agent declaration, making it hard for maintenance and runtime optimization. In this work, we introduce ADL, an agent declarative language for customer service chatbots. ADL abstracts away implementation details, offering a declarative way to define agents and their interactions, which could ease maintenance and debugging. It also incorporates natural language programming at its core to simplify the specification and communication of chatbot designs. ADL includes four basic types of agents and supports integration with custom functions, tool use, and third-party agents. MICA, a multiagent system designed to interpret and execute ADL programs, has been developed and is now available as an open-source project here. Its documentation can be found here.

As language agents increasingly automate critical tasks, their ability to follow domain-specific standard operating procedures (SOPs), policies, and constraints when taking actions and making tool calls becomes essential yet remains underexplored. To address this gap, we develop an automated evaluation pipeline SOPBench with: (1) executable environments containing 167 tools/functions across seven customer service domains with service-specific SOPs and rule-based verifiers, (2) an automated test generation framework producing over 900 verified test cases, and (3) an automated evaluation framework to rigorously assess agent adherence from multiple dimensions. Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions. The original code serves as oracle rule-based verifiers to assess compliance, reducing reliance on manual annotations and LLM-based evaluations. We evaluate 18 leading models, and results show the task is challenging even for top-tier models (like GPT-4o, Claude-3.7-Sonnet), with variances across domains. Reasoning models like o4-mini-high show superiority while other powerful models perform less effectively (pass rates of 30%-50%), and small models (7B, 8B) perform significantly worse. Additionally, language agents can be easily jailbroken to overlook SOPs and constraints. Code, data, and over 24k agent trajectories are released at github.