How to Automate Failure Attribution in LLM Multi-Agent Systems: A Step-by-Step Guide

Introduction

LLM-driven multi-agent systems are revolutionizing how we tackle complex problems—from software development to scientific reasoning. Yet one persistent headache remains: when the system fails, pinpointing which agent caused the failure and when it went wrong feels like searching for a needle in a haystack. Traditional debugging means manually sifting through endless logs and relying on deep system expertise. That's where automated failure attribution comes in. Researchers from Penn State, Duke, Google DeepMind, and other top institutions have formalized this challenge, created the first benchmark dataset (Who&When), and open-sourced their solutions. This guide walks you through the process—from setting up your environment to interpreting results—so you can diagnose failures in your own multi-agent systems quickly and reliably.

What You Need

• Python 3.8+ and a working knowledge of Python scripting
• Access to LLM APIs (e.g., OpenAI, Anthropic) or a local LLM (e.g., via Ollama)
• A multi-agent framework like AutoGen, CrewAI, or a custom orchestration layer
• The open-source code from the research: visit https://github.com/mingyin1/Agents_Failure_Attribution
• The Who&When dataset (download from Hugging Face)
• Basic logging tools to capture agent interactions (e.g., JSON, YAML)
• A notebook or IDE for experimentation (Jupyter, VS Code, etc.)

Step-by-Step Guide

Step 1: Understand the Problem of Failure Attribution

Before diving into code, grasp what automated failure attribution means. In a multi-agent system, a failure (e.g., incorrect final answer) could stem from a single agent's error, a misunderstanding between agents, or a transmission mistake. Attribution is the task of identifying the responsible agent and the critical decision point (time step) that led to the failure. The Who&When dataset provides ground truth for controlled scenarios, making it ideal for learning.

Step 2: Set Up Your Environment

Clone the repository and install dependencies:

git clone https://github.com/mingyin1/Agents_Failure_Attribution.git
cd Agents_Failure_Attribution
pip install -r requirements.txt

Make sure your LLM API keys are set as environment variables (OPENAI_API_KEY, etc.). If you are using a local model, ensure your server is running and the endpoint is accessible.

Step 3: Collect Interaction Logs from Your Multi-Agent System

To attribute failures, you first need a record of everything that happened. Instrument your multi-agent framework to log every agent message, decision, and intermediate output in a structured format (ideally JSON). Each entry should include:

• Timestamp
• Agent ID
• Action or message content
• Inputs/received messages
• Any error flags or termination conditions

For example, using AutoGen’s built-in logging:

from autogen import AssistantAgent, UserProxyAgent

# Enable logging
agent_a = AssistantAgent(name='AgentA', llm_config=llm_config, log_function=my_logger)

Run several test tasks—some likely to fail (e.g., ambiguous instructions or conflicting goals). Save the logs as separate files for each run.

Step 4: Use the Who&When Benchmark Dataset

Download the Who&When dataset from Hugging Face. It contains pre-recorded multi-agent interaction logs along with the ground-truth failure attribution (which agent, which step). Use this dataset to:

• Understand what a “failure” looks like in a controlled setting
• Evaluate attribution methods before applying them to your own logs
• Fine-tune any model (if you are using a learning-based approach)

Load the dataset in Python:

from datasets import load_dataset

dataset = load_dataset("Kevin355/Who_and_When")
print(dataset['train'][0])  # explore first sample

Step 5: Apply Automated Attribution Methods

The researchers developed and evaluated several methods. Here we outline the key approaches you can implement:

• Trajectory Analysis: Compare the successful and failed execution paths. Identify the divergence point. This can be done with sequence alignment or by monitoring agent metrics.
• Counterfactual Reasoning: For each agent, simulate what would have happened if that agent had acted differently (or been removed). If the failure disappears, that agent is the likely cause. This requires a simulator or a causal model.
• Attention/Score-Based Attribution: Use the LLM’s own attention weights or confidence scores to flag unusual low-confidence outputs. A sudden drop often indicates the point of failure.
• Learned Classifiers: Train a classifier (e.g., logistic regression or a small transformer) on the Who&When dataset to predict the failing agent and step from the log sequence.

For a quick start, the repository includes a baseline method. Run it on a sample log:

python attribute_failure.py --log_path logs/my_failure_run.json --method trajectory

Step 6: Interpret the Results

The method will output a report: responsible agent (e.g., “AgentB”) and critical step (e.g., “Step 4 – when AgentB ignored the user’s constraint”). Review the context to confirm plausibility. If the attribution seems off, check your log completeness or try another method. The Who&When dataset provides ground truth, so you can measure your accuracy on it first.

Step 7: Iterate and Improve Your System

With a clear attribution, you can now fix the issue. For example:

• If an agent consistently fails, revise its instruction prompt or ask it to double-check.
• If miscommunication is the cause, add a confirmation step between agents.
• If the failure is due to missing context, adjust the information flow.

After fixing, repeat steps 3–6 to verify the improvement. Use the automated attribution to continuously monitor new runs, catching regressions early.

Tips for Success

• Log everything, but keep it structured. The more detail you have (including intermediate thoughts if your LLM exposes them), the easier attribution becomes.
• Start with simple tasks. Use the Who&When dataset to validate your attribution pipeline before applying it to real, complex workflows.
• Combine multiple methods. No single method is perfect—use trajectory analysis as a quick filter and counterfactual reasoning for high-stakes cases.
• Leverage the community. The code and dataset are open-source. Check the repository for updates, issues, and new methods contributed by others.
• Remember the human. Automated attribution is a tool, not a replacement. Always inspect the flagged step and apply your domain expertise before making changes.
• Stay up-to-date. The paper was accepted as a Spotlight at ICML 2025. Follow the authors' future work for even better attribution techniques.

By following these steps, you turn failure diagnosis from a frustrating hunt into a systematic, automated process. Your multi-agent systems will become more reliable, and you'll spend less time debugging and more time innovating.