BFCL Benchmark Complete Guide 2025: Tool Calling Evaluation, Leaderboard Analysis, Model Comparison

AI Agent Capability Stack:

┌─────────────────────┐
│  Multi-Agent Collab  │  ← Impossible without Tool Calling
├─────────────────────┤
│  Multi-step Planning │  ← Tool calls at each step
├─────────────────────┤
│  ★ Tool Calling ★   │  ← Core capability
├─────────────────────┤
│  Reasoning (CoT)    │  ← Decides which tool to use
├─────────────────────┤
│  Text Generation    │  ← Foundational capability
└─────────────────────┘

Improvement Cycle:

  ┌──────────┐
  │ Evaluate │
  │ (BFCL)   │
  └────┬─────┘
       │
  ┌────▼─────┐    ┌─────────────┐    ┌─────────────┐
  │ Identify │───►│   Improve   │───►│ Re-evaluate │
  │ Weakness │    │ (prompt,    │    │  (BFCL)     │
  │          │    │  fine-tune) │    │             │
  └──────────┘    └─────────────┘    └──────┬──────┘
                                            │
                          ┌─────────────────┘
                          ▼
                    Verify → Repeat

BFCL Key Facts:
─────────────────────────────────────────
Test Cases:       2,000+ (v3)
Categories:       7 major categories
Supported Langs:  Python, Java, JavaScript
Evaluation:       AST + Executable
Leaderboard:      gorilla.cs.berkeley.edu
Latest Version:   BFCL v3 (2025)
Update Cycle:     Quarterly
Participating:    60+ models (commercial + open-source)
─────────────────────────────────────────

# User input
"What is the weather in San Francisco today?"

# Available function
def get_weather(location: str, date: str = "today") -> dict:
    """Get weather information for a specific location and date."""
    pass

# Expected output
get_weather(location="San Francisco", date="today")

# Input: "Find me flights from NYC to LA next Friday under $500"
# Available function:
def search_flights(
    origin: str,        # Airport code or city name?
    destination: str,
    date: str,          # "next Friday" -> actual date conversion?
    max_price: float,   # "$500" -> 500.0
    currency: str = "USD"
) -> list:
    pass

# Expected: search_flights(origin="NYC", destination="LA",
#           date="2025-03-28", max_price=500.0, currency="USD")

# Available functions (similar but different)
def get_current_weather(location: str, unit: str = "celsius") -> dict:
    """Get CURRENT weather conditions for a location."""
    pass

def get_weather_forecast(location: str, days: int = 7) -> dict:
    """Get weather FORECAST for upcoming days."""
    pass

def get_historical_weather(location: str, date: str) -> dict:
    """Get HISTORICAL weather data for a past date."""
    pass

def check_severe_weather_alerts(region: str) -> list:
    """Check for severe weather ALERTS in a region."""
    pass

# Test 1: "What will the weather be like in Tokyo next week?"
# Answer: get_weather_forecast(location="Tokyo", days=7)

# Test 2: "Were there any storms in Florida last month?"
# Answer: get_historical_weather(location="Florida", date="2025-02")

# Test 3: "Is it raining in Seoul right now?"
# Answer: get_current_weather(location="Seoul")

# Input: "What's the weather in Seoul, Tokyo, and New York?"

# Expected: 3 independent parallel calls
[
    get_weather(location="Seoul"),
    get_weather(location="Tokyo"),
    get_weather(location="New York")
]

# More complex case:
# "Send a greeting email to Alice and Bob, and check my calendar for tomorrow"
[
    send_email(to="alice@example.com", subject="Greeting", body="Hello Alice!"),
    send_email(to="bob@example.com", subject="Greeting", body="Hello Bob!"),
    get_calendar(date="2025-03-26")  # Different function but parallelizable
]

# Input: "Book a flight to the cheapest destination from the list"

# Step 1: Get destination prices
destinations = get_destination_prices(origin="Seoul")
# Result: [{"city": "Tokyo", "price": 300}, {"city": "Osaka", "price": 250}]

# Step 2: Book to cheapest
cheapest = min(destinations, key=lambda x: x["price"])
book_flight(origin="Seoul", destination=cheapest["city"])

# Input: "Get the manager's email of the employee who sold the most last quarter"

# Step 1: Get top seller
top_seller = get_top_seller(period="Q4-2024")
# Result: {"employee_id": "EMP-123", "name": "John"}

# Step 2: Get their manager
manager = get_manager(employee_id="EMP-123")
# Result: {"manager_id": "MGR-456", "name": "Jane"}

# Step 3: Get manager's email
email = get_employee_email(employee_id="MGR-456")
# Result: "jane@company.com"

# Scenario 1: Only irrelevant functions available
# User: "What is the meaning of life?"
# Available: get_weather(), search_products(), book_flight()
# Expected: Call no function, respond directly

# Scenario 2: Partially related but insufficient
# User: "How many calories are in a Big Mac?"
# Available: search_restaurants(cuisine, location)
# Expected: No function call (restaurant search, not calorie info)

# Scenario 3: Tempting but misuse
# User: "Tell me a joke about programming"
# Available: search_web(query)
# Expected: No function call (LLM can generate jokes directly)

Consequences of Relevance Detection Failure:
─────────────────────────────────────────
1. Unnecessary API costs
2. Degraded user experience (slow responses)
3. Hallucinations from incorrect results
4. Security risks (unnecessary data access)
─────────────────────────────────────────

# Target for evaluation
generated_call = 'get_weather(location="Seoul", unit="celsius")'

# AST parsing
import ast
tree = ast.parse(generated_call)

# Validation checks:
# 1. Is the function name correct?
# 2. Are parameter names correct?
# 3. Are parameter types correct?
# 4. Are all required parameters included?
# 5. Are there no nonexistent parameters?

# Passes AST but may fail at execution
get_weather(location="Seoull")  # Typo but syntactically valid
get_weather(location="Seoul")    # Valid (Korean might fail on some APIs)

def evaluate_executable(generated_call, expected_result):
    """
    1. Execute the generated code
    2. Compare result with expected
    3. Check for exceptions
    """
    try:
        actual_result = eval(generated_call)
        return compare_results(actual_result, expected_result)
    except TypeError as e:
        return {"status": "fail", "reason": f"Type error: {e}"}
    except Exception as e:
        return {"status": "fail", "reason": f"Execution error: {e}"}

BFCL Metric Structure:
─────────────────────────────────────────────────────

Overall Accuracy
├── AST Accuracy (Structural)
│   ├── Simple AST
│   ├── Multiple AST
│   ├── Parallel AST
│   └── Nested AST
├── Exec Accuracy (Execution)
│   ├── Simple Exec
│   ├── Multiple Exec
│   ├── Parallel Exec
│   └── Nested Exec
├── Relevance Accuracy
│   └── Unnecessary call rejection rate
└── Live Test Accuracy
    └── Against real APIs

─────────────────────────────────────────────────────

# AST Accuracy calculation
def calculate_ast_accuracy(predictions, ground_truth):
    correct = 0
    total = len(predictions)

    for pred, truth in zip(predictions, ground_truth):
        pred_ast = parse_function_call(pred)
        truth_ast = parse_function_call(truth)

        if (pred_ast.function_name == truth_ast.function_name and
            match_parameters(pred_ast.params, truth_ast.params)):
            correct += 1

    return correct / total

# Parameter matching (order-independent, type-matched)
def match_parameters(pred_params, truth_params):
    for key in truth_params:
        if key not in pred_params:
            return False
        if not type_match(pred_params[key], truth_params[key]):
            return False
    return True

# Relevance accuracy
def calculate_relevance_accuracy(predictions, labels):
    tp = sum(1 for p, l in zip(predictions, labels)
             if p == "no_call" and l == "irrelevant")
    fp = sum(1 for p, l in zip(predictions, labels)
             if p != "no_call" and l == "irrelevant")

    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    return precision

Strengths:
  + Best Relevance Detection performance (94.5%)
  + High accuracy with complex parameter extraction
  + Stable in nested call chains

Weaknesses:
  - Occasionally converts parallel calls to sequential
  - Selection accuracy drops with many tools (20+)

Strengths:
  + Best Parallel call performance (91.2%)
  + Very high JSON schema compliance
  + Streaming tool call stability

Weaknesses:
  - Lower Relevance score than Claude
  - Occasional unnecessary tool calls

Strengths:
  + Fast response speed
  + Cost-effective
  + Tool calling combined with multimodal input

Weaknesses:
  - Accuracy drops on complex nested calls
  - Some edge cases with parameter type errors

Strengths:
  + Self-hosting possible (data privacy)
  + Fine-tunable for specific domains
  + Cost savings at scale

Weaknesses:
  - Generally 5-10% lower accuracy than commercial models
  - Weaker Relevance Detection
  - Vulnerable with complex schemas

Cost Efficiency (Accuracy / Cost):
─────────────────────────────────────────
Model                   | Accuracy | Cost/1M tok | Efficiency
GPT-4o-mini             | 80.8%    | ~$0.30      | *****
Claude 3.5 Haiku        | 88.5%    | ~$2.40      | ****
Gemini 2.0 Flash        | 90.1%    | ~$0.40      | *****
Claude 3.5 Sonnet       | 92.4%    | ~$9.00      | ***
GPT-4o                  | 91.8%    | ~$7.50      | ***
Llama 3.1 70B (self)    | 81.5%    | ~$0.10*     | *****
─────────────────────────────────────────
* Estimated self-hosting cost

# Clone BFCL repository
git clone https://github.com/ShishirPatil/gorilla.git
cd gorilla/berkeley-function-call-leaderboard

# Install dependencies
pip install -r requirements.txt

# Or install directly via pip
pip install bfcl

# Basic evaluation
from bfcl import evaluate

# Evaluate an OpenAI model
results = evaluate(
    model="gpt-4o",
    categories=["simple", "multiple", "parallel", "relevance"],
    api_key="your-openai-api-key"
)

print(f"Overall Accuracy: {results['overall']:.2%}")
print(f"Simple: {results['simple']:.2%}")
print(f"Multiple: {results['multiple']:.2%}")
print(f"Parallel: {results['parallel']:.2%}")
print(f"Relevance: {results['relevance']:.2%}")

# CLI execution
python eval.py \
    --model gpt-4o \
    --categories simple multiple parallel relevance \
    --output-dir ./results

# Anthropic models
python eval.py \
    --model claude-3-5-sonnet \
    --categories all \
    --output-dir ./results

# Local model (vLLM server)
python eval.py \
    --model local \
    --api-base http://localhost:8000/v1 \
    --categories all

from bfcl import BFCLEvaluator

class MyModelHandler:
    """Custom model handler"""

    def __init__(self, model_path):
        self.model = load_my_model(model_path)

    def generate(self, prompt, tools, **kwargs):
        """
        Interface called by BFCL.
        prompt: User input
        tools: Available tool definitions
        Returns: Function call string or "NO_CALL"
        """
        formatted_prompt = self.format_prompt(prompt, tools)
        response = self.model.generate(formatted_prompt)
        return self.parse_tool_call(response)

    def format_prompt(self, prompt, tools):
        tool_descriptions = "\n".join([
            f"Function: {t['name']}\n"
            f"Description: {t['description']}\n"
            f"Parameters: {json.dumps(t['parameters'])}"
            for t in tools
        ])
        return f"""Available functions:
{tool_descriptions}

User query: {prompt}

Respond with a function call or "NO_CALL" if no function is relevant."""

# Run evaluation
evaluator = BFCLEvaluator()
handler = MyModelHandler("/path/to/model")

results = evaluator.evaluate(
    handler=handler,
    categories=["simple", "multiple", "parallel", "relevance"],
    output_dir="./my_model_results"
)

evaluator.generate_report(results, "./report.html")

custom_test = {
    "id": "custom_001",
    "category": "simple",
    "prompt": "Send a 'Meeting starting' message to the team Slack channel",
    "available_functions": [
        {
            "name": "send_slack_message",
            "description": "Send a message to a Slack channel",
            "parameters": {
                "type": "object",
                "properties": {
                    "channel": {
                        "type": "string",
                        "description": "Slack channel name"
                    },
                    "message": {
                        "type": "string",
                        "description": "Message text"
                    }
                },
                "required": ["channel", "message"]
            }
        }
    ],
    "ground_truth": 'send_slack_message(channel="team", message="Meeting starting")',
    "acceptable_variants": [
        'send_slack_message(channel="team", message="Meeting starting")',
        'send_slack_message(message="Meeting starting", channel="team")',
    ]
}

evaluator.evaluate_custom(
    handler=handler,
    test_cases=[custom_test],
    output_dir="./custom_results"
)

import json

with open("./results/evaluation_results.json") as f:
    results = json.load(f)

# Category-wise accuracy
for category, accuracy in results["categories"].items():
    print(f"{category}: {accuracy:.2%}")

# Failure case analysis
failures = results["failures"]
for failure in failures[:5]:
    print(f"\nTest ID: {failure['id']}")
    print(f"Category: {failure['category']}")
    print(f"Prompt: {failure['prompt']}")
    print(f"Expected: {failure['expected']}")
    print(f"Got: {failure['predicted']}")
    print(f"Error Type: {failure['error_type']}")
    # error_type: wrong_function, wrong_params, unnecessary_call, missing_call

# API-Bank: 3-level evaluation
# Level 1: API calling ability (single)
# Level 2: API search + call (finding the right API)
# Level 3: API composition + planning (multi-step)

# Example (Level 3):
# "Check if I have a meeting tomorrow morning, and if so, notify the attendees"
# -> Step 1: check_calendar(date="tomorrow", time="morning")
# -> Step 2: if meeting exists, get_attendees(meeting_id=...)
# -> Step 3: send_notification(recipients=..., message=...)

# ToolBench: Based on 16,000+ real APIs from RapidAPI
# Uses actual API documentation for realistic scenarios

# Categories:
# - Single Tool: Single API usage
# - Intra-Category: Multiple APIs within same category
# - Inter-Category: APIs from different categories combined

# Metrics:
# - Pass Rate: Execution success rate
# - Win Rate: Preference vs other models (GPT-4 evaluated)

# T-Eval: Granular evaluation of each stage of tool use
# Measures 6 sub-capabilities:

# 1. Instruct Following: Understanding instructions
# 2. Plan: Formulating action plans
# 3. Reason: Reasoning about correct tools
# 4. Retrieve: Finding appropriate tools
# 5. Understand: Comprehending tool documentation
# 6. Review: Verifying and correcting results

Which benchmark should you use?
─────────────────────────────────────────────────
Purpose                          | Recommended
Comprehensive Tool Calling eval  | BFCL
Large-scale real API testing     | ToolBench
Granular step-by-step analysis   | T-Eval
Multi-step API planning eval     | API-Bank
Quick basic evaluation           | BFCL (Simple only)
Custom model comparison          | BFCL + custom tests
─────────────────────────────────────────────────

import json
from openai import OpenAI

def generate_training_data(tools, num_examples=1000):
    """Generate training data using GPT-4o"""
    client = OpenAI()
    training_data = []

    tool_descriptions = json.dumps(tools, indent=2)

    for i in range(num_examples):
        # 1. Generate natural language query
        query_response = client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {"role": "system", "content": f"""Generate a natural language user query
that would require calling one of these tools:
{tool_descriptions}

Generate diverse, realistic queries. Include edge cases.
Respond with ONLY the query text."""},
                {"role": "user", "content": f"Generate query #{i+1}"}
            ]
        )
        query = query_response.choices[0].message.content

        # 2. Generate correct function call
        call_response = client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": query}],
            tools=[{"type": "function", "function": t} for t in tools],
            tool_choice="auto"
        )

        if call_response.choices[0].message.tool_calls:
            tc = call_response.choices[0].message.tool_calls[0]
            training_data.append({
                "messages": [
                    {"role": "system", "content": f"You have access to: {tool_descriptions}"},
                    {"role": "user", "content": query},
                    {"role": "assistant", "content": None, "tool_calls": [
                        {
                            "type": "function",
                            "function": {
                                "name": tc.function.name,
                                "arguments": tc.function.arguments
                            }
                        }
                    ]}
                ]
            })

    return training_data

data = generate_training_data(my_tools, num_examples=5000)
with open("tool_calling_train.jsonl", "w") as f:
    for item in data:
        f.write(json.dumps(item) + "\n")

# Iterative optimization process

# Step 1: Initial description
v1 = {
    "name": "search_products",
    "description": "Search for products"  # Too simple
}

# Step 2: Add clear purpose
v2 = {
    "name": "search_products",
    "description": "Search for products in the catalog by name, category, or keywords. Returns matching products with price and availability."
}

# Step 3: Add usage conditions
v3 = {
    "name": "search_products",
    "description": """Search for products in the e-commerce catalog.

USE WHEN: User wants to find, browse, or compare products.
DO NOT USE: For order status (use get_order), account info (use get_account), or returns (use create_return).

Returns: List of products with name, price, rating, availability."""
}

# Step 4: Add examples (final)
v4 = {
    "name": "search_products",
    "description": """Search for products in the e-commerce catalog.

USE WHEN: User wants to find, browse, or compare products.
DO NOT USE: For order status, account info, or returns.

EXAMPLES:
- "wireless headphones" -> query="wireless headphones"
- "cheap laptops under $500" -> query="laptops", max_price=500
- "best rated phones" -> query="phones", sort_by="rating"

Returns: List of products with name, price, rating, availability."""
}

system_prompt = """You are an AI assistant with access to tools.

IMPORTANT RULES:
1. ONLY call a function when the user's request CLEARLY requires it.
2. If you can answer directly from your knowledge, do NOT call any function.
3. When calling functions, ensure ALL required parameters are provided.
4. Use the EXACT parameter names and types defined in the function schema.
5. If a user's request is ambiguous, ask for clarification BEFORE calling.

PARAMETER GUIDELINES:
- Dates: Use ISO 8601 format (YYYY-MM-DD)
- Locations: Use the most common English name
- Numbers: Use numeric type, not string
- Booleans: Use true/false, not "yes"/"no"

WHEN NOT TO CALL FUNCTIONS:
- General knowledge questions
- Opinions or advice
- Greetings or small talk
- Math that you can calculate yourself
"""

def analyze_errors(results):
    """Analyze error patterns from BFCL results"""

    error_categories = {
        "wrong_function": [],
        "missing_params": [],
        "wrong_param_type": [],
        "extra_params": [],
        "unnecessary_call": [],
        "missing_call": [],
        "wrong_value": [],
    }

    for failure in results["failures"]:
        error_type = classify_error(failure)
        error_categories[error_type].append(failure)

    print("Error Distribution:")
    print("=" * 50)
    total = sum(len(v) for v in error_categories.values())
    for category, errors in sorted(
        error_categories.items(),
        key=lambda x: len(x[1]),
        reverse=True
    ):
        pct = len(errors) / total * 100 if total > 0 else 0
        print(f"  {category}: {len(errors)} ({pct:.1f}%)")

    # Identify most common error patterns
    print("\nTop Error Patterns:")
    for category, errors in error_categories.items():
        if errors:
            print(f"\n{category}:")
            patterns = find_common_patterns(errors)
            for pattern, count in patterns[:3]:
                print(f"  - {pattern} ({count} occurrences)")

    return error_categories

Tool Calling Improvement Cycle:
─────────────────────────────────────────────────

Step 1: Measure Current Performance
  - Run full BFCL categories
  - Record per-category accuracy

Step 2: Identify Weaknesses
  - Run error analysis
  - Identify most frequent error types
  - Analyze failure patterns

Step 3: Implement Improvements
  ├─ Prompt Improvements (quick wins)
  │   - Improve Tool Descriptions
  │   - Optimize System Prompt
  │   - Add Few-shot examples
  ├─ Tool Design Improvements (medium)
  │   - Simplify schemas
  │   - Consolidate related tools
  │   - Clarify parameter names
  └─ Fine-tuning (long-term)
      - Generate training data from failures
      - LoRA/QLoRA fine-tuning
      - Evaluate + iterate

Step 4: Re-evaluate
  - Re-run same benchmark
  - Verify improvement
  - Identify new weaknesses

-> Repeat from Step 2

Benchmark Limitations:
─────────────────────────────────────────────────
1. Ambiguous User Input
   Benchmark: "Weather in Seoul" (clear)
   Real world: "What's the weather?" (no location/time)

2. Conversation Context
   Benchmark: Single-turn tests
   Real world: "there" refers to location from earlier

3. Error Recovery
   Benchmark: Only tests successful responses
   Real world: API failures, timeouts, bad responses

4. Tool Count Explosion
   Benchmark: 5-10 tools
   Real world: 50-100 tools simultaneously

5. Real-time Performance
   Benchmark: Measures only accuracy
   Real world: Speed, cost, and reliability all matter
─────────────────────────────────────────────────

class ProductionEvalSuite:
    def __init__(self, tools, model):
        self.tools = tools
        self.model = model
        self.test_cases = []

    def add_test_case(self, category, prompt, expected, context=None):
        self.test_cases.append({
            "category": category,
            "prompt": prompt,
            "expected": expected,
            "context": context or []
        })

    def build_standard_suite(self):
        # 1. Basic functionality
        self.add_test_case(
            "basic", "What's the weather in Seoul?",
            "get_weather(location='Seoul')"
        )

        # 2. Ambiguous input
        self.add_test_case(
            "ambiguous", "What's the weather?",
            "ASK_CLARIFICATION"
        )

        # 3. Multi-turn context
        self.add_test_case(
            "multi_turn",
            "What about tomorrow there?",
            "get_weather(location='Seoul', date='tomorrow')",
            context=[
                {"role": "user", "content": "Weather in Seoul?"},
                {"role": "assistant", "content": "Seoul is currently 15C."}
            ]
        )

        # 4. Relevance
        self.add_test_case(
            "relevance", "What is the meaning of life?",
            "NO_CALL"
        )

        # 5. Error recovery
        self.add_test_case(
            "error_recovery",
            "What's the weather in Seoul?",
            "RETRY_OR_FALLBACK",
            context=[
                {"role": "tool", "content": "ERROR: API timeout"}
            ]
        )

    def run(self):
        results = {"total": 0, "correct": 0, "by_category": {}}
        for test in self.test_cases:
            result = self.evaluate_single(test)
            results["total"] += 1
            if result["correct"]:
                results["correct"] += 1
            cat = test["category"]
            if cat not in results["by_category"]:
                results["by_category"][cat] = {"total": 0, "correct": 0}
            results["by_category"][cat]["total"] += 1
            if result["correct"]:
                results["by_category"][cat]["correct"] += 1
        results["accuracy"] = results["correct"] / results["total"]
        return results

class ToolCallingMonitor:
    def __init__(self):
        self.metrics = {
            "total_calls": 0,
            "successful_calls": 0,
            "failed_calls": 0,
            "unnecessary_calls": 0,
            "latency_sum": 0,
            "cost_sum": 0,
            "by_tool": {},
        }

    def record_call(self, tool_name, success, latency, cost,
                    was_necessary=True):
        self.metrics["total_calls"] += 1
        if success:
            self.metrics["successful_calls"] += 1
        else:
            self.metrics["failed_calls"] += 1
        if not was_necessary:
            self.metrics["unnecessary_calls"] += 1
        self.metrics["latency_sum"] += latency
        self.metrics["cost_sum"] += cost

    def get_dashboard_data(self):
        total = self.metrics["total_calls"]
        if total == 0:
            return {}
        return {
            "success_rate": self.metrics["successful_calls"] / total,
            "failure_rate": self.metrics["failed_calls"] / total,
            "unnecessary_rate": self.metrics["unnecessary_calls"] / total,
            "avg_latency": self.metrics["latency_sum"] / total,
            "total_cost": self.metrics["cost_sum"],
        }

    def alert_on_anomaly(self):
        data = self.get_dashboard_data()
        alerts = []
        if data.get("failure_rate", 0) > 0.1:
            alerts.append("HIGH: Tool call failure rate above 10%")
        if data.get("unnecessary_rate", 0) > 0.2:
            alerts.append("MEDIUM: Unnecessary tool calls above 20%")
        if data.get("avg_latency", 0) > 5.0:
            alerts.append("MEDIUM: Average latency above 5 seconds")
        return alerts