Sql Agent Laravel Package

title: Evaluation description: Test agent accuracy with automated evaluations against known test cases. sidebar: order: 6

The evaluation system helps you measure and improve your agent's accuracy by running it against a suite of test cases with known expected outcomes.

Creating Test Cases

Test cases are stored in the sql_agent_test_cases table. You can seed them using the built-in seeder (--seed flag) or create your own:

use Knobik\SqlAgent\Models\TestCase;

TestCase::create([
    'name' => 'Count active users',
    'category' => 'basic',
    'question' => 'How many active users are there?',
    'expected_values' => ['count' => 42],
    'golden_sql' => 'SELECT COUNT(*) as count FROM users WHERE status = "active"',
    'golden_result' => [['count' => 42]],
]);

Running Evaluations

# Run all test cases
php artisan sql-agent:eval

# Run with LLM grading
php artisan sql-agent:eval --llm-grader

# Run a specific category
php artisan sql-agent:eval --category=aggregation

# Generate an HTML report
php artisan sql-agent:eval --html=storage/eval-report.html

# Seed built-in test cases first
php artisan sql-agent:eval --seed

Evaluation Modes

Three evaluation modes are available: