Laravel Llm Observability Laravel Package

Getting Started

Three commands get you running: install, publish the config, migrate.

composer require mubseoul/laravel-llm-observability
php artisan vendor:publish --tag="llm-observability-config"
php artisan migrate

Where to look first: the published config file. This package has a lot of surface area (recording mode, sampling rate, pricing tables, quota scopes, webhook endpoints), and nearly every decision you'll make is expressed there rather than in code. Read it top to bottom before wiring up a single call — it will tell you whether you're opted into async recording and whether sampling defaults to something other than 100%.

Then skim EXAMPLES.md in the repo. The README covers installation; the examples file is where the actual recording call signatures and middleware registration live.

First use case: instrument one call and verify it lands. Pick a single LLM call in your app — not all of them — and record it. Run a real request in local, then check the database table directly with tinker before touching the Filament dashboard. This separates "is the package recording?" from "is Filament rendering?" — two different failure modes you don't want to debug simultaneously.

If you're also installing the dashboard, note it's optional and versioned (Filament 3.2+). Get recording working headless first.

Implementation Patterns

Wrap your LLM calls in a single service, don't sprinkle recording calls. Whether you're using Prism, the OpenAI SDK, or raw Http:: calls, put the recording in one place. LLM observability is only as good as its coverage — a single inline call that bypasses your wrapper becomes a silent hole in your cost numbers, and you won't notice until a dashboard total doesn't reconcile. A thin LlmClient wrapper that records on success and failure is the pattern that survives contact with real code.

Async recording is the right default for anything user-facing. Sync mode adds latency to every LLM request and couples your app's availability to the observability table. Flip to async and make sure a queue worker is actually running in every environment — staging included. Pair this with a monitored queue; a stalled worker means you're flying blind, and the package won't tell you that.

Attribution is a config-and-call-site concern, not a dashboard filter. Set user and team IDs from auth() at the call site (or via a global context resolver if the config supports one). Multi-tenant apps that bolt attribution on later end up with months of unattributable rows. Decide your scoping keys — user, team, API key — before the first deploy.

Sampling is for volume, not for testing. Turn sampling to 100% in local and staging. A 10% sample makes debugging a specific failed request impossible, and the request you care about is never in the sample. Reserve sampling for production traffic where the aggregate is what matters, and document the rate where your team can find it.

Quota middleware belongs on the routes that actually consume the resource. Register it on the LLM-calling endpoints, not globally — a global quota middleware will burn quota checks on health checks and static pages. Match the middleware's scope to the limit type: per-user quotas on authenticated user routes, per-API-key quotas where keys are present. Quota resets are scheduled, so confirm schedule:run is alive in production; a missing cron means quotas never reset and everyone gets locked out on day two.

Custom pricing providers are your escape hatch. The built-in tables will lag behind new models and negotiated rates. If you're on committed-spend pricing or a proxy that rebills at a markup, implement the custom provider early rather than patching costs downstream in SQL.

Use the alert rules for error rate and latency, not just cost. Cost alerts are the obvious one and the least urgent — a cost spike is noticed at the invoice. Error-rate and latency alerts catch a broken provider integration while it's happening. Route them through Laravel Notifications if your team already lives in Slack or email; the webhook retry logic is a nice-to-have but notifications are what people actually read.

Treat Filament as an ops surface, not an end-user feature. It's genuinely useful for support ("why is this customer out of quota?") and for spot-checking model costs. Resist the urge to expose it to customers without scoping queries by tenant.

Gotchas and Tips

Token counts and costs are estimates until the provider tells you otherwise. If the package estimates tokens for streaming or for providers that don't return usage, your cost numbers are approximate. Don't put them in customer-facing invoices without reconciliation against provider billing.

Pricing tables go stale silently. New model releases and provider price changes won't trigger an error — they'll just produce wrong costs. Add a calendar reminder to check the pricing tables, or override them yourself.

Model name mismatches are the most common "cost is zero" bug. If the model string you pass doesn't match what the pricing table expects (date suffixes, provider prefixes, aliases), cost calculation falls through. When a row shows tokens but no cost, check the model string first.

Async mode loses data on failure and there's no built-in backfill. Webhook delivery has retries; recording likely doesn't. If telemetry matters for billing or compliance, consider sync mode for those specific paths or add your own fallback.

Watch table growth. Every request is a row with metadata. On a busy app this is millions of rows within months. Plan a pruning or partitioning strategy before you need it, and index the columns your dashboard actually filters on (user, team, created_at, provider).

Sampling config is easy to leave on after a load test. If you dropped sampling during an incident and forgot to raise it back, you'll have a partial dataset and no indication it's partial. Make it visible somewhere or set it from an env var that's obvious.

Middleware ordering matters if other middleware short-circuits. A quota check that runs after an auth or rate-limit middleware that returns early will never fire — fine — but a quota middleware placed before tenant resolution will check against the wrong scope. Put it after the middleware that establishes user and team context.

Extension points worth knowing: pricing providers (custom costs), the notification channel for alerts, and the recording mode/sampling config. Beyond that, the package is fairly opinionated. If you need to enrich records with custom metadata, do it at the call site when recording rather than post-processing rows.

Laravel 10 and 11 are both supported but pick your Filament version deliberately. The 3.2+ constraint is separate from your Laravel version; check compatibility against your existing Filament install before adding the dashboard to an app that already has it.