Save this content to: .claude/skills/vigilance-development/SKILL.md

# Vigilance Development

Vigilance is a self-contained Livewire control center for Laravel queues, jobs, commands and the scheduler. It is **driver-agnostic** (database, redis, sqs, beanstalkd, sync) and **production-first** (sampling, size caps, secret redaction, a crash-proof capture guard). The dashboard is at `/vigilance`. All config lives in `config/vigilance.php` and is documented inline.

## When to use this skill

Use it when you are: installing/configuring Vigilance, securing the dashboard, deciding what gets captured, enabling manual job/command dispatch, replacing Horizon/`queue:work` with the supervisor, wiring up APM or tracing, enabling the observability layers (error tracking, route/front-end performance, RUM, SLOs, custom metrics, the log explorer), or adding alerting (including custom rules, extra channels and incident tracking).

## Install & authorize

```bash
composer require anousss007/vigilance
php artisan vigilance:install   # publishes config + prints next steps
php artisan migrate             # migrations are auto-loaded
php artisan vigilance:doctor    # diagnose common misconfigurations
```

The dashboard is **local-only until you authorize it** (secure by default). Use a Gate (preferred — same idiom as Horizon's `viewHorizon`) or a closure, in a service provider's `boot()`:

```php
use Illuminate\Support\Facades\Gate;

Gate::define('viewVigilance', fn ($user) => $user->isAdmin());
// or: \Vigilance\Vigilance::auth(fn ($request) => in_array($request->user()?->email, ['[email protected]']));
```

> **`web` middleware caveat:** the dashboard inherits `config('vigilance.middleware')` (default `['web']`). If your `web` group appends global redirects (locale prefixing, maintenance/teaser pages, forced auth) they will rewrite or 404 the dashboard URL — the same caveat Horizon/Pulse/Telescope carry. Skip the `vigilance` path in those middleware, or trim `vigilance.middleware`.

## What is captured — and how to exclude

Capture is automatic via framework events. **Never** add manual tracking. To reduce noise, prefer exclusions over disabling:

- Mark a job/command class with `Vigilance\Contracts\ShouldNotBeMonitored`.
- Or list patterns in `except.jobs` / `except.commands` in config.
- Tune volume with `capture.sample_rate` (fraction of **successful** runs kept; failures are always captured) and keep monitoring writes off your primary DB with `storage.connection`.
- **Storage isolation:** Vigilance writes to the app's default connection unless `VIGILANCE_DB_CONNECTION` names a dedicated one. Prefer a dedicated connection in production. On SQLite especially — it locks per file, so a telemetry burst can block the app (`database is locked`) — give Vigilance its own SQLite file with `'journal_mode' => 'WAL'`, then `VIGILANCE_DB_CONNECTION=that`. A single SQLite file is one-writer-at-a-time; at high volume move to MySQL/Postgres or `VIGILANCE_APM_INGEST=redis`.

```php
// Run a block without Vigilance observing it (e.g. its own writes):
\Vigilance\Vigilance::withoutRecording(fn () => $this->doInternalWork());

// Surface a caught/swallowed exception to the APM exception card + active trace:
try { /* ... */ } catch (\Throwable $e) { \Vigilance\Vigilance::report($e); }
```

## Manual control: dispatch jobs & run commands (treat as RCE)

Off by default. Enable deliberately and govern with allowlists. Every manual action is written to an audit log.

```php
// config/vigilance.php
'control' => [
    'enabled'  => env('VIGILANCE_CONTROL_ENABLED', false),
    'jobs'     => ['mode' => 'marker'], // marker | list | discover | all
    'commands' => [
        'mode'  => 'list',              // list | all
        'allow' => ['mail:*', 'cache:clear'],
        'deny'  => ['migrate:fresh', 'db:wipe', 'tinker'], // deny always wins
    ],
],
```

Opt a job in (when `jobs.mode` is `marker`) and give it a friendly label; the constructor is reflected to build the form:

```php
use Illuminate\Contracts\Queue\ShouldQueue;
use Vigilance\Contracts\Dispatchable;

class ProcessPodcast implements ShouldQueue, Dispatchable
{
    public static string $vigilanceLabel = 'Process a podcast';
    public static string $vigilanceDescription = 'Transcode and publish a podcast episode';

    public function __construct(public Podcast $podcast, public bool $notify = true) {}
}
```

In `discover` mode, hide a job with side effects using `Vigilance\Contracts\ShouldNotBeDispatchedManually`. Vigilance's own `vigilance:*` commands are always blocked.

## Worker supervision (the Horizon replacement)

Run `php artisan vigilance:supervise` as your long-running worker process (under systemd / Supervisor / your platform manager) **in place of** `php artisan horizon` or `queue:work`. It works on every supervisable driver and auto-scales by backlog. Define supervisors per environment:

```php
// config/vigilance.php
'environments' => [
    'production' => [
        'payments' => [
            'connection'    => 'redis',
            'queue'         => ['payments'],
            'balance'       => 'auto',   // auto | simple | false
            'min_processes' => 2,
            'max_processes' => 10,
            'max_time'      => 3600,
        ],
    ],
],
```

Control it with `vigilance:status`, `vigilance:pause` / `vigilance:continue`, `vigilance:restart` (e.g. after a deploy), `vigilance:terminate`. It is Windows-safe (cache control-plane + tagged worker reaping). Do not point it at the same queues Horizon is already draining.

## APM & tracing

- Run `php artisan vigilance:check` as a per-server heartbeat (captures server stats and flushes telemetry; recorders capture cheaply and flush **after** the response — zero request latency).
- Per-recorder `sample_rate` and `threshold` (ms) bound overhead/storage; configure under `apm` in config.
- The `Requests` recorder rolls **all** requests up per route for the **Routes** page (throughput, error rate, Apdex, p50/p95/p99) — distinct from `SlowRequests` (threshold-only). Apdex target: `VIGILANCE_APM_APDEX_MS`.
- Optional Redis write-behind ingest: drain with `php artisan vigilance:apm-work`.
- Tracing is off by default. Enable with `VIGILANCE_TRACING=true`; it tail-samples (keeps only slow / errored / sampled traces). Spans cover query / cache / HTTP / redis / mail / notification.
- **What a page costs** (opt-in, `VIGILANCE_APM_REQUEST_PROFILE=true`): the `RequestProfile` recorder adds queries, database time, peak memory and hydrated models per route to the Routes page. Not built on tracing on purpose — a trace is only kept when sampled/slow/errored, so a route quietly running 180 queries in 400ms is invisible there. Alert on it with `alerts.rules.heavy_request`.
- **Incident mode** (`VIGILANCE_INCIDENT_MODE=true` makes it available): engage from the dashboard to keep every trace, stop sampling and lower the log floor for N minutes. The timer is a cache TTL, so it expires even across a redeploy — never raise `sample_rate` by hand "temporarily".
- Uptime checks: `php artisan vigilance:health` records availability + latency for configured URLs.

## Observability suite

Front-to-back observability, each layer on its own dashboard page. Full reference: `docs/observability.md`.

- **Issues — error tracking** (`/vigilance/issues`, on by default). Every exception — HTTP requests, `Vigilance::report()`, queued jobs, console commands and uncaught **browser** errors — is fingerprinted into a grouped inbox with stacktrace + context. Tune with `issues.sample_rate` / `issues.except`; `issues.capture_request_input` is off for privacy.
- **Per-route performance** (`/vigilance/routes`, on). See APM above.
- **RUM / Web Vitals** (`/vigilance/vitals`, **off** by default). Set `VIGILANCE_RUM=true` and add the `@vigilanceRum` Blade directive to your layout `<head>`. Collects LCP/INP/CLS/FCP/TTFB + JS errors (which land in Issues as source `browser`) via a public, throttled, validated beacon.
- **SLOs & error budgets** (`/vigilance/slos`, off until defined). Add objectives under `slos`; `sli` is `success_rate` or `latency` (Apdex). A `slo_burn` alert fires on a high short-window burn rate.
- **Custom business metrics** (`/vigilance/custom-metrics`, on). Record KPIs anywhere — they auto-appear:

```php
use Vigilance\Vigilance;

Vigilance::increment('signups');                 // counter (sum + count)
Vigilance::gauge('cart_value', $cart->total());  // gauge (avg / peak / min)
```

- **Log explorer** (`/vigilance/logs`, **off** by default). Set `VIGILANCE_LOGS=true` to capture `Log::*` records into a searchable explorer **correlated to the trace that emitted them** (the trace detail page lists its logs, and each log links back). Set the minimum level with `VIGILANCE_LOGS_LEVEL` (default `debug`; raise to `warning`/`error` at volume). Buffered + flushed after the response; trimmed by `vigilance:prune`.
- **Release health** (`/vigilance/releases`, on). After each `php artisan vigilance:deploy --release=...`, Vigilance compares error-rate/latency/throughput in the window after the deploy vs before it and assigns a healthy/degraded/regressed verdict. A regression fires the `deploy_regression` alert (route a webhook at it to roll back). Set the release via `VIGILANCE_RELEASE` (or `app.version`) so issues are tagged with `first_release` / `regressed_release`. Tune under `release_health`. Symbolicate RUM browser errors by uploading source maps: `php artisan vigilance:sourcemaps <build-dir> --release=...`.

> **Exclude noisy endpoints once:** `vigilance.ignore_paths` (e.g. `['/admin/*', 'livewire/*']`) drops a path from APM, tracing, RUM and request-error capture together — use it instead of editing each recorder's `ignore`.

## MCP server (query telemetry from an AI agent)

Beyond the per-page tools: **`footprint`** (what Vigilance itself stores, and whether pruning keeps up), **`suppressions`** (list — always check before concluding a route has no data — plus create/remove with writes on), and **`incident-mode`** (status read-only; `engage`/`end` need writes **and** `vigilance.incident_mode.enabled`). Engaging is safe to hand to an agent precisely because it cannot be left on: the duration is capped and it expires itself.

Optional, **off by default**, built on `laravel/mcp` (`composer require laravel/mcp`). Set `VIGILANCE_MCP_ENABLED=true` and Vigilance registers a local stdio server your AI client launches with `php artisan mcp:start vigilance`. It exposes a tool for **every dashboard page**, so an agent can investigate (and, when allowed, fix) against live data:

- **Errors / runs:** `overview`, `issues`/`issue`, `exceptions`, `runs`/`run`, `job-metrics`.
- **APM:** `performance`, `slow-requests`, `slow-queries`, `slow-jobs`, `slow-http`, `cache`, `servers`, `usage`.
- **Front-end / tracing / logs:** `vitals` (RUM Web Vitals), `traces`/`trace`, `logs`.
- **Reliability:** `slos`, `incidents`, `releases`.
- **Queues & workers:** `workers`, `queues`, `pending`, `schedule`, `batches`, `tags`.
- **Business metrics:** `custom-metrics`.

- **Read-only by default.** Output is redacted (same `redact` keys as storage) and bounded by `mcp.max_results` / `mcp.max_field_length`, so a tool can't leak a secret or dump the DB into the agent's context.
- **Writes are opt-in:** `VIGILANCE_MCP_ALLOW_WRITES=true` adds `resolve-issue`, `acknowledge-issue`, `mute-issue`, `reopen-issue`, `retry-run`, `retry-issue`. While off they aren't even advertised to the client. Every write is recorded in the same audit log as a dashboard action.
- **Manual control is double-gated:** `dispatchable-jobs` / `runnable-commands` (discovery) need `VIGILANCE_CONTROL_ENABLED=true`; `dispatch-job` / `run-command` need that **and** `VIGILANCE_MCP_ALLOW_WRITES=true`, and obey the same `control` allowlist as the dashboard.
- **Remote access** is off by default; `VIGILANCE_MCP_WEB_ENABLED=true` serves over HTTP wrapped in the `viewVigilance` gate (add Sanctum/Passport for real remote auth).

```bash
composer require laravel/mcp
# .env: VIGILANCE_MCP_ENABLED=true  (+ VIGILANCE_MCP_ALLOW_WRITES=true for triage tools)
php artisan mcp:start vigilance       # the stdio command your MCP client launches
php artisan mcp:inspector vigilance   # sanity-check the server
```

Full guide: `docs/mcp.md`. Complements Laravel Boost: Boost teaches conventions, MCP serves live data.

## Alerting (mail / Slack / Discord / Teams / webhooks / custom) + incidents

Rules are evaluated at `vigilance:snapshot` time and throttled per key: `queue_long_wait`, `error_rate`, `exception_spike`, `slow_request_rate`, `scheduled_task_late` (a **dead-man's-switch**), `slo_burn`, `new_issue`, `issue_regression`, `anomaly` (dynamic baselines) and `deploy_regression` (bad deploy). Configure them under `alerts.rules`; route delivery from `.env` or code. Fired alerts are persisted as **incidents** (`alerts.incidents`, on by default) — opened on first fire, auto-resolved when they stop recurring — with occurrence counts and MTTR on the **Incidents** page (`/vigilance/incidents`). You're notified **once per incident** (open / escalate / recur), so a sustained condition doesn't spam every window; set `alerts.renotify_minutes` for periodic reminders.

```ini
# .env — simplest path, no provider needed
[email protected],[email protected]   # single or comma-separated
VIGILANCE_SLACK_WEBHOOK=https://hooks.slack.com/services/...
VIGILANCE_DISCORD_WEBHOOK=https://discord.com/api/webhooks/...
VIGILANCE_TEAMS_WEBHOOK=https://outlook.office.com/webhook/...
VIGILANCE_ALERT_WEBHOOKS=https://events.pagerduty.com/...   # one or comma-separated
```

```php
// ...or in a service provider's boot() (overrides the .env values)
use Vigilance\Vigilance;

Vigilance::routeMailNotificationsTo(['[email protected]', '[email protected]']);
Vigilance::routeSlackNotificationsTo('https://hooks.slack.com/services/...');
Vigilance::routeDiscordNotificationsTo('https://discord.com/api/webhooks/...');
Vigilance::routeTeamsNotificationsTo('https://outlook.office.com/webhook/...');
Vigilance::routeWebhooksTo(['https://events.pagerduty.com/...']);

// Custom channel (SMS, a Notification, …):
Vigilance::alertUsing(fn (\Vigilance\Notifications\Alert $alert) =>
    Notification::route('slack', $url)->notify(new QueueAlert($alert))
);
```

Add your own rule by implementing `AlertRule` and registering it under `alerts.custom`:

```php
namespace App\Vigilance\Alerts;

use Vigilance\Notifications\Alert;
use Vigilance\Notifications\Contracts\AlertRule;

class TooManyRefunds implements AlertRule
{
    public function evaluate(): iterable
    {
        $count = Refund::whereBetween('created_at', [now()->subHour(), now()])->count();

        if ($count > 50) {
            // Alert(key, title, message, level = info|warning|critical)
            yield new Alert('refund-spike', 'Refund spike', "{$count} refunds in the last hour.", 'critical');
        }
    }
}
```

```php
// config/vigilance.php
'alerts' => [
    'custom' => [
        \App\Vigilance\Alerts\TooManyRefunds::class,
    ],
],
```

## Forwarding telemetry (the Ingest seam)

There is a ready-made `http` exporter: set `apm.ingest.exporters => ['http']` and `VIGILANCE_APM_HTTP_ENDPOINT`. It POSTs the batched `Entry`/`Value` feed (aggregations included) and is strictly additive — a failing sink never breaks local capture. There is deliberately **no Nightwatch driver**: Nightwatch ingests through its own agent and publishes no third-party format, so one could only be a reverse-engineered protocol.

To fan APM telemetry out to an external system, bind a custom implementation of `Vigilance\Apm\Contracts\Ingest`:

```php
interface Ingest
{
    public function ingest(\Illuminate\Support\Collection $items): void; // Collection<Entry|Value>
    public function digest(\Vigilance\Apm\Storage $storage): int;        // drain buffered items
    public function trim(): void;
}
```

## Commands reference

| Command | Purpose |
| --- | --- |
| `vigilance:install` | Publish config, optionally migrate, print next steps |
| `vigilance:doctor` | Diagnose the install; **exits non-zero when `vigilance:snapshot` has gone silent** — point an uptime check at it |
| `vigilance:prune` | Delete old runs (`--days`, `--failed-days`, `--dry-run`) + trim snapshots |
| `vigilance:snapshot` | Capture a metric snapshot **and evaluate alerts** |
| `vigilance:schedule-sync` | Sync defined scheduled tasks into monitors |
| `vigilance:deploy` | Record a deployment marker on the timeline |
| `vigilance:supervise` | Run & auto-scale workers (replaces `queue:work`); `--once` / `--max-time=N` |
| `vigilance:status` / `pause` / `continue` / `restart` / `terminate` | Supervisor control |
| `vigilance:check` | APM heartbeat (daemon; `--once` for cron/testing) |
| `vigilance:apm-work` | Drain Redis write-behind telemetry |
| `vigilance:health` | Ping uptime URLs; record availability + latency |
| `vigilance:deploy` | Record a deployment marker (`--release`, `--commit`) — feeds release health |
| `vigilance:sourcemaps` | Upload JS source maps (`--release`, `--prune`) for RUM symbolication |

## Gotchas

- It's **secure by default** — if the dashboard 404s/403s, you haven't defined `viewVigilance` / `Vigilance::auth()`, or `web` middleware is rewriting the path.
- Failures are always captured regardless of `sample_rate`; only successes are sampled out.
- The supervisor only drains *supervisable* drivers (not `sync` / `null` / push-only "after response" drivers).
- Don't run the supervisor and Horizon on the same queues.
- **RUM and the log explorer are off by default** — they need `VIGILANCE_RUM=true` (+ the `@vigilanceRum` directive in your layout) and `VIGILANCE_LOGS=true` respectively. SLOs show nothing until you define one under `slos`.
- Log–trace correlation only happens when **both** logs and tracing are enabled (a log's `trace_id` is the in-flight trace).
- If you published `config/vigilance.php`, re-publish (or merge new keys — `issues`, `rum`, `slos`, `logs`, `alerts.incidents`, `mcp`, `notifications.discord` / `teams` / `webhooks`) after upgrading so `.env` settings resolve.
- **A dead-man's switch cannot watch its own process.** `MonitoringHealthRule` is evaluated *by* `vigilance:snapshot`, so it can never report the snapshotter dying — that is what `vigilance:doctor`'s exit code is for.
- **`aggregateTotal()` only answers for windows with a matching bucket period** (15m/1h/6h/24h/7d). Any other interval reads back as **zero**, not an error. Add a period in `DatabaseStorage::periods()` before adding a range.
- **Alpine's `:attr` and Blade's component prop binding share a syntax.** On a `<x-vigilance::ui.*>` component, `:aria-expanded="open"` is evaluated as PHP. Alpine-driven markup stays a plain element.
- The **MCP server is off by default** and needs `laravel/mcp` installed; its write/triage tools require `VIGILANCE_MCP_ALLOW_WRITES=true` and aren't advertised to the client otherwise.