Save this content to: .claude/skills/watchtower-error-tracking-reference/SKILL.md

# Watchtower install — reference

Companion to [`SKILL.md`](SKILL.md). This is the canonical install guide. Covers Laravel + browser JavaScript end-to-end.

## What Watchtower is

Self-hosted, Sentry-compatible exception tracker. Client apps report errors through the standard Sentry SDKs (`sentry/sentry-laravel` on the backend, `@sentry/browser` in the browser) pointed at your Watchtower instance. **Exception-only** — Watchtower drops trace / transaction (performance) items at ingest, so do not enable performance monitoring on the client side.

## Pick the right Watchtower project for this codebase

Default: **one Watchtower project per client app**. The same DSN authenticates the Laravel backend (`SENTRY_LARAVEL_DSN`) and the browser frontend (`VITE_SENTRY_DSN`). `watchtower:install` writes both env keys to that DSN. Backend exceptions and JS errors share one inbox; project name in the breadcrumb tells them apart at triage time. On a central server (relay mode; `<watchtower-host>` is the host in `WATCHTOWER_DSN`), create the project once at `https://<watchtower-host>/projects/create`, pick the primary runtime (usually `php-laravel`) as `platform`, copy the DSN.

### When to split into two projects

The trade-off of the shared-DSN default is that the frontend DSN ships in the JS bundle, so a leaked key can be used to forge events against the same project. Watchtower's per-project rate limit caps the blast radius. Split into a second project when one of these applies:

| Situation | Why split |
|---|---|
| Regulated environment | Auditors want server-only and browser-only signals isolated. |
| Public-facing / high-traffic frontend | A forged-events flood would drown out backend triage. |
| Very noisy frontend (third-party scripts, extensions) | Different cooldowns, different recipients, different rate limits than backend. |
| Different alert recipients per runtime | Frontend team owns `javascript`; backend team owns `php-laravel`. |

To split: after `watchtower:install`, edit `.env` to point `VITE_SENTRY_DSN` at a second Watchtower project's DSN. Register a second MCP server for the frontend project with `claude mcp add --transport http --scope project watchtower-frontend <host>/mcp/<frontend-project-id> --header 'Authorization: Bearer ${WATCHTOWER_TOKEN}'` so Claude can triage both inboxes.

## DSN format — must be numeric

```
https://{public_key}@{host}/{numeric_project_id}                    # central Watchtower
https://{public_key}@{host}/{path_prefix}/{numeric_project_id}      # embedded (standalone/dual)
```

The project segment **MUST** be the project's numeric id. Stock Sentry SDKs validate at `Dsn::parse()` time and silently no-op the transport if the segment isn't `\d+`. The slug works in cURL pokes but breaks every real SDK. The settings page renders the numeric form by default — copy it verbatim.

An **embedded** host serves Watchtower under a URL prefix (`watchtower.server.path`, default `watchtower`), and that prefix goes into the DSN path in front of the project id — with **no `api` segment of its own**. Every SDK appends `/api/{project_id}/envelope/` to the DSN path itself, so `https://[email protected]/watchtower/2` is what reaches `https://shop.test/watchtower/api/2/envelope/`, which is the registered route. Writing the `api` segment yourself, or dropping the prefix, gives a silent 404 per event. `php artisan watchtower:project list` prints the correct DSN for every project — copy it rather than assembling one.

## Laravel install (one command)

For any Laravel 12 / 13 project (with or without a Vite frontend), use the install package. This section describes the default **relay** install; add `--standalone` or `--mode=dual` for the embedded server ([Standalone / dual mode](#standalone--dual-mode-embedded-server)).

```bash
composer require phattarachai/watchtower-laravel
php artisan watchtower:install
```

The package is published at <https://packagist.org/packages/phattarachai/watchtower-laravel> (GitHub source: <https://github.com/phattarachai/watchtower-laravel>).

`watchtower:install` does eight things:

1. **Resolves the DSN.** From `--dsn=…`, then `WATCHTOWER_DSN`, then `SENTRY_LARAVEL_DSN`, otherwise prompts. Validates the DSN format (numeric project id required).
2. **Writes env keys.** Sets `WATCHTOWER_DSN` and `SENTRY_LARAVEL_DSN` in `.env`. Empty in `.env.example`. For `APP_ENV=local`, prompts whether to use `null` (recommended — don't flood Watchtower from dev).
3. **Patches `bootstrap/app.php`.** Adds `use Sentry\Laravel\Integration;` and inserts `Integration::handles($exceptions);` inside `withExceptions(...)`. **Critical** — without this, unhandled exceptions in HTTP requests, jobs, and commands never reach Watchtower. If the package can't recognize the file's shape, it prints the diff and bails — apply it by hand.
4. **Publishes `config/watchtower.php`.** Knobs: relay path, timeout, async forwarding flag, queue name, plus the `user_context` + `before_send` sections that power the [Smart defaults](#smart-defaults).
5. **Detects Vite.** If `vite.config.{js,ts}` exists, writes `VITE_SENTRY_DSN` (same value as `SENTRY_LARAVEL_DSN`), `VITE_SENTRY_TUNNEL=/api/watchtower-relay`, and `VITE_SENTRY_ENVIRONMENT="${APP_ENV}"` to `.env` and `.env.example`. Prints the `Sentry.init(...)` snippet to paste into the entry JS file (see [Browser-side init](#browser-side-init) below). To point the browser side at a *different* Watchtower project, edit `VITE_SENTRY_DSN` in `.env` after install — see [When to split into two projects](#when-to-split-into-two-projects).
6. **Confirms `SENTRY_SEND_DEFAULT_PII`.** Prompts `Enable SENTRY_SEND_DEFAULT_PII (attach request data + IP — Watchtower scrubs secrets via BeforeSend)?` — **opt-in**, defaults to `no` across every env (regulated industries and casual installs both get the conservative default). Writes `SENTRY_SEND_DEFAULT_PII=true|false`. Without PII the SDK strips the request body + IP before any local filter sees them — the User tab and Request tab in Watchtower stay empty. Flip to `yes` only after confirming `BeforeSend`'s scrub coverage (step 4 config) is sufficient for your app's data; that's what makes it safe to leave on in production.
7. **Writes Sentry breadcrumb env keys** (only when absent — re-runs preserve customization): `SENTRY_BREADCRUMBS_SQL_QUERIES_ENABLED=true`, `SENTRY_BREADCRUMBS_SQL_BINDINGS_ENABLED=false` (bindings can leak PII even after scrubbing — opt in if you need them), `SENTRY_BREADCRUMBS_CACHE_ENABLED=true`, `SENTRY_BREADCRUMBS_HTTP_CLIENT_REQUESTS_ENABLED=true`, `SENTRY_BREADCRUMBS_REDIS_COMMANDS_ENABLED=true`. Result: events arrive with the last ~100 query/cache/HTTP/Redis ops in Watchtower's Breadcrumbs tab.
8. **Publishes the browser helper** (Vite path only). Calls `vendor:publish --tag=watchtower-js --force=false` → drops `resources/js/vendor/watchtower.js` plus the `beforeSend` rule files it imports: `resources/js/vendor/livewire.js` (Livewire) and `resources/js/vendor/noise.js` (crawlers and chunk loads). The printed snippet is a 3-line import + `initWatchtower()` call — the actual `Sentry.init(...)` config lives inside the published helper, so per-entry duplication is gone. A separate `<meta name="watchtower-user-*">` snippet is printed for the root Blade layout. See [Browser user context](#browser-user-context) below.

Then it registers the project-scoped MCP server with Claude Code if the `claude` CLI is on PATH (see [Querying via MCP](#querying-via-mcp)).

Flags:
- `--dsn=…` — Watchtower DSN. Skips the prompt.
- `--standalone` / `--mode=relay|standalone|dual` — pick the mode. Default `relay`.
- `--dry-run` — print intended changes (env diff + `bootstrap/app.php` unified diff) without writing.
- `--no-mcp` — skip registering the Watchtower HTTP MCP server with Claude Code.
- `--patch-js` / `--patch-views` — inject the frontend snippets instead of printing them (idempotent, sentinel-guarded).

After install: `php artisan watchtower:test` sends a synthetic exception through both the Laravel SDK path and the local relay path.

### How the relay works

The package registers a route at `POST /api/watchtower-relay`. Browser SDKs post envelopes to this same-origin path with `Sentry.init({ tunnel: '/api/watchtower-relay' })`. The relay reads the body and forwards it to `{WATCHTOWER_DSN host}/api/watchtower-relay`. Two reasons:

1. **Ad blockers.** uBlock, AdGuard, and Brave Shields drop URLs containing `sentry_key=` (the default Sentry SDK URL format). A relay path under your own domain has neither `sentry_key=` nor a recognizable Sentry hostname, so filter lists don't match.
2. **CORS.** Same-origin means no CORS preflight, no `OPTIONS` requests.

#### Sync vs async forwarding

Default: sync (Guzzle POST to upstream within the request lifecycle). The browser SDK uses `keepalive: true` so it doesn't block the page, but the relay still spends a Laravel worker thread forwarding bytes. Latency typically 50–200ms.

Opt in to async by setting `WATCHTOWER_RELAY_ASYNC=true` — the relay dispatches a `ForwardEnvelope` job and returns 202 immediately. Requires a real queue (Redis / Horizon / database driver). Set `WATCHTOWER_RELAY_QUEUE` if you want a non-default queue name.

Both paths go through one forwarder that:

- **reuses its HTTP client**, so a queue worker (or Octane) keeps the upstream TLS connection alive across envelopes instead of handshaking per envelope. Under PHP-FPM each request still opens its own connection — async forwarding is the way to amortize it;
- **gzips** envelopes of 1 KB and up that the browser sent uncompressed (`WATCHTOWER_FORWARD_GZIP=false` to turn off). Every Watchtower relay endpoint inflates `Content-Encoding: gzip`.

`ForwardEnvelope` retries an unreachable upstream or a 5xx twice (10 s, then 60 s), drops a 4xx — including a 429, which is the upstream asking for less traffic — and on its last attempt logs a warning and completes, so an upstream outage never parks envelope bodies in `failed_jobs`.

### Configuration reference

`config/watchtower.php` exposes all knobs through env:

| Env key | Default | Purpose |
|---|---|---|
| `WATCHTOWER_DSN` | (required) | Upstream DSN. Host is derived from this. |
| `SENTRY_LARAVEL_DSN` | (alias) | The package sets both to the same value for SDK compatibility. |
| `WATCHTOWER_RELAY_ENABLED` | `true` | Set to `false` to skip route registration entirely. |
| `WATCHTOWER_RELAY_PATH` | `/api/watchtower-relay` | Path the browser SDK posts to. Change only if it collides with an existing route. |
| `WATCHTOWER_RELAY_ASYNC` | `false` | Forward via queue instead of inline. |
| `WATCHTOWER_RELAY_QUEUE` | `null` | Queue name when async. |
| `WATCHTOWER_RELAY_TIMEOUT` | `5` | Guzzle request timeout (seconds). |
| `WATCHTOWER_CONNECT_TIMEOUT` | `3` | Guzzle connect timeout (seconds, float). |
| `WATCHTOWER_VERIFY_SSL` | `true` | Disable for self-signed dev Watchtower instances only. |
| `WATCHTOWER_FORWARD_GZIP` | `true` | gzip uncompressed envelopes (≥ 1 KB) before forwarding upstream. |
| `WATCHTOWER_USER_CONTEXT` | `true` | Set `false` to disable the auto-pushed [user context middleware](#user-context-middleware) entirely. |
| `WATCHTOWER_USER_CONTEXT_GUARDS` | `auto` | `auto` walks `array_keys(config('auth.guards'))`; or a comma-separated list (`web,admin,api`) to fix priority and narrow the set. First authenticated guard wins. |
| `WATCHTOWER_BEFORE_SEND` | `true` | Set `false` to skip the [BeforeSend pipeline](#beforesend-pipeline) (no noise drop, no scrubbing). |
| `SENTRY_SEND_DEFAULT_PII` | (prompted at install) | Sentry SDK key. With it on, request body + IP attach to events (Watchtower's BeforeSend is the safety net that strips secrets). Without it, User tab + Request tab stay empty. |
| `SENTRY_BREADCRUMBS_SQL_QUERIES_ENABLED` | (`true` after install) | Sentry SDK key. Attach SQL query crumbs. |
| `SENTRY_BREADCRUMBS_SQL_BINDINGS_ENABLED` | (`false` after install) | Sentry SDK key. Bindings can leak PII even after scrubbing — opt in deliberately. |
| `SENTRY_BREADCRUMBS_CACHE_ENABLED` | (`true` after install) | Sentry SDK key. Cache get/set crumbs. |
| `SENTRY_BREADCRUMBS_HTTP_CLIENT_REQUESTS_ENABLED` | (`true` after install) | Sentry SDK key. Outbound HTTP crumbs via Laravel's HTTP client. |
| `SENTRY_BREADCRUMBS_REDIS_COMMANDS_ENABLED` | (`true` after install) | Sentry SDK key. Redis command crumbs. |

Two more knobs live in `config/watchtower.php` only (arrays — too much for an env var): `watchtower.user_context.fields` (see [User context middleware](#user-context-middleware) below) and `watchtower.before_send.{ignored_exceptions, scrub_keys, scrub_url_patterns}` (see [BeforeSend pipeline](#beforesend-pipeline) below).

## Standalone / dual mode (embedded server)

`WATCHTOWER_MODE` (`relay` | `standalone` | `dual`) decides whether the app reports to a Watchtower or *is* one. `relay` is the default and registers nothing extra. See `SKILL.md` § "Pick a mode first" for the choice and the install commands; this section is the configuration and operations detail.

### What standalone/dual adds

Six `watchtower_*` tables (json columns — SQLite, MySQL and Postgres all work), an ingest endpoint at `POST /{prefix}/api/{project}/envelope`, an Inertia + React console at `/{prefix}` (mounted only when `inertiajs/inertia-laravel` is installed and `WATCHTOWER_UI_ENABLED` is true), per-project alert-rule email, and an MCP server at `/{prefix}/mcp`. `ProcessEventJob` does the same scrub → normalize → fingerprint → group-upsert pipeline as the central server.

This app's own exceptions reach the store through `watchtower.server.self_capture`:

| Value | How |
|---|---|
| `transport` (default) | The SDK's HTTP transport is swapped for an in-process one — no socket, no round trip. `BeforeSend` still runs first. |
| `loopback` | The SDK posts over HTTP back into the app's own ingest route. Use when you specifically want the wire path exercised. |
| `false` | No self-capture. |

Either way, **Watchtower's own queue failures are never self-captured.** Anything a worker reports while running, failing, or recording the failure of a `ProcessEventJob`, a `ForwardEnvelope` or a queued alert mail (`IssueAlertMail`, which rides the same queue) — the job's exception, `MaxAttemptsExceededException` / `TimeoutExceededException` naming it, even the `failed_jobs` insert blowing up — is dropped before it reaches the SDK transport. This guard runs even with `WATCHTOWER_BEFORE_SEND=false`. Those failures still land in `failed_jobs` and the worker log.

### Env keys

| Env key | Default | Purpose |
|---|---|---|
| `WATCHTOWER_MODE` | `relay` | `relay` / `standalone` / `dual`. |
| `WATCHTOWER_PATH` | `watchtower` | URL prefix for ingest, console and MCP. Also the DSN path segment. |
| `WATCHTOWER_DB_CONNECTION` | (host default) | Put the tables on a separate connection if you want them out of the main database. |
| `WATCHTOWER_RETENTION_DAYS` | `90` | Event retention; a project row's `retention_days` overrides it. Groups are kept forever. |
| `WATCHTOWER_SELF_CAPTURE` | `transport` | See table above. |
| `WATCHTOWER_UI_ENABLED` | `true` | `false` for a headless install — ingest + alerts + MCP, no console, no Inertia/Tailwind requirement. |
| `WATCHTOWER_UI_DOMAIN` | `null` | Serve the console on its own domain. |
| `WATCHTOWER_UI_LOGIN_ROUTE` | `login` | Route name (or URL) a rejected **guest** is redirected to. `null` 403s instead. |
| `WATCHTOWER_UI_THEME` | `auto` | `auto` turns the console dark under a `.dark` ancestor (the starter kits put it on `<html>`); `light` or `dark` pins it. |
| `WATCHTOWER_MCP_ENABLED` | `true` | Registration is skipped anyway when `laravel/mcp` isn't installed. |
| `WATCHTOWER_RATE_LIMIT_PER_MIN` | `300` | Per-project event budget per minute, over the host's cache store — no Redis needed. Applies to **every** ingest path: the HTTP endpoint (429 once spent), the relay, and in-process self-capture. `0` disables. |
| `WATCHTOWER_RATE_LIMIT_PER_FINGERPRINT_PER_MIN` | `20` | Per-issue budget, so one hot error can't spend the project's budget or flood the queue. `0` disables. |
| `WATCHTOWER_MAX_PAYLOAD_BYTES` | `1048576` | Reject bigger envelope bodies (as sent). A gzip body may inflate to at most 20× this. |
| `WATCHTOWER_MAX_EVENT_BYTES` | `200000` | Every event is trimmed to this JSON size *before* it is queued. `0` disables. |
| `WATCHTOWER_MAX_STRING_BYTES` | `8192` | Any single string (an exception message quoting SQL, a breadcrumb) is capped at this. |
| `WATCHTOWER_DROP_BOT_EVENTS` | `true` | Drop browser events whose `User-Agent` is a crawler (Googlebot, GoogleOther, bingbot, AdsBot, …) at ingest, for clients that predate the SDK-side filter. Server-side events and HeadlessChrome browser tests still report. |
| `WATCHTOWER_MAX_QUEUE_DEPTH` | `5000` | Once this many jobs wait on the Watchtower queue, new events are counted on their issue instead of queued — a dead Horizon can't fill Redis. Checked at most every 5 s per process. `0` disables; never applies to `sync`. |
| `WATCHTOWER_REDACT_SQL_VALUES` | `true` | Strip row values from SQL error messages (Laravel's QueryException inlines every binding) and drop `bindings` from query breadcrumbs. The query template still arrives in the SQL breadcrumbs. |
| `WATCHTOWER_QUEUE_CONNECTION` / `WATCHTOWER_QUEUE_NAME` | (host default) | Where `ProcessEventJob` runs. `sync` is fine for a small install — events are normalized inline. On Redis/Horizon use a dedicated `watchtower` queue — see [Production queue and Redis](#production-queue-and-redis). |

An event over either budget — or arriving while the queue is past `WATCHTOWER_MAX_QUEUE_DEPTH` — is not queued, but if its issue group exists it is still **counted** — `event_count` and `last_seen_at` move without a payload being stored — so the issue list shows the real volume.

**Trimming** follows Sentry's approach: strings are capped and only the last 100 breadcrumbs kept on every event; then, only while the event is still too big: the oldest breadcrumbs and their `data`, then `extra` + the request body/cookies, then frame locals and the middle of stacks deeper than 50 frames, and finally everything but the exception. A trimmed event carries the tag `watchtower.truncated: true`. Scrubbing and the field allow-list also run before dispatch now, so secrets and `modules` never reach the queue backend. The scrub → normalize → trim pipeline and the gzip inflate cap live in `phattarachai/watchtower-core`, shared with the central server.

`ProcessEventJob` is idempotent: a retry, a job restored from a Redis snapshot, or two workers racing on a new fingerprint converge on one event row and one group. It has `tries = 3`, `backoff = [5, 30]`, `timeout = 30`, `failOnTimeout = true`.

Array-only knobs under `watchtower.server.ingest`: `allowed_event_fields`, `allowed_context_keys`, and `scrub.{header_keys, body_keys, placeholder}` — the allow-list applied before the row is written.

### Access control

The console is open in the `local` environment and closed everywhere else until the host registers a check — `Watchtower::auth(fn ($request) => …)`, or a `viewWatchtower` gate. The `AuthorizeUi` middleware is appended by the service provider and cannot be removed by editing `watchtower.server.ui.middleware`.

### Alerts

Rules live on `watchtower_alert_rules` and use the same types and suppression semantics as the central server. The one difference: **recipients are the literal addresses in the rule's `targets.emails`** — there is no user directory, so `targets.user_ids` and per-user mute preferences have no meaning. Nothing is seeded, so a fresh project alerts on nothing until you add a rule (console → Alerts, or `POST /{prefix}/alerts`). Publish `--tag=watchtower-views` to override the alert mailable.

### Operations

| Command | Use |
|---|---|
| `watchtower:doctor` | The verification step for embedded installs. Checks tables, routes, the published Inertia page, the Vite alias, the Tailwind lines, an active project, and that self-capture is bound. Advisories for mailer, queue driver and missing `laravel/mcp`. |
| `watchtower:project {list\|create\|rotate-key\|activate\|deactivate}` | Project management without a browser. `list` prints keys and DSNs. |
| `watchtower:prune` | Scheduled daily by the package. Deletes events past retention and notification rows older than 90 days; issue groups survive. |

### Production queue and Redis

**Dedicated queue.** `ProcessEventJob` defaults to the host's default queue, because a queue name no worker listens on would leave events unprocessed. On Redis + Horizon, move it to its own queue so a burst of errors can't starve the app's jobs (or the reverse), and so a runaway can be flushed without touching them. Supervisor first, env key second:

```php
// config/horizon.php
'defaults' => [
    // …the app's existing supervisor(s)…
    'app-supervisor-watchtower' => [
        'connection' => 'redis',
        'queue' => ['watchtower'],
        'balance' => false,
        'maxProcesses' => 1,
        'tries' => 3,
        'timeout' => 60,   // > ProcessEventJob's 30 s timeout, < the redis connection's retry_after (90)
        'memory' => 256,
    ],
],

'environments' => [
    'production' => [
        // …
        'app-supervisor-watchtower' => ['maxProcesses' => 2],
    ],
    'local' => [
        // …
        'app-supervisor-watchtower' => [],
    ],
],
```

```dotenv
WATCHTOWER_QUEUE_NAME=watchtower
```

Deploy, `php artisan horizon:terminate` so the new supervisor starts, then `php artisan watchtower:doctor` — it **fails** when no supervisor in the current environment consumes `[connection:queue]`, and warns when the queue's Redis has no `maxmemory` or an evicting policy. Alert mail is queued on the same connection and queue. Without Horizon, run a worker with `--queue=watchtower` (or append it: `--queue=default,watchtower`).

**Redis memory guard.** An unbounded Redis on a box that also serves the app turns any queue runaway into an outage: it grows until the kernel OOM-kills it, and every request that touches cache, session or queue 500s. Cap it:

```conf
# redis.conf
maxmemory 1gb
maxmemory-policy noeviction   # queues must not be evicted — fail writes instead
```

With `noeviction` a full Redis rejects writes; self-capture swallows the failed dispatch and the request carries on.

**Horizon's own copies.** Horizon stores a hash with the full payload of every job and keeps it for `trim.recent` / `trim.completed` minutes (60 by default) — that, not the queue, was 3.7 GB in the incident. Worst case is roughly events-per-minute × event size × 60: at the default 300/min and a 200 KB cap that is 3.6 GB, typical events are 10–30 KB. If memory is tight, lower `trim.recent` and `trim.completed` (they apply to all jobs), or lower `WATCHTOWER_RATE_LIMIT_PER_MIN` / `WATCHTOWER_MAX_EVENT_BYTES`. Marking the job `Silenced` does **not** help — Horizon keeps silenced job hashes for the same `trim.completed` window. If cache and queues share one Redis, prefer a second instance (or DB on a separate server) for cache with `allkeys-lru` rather than an evicting policy on the queue instance. Also set `vm.overcommit_memory = 1` so RDB snapshots can fork.

### Self-capture loop — triage

**Symptom:** Redis memory climbing, `redis-cli llen` on the queue in the tens of thousands, Horizon full of `ProcessEventJob`, `watchtower_events` / `failed_jobs` / `laravel.log` growing by GB, possibly Redis OOM kills. Before 1.2.0 the cause was a failing `ProcessEventJob` being captured into a new one; the payload nests every generation (each SQL error quotes the previous job).

```bash
# 1. Confirm what holds the memory (keys carry the REDIS_PREFIX, e.g. myapp_database_)
redis-cli info memory | grep used_memory_human
redis-cli --memkeys                                  # biggest keys by memory (redis-cli ≥ 6)
redis-cli llen myapp_database_queues:default         # or :watchtower

# 2. Stop the consumers so the loop can't refill the queue
php artisan horizon:pause                            # or: supervisorctl stop horizon

# 3. Break the loop at the source until the package is upgraded
#    .env: WATCHTOWER_SELF_CAPTURE=false, then:
php artisan config:clear

# 4. Drop the runaway queue. UNLINK frees memory asynchronously.
#    On the shared `default` queue this also drops the app's own pending jobs — that is
#    the main argument for the dedicated `watchtower` queue.
redis-cli UNLINK myapp_database_queues:default myapp_database_queues:default:reserved myapp_database_queues:default:delayed

# 5. Horizon keeps a hash per job (full payload) for its recent/failed lists, expired after
#    `horizon.trim` minutes. Free the failed ones now; the rest age out (or lower `trim` briefly).
php artisan horizon:forget --all

# 6. Clean up the database side
php artisan queue:flush                              # failed_jobs
php artisan watchtower:prune                         # events past retention (or delete the runaway groups in the console)

# 7. Upgrade, move to the dedicated queue, re-enable, resume
composer update phattarachai/watchtower-laravel
php artisan horizon:continue                         # or horizon:terminate to pick up new config
```

The runaway usually shows as one or two issue groups with tens of thousands of events — deleting those groups in the console removes their events. Finally, add the `maxmemory` guard above so the next runaway of any kind degrades instead of taking the box down.

### Host build requirements

The console is an Inertia page, so the host needs `inertiajs/inertia-laravel` (the package only suggests it), `@inertiajs/react` and React. Without it the UI routes are not registered, `watchtower:install` skips the files below, and `watchtower:doctor` fails until you install it or set `WATCHTOWER_UI_ENABLED=false`. With it, `watchtower:install` handles all three, but they're what a manual wire-up must reproduce:

1. `resources/js/pages/Watchtower.jsx` published into the host (the host's `import.meta.glob` never leaves `./pages`).
2. A `@watchtower` alias in `vite.config.{js,ts}` pointing at `./vendor/phattarachai/watchtower-laravel/resources/js/watchtower`.
3. `resources/css/watchtower.css`, imported by the page stub: `@import 'tailwindcss' prefix(tw) source(none);`, `@source '../../vendor/phattarachai/watchtower-laravel/resources/js/watchtower/**/*.jsx';` and `@custom-variant dark (&:where(.dark, .dark *));`. Without the `@source`, Tailwind emits none of the module's classes and the console renders unstyled.

The module's colours come from `--wt-*` variables on `.wt-root` — override them to re-skin. Writes send the `csrf-token` meta tag's value as `X-CSRF-TOKEN` when the layout has one, and otherwise the `XSRF-TOKEN` cookie as `X-XSRF-TOKEN`, so the React starter kit's layout (no meta tag) needs nothing added. `watchtower:install` also finds that kit's `resources/views/app.blade.php` (any top-level view rendering `@inertia`) when placing `@watchtowerUser`.

## Smart defaults

Out of the box, `watchtower:install` produces events that already populate Watchtower's User tab, hide framework noise from the inbox, and never leak secrets in request bodies / headers. Each behavior is reversible via env or config.

### User context middleware

`Phattarachai\WatchtowerLaravel\Http\Middleware\WatchtowerUserContext` is auto-pushed onto both the `web` and `api` route groups when the service provider boots (unless `WATCHTOWER_USER_CONTEXT=false`). On every request it:

1. Walks the configured guards in order — `config('auth.guards')` keys when `WATCHTOWER_USER_CONTEXT_GUARDS=auto`, or the explicit comma-separated list when set (`web,admin,api`).
2. First authenticated guard wins. The middleware grabs the `Authenticatable` and remembers the guard name.
3. Builds a payload and calls `\Sentry\configureScope(fn ($scope) => $scope->setUser($payload)->setTag('auth.guard', $guard))`.

Payload behavior depends on `watchtower.user_context.fields`:

- **`[]` (default — auto-discover)** — every attribute on the user model except (a) anything in the model's `$hidden` array and (b) the package's hard-coded deny-list (`password`, `password_hash`, `remember_token`, `api_token`, `two_factor_secret`, `two_factor_recovery_codes`). `id` falls back to `getAuthIdentifier()` (survives renamed primary keys), `ip_address` always comes from `$request->ip()`. The `name` column is mapped to Sentry's `username` (Sentry's user schema uses `username` for the display name, not `name`).
- **Explicit list (`['id', 'email']`)** — only those fields. Allowed values: `id`, `email`, `name` (mapped to `username`), `ip_address`.

The middleware swallows exceptions from broken auth drivers — a misconfigured guard never breaks a request, it just means the User tab won't populate for that request.

Result in Watchtower: `IssueDetail` → User tab shows id / email / username / ip rows; Custom tab shows the `auth.guard` pill. Multi-guard apps (web humans + API mobile clients + admin back-office) get an at-a-glance signal of which surface produced the exception.

### BeforeSend pipeline

`Phattarachai\WatchtowerLaravel\Sentry\BeforeSend` is chained in front of any existing `before_send` callback from `config/sentry.php` at service-provider boot. Composition order: ours runs first (drop + scrub); if the event survives, the user's existing callback runs after. Either can return `null` to drop. The same scrub (never the drop) is chained in front of `before_send_transaction`, so sampled transactions leave the SDK scrubbed too. Keep any host callback in `config/sentry.php` as a `[Class::class, 'method']` array: a closure breaks `php artisan optimize`.

Two responsibilities:

**Drop framework noise.** If the event's hinted exception is an instance of any class in `watchtower.before_send.ignored_exceptions`, the event is dropped (returns `null`). Defaults:

```
Illuminate\Validation\ValidationException
Illuminate\Auth\AuthenticationException
Illuminate\Auth\Access\AuthorizationException
Illuminate\Database\Eloquent\ModelNotFoundException
Illuminate\Session\TokenMismatchException
Symfony\Component\HttpKernel\Exception\NotFoundHttpException
Symfony\Component\HttpKernel\Exception\MethodNotAllowedHttpException
Symfony\Component\HttpKernel\Exception\AccessDeniedHttpException
Symfony\Component\HttpFoundation\Exception\SuspiciousOperationException
```

Laravel's HTTP handler already excludes most of these via `$internalDontReport`, but the BeforeSend list is belt-and-braces — it also catches direct `\Sentry\captureException()` calls from queue jobs or console code that bypasses the HTTP handler. Extend per project. Don't subtract unless you actually want validation / auth-fail / 404 noise in the inbox.

**Scrub secrets.** For each event that survives the drop check, the middleware walks `event.request.data`, `event.request.headers` and `event.extra`. For each entry whose key (case-insensitive) matches `watchtower.before_send.scrub_keys`, the value becomes the literal string `[Filtered]`. Defaults:

```
password, password_confirmation, current_password,
token, _token, api_key, secret, authorization, cookie,
x-csrf-token, x-xsrf-token,
credit_card, card, cvv, cvc
```

`x-csrf-token`, `x-xsrf-token` and `_token` are scrubbed even when a published config predates them. The Sentry SDK filters only `Authorization` and `Cookie` headers, so otherwise the CSRF token that Inertia and axios send on every request would reach the event. Every value in `event.request.cookies` becomes `[Filtered]`; the cookie names are kept.

**Scrub URLs.** In `event.request.url` and `event.request.query_string`, a query value whose key (or any bracketed segment of it, `user[token]`) matches `scrub_keys` becomes `[Filtered]`; a signed URL's `signature` always does. Every match of a regex in `watchtower.before_send.scrub_url_patterns` (default `[]`) is replaced with `[Filtered]` too — that is how to hide a token that lives in the URL path, e.g. `'#/feeds/\K[^/?]+#'` turns `/feeds/s3cr3t/orders.csv` into `/feeds/[Filtered]/orders.csv` (`\K` keeps the prefix). A pattern that does not compile filters the whole value.

Then a credit-card-shape regex (`/\b\d(?:[ -]?\d){12,18}\b/`) sweeps the remaining string values and replaces any 13–19-digit number with `[Filtered]`. Intentionally permissive — false positives just redact an innocent number; false negatives leak a real card.

Result in Watchtower: matching exceptions never reach the inbox at all. Surviving events render the literal string `[Filtered]` in the Request tab's headers `<dl>` and body `<pre>`.

To extend either list, publish `config/watchtower.php` and edit the arrays directly. Both are merged from the package defaults at boot, so a published copy is the full picture.

### Breadcrumb env keys

`watchtower:install` writes five `SENTRY_BREADCRUMBS_*` env keys on first run (`setIfAbsent` — re-runs preserve user customization):

| Env key | Default after install | Why |
|---|---|---|
| `SENTRY_BREADCRUMBS_SQL_QUERIES_ENABLED` | `true` | The query itself is the highest-signal breadcrumb when triaging a DB-shaped exception. |
| `SENTRY_BREADCRUMBS_SQL_BINDINGS_ENABLED` | `false` | Bindings carry the actual user data — emails, IDs, sometimes free text. Even with BeforeSend scrubbing, prefer off by default. Opt in when bindings are essential. |
| `SENTRY_BREADCRUMBS_CACHE_ENABLED` | `true` | Cache key + hit/miss is cheap and useful. |
| `SENTRY_BREADCRUMBS_HTTP_CLIENT_REQUESTS_ENABLED` | `true` | Outbound HTTP calls via Laravel's HTTP client; URL + status + duration. |
| `SENTRY_BREADCRUMBS_REDIS_COMMANDS_ENABLED` | `true` | Redis commands (excluding payloads); useful for queue / cache / lock debugging. |

Set any specific key to `false` in `.env` to disable that family without disabling the rest.

### Browser user context

The install command publishes `resources/js/vendor/watchtower.js` and prints two snippets to paste:

1. Per-entry init block — imports `initWatchtower` from the published helper and calls it. The helper runs `Sentry.init(...)` with the Watchtower-tuned defaults (same-origin tunnel, `denyUrls` for browser extensions, no PII) and then reads `<meta name="watchtower-user-{id,email,name}">` from the document and calls `Sentry.setUser({id, email, username})` with whatever values are present (no-op when no meta tags are found). The Sentry config lives inside the helper, so adding the snippet to multiple Vite entries does not duplicate the config.
2. `@watchtowerUser` Blade directive — paste into the root Blade layout's `<head>`. The directive compiles to the three meta tags read by the browser helper, sourced from `auth()->id() / ->user()?->email / ->user()?->name`. For Filament admin pages, the install command prints a `PanelsRenderHook::HEAD_END` snippet that returns `Blade::render('@watchtowerUser')` so the same directive emits inside Filament's `<head>`.

The result: browser-side exceptions get the same User tab population as server-side, with the same `id`/`email`/`username` mapping.

### Livewire: dropped noise and legible failures

Livewire/Filament apps generate a family of browser captures that are either pure noise or unreadable, so the helper ships a `beforeSend` for them. It lives in a second published file, `resources/js/vendor/livewire.js`, kept import-free so it is unit-testable — the package tests it with `composer test:js` (`node --test`, no npm dependency).

**Dropped** — transient, never a real bug: a 419 page-expired rejection (idle session / stale CSRF); a `wire:navigate` transition the user clicked through (`isFromCancelledTransition`); a component torn down mid-navigation (`Component not found: undefined` — a *named* missing component still reports); and a `{status, body, json, errors}` response with every key null, which means the request was superseded or aborted before it settled rather than failed.

**Made legible** — everything else with those four keys. Livewire rejects a failed request with its raw response object instead of an `Error`, so Sentry can only title it *"Object captured as exception with keys: …"* and fingerprint on that key list: a 502 on a lazy-load and a 500 on a table filter collapse into one meaningless issue, with the status buried in `extra.__serialized__`. Worse, the stack is entirely the minified SDK's own capture machinery, so the issue looks like *Watchtower itself* is crashing. The hook restates these as `LivewireRequestFailed: Livewire request failed with HTTP <status>` and fingerprints by status, giving each failure mode its own issue. It filters nothing — it only surfaces data the event already carries.

Because the fingerprint changes, any pre-existing "Object captured as exception with keys: …" issue stops receiving events after you deploy this; resolve it once the replacement issues appear.

### Crawlers and chunk loads

The third published file, `resources/js/vendor/noise.js`, runs first in the helper's `beforeSend`:

- **Crawlers are dropped.** Googlebot, GoogleOther, Google-InspectionTool, bingbot, AdsBot and other crawlers execute the site's JavaScript with a frozen clock and a stale asset cache, so they throw errors no visitor sees (typically chunk loads for a hash from an earlier crawl). The test is `navigator.userAgent` against a `+http://` contact link, a `…bot/2.1` token, a `compatible; …bot` comment, or a named crawler. HeadlessChrome is not matched: browser tests use it, and what they find is real.
- **Chunk-load failures share one issue.** Chrome's `Failed to fetch dynamically imported module: <url>`, Safari's `Importing a module script failed.`, Firefox's `error loading dynamically imported module: <url>` and Vite's `Unable to preload CSS for <url>` are sent with `fingerprint: ['chunk-load-error']`. Without it every deploy's hashed chunk name opens its own issue.

The Watchtower server applies the same two rules at ingest (`WATCHTOWER_DROP_BOT_EVENTS`, and chunk-load grouping in `watchtower-core`), so clients that haven't re-published the helper are covered too. The rules only group and drop; reloading the page on `vite:preloadError` so the visitor gets the new build is up to the app.

The published helper is yours to edit — `--force=false` on re-runs means customizations are preserved. The package version is the source of truth in `vendor/phattarachai/watchtower-laravel/resources/js/{watchtower,livewire,noise}.js`. To customize the Sentry options (e.g. add `ignoreErrors`), edit `Sentry.init({ ... })` inside the published helper, not in every entry. To customize the meta-tag template, publish the view with `php artisan vendor:publish --tag=watchtower-views` and edit `resources/views/vendor/watchtower/user-meta.blade.php` — the `@watchtowerUser` directive will pick up the override automatically.

### Recommended `.env` per environment

| Environment | `SENTRY_LARAVEL_DSN` | `VITE_SENTRY_DSN` | `WATCHTOWER_RELAY_ASYNC` |
|---|---|---|---|
| Production | full DSN | same DSN | `true` (if Horizon present) |
| Staging | full DSN | same DSN | `true` |
| Local dev | `null` | full DSN (for browser dev) or `null` | `false` |

For the rare cases that warrant splitting backend and frontend into two Watchtower projects, set `VITE_SENTRY_DSN` to the second project's DSN — see [When to split into two projects](#when-to-split-into-two-projects).

**Do not set `SENTRY_TRACES_SAMPLE_RATE`.** Watchtower drops trace items.

## Browser-side init

The install command writes the env keys, publishes the helper, and prints two snippets — but does NOT modify your entry JS file or your Blade layout. Those one-time pastes are on you (or pass `--patch-js` / `--patch-views` to auto-inject them).

In `resources/js/app.js` (or wherever your bundler entry is), add at the top:

```javascript
import { initWatchtower } from './vendor/watchtower.js';

initWatchtower();
```

That's the entire per-entry footprint. Repeat for every Vite entry that should report browser errors — the import is cheap and `initWatchtower()` is idempotent (only the first call performs `Sentry.init`).

The Sentry config — tunnel, environment, `sendDefaultPii: false`, browser-extension `denyUrls` — lives inside `resources/js/vendor/watchtower.js`. Customize there once instead of per entry:

```javascript
// resources/js/vendor/watchtower.js
Sentry.init({
    dsn: import.meta.env.VITE_SENTRY_DSN,
    tunnel: import.meta.env.VITE_SENTRY_TUNNEL,  // /api/watchtower-relay
    environment: import.meta.env.VITE_SENTRY_ENVIRONMENT,
    sendDefaultPii: false,
    tracesSampleRate: 0,
    denyUrls: [
        /^chrome-extension:\/\//i,
        /^moz-extension:\/\//i,
        /^safari-extension:\/\//i,
        /^safari-web-extension:\/\//i,
    ],

// ignoreErrors: [
    //     'Network request failed', 'NetworkError', 'Failed to fetch',
    //     'ResizeObserver loop limit exceeded',
    //     'ResizeObserver loop completed with undelivered notifications',
    //     'Non-Error promise rejection captured',
    // ],
});
```

Then in your root Blade layout's `<head>` — `resources/views/components/layouts/app.blade.php` or equivalent — add the package directive:

```blade
@watchtowerUser
```

`@watchtowerUser` compiles to the three `<meta name="watchtower-user-{id,email,name}">` tags that the browser helper reads. To customize, publish the source view: `php artisan vendor:publish --tag=watchtower-views` → edit `resources/views/vendor/watchtower/user-meta.blade.php`.

For Filament admin panels (which bypass the root Blade layout), register a render hook in each panel provider:

```php
use Filament\View\PanelsRenderHook;
use Illuminate\Support\Facades\Blade;

$panel->renderHook(
    PanelsRenderHook::HEAD_END,
    fn (): string => Blade::render('@watchtowerUser'),
);
```

Then install the browser SDK using whichever package manager the consumer already uses — `watchtower:install` detects the lockfile (`bun.lockb`/`bun.lock`, `pnpm-lock.yaml`, `yarn.lock`, `package-lock.json`) and prints the matching one-liner, or reports "already in package.json" when the dep is present. If you're running this manually:

```bash
# pick the one that matches the project's lockfile
npm install --save @sentry/browser
pnpm add @sentry/browser
yarn add @sentry/browser
bun add @sentry/browser

npm run build   # or pnpm build / yarn build / bun run build
```

The `tunnel` option is non-negotiable for production deployments. Without it, ~10–30% of users with ad blockers will silently fail to report errors. The user-context step inside `initWatchtower()` is what fills Watchtower's User tab for browser-side exceptions — without it, you get the same User tab population gap on the browser side that disabling the middleware causes on the server side. Leave `ignoreErrors` commented at first; turn individual lines on once you've seen the actual noise in your project's Watchtower inbox.

### Verify the browser side

Open the app in a browser, then in DevTools console:

```javascript
setTimeout(() => { throw new Error('watchtower probe ' + Date.now()) }, 0)
```

Async throw (not a sync `throw` in the console — the console swallows those). In DevTools → Network, filter for `watchtower-relay` — you should see a `POST /api/watchtower-relay` returning 200.

## Browser-only setup (no Laravel host)

If the project is a pure JavaScript app (Next.js, Astro, Vite SPA, static site with a Node server, etc.) with no Laravel runtime, you have two sub-cases:

### Same-origin Watchtower

If you control DNS and want the JS app served from the *same domain* as Watchtower (e.g. `app.example.com` for the app, `errors.example.com` for Watchtower, both behind a reverse proxy that routes `/api/watchtower-relay` to Watchtower): set the SDK tunnel to point directly at Watchtower's own `/api/watchtower-relay` endpoint. No app-side proxy needed.

```javascript
Sentry.init({
    dsn: import.meta.env.VITE_SENTRY_DSN,
    tunnel: 'https://errors.example.com/api/watchtower-relay',  // direct
    // ...
});
```

### Cross-origin (your own server forwards)

Most cases. You need a tiny relay endpoint in whatever runtime hosts the JS. The job: receive POST `/api/watchtower-relay`, forward the raw body and `Content-Type` / `Content-Encoding` headers to `https://<watchtower-host>/api/watchtower-relay`, return the upstream response verbatim. ~20 lines in any runtime. Express example:

```javascript
app.post('/api/watchtower-relay', express.raw({ type: '*/*', limit: '1mb' }), async (req, res) => {
    const upstream = await fetch('https://<watchtower-host>/api/watchtower-relay', {
        method: 'POST',
        headers: {
            'Content-Type': req.get('Content-Type') ?? 'application/octet-stream',
            ...(req.get('Content-Encoding') ? { 'Content-Encoding': req.get('Content-Encoding') } : {}),
        },
        body: req.body,
    });
    res.status(upstream.status).set('Content-Type', upstream.headers.get('Content-Type') ?? 'application/json');
    upstream.body.pipe(res);
});
```

## Manual install (Laravel — fallback when the package isn't available)

Use only when `composer require phattarachai/watchtower-laravel` isn't an option (locked composer.json, air-gapped environment, Laravel 10 or older).

### 1. Install the SDK

```bash
composer require sentry/sentry-laravel
```

### 2. Wire the exception handler (Laravel 11+) — REQUIRED

Edit `bootstrap/app.php`:

```php
use Sentry\Laravel\Integration;

return Application::configure(basePath: dirname(__DIR__))
    // ...
    ->withExceptions(function (Exceptions $exceptions) {
        Integration::handles($exceptions);
    })
    ->create();
```

Without this, real unhandled exceptions are silently dropped. `php artisan sentry:test` still works (it sends a synthetic event directly), so don't take a successful `sentry:test` as proof of wiring.

For Laravel 10, register in `app/Exceptions/Handler.php` instead — see the sentry-laravel docs.

### 3. Publish config + set the DSN

```bash
php artisan sentry:publish --dsn=https://YOUR_PUBLIC_KEY@<watchtower-host>/NUMERIC_PROJECT_ID
```

### 4. (Optional) Browser tunnel

If you also serve browser JS from this app and want ad-blocker-safe reporting, you'll need a relay route. Either install the package (which provides it for free) or copy this manual controller:

```php
// routes/api.php
Route::post('/watchtower-relay', function (\Illuminate\Http\Request $request) {
    $upstream = (string) env('WATCHTOWER_RELAY_UPSTREAM', 'https://<watchtower-host>/api/watchtower-relay');
    $response = \Illuminate\Support\Facades\Http::withHeaders(array_filter([
        'Content-Type' => $request->header('Content-Type'),
        'Content-Encoding' => $request->header('Content-Encoding'),
    ]))->withBody($request->getContent(), $request->header('Content-Type') ?? 'application/octet-stream')
       ->post($upstream);

    return response($response->body(), $response->status())
        ->header('Content-Type', $response->header('Content-Type') ?? 'application/json');
})->name('watchtower.relay.manual');
```

### 5. (Recommended) PHP setting for richer stack traces

`zend.exception_ignore_args = Off` so Sentry can capture argument values in frames.

## Querying via REST — verify + triage

Watchtower exposes `/api/v1/projects/{project}/...` endpoints for verifying ingestion and triaging issues without opening the web UI. They authenticate with your **personal access token** (PAT), minted at `/me/access-tokens` on the Watchtower server.

### Auth

```
Authorization: Bearer $WATCHTOWER_TOKEN        # your wt_pat_… token
```

`{project}` is the project's numeric id (the DSN's last segment) or its slug. The PAT reaches every project in the teams you belong to; a project outside them answers 404. The DSN public key is **not** a read credential: it ships to every browser. The old flat routes (`/api/v1/issues`, …) still accept it during a deprecation window and answer with a `Deprecation` header and a `Link` to the new URL.

Rate limit: 60 req/min per token.
Pagination: `?page=N`, 25/page.
Errors: standard Laravel `{"message": "...", "errors": {...}}` shape; 401 unauth, 404 not-in-scope, 422 validation, 429 rate-limited.

### Endpoints

Prefix every URL with `/api/v1/projects/{project}`.

| Method | URL | Purpose |
|---|---|---|
| GET | `/` | Show the project (id, slug, name, platform). |
| GET | `/issues` | List issue groups. Filters: `environment`, `level`, `status`, `fingerprint`, `since` (`24h`/`7d` or ISO), `q`. |
| GET | `/issues/{id}` | Show one issue group. |
| PATCH | `/issues/{id}` | Triage. Body: `{"status":"resolved"}` etc. Snooze requires `snooze_duration`. |
| GET | `/events` | List recent events (slim shape — id, level, env, top_frame). Filters: `environment`, `release`, `level`, `since`, `group_id`. Pagination: `page`, `per_page` (1–100, default 25). |
| GET | `/events/{event_id}` | Full event payload (stacktrace frames, breadcrumbs, request, contexts, tags, extra, sdk, mechanism) — verification core AND debugging entry point. |

### Examples

`PROJECT_ID=$(grep -E '^SENTRY_LARAVEL_DSN=' .env | grep -oE '[0-9]+$')` — the DSN's numeric project segment.

Verify a specific exception arrived:

```bash
curl -fsSL -H "Authorization: Bearer $WATCHTOWER_TOKEN" \
  https://<watchtower-host>/api/v1/projects/$PROJECT_ID/events/$EVENT_ID
```

List unresolved errors in production from the last 24h:

```bash
curl -fsSL -H "Authorization: Bearer $WATCHTOWER_TOKEN" \
  "https://<watchtower-host>/api/v1/projects/$PROJECT_ID/issues?status=unresolved&environment=production&level=error&since=24h"
```

Resolve an issue:

```bash
curl -fsSL -X PATCH -H "Authorization: Bearer $WATCHTOWER_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"status":"resolved"}' \
  https://<watchtower-host>/api/v1/projects/$PROJECT_ID/issues/39
```

Snooze for 24h:

```bash
curl -fsSL -X PATCH -H "Authorization: Bearer $WATCHTOWER_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"status":"snoozed","snooze_duration":"24h"}' \
  https://<watchtower-host>/api/v1/projects/$PROJECT_ID/issues/39
```

Valid `snooze_duration` values: `1h`, `24h`, `7d`, `30d`, `until_event`.

> **Privilege scope.** A PAT can read **and** triage every project its user can reach. Keep it in your shell environment, never in a repo.

## Querying via MCP

Watchtower exposes a Model Context Protocol server at `/mcp/{project}` for Claude Code (and any other MCP client). Same auth model as REST — your PAT is the bearer token — and **project-scoped**: each MCP server gives Claude access to one Watchtower project, matching how the project-scoped REST endpoints work. With the package default of one Watchtower project per client app (backend + browser share the DSN), a single MCP registration covers both runtimes.

### Add the server

`php artisan watchtower:install` registers the MCP server with Claude Code automatically when the `claude` CLI is on PATH (use `--no-mcp` to skip). It registers with `--scope project`, so the server is written to a committed `.mcp.json` at the repo root rather than the per-machine local config — anyone who pulls the project and runs `composer install` gets the MCP after a one-time approval prompt, without re-running the installer. To register manually after the fact:

```bash
claude mcp add --transport http --scope project watchtower https://<watchtower-host>/mcp/<project-id-or-slug> \
  --header 'Authorization: Bearer ${WATCHTOWER_TOKEN}'
```

The project id is the DSN's last segment: `grep -E '^SENTRY_LARAVEL_DSN=' .env | grep -oE '[0-9]+$'`. The single quotes keep
`${WATCHTOWER_TOKEN}` literal in `.mcp.json`; Claude Code expands it from its own environment, so export the PAT in the
shell profile Claude Code starts from and restart it.

If you actually split backend and browser into two Watchtower projects (rare — see [When to split into two projects](#when-to-split-into-two-projects)), register one MCP server per project with distinct names so Claude can address each inbox:

```bash
claude mcp add --transport http --scope project watchtower-backend  <host>/mcp/<backend-project-id>  --header 'Authorization: Bearer ${WATCHTOWER_TOKEN}'
claude mcp add --transport http --scope project watchtower-frontend <host>/mcp/<frontend-project-id> --header 'Authorization: Bearer ${WATCHTOWER_TOKEN}'
```

### Tool reference

| Tool | Args | Returns |
|---|---|---|
| `get_stats` | `window?` (`24h`/`7d`/`30d`, default `7d`) | Event volume + per-level breakdown + top-5 issues in window + status_mix |
| `list_issues` | `status?`, `environment?`, `level?`, `q?`, `since?`, `page?` | Paginated issue groups for this project, newest-seen first |
| `get_issue` | `issue_id` | One issue group + permalink to the Watchtower UI + **`latest_event_id`** (skip the `list_events` round-trip — feed it straight to `get_event`) |
| `list_events` | `environment?`, `release?`, `level?`, `group_id?`, `since?`, `page?`, `per_page?` (1–100, default 25) | Paginated events for this project (slim shape — id, level, env, top_frame), newest first |
| `get_event` | `event_id` (UUID) | **Full event payload** — slim fields plus `stacktrace` frames, `breadcrumbs`, `request`, `contexts`, `tags`, `extra`, `sdk`, `mechanism`. Verification core AND debugging entry point. |
| `resolve_issue` | `issue_id` | Marks resolved; re-opens automatically on regression in a higher release |
| `ignore_issue` | `issue_id` | Marks ignored; does NOT re-open on new events |
| `unresolve_issue` | `issue_id` | Re-opens a resolved or ignored issue |
| `snooze_issue` | `issue_id`, `duration` (`1h`/`24h`/`7d`/`30d`/`until_event`) | Snoozes; `until_event` un-snoozes on the next event |

### Verifying an exception via MCP

After capturing an exception in the client app, ask Claude:

> "Use Watchtower to verify event_id `<uuid>` arrived."

Claude calls `get_event`. Success → ingested (and the response includes the full stacktrace + breadcrumbs, so the same call doubles as a debugging starting point). An error response saying *"Event not found in this project. The ingest pipeline is async — retry after a few seconds…"* → the queue hasn't drained yet, try again in a moment.

### Debug an issue end-to-end via MCP

The common "fix issue N" flow is a 2-call path:

> "Fix Watchtower issue #44."

1. `get_issue(issue_id=44)` — title, status, counts, permalink, **and `latest_event_id`**.
2. `get_event(event_id=<latest_event_id>)` — full payload. The agent now has the stacktrace (with `in_app` flags), breadcrumbs (the user actions leading up to the error), request URL/method, contexts, tags, and SDK info needed to open the right file and propose a fix.
3. After the fix lands and is verified, `resolve_issue(issue_id=44)`. Watchtower auto-reopens the issue if a new event arrives in a later release.

### Privilege scope

The PAT holder gets read **and** mutate access to every project their teams own — same boundary as the REST endpoints.

## Troubleshooting

| Symptom | First thing to check | Fix |
|---|---|---|
| Laravel: `php artisan sentry:test` works, but real exceptions never appear | `Integration::handles($exceptions)` not wired in `bootstrap/app.php` | Re-run `php artisan watchtower:install`, or apply the diff manually |
| Laravel: `php artisan sentry:test` returns 404 | DSN project segment is non-numeric (slug instead of id) | Re-copy the DSN from project settings — it should end in a number |
| Laravel: `php artisan sentry:test` returns 401 | DSN public key stale | Re-copy DSN |
| JS: no network request at all | `import.meta.env.VITE_SENTRY_DSN` was empty at *build* time | Set the env, then `npm run build` |
| JS: request blocked / "Failed to fetch" / status 0 / `503` and no nginx hit | Ad blocker matching `sentry_key=` | The `tunnel` option fixes this — verify it's set |
| JS: `window.Sentry.getClient().getDsn()` returns `null` | DSN project segment non-numeric | Use numeric id in DSN |
| Tunnel: `502 upstream_unreachable` | Client app can't reach upstream Watchtower | DNS / firewall — check `WATCHTOWER_DSN` host is reachable from the app server |
| Tunnel: `503 watchtower_dsn_missing` | `WATCHTOWER_DSN` not set in `.env` | Set it, then `php artisan config:clear` |
| Either: ingest returns 200 but issue page stays empty | Watchtower's Horizon `ingest` queue is stalled | Contact the Watchtower admin |
| Either: `429 Too Many Requests` from ingest | Per-minute rate limit hit | Back off, or ask admin to raise `rate_limit_per_min` for the project |
| `/api/v1/...` returns 401 | Sending the DSN public key (ingest-only), or the PAT is revoked / expired / not exported | Send your PAT; mint a new one at `/me/access-tokens` |
| `/api/v1/events/{id}` returns 404 right after sending | Async ingestion hasn't drained yet | Retry after 2–5s |
| MCP: `claude mcp add` succeeds but tool calls fail with `Unauthenticated` | `WATCHTOWER_TOKEN` not exported in the shell Claude Code started from, or the PAT is revoked | Export it (e.g. in `~/.zshrc`), restart Claude Code; mint a new PAT if revoked |
| MCP: 404 `Project not found` | The `/mcp/{project}` segment names a project outside your teams, or a typo | Use the DSN's numeric project id, or a slug from `GET /api/v1/me/projects` |
| MCP: tool says "Issue not found in this project" but the issue is visible in the Watchtower UI | The issue belongs to a different Watchtower project than the one in your `/mcp/{project}` URL | Re-run `claude mcp add` with that project's id, or register a second MCP server alongside the first with a distinct name |
| Embedded MCP returns 401 after upgrading | `WATCHTOWER_MCP_LEGACY_KEYS=false` and `.mcp.json` still sends the public key | `php artisan watchtower:project mcp-token <project>`, export the token as `WATCHTOWER_MCP_TOKEN` |
| User tab in Watchtower is empty despite a logged-in user triggering the exception | Smart-defaults chain broken at one of three points | Check (a) `SENTRY_SEND_DEFAULT_PII=true` — without it Sentry strips request data + IP before BeforeSend runs; (b) `WATCHTOWER_USER_CONTEXT` not set to `false`; (c) the right guard is listed in `WATCHTOWER_USER_CONTEXT_GUARDS` (or it's `auto`); (d) for browser-side, the `<meta name="watchtower-user-*">` tags are in `<head>` (via `@watchtowerUser`) AND `initWatchtower()` is called. See [Smart defaults](#smart-defaults). |
| Inbox is suddenly missing `ValidationException` / `404` / auth-fail events | BeforeSend smart-default is dropping them — this is the design | If you actually want these in the inbox, edit `watchtower.before_send.ignored_exceptions` and remove the relevant class, or set `WATCHTOWER_BEFORE_SEND=false` to disable the filter entirely. |
| Request body in event payload shows `[Filtered]` for a non-secret field | The field name matches an entry in `watchtower.before_send.scrub_keys` (case-insensitive) | Edit `watchtower.before_send.scrub_keys` to remove the entry, or rename the field. |
| Standalone: a child app's events 404 at ingest | Its DSN has an `api` segment, or is missing the path prefix | Copy the DSN from `php artisan watchtower:project list` verbatim — see [DSN format](#dsn-format--must-be-numeric) |
| Standalone: `/watchtower` renders unstyled | The Tailwind `@source` line for the package is missing, so none of the module's classes were emitted | `php artisan watchtower:doctor`, then add the line it names and rebuild |
| Standalone: `/watchtower` 403s for an admin | No auth check registered — the gate is closed outside `local` by default | Register `Watchtower::auth(...)` or a `viewWatchtower` gate |
| Dual: backend exceptions never reach the central Watchtower | By design — the install points `SENTRY_LARAVEL_DSN` at the local project | Use `relay` mode if the central inbox must see server-side events |
| Standalone: events accepted but the issue list stays empty | A queue worker isn't running for `WATCHTOWER_QUEUE_CONNECTION`, or nothing consumes `WATCHTOWER_QUEUE_NAME` | `php artisan watchtower:doctor`; start a worker / add the Horizon supervisor, or set the connection to `sync` for a small install |
| Standalone: Redis memory balloons with `ProcessEventJob` payloads | The self-capture loop (pre-1.2.0) or a flood with no dedicated queue | [Self-capture loop — triage](#self-capture-loop--triage) |
| Standalone: an issue's event count keeps rising but no new events appear | The per-fingerprint budget is dampening it — counted, not stored | Expected. Raise `WATCHTOWER_RATE_LIMIT_PER_FINGERPRINT_PER_MIN` if you need more samples |
| Standalone: an event shows the `watchtower.truncated` tag | It was over `WATCHTOWER_MAX_EVENT_BYTES` / had strings over `WATCHTOWER_MAX_STRING_BYTES` | Expected; raise the limits if a specific field matters |
| Standalone: a failing `ProcessEventJob` never shows up in the Watchtower inbox | By design — Watchtower's own job failures are not self-captured | Check `failed_jobs` / Horizon's failed list and the worker log |

## Updating this skill

This skill is shipped by the `phattarachai/watchtower-laravel` composer package and discovered by Laravel Boost from `vendor/phattarachai/watchtower-laravel/resources/boost/skills/`. To refresh:

```bash
composer update phattarachai/watchtower-laravel
php artisan boost:install --skills
```

Boost re-copies the skill files into `.claude/skills/watchtower-error-tracking/` (and the agent-specific equivalents for Cursor, Codex, etc.). The `version:` field in the frontmatter tracks releases.

## Roadmap

This skill is feature-complete for Phase 6. Remaining items, deferred until real need shows up:

- Read-only tokens. A PAT reads, triages and provisions; an embedded MCP token reads and triages its project. Add narrower abilities if tokens ever go to people outside the owner's teams.
- Bulk triage tools (`resolve_many`, `snooze_many`). Not needed until agents complain about one-at-a-time mutations.