Filament Ops Notify Laravel Package

Technical Evaluation

What this is. A Laravel package for operational alerting: a facade-style sender (OpsMessage::send()), a custom ops notification channel that plugs into Laravel's notification system, a burst guard (per-event, per-destination rate limiting), a digest that reports suppressed volume, a chat-destination layer with a paged discovery scan, and a Filament-style panel plus an artisan command (ops-notify:test) for operator self-service. It sits on the critical path of business events — the release notes explicitly name "saving an inquiry" and "finishing a build" as the host requests that could be affected.

Architecture fit. The package is well-behaved Laravel: it uses the cache for dedupe/burst state, the queue for digest and resend work, the scheduler for digest timing, and the DB for the suppressed-row log. That is idiomatic, but it means the package inherits the failure characteristics of every one of those subsystems. The most important architectural property of v1.3.1 is that it is now fail-open on the alerting path: an unreachable cache degrades to "log and send anyway" rather than throwing into the host request. For a package whose blast radius previously included 500s on unrelated user actions, this is the single most valuable change in the release.

Integration feasibility. Very low friction. No config changes, no migrations, no breaking changes for normal use. The one internal API change — BurstGuard::pullHeld() replaced by heldCount() + clearHeld() — matters only if our application (or another package) has subclassed, decorated, or reflected into BurstGuard. That is checkable in minutes and should be a pre-merge grep, not a design discussion.

Technical risk.

• Low: upgrade itself, given the absence of schema and config deltas and the stated 285-test / Larastan level 8 baseline.
• Moderate, and the one to think about: semantics shift from at-most-once toward at-least-once. During a cache outage, dedupe is bypassed by design ("a duplicate beats a lost alert"). That is the right trade for paging, but it means our alert-noise tolerance and any downstream consumer that keys on alert uniqueness should be checked.
• Moderate, easily missed: the burst key now includes the destination, where it previously did not. One event fanned out to two chats/topics will now maintain two independent counters instead of sharing one. That is a correctness fix, but it is also a volume change — multi-destination events will fire more often per unit time than they did on v1.3.0. Anyone who tuned thresholds against the old behaviour should re-baseline.
• Low: operator-visible workflow changes — resend now honestly reports Failed instead of a false "Queued", and ops-notify:test --queue returns non-zero when nothing was queued. Both are improvements; both will look like a regression to whoever wrote a runbook around the old output.
• Low: the chat-discovery lock is now held for the whole paged scan (~100 s worst case) instead of a fixed 60 s. If discovery runs as a queued job, the worker timeout must exceed that, or the lock outlives the worker.

Key questions.

1. Is BurstGuard treated as internal by the maintainer, and does anything in our codebase or our other packages extend it?
2. Which cache drivers are supported and tested? The dedupe/burst logic depends on Cache::add + increment atomicity and TTL re-arming; Redis and Memcached are the safe assumptions, and we should confirm the database cache driver is either supported or explicitly out of scope.
3. What is the expected duplicate rate during a cache outage, and what is the alert channel's tolerance for it?
4. How many events currently fan out to more than one destination, and what is their current alert frequency? This sets the re-baselining effort for the burst-key change.
5. What Laravel and PHP versions are supported, and is this an internal package with an owner/SLA, or effectively unowned infrastructure?
6. Does the digest have a documented retry policy and dead-letter behaviour, given the release notes mention "with retries"?

Integration Approach

Stack fit. Assumes a conventional Laravel deployment: queue worker, scheduler, cache, DB, plus Horizon for the panel's status widget. No new services are introduced by this release. The package is a leaf dependency — nothing else depends on it — which keeps the integration boundary clean.

Migration path. Treat as a routine dependency bump: update the constraint, composer update, deploy. No migration step, no config republish, no cache flush strictly required — though flushing the burst/digest keyspace is harmless and gives a clean baseline for the changed key shape. The only pre-merge gate is the pullHeld() grep. If it comes back clean, this is a one-line change and a pipeline run.

Compatibility. No breaking changes for normal use, per the maintainer. The residual compatibility surface is the burst-key shape (behavioural, not API), the discovery lock duration (infrastructure, not code), and the panel's resend status semantics (operator-visible). None of these are compile- or boot-time concerns.

Sequencing.

1. Static check for pullHeld() / BurstGuard extension points; clear the internal-API gate.
2. Confirm worker timeout > ~120 s if chat discovery is queued, and confirm the scheduler is running the digest job.
3. Deploy in a normal window — this is a reliability release, so it does not need a change freeze, and delaying it leaves the host-request 500 exposure open.
4. Immediately post-deploy: run ops-notify:test --queue and confirm exit code 0 in the pipeline; confirm a non-zero exit when the queue is deliberately unreachable.
5. Within the first day: watch alert volume per event, specifically multi-destination events, to validate the burst-key change did not breach anyone's noise budget.
6. Within the first week: exercise a resend from the panel and confirm the row stays Failed with its button when the resend is burst-held — i.e. verify the honest-status behaviour an operator now depends on.

Rollback. Clean, because there is no schema or config delta. Reverting the constraint restores prior behaviour, with the caveat that in-flight burst counters carry the new destination-scoped keys and will look like cold keys to the old version, temporarily letting suppressed volume through. Acceptable for a reliability-only release; worth stating rather than discovering.

Operational Impact

Maintenance. Low and decreasing. The package appears actively maintained — this release is the output of a full src/ review, which is a good signal of maintainer attention rather than a one-off patch. The 285-test suite with Larastan level 8 gives reasonable regression confidence for a package of this size, and the release notes are unusually specific about root cause, which lowers the cost of future incident diagnosis.

Support. Support burden shifts decisively in our favour with this release. Two classes of tickets go away: "the app 500'd while saving an inquiry because alerts broke", and "the panel said Queued but nothing was sent". Both were the kind of bug that erodes trust in the alerting system and, by extension, in the panel. The honest Failed state and the meaningful exit code on ops-notify:test --queue also mean we can assert on alerting health in CI rather than discovering it in production.

Scaling.

• Cache becomes the throughput and correctness bottleneck: every send does a dedupe check, every event does a burst check, and the design now degrades to bypass-on-failure rather than fail. Size and monitor Redis accordingly, and alarm on cache unavailability as an alerting-degradation signal, not just a platform signal.
• Digest generation does a bounded, destination-filtered, truncated-title query. That is now cheaper and more correct than the previous unbounded lookup — good for large log tables, but the suppressed-row table still grows and needs a retention policy.
• Chat discovery holds a lock for up to ~100 s. That is fine for a scheduled job at our scale; it becomes a scheduling consideration if event count or chat count grows substantially, since discovery and digest compete for the same worker pool.

Failure modes, post-upgrade.

• Cache down: alerts still send, dedupe is skipped, duplicates possible — by design. Host requests are unaffected. This is the correct failure direction.
• Queue down or backed up: digests accumulate or fail; the held count is retained until the digest is queued, so the eventual digest reports the real number rather than a partial one. Alerts themselves are delivered inline.
• DB hiccup during suppression logging: digest is scheduled before the row is logged, and if scheduling fails the message is sent instead of dropped. No silent loss.
• Scheduler not running: digests never fire, and suppressed events stay suppressed with no summary. This is now the most plausible silent-failure mode and the one to monitor explicitly.
• Worker timeout shorter than the discovery scan: lock held past job death, blocking subsequent discovery runs. Concrete misconfiguration to check at deploy.
• Residual: the digest's title query now matches the truncated event name and bounds the window, fixing both long-name listing and late-digest cross-window contamination. Low-risk, but verify on an event with a genuinely long name.

Ramp-up. Minimal for developers — no new concepts, no config. The ramp-up burden is on operators, in two small ways: the resend button's behaviour is now truthful and therefore sometimes stays Failed (expect one round of "why didn't it queue?"), and multi-destination events may alert more often than the previous baseline. A short note in the runbook covering the honest resend state, the ops-notify:test --queue exit-code check, and the expectation of duplicates during a cache incident covers essentially all of it.

Recommendation. Upgrade. This release removes a production-availability defect that reached past the alerting subsystem into unrelated user-facing requests — the highest-value class of fix a package like this can ship. The cost is one grep, one worker-timeout check, and a brief re-baseline of alert volume for fanned-out events.