Corewatch Laravel Package

Technical Evaluation

Architecture Fit

• Embedded DevOps Dashboard: Aligns well with Laravel’s monolithic/traditional architecture, eliminating the need for external monitoring tools (e.g., Prometheus, Grafana, ELK). Reduces operational complexity by consolidating observability into the application layer.
• Zero-Dependency Design: Leverages Laravel’s built-in components (queues, caching, logging, scheduler) without introducing external services, minimizing integration overhead.
• Filament Integration: Tight coupling with Filament (admin panel) suggests a UI-first approach, ideal for teams already using Filament. May require Filament adoption if not already in use.
• Safe Ops Panel: Provides a self-service debugging layer for non-technical stakeholders (e.g., support teams), reducing reliance on manual log parsing or SSH access.
• New in 2.2.0 — SystemDoctor diagnostics service: php artisan corewatch:doctor and the programmatic CoreWatch::doctor() / CoreWatchManager::doctor() API add a pre-flight and runtime health-check capability across System, Database, Cache, Queue, Scheduler, Alerts, and Security. This is architecturally significant: the package now exposes a machine-readable health surface, which makes it usable as a deployment gate and as a liveness/readiness probe rather than purely a human-facing dashboard. It is also the first API in the package that is useful in CI without a browser.
• New in 2.2.0 — Asynchronous alerting: SendQueuedAlertJob (ShouldQueue, backoff retries, queue selectable via COREWATCH_NOTIFICATIONS_QUEUE) moves alert delivery off the request path. This is a correct architectural direction (alerting should never block a request), but it introduces a hard runtime dependency on a functioning queue worker for alert delivery — a shift from "alerts are synchronous side effects" to "alerts are a background pipeline."

Integration Feasibility

• Laravel Native: Designed for Laravel 10+ (PHP 8.1+), with minimal configuration required. Compatible with:
  • Queues (async log processing; now also async alert dispatch).
  • Cache (metrics storage).
  • Database (alert history).
  • Scheduler (health-check and retention routines).
• Lightweight: Metrics collection is non-blocking (event-listener based), ensuring minimal performance impact.
• Alerting: Custom thresholds plus Laravel notification channels (Slack, email) — and now Telegram is explicitly supported and testable.
• New in 2.2.0 — corewatch:test-alert: An interactive diagnostic command that verifies Slack and Telegram channel delivery end-to-end. This materially de-risks the single most failure-prone part of any observability rollout (silently misconfigured alert channels) and shortens the "is it wired up correctly?" validation loop from a production incident to a one-minute CLI check.
• New in 2.2.0 — Resilience hardening: Exception handling across collectors, controllers, repositories, and actions widened from \Exception to \Throwable. Practically, this means PHP Errors (TypeError, out-of-memory-adjacent fatals, bad method calls) no longer propagate out of the observability layer into user requests. For a TPM this is the difference between "monitoring package can take down checkout" and "monitoring package degrades quietly."
• New in 2.2.0 — Quality signals: PHPStan Level 8 with zero errors, 32 tests / 211 assertions, resolved Composer security advisories. These are the strongest procurement-grade signals in this release: they are auditable, they raise confidence in upgrade safety, and they answer the "is this maintained?" question that a 5-star GitHub repo cannot.

Technical Risk

• Single-Point-of-Failure: Embedded design means dashboard uptime is tied to the app. Downtime in production = lost observability. (Partially mitigated in 2.2.0 — doctor() can be called from an external prober, but the prober is still your own app.)
• Resource Contention: High-traffic apps may experience cache/database load from metrics storage. Requires tuning (e.g., TTL for cached metrics).
• Filament Dependency: If not using Filament, requires either custom UI integration (higher effort) or forking to replace Filament with another admin panel (Nova, Backpack).
• Log Volume: Unbounded log streaming could bloat storage if not managed (log rotation, retention policies).
• Security: Exposes server metrics via web UI — must be IP-restricted or behind auth (e.g., Laravel Sanctum). Note 2.2.0's doctor command explicitly reports on Security, which gives a verification hook for this control rather than relying on code review.
• New in 2.2.0 — Alert delivery latency and queue coupling: Alerts are now queued. If the worker is down, alerts are silently delayed (or lost without a dead-letter/replay path being confirmed). This is a new single point of failure for the alerting path specifically, distinct from the dashboard SPOF. Backoff retries help on transient failures, not on "no worker running."
• New in 2.2.0 — \Throwable as a double-edged sword: Catching all Throwables prevents application crashes, but can convert loud failures into quiet ones. If the package swallows an Error without logging at an appropriate level, you lose signal about your observability layer being broken. Mitigation: verify what the package logs when it catches; add an alert on "collector error" events if exposed.
• Minor version, behavioral change: 2.2.0 is a minor bump with no listed deprecations, so API compatibility risk is low; however, async alerting is a behavioral change even if the config surface is unchanged (delivery is no longer synchronous). Treat it as a behavior change in the change record, not a drop-in patch.
• Residual risk from prior version persists: nothing in 2.2.0 addresses the Filament coupling, distributed/microservice unsuitability, or serverless unsuitability.

Key Questions

1. Do we use Filament? If no, what's the UI integration strategy (custom routes, alternative admin panels)?
2. Alert delivery SLA: What is our acceptable p95 latency from threshold breach to Slack/Telegram message now that alerts are queued? Does the retry/backoff profile fit our on-call expectations?
3. Queue ownership: Which queue and connection will COREWATCH_NOTIFICATIONS_QUEUE point to, who owns its worker supervision, and what happens to alerts during a deploy that drains workers?
4. Failure observability of the observability layer: When a collector throws a Throwable and is caught, where does that surface? Do we get a log entry, a metric, or nothing?
5. Health-check integration: Can CoreWatch::doctor() back our /health endpoint or a synthetic external probe, and what is its own runtime cost?
6. Log and metric retention: Need explicit policies (e.g., delete logs >30 days, TTL on cached metrics).
7. Alerting strategy: Built-in CoreWatch alerts, or forward into an existing tool (PagerDuty/Opsgenie) to avoid a second on-call surface?
8. Performance impact: Benchmark metrics collection and the alert job under load (e.g., 10K RPS) — the 2.2.0 test suite is unit-level confidence, not a load test.
9. Deployment/rollout gating: Will we use corewatch:doctor as a pre-cutover gate in staging and production?
10. Compliance: Are server metrics (CPU, memory, queue depth) sensitive data? If yes, how is access secured and audited?

Integration Approach

Stack Fit

• Best Fit For:

  • Laravel monoliths (not microservices).
  • Teams using Filament for admin panels.
  • Projects needing embedded observability without external tooling.
  • Teams that want a pre-deploy verification step (2.2.0 doctor + test-alert make the package CI-adjacent for the first time).

• Stack Requirements:

  Component	Compatibility	Notes
  Laravel	10.x+ (PHP 8.1+)	Uses Laravel’s service container.
  PHP Extensions	pcntl, posix (for process metrics)	May need enabling on shared hosting.
  Database	MySQL, PostgreSQL, SQLite	Alert history stored in DB.
  Cache	Redis, Memcached, Database	Metrics caching layer.
  Queues	Database, Redis, SQS	Async log processing and (2.2.0) async alert jobs.
  Scheduler	Laravel scheduler	Required for health/recurring checks per 2.2.0 doctor.
  Filament	v3.x	UI integration.
  Queue worker	Supervisor / Horizon	Now required for alert delivery (2.2.0).

• Anti-Patterns:

  • Microservices: Not designed for distributed systems (no service discovery).
  • Serverless: Assumes persistent PHP processes (no AWS Lambda support); 2.2.0's queued alerts make a long-lived worker even more of a prerequisite.
  • "Set and forget" alerting: with async dispatch, an unsupervised queue is now an undetected outage of your alerting.

Migration Path

1. Pilot Phase (Staging):
   • Install/upgrade via Composer: composer require hamzi/corewatch:^2.2.
   • Publish/review config/corewatch.php; set COREWATCH_NOTIFICATIONS_QUEUE explicitly rather than relying on defaults.
   • Run php artisan corewatch:doctor immediately post-install to establish a baseline and catch missing PHP extensions/permissions early.
2. Core Features Rollout:
   • Enable metrics dashboard (CPU, memory, queries).
   • Enable log streaming (tail -f equivalent).
   • Enable alerts (5xx errors, high memory) and validate channels with php artisan corewatch:test-alert.
3. Gradual Adoption:
   • Start read-only (metrics + logs, no alerts) to establish baselines.
   • Enable alerts to a non-paging channel (team Slack) for one to two weeks before routing to on-call.
4. Production Cutover:
   • Confirm a supervised worker consumes the alerts queue; verify a real alert traverses queue → job → channel.
   • Monitor performance impact (top/htop, queue depth) during peak load.
   • Set retention policies for logs/metrics.
5. Post-Cutover Verification (new):
   • Schedule corewatch:doctor (or CoreWatch::doctor()) as a recurring check and alert on its failure — the observability layer should be able to report on itself.

Compatibility

• Backward Compatibility: 2.2.0 is a minor release with no listed deprecations or breaking config changes; expect a low-friction upgrade from 2.x. The practical exception is behavioral: alert delivery becomes asynchronous, so any test or runbook that assumes an immediate send will need updating.
• Customization Points:
  • Override metric collectors (add custom business metrics).
  • Extend alert rules via Laravel events.
  • Replace Filament UI with custom Blade views.
  • New: consume CoreWatch::doctor() programmatically in health endpoints, deploy scripts, and pipelines.
  • New: bind alerts to a dedicated queue connection for isolation.
• Known Conflicts:
  • Other Telescope-like packages (e.g., laravel-debugbar) may duplicate metrics.
  • Log drivers (Monolog handlers) may need configuration to avoid duplication.
  • New: alerting now competes for worker capacity with business jobs — isolate via a dedicated queue to prevent observability from starving checkout/email/job throughput (and vice versa).

Sequencing

1. Pre-Reqs:
   • Laravel 10+ with Filament (if using UI).
   • PHP extensions (pcntl, posix) enabled.
   • A supervised queue worker and an assigned queue name for alerts.
2. Step 1: Core Setup
   • Publish config: php artisan vendor:publish --provider="Hamzi\CoreWatch\CoreWatchServiceProvider".
   • Configure config/corewatch.php incl. COREWATCH_NOTIFICATIONS_QUEUE.
3. Step 2: Pre-Flight Diagnostics (new)
   • Run php artisan corewatch:doctor; resolve anything flagged under System, Database, Cache, Queue, Scheduler, Alerts, Security.
4. Step 3: UI Integration
   • Register Filament resources (if applicable); customize dashboard layout (Blade).
5. Step 4: Alerting
   • Define thresholds in config/corewatch.php.
   • Configure Slack/Telegram/email channels.
   • Verify with php artisan corewatch:test-alert (interactive feedback) — do this before enabling production thresholds.
6. Step 5: Monitoring Verification
   • Confirm metrics appear in the dashboard; test log streaming via php artisan corewatch:tail.
   • Confirm an alert round-trips through the queue under load.
7. (Step 6 in prior assessment removed as a numbering artifact.)
8. Step 6: Production Hardening
   • Secure dashboard routes (middleware/IP allowlist) and confirm via doctor's Security check.
   • Set up log retention (scheduled purge) and metrics TTL.
   • Wire doctor() into a recurring health check.

Operational Impact

Maintenance

• Pros:
  • No external dependencies: Updates via Composer; no infrastructure management.
  • Laravel-native: Leverages familiar tooling (queues, caching, events, scheduler).
  • Rising quality bar (2.2.0): PHPStan Level 8 at zero errors, 32 tests / 211 assertions, and resolved Composer security advisories materially reduce the cost and risk of routine upgrades. Static analysis at L8 is a credible proxy for "this will not explode on a Laravel minor bump."
• Cons:
  • Vendor Lock-in: Custom metrics/alerts tied to CoreWatch’s schema.
  • Upgrade Risk: Breaking changes may require migration (config structure, Throwable semantics in custom collectors).
  • Dual maintenance surface: you now maintain both the package and a queue worker dedicated to it.
• Maintenance Tasks:
  • Monitor package updates (MIT license allows forks if needed).
  • Prune log/metric data (monthly cleanup job).
  • Update Filament dependencies if using UI.
  • Keep the alerts queue worker supervised and monitored (Supervisor/Horizon).
  • Re-run corewatch:doctor after infrastructure changes (new DB, cache driver swap, queue backend move).

Support

• Pros:
  • Self-Service Debugging: Support teams can access logs/metrics via UI.
  • Reduced Context Switching: No jumping between SSH, logs, and metrics.
  • Faster triage of the package itself: corewatch:doctor gives support/first-line a scripted diagnostic instead of escalating to engineering — a genuine tier-1 deflection win. corewatch:test-alert similarly converts "alerts aren't firing" from an engineering ticket into a self-service check.
  • Fewer "monitoring broke the app" escalations: Throwable hardening means the dashboard failing should no longer take down user-facing features.
• Cons:
  • Learning Curve: Teams must learn CoreWatch’s UI, the doctor output vocabulary, and queue concepts.
  • Limited Community: Low GitHub stars (5) means slower issue resolution; 2.2.0's test suite partly compensates by making self-diagnosis viable.
  • New failure question support must learn to ask: "Is the queue worker running?" — currently absent from most Laravel runbooks and an easy triage miss.
• Support Workflows:
  • Incident Response: Use Safe Ops panel to triage (e.g., "Why is this endpoint slow?"); run corewatch:doctor as step zero.
  • On-Call Rotation: Alerts trigger Slack/PagerDuty; verify channel health with test-alert after any credential rotation.
  • Runbooks needed: "How to check queue backlog," "How to verify alert channels," "What to do when doctor reports a failed subsystem," "What to do if alerts stop arriving."

Scaling

• Performance:
  • Metrics Collection: Low overhead (event listeners); high-frequency data may require sampling.
  • Log Streaming: Async via queues; risk of lag under heavy load.
  • Alert Dispatch (2.2.0): Off the request path — a clear improvement — but throughput is now bounded by worker concurrency and retry/backoff behavior during channel outages (e.g., Slack 5xx storms).
  • Database Load: Alert history scales with usage; archive old data.
• Scaling Strategies:
  • Horizontal Scaling: Deploy on all app instances (metrics aggregated via cache); be aware doctor reports per-instance system state, so multi-instance deployments need per-instance invocation or aggregation.
  • Queue Isolation: Dedicated connection/queue for SendQueuedAlertJob to prevent cross-contamination with business jobs.
  • Log Retention: Laravel log-max-files or custom cleanup.
  • Caching: Adjust TTL (1-minute averages instead of per-second).
• Bottlenecks:
  • Queue Backlog: Log processing and now alert delivery may lag if workers are saturated.
  • Alert Retry Storms: Backoff retries are correct but can stack up during extended channel outages; cap and monitor.
  • Cache Eviction: High-cardinality metrics (per-user) could bloat Redis.

Failure Modes