Laravel Pennant Experiments Laravel Package

Technical Evaluation

Architecture fit. The package is a thin decision-and-measurement layer on top of Laravel Pennant, which is the right place for it: variant assignment rides on Pennant's existing scope resolution, cache, and database driver, so there is no new infrastructure, no assignment service, and no per-request network hop. Assignment is hash-based and sticky, which suits a Laravel monolith serving users, teams, and arbitrary scopes. The trade-off is coupling: the experiment variant is a Pennant feature value, so feature-flag resolution semantics, cache invalidation, and the features table become part of your experiment-correctness surface. Winner rollout rewrites stored Pennant values, meaning the package has a write path into flag state — a "graduation" operation that product and engineering should agree is owned by one team, not two.

Integration feasibility. Low for the server side: define experiments in code, render @variant in Blade, call Experiments::convert() where the goal occurs. Client-side conversions require same-origin session cookies (the JS endpoints resolve the participant from session or cookie, never the request body — a sound choice), so this fits server-rendered Blade, Livewire, and same-origin Inertia/Vue/React apps, and does not fit headless backends serving native mobile clients or cross-origin SPAs without explicit CORS/credentials work. The optional dashboard tab is genuinely optional, verified against both v1.0.2 and v1.1.0 — a good sign for dependency hygiene.

Technical risk.

• Exposure is explicit, and that is a footgun. variant() alone never counts, and conversions are only recorded for exposed scopes. A team that implements assignment but forgets @variant gets zero exposures and zero conversions — indistinguishable from "no one converted." Needs a lint/QA checklist, not just docs.
• Scope of measurement is narrow. Unique converters per goal, binary only. No revenue, no continuous metrics, no per-participant value. That excludes most pricing, ARPU, and engagement-depth experiments. Confirm this matches the first 6–12 months of the roadmap before committing.
• Identity gaps. The guest→user handoff on login is well handled (counted once, same variant), but a guest signing in on a second device keeps that device's separate visitor exposure. For logged-out-heavy products this dilutes measured effect and inflates participant counts; acceptable for a first release, but it is a permanent caveat in results you publish.
• Decision policy is baked in. Z-test with difference interval, Bayesian probability to beat control, Wilson intervals, Bonferroni across challengers, minimum-data rule, planned sample size. The verification story is unusually strong for a v0.1.0 — textbook values, numpy cross-check, and a 4,000-visitor browser run independently reproduced in Python. But the package is choosing your stopping rule. Bonferroni is conservative (the 3-variant run correctly said "Keep running" at p=0.0006 because the planned n wasn't reached), and nothing in the notes addresses peeking or correction across multiple goals — only across challengers. Tracking five goals uncorrected will eventually produce a false "Better."
• Maturity. v0.1.0, unofficial, MIT, single-maintainer profile is likely. Pre-1.0 means breaking changes are expected, including potentially to stored data shapes.
• Compliance. A long-lived http-only cookie holding a random assignment id is plausibly "strictly necessary" but is not unambiguously so under ePrivacy. Legal review needed before EU/UK traffic, and the cookie needs to appear in your consent/cookie inventory.
• Admin surface. Gates default to local-only, read-only mode exists, and end is a destructive-ish operation. Defaults are appropriately paranoid for a package shipping a results UI plus a data-rewriting command.

Key questions. How do exposures and conversions scale at 10M+ rows — table growth, indexes, and results-page query cost? Is there retention/pruning? Can assignment weights be changed safely, or is a guard that blocks mid-test weight edits warranted? Is rollback of a winner rollout defined? What is the maintainer's release cadence and Laravel-version support policy? Does the results UI invite early stopping in practice, and what guidance ships with "Keep running"?

Integration Approach

Stack fit. Best fit: Laravel 12/13 monolith or modular monolith, PHP 8.3+, server-rendered or same-origin front end, Pennant already present or willing to be added, moderate experiment velocity (a handful concurrent). Poor fit: mobile-client products, multi-origin SPA estates, high-concurrency experimentation programs needing holdouts, mutual exclusion, or layered/orthogonal assignment — none of which are in scope here.

Migration path. Additive, not a rip-and-replace. Pennant flags that are pure feature gates keep working untouched; experimentation is opted into per feature. If you are coming off LaunchDarkly Experiments, Statsig, Optimizely, or GA4 experiments, treat this as a downgrade in capability, upgrade in cost and data ownership — you lose the identity graph, holdouts, mutual exclusion, and vendor-managed sequential testing. If you currently do nothing but ship flags and eyeball dashboards, this is a clear upgrade: real intervals and a documented verdict rule replace vibes.

Compatibility. Verified on PHP 8.3/8.4/8.5 × Laravel 12/13, lowest-deps run against Pennant v1.18.0, 66 tests, Pint, PHPStan level 6. Laravel 11 is unsupported, which is fine given the release date. Dashboard integration is optional and version-tolerant in both directions.

Sequencing.

1. Legal/privacy review of the guest cookie and exposure logging; add to cookie inventory. Blocking for EU traffic, parallel otherwise.
2. Install with read-only mode on and gates limited to the platform team; confirm php artisan about and the results routes come up.
3. First experiment: one low-blast-radius change on an existing Pennant feature, one goal, @variant in one template, one PHP conversion. Diff exposure and conversion counts against raw queries before trusting the UI.
4. Reconcile the package's stats output against an independent calculator on that same dataset before anyone reads a verdict.
5. Expand to 2–3 concurrent experiments; define the review cadence and the "who may call end" rule at this point.
6. Enable winner rollout and the end command in shared environments only after a backup/restore drill on the features table.

Operational Impact

Maintenance. MIT community package at v0.1.0 — pin the exact version and treat upgrades as code review, not composer update. It writes to flag storage, so upgrades are data-affecting. Budget ~1–2 engineer-days per quarter for upgrades and Pennant compatibility checks, plus the cost of re-validating stats whenever the verdict logic changes.

Support. No vendor SLA, no support contract, plausible single-maintainer bus factor. Mitigating factors are real: small surface area, unusually thorough test and verification story for a first release, and a fallback path that is genuinely cheap — Pennant plus a ~200-line wrapper gets you assignment and counting, and the statistics are the only part worth not rewriting. Given the write path into flag data, identify an internal owner and consider vendoring/forking if adoption spreads beyond one team.

Scaling. Request-time cost is negligible (local hash, no I/O). The pressure points are storage and reads: one row per exposure and per conversion per scope, plus results-page aggregation. The release notes verify 4,000 participants in a real browser but say nothing about 4M. Get an answer on indexing and retention before running a test on high-traffic surfaces, and plan a pruning job or archival table from day one.

Failure modes to instrument.

• Missing @variant → zero exposures and zero conversions, looks like a dead experiment. Alert on "experiment defined, no exposures in 24h."
• Cookie blocked or cleared on the cache driver → guest re-assignment; DB driver mitigates stickiness but not identity.
• Mid-test weight changes → split shifts for new scopes only; existing participants stay. Aggregate results then mix assignment regimes. Add a change-log convention and freeze weights once a test starts.
• Winner rollout rollback → participants retain the rewritten value; confirm the revert story before you need it.
• Multi-goal testing without correction → inflated false positives; standardize on one primary goal plus secondary goals treated as exploratory.
• Bot exclusion is best-effort; server-side conversions can still be triggered by non-browser traffic.

Ramp-up. PHP engineers familiar with Pennant: half a day to first experiment, one to two days to production-ready. Analysts and PMs: the verdict labels are approachable, but "Keep running" vs "No significant difference" vs "Better" requires a short training session plus a written policy on planned sample size, primary goal, and who is allowed to stop a test. Total adoption cost for a two-engineer team is roughly one sprint end-to-end including privacy review, first experiment, and the stats reconciliation — which is the single most valuable step, because it converts "a community package says we won" into a number your organization will actually act on.