Laravel Pennant Experiments Laravel Package

Product Decisions This Supports

• Whether to ship a change at all, and to whom. This turns "should we launch feature X?" into a measured question: define a control and challenger, run it, and get a verdict of Better / Worse / No significant difference instead of an opinion. The verdicts are the product decision, not the dashboard.
• Roadmap sequencing under uncertainty. The planned-sample-size rule (set explicitly, or derived from a minimum detectable effect) forces the PM to state up front how big a lift matters. That is a roadmap conversation: if a 1% lift isn't worth the engineering, the test isn't worth running.
• Kill criteria for in-flight work. "Too early / Keep running" prevents the most expensive PM error in experimentation — calling a winner on day two. Bonferroni correction across multiple challengers means a three-variant test doesn't quietly inflate false positives.
• Rollout mechanics as one tool, not two. Winner rollout rewrites stored Pennant values so new and existing visitors get the winner, and recording stops. That makes "we have a winner" and "we have shipped the winner" the same action, which removes the usual gap where a winning variant is measured but never actually becomes the default.
• Guest-first funnels. The guest → user handoff (exposures, conversions and the variant move to the user, counted once, same variant) supports the decision most consumer products actually care about: testing on logged-out traffic — landing pages, pricing pages, signup flows — without poisoning the data when the visitor registers.
• Build vs buy. This is the "buy-ish but free, self-hosted" middle path. Against SaaS flags/experimentation platforms you trade vendor support, warehouse-native analysis, and non-engineer tooling for zero marginal cost, no data leaving your app, and no SDK surface in the front end beyond @experimentsScript.
• Instrumentation ownership. Goals can be fired from PHP (Experiments::convert()) or JS (data-experiment-convert), so the PM can choose whether conversion definitions live in server code (harder to tamper with) or in templates (faster to iterate). The JS endpoints resolving the participant from session/cookie and never the request body is a product-trust decision as much as a security one.

When to Consider This Package

Consider it when most of the following hold:

• You are on Laravel 12/13 with Pennant already in the stack (or willing to add it). This package is a layer on Pennant, not a replacement; if you don't already use feature flags, you're adopting two things.
• Your metrics are binary conversions — signup completed, upgrade clicked, invite sent. That is the supported shape today. Revenue, session time, and other continuous metrics are explicitly out of scope in v0.1.0.
• You have engineering capacity to own a v0.1.0 community package. It is MIT-licensed, unofficial, single-maintainer-style, and has no long-term support commitment. Budget for reading the code, pinning versions, and being your own support desk.
• Your experimentation volume is modest and deliberate — a handful of tests a quarter where statistical correctness matters more than throughput. The minimum-data rules and planned sample sizes will feel slow to a team running twenty tests a month.
• You need self-hosting for data-residency or compliance reasons, or you're avoiding per-seat SaaS pricing on low-traffic apps.
• You want the optional Pennant dashboard tab and already use amjadiqbal/laravel-pennant-dashboard v1.1+, or you're fine with the artisan commands and results pages alone.

Look elsewhere when:

• You need continuous or revenue metrics, per-scope monetary impact, or arbitrary aggregation. Nothing here supports it yet.
• Identity resolution across devices is core to your funnel. The stated limitation — a guest signing in on a second device keeps that device's own visitor exposure as a separate participant — will silently split a real user into two participants. For a B2C product with heavy multi-device traffic, that is a data-quality problem you'd be accepting on day one.
• You want non-engineers to launch and read tests. There is no PM-facing authoring UI; definitions are code and results are artisan/Blade.
• You need vendored support, SLAs, SOC 2, or an audit trail for who changed a test. The managePennantExperiments gate is local-only by default and is not a governance story.
• You want bandits, holdouts, mutually exclusive experiment layers, or warehouse-native analysis. None of these are present; the deliberate frequentist/Bayesian reporting is aimed at fixed-horizon tests.
• Your team will change weights mid-test and expect the whole population to re-split. Weights only affect new scopes; existing participants stay put. That's the correct behavior, but teams expecting a live re-randomization will misread it.

How to Pitch It (Stakeholders)

To executives: This is a free, MIT-licensed Laravel package that lets us run proper A/B and multivariate tests — control versus challengers, with real statistics — without buying an experimentation platform. It reports a verdict rather than a chart: Better, Worse, No significant difference, or Keep running. It protects us from the classic mistake of declaring a winner too early, and it includes the guardrails (minimum sample sizes, correction for running several variants at once) that make the numbers defensible. Cost is engineering time, not licence fees, and no customer data leaves our infrastructure. The main caveat is maturity: it's a v0.1.0 community release, so we adopt it as a tool we can read and maintain, not a vendor we can escalate to.

To engineering: It sits directly on Laravel Pennant — an experiment is a Pennant feature whose value is the variant, so there's no parallel flag system to reconcile. Assignment is sticky and hash-based across users, teams, arbitrary scopes and guests (random id in a long-lived http-only cookie), with the first answer also persisted under the database driver. Exposure tracking is explicit: variant() never counts, only @variant or Experiments::expose(), which means you can resolve a variant for internal logic without polluting results. Conversions fire from PHP or JS, are once-per-scope-per-goal, and only count for exposed scopes; the JS endpoints read the participant from session or cookie and never from the request body. Bot exclusion, eligibility rules, and viewPennantExperiments / managePennantExperiments gates (local-only by default) are included, plus a read-only mode. Statistics are a two-proportion z-test with a difference interval, exact Beta probability to beat control, Wilson intervals, Bonferroni correction, and a minimum-data rule — and they're unit-tested against textbook values, so you're not asked to trust an unverified formula. It's tested across PHP 8.3–8.5 and Laravel 12–13, including a lowest-deps run against Pennant v1.18.0, and validated end-to-end in a real browser against a 4,000-visitor simulation. Known gaps to design around: binary metrics only (no revenue or continuous measures), guest-to-user handoff does not span devices, and weight changes mid-test apply to new scopes only.

To the wider business: The pitch isn't "we can test things" — it's "our test results will be trustworthy, and a winner will actually get shipped." The winner-rollout command rewrites stored values and stops recording, so the moment we call a result we can act on it. That closes the loop most teams leave open.