Php Token Normalize Laravel Package

Product Decisions This Supports

• Build vs. buy for code-quality tooling — still yes, but the scope of what you get just narrowed. Normalizing PHP token streams remains the prerequisite for any feature that compares code semantically rather than textually: clone detection, duplicate-code (DRY) CI gates, semantic diffing, and code-search ranking. The normalization layer is still 1–3 weeks of work plus edge cases you would otherwise own.
• v3.0.0 changes what "equivalent" means — this is a product decision, not just a version bump. The normalizer no longer enumerates variables ($a → $var1, $b → $var2, preserving identity); it now collapses every variable to $x. Concretely, $x + $x and $x + $y become identical token streams. That is a deliberate loosening of the equivalence relation, and it moves the feature you are building:
  • Duplicate-code CI gate: higher recall, materially higher false-positive rate. Code that differs only in whether it reuses one variable or uses two now reads as a clone. Expect noisier reports and more developer pushback; budget for threshold tuning or a strict/strict-ish mode.
  • "Did this refactor change behavior?" checks: this is the riskier direction. Collapsing variables makes behavior-changing edits look unchanged — a false negative on exactly the check you built. $a = $b; $c = $a; and $a = $b; $c = $b; now normalize the same. If this gate is supposed to catch real regressions, v3.0.0 weakens it.
  • Plagiarism / similarity scoring: generally acceptable, arguably desirable — it defeats variable renaming as an obfuscation strategy. Similarity scores will inflate versus v2; recalibrate any thresholds before shipping a user-facing score.
• Roadmap: migration and modernization assistants. Still supported, but pin the version and state the normalization contract in the feature spec. A normalization change in a minor-adjacent release would silently move your diff results; you need snapshot tests on the normalizer output, not just on the features above it.
• Maturity signal is now mixed, not negative. The 2022-era "no releases" concern is stale: the project has shipped through v2.0.11 to v3.0.0, and it uses a major version bump for a behavior-breaking change — that is correct semver discipline and a point in its favor. What we still cannot verify from this release alone is stars, dependents, or a security-response process. The prior recommendation therefore holds with an amendment: vendor it, own it, and pin to the exact commit you vetted.
• Build vs. buy for a static-analysis pipeline. Unchanged: if the pipeline is syntax-aware rather than type-aware, normalized tokens remain cheaper than a full AST and sufficient for many heuristics.

When to Consider This Package

Consider it when:

• The requirement is specifically token-level PHP normalization — whitespace, comments, string quoting, and equivalent-syntax collapsing — rather than type inference or dataflow analysis.
• You need to answer "are these two PHP snippets equivalent modulo formatting and variable naming?" cheaply. That second clause is now load-bearing: v3.0.0 gives you variable-agnostic comparison only, with no distinct-variable mode.
• Your feature tolerates the false positives that come from collapsing variable identity, or you are prepared to patch the normalizer to restore enumeration. The change is small and self-contained (it was a single PR), which is a point in favor of vendoring rather than a reason to fork the whole concept.
• You are comfortable treating the code as a vendored dependency you maintain, not an upstream you rely on. BSD-3-Clause makes that legally straightforward.
• The team's PHP version floor matches a 2022-era normalization baseline — re-validate against PHP 8.2/8.3 tokenizer behavior before committing, as before.

Look elsewhere when:

• You need semantic/type-aware analysis — nikic/php-parser, PHPStan, Psalm. Token normalization is the wrong abstraction for name resolution and type checking.
• You need formatting or auto-fixing — php-cs-fixer or PHP_CodeSniffer own that end to end.
• You need variable identity preserved (e.g., a behavior-change gate, or copy-paste detection where "used twice vs. two different names" is a meaningful difference). Out of the box in v3.0.0, it is not. Either patch it or write the normalizer yourself — do not silently accept the coarser relation.
• You need an actively maintained dependency with a documented security response process. Release activity is better than we previously assumed, but that is not the same as a response process; do not put it on a critical path you cannot patch yourself.
• Your use case is a one-off script — a small internal function over PHP's native token_get_all() may be simpler than adopting any dependency.

How to Pitch It (Stakeholders)

For executives (one paragraph): This is a small, permissively licensed (BSD-3) utility that does one narrow job: normalizing PHP token sequences so two pieces of code can be compared for semantic rather than textual equivalence. It remains a cheap enabler for code-health features on the roadmap — duplicate-code detection in CI, semantic diffing for migration work, similarity scoring. Two things changed with the new 3.0 release. First, the maintenance picture is better than we thought: the project has continued shipping releases and versioned a behavior change correctly as a major version. Second, that change made the comparison deliberately looser — it now treats every variable as interchangeable, which will make duplicate-code detection noisier and, more importantly, makes a "did this refactor change behavior?" check less reliable. The recommendation is unchanged in shape and firmer in reason: do not adopt as a live dependency, vendor the source, pin the exact version we review, add our own tests, and own it. If we need the previous, stricter behavior, we patch it — the change is small.

For engineering (the ask):

1. Re-run the spike against v3.0.0 specifically (1–2 days). The behavior table produced last time is now out of date. Document the new rule — all variables collapse to $x — and mark every use case where variable identity mattered as a behavioral regression versus what we specified.
2. Decide, per use case, strict vs. loose. Duplicate-code detection probably tolerates (or wants) the loose relation with tuned thresholds; the migration/behavior-change check probably does not. If it needs identity preserved, restore enumeration as a local patch — it was a single upstream PR, so the diff is small and reviewable.
3. Compatibility check: run against our supported PHP versions (especially 8.2+) and add fixtures for tokenizer changes since the 2022 baseline.
4. Vendor it, pinned: copy into an internal namespace under BSD-3 with attribution, record the upstream commit hash, add golden/snapshot tests over normalizer output so a future upstream change can never silently alter our results, and name a maintainer.
5. Fallback: if the spike shows the rules are a poor fit even with a patch, write a thin internal wrapper over token_get_all() — a few days — rather than seeking another third-party package, since we would be maintaining this either way.

Bottom line: still useful, still ours to own — and the new release changes the equivalence semantics enough that our feature specs and thresholds must be revisited before we ship anything on top of it.