leovie/php-token-normalize
A PHP library for normalizing PHP tokens, making it easier to compare, analyze, or process code consistently. Useful for static analysis, duplicate detection, obfuscation checks, and tooling that needs a stable token representation.
Install and inspect the surface area first.
composer require leovie/php-token-normalize:^3.0
The package is tiny, so the cheapest orientation is:
README.md — the usage example is usually the whole API.src/ — expect a facade/aggregator class (e.g. a *Normalizer) plus one small class per rule (whitespace, comments, variables). The aggregator is the entry point; the individual normalizers are the extension points.tests/ — the fixtures double as a spec for what "normalized" means (which tokens are dropped, merged, or re‑emitted). Check the variable fixture specifically: as of v3.0.0 every variable token is rewritten to $x.First real use case: "are these two files the same code?" Normalize both token streams and compare hashes. This is what the package is effectively for — ignoring formatting, comments, and (since v3.0.0) variable naming so that copy‑pasted, reformatted, and mechanically renamed code collapses to the same key.
use LeoVie\PhpTokenNormalize\Normalizer\TokenNormalizer; // confirm exact class in src/
$normalizer = new TokenNormalizer();
function fingerprint(string $code, TokenNormalizer $n): string
{
$tokens = $n->normalize(token_get_all($code));
// \x1f is a safe separator: it can't appear in a token's text.
$canonical = implode("\x1f", array_map(
fn($t) => is_array($t) ? $t[0] . ':' . $t[1] : $t,
$tokens
));
return hash('xxh128', $canonical);
}
$same = fingerprint($original, $normalizer) === fingerprint($reformatted, $normalizer);
If your installed version returns a normalized string instead of a token array, drop the array_map/implode and hash the result directly.
1. Duplicate / plagiarism detection in CI.
Walk the repo with Symfony Finder, fingerprint each file, group by hash. Exact duplicates fall out immediately. v3.0.0 materially improves this: because variables normalize to a single placeholder, copy‑pastes where someone renamed $order → $invoice (or $items → $rows) now hash identically without any extra work. For near duplicates, fingerprint sliding windows of N tokens (e.g. 50–100) rather than whole files — whole‑file hashes only catch verbatim copies, and any one‑line edit changes the whole hash.
$tokens = $normalizer->normalize(token_get_all($code));
$windows = [];
for ($i = 0; $i + 80 <= count($tokens); $i += 20) {
$windows[] = hash('xxh128', serialize(array_slice($tokens, $i, 80)));
}
2. "Did the logic change?" gates for refactors. This is the pattern v3.0.0 unlocks. Fingerprint the file before and after a pure rename refactor and assert equality — it's now a cheap, high‑signal guard for Rector/PHPStan auto‑fix passes and IDE rename operations. Store the pre‑refactor fingerprint in the test fixture, not in a DB, so it travels with the code.
3. Cache keys for generated/templated PHP. If you compile templates or generate classes, fingerprint the generated source to decide whether the on‑disk artifact is stale. This is more robust than mtime and survives whitespace‑only changes. Note that with v3 variable collapsing, changing a template that only alters variable names will not invalidate the cache — usually what you want, occasionally not (see Gotchas).
4. Snapshot tests for a code generator. Assert the fingerprint of the fixture against the fingerprint of the generator output. It keeps the test green when you only reformat output or rename locals, and red when semantics change. Pair it with a normal textual diff for debuggability.
5. Storage pattern for a scan.
Persist two columns: hash (indexed, char(32)/binary(16)) and tokens (JSON‑encoded normalized tokens). Look up candidates by hash, then confirm with a token‑level diff so hash collisions are never treated as proof. Add a third column for the normalizer version (see Gotchas) so you can re‑scan selectively after upgrading.
6. Composing rules instead of taking the whole normalizer. If you don't want to erase docblocks (annotations, route attributes, OpenAPI) or you do need real variable names preserved for a downstream tool, use the individual normalizers so only whitespace/comments are collapsed:
// illustrative: chain just what you need, skipping the variable normalizer
$tokens = $whitespaceNormalizer->normalize(token_get_all($code));
$tokens = $commentNormalizer->normalize($tokens);
7. Extension. Add your own rule as a decorator around the package's normalizer — normalize identifier case, sort use statements, strip trailing ?> — then feed the result into the aggregator. Normalizers are small enough that a decorator is usually clearer than forking. If you need "variables distinct but consistently numbered" (v2 behaviour), that decorator is now yours to write: walk the normalized output and re‑expand $x into $v0, $v1, … by first‑appearance order.
8. Batch jobs. For repo‑wide scans in a queue worker/Octane, process one file per iteration and unset($tokens) — token arrays are memory‑heavy for large files.
$x. This is not a bug fix, it's an information‑erasing change. Consequences you must plan for:
$a = $b; $b = $a; and $x = $x; $x = $x; normalize to the same token stream. Also $first = $row['a']; $second = $row['b']; vs both reading ['a']. Fine for duplicate detection, dangerous if you were using fingerprints to detect behavioural changes in code that differs only in which variable is used where.$this and object variables alike, $this->save() and $service->save() look identical, and $a->b vs $a->c still differ (property names are T_STRING, not variables) — so you get a lopsided fingerprint. Verify against src/ whether $this, $GLOBALS, and superglobals ($_POST, $_SERVER) are rewritten; rewriting $this is usually undesirable.token_get_all() gained token IDs across 8.x (T_NAME_QUALIFIED/T_NAME_FULLY_QUALIFIED in 8.0, T_READONLY and T_ENUM in 8.1, and more since). Newer token types may pass through untouched or trip a match/switch that never saw them. Pin the PHP version in CI and regenerate stored hashes whenever you bump PHP.implode('', $tokens). Without a separator, adjacent tokens fuse and different code collides ($ab vs $a + $b, or ab as one T_STRING vs two). Use a separator or serialize the array.a . b vs ab, or yield from vs yieldfrom. Sanity‑check with a few adversarial fixtures before trusting results on numeric/string adjacency.#[Attribute] tokens are not comments — verify your normalizer keeps them. Note #[Attribute] arguments can contain variables, which now become $x."{$name}") may or may not be caught by the v3 variable rule depending on whether the normalizer walks T_STRING_VARNAME/T_VARIABLE inside string parts. Compare a before/after dump on a heredoc containing {$var} — this is the classic place normalization surprises you.T_INLINE_HTML, T_OPEN_TAG, and declare handling vary; a .blade.php‑style file will produce noisy tokens. Normalize whole files, not mixed fragments, or strip PHP blocks first.PhpToken::tokenize($code) for readable token names, var_export the array before and after normalize(), and diff. Most "wrong fingerprint" reports are either (a) a token ID the package didn't recognize, or (b) since v3, an expectation that variable names survive normalization.src/ rather than copying snippets blindly; wrap the package behind your own one‑method interface so a future fork or replacement is a single‑file change. That wrapper is also where you'd pin the normalizer version into the fingerprint input.How can I help you explore Laravel packages today?