phug/lexer
A fast, extensible PHP lexer for Pug (Jade) templates and the Phug ecosystem. It converts template source into structured tokens for parsing, supporting indentation, tags, attributes, interpolation, code, and comments with configurable scanners.
phug/lexer is the tokenizer of the Phug stack: it turns Pug/Jade source text into a TokenStream of typed tokens (TagToken, TextToken, IndentToken, YieldToken, ExpressionToken, …). You consume it directly only when you need tokens — outline extraction, linting, formatting, syntax highlighting, source maps — or when you are extending Pug's syntax. For rendering, go through phug/parser / phug/renderer; the lexer sits underneath them.
composer require phug/lexer
use Phug\Lexer;
$lexer = new Lexer(); // options optional
$tokens = $lexer->lex("doctype html\nh1.title Hello\n p World\n", 'demo.pug');
foreach ($tokens as $token) {
printf(
"%-16s lvl=%d line=%d value=%s\n",
$token->getName(),
$token->getLevel(),
$token->getLine(),
json_encode($token->getValue())
);
}
// doctype lvl=0 line=1 value="html"
// newline lvl=0 line=1 value="\n"
// tag lvl=0 line=2 value="h1"
// attribute_start lvl=0 line=2 value="."
// ...
Where to look first, in this order:
src/Lexer.php — a thin orchestrator; lex() is short and tells you the whole story.src/Scanner/ — one class per Pug construct (TagScanner, TextScanner, IndentScanner, CommentScanner, EachScanner, …). Alphabetical scanning; this directory is the language reference.src/Token/ — the token shapes you will instanceof against. Use instanceof TagToken, not string comparison; getName() gives the short name (tag, text, indent, outdent, newline, doctype, expression, …). If getName() is absent on your version, fall back to (new ReflectionClass($token))->getShortName().src/State.php — the mutable indentation cursor shared across scanners.First use case — a template outline / structure dump. Walk the stream, keep only TagTokens, and print str_repeat(' ', $token->getLevel()) . $token->getValue(). That single exercise forces you to meet the three things that matter: indentation levels, the indent/outdent/newline noise tokens, and the difference between getValue() and getSource().
Token-shape assertions over strings. Always branch on token classes:
use Phug\Lexer\Token\TagToken;
use Phug\Lexer\Token\TextToken;
use Phug\Lexer\Token\YieldToken;
foreach ($lexer->lex($source) as $token) {
if ($token->isVisible() && ($token instanceof TagToken || $token instanceof TextToken)) {
$out .= $token->getValue();
}
}
isVisible() filters out structural tokens (indent, outdent, newline) so you don't have to whitelist them by hand — but if your tool genuinely cares about layout, ignore isVisible() and use the level field instead.
Two modes: semantic value vs. raw source. getValue() gives the cooked content (tag name without attributes, text without surrounding syntax). getSource() plus getOffset() gives you the exact byte slice from the original template. For formatters and linters, getSource()/getOffset()/getLine() are the only safe way to build edit ranges. getLine() is 1-based; getOffset() is a 0-based byte offset — be careful with multibyte templates when you translate offsets into mb_substr positions.
Nested token sets. Interpolations (#{...}), attribute values, and inline code produce tokens that carry their own child tokens (getTokens() / getChildren(), depending on version). Recursively walk them; a flat foreach over the top-level stream will silently miss everything inside class="a #{b} c".
Registering a custom scanner instead of pre-processing the string. Both approaches work; the scanner is better because it composes with the rest of Pug (nesting, indentation, line numbers stay correct). Extension points:
use Phug\Lexer;
use Phug\Lexer\State;
use Phug\Lexer\ScannerInterface;
use Phug\Lexer\TokenInterface;
class TodoScanner implements ScannerInterface
{
// Signature in current versions: scan(State $state, TokenInterface $token),
// returning a TokenInterface (or null to fall through). Check the installed
// interface first — it is a single small method and stable-ish, but has
// changed names between majors.
public function scan(State $state, TokenInterface $token)
{
// inspect $state->getSource() / $state->getLevel(), advance, return token
}
}
$lexer = new Lexer([
'scanners' => [My\TodoScanner::class],
]);
Prefer a token handler over a scanner when you only want to post-process tokens the existing scanners already produce (rewrite a TextToken, attach metadata, split a token). Scanners change what is recognized; handlers change what happens to already-recognized tokens. php_token_handlers is the same idea keyed by token class. Reaching for a scanner when a handler would do is the most common over-engineering in this package.
Integrate at the right layer. If you need the tree, use phug/parser ((new Parser())->parse($source)) — it consumes the lexer and saves you reimplementing nesting. Drop to the lexer only for stream-shaped tools: dependency extraction from include:/extends, lint rules, keyword highlighting, "find all mixin calls", diff-friendly formatting.
Snapshot testing. Serialize a compact tuple per token and assert on the JSON:
$snapshot = array_map(fn ($t) => [
$t->getName(), $t->getValue(), $t->getLevel(), $t->getLine(), $t->isVisible(),
], iterator_to_array($lexer->lex($template, 'fixture.pug'), false));
This is far more maintainable than string-comparing a var_dump, and it survives renames of internal scanner classes.
Reuse the State for style policy. After lex(), $lexer->getState() still exposes the indentation statistics (getIndentStyle(), getIndentWidth(), getLevel(), isIndentConsistent()). That is a free "tabs-vs-spaces / indent-width" linter without writing a single scanner.
indent and outdent tokens are synthesized, not read from the source, and any outdents still open at EOF are emitted at the end. Token count never equals line count. Never infer nesting from getLevel() deltas alone — track the indent/outdent stream.State tracks indent style and consistency; inconsistent indentation is where "my template works but the lexer output looks wrong" reports come from. Normalize on input or validate with isIndentConsistent().|), block text (. after a tag), escaped #{}, and inline HTML all fragment differently. Always test on a real template from the project, not a synthetic one-liner.setOption() after construction may not rebuild already-decided scanners — construct with options, or set them before the first lex() call. When used through phug/compiler, the same keys arrive nested under 'lexer' => [...]; don't be surprised that the flat standalone form and the nested compiler form look different.lex($source, 'views/card.pug') populates getPath() on every token, which is what error messages and source maps key off of. Omitting it makes downstream errors untraceable.getTokens() exposes it), but don't write code that assumes it. If you need two passes, iterator_to_array($stream, false) once and keep the array.phug/parser pulls it in), so breakage will not be caught by the wider ecosystem — it will be caught by you. Pin the exact minor version and pin phug/token alongside it; tokens were extracted into that package in the 2.x line, and the token/stream APIs have churned across majors (generator-based → array-backed streams). It is MIT-licensed and still shipping releases, so vendoring/forking is a realistic escape hatch.[getName(), getValue(), getLevel(), getLine(), isVisible()] per token first, then read the specific scanner. Nine times out of ten the surprise is an outdent you didn't expect or a text token you assumed was a tag attribute.TextTokens is ten lines and survives upgrades; a subclass of TagScanner is a merge conflict waiting to happen.count($tokens) in tooling before assuming a parse failure.How can I help you explore Laravel packages today?