Getting Started

The Doctrine Lexer package provides a base AbstractLexer class to build custom tokenizers for parsing domain-specific languages (DSLs) — common in annotation systems (e.g. Doctrine Annotations), query languages (e.g. DQL), or configuration formats.

First step: Create a subclass of Doctrine\Common\Lexer\AbstractLexer and implement:

protected function getMatch(): string — the regex alternation used to match the next token (e.g. '[a-zA-Z_][a-zA-Z0-9_]*' for identifiers)
protected function getType(string $value): int|string|null — maps the matched string to a token type (integer constant, string, or enum case); return null to skip the token

Then instantiate the lexer with the input text and iterate with moveNext(), reading getValue() (matched string) and getToken() (token type).

use Doctrine\Common\Lexer\AbstractLexer;

class MyLexer extends AbstractLexer
{
    public const TOKEN_WORD   = 1;
    public const TOKEN_NUMBER = 2;

    protected function getMatch(): string
    {
        return '[a-zA-Z_\x80-\xff][a-zA-Z0-9_\x80-\xff]*|[0-9]+';
    }

    protected function getType(string $value): int|string|null
    {
        return ctype_digit($value) ? self::TOKEN_NUMBER : self::TOKEN_WORD;
    }
}

$lexer = new MyLexer('SELECT 42 FROM users');

while ($lexer->moveNext()) {
    // getValue() = raw matched string, getToken() = type from getType()
    printf("%-8s %s\n", $lexer->getValue(), var_export($lexer->getToken(), true));
}

Check src/AbstractLexer.php first — the useful helpers are moveNext(), getToken(), getValue(), lookahead-based peek() / isNextToken() / isNextTokenAny(), getPrevious(), getPosition(), reset(), and resetPosition().

Implementation Patterns

1. Feeding a Recursive Descent Parser

The lexer is a token producer; keep it as a private dependency of your parser and drive it with moveNext() + a match() helper.

class MyParser
{
    private MyLexer $lexer;

    public function parse(string $input): void
    {
        $this->lexer = new MyLexer($input);
        $this->lexer->moveNext();
        $this->statement();
    }

    private function statement(): void
    {
        if ($this->lexer->isNextToken(MyLexer::TOKEN_WORD)) {
            $this->match(MyLexer::TOKEN_WORD);
            // ... consume further tokens
        }
    }

    private function match(int|string $type): void
    {
        if ($this->lexer->getToken() !== $type) {
            throw new ParseError(sprintf(
                'Expected %s, got %s at offset %d',
                $type,
                var_export($this->lexer->getToken(), true),
                $this->lexer->getPosition()
            ));
        }

        $this->lexer->moveNext();
    }
}

peek() (and isNextToken()) require a lookahead pass, so call them before consuming — they answer "what comes after the current token", not "what is the current token".

2. Backtracking with `resetPosition()`

For grammar rules that need speculative parsing (optional clauses, ambiguity resolution), snapshot the character offset, try the rule, and rewind on failure. getPosition() returns the character offset of the current token, and resetPosition() rewinds to a given character offset.

$savepoint = $lexer->getPosition();

try {
    $this->parseOptionalOrderBy();
} catch (ParseError) {
    $lexer->resetPosition($savepoint); // 3.0.3: accepts a token character offset
    $lexer->moveNext();
}

This is the pattern the 3.0.3 patch targets — if you rely on rewind-based backtracking, pin ^3.0.3; earlier 3.0.x releases mishandled the offset passed to resetPosition().

3. Type-Safe Token Enums (v2.0+)

Return enum cases from getType() instead of magic ints — strict typing plus IDE autocomplete in both lexer and parser.

enum TokenType: string
{
    case IDENTIFIER = 'identifier';
    case INTEGER    = 'integer';
}

class MyLexer extends AbstractLexer
{
    protected function getMatch(): string
    {
        return '[a-zA-Z_][a-zA-Z0-9_]*|[0-9]+';
    }

    protected function getType(string $value): ?TokenType
    {
        return ctype_digit($value) ? TokenType::INTEGER : TokenType::IDENTIFIER;
    }
}

4. Skipping Whitespace and Comments

Return null from getType() for tokens that should not surface to the parser (whitespace, comments). The lexer still matches them, so offsets and getPosition() remain accurate — you just never see them in the token stream.

protected function getMatch(): string
{
    return '\s+|\/\/[^\n]*|[a-zA-Z_][a-zA-Z0-9_]*|[0-9]+';
}

protected function getType(string $value): int|string|null
{
    if (str_starts_with($value, '//') || trim($value) === '') {
        return null;
    }

    return ctype_digit($value) ? self::TOKEN_NUMBER : self::TOKEN_WORD;
}

5. Context-Aware Lexing via Instance State

Keep mode flags on the subclass when the same characters mean different things in different contexts (e.g. inside an annotation, inside a string, inside a block).

class AnnotationLexer extends AbstractLexer
{
    private bool $inAnnotation = false;

    protected function getType(string $value): int|string|null
    {
        if ($value === '@') {
            $this->inAnnotation = true;

            return self::TOKEN_AT;
        }

        if ($value === ')' && $this->inAnnotation) {
            $this->inAnnotation = false;
        }

        return $this->inAnnotation ? self::TOKEN_ANNOTATION : self::TOKEN_DEFAULT;
    }
}

Remember the state does not survive reset()/resetPosition() — reset it yourself if you reuse a lexer instance for a second pass.

Gotchas and Tips

⚠️ Version Compatibility

3.0.x requires PHP 8.1+ and drops the legacy BC layers; token types must be int|string|null or an enum case (arrays/objects are rejected).
3.0.3 is a patch: no API breaks, but it changes the semantics of resetPosition() so that it accepts a token character offset. Verify any custom backtracking against 3.0.3 specifically rather than "any 3.0.x".
getToken() returns the type and getValue() returns the string. Mixing them up is the most common source of Unexpected token errors in a hand-rolled parser.

⚠️ Whitespace Is Still a Token by Default

Nothing is skipped implicitly: the regex alternation decides what is matched, and getType() returning null decides what is discarded. If your getMatch() doesn't include \s+, whitespace simply won't be matched and lexing will stop early with a trailing position you should assert on in tests.

🛠️ Debugging Tips

Dump tokens by looping moveNext() and var_export([$lexer->getValue(), $lexer->getToken(), $lexer->getPosition()]) — this is faster than attaching a debugger and pins down exactly where a pattern diverges.
Assert getPosition() equals strlen($input) after the final moveNext(); an unreached tail almost always means a missing case in getMatch().
Wrap moveNext() in your own try/catch (or subclass) to add filename/line context, since the lexer only knows character offsets.
For multibyte input, add the u modifier to your pattern and use \p{L} / \p{N} character classes; offsets are then byte-based — keep that in mind when mapping offsets back to source substrings.

🛠️ Performance Optimization

Compile the regex once per class: keep getMatch() returning a constant string and, if the same pattern is reused across thousands of instances, memoize the composed pattern (or the resulting enum/type map).
Keep getType() allocation-free for hot tokens — a ctype_digit() / match chain beats regex checks inside the mapper.
The $value argument should be treated as read-only: recognise, don't mutate.

🔌 Extension Points

getMatch() — the single place to change the token alphabet; keep it ordered from most to least specific, since alternation is first-match-wins.
getType() — the natural place to add token classification, keyword promotion (e.g. SELECT → TOKEN_SELECT), and skip logic.
resetPosition() / getPosition() — the supported rewind surface for backtracking parsers; build your savepoint API on top of these rather than poking at private state.
Subclass state + overridden moveNext() — for pre-tokenisation hooks (e.g. normalising the input, tracking nesting depth) without duplicating the base scanning loop.

Lexer Laravel Package