Lexer Contracts Laravel Package

Getting Started

You almost never install this directly. phplrt/lexer-contracts is a read-only subtree split of the phplrt/phplrt monorepo. It arrives transitively when you composer require phplrt/phplrt (or a component like phplrt/lexer or phplrt/parser). Install it standalone only when you are writing a library that consumes tokens and you deliberately want a dependency tree free of the full parser.

composer require phplrt/lexer-contracts   # consumers only
composer show phplrt/lexer-contracts      # check what you actually got

Where to look first: vendor/phplrt/lexer-contracts/src/. It is a handful of interfaces and nothing else — no traits, no base classes, no runtime dependencies. Read all of them in one sitting; that is the entire surface area you are programming against.

The two you will touch daily:

• Phplrt\Contracts\Lexer\LexerInterface — lex(SourceInterface $source): iterable, yielding TokenInterface instances.
• Phplrt\Contracts\Lexer\TokenInterface — getName(), getValue(), getOffset(), getLength() (plus channel accessors in recent versions).

The SourceInterface itself lives in the sibling phplrt/source-contracts package.

First use case — a token dumper. This is the fastest way to build intuition for how a lexer behaves, and it is genuinely useful in production as a debug endpoint or CLI command:

use Phplrt\Contracts\Lexer\LexerInterface;
use Phplrt\Contracts\Source\SourceInterface;

final class TokenDumper
{
    public function __construct(private readonly LexerInterface $lexer) {}

    public function dump(SourceInterface $source): void
    {
        foreach ($this->lexer->lex($source) as $token) {
            printf(
                "%-14s %-24s @%d+%d\n",
                $token->getName(),
                var_export($token->getValue(), true),
                $token->getOffset(),
                $token->getLength(),
            );
        }
    }
}

Wire it up with any concrete lexer (phplrt's Lexer with a GrammarInterface, or a hand-rolled RegexLexer/StatefulLexer) and you have a working inspection tool in twenty lines.

Implementation Patterns

Constructor-inject the interface, never the concrete class. This is the whole reason the split package exists. Your domain services should ask for LexerInterface; the composition root (a Laravel service provider, a small container binding) decides whether that is phplrt's grammar-driven lexer, a RegexLexer, or a test fake.

// AppServiceProvider
$this->app->bind(LexerInterface::class, fn () => new Lexer($grammar));

Decorators are the workhorse pattern. Because the contract is a single method returning iterable, wrapping is trivial and composable:

final class FilteringLexer implements LexerInterface
{
    public function __construct(private readonly LexerInterface $inner) {}

    public function lex(SourceInterface $source): iterable
    {
        foreach ($this->inner->lex($source) as $token) {
            if ($token->getName() === 'T_WHITESPACE') {
                continue;
            }
            yield $token;
        }
    }
}

Use this for comment stripping, whitespace elision, EOF token injection, offset rewriting when lexing an embedded fragment, or logging token counts per request. Chain several decorators; each stays small and testable.

Stream, don't collect. lex() is typed iterable and implementations return generators. Consume lazily and break early when you only need the first few tokens (e.g. sniffing a file header or detecting a shebang). Memory stays flat on large inputs — relevant if you lex uploaded files.

Test doubles without a grammar. Implement TokenInterface as a tiny readonly value object, then hand your code a lexer stub:

final class FakeLexer implements LexerInterface
{
    public function __construct(private readonly array $tokens) {}

    public function lex(SourceInterface $source): iterable
    {
        return $this->tokens; // arrays satisfy `iterable`
    }
}

This decouples your token-processing unit tests from grammar files entirely — no fixtures to maintain, no parser bootstrap in setUp().

Boundary discipline. Depend on the contracts in your domain layer, and require phplrt/lexer-contracts (not phplrt/phplrt) from library packages. If you only ever read tokens, you should not be pulling the parser into a downstream consumer's vendor directory.

Buffer when order matters more than size. If you need lookahead or count()-style access, wrap the iterable into an array-backed token stream of your own rather than assuming the contract offers it. The contract deliberately does not promise random access.

Gotchas and Tips

Version skew is the top footgun. This package is auto-generated by a subtree split and its tags track the monorepo. If you pin it independently you can resolve a version whose interface signatures disagree with the phplrt/phplrt you have installed — typically surfacing as a fatal "declaration must be compatible" error or a missing getChannel(). Let the monorepo pull it transitively, or constrain it with a broad ^3 range that matches your phplrt version.

It is read-only. Do not PR here. The description says so explicitly. File issues and patches at phplrt/phplrt; changes to this repo are overwritten by the split workflow. If you need a signature changed, you are changing the monorepo.

iterable means "not an array". Do not count() it, index it, or foreach twice. Generators throw Cannot traverse an already closed generator on a second pass. If your code needs two passes, materialise once into an array or a stream object.

Offsets are byte offsets. getOffset() and getLength() count bytes, not characters or grapheme clusters. Any multibyte source — UTF-8 identifiers, emoji in strings — will make naive substr() splicing wrong. Use mb_* / grapheme_* functions and recompute if you map back to character positions.

Token names are strings owned by the implementation. The T_* constants and the vocabulary (T_WHITESPACE, T_COMMENT, EOF) live in the concrete lexer or grammar, not in this contract. Never hardcode a name in a library that accepts an arbitrary LexerInterface — make the skip-list configurable.

Whitespace semantics are undefined by the contract. Some lexers emit whitespace tokens, some hide them on a separate channel, some skip them entirely. Do not assume either behaviour; if your consumer cares, normalise in a decorator.

Catch the contract exception, not \Throwable. Implementations throw exceptions implementing Phplrt\Contracts\Exception\ExceptionInterface (sibling contracts package). Catching that interface lets you separate "the input is malformed" from "my code has a bug" — and lets you swap lexers without rewriting error handling.

Mocking generators in PHPUnit: ->willReturn([$token]) works fine because arrays satisfy iterable. Reach for willReturnCallback(fn () => yield $token) only when you are specifically testing lazy-consumption behaviour.

Extending the contract: the interface is intentionally minimal. If you need extra capabilities, define your own sub-interface and an adapter that wraps any LexerInterface — concrete implementations will not implement your interface, so type-hint the narrow contract at the edges and adapt at the boundary.

Debugging checklist when tokens look wrong: dump getName()/getValue()/getOffset() triples; check the grammar's token definition order (longest-match vs first-match rules); confirm whether your source object normalises line endings or BOMs before lex() is called — those shift every offset downstream.