Tokenizer Laravel Package

Getting Started

Install via Composer: composer require theseer/tokenizer. Start by instantiating TheSeer\Tokenizer\Tokenizer and passing PHP source code to tokenize:

use TheSeer\Tokenizer\Tokenizer;

$tokenizer = new Tokenizer();
$tokens = $tokenizer->tokenize('<?php echo "Hello";');
foreach ($tokens as $token) {
    echo sprintf("[%s] %s\n", $token->getName(), $token->getValue());
}

The first use case is often static inspection — e.g., counting function declarations or finding @deprecated docblocks. The tokenizer returns an iterable of Token objects with methods getName(), getValue(), getLine(), and getType().

Implementation Patterns

• Stream-based processing: Use TokenStream to group tokens (e.g., class/method blocks) with TokenStream::filter(), TokenStream::nextUntil(), or custom iterators.
• Replaying token sequences: For refactoring, reconstruct code using TokenStream::serialize() or manually emit tokens back to code.
• Context-aware filtering: Combine with TokenFilter to skip whitespace/comments via Tokenizer::removeWhitespace() for cleaner analysis.
• Integration in tools: Often used in custom sniffers (e.g., alongside PHPCS), code quality metrics, or AST-less analyzers where speed matters more than full parse fidelity.
• Version-safe parsing: Leverage the library’s handling of token name differences across PHP versions (e.g., T_INLINE_HTML, T_FN).

Gotchas and Tips

• No position tracking: Unlike full ASTs, this provides no column offset — only line numbers. Use Token::getLine() for debugging but avoid fine-grained edits.
• Whitespace removal: By default, whitespace is included. Use $tokenizer->removeWhitespace() if you need pure semantic tokens — but be aware this may change token positions if you later reconstruct code.
• Open tag expectations: The tokenizer expects <?php — omitting it may cause unexpected T_INLINE_HTML tokens. Always normalize input.
• Token naming quirk: getName() returns PHP userland token names (e.g., "T_STRING"), not human-friendly labels — you may need to map these.
• Extensibility: Extend Token or Tokenizer to add custom token types if building domain-specific tooling (e.g., custom annotations), but maintain compatibility by not overriding TokenStream logic lightly.
• Debugging tip: Use Token::export() or dump the token stream to a file for visual inspection during development.