Tokenizer Laravel Package

Getting Started

Start by installing the package via Composer (composer require spiral/tokenizer). Despite its low star count and outdated last release (2019), it remains useful in legacy Spiral Framework projects for static analysis. Its primary use case is scanning codebase files to locate classes implementing a given interface, extending a parent class, or using a specific trait—without executing PHP code. Begin by instantiating the Tokenizer class, providing the root directory to scan, then call getClassesByInterface(), getClassesByParent(), or getClassesByTrait() with the fully qualified name.

Implementation Patterns

The package is typically used in console commands or build tools for code introspection. Common patterns include:

• Dependency Registration: Dynamically register services by scanning for classes implementing a common interface (e.g., CommandInterface).
• Code Quality Checks: Build custom analyzers to enforce architectural rules (e.g., ensure all entities extend a base model).
• Metadata Harvesting: Extract annotations or attributes across many classes at build time for caching or documentation generation. Workflows often integrate with Symfony Console, running as a CLI tool. Use TokenizedFile::getTokens() for lower-level token access when custom logic is needed beyond the provided filters.

Gotchas and Tips

• PHP Version Constraints: Supports PHP 7.2–7.4; incompatible with PHP 8+ due to reliance on deprecated token_get_all() parsing (no AST). Expect issues with modern syntax like union types or attributes.
• Performance: Scans are file-system heavy—cache results if reused. Consider limiting scope via Tokenizer::$excludedDirectories.
• Incomplete Parsing: May miss classes in eval() or dynamically generated code; strictly static analysis only.
• Extension Points: Extend Tokenizer or TokenizedFile to customize scanning rules or add class filters (e.g., skip tests).
• Alternatives: For new projects, consider nikic/php-parser for robust AST-based analysis or Spiral’s newer spiral/annotated in modern frameworks. Given the age of this package, treat it as legacy tooling—great for maintaining older Spiral apps, but avoid in greenfield development.