Php Token Stream Laravel Package

Getting Started

Minimal Steps

1. Installation:

   composer require --dev phpunit/php-token-stream
   

   Add to composer.json under require-dev if only needed for testing/analysis.

2. First Use Case: Parse a PHP file to detect specific syntax patterns (e.g., echo statements):

   use PHPUnit\Util\Token\TokenStream;
   
   $code = '<?php echo "Hello"; ?>';
   $tokens = TokenStream::createFromCode($code);
   
   foreach ($tokens as $token) {
       if ($token->getId() === T_ECHO) {
           echo "Found an echo statement!";
       }
   }
   

3. Where to Look First:

   • Token IDs: Reference PHP’s token constants (e.g., T_FUNCTION, T_CLASS).
   • TokenStream API: Focus on createFromCode(), createFromFile(), and iteration methods.
   • PHPUnit Tests: The package’s test suite demonstrates edge cases (e.g., heredoc, dynamic code).

Implementation Patterns

Usage Patterns

1. Token Stream Iteration: Process tokens sequentially to avoid memory overload:

   $tokens = TokenStream::createFromFile('src/Example.php');
   foreach ($tokens as $token) {
       if ($token->isGiven(T_VARIABLE)) {
           // Handle variable usage
       }
   }
   

2. Context-Aware Parsing: Track state (e.g., inside a class/method) to enforce rules:

   $tokens = TokenStream::createFromCode($code);
   $inClass = false;
   foreach ($tokens as $token) {
       if ($token->isGiven(T_CLASS)) {
           $inClass = true;
       } elseif ($token->isGiven(T_FUNCTION) && $inClass) {
           // Parse method body
       }
   }
   

3. Token Manipulation: Rebuild code by collecting and rewriting tokens (e.g., for refactoring):

   $tokens = TokenStream::createFromCode($code);
   $rewrittenTokens = [];
   foreach ($tokens as $token) {
       if ($token->getId() === T_ECHO) {
           $rewrittenTokens[] = new Token(T_PRINT, 'print');
       } else {
           $rewrittenTokens[] = $token;
       }
   }
   $rewrittenCode = Token::toCode($rewrittenTokens);
   

4. File-Level Analysis: Process entire directories with SplFileInfo:

   $iterator = new \RecursiveIteratorIterator(new \RecursiveDirectoryIterator('src'));
   foreach ($iterator as $file) {
       if ($file->isFile() && $file->getExtension() === 'php') {
           $tokens = TokenStream::createFromFile($file->getPathname());
           // Analyze tokens...
       }
   }
   

Workflows

1. Custom Linter:

   • Integrate with PHPCS/PHPStan by extending their rule systems.
   • Example: Flag deprecated mysql_* functions:

     $tokens = TokenStream::createFromCode($code);
     $deprecatedFunctions = ['mysql_connect', 'mysql_query'];
     foreach ($tokens as $token) {
         if ($token->isGiven(T_STRING) && in_array($token->getValue(), $deprecatedFunctions)) {
             throw new \RuntimeException("Deprecated function used: {$token->getValue()}");
         }
     }
     

2. Test Data Extraction:

   • Parse test files to extract @dataProvider arguments dynamically:

     $tokens = TokenStream::createFromFile('tests/ExampleTest.php');
     $dataProviders = [];
     $inDocBlock = false;
     foreach ($tokens as $token) {
         if ($token->isGiven(T_DOC_COMMENT)) {
             $inDocBlock = true;
             if (preg_match('/@dataProvider\s+(\w+)/', $token->getValue(), $matches)) {
                 $dataProviders[] = $matches[1];
             }
         } elseif ($token->isGiven(T_WHITESPACE)) {
             $inDocBlock = false;
         }
     }
     

3. Code Metrics:

   • Count tokens by type (e.g., control structures) for complexity analysis:

     $tokens = TokenStream::createFromCode($code);
     $complexity = 0;
     foreach ($tokens as $token) {
         if (in_array($token->getId(), [T_IF, T_FOR, T_WHILE, T_SWITCH])) {
             $complexity++;
         }
     }
     

Integration Tips

1. Combine with Other Tools:

   • Use nikic/php-parser for AST-level operations after token analysis.
   • Pair with symfony/finder for file discovery:

     use Symfony\Component\Finder\Finder;
     $finder = Finder::create()->files()->in('src')->name('*.php');
     foreach ($finder as $file) {
         $tokens = TokenStream::createFromFile($file->getPathname());
         // Analyze...
     }
     

2. Performance Optimization:

   • Stream tokens incrementally for large files:

     $tokens = TokenStream::createFromFile('large_file.php');
     while ($tokens->valid()) {
         $token = $tokens->current();
         // Process token...
         $tokens->next();
     }
     

   • Cache token streams if reprocessing the same files frequently.

3. Error Handling:

   • Wrap token stream creation in try-catch:

     try {
         $tokens = TokenStream::createFromCode($code);
     } catch (\RuntimeException $e) {
         // Handle malformed PHP (e.g., log or skip file)
     }
     

Gotchas and Tips

Pitfalls

1. Tokenizer Extension Dependency:

   • Issue: The tokenizer extension must be enabled. Some shared hosting environments disable it.
   • Fix: Add a runtime check:

     if (!extension_loaded('tokenizer')) {
         throw new \RuntimeException('Tokenizer extension is required. Enable it in php.ini.');
     }
     

2. PHP Version Incompatibilities:

   • Issue: New PHP features (e.g., attributes, match expressions) introduce new tokens. Older versions of php-token-stream may not recognize them.
   • Fix: Test against target PHP versions early. Example for PHP 8.1 attributes:

     $tokens = TokenStream::createFromCode('[Attribute] class Example {}');
     // Check for T_ATTRIBUTE (PHP 8.1+)
     

3. Token Stream State:

   • Issue: Iterating over a TokenStream consumes it. Reusing the same stream requires re-creation.
   • Fix: Store tokens in an array if multiple passes are needed:

     $tokens = TokenStream::createFromCode($code)->toArray();
     

4. Heredoc and Nowdoc:

   • Issue: Tokens inside heredoc/nowdoc blocks may not be parsed as expected (e.g., variables are not interpolated).
   • Fix: Handle these blocks as opaque strings if precise analysis isn’t required.

5. Dynamic Code:

   • Issue: Code generated via eval(), create_function(), or $$vars may not be tokenized predictably.
   • Fix: Avoid analyzing dynamic code or use runtime inspection (e.g., reflection) as a fallback.

6. Whitespace Sensitivity:

   • Issue: Token streams preserve whitespace, which can affect reformatting logic.
   • Fix: Use Token::toCode() carefully—it may not match original formatting.

Debugging

1. Token Dumping:

   • Inspect raw tokens for debugging:

     $tokens = TokenStream::createFromCode($code)->toArray();
     print_r(array_map(function($token) {
         return [
             'id' => $token->getId(),
             'value' => $token->getValue(),
             'line' => $token->getLine(),
         ];
     }, $tokens));
     

2. Line/Column Tracking:

   • Use getLine() and getColumn() to pinpoint issues:

     foreach ($tokens as $token) {
         if ($token->getId() === T_ERROR) {
             echo "Error at line {$token->getLine()}, column {$token->getColumn()}";
         }
     }
     

3. Edge Cases:

   • Test with:
     • Empty files.
     • Files with syntax errors (e.g., unclosed braces).
     • Multi-byte characters (e.g., UTF-8 comments).
     • PHP 8+ features (e.g., attributes, match expressions).

Config Quirks

1. TokenStream Creation:
   • From Code: Use TokenStream::createFromCode($phpCode) for strings.
   • From File: Use TokenStream::createFromFile($filePath) for files. Note: Files are read once and cached in memory.
   • From Array: Re