Typo3 Typoscript Parser Laravel Package

Getting Started

Minimal Setup

1. Installation:

   composer require helmich/typo3-typoscript-parser
   

   Ensure your project uses PHP 8.1+ (required since v2.6.0).

2. First Use Case: Parse a TypoScript file (typoscript.ts) into an Abstract Syntax Tree (AST):

   use Helmich\TypoScriptParser\Parser\Parser;
   use Helmich\TypoScriptParser\Tokenizer\Tokenizer;
   
   $typoscript = file_get_contents('path/to/typoscript.ts');
   $parser = new Parser(new Tokenizer());
   $statements = $parser->parse($typoscript);
   

3. Key Classes to Know:

   • Parser: Converts TypoScript to AST.
   • Tokenizer: Splits raw TypoScript into tokens.
   • PrettyPrinter: Converts AST back to TypoScript.
   • Traverser + Visitor: Analyze/modify AST (e.g., linting, transformations).

Implementation Patterns

1. Parsing and Traversal Workflow

• Parse TypoScript:

  $parser = new Parser(new Tokenizer());
  $statements = $parser->parse(file_get_contents('Config.ts'));
  

• Traverse AST with Visitors:

  $traverser = new Traverser($statements);
  $traverser->addVisitor(new MyCustomVisitor());
  $traverser->walk();
  

  Example Visitor:

  class MyCustomVisitor implements Visitor {
      public function enterNode(Statement $node) {
          if ($node instanceof Assignment) {
              // Modify or log assignments
          }
      }
  }
  

2. Transforming TypoScript

• Modify AST and Reprint:

  $printer = new PrettyPrinter();
  $printer->printStatements($modifiedStatements, new StreamOutput(fopen('output.ts', 'w')));
  

• Customize Output:

  $config = PrettyPrinterConfiguration::create()
      ->withSpaceIndentation(4)
      ->withConditionTermination(PrettyPrinterConditionTermination::EnforceEnd);
  $printer = new PrettyPrinter($config);
  

3. Integration with Laravel

• Service Provider: Bind the parser as a singleton in AppServiceProvider:

  $this->app->singleton(Parser::class, function ($app) {
      return new Parser(new Tokenizer());
  });
  

• Command for Linting/Validation:

  use Illuminate\Console\Command;
  
  class TypoScriptLintCommand extends Command {
      protected $signature = 'typo3:lint {file}';
      public function handle() {
          $statements = app(Parser::class)->parse(file_get_contents($this->argument('file')));
          // Traverse and validate
      }
  }
  

4. Handling Edge Cases

• Error Handling:

  try {
      $statements = $parser->parse($typoscript);
  } catch (ParseError $e) {
      $this->error("TypoScript error: " . $e->getMessage());
  }
  

• Debugging AST: Use StatementDumper to inspect the tree:

  $dumper = new StatementDumper();
  echo $dumper->dump($statements);
  

Gotchas and Tips

Pitfalls

1. PHP Version Compatibility:

   • Minimum PHP 8.1 (required since v2.6.0). Older versions will fail.
   • Symfony 7+ is supported; older versions may break.

2. Tokenization Quirks:

   • Comments: Preserved but stripped of whitespace during parsing. Use StatementDumper to debug.
   • Empty Object Keys: Allowed (e.g., = .), but may cause unexpected behavior in older TYPO3 versions.

3. PrettyPrinter Pitfalls:

   • Indentation: Defaults to 1 space. Use withSpaceIndentation(4) for readability.
   • Conditions: Without withIndentConditions(), nested conditions may collapse.
   • Global Statements: Ensure withClosingGlobalStatement() is set to avoid missing ] in output.

4. Visitor Traversal:

   • Order Matters: Visitors run in the order added. Place critical checks first.
   • State Management: Visitors are stateless by default. Use closures or class properties for shared state.

Debugging Tips

1. Dump Tokens: Extend Tokenizer to log tokens during development:

   $tokenizer = new Tokenizer();
   $tokenizer->setLogger(function ($token) {
       error_log("Token: " . print_r($token, true));
   });
   

2. AST Validation: Compare parsed AST against expected structure using StatementDumper:

   $dumper = new StatementDumper();
   $this->assertStringContainsString('page = PAGE', $dumper->dump($statements));
   

3. Performance:

   • Large Files: Parse incrementally if memory is a concern (stream the file line-by-line).
   • Caching: Cache parsed ASTs if the same TypoScript is reused (e.g., in a CLI tool).

Extension Points

1. Custom Token Types: Extend Tokenizer to handle non-standard TypoScript syntax:

   class CustomTokenizer extends Tokenizer {
       protected function getTokenRegex(): string {
           return parent::getTokenRegex() . '|/\*custom\*/';
       }
   }
   

2. Visitor Patterns:

   • Composite Visitors: Combine multiple visitors for complex logic.
   • Stateful Visitors: Use class properties to track context (e.g., current path in the AST).

3. PrettyPrinter Extensions: Override PrettyPrinter to customize output (e.g., add TYPO3 version headers):

   class CustomPrinter extends PrettyPrinter {
       public function printStatements(array $statements, OutputInterface $output) {
           $output->writeln('# TYPO3 v12 Configuration');
           parent::printStatements($statements, $output);
       }
   }
   

Configuration Quirks

1. Tokenizer Options: Configure preprocessing (e.g., trim whitespace):

   $tokenizer = new Tokenizer();
   $tokenizer->setPreprocessCallback(function ($code) {
       return preg_replace('/\s+/', ' ', $code); // Normalize whitespace
   });
   

2. PrettyPrinter Flags:

   • withConditionTermination(EnforceEnd): Ensures conditions end with }.
   • withIndentConditions(): Adds indentation for nested conditions.

3. Symfony Integration: If using Symfony DI, leverage the parser service directly:

   # config/services.yaml
   services:
       Helmich\TypoScriptParser\Parser\Parser:
           arguments:
               $tokenizer: '@Helmich\TypoScriptParser\Tokenizer\Tokenizer'