Json Streaming Parser Laravel Package

Getting Started

Minimal Setup

1. Install the package:

   composer require salsify/json-streaming-parser
   

2. Create a listener implementing \JsonStreamingParser\Listener\ListenerInterface:

   use JsonStreamingParser\Listener\ListenerInterface;
   
   class MyJsonListener implements ListenerInterface {
       public function enterObject(array $properties) {}
       public function leaveObject() {}
       public function enterArray() {}
       public function leaveArray() {}
       public function key($key) {}
       public function value($value) {}
       public function complete() {}
   }
   

3. Parse a JSON file:

   $stream = fopen('large_file.json', 'r');
   $parser = new \JsonStreamingParser\Parser($stream, new MyJsonListener());
   $parser->parse();
   fclose($stream);
   

First Use Case: Large JSON Processing

Use this package when dealing with JSON files too large to load into memory (e.g., >100MB). Example:

// Process a 500MB JSON file without memory overload
$listener = new class implements ListenerInterface {
    public function value($value) {
        // Process each value as it's parsed (e.g., write to DB)
    }
};
$parser = new \JsonStreamingParser\Parser(fopen('huge_data.json', 'r'), $listener);
$parser->parse();

Implementation Patterns

Core Workflow

1. Streaming Pipeline:

   $stream = fopen('data.json', 'r');
   $listener = new MyListener();
   $parser = new \JsonStreamingParser\Parser($stream, $listener);
   $parser->parse(); // Triggers listener callbacks
   fclose($stream);
   

2. Listener Patterns:

   • Stateful Parsing: Track nested structures (e.g., current array/object depth) in your listener.

     class MyListener implements ListenerInterface {
         private $depth = 0;
     
         public function enterObject(array $properties) { $this->depth++; }
         public function leaveObject() { $this->depth--; }
     }
     

   • Selective Processing: Ignore irrelevant data by checking keys/values:

     public function key($key) {
         if ($key === 'metadata') return; // Skip metadata
     }
     

3. Integration with Laravel:

   • Service Provider: Register a parser service:

     $this->app->bind('json.parser', function() {
         return new \JsonStreamingParser\Parser(
             fopen(storage_path('app/large.json'), 'r'),
             new MyListener()
         );
     });
     

   • Queue Jobs: Process large JSON files asynchronously:

     ParseLargeJsonJob::dispatch('path/to/file.json');
     

     class ParseLargeJsonJob implements ShouldQueue {
         protected $filePath;
     
         public function handle() {
             $parser = new \JsonStreamingParser\Parser(
                 fopen($this->filePath, 'r'),
                 new MyListener()
             );
             $parser->parse();
         }
     }
     

4. Error Handling:

   • Wrap parsing in a try-catch to handle malformed JSON:

     try {
         $parser->parse();
     } catch (\JsonStreamingParser\Exception\ParsingException $e) {
         Log::error("JSON parse error at position {$e->getPosition()}: {$e->getMessage()}");
     }
     

5. Position Tracking (v8.0+):

   • Use PositionAwareInterface to log error positions:

     class MyListener implements ListenerInterface, PositionAwareInterface {
         public function setFilePosition($position) {
             $this->currentPosition = $position;
         }
     }
     

Gotchas and Tips

Common Pitfalls

1. Resource Leaks:

   • Always fclose() streams in finally blocks or use Laravel's Storage facade:

     $stream = fopen(storage_path('app/data.json'), 'r');
     try {
         $parser->parse();
     } finally {
         fclose($stream);
     }
     

   • Tip: Use Laravel\Filesystem\Filesystem::get() for automatic cleanup:

     $stream = Storage::get('path/to/file.json');
     $parser = new \JsonStreamingParser\Parser(fopen('php://temp', 'r+'), $listener);
     fwrite($stream, $parser->getStream());
     

2. UTF-8 BOM Issues:

   • Strip BOM manually if parsing fails:

     $stream = fopen('file.json', 'r');
     $bom = pack('H*', 'EFBBBF');
     if (fread($stream, 3) === $bom) {
         rewind($stream);
     }
     

3. Nested Structures:

   • Gotcha: Forgetting to track depth can lead to misaligned enter/leave calls.
   • Fix: Use a counter in your listener:

     private $depth = 0;
     public function enterObject(array $properties) { $this->depth++; }
     public function leaveObject() { $this->depth--; }
     

4. Large Arrays/Objects:

   • Performance Tip: For deeply nested structures, batch writes to a database:

     private $batch = [];
     public function value($value) {
         $this->batch[] = $value;
         if (count($this->batch) >= 100) {
             DB::table('data')->insert($this->batch);
             $this->batch = [];
         }
     }
     

5. PHP 8.0+ Features:

   • Tip: Use named arguments in listener methods (PHP 8.0+):

     public function value(string $value) { /* ... */ }
     

Debugging Tips

1. Log Events:

   • Implement a debug listener to log all events:

     class DebugListener implements ListenerInterface {
         public function enterObject(array $properties) {
             Log::debug('Enter object', ['properties' => $properties]);
         }
         // Implement all other methods similarly
     }
     

2. Test with Small Files:

   • Validate logic with tiny JSON files before scaling up:

     {"key": "value"}
     

3. Check for Trailing Commas:

   • The parser may fail on trailing commas in arrays/objects. Validate input with:

     if (preg_match('/[,\[\]{}]\s*,\s*[^\s]/', file_get_contents('file.json'))) {
         throw new \Exception('Trailing comma detected!');
     }
     

Extension Points

1. Custom Exceptions:

   • Extend \JsonStreamingParser\Exception\ParsingException for domain-specific errors:

     class InvalidDataException extends ParsingException {}
     

2. Parser-Aware Listeners (v8.1+):

   • Use ParserAwareInterface to access parser state:

     class MyListener implements ListenerInterface, ParserAwareInterface {
         public function setParser(\JsonStreamingParser\Parser $parser) {
             $this->parser = $parser;
         }
     }
     

3. Position Tracking:

   • Override setFilePosition() to log line/column numbers:

     public function setFilePosition($position) {
         $this->line = substr_count($this->rawContent, "\n", 0, $position) + 1;
     }
     

4. Stream Wrappers:

   • Parse from remote sources using fopen() wrappers:

     $parser = new \JsonStreamingParser\Parser(
         fopen('https://example.com/large.json', 'r'),
         $listener
     );