Json Streamer Laravel Package

Getting Started

Minimal Steps

1. Installation:

   composer require symfony/serializer symfony/json-streamer
   

   • Laravel 10+ users can leverage Symfony’s built-in Serializer component.

2. First Use Case: Replace memory-intensive json_decode() with streaming for large JSON files:

   use Symfony\Component\Serializer\JsonStreamer;
   
   $streamer = new JsonStreamer();
   $streamer->read('large_dataset.json', function ($data) {
       // Process each chunk incrementally
       // Example: Save to DB, transform, or emit via WebSocket
   });
   

3. Where to Look First:

   • Symfony Documentation for API reference.
   • JsonStreamer::read() and JsonStreamer::write() methods for core functionality.
   • Laravel’s StreamedResponse for HTTP streaming integration.

Implementation Patterns

Core Workflows

1. Reading Large JSON Files

• Use Case: Processing API exports, logs, or database dumps (>100MB).
• Pattern:

  $streamer = new JsonStreamer();
  $streamer->read('data.json', function ($item) {
      // Process $item (e.g., store in DB, emit via Pusher)
  }, [
      'include_null_properties' => true, // Optional: Include null values
  ]);
  

• Laravel Integration: Wrap in a service for reusability:

  // app/Services/JsonStreamerService.php
  class JsonStreamerService {
      public function streamFile(string $path, callable $callback): void {
          $streamer = new JsonStreamer();
          $streamer->read($path, $callback);
      }
  }
  

2. Streaming JSON Responses

• Use Case: Large API responses (e.g., reports, exports).
• Pattern:

  use Symfony\Component\HttpFoundation\StreamedResponse;
  
  return new StreamedResponse(function () use ($data) {
      $streamer = new JsonStreamer();
      $streamer->write($data, $this->getEmitter());
  }, 200, ['Content-Type' => 'application/json']);
  

• Laravel Tip: Use Http\StreamedResponse (Laravel 10+) or Illuminate\Http\StreamedResponse.

3. Custom Value Transformers

• Use Case: Modify data during streaming (e.g., sanitize, transform).
• Pattern:

  $streamer = new JsonStreamer();
  $streamer->setValueTransformer('App\\Entity\\User', function ($value, $class, $format) {
      return $value->toArray(); // Convert entity to array
  });
  

4. Self-Referencing Objects

• Use Case: Circular references (e.g., nested comments, trees).
• Pattern:

  $streamer = new JsonStreamer();
  $streamer->write($rootObject, $emitter, [
      'ignore_circular_references' => false, // Default: true
  ]);
  

5. CLI Batch Processing

• Use Case: ETL pipelines or data migrations.
• Pattern:

  // In an Artisan command
  $streamer = new JsonStreamer();
  $streamer->read('input.json', function ($item) {
      // Process and save to CSV/DB
  });
  

Integration Tips

Laravel-Specific Patterns

1. Service Container Binding:

   // config/app.php
   'bindings' => [
       Symfony\Component\Serializer\JsonStreamer::class => function () {
           return new JsonStreamer();
       },
   ];
   

2. Queue Jobs for Large Files:

   // app/Jobs/ProcessJsonStream.php
   public function handle() {
       $streamer = resolve(JsonStreamer::class);
       $streamer->read(storage_path('large.json'), [$this, 'processChunk']);
   }
   
   public function processChunk($data) {
       // Process incrementally
   }
   

3. Middleware for API Streaming:

   // app/Http/Middleware/StreamJsonResponse.php
   public function handle($request, Closure $next) {
       if ($request->wantsJson() && $request->query('stream')) {
           return $this->streamResponse($request);
       }
       return $next($request);
   }
   
   private function streamResponse($request) {
       $data = $this->fetchLargeData();
       return new StreamedResponse(function () use ($data) {
           $streamer = new JsonStreamer();
           $streamer->write($data, $this->getEmitter());
       });
   }
   

Performance Optimizations

• Cache Metadata: Use JsonStreamer with Symfony’s Serializer to cache class metadata:

  $serializer = Serializer::create([], [new JsonStreamer()]);
  

• Lazy Loading: For nested objects, enable lazy loading to avoid deep cloning:

  $streamer->write($object, $emitter, ['lazy' => true]);
  

Gotchas and Tips

Pitfalls

1. Memory Leaks:

   • Issue: Older versions (<8.0.4) had memory leaks with stream readers/writers.
   • Fix: Always use the latest version (8.0.x or 7.4.x).
   • Workaround: Explicitly close streams:

     $streamer->read($file, function ($data) { /* ... */ });
     $streamer->close(); // Manually close if needed
     

2. Circular References:

   • Issue: Default behavior ignores circular references (ignore_circular_references: true).
   • Fix: Set ignore_circular_references: false and handle cycles manually:

     $streamer->write($object, $emitter, ['ignore_circular_references' => false]);
     

3. Null Properties:

   • Issue: By default, JsonStreamer skips null properties.
   • Fix: Enable include_null_properties:

     $streamer->read($file, $callback, ['include_null_properties' => true]);
     

4. PHP 8.4+ Compatibility:

   • Issue: Some features (e.g., synthetic properties) require PHP 8.4+.
   • Fix: Check your PHP version and use fallbacks if needed.

5. Generator Overhead:

   • Issue: Deeply nested generators can cause stack overflows.
   • Fix: Limit recursion depth or flatten data structures.

Debugging Tips

1. Validate JSON Structure: Use JsonStreamer::read() with a debug callback to inspect chunks:

   $streamer->read('data.json', function ($chunk) {
       \Log::debug('Chunk:', ['data' => $chunk]);
   });
   

2. Check for Lazy Loading Issues: If objects aren’t hydrated correctly, ensure:

   • The Serializer is properly configured.
   • No custom Normalizer interferes with lazy loading.

3. Monitor Memory Usage:

   $streamer->read($file, function ($data) {
       \Log::info('Memory usage:', memory_get_usage(true));
   });
   

Extension Points

1. Custom Emitters: Extend JsonStreamer to support custom output (e.g., WebSocket, Kafka):

   $streamer = new JsonStreamer();
   $streamer->write($data, new CustomEmitter());
   

2. Value Transformers: Override default serialization/deserialization:

   $streamer->setValueTransformer('App\\Model\\User', function ($value) {
       return ['id' => $value->id, 'name' => $value->name];
   });
   

3. Event Listeners: Use Symfony’s EventDispatcher to hook into streaming events (e.g., pre/post chunk processing).

4. Custom Metadata: Annotate classes with #[Serializer\Groups] or #[Serializer\Ignore] to control serialization:

   #[Serializer\Ignore]
   private $sensitiveData;
   

Laravel-Specific Quirks

1. Service Provider Setup: If using Symfony’s Serializer, register it in AppServiceProvider:

   public function register() {
       $this->app->singleton(Symfony\Component\Serializer\Serializer::class, function () {
           return Serializer::create([], [new JsonStreamer()]);
       });
   }
   

2. Caching: Laravel’s cache can interfere with JsonStreamer's internal caching. Use null driver for testing:

   $streamer = new JsonStreamer();
   $streamer->setCache(new \Symfony\Component\Cache\NullCache());
   

3. Testing: Mock JsonStreamer