Json Streamer Laravel Package

Getting Started

Minimal Setup

1. Installation:

   composer require symfony/json-streamer
   

   Add to composer.json if using Laravel’s Symfony components:

   "require": {
       "symfony/json-streamer": "^8.0"
   }
   

2. First Use Case: Stream a large JSON file without loading it entirely into memory:

   use Symfony\Component\JsonStream\JsonStreamer;
   
   $streamer = new JsonStreamer();
   $stream = fopen('large_file.json', 'r');
   
   $streamer->read($stream, function ($data) {
       // Process each JSON object/array incrementally
       if (is_array($data)) {
           // Handle array data
       } elseif (is_object($data)) {
           // Handle object data
       }
   });
   

3. Key Classes to Know:

   • JsonStreamer: Core class for reading/writing streams.
   • JsonStreamer\JsonStreamer::read(): Parse JSON incrementally.
   • JsonStreamer\JsonStreamer::write(): Generate JSON streams.
   • JsonStreamer\JsonStreamer::writeObject(): Stream objects with custom logic.

4. Laravel Integration: Use with Laravel’s StreamedResponse for chunked API responses:

   use Symfony\Component\HttpFoundation\StreamedResponse;
   
   return new StreamedResponse(function () {
       $streamer = new JsonStreamer();
       $streamer->writeObject($this->getLargeData(), function ($chunk) {
           echo $chunk;
       });
   }, 200, ['Content-Type' => 'application/json']);
   

Implementation Patterns

Common Workflows

1. Streaming JSON Responses in APIs

Use Case: Return large datasets (e.g., paginated results, logs) without memory overload.

public function streamData(Request $request)
{
    $data = $this->repository->fetchLargeDataset(); // Generator or iterable
    $response = new StreamedResponse(
        fn() => JsonStreamer::writeObject($data, fn($chunk) => echo $chunk),
        200,
        ['Content-Type' => 'application/json']
    );
    return $response->withHeader('X-Content-Type-Options', 'nosniff');
}

2. Processing Large JSON Files

Use Case: Parse CSV/JSON files line-by-line (e.g., imports, ETL).

public function processJsonFile($filePath)
{
    $streamer = new JsonStreamer();
    $stream = fopen($filePath, 'r');

    $streamer->read($stream, function ($data) {
        if (is_array($data)) {
            $this->processArrayChunk($data);
        }
    });
}

3. Custom Value Transformers

Use Case: Modify how properties are serialized/deserialized (e.g., dates, nested objects).

use Symfony\Component\JsonStream\ValueTransformer\ValueTransformerInterface;

class DateTimeTransformer implements ValueTransformerInterface
{
    public function transform($value, string $type, array $context = []): ?string
    {
        return $value instanceof \DateTime ? $value->format('Y-m-d\TH:i:sP') : null;
    }
}

// Register globally or per-stream
$streamer = new JsonStreamer();
$streamer->addValueTransformer(new DateTimeTransformer());

4. Self-Referencing Objects

Use Case: Handle circular references (e.g., graphs, trees).

$streamer = new JsonStreamer();
$streamer->writeObject($graphNode, function ($chunk) {
    echo $chunk;
}, ['include_null_properties' => true]);

5. Event-Driven Streaming

Use Case: Process JSON streams in real-time (e.g., webhooks, SSE).

use Symfony\Component\JsonStream\JsonStreamer;

$streamer = new JsonStreamer();
$streamer->read($webhookStream, function ($event) {
    dispatch(new ProcessWebhookEvent($event));
});

Laravel-Specific Patterns

Queue Jobs for Batch Processing

Use Laravel queues to offload heavy JSON processing:

class ProcessLargeJsonJob implements ShouldQueue
{
    public function handle()
    {
        $streamer = new JsonStreamer();
        $streamer->read($this->fileStream, function ($data) {
            // Process chunk in database/queue
            ProcessedData::create($data);
        });
    }
}

Middleware for Chunked Responses

Add middleware to auto-stream large responses:

public function handle($request, Closure $next)
{
    $response = $next($request);
    if ($response->getContent() && strlen($response->getContent()) > 1024 * 1024) {
        $streamer = new JsonStreamer();
        $streamer->writeObject(json_decode($response->getContent()), function ($chunk) {
            echo $chunk;
        });
        return new StreamedResponse(fn() => echo $chunk, 200, ['Content-Type' => 'application/json']);
    }
    return $response;
}

API Resource Streaming

Extend Laravel’s JsonResource to support streaming:

use Symfony\Component\JsonStream\JsonStreamer;

class StreamedResource extends JsonResource
{
    public function toStreamedResponse($request)
    {
        return new StreamedResponse(
            fn() => JsonStreamer::writeObject($this->resource, fn($chunk) => echo $chunk),
            200,
            ['Content-Type' => 'application/json']
        );
    }
}

Gotchas and Tips

Pitfalls

1. Memory Leaks in Older Versions:

   • Pre-8.0.4 versions had memory leaks with stream readers/writers.
   • Fix: Always use symfony/json-streamer:^8.0 or ^7.4.8+.
   • Workaround: Manually close streams or use dependency injection to manage lifecycles.

2. Self-Referencing Objects:

   • Circular references (e.g., User->posts->author->user) may cause infinite loops.
   • Fix: Use the include_null_properties option or implement custom ValueTransformer:

     $streamer->writeObject($object, null, ['include_null_properties' => true]);
     

3. DateTime Handling:

   • Union types (e.g., string|DateTime) may fail to serialize/deserialize.
   • Fix: Register a ValueTransformer for DateTime:

     $streamer->addValueTransformer(new DateTimeTransformer());
     

4. Nested Generators:

   • Deeply nested generators (e.g., foreach inside foreach) can crash with RecursionLimitException.
   • Fix: Limit recursion depth or flatten data structures.

5. Stream Positioning:

   • JsonStreamer does not support seeking in streams (unlike fseek).
   • Workaround: Rewind streams or buffer chunks manually.

Debugging Tips

1. Validate JSON Streams: Use JsonStreamer::validate() to check for malformed JSON:

   try {
       $streamer->validate($stream);
   } catch (\Symfony\Component\JsonStream\Exception\JsonException $e) {
       Log::error("Invalid JSON: " . $e->getMessage());
   }
   

2. Log Streamed Data: Inspect chunks during development:

   $streamer->read($stream, function ($data) {
       Log::debug('Stream chunk:', ['data' => $data]);
       // Process data...
   });
   

3. Profile Memory Usage: Compare memory usage with/without streaming:

   $memoryBefore = memory_get_usage();
   $streamer->read($stream, fn($data) => null);
   $memoryAfter = memory_get_usage();
   Log::info("Memory saved: " . ($memoryBefore - $memoryAfter) / 1024 / 1024 . "MB");
   

Configuration Quirks

1. Caching:

   • JsonStreamer caches metadata for classes. Clear cache if classes change:

     $streamer->getMetadataFactory()->clearCache();
     

2. PHPDoc Generation:

   • Newer versions (v8.0+) auto-generate PHPDoc for streamed objects.
   • Tip: Use JsonStreamer::writeObject($obj, null, ['generate_phpdoc' => true]).

3. Union Types:

   • Explicitly handle unions (e.g., string|int) with custom transformers:

     $streamer->addValueTransformer(new UnionTypeTransformer());
     

Extension Points

1. Custom Metadata Factories: Extend MetadataFactoryInterface to add custom type hints or annotations