Json Stream Laravel Package

Getting Started

Minimal Steps

1. Installation:

   composer require bcncommerce/json-stream
   

   Add to composer.json if using a monorepo or custom package.

2. First Use Case: Replace a memory-heavy json_encode() call for a large dataset (e.g., exporting 100K+ records).

   use Bcncommerce\JsonStream\Writer;
   
   $fh = fopen('large_export.json', 'w');
   $writer = new Writer($fh);
   
   $writer->enter(Writer::TYPE_OBJECT);
   foreach ($hugeArray as $item) {
       $writer->write(null, $item); // Streams incrementally
   }
   $writer->leave();
   fclose($fh);
   

3. Where to Look First:

   • Writer API: Writer class for generating JSON.
   • Reader API: Reader class for parsing JSON.
   • Examples: README.md for product catalog use cases.

Implementation Patterns

Usage Patterns

1. Streaming Exports:

   • Laravel Artisan Commands: Export Eloquent collections to JSON without memory overload.

     public function handle() {
         $products = Product::all();
         $fh = fopen(storage_path('app/exports/products.json'), 'w');
         $writer = new Writer($fh);
     
         $writer->enter(Writer::TYPE_ARRAY);
         foreach ($products as $product) {
             $writer->write(null, $product->toArray());
         }
         $writer->leave();
         fclose($fh);
     }
     

2. Streaming Imports:

   • Queue Jobs: Parse large JSON files in chunks (e.g., webhook payloads).

     public function handle() {
         $fh = fopen(storage_path('app/uploads/large_payload.json'), 'r');
         $reader = new Reader($fh);
     
         $reader->enter(Reader::TYPE_ARRAY);
         while ($item = $reader->read()) {
             // Process $item in chunks (e.g., save to DB)
             Model::create($item);
         }
         $reader->leave();
         fclose($fh);
     }
     

3. API Responses:

   • Chunked JSON Responses: Stream large API responses (e.g., paginated data).

     public function show(Request $request) {
         $fh = fopen('php://output', 'w');
         $writer = new Writer($fh);
     
         $writer->enter(Writer::TYPE_OBJECT);
         $writer->write('data', $this->streamDataGenerator());
         $writer->leave();
     
         return response()->stream(fn() => $fh);
     }
     
     private function streamDataGenerator() {
         foreach ($largeDataset as $item) {
             yield $item;
         }
     }
     

4. Hybrid with Laravel Storage:

   • S3/Cloud Storage: Stream JSON directly to cloud storage without local files.

     use Illuminate\Support\Facades\Storage;
     
     $writer = new Writer(Storage::disk('s3')->open('exports/large.json', 'w'));
     // ... write logic ...
     $writer->leave();
     

Workflows

1. ETL Pipelines:

   • Reader + Writer: Transform JSON formats (e.g., CSV → JSON → Elasticsearch).

     $reader = new Reader(fopen('input.csv', 'r')); // Custom CSV parser
     $writer = new Writer(fopen('output.json', 'w'));
     
     $writer->enter(Writer::TYPE_ARRAY);
     while ($row = $reader->read()) {
         $writer->write(null, $this->transformRow($row));
     }
     $writer->leave();
     

2. Real-Time Processing:

   • Event Listeners: Stream JSON from webhooks (e.g., Stripe, GitHub).

     public function handle(WebhookPayload $payload) {
         $fh = fopen('php://temp', 'r+');
         fwrite($fh, $payload->getContent());
         rewind($fh);
     
         $reader = new Reader($fh);
         $reader->enter(Reader::TYPE_OBJECT);
         $data = $reader->read('data'); // Parse incrementally
         $reader->leave();
         // Process $data
     }
     

3. Testing:

   • Mock Streams: Use php://memory for unit tests.

     $fh = fopen('php://memory', 'r+');
     $writer = new Writer($fh);
     $writer->enter(Writer::TYPE_OBJECT);
     $writer->write('test', 'value');
     $writer->leave();
     rewind($fh);
     
     $reader = new Reader($fh);
     $result = $reader->read('test');
     $this->assertEquals('value', $result);
     

Integration Tips

1. Laravel Facades:

   • Wrap the package in a facade for consistency.

     // app/Facades/JsonStream.php
     public static function export(array $data, string $path) {
         $fh = Storage::disk('public')->open($path, 'w');
         $writer = new Writer($fh);
         $writer->enter(Writer::TYPE_OBJECT);
         foreach ($data as $key => $value) {
             $writer->write($key, $value);
         }
         $writer->leave();
         fclose($fh);
     }
     

2. Service Providers:

   • Bind the package to Laravel’s container for dependency injection.

     $this->app->singleton('json.stream.writer', function () {
         return new Writer(fopen('php://temp', 'w'));
     });
     

3. Middleware:

   • Stream JSON responses dynamically (e.g., for large downloads).

     public function handle($request, Closure $next) {
         if ($request->wantsJson() && $request->largePayload) {
             $response = $next($request);
             $response->setContent($this->streamJson($response->getContent()));
         }
         return $next($request);
     }
     
     private function streamJson($data) {
         $fh = fopen('php://temp', 'r+');
         $writer = new Writer($fh);
         $writer->enter(Writer::TYPE_OBJECT);
         foreach ($data as $key => $value) {
             $writer->write($key, $value);
         }
         $writer->leave();
         rewind($fh);
         return stream_get_contents($fh);
     }
     

4. Queue Jobs:

   • Process streaming JSON in background jobs (e.g., parsing logs).

     public function handle() {
         $fh = fopen(storage_path('app/logs/large.log.json'), 'r');
         $reader = new Reader($fh);
     
         $reader->enter(Reader::TYPE_ARRAY);
         while ($logEntry = $reader->read()) {
             $this->processLogEntry($logEntry);
         }
         $reader->leave();
         fclose($fh);
     }
     

Gotchas and Tips

Pitfalls

1. Resource Leaks:

   • Unclosed Handles: Always call fclose() or use finally blocks.

     $fh = fopen('file.json', 'r');
     try {
         $reader = new Reader($fh);
         // ... read logic ...
     } finally {
         fclose($fh);
     }
     

   • Laravel Context: Use Storage::disk()->close() for Laravel’s storage facades.

2. Memory Spikes:

   • Buffering: Avoid writing large arrays at once. Stream item-by-item.

     // Anti-pattern: Memory spike
     $writer->write(null, $hugeArray);
     
     // Pattern: Memory-efficient
     $writer->enter(Writer::TYPE_ARRAY);
     foreach ($hugeArray as $item) {
         $writer->write(null, $item);
     }
     $writer->leave();
     

3. Nested Structures:

   • Mismatched enter()/leave(): Causes malformed JSON. Validate with a tool like JSONLint.

     // Example of mismatch
     $writer->enter(Writer::TYPE_OBJECT);
     $writer->enter('items', Writer::TYPE_ARRAY); // Enter array
     // ... missing leave for array ...
     $writer->leave(); // Only leaves object, not array
     

4. PHP Version Quirks:

   • PHP 8.2+: Strict typing may cause issues with null values in read().

     // Workaround for PHP 8.2+
     $value = $reader->read('key') ?: null;
     

5. Encoding Issues:

   • UTF-8 BOM: Ensure streams are UTF-8 encoded. Use fopen($path, 'w+') with explicit encoding.

     $fh = fopen('file.json', 'w+');
     stream_set_blocking($fh, true);