Filesystem Laravel Package

Getting Started

Minimal Setup

1. Installation

   composer require flow-php/filesystem
   

   Register the service provider in config/app.php:

   'providers' => [
       Flow\Filesystem\FilesystemServiceProvider::class,
   ],
   

2. Basic Usage Initialize the filesystem adapter (e.g., for AWS S3):

   use Flow\Filesystem\Filesystem;
   
   $filesystem = app(Filesystem::class)->adapter('s3');
   

3. First Use Case: Stream a Remote File

   $file = $filesystem->read('bucket/path/to/file.txt');
   $file->stream(); // Stream the file without loading it entirely into memory
   

Implementation Patterns

Core Workflows

1. Streaming Reads (ETL Pipelines) Use byte-range requests for large files (e.g., processing logs or CSV exports):

   $stream = $filesystem->read('remote/file.csv')->stream();
   while (!$stream->eof()) {
       $chunk = $stream->read(1024);
       // Transform chunk (e.g., parse CSV)
   }
   

2. Chunked Writes (ETL Outputs) Write files in chunks to avoid memory overload:

   $writer = $filesystem->write('remote/output.json');
   foreach ($data as $chunk) {
       $writer->write(json_encode($chunk));
   }
   $writer->close();
   

3. Adapter Switching Dynamically switch between local (e.g., local) and cloud (e.g., s3, gcs) adapters:

   $adapter = $filesystem->adapter($env('STORAGE_DRIVER'));
   

Integration Tips

• Laravel Filesystem Bridge Extend Laravel’s FilesystemManager to use flow-php/filesystem:

  $this->app->extend('filesystem', function ($app, $filesystem) {
      return new FlowFilesystemAdapter($filesystem->adapter('s3'));
  });
  

• ETL Pipelines Combine with spatie/flysystem or league/flysystem for hybrid workflows:

  $flysystem = new FlysystemAdapter($filesystem->adapter('s3'));
  

• Telemetry Disable telemetry in config/filesystem.php if needed:

  'telemetry' => [
      'enabled' => env('FILESYSTEM_TELEMETRY', false),
  ],
  

Gotchas and Tips

Pitfalls

1. Adapter Configuration

   • Cloud adapters (e.g., s3, gcs) require explicit credentials in .env or config:

     AWS_ACCESS_KEY_ID=your_key
     AWS_SECRET_ACCESS_KEY=your_secret
     

   • Fix: Use Laravel’s config/filesystems.php to centralize settings.

2. Stream Handling

   • Unclosed streams may leak resources. Always call:

     $stream->close(); // Critical for remote adapters
     

   • Tip: Use PHP’s finally blocks or context managers (e.g., Symfony\Component\Filesystem\Stream).

3. Byte-Range Limits

   • Some cloud providers (e.g., Azure Blob) have minimum/maximum range sizes.
   • Workaround: Pad chunks or use provider-specific adapters.

4. Telemetry Overhead

   • Enabled by default. Disable in config/filesystem.php for production:

     'telemetry' => ['enabled' => false],
     

Debugging

• Logs Enable debug mode in config/filesystem.php:

  'debug' => env('FILESYSTEM_DEBUG', false),
  

  Logs appear in Laravel’s default log channel.

• Stream Errors Wrap streams in try-catch:

  try {
      $stream = $filesystem->read('file.txt')->stream();
  } catch (Flow\Filesystem\Exception\StreamException $e) {
      Log::error('Stream failed: ' . $e->getMessage());
  }
  

Extension Points

1. Custom Adapters Implement Flow\Filesystem\Adapter\AdapterInterface for unsupported storages (e.g., FTP):

   class FtpAdapter implements AdapterInterface {
       public function read(string $path): Stream {
           // Custom logic
       }
       // ...
   }
   

   Register via service provider:

   $this->app->bind('flow.filesystem.adapter.ftp', function () {
       return new FtpAdapter();
   });
   

2. Event Hooks Listen for stream events (e.g., stream.opened, stream.closed) via Laravel’s event system:

   event(new StreamOpened($stream, $path));
   

3. Chunk Transformers Extend Flow\Filesystem\Stream to add preprocessing:

   $stream->pipeThrough(function ($chunk) {
       return gzdecode($chunk); // Decompress on-the-fly
   });