Vdm Library Ftp Transport Bundle Laravel Package

Getting Started

Minimal Setup

1. Installation:

   composer require 3slab/vdm-library-ftp-transport-bundle
   

   (Note: Prefer migrating to VdmLibraryFlysystemTransportBundle for new projects.)

2. Configuration: Add to config/packages/vdm_library_ftp_transport.yaml (or merge into config/packages/messenger.yaml):

   framework:
       messenger:
           transports:
               ftp_consumer:
                   dsn: "ftp://username:password@ftp.example.com/path/to/files/"
                   options:
                       mode: move
                       ftp_options:
                           dirpath: "/remote/files/"
                           storage: "/local/processed/"
                       monitoring:
                           enabled: true
   

3. First Use Case:

   • Define a VDM (Virtual Data Model) message handler to process files fetched via FTP.
   • Example handler (src/MessageHandler/FtpFileHandler.php):

     use Symfony\Component\Messenger\Attribute\AsMessageHandler;
     
     #[AsMessageHandler]
     public function __invoke(string $filePath) {
         // Process the file (e.g., parse CSV, upload to S3, etc.)
         file_put_contents('/local/processed/' . basename($filePath), file_get_contents($filePath));
     }
     

   • Trigger the consumer via CLI:

     php bin/console messenger:consume async ftp_consumer -vv
     

Implementation Patterns

Workflows

1. File Collection Modes:

   • move: Files are moved to a storage directory after processing (default for idempotency).

     options:
         mode: move
         ftp_options:
             storage: "/local/processed/"
     

   • delete: Files are deleted post-processing (use cautiously for critical data).

     options:
         mode: delete
     

2. Monitoring Integration: Enable monitoring to log FTP operations (e.g., file transfers, errors) via VDM Library’s monitoring system:

   options:
       monitoring:
           enabled: true
   

   • Access logs via:

     php bin/console vdm:monitoring:list
     

3. Custom DSN Formats:

   • FTP: ftp://user:pass@host:port/path/
   • SFTP: sftp://user:pass@host:port/path/ (requires PHP SFTP extension).

4. Batch Processing:

   • Use Symfony Messenger’s batch handling to process multiple files in a single message:

     #[AsMessageHandler]
     public function handleBatch(array $filePaths) {
         foreach ($filePaths as $path) {
             // Process each file
         }
     }
     

5. Integration with VDM Library:

   • Pair with VdmLibraryBundle for structured data modeling. Example:

     # config/packages/vdm_library.yaml
     vdm_library:
         models:
             ftp_file:
                 path: "%kernel.project_dir%/config/vdm/ftp_file.yaml"
     

   • Define a VDM model to map FTP files to domain objects.

Integration Tips

• Laravel-Specific:

  • Register the bundle in config/app.php:

    'extra.bundles' => [
        VdmLibrary\FtpTransportBundle\VdmLibraryFtpTransportBundle::class,
    ],
    

  • Use Laravel’s Artisan commands to manage consumers:

    php artisan messenger:consume ftp_consumer --limit=10
    

• Environment Variables: Store credentials in .env:

  FTP_TRANSPORT_DSN=ftp://${FTP_USER}:${FTP_PASSWORD}@${FTP_HOST}/files/
  

  • Reference in config:

    dsn: "%env(FTP_TRANSPORT_DSN)%"
    

• Testing:

  • Mock the FTP transport in PHPUnit:

    $this->messenger->setTransport('ftp_consumer', new MockTransport());
    

  • Use league/flysystem adapters for local testing:

    $adapter = new LocalAdapter('/path/to/test/files');
    $filesystem = new Filesystem($adapter);
    

Gotchas and Tips

Pitfalls

1. Retry Strategy:

   • Hard Requirement: retry_strategy.max_retries must be 0. FTP transport ignores retries.

     retry_strategy:
         max_retries: 0  # <-- Critical
     

   • Workaround: Implement retries at the application level (e.g., exponential backoff in the message handler).

2. SFTP Dependencies:

   • Missing Extension: SFTP requires the php-sftp extension (not php-ftp).
     • Install via:

       pecl install sftp
       

     • Or use league/flysystem-sftp as a fallback.

3. File Permissions:

   • Storage Path Issues: Ensure the storage directory (for mode: move) is writable by the PHP process.

     chmod -R 775 /local/processed/
     

   • Remote Permissions: FTP servers may block moves/deletes in certain directories.

4. Monitoring Overhead:

   • Performance Impact: Enabling monitoring adds logging overhead. Disable in production if unnecessary:

     options:
         monitoring:
             enabled: false
     

5. Deprecation Warning:

   • Active Maintenance: This bundle is unmaintained. Migrate to VdmLibraryFlysystemTransportBundle for long-term support.
   • Key differences:
     • Flysystem supports retries, SFTP, and more transports.
     • Example migration config:

       transports:
           ftp_consumer:
               dsn: "flysystem://ftp://user:pass@host/path"
               options:
                   adapter: ftp
                   mode: move
       

Debugging

1. Connection Issues:

   • Symptoms: Files not fetched, timeouts.
   • Debug Steps:
     • Verify DSN format (e.g., ftp:// vs. sftp://).
     • Test connectivity manually:

       ftp ftp.example.com
       

     • Check PHP error logs for phpseclib (underlying library) errors.

2. File Processing Failures:

   • Symptoms: Files stuck in dirpath or missing in storage.
   • Debug Steps:
     • Enable verbose logging:

       php bin/console messenger:consume ftp_consumer -vv
       

     • Check VDM monitoring logs:

       php bin/console vdm:monitoring:show ftp_consumer
       

3. Configuration Errors:

   • Symptoms: "Invalid configuration" or "Unknown option".
   • Debug Steps:
     • Validate YAML syntax (use yaml-lint.com).
     • Ensure dirpath and storage paths are absolute (relative paths may fail).

Extension Points

1. Custom FTP Executor:

   • Override the default executor by binding a service:

     # config/services.yaml
     VdmLibrary\FtpTransportBundle\Executor\FtpExecutorInterface: '@app.custom_ftp_executor'
     

   • Example implementation:

     use VdmLibrary\FtpTransportBundle\Executor\FtpExecutorInterface;
     use League\Flysystem\FTP\FTPConnectionOptions;
     
     class CustomFtpExecutor implements FtpExecutorInterface {
         public function connect(string $dsn): void {
             // Custom logic (e.g., proxy, logging)
             $options = FTPConnectionOptions::fromDsn($dsn);
             $client = new \phpseclib\Net\SFTP($options['host']);
             $client->login($options['username'], $options['password']);
         }
     }
     

2. Pre/Post-Processing Hooks:

   • Use Symfony’s event system to intercept file operations:

     // src/EventSubscriber/FtpFileSubscriber.php
     use Symfony\Component\EventDispatcher\EventSubscriberInterface;
     use VdmLibrary\FtpTransportBundle\Event\FileProcessedEvent;
     
     class FtpFileSubscriber implements EventSubscriberInterface {
         public static function getSubscribedEvents() {
             return [
                 FileProcessedEvent::class => 'onFileProcessed',
             ];
         }
     
         public function onFileProcessed(FileProcessedEvent $event) {
             // Log, transform, or validate the file
             $event->setMetadata(['processed_at' => now()]);
         }
     }
     

3. Dynamic DSN Configuration:

   • Fetch DSN from a database or API:

     $dsn = $this->ftpConfigRepository->getDsnForClient($clientId);
     $transport = new FtpTransport($dsn, $options);