Getting Started

Install via Composer:

composer require cleverage/flysystem-process-bundle

return [
    // ...
    CleverAge\FlysystemProcessBundle\FlysystemProcessBundle::class => ['all' => true],
];

Correction to the v3.0 assessment: this bundle does not ship a ProcessAdapter with a run() method. The v3.2 release notes confirm the real public surface is a set of tasks that operate on Flysystem storages: FileFetchTask, ListContentTask, and RemoveFileTask. Build your first workflow around those, not around an invented adapter wrapper.

Where to look first:

The bundle's DependencyInjection extension and its configuration tree — v3.2 added dedicated tests for the bundle and DI extension (#33), so the test fixtures are the most reliable, up-to-date documentation of valid config keys.
The task classes themselves (FileFetchTask, ListContentTask, RemoveFileTask) — v3.2 fixed several of their edge cases and updated the docs alongside each fix (#37–#41).
The local-storage test cases added in #33 — they show the shortest path from a configured adapter to a working task.

First use case — pull files off a storage so local tooling can process them:

# config/packages/cleverage_flysystem_process.yaml
# Shape only — confirm exact keys against the bundle's DI extension test/config reference.
cleverage_flysystem_process:
    storages:
        source: '@flysystem.adapter.s3_source'
        target: '@flysystem.adapter.local_workdir'
    tasks:
        fetch_invoices:
            type: file_fetch
            source: source
            target: target
            file_pattern: 'invoices/*.pdf'
            ignore_missing: true

Then list, process, and clean up:

$files   = $listContentTask->run($sourceStorage, 'invoices/');  // discover inputs
$fetched = $fileFetchTask->run($files);                         // copy to local workdir
// ... run ffmpeg / convert / pdftotext locally on $fetched ...
$removeFileTask->run($files);                                   // delete sources once processed

Implementation Patterns

The canonical pipeline: list → fetch → process → remove

The tasks compose into a four-stage job. Each stage is independently runnable, which makes retries and partial failures manageable:

ListContentTask — enumerate the source storage (optionally filtered by pattern) and produce the input list.
FileFetchTask — copy the matched files from source to target storage. Accepts a single path or a list of paths (v3.2, #41 for RemoveFileTask, #38 for FileFetchTask), and honours file_pattern and ignore_missing.
Local processing — this is where Symfony's Process component (or your own tooling) actually runs: ffmpeg, ImageMagick, pdftotext, linters, archive tooling.
RemoveFileTask — delete the processed sources. Accepts a list of paths as of #41.

Batch inputs, don't loop the task

Before v3.2, FileFetchTask only copied files for the first input when file_pattern was used, and a path passed twice was silently dropped. If your code worked around this with a foreach over inputs, collapse it into one call now — you'll get the fixed behaviour and one source listing per input instead of one per file (#37).

// v3.2 — one call, correct handling of patterns, duplicates and missing files
$fileFetchTask->run($inputs, [
    'file_pattern'   => 'reports/*.csv',
    'ignore_missing' => true,
]);

Make local storage the default in dev and test

#33 added FileFetchTask, ListContentTask and RemoveFileTask coverage against local storages. That means you can run your entire pipeline against a local Flysystem adapter in tests and CI with no S3/SFTP emulator:

# config/packages/test/cleverage_flysystem_process.yaml
cleverage_flysystem_process:
    storages:
        source: '@flysystem.adapter.local_fixtures'
        target: '@flysystem.adapter.local_workdir'

Keep the storage names identical between dev, test and prod and only swap the adapter they point to — that way task configuration never diverges by environment.

Dependency injection

Tasks and storages are wired by the bundle's DI extension, so inject the task services rather than constructing them:

# config/services.yaml
services:
    App\Pipeline\InvoicePipeline:
        arguments:
            $listContentTask: '@cleverage_flysystem_process.task.list_content'
            $fileFetchTask:   '@cleverage_flysystem_process.task.file_fetch'
            $removeFileTask:  '@cleverage_flysystem_process.task.remove_file'

(Service IDs are indicative — take the exact ids from the extension's test fixtures added in #33.)

Driving it from the scheduler / Messenger

The tasks are stateless workers, which makes them good candidates for Symfony Scheduler or a Messenger handler. Because RemoveFileTask now correctly reports missing files as not found rather than deleted, a "process and remove" handler can distinguish "already handled by another worker" from "deleted successfully" — useful for idempotent retries.

Laravel usage

There is no Laravel service provider — this is a Symfony bundle. In a Laravel app you'd either (a) composer require the underlying Flysystem packages and instantiate the tasks yourself through the container, or (b) bind the task classes as singletons in a custom ServiceProvider and resolve the storage adapters from Laravel's filesystem disks. In practice (b) is the lower-friction option; write an artisan command per stage and chain them with Bus::chain() or a queued job.

Gotchas and Tips

Upgrade to v3.2 if you hit any of these

These are bug fixes, but several of them are behaviour changes you'll observe after upgrading:

file_pattern + multiple inputs (#38): only the first input used to be copied. Pipelines that "worked" because they only ever passed one input will now actually process all of them — expect more files and longer runtimes.
Duplicate paths (#38): a path supplied twice was previously skipped. It will now be fetched twice. Add a array_unique() upstream if your input list is not deduplicated.
ignore_missing on inputs (#39): previously a missing input file threw UnableToReadFile even with ignore_missing: true. It now skips silently. If you were relying on that exception as an alerting signal, add an explicit check on your input list instead.
Filename "0" (#40): a file literally named 0 was treated as "no file" by both FileFetchTask and RemoveFileTask. Fixed — but if you extend or wrap these tasks, use !== null / !== '' rather than empty() for the same reason.
RemoveFileTask logging (#41): a missing file is now logged as not found, not deleted. If you alert on deletion counts or on deleted log lines, expect a shift and possibly a dip in volume.
RemoveFileTask accepts a list of paths (#41), and the deletion-failure message was corrected — update any log-parsing or monitoring rules that matched the old string.

Performance: listing is per-input, not per-file

#37 fixed FileFetchTask re-listing the source storage before every file — it now lists once per input. That's a big win for large inputs, but if you pass 5,000 individual paths you still get 5,000 listings. Pass a directory or a glob pattern and let file_pattern do the filtering, rather than enumerating files yourself.

Debugging missing files

With ignore_missing: true, nothing throws — files just don't appear. When a fetch "silently does nothing", check in this order:

Is the pattern relative to the storage root, and does ListContentTask return the same paths you're feeding in?
Does the source adapter's visibility/permissions allow reads as the configured user?
Is ignore_missing masking the miss? Reproduce once with ignore_missing: false to surface the UnableToReadFile.

Version requirements

v3.0 moved the floor to PHP 8.2+ and Symfony 7.4+. On 8.1/Symfony 7.3 you must upgrade or pin to v2.x. In Laravel terms, that PHP floor is the binding constraint — the Symfony version only matters if you're running it inside a Symfony app or pulling the bundle in alongside one.

Extension points

Decorate the tasks. Rather than subclassing (they're final-ish workers with fixed contracts), use Symfony's service decoration to add retry, metrics or tracing around FileFetchTask / RemoveFileTask.
Flysystem plugins. Anything the tasks don't cover (checksums, path normalisation) is better handled by a Flysystem plugin on the adapter than by a wrapper task.
Custom file patterns. file_pattern is passed through to the source listing; if you need richer selection, run ListContentTask first, filter the returned array in your own service, then hand the array to FileFetchTask.
Config validation is your friend. The DI extension now has tests (#33); if a config key is silently ignored it's very likely a typo — run bin/console debug:config cleverage_flysystem_process before assuming a bug.

Flysystem Process Bundle Laravel Package