Etl Bundle Laravel Package

Getting Started

Install and pin the version. This is a pre-1.0 package, so pin the exact release rather than floating:

composer require whitedigital-eu/etl-bundle:0.5.10

0.x releases here are small and frequent (0.5.9 → 0.5.10 was a single-line behavioural fix), and patch numbers do occasionally change runtime behaviour. Treat every bump as something to read the changelog for, not something to let Composer pick up silently in CI.

Reconcile the framework mismatch first. Despite the package name, this ships as a Symfony bundle (etl-bundle), not a Laravel package. That has one practical consequence you'll hit within five minutes: there is no Laravel package auto-discovery, no config/etl.php merge, and no service-provider registration of the bundle's DI extension. In Laravel you use the underlying ETL classes directly — either instantiate them yourself or bind the ones you need in your own AppServiceProvider. Don't expect app(SomeEtlService::class) to resolve out of the box.

Where to look, in order:

1. src/ — the reader / transformer / writer contracts. These interfaces are the real API surface; everything else is wiring.
2. DependencyInjection/Configuration.php and Resources/config/services.* — this is the closest thing to documentation for what options exist and how the pieces are composed. Read it even though Laravel won't consume it; it tells you the intended object graph.
3. tests/ — the most reliable usage examples, especially for how rows and column mappings are shaped.
4. CHANGELOG.md / GitHub releases — mandatory reading before each bump.

First use case: CSV → database table. That's the canonical ETL shape and the fastest way to validate the whole pipeline. Build the smallest possible version: a reader over a 200-row fixture CSV, a single transformer that trims strings and casts dates, and a writer targeting a scratch table. Get that green end-to-end before touching a real dataset — most of the friction in this package is in row/column shape mismatches, and a tiny fixture surfaces those in seconds instead of minutes.

Implementation Patterns

Compose the pipeline as an explicit object, not as config. Wire reader → transformer(s) → writer in a small dedicated class (ImportCustomersPipeline) and let it take its dependencies via constructor. This keeps the Symfony-flavoured configuration out of your Laravel code and gives you one obvious place to look when a run misbehaves.

Wrap it in an Artisan command, then a queued job. The command is for developers (php artisan etl:import customers --file=...); the job is for production. Large files should not run in a request or a synchronous command — chunk the read and dispatch per-chunk jobs so a failed chunk can be retried without replaying the whole file.

Batch your writes — this is what 0.5.10 unlocks. Prior to 0.5.10, insert operations did not accept arrays, which pushed everyone into per-row inserts (correct, but slow: one round trip per row). With 0.5.10 you can accumulate transformed rows and hand the writer an array of rows for a single multi-row insert. The pattern:

$buffer = [];
foreach ($reader->read() as $row) {
    $buffer[] = $transformer->transform($row);

    if (count($buffer) >= 500) {
        $writer->write($buffer);   // array of rows — supported as of 0.5.10
        $buffer = [];
    }
}
if ($buffer) {
    $writer->write($buffer);
}

Pick your batch size against the narrowest constraint, not the widest: SQLite's bound-parameter ceiling (~999 total variables, so ~999 divided by column count) and MySQL's max_allowed_packet both bite before PHP's memory limit does.

Keep mapping external to the pipeline. Column maps, type casts, and skip-rules belong in a config array or a small mapping class per source. When a vendor changes their CSV header for the fourth time you want a one-line diff, not a code change inside a transform loop.

Make reruns idempotent. ETL jobs get re-run after partial failure. Either key the write to an external ID with an upsert, or write to a staging table and swap. A plain insert-based writer will happily duplicate every row if someone retries a job — and after 0.5.10, a batched insert duplicates 500 rows at a time.

Test against an in-memory database. Point the writer at :memory: SQLite in tests, feed it a fixture CSV, assert on row counts and a couple of spot values. This catches row/column shape bugs (the most common failure class) without needing a real schema.

Gotchas and Tips

The 0.5.10 fix has an ambiguity worth testing. "Allow inserting array" means arrays are now accepted as insert rows. But an array is also a legitimate value for a JSON/text column. If you have a JSON column and were previously stringifying it to work around the rejection, re-test after upgrading — the same PHP array may now be interpreted as a batch of rows rather than a single value. Wrap JSON payloads explicitly (json_encode(...) into a string column, or an unambiguous wrapper) rather than relying on inference.

Guard the empty-array case. Passing [] to a batched insert is a no-op at best and a malformed INSERT ... VALUES () at worst, depending on the writer. The if ($buffer) check above is not optional.

Column/value count mismatches fail loudly but late. You'll typically get the error from the DB driver mid-batch with no indication of which source row caused it. Log the source line number alongside the buffer contents so a failure points at the CSV row, not at "batch 14 of 90".

Transactions: one per chunk, not one per file. A single transaction over a 500k-row import will bloat the redo log and make rollback expensive. Per-chunk transactions give you resumability — record the last committed offset and skip forward on retry.

Memory: stream, don't slurp. Read via a generator/streaming reader and never file_get_contents() the source. PHP's array overhead is roughly an order of magnitude above the raw bytes, so a 50 MB CSV can easily become 500 MB of resident arrays if you materialise it.

Source-file quirks will cost you more time than the library will. UTF-8 BOMs, Windows line endings, semicolon delimiters, quoted fields containing newlines, and leading-zero values that arrive as integers are the realistic failure modes. Sniff the encoding and delimiter explicitly, and validate the header row against your mapping before processing any data.

Don't assume Laravel's container is helping you. Because the Symfony DI extension isn't loaded, autowiring of bundle services won't work and any configuration defaults defined in the bundle's Configuration.php won't be applied. Bind explicitly in a service provider and pass your own defaults — this eliminates a whole category of "it works in the test, not in the queue worker" confusion.

Extension points live at the interfaces. Custom readers (API pagination, S3, Excel), custom transformers (currency conversion, deduplication), and custom database writers (upsert semantics, soft deletes) should implement the package's contracts and be bound in your own provider. That is the intended extension route, and it keeps you insulated from internal changes between 0.5.x releases — which are frequent enough that you don't want to subclass concrete internals.

Upgrade procedure for 0.5.x. Read the changelog, bump, run the pipeline test suite against the SQLite fixture, and spot-check one real import in staging before deploying. The release cadence is fast and the fixes are occasionally behavioural rather than purely cosmetic, so a green unit suite alone isn't sufficient evidence that a batched import still produces the rows you expect.