Etl Bundle Laravel Package

Getting Started

boutdecode/etl-bundle ships its first tagged release at v0.1.0. Treat this as a pre-stability release: the changelog entry contains no feature list, no migration notes and no BC guarantees, so the package's public surface has to be discovered from the source rather than from documentation. Budget an hour of reading before you write production code against it.

Before installing, check these in order:

1. composer.json — required PHP and illuminate/* constraints. A 0.1.0 package is often pinned to a narrow Laravel range; confirm it supports your version rather than assuming.
2. README.md and any docs/ folder — v0.1.0 releases frequently ship with a README that lags the code. Prefer the source over the prose when they disagree.
3. src/ service provider — this tells you what is registered: config keys, bindings, console commands, publishable assets.
4. config/*.php — the real contract of the package. Every driver, connection and default lives here.
5. tests/ — the fastest way to learn intended usage; feature tests usually show the smallest working configuration.

Install:

composer require boutdecode/etl-bundle:0.1.*

Pin to 0.1.* rather than ^0.1. Under semantic versioning, anything may break between 0.x minors, and you do not want a composer update to silently change extraction or loading semantics.

First use case to attempt: a single-source, single-destination pipeline, no transformations — read rows from one connection, write them to a table, verify row count and column mapping. Prove the plumbing works before layering transforms on top. If the package exposes an Artisan command (likely, given it is a Laravel bundle), run php artisan list after install and look for an ETL/import/export namespace; php artisan vendor:publish --provider="Boutdecode\..." will surface the config if publishing is supported.

If the package has no docs and no tests, that is itself the finding: it is not yet a dependency to build on, only one to evaluate.

Implementation Patterns

Keep pipeline definitions in config, not in controllers. ETL bundles almost always derive their value from declaring extract/transform/load steps as data. Put each pipeline in config/etl.php (or a dedicated config file the provider publishes) and resolve it by name at the call site. This keeps schedules, retries and credentials out of application code and makes pipelines diffable in review.

Wrap every pipeline in a job, then schedule the job. Do not run an ETL synchronously in a request. A thin job that resolves the pipeline and dispatches it gives you queue retries, timeouts, failed() hooks and Horizon visibility for free:

final class RunOrdersSync implements ShouldQueue
{
    use Queueable, InteractsWithQueue;

    public int $timeout = 900;
    public int $tries = 1;          // usually wrong to auto-retry a partial load
    public bool $failOnTimeout = true;

    public function handle(PipelineRunner $runner): void
    {
        $runner->run('orders_sync');
    }

    public function failed(Throwable $e): void
    {
        // alert, mark watermark as unadvanced, page the owner
    }
}

Set tries = 1 unless the package guarantees idempotent loads. Auto-retrying a load that partially committed duplicates rows.

Separate the three stages behind your own seams. Even at 0.1.0, wrap the package's extractor/loader behind small interfaces of your own. When the package renames a method in 0.2.0, you edit one adapter class instead of forty call sites. This is the single highest-value habit with a 0.x ETL dependency.

Model incremental loads with a watermark you own. Store last_synced_at or last_id in your own table, not in package-managed state. Pass it into the pipeline, then advance it only after the load commits successfully. This lets you re-run a failed window and keeps you independent of the package's state handling.

Chunk and stream. For anything beyond trivial volume, configure batch size explicitly and avoid the package's "load everything" path. Confirm whether chunk() returns a generator or a materialized collection — the difference decides whether a 500k-row sync exhausts memory.

Pair with Laravel's own tooling for the edges. Use Http or Storage fakes in tests, DB::transaction() around single-destination loads, and Log::withContext(['pipeline' => ...]) so ETL log noise is filterable.

Gotchas and Tips

No documented surface at v0.1.0. The release notes are a bare changelog link. Verify each of these yourself before relying on them: does the provider auto-register (package discovery), is the config actually publishable, are commands namespaced, are there events you can listen to for per-row failures?

Assume breaking changes at every minor bump until 1.0. Maintain a small integration test of your own that exercises one full pipeline against a SQLite or test MySQL database. Run it on every composer update. That test is your early-warning system for a 0.2.0 that reorders arguments.

Debugging failed loads is the hard case. When a load half-succeeds, you need to know the transaction boundary. Check whether the loader commits per batch (fast, but leaves partial data) or per pipeline (atomic, but memory-hungry). Wrap the load in DB::transaction() yourself if the package does not, and confirm your destination supports it — MySQL InnoDB and Postgres do; MyISAM does not.

Credentials and connection ambiguity. If the bundle accepts a connection name, confirm it resolves through config('database.connections') and not a second, package-specific credential config. Two sources of truth for DB credentials is a common source of "it works locally, fails in staging".

Memory and timeouts. Bump memory_limit and queue worker timeouts before the first large run, not after. ETL failures at 95% completion are the expensive ones.

Extension points worth mapping early. Look for interfaces you can bind your own implementations to — an extractor contract, a transformer contract, a loader contract. If the package only offers concrete classes, favour composition (wrap it) over inheritance (extend it); subclassing a 0.x class couples you to its internals.

Logging and observability. Confirm failures surface as exceptions rather than swallowed return codes. If the package has a quiet or ignore_errors flag, leave it off in production — silently dropped rows are the worst failure mode in ETL.

Do not put this on a critical path yet. At 0.1.0 with no release documentation, use it for internal syncs and reporting pipelines where a failure is recoverable. Hold off on anything that mutates billing, inventory or customer-of-record data until the package reaches 1.0 or demonstrates a stable release cadence.