Php Etl Laravel Package

Getting Started

Install the package and pull it into a project directory:

composer require marquine/php-etl

Before writing any pipeline, read these three directories in the repo, in order — they are the whole mental model:

1. src/Etl.php — the fluent builder (extract(), transform(), load(), run()).
2. src/Extractors, src/Transformers, src/Loaders — every string type you pass is a class name resolved from these folders. Skim the class list; it is the entire feature surface.
3. tests/ — the fastest source of working, copy-pasteable examples for every extractor/transformer combo.

The canonical first use case is CSV → clean → database table:

use Marquine\Etl\Etl;

$etl = new Etl;

$etl->extract('users', 'csv', ['path' => base_path('data/users.csv')])
    ->transform('users', 'trim', ['columns' => ['name', 'email']])
    ->transform('users', 'rename', ['columns' => ['name' => 'full_name']])
    ->load('users', 'insert', ['table' => 'users'])
    ->run();

The first argument ('users') is a step name — it is how extractors, transformers, and loaders are wired together. Multiple transforms can be stacked onto the same step name and they run in declaration order, row by row.

Once run() completes, the Etl instance holds per-step counters (extracted / transformed / loaded), which is what you should log or assert on rather than trusting the absence of exceptions.

Implementation Patterns

Name-based step wiring. Every call takes a pipeline name as the first argument. This lets you build several independent pipelines on one Etl instance and even feed one pipeline's output into another:

$etl->extract('orders', 'query', ['connection' => 'mysql', 'query' => 'select * from orders']);
$etl->transform('orders', 'callback', fn ($row) => [...$row, 'total' => (float) $row['total']]);
$etl->load('orders', 'insert', ['table' => 'orders_archive']);

$etl->run();

Custom logic lives in callback transformers. Anything the built-in transformers (trim, rename, map, filter, convert, json_decode, …) don't cover should be a closure, not a fork of the package:

$etl->transform('users', 'callback', function (array $row) {
    $row['email'] = strtolower($row['email']);
    $row['created_at'] = now();
    return $row;
});

Return null (or use the filter transformer) to drop a row.

Laravel integration. The database loaders accept a connection key that resolves to an Illuminate connection name, so 'connection' => 'mysql' reads from config/database.php. Wrap each pipeline in a queued job and register an artisan command per pipeline — don't run ETLs inline in a request:

class RunUserImport extends ShouldQueue
{
    public function handle(): void
    {
        (new Etl)
            ->extract('users', 'csv', ['path' => storage_path('imports/users.csv')])
            ->transform('users', 'trim')
            ->load('users', 'upsert', ['connection' => 'mysql', 'table' => 'users', 'key' => ['email']])
            ->run();
    }
}

The upsert/update loaders (with a key) are what you want for idempotent nightly syncs; insert is for append-only loads.

Config-driven pipelines. For anything reused across environments, keep the three arrays (extract, transform, load) in config/etl.php and loop over them in a command. This is the same shape the Symfony bundle uses, and it keeps paths/connections out of code.

Testing. Use the array extractor and the array (or memory) loader so the entire pipeline runs in-process with no filesystem or DB:

$etl = (new Etl)
    ->extract('t', 'array', ['data' => [['name' => ' Alice ']]])
    ->transform('t', 'trim')
    ->load('t', 'array');

$etl->run();

Assert on the output array — this is the cheapest way to pin down transformer behaviour.

Chunking for volume. Rows are processed in a stream, so a csv extractor on a 2 GB file will not blow out memory the way array or a raw query extractor will. Prefer csv/generator sources for big imports and let the loaders batch the inserts rather than materialising the whole dataset.

Gotchas and Tips

Extractor output keys are the contract. Transformers and the insert loader address columns by array key, not by position. If your CSV has a header row with spaces or odd casing, normalise it with rename first, before any other transformer, or the DB insert will silently fail on unknown columns.

query extractors load everything. Unlike CSV streaming, a query extractor pulls the full result set into memory before transforming. For large tables, add a limit/offset and loop, or use a chunked query — do not assume the pipeline will throttle it for you.

Transformers are per-row and synchronous. A callback that hits an API or does a per-row DB lookup will make an import of 100k rows take hours. Batch those cases yourself (collect rows, flush every N) rather than relying on the ETL loop.

Loaders need explicit config. insert requires table; DB loaders generally require connection (or fall back to a default), and upsert requires key. Missing keys throw at run time, not at build time, so validate your pipeline config before run() — a cheap try/catch around the whole pipeline with the step name logged will save you a lot of guessing.

Debugging tip: swap the final load for 'array' (or 'csv' to a /tmp file) and dd() inside a callback transformer to see exactly what the pipeline is handing you at each stage. Nine times out of ten the bug is an unexpected key name or a string where you expected an int.

Type coercion is your job. CSV extractors return strings; insert loaders pass them straight through. Boolean/date/null handling belongs in a convert transformer or a callback — don't expect the DB layer to coerce "0"/"1" into booleans.

No Laravel auto-discovery. There is no service provider; bind Etl yourself or just new Etl inside a job. That's fine — the class is stateless per pipeline.

Extension points. Custom extractors, transformers, and loaders are registered through the registry on the Etl instance (see how built-ins are wired in src/Etl.php). Writing a loader that implements the package's loader interface is the right way to target an exotic sink (S3, a message queue) — much better than post-processing the array loader's output.

PHP version floor. Check composer.json for the required PHP version before adding it to an older Laravel app; the package tracks modern PHP and ^8.x syntax may appear in examples.