Getting Started

Install and pin

composer require fab2s/yaetl

Pin the version in composer.json ("fab2s/yaetl": "^1.x" or whatever the current tag is). It is a small, single-maintainer package (65 stars), so a lockfile is your friend.

Where to look first

README.md — the examples are the fastest way to see the fluent API.
src/Yaetl.php — the orchestrator (extract() / transform() / load() chain and run()).
src/Extractors/, src/Transformers/, src/Loaders/ — the shipped PdoExtractor, PdoLoader, CallUserFunc, etc.
The task interfaces / abstract classes at the root of src/ — these are the contracts you will implement for anything custom.

Minimal first pipeline (DB → DB)

use fab2s\Yaetl\Yaetl;
use fab2s\Yaetl\Extractors\PdoExtractor;
use fab2s\Yaetl\Transformers\CallUserFunc;
use fab2s\Yaetl\Loaders\PdoLoader;

$pdo = DB::connection()->getPdo();

$etl = (new Yaetl)
    ->extract(new PdoExtractor($pdo, 'SELECT id, email, created_at FROM legacy_users'))
    ->transform(new CallUserFunc(function (array $record): array {
        $record['email'] = strtolower(trim($record['email']));
        return $record;
    }))
    ->load(new PdoLoader($pdo, 'users', ['id', 'email', 'created_at']));

$etl->run();

First real use case: a CSV/legacy-table backfill into a Laravel table. Start with a read-only extractor, one no-op transformer, and a PdoLoader pointed at a staging table. Verify row counts and sample records before adding transformations.

Implementation Patterns

One responsibility per task. Keep extractors to “yield rows”, transformers to “one row in / one row out”, and loaders to “persist a row or buffer”. This makes each piece trivial to unit test and reorder.

Closures vs classes. CallUserFunc is fine for trivial mapping (trim, strtolower, date parsing). As soon as a transformer needs dependencies (a service, a repository, a lookup cache), write a class implementing the transformer interface and resolve it from Laravel’s container. It also makes it mockable in tests.

Custom extractor for non-PDO sources. Implement the extractor contract and yield rows:

class ApiExtractor implements ExtractorInterface
{
    public function extract(): \Generator
    {
        foreach ($this->client->paginate('/users') as $page) {
            foreach ($page['data'] as $row) {
                yield $row;
            }
        }
    }
}

Custom loader for upserts / bulk inserts. The shipped PdoLoader is an insert-style loader. If you need INSERT ... ON DUPLICATE KEY UPDATE, buffered multi-row inserts, or writes to multiple tables, implement your own loader that accumulates records in an internal buffer and flushes on batch boundary and on completion.

Chaining transformers. Multiple transform() calls run in pipeline order. Put cheap filters first, expensive enrichment last, and normalization (trim/cast/rename) at the front so downstream tasks can assume a consistent shape.

Batch sizing. Tune the batch size to balance memory against round-trips and transaction size. Smaller batches (500–1000) keep transactions short and reduce lock contention; larger batches reduce overhead. Whatever the batch size, make sure the final partial batch is flushed.

Laravel integration. Treat yaetl as a plain library — there is no service provider, facade, or config file. Wrap it:

An Artisan command with $this->output / ProgressBar for manual/backfill runs.
A queued job (ShouldQueue, $timeout, $tries = 1) for scheduled syncs. Use withoutOverlapping() on the scheduler.
Put connection names, batch size, table names, and column maps in config/ so the same pipeline can target staging vs production.
Use DB::connection('source')->getPdo() and DB::connection('target')->getPdo() — see the unbuffered-query gotcha below.

Testing. Use an array/in-memory extractor plus a recording loader (or sqlite :memory:) to assert transformations without touching MySQL. Test transformer classes in isolation; test the full run() against SQLite schema fixtures.

Error handling. Wrap run() in try/catch, log the failing row context from inside a transformer (or a dedicated logging transformer), and design the load to be idempotent so a rerun is safe. If you batch commits, record progress so a partial run can resume.

Observability. Add a lightweight transformer that increments counters / logs every N records. Expose the counts from the command output. A per-stage “tap” transformer is the cheapest debugging tool you have.

Gotchas and Tips

Lazy generators, late exceptions. Construction is cheap; nearly everything (query errors, type errors in transformers) surfaces during run(). Don’t assume a pipeline is valid because it built.
Flush the last batch. Loaders that buffer will silently drop the trailing partial batch if you don’t finalize on pipeline completion. Verify final row counts on a dataset that isn’t a multiple of the batch size.
Don’t reuse stateful task instances. A loader or transformer that holds a prepared statement, buffer, or counter should be constructed per run. Reusing a Yaetl object across jobs in a long-lived queue worker is a classic source of duplicated or missing rows.
MySQL unbuffered queries. Reading with a cursor from one connection while writing on the same connection can throw Cannot execute queries while other unbuffered queries are active. Either use separate source and target connections, or explicitly configure buffered queries. This is the most common production failure.
Transaction scope. Batching implies partial success. If you need all-or-nothing, you will be holding one large transaction and defeating the memory benefits. Prefer idempotent loads (upsert or delete-then-insert by key range) over one giant transaction.
Memory creep from “innocent” transformers. Any sort, group, unique, or “collect then process” transformer materializes the full dataset. The pipeline is streaming only if every stage is streaming.
Skip vs stop semantics. Returning null/false from a transformer may skip, pass through, or abort depending on the implementation — test it explicitly before relying on it for filtering. When in doubt, filter in the extractor’s SQL/query instead.
Column mapping and types. PdoLoader expects record keys to match the target columns exactly. Alias in the extract SQL (SELECT old_email AS email) to reduce transformer noise. PDO returns strings for most types; cast timestamps and booleans deliberately.
No Laravel conventions. No provider, no facade, no published config. Don’t look for one in the vendor directory. Wire it yourself in a command/job and keep the pipeline definition in a small factory or service class.
Concurrency. Two overlapping runs against the same target will race. Guard with Laravel’s withoutOverlapping() or a DB advisory lock, and make loaders idempotent.
Extension points. Custom extractor/transformer/loader interfaces and the package’s abstract base classes are the intended seams. Prefer implementing the interface over subclassing shipped tasks — the package is small and internals are not a stable API.
Debugging recipe. Reproduce with a WHERE id < 1000 extractor (or a limit on the source query), then add a dump/log transformer immediately after extraction and after each transform. The first stage where the record shape goes wrong is your bug.
Version risk. Low star count and a single maintainer mean API changes can land without much ceremony. Lock the version, read the changelog before upgrading, and keep a smoke test that runs a tiny pipeline end-to-end.

Yaetl Laravel Package

Getting Started

Implementation Patterns

Gotchas and Tips