Laravel Chunk Cursor Laravel Package

Getting Started

composer upgrade itsimiro/laravel-chunk-cursor   # or: composer require itsimiro/laravel-chunk-cursor

The package is tiny and (almost certainly) ships without a service provider or config file, so there is nothing to publish or register. Read vendor/itsimiro/laravel-chunk-cursor/src/ before writing any code — it is a handful of files, and the exact method name/signature is the whole API contract. Historically this package exposes a chunkCursor() method made available on the query builder (via a macro or a custom Eloquent builder), called like:

Post::query()
    ->with(['author', 'tags'])      // eager loading works — this is the point
    ->chunkCursor(500, function ($posts) {
        foreach ($posts as $post) {
            // ...
        }
    });

The first real use case is a backfill or export job where you currently have to choose between:

• ->cursor() — flat memory, but relations lazy-load row-by-row (N+1 storm), and
• ->with(...)->chunk(...) — eager loading works, but OFFSET pagination degrades on big tables and can skip/duplicate rows when the table is being written to.

chunkCursor is meant to give you both: eager loading and keyset-style iteration without a growing OFFSET.

Sanity-check it in tinker against a table you know the size of:

DB::enableQueryLog();
Post::with('author')->chunkCursor(200, fn ($posts) => null);
dump(count(DB::getQueryLog())); // expect roughly rows/200 * (1 + #relations)

Implementation Patterns

Chunked export / nightly sync. This is the canonical fit. Keep the callback doing work only, never accumulating:

Post::with('author')
    ->where('updated_at', '>=', $since)
    ->orderBy('id')
    ->chunkCursor(1000, function ($posts) use ($writer) {
        foreach ($posts as $post) {
            $writer->write($post->toExportRow());
        }
    });

Queue fan-out. Use the chunk callback to dispatch one job per chunk (or per model) instead of one per row — this is where memory and queue overhead both drop:

Post::with('author')->chunkCursor(500, function ($posts) {
    ProcessPostChunk::dispatch($posts->pluck('id'));
});

Backfill with idempotent writes. Because you are no longer using OFFSET, a concurrent insert won't shift your window. Still make the per-row operation idempotent (upsert, updateOrCreate, or a whereNull('processed_at') guard) so a re-run is safe.

Compose with existing builder constraints freely — with, withCount, select, whereHas, scopes all work because you're still on an Eloquent builder. Prefer select()-ing only what the callback touches; the eager-loaded relations still fire their own whereIn per chunk.

Tune the chunk size per relation count, not per row count. A chunk of 1000 with 3 eager-loaded relations is 4 queries per chunk; at 1000 rows that's fine, at 50 rows it's wasteful. A good default is (int) (2000 / (1 + count($relations))).

Prefer it over chunk() when: the ordering column is indexed and unique, the table is large, and rows are being written concurrently. Prefer core lazyById() when you don't need eager loading at all — it's maintained by Laravel and needs no extra dependency.

Gotchas and Tips

• Maturity risk is real. 2 stars, 0 dependents, last release April 2024. Pin the exact version in composer.json and write one integration test that asserts the chunking behavior you rely on, so a future core-Laravel upgrade (or the package silently breaking) is caught by CI. If this is load-bearing, budget an afternoon to vendor the logic yourself — it's small.
• A deterministic, unique orderBy is mandatory. Keyset iteration is only correct if the cursor column is unique (usually id). Ordering by created_at alone will silently skip rows that share a timestamp. Use orderBy('id') or a composite (created_at, id) if the package supports it; otherwise filter on id.
• Don't mutate the cursor column (id) inside the callback — and be wary of updating the ordering column at all, since the next chunk's boundary is derived from the last row seen.
• Unbuffered cursor + eager loading can collide. A true PDO cursor holds an open unbuffered statement on the connection; issuing the relation whereIn queries mid-stream can throw "Cannot execute queries while other unbuffered queries are active" on MySQL. If you hit this, the package is (or should be) buffering the chunk before eager-loading. If not, use with()-free iteration plus a second whereIn(...)->get() per chunk, or force a separate connection via ->on('other').
• Memory only stays flat if the callback doesn't retain. $posts->push(...), collect() accumulation, or capturing models in closures will defeat the whole exercise. If you need a running aggregate, keep scalars.
• N+1 sneaks back in via accessors. Eager loading a relation doesn't help if your Blade/export calls $post->author->company — add that nested relation to with() ('author.company').
• Debug query behavior with the query log, not with memory_get_peak_usage alone: DB::listen() and count how many queries fire per chunk. If it's 1 + relations per chunk you're fine; if it's 1 + rows, eager loading isn't being applied.
• Long-running jobs and transactions: don't wrap the whole iteration in a transaction — you'll hold locks for minutes. Commit per chunk.
• Stopping early: most chunk-style APIs honor return false from the callback. Verify that in the source before relying on it for "process until X" semantics; if unsupported, throw and catch.
• Extension point: if chunkCursor is a builder macro, you can Builder::macro('chunkCursorWith', ...) to bake in your team's default chunk size and relation set — a cheap way to standardize usage across jobs.
• Test it against the same DB engine as production. Keyset behavior around NULL cursor values and string ordering differs between SQLite (dev) and MySQL/Postgres (prod).