Laravel Chunk Cursor Laravel Package

Technical Evaluation

Architecture fit

• Targeted at a real, well-known gap in Eloquent: chunk()/chunkById() are offset/keyset-paginated (N queries, re-planning cost on deep offsets), while cursor() streams a single result set but does not support eager loading, producing N+1 queries inside the loop. This package advertises chunked cursor iteration with eager loading — a genuine ergonomic win for ETL, exports, reindexing, and nightly aggregation jobs.
• However, Laravel 8+ already ships lazy() and lazyById(), which provide bounded-memory iteration with eager loading via chunkById semantics. The incremental value of this package over the framework baseline is therefore narrow and must be measured, not assumed. Treat "does this beat lazyById() on our data shape?" as the gating question.
• No published dependents and a single release (2024-04-02) suggest a thin wrapper rather than a differentiated engine; the TPM assumption should be "small surface area, easy to read, easy to replace," which cuts both ways.

Integration feasibility

• Low-friction: Composer install, Laravel package auto-discovery, MIT license, no infrastructure. A spike can be run in hours on a read replica or a restored snapshot.
• The real integration work is not the install — it is wrapping the package behind an internal iterator/repository interface so call sites do not import vendor classes directly. Given the health signals, this wrapper is non-negotiable.

Technical risk

• Supply chain / abandonment (high): 2 stars, 0 dependents, one maintainer, one release. No realistic expectation of security patches or Laravel 12+ compatibility.
• Correctness under concurrency (medium-high): cursor/keyset iteration guarantees depend on the ordering column being unique, immutable, and monotonic. Deletes mid-scan can skip rows; inserts can be missed. If the package does not document its cursor column requirements, silent data loss in exports is plausible and hard to detect.
• Eager-loading semantics (medium): if with() relations are loaded per chunk via load(), chunk size directly controls query fan-out and memory ceiling — misconfiguration trades memory savings for query storms.
• Driver-specific behavior (medium): unbuffered MySQL cursors hold the connection; you cannot issue another query on that connection while the cursor is open. If the package's eager loading happens on the same connection mid-stream, behavior varies by driver and Laravel version.
• No visible CI/test signal in the metadata, so verification cost lands on us.

Key questions to resolve before adoption

1. Is iteration keyset-based on a required unique column (e.g., id), or offset-based under the hood?
2. Does it support compound/custom sort keys, select() subsets, joins, groupBy, and whereHas?
3. How exactly is eager loading applied — with() on each chunk query, or load() post-hydration?
4. What are the PHP/Laravel version constraints, and is there a test suite/CI matrix?
5. Behavior under concurrent inserts/deletes during iteration — documented guarantees?
6. Memory and query-count profile versus lazyById() at our real chunk sizes?
7. Connection lifecycle: does it hold a cursor open, and is that compatible with our pooler (PgBouncer) / replica setup?

Integration Approach

Stack fit

• Fits any standard Laravel service using Eloquent against MySQL/Postgres. Best candidates: nightly CSV/report exports, search-index rebuilds, backfill/migration jobs, batch notification fan-outs, and analytics rollups — i.e., jobs where the row count exceeds a few hundred thousand and memory ceilings are enforced by the queue worker.
• Poor fit for request-path code, anything requiring resumable/idempotent checkpoints (the package provides no checkpointing), or queries without a stable unique ordering key.

Migration path

• Do not swap chunk() calls wholesale. Introduce an internal ChunkedIterator contract (e.g., LargeDatasetIterator::each(callable $fn, int $chunkSize)) with two implementations: one delegating to the package, one to native lazyById(). This gives a one-line rollback and removes the vendor from our public API.
• Migrate opportunistically: only touch call sites where profiling shows offset pagination is the bottleneck (deep offsets, large scans) or where cursor() currently causes N+1 storms.

Compatibility

• Pin an exact version in composer.json, add a CI job that runs the package's code path against our supported Laravel/PHP matrix, and add a PHPStan baseline. Since the package is unmaintained-in-practice, plan to vendor or fork under MIT if a framework upgrade breaks it — the small surface area makes that a day of work, not a quarter.

Sequencing

1. Spike (1 week): clone the source, read the iteration logic, benchmark against lazyById() and chunk() on a production-shaped table (≥10M rows) with eager-loaded relations. Capture peak memory, query count, wall time, and correctness (row count parity, no dupes/skips).
2. Wrapper: land the internal iterator interface with the native implementation as the default (zero behavior change).
3. Pilot: enable the package implementation behind a config flag for one low-risk nightly job. Run shadow comparisons for a week (compare output hashes against the native path).
4. Decide: promote to default only if it shows a material win (e.g., >30% wall-time reduction or a memory ceiling we cannot hit otherwise); otherwise drop the dependency and keep the wrapper.
5. Document: record cursor-column requirements, chunk-size guidance, and known concurrency caveats in the runbook regardless of outcome.

Operational Impact

Maintenance

• Effectively self-maintained. One release, one maintainer, no dependents: assume no upstream fixes. Budget a recurring line item for "own the fork" — tracking Laravel major upgrades and republishing internally. MIT makes this legal and cheap; the risk is calendar surprise, not licensing.
• The wrapper isolates us from the worst case, reducing maintenance from "rewrite jobs" to "swap one implementation class."

Support

• No community, no issue-corpus, no Stack Overflow footprint. Debugging falls entirely on internal engineers reading vendor source. Require the adopting team to produce a short internal design note (iteration semantics, cursor column rules, driver caveats) before the pilot exits.
• On-call playbooks must include: "export produced fewer rows than expected" → check for concurrent deletes and ordering-column mutations.

Scaling

• Value scales with table size and row width: the win is largest on wide tables (many columns, JSON blobs) where offset pagination plus hydration dominates. On narrow, index-only workloads the delta versus lazyById() may be noise.
• Watch DB-side effects: chunked eager loading multiplies queries per chunk by relation count. With chunkSize=1000 and 3 relations, that is 4 queries per 1000 rows — fine, but tune chunk size against query-per-second capacity on the replica, not just memory.
• Long-held read cursors on Postgres/PgBouncer can pin a connection in transaction mode; validate against the pooler configuration before enabling for long-running jobs.

Failure modes

• Silent partial iteration: exception or job timeout mid-scan leaves no checkpoint; the job must be idempotent and re-runnable from scratch. Do not assume the package provides resume semantics.
• Row skips/duplicates under concurrent writes to the ordering column — unacceptable for financial or audit exports; require a stable, immutable cursor key and a snapshot/replica read.
• Memory regression from a mis-set chunk size or a relation that hydrates a large collection per model; enforce chunk size via the wrapper's default, not per-call-site.
• Connection contention if eager loading or lazy loading issues queries while an unbuffered cursor is open on the same connection — driver-dependent, needs explicit testing.
• Framework upgrade breakage with no upstream fix — mitigated only by the wrapper and fork readiness.

Ramp-up

• Low for a mid-level Laravel engineer: the API surface is small and mirrors chunk(). The non-obvious cost is conceptual — teaching the team when cursor chunking is correct (stable ordering key, no mid-scan mutation, idempotent consumers) versus when lazyById() is simply the right default. Budget half a day of team education plus the internal design note.

TPM recommendation: adopt only as an experiment behind an internal abstraction, with a hard gate on the benchmark spike. The package solves a real problem but shows every indicator of a hobby project; the expected value is positive only if we can prove a measurable win over Laravel's built-in lazyById() and can absorb a fork at short notice.