cerbero/lazy-json-pages
Cerbero LazyJsonPages lets you consume paginated JSON APIs as Laravel lazy collections. Point it at an endpoint, describe how pagination works, and iterate results without loading every page into memory. Ideal for large API datasets, sync jobs, and streaming responses efficiently.
Install
composer require cerbero/lazy-json-pages
There is no config file to publish, no migration, and (for a framework-agnostic package) no service provider you need to touch — you interact with it through a single static entry point, Cerbero\LazyJsonPages\LazyJsonPages.
First use case: drain a paginated REST endpoint into a lazy collection
The canonical use is "give me every record from this API, but don't blow up memory and don't take 40 sequential round-trips to do it."
use Cerbero\LazyJsonPages\LazyJsonPages;
$books = LazyJsonPages::from('https://api.example.com/books')
->items('data') // JSON path to the array of items on each page
->totalItems('meta.total') // JSON path to the total record count
->perPage(50) // how many items each page returns
->collect();
// $books is an Illuminate\Support\LazyCollection
$books->each(fn (array $book) => dump($book['title']));
If the API tells you the total up front, totalItems() + perPage() is all the package needs to compute how many pages exist — it then fires those page requests concurrently and yields items in order.
Where to look first
README.md — the pagination "recipes" (total-based, page-number, offset/limit, cursor) are the whole mental model.src/LazyJsonPages.php — this is the fluent builder; skim the public methods and you have the entire API surface.tests/ — the most reliable documentation of edge cases: how missing totals, empty pages, and non-200 responses are handled.Start with a small ->take(5) against a real endpoint before writing any pipeline around it. The whole package is a lazy generator, so ->take(5) stops after the first page or two and gives you a fast feedback loop.
The builder is a set of knobs describing how your particular API paginates. Pick the dialect:
// 1. Total-count APIs (Stripe-ish, GitHub-ish)
LazyJsonPages::from($url)
->items('data')
->totalItems('total_count')
->perPage(100);
// 2. Page-number APIs (?page=1, ?page=2, ...)
LazyJsonPages::from($url)
->items('results')
->pageKey('page')
->firstPage(1)
->perPage(25)
->totalItems('meta.pagination.total');
// 3. Offset/limit APIs
LazyJsonPages::from($url)
->items('rows')
->offset('offset')
->perPage(500)
->totalItems('count');
// 4. "Is there a next page?" APIs (has_more / next_cursor)
LazyJsonPages::from($url)
->items('data')
->cursor('next_cursor') // follow the cursor the API hands back
->hasMorePages('has_more'); // or let the API decide when to stop
Getting this wrong is the #1 cause of infinite loops or silently truncated results, so verify the exact field names against a single curl response first.
Async is the selling point, but concurrency is also the thing that gets you 429'd. Treat it as a rate-limit dial:
LazyJsonPages::from($url)
->items('data')
->totalItems('total')
->perPage(100)
->concurrency(5) // 5 pages in flight at once
->timeout(30)
->retry(3); // retry flaky pages before failing the whole stream
Rule of thumb: concurrency × perPage should stay under the API's documented per-minute budget, and lower concurrency is the correct answer for anything with a shared quota.
Because the result is a LazyCollection, this is where the memory win shows up. Don't ->collect() a million rows:
LazyJsonPages::from($url)
->items('data')
->totalItems('total')
->perPage(100)
->chunk(1000)
->each(function (LazyCollection $chunk) {
Book::upsert(
$chunk->map(fn ($b) => [
'external_id' => $b['id'],
'title' => $b['title'],
'updated_at' => now(),
])->all(),
['external_id'],
['title', 'updated_at'],
);
});
Wrap the whole thing in a queued job so the artisan invocation returns immediately and failures land in your queue's retry/DLQ machinery rather than a request cycle.
mapThe package yields raw decoded arrays/objects, not DTOs. Push transformation into the lazy pipeline so it happens once per item and never gets materialized:
LazyJsonPages::from($url)
->items('data')
->totalItems('total')
->map(fn (array $raw) => BookData::fromApi($raw))
->filter(fn (BookData $b) => $b->isPublished())
->each(fn (BookData $b) => $b->persist());
This keeps the transformation logic testable in isolation from the HTTP concerns.
Because every API needs a slightly different knob combination, don't scatter LazyJsonPages::from() across the codebase. Give each integration its own class:
final class BookApi
{
public function stream(): LazyCollection
{
return LazyJsonPages::from(config('services.books.url') . '/books')
->withHeaders(['Authorization' => 'Bearer ' . config('services.books.token')])
->items('data')
->totalItems('meta.total')
->perPage(100)
->concurrency(4)
->timeout(30)
->retry(3);
}
}
Now the API's pagination dialect, auth, and tuning live in one place, and the caller just gets a LazyCollection it can map/filter/chunk against.
The output is a plain LazyCollection, so it composes with the rest of the framework for free — merge multiple endpoints, union with a DB query, etc. This is the main reason to prefer it over a bespoke Guzzle Pool implementation.
Lazy collections are single-pass. Once you iterate, you can't iterate again. If you need two passes, either ->collect() (accepting the memory cost) or, better, do one pass with a tee-style ->each() that fans out to both consumers.
Http::fake() likely won't intercept these requests. The package does its own async HTTP work rather than going through Laravel's Illuminate\Support\Facades\Http client, so the usual test-time fakes can miss. Verify with a single test before building a suite around it; if it bypasses the fake, you'll need Guzzle MockHandler/HandlerStack injection or a local stub server. Budget an hour for this — it's the most common early stumbling block.
The items() path must point at an array. Point it at the wrong node and you'll get a confusing type error or an empty stream. Similarly, totalItems() returning 0 or null is treated as "unknown" — if the API reports totals inconsistently, prefer the hasMorePages()/cursor strategy over fighting the total.
Infinite pagination is a real failure mode. Any time you use hasMorePages() or a cursor, the stop condition lives in the remote payload. If the API keeps returning has_more: true (a bug on their side) your job runs forever. Add a belt-and-braces guard by chaining ->take($hardLimit) on the lazy collection, and log when you hit it.
Errors surface mid-stream, not at construction. The object builds fine; the exception throws when iteration reaches the failing page. That means try/catch must wrap the consumption (->each, ->chunk), not the LazyJsonPages::from() call. ->retry(3) handles transient 5xx/timeouts, but a persistent 401 will only blow up when the iterator reaches it — which is why a small ->take(5) smoke test in staging is worth the 30 seconds.
Set an explicit timeout. Guzzle's default will cut off slow, heavy pages (large perPage values plus a slow backend) and the resulting exception looks like a random retry storm. ->timeout(30) or higher for anything with joins on the far side.
Don't use it for single-page requests. The async machinery and pagination detection are overhead when the endpoint returns everything at once — a plain Http::get() is simpler and fakeable.
Auth and headers belong on the builder. Use the builder's header/options methods rather than a globally configured Guzzle client, otherwise the concurrency pool and your custom client config can disagree about middlewares, retries, or base URIs.
Low bus factor. 195 stars, zero dependents, one maintainer. It's MIT and small enough to vendor or fork if needed, but pin the version in composer.json and keep a thin wrapper class (as in the pattern above) so replacing it later is a one-file change rather than a codebase-wide refactor. Treat it as a convenience, not infrastructure.
Extension points are minimal by design. There's no pluggable "pagination strategy" interface to implement for exotic APIs (GraphQL cursors, link headers). If your API doesn't fit the supplied knobs, the pragmatic path is to drop to Guzzle's Pool directly for that one integration and keep LazyJsonPages for the endpoints that fit cleanly.
How can I help you explore Laravel packages today?