Getting Started

What you're actually installing. A very thin (a few hundred lines) wrapper around a streaming JSON tokenizer — historically salsify/json-streaming-parser, newer tags may pull a different engine, so confirm with composer show maxakawizard/json-collection-parser right after install. Its single job: read a JSON file whose top level is (or contains) an array of objects and hand you one decoded array at a time, so peak memory stays roughly "one record", not "one file". Everything else — DB writes, validation, queueing — is yours to write.

Install and first call.

composer require maxakawizard/json-collection-parser

use JsonCollectionParser\Parser;

$parser = new Parser();

$parser->parse('/path/to/products.json', function (array $item) {
    // one object from the top-level array, already json_decode'd
});

// wrapped payloads, e.g. {"data": [ {...}, {...} ]}
$parser->parse('/path/to/products.json', $callback, 'data');

There is no return value and no Illuminate\Support\Collection involved — state lives in variables you capture in the closure. That's the whole API surface worth learning on day one.

Where to look first, in order.

vendor/maxakawizard/json-collection-parser/README.md — the usage block is the entire documented contract; the third argument (the collection key) is the only option you'll use daily.
vendor/maxakawizard/json-collection-parser/src/Parser.php and src/Listener/ — read the ~100 lines. Knowing exactly when the listener emits an item (per complete object vs. per token) tells you whether you can trust array $item typing.
The underlying streaming parser's source — every weird error message (offset/line numbers, "unexpected token") originates there, not here.

The canonical first use case. An upstream (ERP, Shopify, Stripe export, partner API dump) hands you a 400 MB products.json. The "obvious" Laravel code — json_decode(file_get_contents($path), true) — dies at ~512 MB on a stock PHP-FPM worker, and Storage::json() is the same thing with nicer syntax. Replace it with:

$parser->parse(Storage::disk('local')->path('imports/products.json'), function (array $item) {
    Product::updateOrCreate(['sku' => $item['sku']], $item);
});

Prove it on a 100-row fixture first, then on a 1 GB file, watching memory_get_peak_usage(true) — if peak tracks file size, you've kept a reference somewhere (usually $items[] = $item).

Implementation Patterns

Isolate it behind one service class. The package has one maintainer, ~130 stars, and zero dependents in the wild — treat it as a swappable dependency. Wrap it once, construct a fresh Parser per run (don't share one across files or across queued jobs; the internal listener holds parse state), and the rest of your app never imports the namespace:

final class JsonCollectionStream
{
    public function each(string $absolutePath, callable $onItem, ?string $collectionKey = null): void
    {
        (new \JsonCollectionParser\Parser())->parse($absolutePath, $onItem, $collectionKey);
    }
}

If you later need LazyCollection semantics, halaxa/json-machine is a drop-in conceptual replacement and only this one file changes.

Always parse into a batched bulk write, never per-row Eloquent. The callback is your foreach body; treat it as a hot loop:

$batch = [];
$size  = 1000;
$now   = now();

$flush = function () use (&$batch, $size, $now) {
    if ($batch === []) {
        return;
    }
    DB::table('products')->upsert($batch, ['sku'], ['name', 'price', 'updated_at']);
    $batch = [];
};

$stream->each($path, function (array $item) use (&$batch, $size, $flush, $now) {
    $batch[] = [
        'sku'        => (string) ($item['sku'] ?? ''),
        'name'       => (string) ($item['name'] ?? ''),
        'price'      => (float)  ($item['price'] ?? 0),
        'created_at' => $now,
        'updated_at' => $now,
    ];

    if (count($batch) >= $size) {
        $flush();
    }
}, 'data');

$flush(); // never forget the tail batch

DB::disableQueryLog() before the loop and set_time_limit(0) (or run it as a queued job) are effectively mandatory for anything over a few hundred thousand rows.

An Artisan command is the right home. Signature like import:products {file} {--key=data} {--batch=1000} {--limit=0} {--dry-run}. The --limit and --dry-run flags pay for themselves: you can smoke-test against a real 400 MB file in 5 seconds instead of importing junk into production. Log progress every N items with row count and peak memory — you have no total count without a second pass over the file, so don't promise a percentage.

Per-item try/catch, not one big try. An exception thrown inside the callback propagates out of parse() and kills the whole run. If a single malformed record shouldn't abort a 20-minute import:

$failures = 0;
$samples  = [];

try {
    // ... build row, validate with Validator::make(...)
} catch (\Throwable $e) {
    $failures++;
    if (count($samples) < 20) {           // cap it — this array is the #1 leak
        $samples[] = $e->getMessage();
    }
    return;
}

Report the failure count and a capped sample at the end; dump the full list to a log/table, not to memory.

Make re-runs safe. Streaming parsers validate lazily — you can successfully process 900k records and then hit a syntax error on the last token. Combine that with a queue worker timeout at minute 58 and you will re-run mid-file. Use upsert on a natural key, or write to a staging table tagged with an import_run_id and swap/promote at the end. This single decision removes most "half-imported" incidents.

Remote files: download, then parse. The parser wants a local, seekable path. Storage::disk('s3')->path() won't help for S3; stream the object down to storage/app/imports/{uuid}.json first, parse, then delete in a finally. Budget the disk space — a 400 MB JSON needs 400 MB free, plus headroom for MySQL's own work.

Queue strategy. Don't dispatch() one job per item (queue serialization cost dwarfs the parse) and don't serialize a 1000-row batch into a job payload either. Two workable shapes: (a) parse synchronously in a single long-running job with a generous $timeout and $tries = 1; (b) parse into a staging table in one job, then Bus::batch() chunked promotion jobs off that table. (b) is the one that survives a mid-import failure.

Testing. Fixtures beat mocks here — the package isn't worth mocking, but the file format is worth pinning:

public function test_imports_products(): void
{
    $this->artisan('import:products', [
        'file' => base_path('tests/Fixtures/products.json'),
    ])->assertSuccessful();

    $this->assertDatabaseCount('products', 3);
    $this->assertDatabaseHas('products', ['sku' => 'ABC-1']);
}

Include a fixture with an empty array [] (callback never fires), one with a missing field, one with a nested object, and one with multi-byte UTF-8. Those four catch almost every regression.

Gotchas and Tips

array $item is a lie you can get away with only sometimes. The type hint is fine for [{...}, {...}], but if any element of the array is a scalar or nested array and the listener hands you that shape, a bare function (array $item) will TypeError. If the source isn't contractually "array of flat objects", drop the hint and check is_array($item) yourself.

Streaming protects you from the file, not from yourself. Peak memory == largest single record + whatever you accumulate. The three classic leaks: $all[] = $item, an unbounded $errors[], and Log::info(json_encode($item)) on every row (the log buffer alone can double your footprint). Cap arrays, use error_log/a channel with a buffer, and periodically let $batch go out of scope via $flush().

Eloquent accessors, casts, mutators, and timestamps do not run. DB::table()->insert() bypasses the model entirely — you set created_at/updated_at by hand, you cast "12.50" strings to floats by hand, and Carbon::parse() on a million rows is a measurable sliver of your runtime. Parse dates with date()/strtotime() or insert the raw ISO string if MySQL accepts it.

JSON key drift will bite you. Records that omit a key produce rows with fewer columns and insert() throws on the inconsistent column set; records with extra keys silently drop data unless you whitelist columns explicitly (do whitelist — never array_keys($item) into a column list for an untrusted file).

No rewind, no resume, no random access. You cannot seek to record 500,000, and a half-finished parse can't be continued. If resumability matters, add your own bookkeeping: record a run_id and a monotonic item index per batch, or split the source file into chunks before parsing and record completed chunks.

Empty arrays and wrong collection keys both look like success. {"data": []} and {"payload": [...]} (parsed without the 'data' key) both produce zero callbacks and zero exceptions. Always assert $count > 0 before reporting success, and treat "0 rows" as a likely configuration error, not a valid outcome.

File hygiene. Byte-order marks, a leading UTF-8 BOM from an Excel export, and trailing commas all fail deep in the tokenizer with an offset rather than a friendly message. When debugging a bad file, bisect by byte offset (the exception's position is your only clue), and verify encoding with mb_check_encoding($contents, 'UTF-8') on a sample — you cannot mb_convert_encoding a file you can't hold in memory, so fix it upstream or convert line-wise before parsing.

NDJSON is not supported. One-JSON-object-per-line files (.jsonl, streaming API dumps, log exports) will fail immediately. That's a fopen/fgets/json_decode loop, not a job for this package.

Transaction scope. One transaction around the whole file means a giant redo log, lock contention with live traffic, and a full rollback on the last-row error. One transaction per flush is the sweet spot — atomic per 1000 rows, re-runnable thanks to upsert.

upsert/insertOrIgnore are not validators. insertOrIgnore swallows all constraint violations, not just duplicate keys — a NOT NULL breach and a duplicate look identical (both silently dropped). If data quality matters, use upsert with explicit update columns, or stage-then-validate.

Extending it. The realistic extension point is your own callback, not the package internals. Subclassing Parser or swapping the listener to get per-item byte offsets (for resumability) is possible but couples you to a private API with one maintainer — if you go there, pin the exact version in composer.json and add a contract test that fails loudly on upgrade.

Benchmark before you trust it. Run one import against a file at least 10× your median production file. Watch peak memory and wall time; if peak memory grows linearly with row count, you have a reference leak in your callback, and no amount of streaming in the parser will save you.

Json Collection Parser Laravel Package

Getting Started

Implementation Patterns

Gotchas and Tips