rodenastyle/stream-parser
Parse large CSV and other data files in PHP without exhausting memory. rodenastyle/stream-parser offers a simple, memory-efficient streaming API to read and process files chunk by chunk—ideal for Laravel imports, exports, and ETL jobs on big datasets.
Install
composer require rodenastyle/stream-parser
Laravel auto-discovers Rodenastyle\StreamParser\StreamParserServiceProvider and registers the StreamParser facade alias. On Lumen or a locked-down config/app.php, register the provider manually and add 'StreamParser' => Rodenastyle\StreamParser\StreamParserFacade::class to your aliases.
There is nothing else to set up for the common path — no migrations, no published config, no service bindings. It is a thin, generator-based façade over league/csv, XMLReader, and a hand-rolled JSON streamer.
Where to look first
src/StreamParser.php — the factory (csv(), json(), xml(), txt()) and the driver entry points.src/Parsers/ — one class per format. Read the driver for the format you actually use; that is where the real behaviour (delimiters, root node handling, JSON shape expectations) lives.from(), each(), get(), toCollection(), and transform() are defined. Confirm the exact terminal methods before designing around them.First use case
A 200 MB customer CSV you cannot file_get_contents():
use Rodenastyle\StreamParser\StreamParser;
StreamParser::csv()->from(storage_path('app/imports/customers.csv'))
->each(function (array $row) {
Customer::insert([
'email' => $row['email'],
'name' => $row['name'],
]);
});
Verify it is actually streaming before you trust it: log memory_get_peak_usage(true) from inside the closure on a large fixture. If the number tracks file size, you have accidentally fallen back to a non-streaming path (usually ->get() instead of ->each()).
Artisan command as the host. This package belongs in CLI, not in a controller. A php artisan import:customers {path} command gives you unlimited execution time, $this->output->progressAdvance(), and clean failure reporting. Web requests that stream multi-hundred-MB files will hit max_execution_time and tie up PHP-FPM workers.
Batch inserts, not per-row saves. Accumulate rows in a local array and flush every 500–1000 records:
$buffer = [];
StreamParser::csv()->from($path)->each(function ($row) use (&$buffer) {
$buffer[] = $this->normalize($row);
if (count($buffer) >= 500) {
Customer::upsert($buffer, ['email'], ['name']);
$buffer = [];
}
});
if ($buffer) {
Customer::upsert($buffer, ['email'], ['name']);
}
upsert() avoids the N+1 and gives you idempotent re-runs — important because a crashed import restarts from row zero.
Map with transform(), then iterate. Keep the mapping step pure and the side-effect step separate:
StreamParser::json()->from($path)
->transform(fn ($item) => new ImportedOrder($item))
->each(fn (ImportedOrder $order) => $order->persist());
This is also the cleanest place to filter: return null or throw a skippable exception from transform() and guard in each(). If the driver exposes an iterator/generator, a plain foreach is often more readable than each() for early continue/break logic.
Queue it. Pass the file path into the job, never the parsed rows. Streaming inside a queued job gives you retries, $timeout, and $tries for free. For multi-file drops, dispatch one job per file and chain a finalization job.
One interface, many formats. Because all drivers share the same fluent API, dispatch on extension with a tiny factory:
$parser = match (strtolower(pathinfo($path, PATHINFO_EXTENSION))) {
'csv' => StreamParser::csv(),
'json' => StreamParser::json(),
'xml' => StreamParser::xml(),
default => StreamParser::txt(),
};
$parser->from($path)->each($handler);
This is the package's main selling point — a single import pipeline that accepts whatever the client uploads.
Storage integration. The drivers want a local path. For local disks, Storage::disk('local')->path($relative) is correct. For S3 you must download first (Storage::readStream() into a tmpfile() or ->get() to a temp path); streaming the S3 body through this package is not supported.
Testing. Use UploadedFile::fake() fixtures or real files in tests/Fixtures/ and call $file->getRealPath(). Because the parser is deterministic and stateless, a small fixture per format gives you solid coverage of your normalization layer — which is where the bugs actually live.
get()/toCollection()/toArray() are the trap. They materialize the entire file into memory. If your code contains ->get() on a user-supplied upload, you have a streaming parser that does not stream. Reserve those terminals for small config-like files only.StreamParser::csv()->from(...) returns a fresh, non-rewindable stream. Do not cache an instance on a service class and iterate it twice; the second pass silently yields nothing.XMLReader requires you to target the element that repeats under the root, and namespaced documents frequently come back with prefixed keys. Dump one record to array before writing any mapping code — the structure is rarely what you expect.; in European locales, and embed \r\n inside quoted fields. If the driver's from() does not expose delimiter/enclosure options, drop to league/csv directly for that format rather than fighting it.DB::transaction() builds a massive undo log, holds locks for minutes, and rolls back everything on row 499,999. Commit in batches.each() closure, record the failure, and continue. Emit a summary (imported / skipped / failed with row numbers) at the end.storage:link is irrelevant here; the PHP process needs read access to the real path. In containerized setups, this is usually a mounted volume — verify with is_readable() before streaming.transform() cheap. It runs per row; anything that does a query, an HTTP call, or instantiates a heavy service inside it will dominate your runtime. Batch external lookups outside the loop.composer why-not rodenastyle/stream-parser before bumping Laravel, and read the driver source rather than the README when behaviour surprises you — the README is thin relative to the format-specific edge cases.How can I help you explore Laravel packages today?